(Premier Reference Source) Zakariya Belkhamza, Zakariya Belkhamza, Syed Azizi Wafa - Measuring Organizational Information Systems Success - New Technologies and Practices (2012, IGI Global) PDF

Measuring Organizational
Information Systems
Success:
New Technologies and Practices
Zakariya Belkhamza
Universiti Malaysia Sabah, Malaysia
Syed Azizi Wafa

Managing Director: Lindsay Johnston
Senior Editorial Director: Heather Probst
Book Production Manager: Sean Woznicki
Development Manager: Joel Gamon
Development Editor: Hannah Abelbeck
Acquisitions Editor: Erika Gallagher
Typesetter: Milan Vracarich, Jr.
Cover Design: Nick Newcomer, Lisandro Gonzalez
Published in the United States of America by

Business Science Reference (an imprint of IGI Global)
701 E. Chocolate Avenue
Hershey PA 17033
Tel: 717-533-8845
Fax: 717-533-8661
E-mail: cust@igi-global.com
Web site: http://www.igi-global.com
Copyright © 2012 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in
any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher.
Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or
companies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark.
Library of Congress Cataloging-in-Publication Data

Measuring organization information systems success: new technologies and practices / Zakariya Belkhamza and Syed Azizi
Wafa, editors.
p. cm.
Includes bibliographical references and index.
Summary: “This book explores new approaches which may better effectively identify, explain, and improve IS assessment
in organizations”--Provided by publisher.
ISBN 978-1-4666-0170-3 (hardcover) -- ISBN 978-1-4666-0171-0 (ebook) -- ISBN 978-1-4666-0172-7 (print & perpetual
access) 1. Information technology--Evaluation. 2. Information technology--Management. I. Belkhamza, Zakariya, 1976-
II. Wafa, Syed Azizi.
HD30.2.M426 2012
658.4’038011--dc23
2011045300
British Cataloguing in Publication Data

A Cataloguing in Publication record for this book is available from the British Library.
All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the
authors, but not necessarily of the publisher.
Editorial Advisory Board
Stefan Smolnik, EBS Business School, Germany
Eldon Y. Li, California Polytechnic State University (Cal Poly), USA
G. Lawrence Sanders, The State University of New York at Buffalo, USA
Anita Lee-Post, University of Kentucky, USA
Alemayehu Molla, Royal Melbourne Institute of Technology RMIT, Australia
Christophe M. Elie-Dit-Cosaque, Université Paris-Dauphine, France
Ismail Ait Saadi, Swinburne University of Technology, Sarawak Campus, Malaysia
List of Reviewers
Stefan Smolnik, EBS Business School, Germany
Anita Lee-Post, University of Kentucky, USA
Alemayehu Molla, Royal Melbourne Institute of Technology RMIT, Australia
Syed Azizi Wafa, Universiti Malaysia Sabah, Malaysia
Christophe M. Elie-Dit-Cosaque, Université Paris-Dauphine, France
Rob J. Kusters, Eindhoven University of Technology, the Netherlands
Maurice Kügler, EBS Business School, Germany
Aileen Cater-Steel, University of Southern Queensland, Australia
Govindan Marthandan, Multimedia University, Malaysia
Anna Marie Balling Høstgaard, Aalborg University, Denmark
Chun Meng Tang, UCSI University, Malaysia
Tameur Nachef, Universiti Sains Malaysia, Malaysia
Ahmad Ghandour, University of Otago, New Zealand
Cees J. Gelderman, Open University of the Netherlands, the Netherlands
Cristina Lopez, University Pablo of Olavide, Spain
O. Tolga Pusatli, Cankaya University, Turkey
Simona Sternad, University of Maribor, Slovenia
Table of Contents
Foreword.............................................................................................................................................. xvi
Preface................................................................................................................................................... xx
Acknowledgment............................................................................................................................... xxiv
Section 1
Information Systems Success: Theories and Models
Chapter 1
A Symbiotic Model for Information Systems Success Determination.................................................... 1
Kieren Jamieson, CQUniversity, Australia
Chapter 2
Measuring Information Systems Success: A Comment on the Use of Perceptions............................... 23
Cees J. Gelderman, Open University of The Netherlands, The Netherlands
Rob J. Kusters, Open University of The Netherlands, The Netherlands
Chapter 3
Information Systems for Organizational Effectiveness Model: A Rationale Alignment....................... 39
Chapter 4
Information Systems Success: A Review from a Bibliometric Analysis Focus..................................... 62
Hugo Martinez, Universidad Industrial de Santander, Colombia
Luis Becerra, Universidad Industrial de Santander, Colombia
Jaime Camacho, Universidad Industrial de Santander, Colombia
Chapter 5
Theoretical Foundations for Information Systems Success in Small- and Medium-Sized
Enterprises.............................................................................................................................................. 80
Jan Devos, Ghent University, Belgium
Hendrik Van Landeghem, Ghent University, Belgium
Dirk Deschoolmeester, Ghent University, Belgium
Section 2
IT Projects, Service Management and Performance
Chapter 6
Measuring IT Service Management Performance: A Model Development......................................... 102
Francis Gacenga, University of Southern Queensland, Australia
Mark Toleman, University of Southern Queensland, Australia
Wui-Gee Tan, University of Southern Queensland, Australia
Chapter 7
Fuzzy Approach for Monitoring Projects Success in the IT/IS Industry............................................. 120
Jose L. Salmeron, University Pablo of Olavide, Spain
Section 3
Information Systems Maintenance and Development
Chapter 8
A Model to Assist the Maintenance vs. Replacement Decision in Information Systems.................... 137
Brian Regan, University of Newcastle, Australia
Chapter 9
A Steady-State Framework for Integrated Business Change and Information Systems
Development and Maintenance............................................................................................................ 158
Simon McGinnes, Trinity College Dublin, Ireland
Section 4
Enterprise Resource Planning Systems
Chapter 10
Enterprise Resource Planning Acceptance Model (ERPAM): Extended TAM for ERP Systems in
Operational Phase of ERP Lifecycle.................................................................................................... 179
Samo Bobek, University of Maribor, Slovenia
Chapter 11
Specific Factors for ERP-Success Measurement in Healthcare........................................................... 205
Stephan Kronbichler, UMIT, Austria
Herwig Ostermann, UMIT, Austria
Section 5
Websites and E-Commerce Systems
Chapter 12
Issues Facing Website Evaluation: Identifying a Gap......................................................................... 233
Kenneth R. Deans, University of Otago, New Zealand
George L. Benwell, University of Otago, New Zealand
Chapter 13
Website Evaluation Criteria: An Owner’s Perspective........................................................................ 253
Chapter 14
Antecedents of Children’s Satisfaction with Company Websites and their Links with
Brand Awareness.................................................................................................................................. 276
Lucie Sermanet, IESEG School of Management Lille-Paris, France
Frank Goethals, IESEG School of Management (LEM-CNRS) Lille-Paris, France
Andrea Carugati, IESEG School of Management (LEM-CNRS) Lille-Paris, France
Aurélie Leclercq-Vandelannoitte, IESEG School of Management (LEM-CNRS) Lille-Paris, France
Section 6
Information Systems Success Applications
Chapter 15
A Model to Measure E-Learning Systems Success............................................................................. 293
Ahmed Younis Alsabawy, University of Southern Queensland, Australia
Jeffrey Soar, University of Southern Queensland, Australia
Chapter 16
End-User Participation in Health IT Development: The EUPHIT Method......................................... 318
Chapter 17
The Development of a Model for Information Systems Security Success.......................................... 341
Kimberley D. Dunkerley, Nova Southeastern University, USA
Gurvirender Tejay, Nova Southeastern University, USA
Chapter 18
Evolutional Patterns of Intranet Applications: Organizational Issues and Information
Systems Success.................................................................................................................................. 367
Pietro Previtali, University of Pavia, Italy
Chapter 19
Designing an Information Systems Performance Management System: The Case of an Insurance
Corporation.......................................................................................................................................... 385
Angela Perego, SDA Bocconi School of Management, Italy
Compilation of References................................................................................................................ 403
About the Contributors..................................................................................................................... 468
Index.................................................................................................................................................... 477
Detailed Table of Contents
Foreword.............................................................................................................................................. xvi
Preface................................................................................................................................................... xx
Acknowledgment............................................................................................................................... xxiv
Section 1
Information Systems Success: Theories and Models
Chapter 1
A Symbiotic Model for Information Systems Success Determination.................................................... 1
Kieren Jamieson, CQUniversity, Australia
Traditional approaches to identifying and measuring Information Systems success or failure typically
suffer from two deficiencies. First, the measures are taken at a single point in time, usually shortly after
the system adoption with a focus on the implementation “success.” Second, the focus is purely on the
organisational net gain or loss. The organisation is treated as a single entity, and the Information System
itself is relegated to a subservient cog. The power relationship between the organisation and the Informa-
tion System is left unexplored: in other words, which entity controls the other? This chapter proposes and
demonstrates an alternate categorisation model that addresses both deficiencies. The model is applied to
a longitudinal study of an implementation of an enterprise system in order to both categorise and explain
the outcomes for the host organisation.
Chapter 2
Measuring Information Systems Success: A Comment on the Use of Perceptions............................... 23
Cees J. Gelderman, Open University of The Netherlands, The Netherlands
Rob J. Kusters, Open University of The Netherlands, The Netherlands
Information System success is difficult to measure directly. Because of the influence of non-controllable
variables, it actually seems to be impossible to directly compute or determine the contribution of Informa-
tion Systems to organizational performance, or to overall organizational effectiveness. As an alternative,
perception of system success is often used as a surrogate measure. However, this raises the question of
the validity of this surrogate measure. In this chapter, the authors describe a survey aiming to investi-
gate the validity of this surrogate measure. Results show that there is reason to doubt the usefulness and
validity of surrogate measures for objective system success.
Chapter 3
Information Systems for Organizational Effectiveness Model: A Rationale Alignment....................... 39
Despite the proposal of various Information Systems (IS) evaluation models and approaches, IS evalu-
ation has never been straightforward. There are issues and challenges in proving the business value of
IS. Adding to the difficulty, a vast number of measures have been employed conveniently for evalua-
tion purposes without going through a rigorous validation process. Recognising the complexity for IS
researchers, IS specialists, and business managers to agree on a common model for the evaluation of
IS business value, this chapter presents an empirically validated IS evaluation model, the IS for organi-
sational effectiveness (ISOE) model, for planning, designing, implementing, and appraising IS. There
also emerges a new theory, the Information System business value (ISBV) theory, from the ISOE model
to establish that IS business value is multifaceted and are observable in the form of improvements in
organisational effectiveness.
Chapter 4
Information Systems Success: A Review from a Bibliometric Analysis Focus..................................... 62
Hugo Martinez, Universidad Industrial de Santander, Colombia
Luis Becerra, Universidad Industrial de Santander, Colombia
Jaime Camacho, Universidad Industrial de Santander, Colombia
This chapter characterizes the publication activity of Information Systems success by data collected
from the Science Citation Index and Social Science Citation Index databases from ISI Web of Science
during 2001 to 2010. It provides insights into the research activities of Information Systems success
research and identifies patterns, tendencies, or regularities existing in the literature related to authors,
journals, institutes, and countries. A co-ocurrence analysis of keywords is made to indicate the core
themes research areas in the literature and new emerging topics. It is found that the IS success literature
has a positive growth rate, and it is likely to continue with this tendency in the future. However, the
analysis shows the need to set up a common language framework that serves as a guide to researchers
to develop a most mature body of knowledge.
Chapter 5
Theoretical Foundations for Information Systems Success in Small- and Medium-Sized
Enterprises.............................................................................................................................................. 80
Jan Devos, Ghent University, Belgium
Hendrik Van Landeghem, Ghent University, Belgium
Dirk Deschoolmeester, Ghent University, Belgium
Small and medium-sized enterprises (SMEs) adopt Information Technology (IT) and Information Sys-
tems (IS) in order to achieve business goals and obtain net benefits. However, adopting IT/IS into an
existing organizational structure is a complex and risky task. Many investments in IT/IS, outsourced as
well as in-sourced, never fully reach the intended objectives and are therefore considered as not being
successful. In this chapter, the authors focused on IS success in small and medium sized-enterprises
(SMEs) in order to find theoretical foundations. They explain four well-known theories, often used in
IS research, which constitute the basics of their thinking. These theories are the technology acceptance
model (TAM), the theory of planned behavior (TPB), the DeLone & McLean IS success model (D&M),
and the transaction cost economy (TCE) model. The authors weaved the constructs of these theories
into a compound framework that delivers explanatory and predicting power for the successful adoption
of IT/IS in SMEs. In order to validate the framework, the authors examined the extent to which the
theoretical model could provide support for the Cobit framework, often used by practitioners as an IT
governance framework, and also suitable for SMEs. Findings show that the framework offers surpris-
ing coherence and proposes a strong theoretical foundation for the normative directions of the methods
used in Cobit by IT practitioners.
Section 2
IT Projects, Service Management and Performance
Chapter 6
Measuring IT Service Management Performance: A Model Development......................................... 102
Francis Gacenga, University of Southern Queensland, Australia
Mark Toleman, University of Southern Queensland, Australia
Wui-Gee Tan, University of Southern Queensland, Australia
Prompted by the realisation that IT is now seen as a service, with a customer focus and process orientation,
the authors propose a model to measure IT service management (ITSM) performance. Measuring ITSM
performance will enable organisations to demonstrate the benefit from their investment. The model is
based on a systematic literature review that progressed from considering the general areas of organisa-
tion performance measurement to examining commonly used performance metrics. Although there are
a number of studies on ITSM implementation, only a few considered the performance measurement of
ITSM. A structured method for the design of the model was adopted through a three-level analysis. A
comparison of existing performance measurement frameworks was first made to identify those that are
suitable for ITSM and that would facilitate communication between the business and IT function. This
was done using appropriate dimensions from past work of various performance measurement researchers.
The frameworks were then classified along these dimensions to identify their completeness, eliminate
unnecessary dimensions, and identify the natural dimensions for ITSM.
Chapter 7
Fuzzy Approach for Monitoring Projects Success in the IT/IS Industry............................................. 120
Jose L. Salmeron, University Pablo of Olavide, Spain
There are many uncertainties that can influence the success of Information Technology (IT) and Informa-
tion Systems (IS) projects. These are characterized to be highly complex and risky, among other issues.
These features explain the high rate of failures in this kind of projects. So, if practitioners want to pre-
vent undesired outcomes in their IT/IS projects, they have to continuously manage the risks existing in
them. In this way, practitioners should monitor risks impacts on IT/IS projects success. However, current
methods used for it, have several limitations that can be overcome by employing artificial intelligence
techniques. Based on the fuzzy theory, this chapter proposes the use of fuzzy approaches to model risks
effects on IT/IS projects success measures. Its applicability is presented through an illustrative case. The
findings highlight that the method proposed give project managers insights into the causes of failure or
delay of their IT/IS projects, in order to develop effective strategies.
Section 3
Information Systems Maintenance and Development
Chapter 8
A Model to Assist the Maintenance vs. Replacement Decision in Information Systems.................... 137
Brian Regan, University of Newcastle, Australia
An extensive literature review shows that Information Systems (IS) are changed and eventually replaced
by substitutes under the influence of productivity, popularity, and specialisation of IS along with as-
sociated available support facilities, maintenance activities, failures, and user feedback. This chapter
compiles those factors emerged from software engineering, IS, software quality assurance measurements,
and computer science literature. A final product is a proposed model bringing those factors together as
they are suspected to raise a need for taking the decision of evaluating change requests that may lead to
a further maintenance or replace the IS. While keeping in mind that limitations on advanced testing ex-
ist, an expected service of such a model may help decision makers to explain maintenance/replacement
decision of IS/component in a more itemized manner, hence diminish overburden pressure of experience
responsibility on them.
Chapter 9
A Steady-State Framework for Integrated Business Change and Information Systems
Development and Maintenance............................................................................................................ 158
Simon McGinnes, Trinity College Dublin, Ireland
Success models often treat Information Systems (IS) as static. Yet most IS evolve continuously, and most
development effort occurs during the so-called maintenance phase. For an IS to succeed, its evolutionary
process must also remain successful. Unfortunately many IS projects fail, particularly when outsourced.
This chapter argues that the practice of managing IS work in project form may itself be implicated in
IS failure. The project model is critically examined, identifying mismatches with the reality of IS work
as a component of business change. The results suggest that merely trying harder to make projects suc-
ceed may be ineffective. An alternative framework for “steady state” development is proposed, which
characterises IS work as evolutionary and inseparable from its context of business change, providing a
blueprint for IS development without the need for projects, and offering improved chances of success
when “big bang” project management would otherwise be the only option.
Section 4
Enterprise Resource Planning Systems
Chapter 10
Enterprise Resource Planning Acceptance Model (ERPAM): Extended TAM for ERP Systems in
Operational Phase of ERP Lifecycle.................................................................................................... 179
Samo Bobek, University of Maribor, Slovenia
Enterprise resource planning (ERP) systems have been implemented in most organizations for few years.
But most of the organizations cannot really expose promised benefits of ERP systems. One of the reasons
might be ERP users who do not accepted and use ERP system properly. In IT/IS literature organizational
users have been exposed as important factor, which has influence on IT/IS acceptance and usage. Tech-
nology acceptance model (TAM) has been most widely used model for researching user acceptance and
usage of IT/IS. While this research is not the first attempt to apply TAM to ERP context, the authors
of this chaptre aim to make more contributions to the topic. First, they focus on the ERP system use in
routine (mature) stage, and because of that, they use construct extended use instead of actual use. In the
latest research of ERP system usage, the relationship between work compatibility and usefulness has
been examined. New relationships between work compatibility and attitude toward using ERP system
have been added. In all TAM studies regarding ERP context, a small number of external factors have
been researched. The groups of external factors that have influence on ERP extended usage have been
researched. The proposed model has been empirically tested using data collected from a survey of 293
ERP users in 44 organizations across country.
Chapter 11
Specific Factors for ERP-Success Measurement in Healthcare........................................................... 205
Stephan Kronbichler, UMIT, Austria
Herwig Ostermann, UMIT, Austria
There are different success factors that influence the outcome of ERP-projects. Each industry has cer-
tain requirements on success measurement, where other industries, in general, do not. In healthcare for
example the legal constraints are different from other branches. Each success measurement model has
a specific aim and covers specific needs. It measures different success dimensions and can be used at
different stages of ERP-projects. Some of the models consider different phases within ERP-projects
others can be used to determine overall ERP-success after the implementation of an IS. One important
factor for the use of such models is if the models covers branch specific needs in a way that allows the
researchers to allocate meaningful results and recommendations for actions. This chapter investigates
the specific needs on ERP-success measurement of the healthcare industry and the fit of the existing
success measurement models for this use case.
Section 5
Websites and E-Commerce Systems
Chapter 12
Issues Facing Website Evaluation: Identifying a Gap......................................................................... 233
As business organisations have become more reliant on Information Technology in achieving success,
Information Systems have become essential. Business organisations now use websites as part of their
Information Systems as a medium for communication and transactions between the business and their
customers. A better understanding of how to evaluate a website is necessary. This chapter explores web-
site evaluation and recognises the current challenges facing website evaluation. It begins by identifying
the type of website the current study is focussed on, namely the e-commerce website. This is followed
by reviewing the literature on website evaluative approaches and anecdotally described issues with the
existing performance measures. Three perspectives were identified when evaluating websites, user, de-
signer, and owner perspectives. While the user and the designer perspectives are well advanced in the
literature, there is a relative dearth of scholarly studies that address the owners’ needs. The provision
of such a perspective may enhance an owner’s ability to increase returns and benefits from their online
activity. Such gap need to be filled, a call from the authors.
Chapter 13
Website Evaluation Criteria: An Owner’s Perspective........................................................................ 253
There is often a need for business organisations to evaluate their current and potential website in order
to maximize the payoffs from website investments. Current evaluative approaches for the performance
of e-commerce websites do not adequately address owners’ concerns regarding the payoffs from their
e-commerce investment. This chapter establishes criteria to evaluate e-commerce websites based on an
owner’s perception rather than the customer’s perception. Drawing upon theories of communication,
resource based view and process oriented approach, an evaluation framework of three dimensions is
developed. The three dimensions are: website offer, usage, and payoff. These three dimensions are used
to explain the performance of a website, culminating in a website evaluation model. Each dimension is
a business process of the website that the organisation needs to monitor.
Chapter 14
Antecedents of Children’s Satisfaction with Company Websites and their Links with
Brand Awareness.................................................................................................................................. 276
Lucie Sermanet, IESEG School of Management Lille-Paris, France
Frank Goethals, IESEG School of Management (LEM-CNRS) Lille-Paris, France
Andrea Carugati, IESEG School of Management (LEM-CNRS) Lille-Paris, France
Aurélie Leclercq-Vandelannoitte, IESEG School of Management (LEM-CNRS) Lille-Paris, France
This chapter focuses on children’s satisfaction with the use of commercial websites. The authors address
two relevant gaps in extant literature: children as users of Information Technology and the concept of
Information System success itself. Children’s use of IT needs focus since this age group has been largely
neglected by extant IS research. The concept of success needs further research because the research results
on this dependent variable have been controversial, and there is still no full agreement on the antecedents
of success. In this chapter, the authors therefore contextualize the classic Information Systems success
model of DeLone and McLean to IS success with children. They carried out a survey of a sample of 76
children aged 11 to 14, located in various cities in France. The results show that children get higher brand
awareness if they are more satisfied with the kids website of the brand. The chapter finds the satisfaction
with the site depends upon the personalization that is possible and the presence of dynamic elements on
the site. Characters and animations that help navigating the site are also appreciated. A good presenta-
tion of the site’s rules could not be shown to be a significant antecedent of satisfaction. Qualitative data
that was gathered suggests that children are not really aware of the link between symbols they see on
the site and products that are sold.
Section 6
Information Systems Success Applications
Chapter 15
A Model to Measure E-Learning Systems Success............................................................................. 293
Ahmed Younis Alsabawy, University of Southern Queensland, Australia
Jeffrey Soar, University of Southern Queensland, Australia
E-learning involves adopting and exploiting the potential of new, advanced Information Technology
in development and delivery of education. In spite of a rapid growth in the e-learning field there still
exists a range of issues facing the stakeholders of e-learning systems. One of the key issues is how to
measure e-learning system success. Although considerable attention has been paid to the Information
Systems success issue, there remain arguments about the factors which are most effective for measur-
ing Information System success. The issue of measuring Information System success has an impact on
evaluating e-learning systems success. This chapter aims to fill this void by proposing an evaluation
methodology model to assess e-learning systems success. The contribution of this study is the proposed
model to evaluate the success of e-learning systems. The model is based on a thorough review of the
e-learning success literature and existing Information Systems success models.
Chapter 16
End-User Participation in Health IT Development: The EUPHIT Method......................................... 318
Despite there being extensive cumulative knowledge and many experiences about factors that contribute
to health Information Technology (HIT) success, lessons are yet to be learned as many HIT developments
still face a number of problems - many of them of an organizational nature. This chapter presents a new
method - the EUPHIT method – for studying and understanding one of the most crucial organizational
success factors in HIT development: end-user participation. The method was developed and used for
the first time throughout a research study of an EHR planning process in a Danish region. It has proved
effective in disclosing the interactions that occur between the different social groups involved in HIT
development, and in understanding the underlying reasons for these. This allows HIT project manage-
ment to explore new avenues during the development process in order to support, facilitate, and improve
real end-user participation.
Chapter 17
The Development of a Model for Information Systems Security Success.......................................... 341
Kimberley D. Dunkerley, Nova Southeastern University, USA
Gurvirender Tejay, Nova Southeastern University, USA
Information security has received a great deal of attention from a number of researchers. However, there
has been little research aimed at understanding the dimensions critical for the success of organizational
information security programs. This chapter considers a large body of information security literature and
organizes the research based on their findings. This taxonomy is used to develop a parsimonious model
for information security success within organizations. Also, the utility of the proposed model within the
contexts of government and healthcare is considered.
Chapter 18
Evolutional Patterns of Intranet Applications: Organizational Issues and Information
Systems Success.................................................................................................................................. 367
Pietro Previtali, University of Pavia, Italy
The objective of this chapter is to propose a theoretical examination, strengthened by an empiric survey
of intranet evolutional patterns and the neologism that designates a communication system, for access
to and searching of business information based on Internet technologies. This chapter analyses intranet
applications and functionalities in order to classify them according to a taxonomy that allows us to dis-
tinguish between an institutional intranet, a knowledge management intranet, and an operating intranet.
The main research hypothesis is the existence of an intranet life cycle, as an evolutional model starting
from an institutional intranet that moves to a knowledge management intranet and then to an operating
one. This last one is considered as a proxy for successful IS implementation. To substantiate the above-
mentioned hypothesis an empirical study was conducted among 110 large Italian corporations, with a
response rate of 66% (73 corporations). The method used was a survey conducted during the months
of March, April, and May 2010, applying a mix of random sampling (randomly selected interviewees
from the directory) and “snowball” sampling (contacting interviewees through leads). The results show
how, basically, companies approach intranet implementation processes in an incremental way, which
begins with the integration of the basic functionalities as “communication channel,” “service platform,”
or “document management.”
Chapter 19
Designing an Information Systems Performance Management System: The Case of an Insurance
Corporation.......................................................................................................................................... 385
Angela Perego, SDA Bocconi School of Management, Italy
Contribution of Information Systems (IS) to business has been widely debated among both business
scholars and practitioners. Even though a consistent body of literature has examined the problem over
a time frame of more than 20 years, and a plethora of theoretical contributions has been produced, the
issue of evaluating IS effectiveness remains unresolved. Starting from the assumption that real-world
experiences differ from theoretical explications, and with the intent to contribute to IS Performance
Management field bringing evidences from the reality, this chapter describes and discusses the design
of an IS performance management system implemented by an insurance corporation.
Compilation of References................................................................................................................ 403
About the Contributors..................................................................................................................... 468
Index.................................................................................................................................................... 477
xvi
Foreword
As early as during the first International Conference on Information Systems, Peter Keen introduced his
perspective on the key challenges of our discipline. In doing so, he emphasized the lack of a theoretical
basis in Information Systems (IS) research and raised the question of what its dependent variable should
be. Surrogates, such as user satisfaction or hours of usage, would mislead researchers and evade the key
issue of theory (Keen, 1980). Motivated by his call for clarification, many researchers have tried to
identify the factors contributing to IS success.
Among them, William H. DeLone and Ephraim R. McLean introduced their take in the early 1990s.
In what we perceive to be one of the most exciting contributions to the IS success discourse, DeLone
and McLean set out to organize and synthesize the diverse studies, theoretical accounts, and experiences
towards a concise and consistent model of IS success. Thus, the (first) IS success model (DeLone &
McLean, 1992) was born. Since its introduction, the D&M IS success model has created a huge response
in our discipline’s literature. The fact that the 1992 article of DeLone and McLean was found to be the
single most heavily cited article in the IS literature (Lowry et al., 2007), serves as impressive evidence.
Since its initial publication, quite a few topical domains have been investigated using the D&M model
of IS success. These include enterprise systems (Gable et al., 2003), knowledge management systems
(Kulkarni et al., 2007), decision support systems (Bharati & Chaudhury, 2004), Web-based systems
(Garrity et al., 2005), and data warehouses (Wixom & Watson, 2001) – amongst others.
However, and luckily not unusual for the scientific discourse, such groundbreaking work also stimu-
lated a controversy among IS researchers about the model’s tenets. Motivated by DeLone and McLean’s
call for further development and validation of their model, many researchers have attempted to extend or
respecify the original model. A number of researchers have even claimed that the D&M IS success model
is incomplete. For example, Peter Seddon (Seddon et al., 1999; Seddon, 1997; Seddon & Kiew, 1994)
can perhaps been seen as one of the more critical contributors to the discourse surrounding D&M’s IS
success model. Overall, authors engaging in the discussion following the initial publication of the model
suggest that more dimensions should be included in the model or present alternative success models.
Yet others focus on the application and validation of the model.
As a result, DeLone and McLean revised their work ten years after its initial publication. Rumor
has it that, at first, the follow-up piece found it difficult to gain the attention the initial article attracted.
However, when the updated version (DeLone & McLean, 2003) was published eventually, it offered an
interesting overview and integration of the discourse described above and proofed to be an important
collection of partially contradicting evidence, findings confirming the model’s underlying thought, and
new facets reflecting the constant change of the IS phenomenon (Urbach & Müller, 2011).
xvii
As of today, the discussion surrounding the IS success model still continues and the review published
by Petter et al., (2008) draws our attention to some of the issues that still remain to be investigated with
the theoretical lens the model affords us. Among the most prominent issues illustrated by what can
probably be referred to as the IS success model’s second revision, the low number of studies on IS suc-
cess on an organizational level probably stands out. Petter et al. (2008) point out that we currently lack
sufficient data to support the model’s proposition from an organizational point of view and encourage
future contributors to extend this area of the literature.
In a more recent review, Urbach et al., (2009) explore the current state of IS success research by
analyzing and classifying recent empirical articles with regard to their theoretical foundation, research
approach, and research design. The results show that the dominant research analyzes the impact that
a specific type of information system has by means of users’ evaluations obtained from surveys and
structural equation modeling. For future research, the authors suggest considering means to increase the
relevance of research in the field of IS success without compromising its rigor.
Looking at the model’s history, as well as the topics still subject to investigation today, we come
to the conclusion that research based on the IS success model is still as relevant as it was in the early
1980s. Perhaps even more so, as the insights and understanding we have gained through three decades of
research in this area now allow us to take a refined, comprehensive, and detailed look at the antecedents,
processes, and consequences that constitute IS success.
But it is not just the model’s content and ongoing discourse that makes DeLone and McLean’s
contribution as exciting as we initially suggested. Through its popularity, DeLone and McLean’s work
also managed to address another of Keen’s key challenges to the IS discipline: the lack of a cumulative
tradition in IS research. In this vein, their model probably serves as an important example of the ongo-
ing theoretical maturation of our discipline and might serve as one of the most prominent examples of
building a cumulative tradition of theoretical knowledge in the field. In doing so, its history can also
help motivate others to engage in a critical, constructive, and cumulative discourse on the concepts at the
heart of our research endeavors, perhaps even beyond IS success. From our point of view, the legitimacy
of the IS field in comparison to its neighboring disciplines (Frank, 2006; Lyytinen & King, 2004), the
search for domain identity (Benbasat, 2001; Benbasat & Zmud, 2003), and the “race for credibility”
in the scientific discourse (Weber, 1997) are good reasons why the IS success model’s history should
perhaps inspire and guide the field’s strive for new insights.
It is in this context that we greatly appreciate contributions that diligently review our current knowl-
edge; both critically and carefully evaluating what we already seem to know and advancing our knowl-
edge beyond its current boundaries. It is this ongoing effort that “Measuring Organizational Information
Systems Success: New Technologies and Practices” and its chapters are contributing to. Based on a mix
of review, conceptual, and theoretical pieces, the authors and editors provide extensions and refinements
that help us extend what we already know about success in various domains of the IS discipline.
Benjamin Müller
University of Mannheim, Germany
Nils Urbach
EBS Universität für Wirtschaft und Recht, Germany
xviii
Benjamin Müller is an Assistant Professor at the University of Mannheim, Germany. He received his Doctorate from the EBS
Business School in Wiesbaden, Germany. He studied Business Administration and Information Systems at both EBS Business
School and at Georgia State University, Atlanta, and holds graduate degrees from both institutions. Benjamin is currently
researching value-augmented enterprise systems. He and his group at the Chair of Information Systems IV of the University of
Mannheim are particularly interested in the organizational impacts of Enterprise Systems and their contribution to IT’s busi-
ness value. His work has been published in the Journal of Management Information Systems (JMIS) as well as in proceedings
of key international conferences. Beyond his research, Benjamin has worked as a consultant in the area of IT strategy and IT
benchmarking and gathered practical experience with corporations in the US and Europe.
Nils Urbach is Research Director at EBS Business School in Wiesbaden, Germany. He is also Head of the Strategic IT Man-
agement Competence Center at the Institute of Research on Information Systems (IRIS). He received his Doctorate from this
institution, and holds a Diploma in Information Systems from the University of Paderborn. In 2008, he stayed at the University
of Pittsburgh as Visiting Scholar at the Katz Graduate School of Business. Complementary to his academic work, he is a Man-
aging Consultant with Horváth & Partners in Stuttgart. He has also worked as a consultant with Accenture in Frankfurt. Nils
Urbach has been working in the fields of information management and collaborative Information Systems for several years.
In his current research, he focuses on IS success, IT governance, and IT outsourcing. His work has been published in several
journals such as the Journal of Strategic Information Systems (JSIS), Journal of Information Technology Theory and Applica-
tion (JITTA), Business Process Management Journal (BPMJ), and Business & Information Systems Engineering (BISE), as
well as in several international conference proceedings.
REFERENCES
Benbasat, I. (2001). Editorial notes. Information Systems Research, 12(2), iii–iv.

Benbasat, I., & Zmud, R. W. (2003). The identity crisis within the IS discipline: Defining and commu-
nicating the discipline’s core properties. Management Information Systems Quarterly, 27(2), 183–194.
Bharati, P., & Chaudhury, A. (2004). An empirical investigation of decision-making satisfaction in
Web-based decision support systems. Decision Support Systems, 37(2), 187–197. doi:10.1016/S0167-
9236(03)00006-X
DeLone, W. H., & McLean, E. R. (1992). Information Systems success: The quest for the dependent
variable. Information Systems Research, 3(1), 60–95. doi:10.1287/isre.3.1.60
DeLone, W. H., & McLean, E. R. (2003). The Delone and Mclean model of Information Systems Suc-
cess: A ten-year update. Journal of Management Information Systems, 19(4), 9–30.
Frank, U. (2006). Towards a pluralistic conception of research methods in Information Systems research.
Essen, Germany: Institut für Informatik und Wirtschaftsinformatik (ICB).
Gable, G., Sedera, D., & Chan, T. (2003). Enterprise systems success: A measurement model. In 24th
International Conference on Information Systems (ICIS 03), Seattle, Washington.
Garrity, E. J., Glassberg, B., Kim, Y. J., Sanders, G. L., & Shin, S. K. (2005). An experimental investiga-
tion of Web-based Information Systems success in the context of electronic commerce. Decision Support
Systems, 39(3), 485–503. doi:10.1016/j.dss.2004.06.015
xix
Keen, P. G. W. (1980). Reference disciplines and a cumulative tradition. In 1st International Conference
on Information Systems (ICIS 80), Philadelphia, PA, (pp. 9-18).
Kulkarni, U. R., Ravindran, S., & Freeze, R. (2007). A knowledge management success model: Theoreti-
cal development and empirical validation. Journal of Management Information Systems, 23(3), 309–347.
doi:10.2753/MIS0742-1222230311
Lowry, P. B., Karuga, G. G., & Richardson, V. J. (2007). Assessing leading institutions, faculty, and
articles in premier Information Systems research journals. Communications of the Association for In-
formation Systems, 20(16).
Lyytinen, K., & King, J. L. (2004). Nothing at the center? - Academic legitimacy in the Information
Systems field. Journal of the Association for Information Systems, 5(6), 220–246.
Petter, S., DeLone, W., & McLean, E. (2008). Measuring Information Systems success: Models, dimen-
sions, measures, and interrelationships. European Journal of Information Systems, 17(3), 236–263.
doi:10.1057/ejis.2008.15
Seddon, P., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions of Information Systems
success. Communications of the Association for Information Systems, 2(20), 1–61.
Seddon, P. B. (1997). A respecification and extension of the Delone and Mclean Model of IS success.
Information Systems Research, 8(3), 240–253. doi:10.1287/isre.8.3.240
Seddon, P. B., & Kiew, M.-Y. (1994). A partial test and development of the Delone and Mclean model
of IS success. In 15th International Conference on Information Systems (ICIS 94), Vancouver, Canada,
(pp. 99-110).
Urbach, N., & Müller, B. (2011). The updated Delone and Mclean model of Information Systems suc-
cess . In Dwivedi, Y. K., Wade, M., & Schneberger, S. L. (Eds.), Information Systems theory: Explaining
and predicting our digital society (Vol. 1, pp. 1–18). New York, NY: Springer. doi:10.1007/978-1-4419-
6108-2_1
Urbach, N., Smolnik, S., & Riempp, G. (2009). The state of research on Information Systems success –
A review of existing multidimensional approaches. Business & Information Systems Engineering, 1(4),
315–325. doi:10.1007/s12599-009-0059-y
Weber, R. (1997). Ontological foundations of Information Systems. Melbourne, Australia: Coopers &
Lybrand.
Wixom, B. H., & Watson, H. J. (2001). An empirical investigation of the factors affecting data warehous-
ing success. Management Information Systems Quarterly, 25(1), 17–41. doi:10.2307/3250957
xx
Preface
The tendency for IT decision-makers to ignore low probability but high-impact risks to project plans is
often an overlooked issue to be taken seriously into account. This notion of “black swan blindness” as
coined by Flyvbjerg and Budzier (2011) provides what we can consider a beginning of a new chapter
in the evaluation of organizational IT projects. IT managers tend not to see these black swans assuming
that they’re rare. In fact, they happen to the extent that IT managers often blindly run into them without
being able to identify or prevent them. This new concept in the IT risk management comes from the
examination of 1,471 IT projects carried out by Bent Flyvbjerg and Alexander Budzier of Oxford Uni-
versity and McKinsey. They found out that large IT projects are on average 27% over budget and take
55% longer to complete than planned. The study also reported that IT projects are 20 times more likely
to fail than other business projects, with one in six IT projects run out of control with average cost over-
runs of 200% (Budzier & Flyvbjerg, 2011). This means that failed IT projects are too frequent and
failure can have major consequences. These results presage to IT managers that poorly conceived initia-
tives, those that are excessively complex, and those that are poorly managed or coordinated can be fatal
to both IT managers and organization.
Not only IT projects which are actually reported to fail, many IT practitioners however, anticipate
their projects will fail. According to Geneca recent survey, about 75% of IT executives lack of confidence
in IT project success, admitting that their projects are either always or usually “doomed right from the
start,” with 27% of them feel always this way. The survey also revealed that challenges reflect difficulty
in defining project success (Geneca LLC, 2011).
These two main issues, the IT project failure and the IT practitioners’ anticipation of failure, con-
tribute to advocate the ongoing debate in the academia on the information systems success measure-
ment. Although this research stream has proliferated and nearly become a standalone stream within the
information systems research field, more research is still needed to identify factors that contribute to
information systems effectiveness and evaluation. To date, a number of information systems success
models have been introduced. However, the scope and approach of these studies has little consensus on
the assessment of information systems success, which urge for more exploration to better identify and
understand the various measurements of these models.
This book is divided into six sections, which cover various aspects of Information Systems success.
The first section, “Information Systems Success: Theories and Models” contains five chapters presenting
new models and frameworks of the information systems success. The first chapter, “A Symbiotic Model
for Information Systems Success Determination” by Kieren Jamieson addresses the issues of whether
an information system has been a success or failure for the organization at a given point of time, notic-
ing the deficiencies of the traditional approaches of information systems success measures. Adapting
xxi
biological relationship descriptions, a symbiotic model for the classification of information systems
success and failure is proposed and used to evaluate the success of an information systems implementa-
tion by examining the results of a ten year longitudinal study in to an organization’s enterprise system
(ES) implementation.
The second chapter, “Measuring Information Systems Success: a Comment on the Use of Perceptions”
by Cees J. Gelderman and Rob J. Kusters investigates another issue on the information systems success,
the perception of information systems as a surrogate measure, in order to determine the contribution
of information systems to the overall organizational effectiveness. In doing so, Gelderman and Kusters
developed and tested measurements to assess the validity of perceived system success as a surrogate
measure for actual system success.
In the third chapter, “Information Systems for Organizational Effectiveness Model: A Rationale
Alignment,” Govindan Marthandan and Chun Meng Tang expand our knowledge on information sys-
tems evaluation by introducing and validating the information systems for organizational effectiveness
model. The model suggests that IS business value will be evident in the organization when there is a
clear distinction between the various dimensions of organizational effectiveness, and the recognition of
positive IS improvements within the individual dimensions of organizational effectiveness.
The fourth chapter entitled ” Information Systems Success: A Review from a Bibliometric Analysis
Focus” by Hugo Martinez, Luis Becerra, and Jaime Camacho presents and characterizes the publication
activity of information systems success by analyzing data collected from the Science Citation Index
and Social Science Citation Index databases from ISI Web of Science from 2001 to 2010. The findings
suggest that information systems success literature has a positive growth rate and it is likely to continue
with this tendency in the future. This chapter also propose to set up a common language framework that
serves as a guide to researchers to develop a most mature body of knowledge.
The last chapter of this section, “Theoretical Foundations for Information Systems Success in Small
and Medium-sized Enterprises” by Jan Devos, Hendrik Van Landeghem, and Dirk Deschoolmeester
investigates four theories, namely technology acceptance model (TAM), theory of planned behaviour
(TPB), the DeLone & McLean IS success model (D&M) and transaction cost economy (TCE) model
in order to find theoretical foundations for information systems success in small and medium sized
enterprise. This resulted to the introduction of a compound framework that delivers explanatory and
predicting power for the successful adoption of IT/IS in SMEs.
The second section “IT Projects, Service Management and Performance” consists of two chapters.
In the first chapter “Measuring IT Service Management Performance: A Model Development,” Francis
Gacenga, Aileen Cater-Steel, Mark Toleman, and Wui-Gee Tan propose a model to measure IT service
management (ITSM) performance based on a systematic literature review of the general areas of organi-
sation performance measurement and the commonly used performance metrics. The authors have taken
a holistic view in the development of the model, by integrating various approaches such as the balanced
scorecard (BSC), broad economic perspectives and service oriented ITSM.
The second chapter in this section “Fuzzy Approach for Monitoring Projects Success in the IT/IS
Industry” by Jose L. Salmeron and Cristina Lopez proposes monitoring risks effects on IT/IS projects
success measures using the Fuzzy approach. It presents its applicability through an illustrative case.
The method proposed give project managers insights into the causes of failure or delay of their IT/IS
projects in order to develop effective strategies.
In the third section, “Information Systems Maintenance and Development,” two chapters are presented.
The first chapter, “A Model to Assist the Maintenance vs Replacement Decision in Information Systems”
xxii
by O. Tolga Pusatli and Brian Regan extensively reviews and compiles factors emerged from software
engineering, software quality assurance measurements and computer science literature, and proposes a
model to help decision makers to explain maintenance and replacement decision of IS/component in a
more itemized manner hence diminish overburden pressure of experience responsibility on them.
The second chapter, “A Steady-State Framework for Integrated Business Change and Information
Systems Development and Maintenance” by Simon McGinnes investigates the information systems
development during the maintenance phase, and proposes an alternative framework for “steady state”
development which characterises IS work as evolutionary and inseparable from its context of business
change, providing a blueprint for IS development without the need for projects, and offering improved
chances of success when “big bang” project management would otherwise be the only option.
Section four, “Enterprise Resource Planning Systems” comes with two chapters. The first chapter,
“Enterprise Resource Planning Acceptance Model (ERPAM): Extended TAM for ERP Systems in The
Operational Phase of ERP Lifecycle” by Simona Sternad and Samo Bobek applies technology accep-
tance model in the enterprise resource planning context to examine groups of external factors which
have impact on actual ERP system use.
The second chapter, “Specific Factors for Enterprise Resource Planning Success Measurement in
Healthcare” by Stephan Kronbichler and Herwig Ostermann discusses the conceptual design of pos-
sibilities of how ERP projects can be more successful when considering critical success factors and
success measurement models which were identified through reviewing the literature and incorporating
practical experiences in the healthcare industry.
The fifth section, “Websites and E-commerce Systems” contains three chapters. The first chapter,
“Website Evaluation: Issues and Challenges,” Ahmad Ghandour, Kenneth R. Deans, and George L.
Benwell explore website evaluation and recognises the current challenges facing website evaluation.
From reviewing the literature, this chapter identifies three perspectives when evaluating websites: user,
designer and owner perspectives. While the user and the designer perspectives are well advanced in the
literature, there is a relative dearth of scholarly studies that address the owners’ needs, in which authors
studied in their second chapter of this section, “Website Evaluation Criteria: an Owner’s Perspective”
in which they establish criteria to evaluate e-commerce websites based on an owner’s perception rather
than the customer’s perception, which lead to the development of an evaluation framework of three
dimensions of website offer, usage and payoff.
In the third chapter, “Antecedents of Children’s Satisfaction with Company Websites and Their Links
with Brand Awareness” by Lucie Sermanet, Frank Goethals, Andrea Carugati, and Aurélie Leclercq-
Vandelannoitte focuses on children’s satisfaction with the use of commercial websites. The chapter
contextualizes the classic information systems success model of DeLone and McLean (1992, 2003) to
information systems success with children in order to study children as users of information technology.
The last section of the book, “Information Systems Success Applications” consists of five chapters
covering various applications of information systems success in practice. The first chapter of this section,
“A Model to Measure E-Learning Systems Success” by Ahmed Younis Alsabawy, Aileen Cater-Steel, and
Jeffrey Soar argues about the factors which are most effective for measuring e-learning systems success
and proposes an evaluation methodology model to assess e-learning systems success.
The second chapter, “End-User Participation in Health IT Development: The EUPHIT Method” by
Anna Marie Balling Høstgaard presents a new method for studying and understanding the end-user par-
ticipation in the health IT development. The method was developed and used for the first time throughout
a research study of an EHR planning process in a Danish region.
xxiii
The third chapter, “The Development of a Model for Information Systems Security Success”by Kimber-
ley D. Dunkerley and Gurvirender Tejay develops a parsimonious model for information security success
within organizations, and consider its applicability within the contexts of government and healthcare.
The fourth chapter of this section, “Evolutional Patterns of Intranet Applications: Organizational
Issues and Information Systems Success” by Pietro Previtali proposes a theoretical and empirical exami-
nation of intranet evolutional patterns. It analyses intranet applications and functionalities and classify
them according to taxonomy in order to differentiate between an institutional intranet, a knowledge
management intranet and an operating intranet.
The last chapter of this section, “Designing an Information Systems Performance Management Sys-
tem: The Case of an Insurance Corporation” by Angela Perego describes and discusses the design of an
IS performance management system implemented by an Insurance Corporation, bringing a real-wolrd
experience and evidence that support the existing theoretical explanation of the information systems
performance evaluation.
The nineteen chapters of the book present a quality collection of various scholars presenting their
recent research on theories and practices which will benefit both academic researchers and practitioners.
As such, Measuring Organizational Information Systems Success: New Technologies and Practices
claims to be a definitive state-of-the-art collection and to prompt the future direction for IT managers
to identify applicable theories and practices in the evaluation of information systems and minimize the
IT failure in organization.
The editors hope this book will become instrumental in the expansion of the Information Systems
success stream and will promote the continued growth of the information systems development in general.
Zakariya Belkhamza
Syed Azizi Wafa

REFERENCES
Budzier, A., & Flyvbjerg, B. (2011). Double whammy – How ICT projects are fooled by randomness
and screwed by political intent. Saïd Business School working papers. University of Oxford.
Flyvbjerg, B., & Budzier, A. (2011). Why your IT project might be riskier than you think. Harvard
Business Review, 89(9), 23–25.
Geneca, L. L. C. (2011). Doomed from the start? Why a majority of business and IT teams anticipate
their software development projects will fail. Winter 2010/2011 Industry Survey. Retrieved October 26,
2011, from http://www.genecaresearchreports.com/GenecaSurveyReport.pdf
xxiv
Acknowledgment
Editing a book is something that requires a lot of collaboration and the support of many people who
have been involved in the development and production of this book.
First, we would like to thank all the authors who devoted so much of their time preparing and writing
their chapters, and responding to numerous comments and suggestions made from the reviewers and
ourselves. We are extremely indebted to the editorial advisory board members and reviewers for their
support, comments, and suggestions.
We are very grateful to Dr. Benjamin Müller and Dr. Nils Urbach for having accepted to write the fore-
word of this book. Our appreciations are also extended to Dr. Stefan Smolnik and his team at the EBS
Business School in Wiesbaden, Germany.
We also wish to gratefully acknowledge that great support and assistance of the excellent team at IGI
Global, especially Hannah Abelbeck who was always delighted to assist us throughout the process of
producing this book.
Finally, family is pivotal in encouraging us to carry on with this project, we are indebted to them for
their love, support, and understanding.
Zakariya Belkhamza
Syed Azizi Wafa

Section 1
Information Systems Success:
Theories and Models
1
Chapter 1
A Symbiotic Model for
Information Systems
Success Determination
Kieren Jamieson
CQUniversity, Australia
ABSTRACT
Traditional approaches to identifying and measuring Information Systems success or failure typically
suffer from two deficiencies. First, the measures are taken at a single point in time, usually shortly af-
ter the system adoption with a focus on the implementation “success.” Second, the focus is purely on
the organisational net gain or loss. The organisation is treated as a single entity, and the Information
System itself is relegated to a subservient cog. The power relationship between the organisation and
the Information System is left unexplored: in other words, which entity controls the other? This chapter
proposes and demonstrates an alternate categorisation model that addresses both deficiencies. The
model is applied to a longitudinal study of an implementation of an enterprise system in order to both
categorise and explain the outcomes for the host organisation.
INTRODUCTION 2002; Standish Group, 1995) with the desired

functionality (Mahaney & Lederer, 1999). How-
Traditional measures of information system (IS) ever, there appears to be very little literature that
success can be determined in a number of ways; discusses the long-term measures of the system’s
however, a general measure is that the system was success in terms of what it has really achieved and
implemented on time and on budget (IT Cortex, its ongoing consequences for the organisation. It
is the relationship between the information system
DOI: 10.4018/978-1-4666-0170-3.ch001 entity and the remainder of the organisation that
Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
A Symbiotic Model for Information Systems Success Determination
provides an alternate lens to the measurement of sational outcomes are perceived and measured is
success. However, due to the nature of disruptive performed. The background to the study is fol-
change, the true relationship may not be initially lowed by the justification and presentation of the
evident and may take time to become apparent. proposed model. The ethnographic nature of the
The issue of how to measure success or failure study is then outlined and the data collection and
is not trivial, as the success or failure of a system hermeneutic analysis techniques are described.
is seen as a matter of interpretation and that inter- The details of the organisation, enterprise sys-
pretation can change over time (Myers, 1994a). tem implementation and subsequent upgrades are
Thus, the success of a system is still largely per- then described. Following an analysis of how and
ception based and will depend on which actor is why organisational structures, staffing, processes
performing the assessment and when. While much and culture changed, the organisational outcomes
of the extant literature has focused on how and why are evaluated and analysed using conventional
information systems are implemented, long-term measures of success and then in the context of
organisational effects of IS implementations have the symbiotic model. The chapter concludes by
not been widely studied. Short-term studies have discussing future research directions.
given some indication of immediate implementa-
tion outcomes, however little is know about the
true costs, benefits and issues in organisations over BACKGROUND
substantially longer periods during the operational
lifespan of systems. While this chapter examines an enterprise system
This poses a problem: how can we determine implementation, the focus of the model is broader
if an information system has been a success or information system success and failure. Even so,
failure for the organisation at a given point of this section provides a brief discussion of enter-
time? This lack of a simple measurement model prise system technology, moving to a broader
not only impairs IS adoption decision-making, examination of organisational issues associated
but it also limits the understanding of what an with information system adoption and use. The
ongoing commitment to an information systems section concludes with a discussion of extant in-
entails. Worryingly, it ultimately means many formation system success and failure measures and
organisations are unable to answer the simple why there is a need to study theses systems over a
question: has an IS investment resulted in good longer period of time in order to be able to reach
or bad outcomes? The objective of this chapter conclusions about their organisational effects.
is to address this problem. By borrowing from
biological relationship descriptions, a symbiotic Enterprise Systems
model for the classification of IS success and
failure is proposed. The model is then used to Enterprise systems (ES), sometimes known as
evaluate the success of an IS implementation by enterprise resource planning systems (ERPs),
examining the results of a ten year longitudinal evolved from a long line of systems designed to aid
study in to an organisation’s enterprise system manufacturing and production. The aim of these
(ES) implementation. systems was to ensure availability of components
The paper begins with an examination of back- for smooth and consistent mass production. Enter-
ground literature concerning the specific reasons prise systems integrate management information
why organisations adopt enterprise systems and and processes, such as financial, manufacturing,
the expected and observed impacts. A review of distribution and human resources (Shang & Sed-
how more generalized information systems organi- don, 2002) as well as industry specific software
2
modules. Towards the mid 1990s, enterprise sys- the development cycle, or the difficulties involved
tems began to ‘come into vogue’ (Scott & Vessey, in assessing costs (Scheer & Habermann, 2000).
2002, p.78), delivering the promise of resource Yet, implementing enterprise systems has
planning focussing on the need of executives to disadvantages too. Significant investments in
control and strategically assess the position of hardware in terms of processing, storage and
the organisation in terms of financial, human networking requirements as well as software
resource and customer data. Many companies licensing training overheads are regarded as nega-
joined the rush to implement them, taking the tive aspects of enterprise investments (Scheer &
opportunity to reorganise their business practices. Habermann, 2000). Sheer and Habermann (2000,
By the late 1990s and onwards, enterprise systems p.57) also note that ‘the scale of business process
implementation had become more targeted, with reengineering and customisation tasks involved
organisations often adopting multiple Enterprise in the software implementation process are the
systems from different vendors to meet their needs. major reasons for ERP dissatisfaction’.
Late in the 1990’s, many large companies Implementing any information systems, espe-
faced with the prospect of Y2K chose to imple- cially an enterprise system, is a major undertaking
ment enterprise systems (Bagranoff & Brewer, given the size and complexity of the task. The
2003). However, this was not the only justifica- complexity of the implementation depends on
tion that saw the rise of the enterprise systems a number of operational and functional factors
during the 1990s. Organisational justifications including the number and type of modules de-
for enterprise system implementations have tra- ployed and by the strategy chosen to roll out the
ditionally centred on three inter-related factors. modules (for example, Big Bang, phased, pilot,
Firstly, cost savings are typically nominated as or some combination thereof) (Scott & Vessey,
being a key driver behind radical IS overhaul 2002). Moreover, organisational factors, such
(see for example Bagranoff and Brewer (2003), as culture and ability to change are indicators of
Lozinsky (1998)). Typically cost savings are the effort required to successfully implement the
predicted from reduced support costs in direct IS product (Jamieson, 2007; Scott & Vessey, 2002).
staff terms, efficiencies resulting from business Scheer and Habermann (2000) have noted that
process re-engineering (BPR) and standardising major enterprise system vendors such as Baan,
hardware between systems. Secondly, an oft-stated Peoplesoft (now Oracle), as well as SAP estimate
advantage of enterprise systems is their integrated, customers spend between three and seven times
standardised approach (Scott & Vessey, 2002) and more money on enterprise system implementa-
while the exact benefits to organisations from this tions compared to the initial software license
paradigm are not necessarily clear, there are broad purchase. Furthermore, Scheer and Habermann’s
gains from consistent look-and-feel, lack of data own research indicates that the ratio between
duplication and the reduced need for interfaces enterprise system implementation and software
between systems. However, it should be noted purchase costs is approximately five to one. More
that enterprise systems frequently fail to eliminate recent research gives examples of cases where
data duplication and the use of shadow systems implementation costs have exceeded ten times
(Behrens, 2009). Finally, organisations implement that of the initial purchase price and reinvestment
enterprise systems in order to reduce complexity of 500 percent of the initial cost within ten years
and uncertainty. Internal development is often in upgrade costs (Jamieson, 2007).
plagued by uncertainties, such as the selection While much effort has been focused on reduc-
of appropriate development tools, the duration of ing the high rate of enterprise system implementa-
tion failure by examining critical success factors
3
(CSFs) (Häkkinen & Himola, 2008), studies of the high levels of IS strategic alignment perform bet-
short, medium and long term effects of enterprise ter. Chan et al. (1997) also supported Baets (1992)
system implementations are few. Indeed, Shang and proposed that intended IS strategy (direction)
and Seddon (2002) noted that most studies are decisions should be checked for IS alignment.
snapshots and none offer a view of long term Alignment between information technology
benefits. Murphy and Simon (2002) confirmed the departments and other business units is also re-
difficulty in measuring enterprise system benefits ferred to as IS strategic fit (Chan & Huff, 1992).
and noted that: Organisational factors such as shared domain
knowledge between business and information
With ERP systems, success has been determined technology executives, existing levels of tech-
based on the organization’s acceptance of the nology implementation success, communication
changes that the system introduces. Publicized between business and information technology
implementation failures have been caused not executives and connections between business
because the system failed to perform as designed and technology planning processes are indicators
or desired cost savings targets could not be of alignment (Reich & Benbasat, 2000). Luft-
obtained, but as a result of a failure to change man and Brier (1999) found that other enablers
the organization’s culture or in business process of alignment were senior executive support for
redesign (BPR), organizational and strategic information systems, involvement in IS strategy
issues. Organizational and managerial classifi- development, IS understanding of business pur-
cation benefits are not only the most difficult to pose, business/IS partnerships, well prioritised IS
obtain but also the hardest to quantify (Murphy projects and strong IS leadership. Without these
& Simon, 2002, p.317). alignment indicators and enablers, the way in
which IS decision making was affected, which
This focus on organisational-strategic align- in turn, affected the perception of success and
ment and system success is noteworthy and forms failure of the system.
the pre-cursor to system success and failure. Information system decision-making is af-
fected by a lack of IS strategic fit in two ways.
Organisational Issues Firstly, if decision makers have a poor under-
standing of the needs of the organisation, then
As previously noted, alignment issues are a sig- misaligned technology selections, no matter how
nificant underlying cause of information system well implemented, can lead to negative outcomes.
failure. However, these alignment issues manifest Baets (1992) supported this by noting that attempt-
themselves at different times within the technol- ing IS systems alignment post-implementation is
ogy adoption, implementation and use life cycle. seldom a success. Secondly, decisions become
There is recognition that alignment issues, such politicised and focus is shifted away from achiev-
as organisational alignment, are critical for the ing optimal organisational solutions. In these
successful use of IS and that alignment is pivotal cases, decision-making occurs at higher levels of
in contributing to organisational profitability and the organisation where IS representation is limited
productivity (Martin, Gregor, & Hart, 2005). Mar- or lacking in power. In such instances, poorly
tin, Gregor and Hart (2005) examined enablers informed selection decisions can occur that can
within a contextual model and concluded the ‘soft’ result in poor technological alignment and lead to
aspects of IS alignment are important. Chan et al. implementation and ongoing support problems.
(1997) found that IS strategic alignment is a better Chan (2002) described a number of predic-
predictor of IS effectiveness and organisations with tors of IS strategic fit. Although specifically
4
addressing the issue of aligning the operation goals will be found at the executive levels. Chan
of IT departments with the wider organisation, (2002) noted that poor business-information sys-
the findings have relevance to both aspects of tem alignment can result in costly investments with
intra-organisational alignment. Importantly, trust, poor yields and returns. Chan (2002) described
credibility and faith were identified as alignment two cases in which this occurs. Firstly, where
facilitators. As discussed previously, decision business executives are unable to communicate
makers need to trust each other, but importantly, or articulate their technology needs to those
there has to be wider organisational trust in IS choosing and implementing information systems.
decision makers and IS departments. Informal Secondly, when technical staff lack the business
structures and relationships were important within vision and knowledge to make information sys-
organisations. In IS pre-implementation decision tems decisions with significant organisational
making, these allow decision makers to identify impacts. Chan (2002) also drew relationships
problems, gain an understanding of organisational between the informal networks, culture and trust
requirements and gather information. These social and alignment conditions. Findings relating to the
structures and ties consist of social networks, use of organisational alignment factors were also
communities of practice and unofficially agreed supported by Ba, Stallaert and Whinston’s (2001)
upon practices. Chan (2002) noted that a strong study which found that system success is related
organisational culture was a facilitator of the use to inbuilt incentives for the users to correctly use
of informal social structures. systems. This discouraged users not to cheat and
Another key indicator and predictor of suc- perform operations which distort processes or
cess is relationship alignment (Jamieson, 2007). data. They argued that systems need to match the
Relationship alignment is an assessment of how way organisations and users perform operations:
well the organisation adopting the information
system will be able to interact with the vendor A system incentive is aligned when a user’s
during and after the IS project implementation. dominant strategy and preferred user behaviour
Strong vendor relationships are essential for posi- correspond from an organisational perspective.
tive long-term outcomes. Without vendor support That is, the agent can still freely determine his
and strong relationships, implementations can own behaviour and use of the system, but the most
fail and post-implementation problems become rational action, i.e., the action that is in his best
difficult to resolve. A shared understanding of interest, coincides with the action that benefits
strategic and organisational goals is a condition the organisation the most (Ba et al., 2001 p. 227).
for relationship alignment. Moreover, confidence,
trust and existing relationships with the vendor Fan, Stallaert and Whinston (2003) supported
can provide indicators of the level of relationship these findings, noting the need to consider this
alignment. Vendor culture should also be closely aspect of alignment in supply chain investment
matched with the organisation, and vendor support decisions. The authors argued for the inclusion
is intrinsically dependent on inter-personal rela- of incentive systems as part of the design or con-
tions. However, vendor relationships need balance sideration of new IS. Bendoly and Jacobs (2004)
and there are risks associated with vendor-led or concurred and noted the importance of operational
vendor-dominated interactions. alignment in ERP systems with business processes
Chan and Huff (1992) noted that it should not for optimal organisational outcomes. Whyte and
be assumed that a conscious strategic awareness Bytheway (1996) also established a link between
exists amongst all actors in organisations. They outcome success and alignment. They found that
found that a greater awareness of organisational post implementation perception of system align-
5
ment with business objectives is a contributor to large IS implementation projects. Factors such
success. However, this is a perception and does as organisational size, project cost and project
not necessarily mean the system is effective in duration have an effect on project success (Ma-
delivering its intended functionality. Specific haney & Lederer, 1999; Standish Group, 1998).
factors associated with negative outcomes also In particular, the traditional problems of lack of
been noted in other studies as having relation- user involvement (OASIG, 1996; Standish Group,
ships with information systems failure, but the 1995), poor project management (Ainsworth,
actual measures of success and failure are still 1999; Ewusi-Mensah, 1997; Kippenberger, 2000),
problematic and subject to debate. resourcing, risk management (Standish Group,
1995; Whittaker, 1999), organisational expertise
Success and Failure and project size (Cannon, 1994) typically affect
enterprise system projects. Again, the focus on
It is important to understand why organisational achieving enterprise systems success has focused
outcomes are important in gauging the ultimate on preparatory and implementation issues, not the
success or failure of any information system long-term effect and usage of the system.
implementation. However, these can be measured In taking a more organisation impact focused
at a number points in time, but traditionally im- view, DeLone and McLean (1992 p. 61) noted
mediately after the information system has been that ‘there are nearly as many measures of suc-
implemented. As such, the success or failure of the cess as there are studies’. As such, an informa-
information system is frequently too often framed tion system may be successful in some criteria
by the implementation project itself, not the sys- and fail in others. The problem remains, as but
tem’s impact on and use within the organisation. Meijden (2003) points out, that the exact criteria
These implementation project focused measures that predict success or failure is unclear and that
of IS success can be determined in a number of it is likely that no single criterion can account
ways; however, a general measure is that the for success or failure of an information system.
system was implemented on time and on budget Furthermore, Meijden believes each evaluation
(IT Cortex, 2002; Standish Group, 1995) with the criterion must be measured in an appropriate way,
desired functionality (Mahaney & Lederer, 1999). but can all stakeholders agree on what is appro-
The 1997 KPMG survey discussed by Whittaker priate? DeLone and McLean (1992) proposed six
(1999) focused on what constituted an immedi- interdependent measurements of system success:
ate IS implementation failure. They deemed an system quality, information quality, use, user
IS project as having failed if it overran its budget satisfaction, individual impact and organisational
or schedule by thirty percent, or if the project impact. All of these factors should be considered
was cancelled or deferred due to non-delivery of when trying to measure success under the model
planned benefits. Mahaney and Lederer (1999) and that no single measure is intrinsically better
argued that there are degrees of failure and that than any other. While attempts have been made to
a project that overruns budget by five percent is refine and expand on their model (see for example
less of a failure than one that overruns by fifty Seddon et al. (1999)) and have included a further
percent. It is notable that much of the focus on refinement from DeLone and McLean (2003;
achieving information success has been linked to 2002) with the inclusion of net benefit, different
build and implementation factors, as there are a actors have different opinions as to what a benefit
lack of long-term success measures. is (DeLone & McLean, 2002) thus the success of
Enterprise system implementations also suf- a system is still largely perception based and will
fer from the traditional issues associated with depend on which actor is performing the assess-
6
ment (Middleton, 1995). As Myers (1994 b p. 65) outcomes and their ultimate negative or positive
stated, ‘information systems success is achieved effects. Without an understanding of the long-term
when an information system is perceived to be organisational effects of IS adoption, or a way
successful by the stakeholders and other observ- of using them to assess the ultimate success or
ers’. However, this definition is lacking given failure of an information system, organisations
that it would be unlikely for all stakeholders and are potentially making ill-informed IS investment
observers to agree on the perception of the success decisions.
of an IS project at any given time.
Thus in multi-stakeholder environments, the
notion of ‘‘success’’ or ‘‘failure’’ has become in- THEORETICAL MODEL
creasingly subjective. Connell and Young (2007)
found that studies of senior IS professionals One particular problem with assessing the “suc-
highlighted the distinction between technical and cess” or “failure” of an information system over
organizational issues. They found that the latter an extended period is that once embedded and op-
group were perceived by their respondents as erational, it is easy to assume that it is, or remains,
more important and that these results appeared organisationally beneficial. It is this assumption
independent of organisation type (Connell & of benefit that suggests separating the entities
Young, 2007). It is this focus on organisational of the information system and the organisation
outcomes that will be examined in this study. may provide an insight into which, if either, are
obtaining benefit. While entirely theoretical, this
Long Term versus Short structural separation permits the examination
Term Studies of the entity relationships using paradigms and
theories from other domains. In this instance, the
The literature is largely silent on the impacts of theoretical domain is biological science.
information systems, particularly enterprise sys- One way of examining the concept of benefit
tems, on organisations over any substantial periods can be best explained in the context of symbio-
of time, with the focus being on the immediate sis. From a biological perspective, symbiosis is
effects and outcomes, yet as noted, realisation and a long-term living arrangement between differ-
understanding of limitations and benefits change ent species resulting in positive (beneficial) and
over time. This chapter seeks to address this gap negative (unfavourable or harmful) associations
by examining outcomes in an organisation with (Encyclopaedia Britannica, 2011b). In the context
a substantially embedded system. of IS within organisations, these association clas-
This study does not focus on measuring benefits sifications depend on the effect each entity has
(as described in Shang and Seddon’s (2002) frame- on the other, be it beneficial, harmful or neutral.
work), but the long-term organisational effects of Drawing from biological descriptions, the four
an information system. This study also examines most common symbiotic relationships, mutualis-
perceptions, and perceptions of aspects of systems tic, commensal, parasitic and amensalistic, pro-
implementations are a matter of interpretation that vide useful explanations of common interactions
can change over time (Myers, 1994a). As Meijden seen between organisations and their information
(2003, p.241) notes ‘A system that is successful systems. As such, using the biological definitions
today may be considered a failure in a decade and determinations of benefit and harm, these
due to technical limitations or changed demands relationships has potential to classify systems as
and expectations’. Moreover, this chapter seeks successes or failures.
to explain the relationships with organisational
7
Mutualism is a symbiotic relationship whereby ever, if the quantity and timing of this investment
both organisms derive benefit (Ahmadjian & is dictated by the requirements of the technology,
Paracer, 2000). Thus, from the perspective of for example, patching, version upgrades to avoid
either entity, or viewed overall, the relationship obsolescence or to remain supported, then the or-
is a positive one. This is perhaps the assumed ganisation is at a distinct disadvantage. This type
default position on IS within organisations. It of investment regime would clearly cause harm
may be assumed that organisational investment in to an organisation, and the relationship between
information systems, both initially and continually, the entities classed as a failure.
adds value (benefit) to the technology, while the Finally, there is one form of symbiotic rela-
technology continues to deliver efficiencies or tionship that may be far more common between
competitive advantage to the organisation. This information systems and their host organisations
ongoing operation of the information systems than might initially be thought. Amensalism is a
would be perceived as a success. relationship between organisms in which one is
Commensalism is a symbiotic relationship inhibited or destroyed and the other is unaffected
whereby one organism benefits and the other is (Encyclopaedia Britannica, 2011a). This can be
not significantly harmed or helped (Ahmadjian best demonstrated in the relationship between an
& Paracer, 2000). Thus, from the perspective of information system and an organisation whereby
the entities, the relationship is either positive or the organisation can no longer afford to invest in
neutral. This poses a question for classification system in order to maintain it and meet the needs
purposes: can a neutral effect be classified as a of the system. In failing to invest, the information
success or failure? In other words, does the absence system remains in stasis while the organisation
or benefit indicate a failure or does the absence is unable to obtain the services it needs from the
of harm indicate success? One way to answer this system. An example would be being unable to
in a symbiotic model is to examine which entity dedicate resources to patching software in order
has the power in the relationship. In the case of to comply with changes to government legislation,
an information system that undergoes periodic such as a change to income tax rates. The reason
investment to maintain existing services, without for being unable to invest in the system could
delivering the organisation any extra efficiency be as simple as failing to keep the system up to
for the investment, then at an initial assessment, date, thus losing vendor support and requiring any
the organisation may seem at a disadvantage, thus changes to the system to be performed “in-house.”
the system may be seen as a failure. However, if The organisation may lack the expertise, corporate
this investment is planed and the timing is entirely knowledge or resources to make the required
within the control of the organisation, then no changes. Another example is an organisation that
harm is being done, however the organisation has made a long-term commitment to an informa-
does not greatly benefit: even so, this commensal tion system and has altered its processes, practices
relationship may be seen as a success. and workflows to comply with the technology. In
In contrast, a parasitic relationship is one this case, the organisation may not be able to re-
whereby one organism benefits while the other is main competitive with changing market demands
harmed (Ahmadjian & Paracer, 2000). Thus, from that the information system cannot address, yet
the perspective of the entities, the relationship is can not afford the massive investment to replace
either positive or negative. Again, in the case of the system: in such a case, the organisation may
an information system that undergoes periodic slowly contract and die. As such, this form or
investment in order to maintain existing services, relationship is a failure for the organisation.
the maintenance can be viewed as beneficial. How-
8
Table 1. Symbiotic success categorisations
Relationship Effect on Information System Effect on Organisation

Mutualism Benefit Benefit
The organisation invests resources into the Information System in order to derive efficiencies and/or com-
petitive advantages.
The organisation controls the timing and quantity of the resource allocation.
Commensalism Benefit Neutral
The organisation invests resources into the Information System in order to maintain the current services.
The organisation controls the timing and the quantity of the resource allocation.
Parasitism Benefit Harm
The organisation invests resources into the Information System in order to maintain current services. The
Information System dictates the timing and quantity of the required resource allocation.
Amensalism Neutral Harm
The organisation is unable to invest additional resources in the Information Systems order to maintain cur-
rent services AND meet the needs of the Information System.
The Information System dictates the timing and quantity of the required resource allocation. The lack of
investment in the Information System poses significant risk to the Organisation
Based on these biological categorisations of tive. Longitudinal case study evidence is used to
symbiotic relationships, an alternative model demonstrate an example of an information system
of information systems success measurement is that by conventional models is successful, but is
summarised in Table 1. a failure in the symbiotic model.
It is proposed that information systems can Ethnography is the science of describing a
only be categorised as successful if they form group or culture (Fetterman, 1998). Myers (1999,
mutualistic or commensalistic relationships with p.2) notes the relevance of this research method
the implementing or “host” organisations. Con- to observing organisational aspect of information
versely, failed information systems form para- systems:
sitic and amensalistic relationships with the imple-
menting organisations. This model is a more Ethnographic research is one of the most in-depth
effective in classifying success or failure over an research methods possible. Because the researcher
extended period of time as it provides simple test is at a research site for a long time - and sees
relating to control of expenditure of resources on what people are doing as well as what they say
the information system and the benefits to the they are doing – an ethnographer obtains a deep
organisation. The model also captures the power understanding of the people, the organization,
relationship between the entities in order to de- and the broader context within which they work.
termine which is dictating resource expenditure Ethnographic research is thus well suited to pro-
quantity and schedule. viding information systems researchers with rich
insights into the human, social, and organizational
aspects of information systems.
METHOD
While it has been acknowledged that ethno-
This study has used an ethnographic data gathering graphic studies are site and context specific, the
technique combined with hermeneutic analysis in value in such studies is not generalizable knowl-
order to construct and explain the derived narra- edge, rather the ability, as in single case studies, to
9
proffer a generalizable theory (Harvey & Myers, 1995). It assumes that the meaning of the text can
1995) and a source of rich data. be obscured by a number of factors. Tradition-
ally, these factors include context and ideology,
Data Gathering although in an organisational context they include
motivation, hidden agendas and power structures.
This study has been conducted over a ten-year These motivations and influences have to be identi-
period in an Australian higher educational insti- fied to see why the actions within the text occurred
tution between mid-1998 to mid 2008. Data was or what they really meant. In an organisational
collected throughout the period using a mixture context, critical hermeneutics focuses on the com-
of techniques. These included participant obser- munication between actors, particularly managers,
vations, semi-structured interviews with key aca- and is useful in understanding and analysing the
demic, IT and senior decision makers, corporate interactions between organisational culture and
emails and organisational documents including power (Phillips & Brown, 1993).
plans, RFPs and internal reports and project docu- In this study, a multi-parse analysis follow-
mentation. The author participated both as an IS ing Eisenhardt’s (1989) analysis guidelines was
professional working on pre-enterprise systems, performed. Firstly, a case ‘write-up’ (Eisenhardt,
as an IS professional implementing the enterprise 1989 p. 540) containing a combination of inter-
system and as an IS professional completing the view data, observations and documentation was
last significant upgrade in 2008, however, it should developed. The write-up contains direct quotes and
be noted that for most of the period of the study, was rich in raw information and contextual data.
the author was an observer/researcher. The author This was purely descriptive and did not contain
also worked as an academic user of the system linkages, propositions or identification of trends.
and as a specialised consultant tasked to review The purpose of the write-up was to establish a
an upgrade in 2006. The author had access to textual source for deeper recursive analysis. From
personnel ranging from senior decision makers the case write-up, a narrative, the findings of this
to general users and was party to some planning study, was developed. This presents the reader with
and decision making meetings. the rich contextual background and understand-
ing of the problems and in order to explain what
Analysis occurred over the ten-year period and justify the
categorisation within the symbiotic model.
In order to make sense of the large amount of
data gathered and the complex cause and effect
relationships, an interpretive critical hermeneutic PRE-IMPLEMENTATION SNAPSHOT
technique was employed. Myers (1994a, 1994b)
believes that hermeneutics, particularly critical This study examines an Australian Higher Edu-
hermeneutics, provides a valuable interpretive cation entity, hereafter ‘the organisation’. The
approach to researching IS implementations. organisation had traditionally maintained core
Hermeneutics is a method of analysis and inter- information systems for financial, human resource
pretation of text for the purpose of understanding and student management; with a centralised IS
and in some cases, exposing underlying meaning department that had evolved from a small team
(Byrne, 1998). into a large organisational unit. The IS department
Critical hermeneutics looks for the underlying handled most of the organisation’s IS needs. Be-
meaning, motivations and influences within the fore the implementation of the enterprise system,
text or narrative (Byrne, 1998; Harvey & Myers, the core information systems were separate and
10
loosely coupled. These consisted of a legacy, in- was little organisational experience with large
house developed student records system (SRS) and corporate systems or the implications of selecting
commercial finance and human resource systems and implementing them. An IS executive explained
IS implementations had had mixed success within that ‘Nobody outside [of the IS department] had
the organisation. While rollouts of off-the-shelf heard of [an] enterprise system’.
systems had been generally successful, projects In March 1999 after an evaluation of three
developed in-house had mixed outcomes. One IS systems, the organisation approved the purchase
professional attributed failures of internal projects of the enterprise system finance, human resource
to the ‘overestimation of the programmers’ abil- and student system modules costing approximately
ity or experience’ and the culture of ‘just in time’ one and a half million dollars. Following a detailed
with minimal resources within the IS department. five-week implementation planning study, it was
The organisation was aware as early as 1989 determined that the enterprise system was missing
that it needed to replace its student records system. functionality and that the implementation would
There were considerable pressures on the legacy be more costly and difficult than predicted. This
student records system as it was not designed meant that to achieve the functional outcomes, the
to meet the operational needs of a modern aca- organisation would need to modify the product. At
demic institution. The system suffered from the this point, an estimate of fourteen and a half million
constant level of change required, for example, dollars was budgeted for the implementation. This
the introduction of private, fee-paying students. budget concerned many in the organisation, as an
This, in combination with organisational growth Academic, who was part of the decision-making
and the need for more efficient systems, initiated team explained:
the need for organisational-wide information sys-
tems change. Apart from the imminent collapse Later, when the presentation went to Council it
of the existing SRS, three key justifications were turned out that it was going to be fifteen million
provided to replace the existing system: bucks or fourteen and a half million dollars and
… everybody just freaked out. And certainly whilst
1. Software vendor driven support for change I now understand that that should have been a
required by government legislation. reasonable expectation, at the time it seemed like
2. Cost savings generated by the reduction in it had gone from something that was going to be a
manual processing and the removal of three couple of hundred grand to fix to something that
hundred casual, seasonal and temporary staff was an enormous monetary project.
positions
3. The need for business process re-engineering The implementation was executed with the
(BPR) within the organisation and the belief assistance of an implementation partner and large
that an enterprise system would solve many numbers of external contractors. The project was
systems problems, including the need for separate to the existing IS department but main-
workflow. tained linkages through secondments and some
hardware/systems service provision. However,
In order to replace the existing system, a there was a high level of disengagement and
working party of senior executives was formed. resentment between the IS department and the
There was strong support for the integration of implementation project.
the organisation’s administrative systems and the It is not the purpose of this chapter to explore
re-engineering and development of better admin- the implementation process itself. It should be
istrative procedures and practices. However, there noted that a robust project methodology was used
11
and the project team were generally highly skilled. cycle. Higher numbers of IS staff were required
However, as a result of the planning study, pressure to maintain the system. By 2004, a team of three
from key stakeholders, including senior managers, analyst programmers had grown to over fifteen
dictated that the enterprise system would undergo programmers, functional analysts, database
extensive modifications and customisations as administrators and systems administrators. One
opposed to re-engineering business processes. IS Professional stated that ‘We need a lot more
people to look after it than we ever had before.
One programmer cannot cope and look after the
POST-IMPLEMENTATION PERIOD [enterprise system]’.This was a notable comment
given a key justification for the enterprise system
Functionally, the implementation project only adoption was the reduction in staff.
delivered the student and half of the finance Perhaps the most negative outcome was that
systems. Extensive work in the months after the the organisation had not realised the benefits
implementation was required to address functional from using enterprise system technology. Busi-
issues and implement remaining modules. It is ness processes had not been reformed nor had
estimated that this cost in excess of three million workflow been implemented. Much of the system
dollars. At this point, the perceptions surround- was simply not used. An IS Executive explained:
ing decision outcomes in the organisation were
mixed. However, the undisputed facts are that the Essentially what we’ve got are two components
project was completed over time, over budget and of an enterprise system that have a similar look
without all the anticipated functionality. and feel… They could have been separate best-
By 2004, interviewees were reporting some of-breed systems … the way we’re using them…
positive longer-term impacts on the organisation. I don’t think we’ve realised the benefits yet of an
One IS Professional felt the decision to adopt enterprise system.
the enterprise system had affected the way the
organisation approached its rollout and imple- Organisational support for the enterprise
mentation decisions. She stated that ‘We are system implementation was primarily limited
much, much more vigorous in how we put things to the executive levels. Most regular members
from a development environment to a production of the organisation were at best suspicious or at
environment. That’s been a big cultural change in worst openly hostile. One IS professional com-
the area I work in’. One senior Academic agreed, mented that:
believing that the way in which decisions were
approached in the organisation were changed and [The members of the organisation] were unsup-
that it had substantially altered the landscape of portive about the whole process of how the project
what could and could not be done in the organisa- went about implementing it.
tion. He believed that the outcomes from adopting
the enterprise system had strengthened internal One particular concern was value for money
process compliance and the ability to access spent. The IS Executive explained:
information - but at a cost to the organisation.
Interviewees also believed the system provided I think the formal [Senior] part of the organisa-
more stringent financial controls and enabled tion thinks it was a good decision. There are areas
timely student enrolments. within the organisation that probably think that
The adoption of the enterprise system led for the amount of money we spent, we didn’t get
to a perpetual enhancement and maintenance an appropriate return.
12
The cost element became a critical post- added a significant burden to the organisation’s
implementation factor once the senior executives budget given that allocations now have to be
realised how much it would take to support an regularly made for high-cost, high-risk projects.
enterprise system. One IS professional explained At issue is the organisation’s total dependence on
that the senior executives in the organisation were maintaining vendor support. Without following
‘flabbergasted by the post-implementation costs’. the vendor prescribed patching requirements and
She explained that at one point, the senior execu- upgrade path, the product would become unsup-
tives wanted an exit strategy prepared to rollback ported: given the need for frequent legislative
the enterprise system implementation, noting driven modifications, this is contrary to one of the
that ‘They wanted an exit strategy. They really initial justifications for the ES adoption.
wanted us to develop a strategy of us jettisoning
the software and going forward with another one’. Higher IS Resourcing Levels
However, due to the massive investment as well
as the lack of a viable alternative, the organisation Since the enterprise system implementation, the
maintained its commitment to the system. organisation has faced a number of challenges
concerning the resourcing required to maintain,
patch and upgrade the system on a regular basis.
FINDINGS: TEN YEARS ON Of particular note is the number of staff required
to support and maintain the enterprise system. By
At the final observation point in mid 2008, the or- 2008, IS support staff numbers had increased to
ganisation had undergone significant change since approximately thirty functional, development and
the inception of the enterprise system implemen- dedicated technical personnel from an original
tation in 1998. Moreover, since the initial period team of three. This is in direct contrast with a
post go-live, the enterprise system had undergone key justification for implementation: reduced IS
two major upgrades at considerable expense to the support costs.
organisation. The first, an upgrade to the finance
system between 2004 and 2005, cost $2.7 million Vendor Lock-In
and exceeded both time and budget. The second
upgrade, that of the student system that occurred Vendor lock-in is perhaps the most insidious out-
between early 2007 and mid 2008, at a cost of ap- come from the enterprise system implementation.
proximately $10 million, also exceeded both time The effect is that given the organisation’s signifi-
and budget. In 2011 at the time of writing, the next cant financial and human resource investments
significant upgrade is underway. Following are in the software, hardware, training and (albeit
some key organisational outcomes and impacts limited) BPR, any move to replace the system is
of the enterprise system investment. virtually impossible. The deliberate decision to
choose a single vendor with common development
Ongoing Costs tools, look-and-feel and integration is offset by
an inability to easily select an implement better
Excluding basic vendor license support costs and suited commercial software modules without
day-to-day production support requirements, it significant integration costs. This is evidenced
is estimated that approximately $13 million has by the continued use of the finance module and
been spent since 2001 on the enterprise system the reluctance to consider other systems. The
upgrades. This is substantial given that the original finance module was initially problematic and is
system cost approximately $20 million. This has arguably the weakest component of the system. It
13
replaced what was regarded as a “best-of-breed” modifications to the system in order to make it
system and somewhat ironically, the most likely conform to their processes. Even after upgrades,
replacement contender would be an updated ver- there is still evidence that the product is being bent
sion of the original system. However this is not to conform to the organisation, not visa-versa. In
possible given the tight integration between the a post-implementation upgrade review in 2006, it
student and finance modules. This problem has was found that “organisational process efficien-
been compounded by investment in integration of cies were not made.” In the 2007/2008 upgrade
other 3rd party modules such as the human resource project, little additional effort was made to affect
system. To remove the enterprise system would BPR, with the focus on fit-gap and modifying
also mean decoupling it with other subservient the product to fit organisational needs. It should
systems, and this poses significant financial and be noted that very few of the modifications are
logistical challenges. site specific or add competitive advantage: they
merely reflect an inability of the organisation to
Reduced Customisations grasp a basic understanding of enterprise system
use and best practice.
One of the more painful lessons learnt was that
the high level of initial customisations was un- Confidence in the IS Department
sustainable. Patching the product with vendor
updates became an expensive and time-consuming One significant positive outcome from the enter-
process whereby customisation had to be recoded prise system adoption has been the up-skilling
over any patched code. At one point, the organi- of the IS department and the inclusion of more
sation had reached patch paralysis whereby the formal project management, design and imple-
overall number of patches to be applied exceeded mentation methodologies. This has led to a skill
the time and resources available. Each system and decision-making maturity, and as noted in a
upgrade since go-live has focussed on reducing review of the most recent upgrade project, has in
customisation. For example, it was estimated that turn produced more successful IS projects. The
the finance system upgrade resulted in 50% of organisation now has increased confidence in the
the modifications added three years previously IS department and its future projects. However,
being removed. The more recent student system IS governance is still very poor. A review of the
upgrade also reduced modifications, however most recent upgrade project noted significant
because of entrenched beliefs and an inability and governance weaknesses, especially in budget
unwillingness to change business practices, many and planning oversight. During a recent project
new modifications were introduced: this will be management role, the author was a close observer
at an ongoing cost to the organisation over the of governance processes and noted the lack of
coming years. understanding surrounding the role, function and
implications of maintaining an enterprise system
Missing the Benefits from BPR in the organisation.
While a justification for the implementation was Reliance on Contractors

the opportunity for business process re-engineer-
ing, the initial decision to heavily customise the Due to the greater need for specialised develop-
enterprise system was counter-productive. While ment and functional resources, the organisation
some BPR was effected, a significant number of now has to rely on contracting staff to complete
business units demanded (and received) major upgrade projects. It should be noted that while a
14
full-time employee costs approximately $70,000 in order to fill this gap, a faculty-based “shadow
for one year, this is equivalent to less than 60 skilled system” was developed. This web-based system
contractor workdays. As noted, the organisation interfaced to the enterprise system and obtained
is committed to continuous upgrades in order to the necessary data academics required. As such,
maintain vendor support. The extensive reliance academics have felt little need to use the enterprise
and lack of internal expertise significantly adds system. This may change in the near future with
to the cost of such projects. moves to abolish the shadow system, which has
gained university-wide usage (Behrens, 2009).
System Reliability and Integration
A key motivation for the move to an enterprise SUCCESS OR FAILURE?

system was the need for a more robust, reli-
able student records system. Certainly, a more In this section, the system is examined at the ten-
functional and scalable system is now in place, year point using conventional measures of success
with regular vendor patches to match regulatory and then classified using the symbiotic model.
changes. However, as previously noted, much of
the functionality has been customised and requires Conventional Measures
considerable ongoing maintenance to support. The
most recent upgrades have seen to a transition to Various elements in the organisation had pro-
a highly scalable, highly available platform that vided differing prioritisations or justifications
has provided a greater degree of availability and for selecting the enterprise system that could be
stability. Moreover, the integration between the used as retrospective measures. While these do
Student and Finance modules is tight. not strictly map to DeLone and McLean’s (1992)
six interdependent measurements (system qual-
System Use and Uptake ity, information quality, use, user satisfaction,
individual impact and organisational impact), the
The system is widely used by administrative overall discussion of these factors in the context
staff and students within the organisation. Great of net benefit provides a conventional measure
emphasis has been placed on the self-service as- of systems success.
pects of the system that has enabled some reform Senior executives were focused on cost sav-
of business practices and genuine cost savings. ings and re-engineering business processes. This
However, as with many other enterprise systems, behaviour supports the findings of authors such
reporting is limited so has spawned the need for as Bagranoff and Brewer (2003). Users of the
a costly data warehousing and online reporting system were focussed on having a product that
system for senior managers and other staff requir- functioned and would service their basic needs.
ing data manipulation and extraction facilities. It Members of the IS department were concerned
is also worth noting that a key user community, with technical issues, vendor support and vendor
academics, have failed to take up the system. This presence. It should be noted that many of these
was primarily for two reasons. Firstly, the original justifications were disconnected and unrelated
student records system provided a web interface to to the abilities of the specific product selected.
academics for student and course enquiries. This From a cost perspective, there is some irony
system was dropped with the implementation of the that by spending a considerable amount on the
enterprise system and replaced with limited func- enterprise system project, only later has the or-
tionality. Secondly, shortly after the go-live, and ganisation realised how much value for money the
15
IS department had given them in the past. This systems justifications (Scott & Vessey, 2002),
is especially important considering the number was ultimately never achieved given the use of a
of staff and ongoing budget required to support, separate human resource system.
patch and upgrade the product. The level of com- Ten years after go-live, after internal restruc-
plexity and lack of control of when upgrades and tures, with new staff and in the light of increased
patching need to occur poses medium to long term competition in the sector, some in the organisa-
planning problems for the organisation. This runs tion still maintained negative perceptions of the
contrary to the arguments that enterprise systems enterprise system implementation. This perception
are implemented in order to reduce uncertainty centred on cost and the ongoing drain on the organi-
and complexity (Scheer & Habermann, 2000). sation’s resources. However, from a customer and
While the organisation now has a stable, reli- corporate perspective, the system has delivered a
able system and has also gained a more experi- robust solution that enables self-service. Although
enced, it has also resisted change. The organisation the initial implementation was not on time, on
has resisted BPR and has instead customised the budget or with all the desired functionality, thus
enterprise system product. The organisation has a failure, most in the organisation described the
also failed to understand the complexity of the system ten years on as a success given the benefits
enterprise system product and the need for good the organisation had reaped.
IS governance.
After the initial change, most users generally Symbiotic Classification
accepted the system. It delivered the required
functionality and continues to do so. The notable While the points discussed previously give an
exception is the academic user cohort, the major- overall impression of system success as a func-
ity of whom have no access to the system, which tion of net benefit, does the same apply in the
is unable to provide them with functionality they symbiotic model? In order to answer this, the
require anyway. As discussed earlier, the com- relationship between the information system
petitive advantages and efficiencies that would (enterprise system) and the organisation needs to
be achieved by using modules from best-of-breed be separated and each entity examined for signs
systems rather than those the existing enterprise of harm and benefit. As part of this analysis, the
system cannot occur because of the reliance on power relationship needs to be clarified: in other
a single “integrated” system. This single system words, which entity exerts the greater control.
reliance poses risks for the organisation in the From the perspective of the information sys-
longer term. tem, the relationship is clearly beneficial. The
From a technical perspective, the system system is subject to regular significant invest-
has been largely a success and has fulfilled the ment projects, including upgrades and patching.
requirement of a vendor-supported product. The Moreover it has a large support structure of IT
implementation has given the organisation more staff that tend to its requirements. There are no
robust IS project management an implementation signs that there is harm being inflicted on the
practices and change management. However, it information system.
should again be noted this has come at a resource With respect to the organisation, it has already
cost to the organisation: the number of staff been noted that it continues to invest in both sup-
required to support the system has increased porting resources and on one-off upgrade projects
dramatically and there is strong vendor lock-in. in order to maintain the system. However, do
Another point to note was that the desire for a those investments give additional efficiencies or
standard integrated system, common in enterprise competitive advantages? While these have been
16
sometimes cited as reasons for upgrades, BPR organisation is not only committed financially,
resulting in efficiencies is consistently absent. but also in terms of the vendor, technology (both
No particular competitive advantage is gained hardware and software) and the skill-base of the
by these upgrades, and as noted previously, the technical and administrative staff. This potential
impetus for the investments is purely to remain inability to change products could easily lead to
vendor-supported and meet legislative require- a situation where the organisation cannot gain
ments. This neutral effect rules out a mutualism efficiencies or competitive advantage by imple-
relationship. menting other best-of-breed systems or modules.
However, before classifying the relationship as Ultimately; this could inhibit and erode organi-
commensalism, the presence of harm needs to be sational growth.
excluded. While the negative aspects of the system
have been previously detailed, are they actually
doing the organisation harm? It is here that the FUTURE RESEARCH DIRECTIONS
power relationship must be used to determine
harm. Given the information system dictates the While the symbiotic model has been demonstrated
schedule for patching and upgrades, and that the as an alternate method of examining the role of
resource requirement to complete this is entirely information systems within organisations, limited
dependant on the complexity of the work required, examples of the symbiotic classification of success
the power is with the information system entity, and failure have been described. As such, there
and as such, is harmful to the organisation. It is has not been the opportunity within this chapter
also noteworthy that the dependency on the system to explore other organisational IS implementation,
also manifests as a dependency on the vendor, i.e. especially where the power relationship is reversed
vendor lock-in. It could then be argued that the between entities within classifications: such cases
enterprise system has formed a parasitic relation- would be useful to test and refine the model. It
ship with the organisation, where the information would also be useful to spend more time examin-
system is the beneficiary. As such, this informa- ing the organisation over multiple points in time
tion system implementation would be classified to see how, if and why the categorisation of suc-
as a failure. cess changed. For example, in the case examined,
Although amensalism can be excluded on the there is evidence to suggest that the relationship
basis that the information system still continues was mutualistic after the initial implementation;
to benefit and that the organisation is still able however, this has changed to parasitic over time.
to invest in it, there are warning signs evident. The mechanism for this change provides useful
The organisation is heavily committed to the research opportunities.
enterprise system in a number of ways that make
it increasingly more and more difficult to end the
relationship. Most significantly, the organisation CONCLUSION
is highly, if not intrinsically reliant on the system
and vendor support: this reliance has led to a con- This chapter has proposed an alternative informa-
stant drain on the organisation’s resources. The tion systems success classification model. Like
requirement to meet vendor driven patching and many success and failure frameworks, the model
upgrade requirements, as well as the significant suffers from the element of subjective assessment.
investment in staff training and technology makes However, its value lies in giving an alternative
it virtually impossible to consider or justify change, way in which to examine the role of information
for example, invest in bespoke technology. The systems in organisations, acknowledging that the
17
success or failure of such systems may change Bagranoff, N. A., & Brewer, P. C. (2003). PMB in-
in time. In examining the ten-year period after vestments: An enterprise system implementation.
an implementation of an enterprise system at a Journal of Information Systems, 17(1), 85–106.
higher education organisation, the organisational doi:10.2308/jis.2003.17.1.85
outcomes have been used as conventional success
Behrens, S. (2009). Shadow systems: The good,
and failure measures. While some positive effects
the bad and the ugly. Communications of the ACM,
have been observed, the organisation has under-
52(2), 124–129. doi:10.1145/1461928.1461960
gone technological change without achieving all
the desired benefits. Even so, by conventional Bendoly, E., & Jacobs, F. R. (2004). ERP archi-
standards, this information system implementa- tecture/operational alignment for order-processing
tion could be classified as successful. However, performance. International Journal of Opera-
re-examining the outcomes and using the sym- tions & Production Management, 24(1), 99–117.
biotic model to identify harmful and beneficial doi:10.1108/01443570410511013
effects provided another perspective. It became
Byrne, M. M. (1998). Hermeneutics 101. Paper
clear that the information system had formed
presented at the Qualitative Interest Group.
a parasitic relationship with the organisation,
where it had the power and gained the benefit. Cannon, J. A. (1994). Why IT applications succeed
At the conclusion of the study, it was clear that or fail: The interaction of technical and organiza-
the organisation has been left entirely dependant tional factors. Industrial and Commercial Training,
on the system with little chance, or advantage in 26(1), 10–15. doi:10.1108/00197859410051226
considering other best of breed modules. This is
Chan, Y. E. (2002). Why haven’t we mastered
a cautionary lesson for organisations considering,
alignment? The importance of the informal or-
what is effectively, an organisational partnership
ganizational structure. MIS Quarterly Executive,
with any large information system.
2(2), 97–112.
Chan, Y. E., & Huff, S. L. (1992). Strategy: An
REFERENCES Information Systems research perspective. The
Journal of Strategic Information Systems, 1(14),
Ahmadjian, V., & Paracer, S. (2000). Symbiosis: 191–204. doi:10.1016/0963-8687(92)90035-U
An introduction to biological associations. Oxford,
UK: Oxford University Press. Chan, Y. E., Huff, S. L., Barclay, D. W., & Cope-
land, D. G. (1997). Business strategic orientation,
Ainsworth, H. (1999). Why do projects break information systems strategic orientation, and stra-
down? Charter, 70(February), 46. tegic alignment. Information Systems Research,
Ba, S., Stallaert, J., & Whinston, A. B. (2001). Re- 8(2), 125–150. doi:10.1287/isre.8.2.125
search commentary: Introducing a third dimension Connell, N. A. D., & Young, T. P. (2007). Evaluat-
in information systems design - The case for in- ing healthcare information systems through an “en-
centive alignment. Information Systems Research, terprise” perspective. Information & Management,
12(3), 225–239. doi:10.1287/isre.12.3.225.9712 44(4), 433–440. doi:10.1016/j.im.2007.04.002
Baets, W. (1992). Aligning information sys- Cortex, I. T. (2002). Success assessment: Where is
tems with business strategy. The Journal of the limit between success and failure? Retrieved
Strategic Information Systems, 1(4), 205–213. 18 October, 2002, from http://www.it-cortex.
doi:10.1016/0963-8687(92)90036-V com/Failure.htm
18
DeLone, W. H., & McLean, E. R. (1992). Informa- Häkkinen, L., & Himola, O.-P. (2008). ERP
tion systems success: The quest for the dependent evaluation during the shakedown phase: Les-
variable. Information Systems Research, 3(1), sons from an after-sales division. Information
60–95. doi:10.1287/isre.3.1.60 Systems Journal, 18, 73–100. doi:10.1111/j.1365-
2575.2007.00261.x
DeLone, W. H., & McLean, E. R. (2002). Informa-
tion systems success revisited. Paper presented at Harvey, L. J., & Myers, M. D. (1995). Scholar-
the 35th Annual Hawaii International Conference ship and practice: The contribution of ethno-
on Systems Sciences, Hawaii. graphic research methods to bridging the gap.
Information Technology & People, 8(3), 13–27.
Delone, W. H., & McLean, E. R. (2003). The
doi:10.1108/09593849510098244
DeLone and McLean model of Information
Systems success: A Ten-year update. Journal of Jamieson, K. (2007). Information systems deci-
Management Information Systems, 19(4), 9–30. sion making: Factors affecting decision makers
and outcomes. Central Queensland University,
Eisenhardt, K. M. (1989). Building theories from
Rockhampton.
case study research. Academy of Management
Review, 14(4), 532–550. Kippenberger, T. (2000). Management’s role
in project failure. The Antidote, 5(4), 30–33.
Encyclopaedia Britannica. (2011a). Amensal-
doi:10.1108/EUM0000000006793
ism. Retrieved 5th May, 2011, 2011, from http://
www.britannica.com/EBchecked/topic/19211/ Lozinsky, S. (1998). Enterprise wide software
amensalism solutions: Integration practices and strategies.
Reading, MA: Addison-Wesley.
Encyclopaedia Britannica. (2011b). Symbiosis.
Retrieved 5th May, 2011, 2011, from http:// Luftman, J., & Brier, T. (1999). Achieving and
www.britannica.com/EBchecked/topic/577677/ sustaining business-IT alignment. California
symbiosis Management Review, 42(1), 109–122.
Ewusi-Mensah, K. (1997). Critical issues in Mahaney, R. C., & Lederer, A. L. (1999). Runaway
abandoned information systems development Information Systems projects and escalating com-
projects. Communications of the ACM, 40(9), mitment. Paper presented at the Special Interest
74–80. doi:10.1145/260750.260775 Group on Computer Personnel Research Annual
Conference, New Orleans, Lousiana, USA.
Fan, M., Stallaert, J., & Whinston, A. B. (2003).
Decentralized mechanism design for supply chain Martin, N., Gregor, S., & Hart, D. (2005). The
organisations using an auction market. Informa- social dimensions of business and IS/IT alignment:
tion Systems Research, 14(1), 1–22. doi:10.1287/ Case studies of six public-sector organisations.
isre.14.1.1.14763 Australian Accounting Review, 15(3), 28–38.
Fetterman, D. M. (1998). Ethnography: Step-by- Meijden, M. J. d., Tange, H. J., Troost, J., &
step (applied social research methods) (2nd ed., Hasman, A. (2003). Determinants of success of
Vol. 17). Sage Publications. inpatient clinical information systems: A literature
review. Journal of the American Medical Infor-
matics Association, 10(3), 235–243. doi:10.1197/
jamia.M1094
19
Middleton, C. (1995, August 25 - 27). A tale of Scott, J., E., & Vessey, I. (2002). Managing
two systems? Success and failure in a single In- risks in enterprise systems implementations.
formation System implementation. Paper presented Communications of the ACM, 45(4), 74–81.
at the Association for Information Systems 1995 doi:10.1145/505248.505249
Inaugural Conference, Pittsburgh, PA.
Seddon, P. B., Staples, S., Patnayakuni, R., &
Murphy, K. E., & Simon, S. J. (2002). Intangible Bowtell, M. (1999). Dimensions of Information
benefits valuation in ERP projects. Information Systems success. Communications of the AIS,
Systems Journal, 12, 301–320. doi:10.1046/ 2(20).
j.1365-2575.2002.00131.x
Shang, S., & Seddon, P. B. (2002). Assessing and
Myers, M. D. (1994a). A disaster for everyone to managing the benefits of enterprise systems: The
see: An interpretive analysis of a failed IS proj- business managers perspective. Information Sys-
ect. Accounting. Management and Information tems Journal, 12, 271–299. doi:10.1046/j.1365-
Technology, 4(4), 185–201. doi:10.1016/0959- 2575.2002.00132.x
8022(94)90022-1
Standish Group. (1995). The CHAOS report. Re-
Myers, M. D. (1994b). Dialectical hermeneutics: trieved 12 April, 2005, from http://www.standish-
A theoretical framework for the implementation group.com/sample_research/chaos_1994_1.php
of information systems. Information Systems
Standish Group. (1998). CHAOS: A recipe for
Journal, 5, 51–70. doi:10.1111/j.1365-2575.1995.
success. Retrieved 12 April, 2005, from http://
tb00089.x
www.standishgroup.com/sample_research/
Myers, M. D. (1999). Investigating Information chaos1998.pdf
Systems with ethnographic research. Communica-
Whittaker, B. (1999). What went wrong? Unsuc-
tions of the Association for Information Systems,
cessful Information Technology projects. Infor-
2(23), 1–20.
mation Management & Computer Security, 7(1),
OASIG. (1996). The performance of information 23–30. doi:10.1108/09685229910255160
technology and the role of human and organiza-
Whyte, G., & Bytheway, A. (1996). Factors affect-
tional factors (Report). Sheffield, UK: University
ing Information Systems’ success. International
of Sheffieldo. Document Number.
Journal of Service Industry Management, 7(7),
Phillips, N., & Brown, J. L. (1993). Analyzing com- 74–93. doi:10.1108/09564239610109429
munication in and around organizations: A critical
hermeneutic approach. Academy of Management
Journal, 36(6), 1547. doi:10.2307/256821
ADDITIONAL READING
Reich, B. H., & Benbasat, I. (2000). Factors
that influence the social dimension of alignment Al-Mashari, M., & Al-Mudimigh, A. (2003).
between business and information technology ERP implementation: Lessons from a case study.
objectives. Management Information Systems Information Technology & People, 16(1), 21–33.
Quarterly, 24(1), 81–113. doi:10.2307/3250980 doi:10.1108/09593840310463005
Scheer, A.-W., & Habermann, F. (2000). Making Avila, V. L. (1995). Biology: Investigating life on
ERP a success. Communications of the ACM, Earth. Boston, MA: Jones and Bartlett.
43(4), 57–61. doi:10.1145/332051.332073
20
Behrens, S., Jamieson, K., Jones, D., & Cranston, Jamiesion, K. (2007). Unearthing the hidden: The
M. (2005). Predicting systems success using the use of a critical hermeneutic case study approach.
Technology Acceptance Model: A case study. Paper Paper presented at the QualIT 2007 - Qualitative
presented at the 16th Australian Conference on Research: From the Margins to the Mainstream.
Information Systems, Manly, Australia.
Jamieson, K., & Hyland, P. (2004). IS failure:
Boland, R. J. Jr. (1991). Information System use as Just too much information? Paper presented at
a hermeneutic process. In Nissen, H.-E., Klein, H. the 8th World Multi-Conference on Systemics,
K., & Hirschheim, R. (Eds.), Information Systems Cybernetics and Informatics, Orlando, U.S.A.
research: Contemporary approaches & emergent
Jones, D., Cranston, M., Behrens, S., & Jamieson,
trends (pp. 439–458). Copenhagen, Denmark:
K. (2005). What makes ICT implementation suc-
Elsevier Science Publishers B.V.
cessful: A case study of online assignment submis-
Charette, R. N. (2005). Why software fails? Spec- sion. Paper presented at the Open and Distance
trum, IEEE, 42, 42-49. Learning Association of Australia Conference,
Adelaide, Australia.
Cozijnsen, A. J., Vrakking, W. J., & Ijzerloo, M.
v. (2000). Success and failure of 50 innovation Laudon, K. C., & Laudon, J. P. (1996). Manage-
projects in Dutch companies. European Jour- ment Information Systems. New Jersey: Prentice-
nal of Innovation Management, 3(3), 150–159. Hall.
doi:10.1108/14601060010322301
Lemon, W. F., Liebowitz, J., Burn, J., & Hackney,
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. R. (2002). Information Systems project failure: A
(1989). User acceptance of computer technology: comparative study of two countries. Journal of
A comparison of two theoretical models. Man- Global Information Management, 10(2), 28–39.
agement Science, 35(8), 982–1003. doi:10.1287/ doi:10.4018/jgim.2002040103
mnsc.35.8.982
March, J. G., & Simon, H. A. (1958). Organiza-
Eisenhardt, K. M., & Graebner, M. E. (2007). tions. New York, NY: John Wiley & Sons.
Theory building from cases: Opportunities and
Markus, M. L., & Tanis, C. (2000). The enterprise
challenges. Academy of Management Journal,
system experience – From adoption to success.
50(1), 25–32. doi:10.5465/AMJ.2007.24160888
In Zmud, R. W. (Ed.), Framing the domains of
Field, T. (1997). When bad things happen to good IT management: Projecting the future through the
projects. CIO, 11(2), 54–62. past (pp. 173–207). Cincinnati, OH: Pinnaflex
Educational Resources, Inc.
Gable, G. G., Sedera, D., & Chan, T. (2003). En-
terprise systems success: A measurement model. Martin, N., Gregor, S., & Hart, D. (2005). The
Paper presented at the Twenty-Fourth International social dimensions of business and IS/IT alignment:
Conference on Information Systems. Case studies of six public-sector organisations.
Australian Accounting Review, 15(3), 28–38.
Goulielmos, M. (2003). Outlining organisa-
tional failure in information systems. Disaster Mintzberg, H. (1990). Studying deciding: An ex-
Prevention and Management, 12(4), 319–327. change of views between Mintzberg and Waters,
doi:10.1108/09653560310493132 Pettigrew, and Butler. Organization Studies, 11(1),
1–15. doi:10.1177/017084069001100101
21
Nahar, N., Lyytinen, K., Huda, N., & Muravyov, Weick, K. E. (1995). Sensemaking in organisa-
S. V. (2006). Success factors for information tions. Thousand Oaks, USA: SAGE Publications.
technology supported international technology
Whyte, G., & Bytheway, A. (1996). Factors affect-
transfer: Finding expert consensus. Information
ing information systems’ success. International
& Management, 43, 663–677. doi:10.1016/j.
Journal of Service Industry Management, 7(7),
im.2005.02.002
74–93. doi:10.1108/09564239610109429
Nulden, U. (1996). Failing projects: Harder to
Yourdon, E. (1997). Death march. New Jersey:
abandon than to continue. Paper presented at the
Prentice-Hall.
Projectics, Bayonne, France.
Rai, A., Lang, S. S., & Welker, R. B. (2002).
Assessing the validity of IS success models: An
KEY TERMS AND DEFINITIONS
empirical test and theoretical analysis. Informa-
tion Systems Research, 13(1), 50–69. doi:10.1287/ Amensalism: A relationship between organ-
isre.13.1.50.96 isms in where one is inhibited or destroyed and
Sarker, S., & Sarker, S. (2000). Implementation the other is unaffected.
failure of an integrated software package: A case Commensalism: A symbiotic relationship
study from the far east. In Khosrow-Pour, M. where one organism benefits and the other is not
(Ed.), Organizational achievement and failure significantly harmed or helped
in Information Technology management (pp. Enterprise System: An integrated Information
169–186). Hershey, PA: Idea Group Publishing. System that manages organisational information
doi:10.4018/978-1-878289-83-4.ch014 and processes, such as financial, manufacturing,
distribution and human resources.
Smyth, R. W. (2001, 20-22 June). Threats to Information System: The systems that encom-
ERP success: A case study. Paper presented at pass the Information Technology, infrastructure,
the 5th Pacific Asia Conference of Information systems, procedures and human resources that
Systems, Seoul. are required to collect, store, manage and com-
Swan, J. A., & Newell, S. (1994). Managers’beliefs municate information that support and enhances
about factors affecting the adoption of technologi- the operations of an organisation. IS includes
cal innovation. Journal of Managerial Psychology, Enterprise Resource Planning systems, Enterprise
9(2), 1–3. doi:10.1108/02683949410059244 Systems, Electronic Commerce, Enterprise Infor-
mation Technologies, Computer Based Informa-
van der Zee, H. (2002). An IT measurement tion Systems and hardware infrastructure.
program. In Measuring the value of Information Mutualism: A symbiotic relationship where
Technology (pp. 115–141). Hershey, PA: Idea both organisms derive benefit.
Group Publishing. doi:10.4018/978-1-930708- Parasitism: A symbiotic relationship where
08-2.ch007 one organism benefits while the other is harmed.
Venkatesh, V., Morris, M. G., Davis, G. B., & Da- Symbiosis: A long-term living arrangement
vis, F. D. (2003). User acceptance of Information between different species resulting in positive
Technology: Towards a unified view. Management and negative associations.
Information Systems Quarterly, 27(3), 425–478.
22
23
Chapter 2
Measuring Information
Systems Success:
A Comment on the Use of Perceptions
Cees J. Gelderman
Open University of The Netherlands, The Netherlands
Rob J. Kusters
Open University of The Netherlands, The Netherlands
ABSTRACT
Information System success is difficult to measure directly. Because of the influence of non-controllable
variables, it actually seems to be impossible to directly compute or determine the contribution of
Information Systems to organizational performance, or to overall organizational effectiveness. As an
alternative, perception of system success is often used as a surrogate measure. However, this raises the
question of the validity of this surrogate measure. In this chapter, the authors describe a survey aim-
ing to investigate the validity of this surrogate measure. Results show that there is reason to doubt the
usefulness and validity of surrogate measures for objective system success.
INTRODUCTION puzzled many researchers and practitioners for

years. The prerequisites for the measurement of
Today, information technology plays an important IS success has been a subject of controversy (e.g.
role in almost every organization. Considerable Delone & McLean, 1992; Gelderman, 1998)).
budgets have been spent on IT in order to gain Because of the influence of non-controllable
a (sustainable) competitive advantage in today’s variables, it actually seems to be impossible to
markets. However, the measurement and expla- directly compute or determine the contribution
nation of information systems (IS) success has of information systems to organizational perfor-
mance or on overall organizational effectiveness.
DOI: 10.4018/978-1-4666-0170-3.ch002 As an alternative, the perception of system success
Measuring Information Systems Success
is often used as a surrogate measure. However, of IT investments are very difficult to measure
in the literature, a number of questions as to the (e.g. Bharadwaj et al., 1999; Davis et al., 2003;
validity of this surrogate measure arise. In this Gelderman, 1998). Because of the influence of
chapter, we describe a survey aimed at investi- non-controllable variables, such as changes in
gating the validity of this surrogate measure. A competitive offering or technology developments
model that measures and explains the (perception in the current product mix, it actually seems to be
of) success of an information system is also in- impossible to directly compute or determine the
troduced and tested, using empirical data derived contribution of information systems to organiza-
from the survey. This survey is described, and the tional performance or on overall organizational
resulting data are used to fit the model. Finally, effectiveness (e.g. Gallagher, 1974, Kusters et al.,
the results are presented and discussed, ending 2002; Skok et al., 2001). As a consequence of these
with a number of conclusions. measurement problems, proxy measurements for
IS evaluation purposes have gained considerable
acceptance. In the absence of objective measures,
INVESTIGATING SYSTEM SUCCESS many researchers and practitioners have turned to
perceptual surrogates (McHaney et al., 2002) by
A notion of ‘system success’ that is widely used, focusing on perceptions of IS success.
was defined, by Delone and Mclean in their Many studies have been aimed at develop-
original 1992 paper (Delone & McLean, 1992), ing and validating (multiple-item) constructs
as ‘effectiveness’ and ‘organizational impact’. for describing and evaluating perceptions of IS
Similar definitions can be found in Doll and success. For instance, Bailey and Pearson (1983)
Tokzadeh (1998) and Ives et al. (1983). We will identified 39 factors that constitute a user satis-
adopt the definition of Delone and Mclean (1992) faction measure, Li (1997) added another 7 items
and therefore assume that information system to this instrument, Doll and Torkzadeh (1988)
success indicates the degree to which the goals proposed a 5-factor model of end-user computing
of the organization have been furthered. Assum- satisfaction, Delone and Mclean (1992) proposed
ing that an organization has decided on its goals, 6 major dimensions or categories of IS success,
these can be translated into objective measures. and Saarinen (1996) identified 4 main dimensions
When investigating system success, direct of IS success.
measurement is therefore an obvious approach. Another related stream of research provides a
However, empirical studies reported mixed find- number of studies based on these proxy measures
ings. There is some evidence to support the idea and developed models, that identified a number
that information systems (IS) improve the ef- of explanatory variables which are believed to
ficiency and effectiveness of a firm’s operations impact and explain (perceptions of) IS success.
(e.g. Li, 1997), and that IT-related resources have Table 1 provides an overview of a number of
indeed a potential for competitive advantage (e.g. explanatory studies.
Lüneborg & Nielsen, 2003; Mata et al., 1995). In each case, the information used to quantify
However, most research in this area found little and validate these models has been derived from
evidence for a convincing, consistent relation- surveys and interviews. This means that the mod-
ship between IT investment and organizational els did not show the existence of relationships
performance (e.g. Farbey et al., 1999; Gallagher, between explanatory variables and system success,
1974; Hitt & Brynloffson, 1996; Raymond et al., but the existence of a relationship between the
2001; Saunders & Jones, 1992). Many publica- perceptions of the explanatory variables on the
tions underscore the proposition, that the benefits perception of system success. This in itself does
24
Table 1. Some recent studies aimed at explaining IS success
Source IS success Explanatory variables

Wierenga & Bruggen, • system usage • implementation process
1998 • user evaluation • system quality
• performance • match between demand for and supply of decision
support
Krishnan & Ramas- • overall satisfaction • cost of ownership
wamy, 1998 • competitive advantage
• process productivity
Ishman et al., 2001 satisfaction on • user involvement
• information quality • user participation
• confidence • perceive equity
• timeliness
Poon & Wagner, 2001 • access • 10 critical success factors
• usage
• satisfaction
• impact
• number of users)
Jiang et al., 2001 • perception of job performance • user satisfaction
• IS service quality
Gelderman, 2002 • user satisfaction • task variability
• task difficulty
• IS functionality
McHaney et al., 2002 • end-user computing satisfaction • information quality
• accuracy
• output format
• ease of use
• timelines
Heo & Han, 2003 • multiple dimensions of IS success • IS structure
not have to cause any problems. The well-known Ives et al., 1983). In Jiang et al. (2001), it is argued
technology acceptance model (Davis et al., 1989; that users are the legitimate source for final judg-
Davis, 1989) explicitly acknowledges the exis- ment, because they are indeed the customer.
tence of a direct link between these perceptions On the other hand, there is also literature that
and actual system use. This model has been vali- indicates that user perceptions might not be such
dated since then and, although some problems a good proxy for actual performance. Straub et al.
have been identified Legris et al., 2003), is still (1995) reported that users’ self-reports of utiliza-
generally accepted. The widely used DeLone and tion might be a poor measure of actual utilization.
McLean dimensions of IS success are also Davis and Kotteman (1994) however have shown
mainly based on the views of users. Similarly, the that the user assessment of system performance
innovation diffusion theory Rogers (1983; 1995) may conflict with reality, even when they are given
sees a direct link between the acceptance of an feedback. A similar result was found confirmed
innovation, and user perceptions regarding a by bij Caniëls and Bakens (in press) when inves-
variety of issues. Many researchers have argued tigating project management information systems.
that user evaluations of IS systems are an appro- Goodhue et al. (2000) also investigated the ques-
priate surrogate for IS success (e.g. Doll & tion of whether user evaluations could serve as
Torkzadeh, 1988; Downing, 1999; Etezadi- surrogates for performance. They concluded that
Amoli & Farhoomand, 1996; Gelderman, 1998; user evaluations should be used very cautiously
25
Figure 1. Research design

success of systems through a number of percep-
tions of explanatory variables can be seen as a
valid proxy of an actual model between the actual
explanatory variables and actual system success
(represented by arrow 4).
This line of reasoning only holds when arrows
1 and 2 represent valid measures. However, what
if the perception of system success is not only
influenced by some objective notion of system
success, but also by an assessment of the utility
of the system for that individual? In this case, the
assumption that perceived system success is a
as surrogates for performance. Finally, Seddon et valid measure for actual system success can no
al. (1999) noted that different stakeholders may longer be supported. Moreover, the model repre-
validly come to different conclusions about the sented by arrow 3 will have a different meaning
success of the same information system. It would from that represented by arrow four and can no
seem that using models based on user perceptions longer function as a proxy for that model.
as a proxy for actual performance might not be In order to isolate the effect of using percep-
so useful after all. tions, we obtained data of a single system from a
number of people involved in using that system.
In this study, we test the assumption that arrow
RESEARCH DESIGN 2 represents a valid measure. That is, we will
investigate whether the perceptions of system
The mixed findings on the usefulness of surrogate success provides a real valid view on actual
measures prompted this study on the validity system success. For this, data are collected for
of perceptions of system success as a surrogate a single system, relying on user perceptions of
measure for actual IS success (or the actual de- system success. Therefore, since actual system
gree to which the goals of the organization have success is now a single value, we would expect
been furthered by the system). Following the that the variance found in the measurement error
usual course, when trying to develop a model to contains no structural element. Hence we should
explain system success, data would be gathered not be able to quantify a model from these data
from different systems in such a way that each explaining perceived system success based on
data record contains data from a single system. perceptions of explanatory variables. So, this is
This allows statistical treatment of data and ex- our working hypothesis:
planation of differences between the degrees of
success reported for the different systems. • If perceived system success can be used as
As illustrated in Figure 1, the assumption is a valid surrogate measure for actual sys-
made that perceptions of explanatory variables tem success,
provide a valid measure for the actual variables • and we measure perceived system success
(arrow 1) and that the perception of system success of a single system where actual system suc-
is a valid measure for an objective phenomenon cess performance is constant,
of actual system success (arrow 2). If this is true, • Then the resulting measurement error
then a resulting model (represented by arrow 3) should have no structural component.
that explains variation between the perceived
26
Figure 2. Conceptual model of IS success
If on the other hand a significant model does In the remainder of this section, we will explain
appear, this would tend to throw doubt on the the design choices and identify the variables for
validity of using perceptions of system success as each of these three clusters.
a proxy measurement for actual system success.
Apparently in that case, that model represented Measures of Success: The
by arrow 3 provides information on the way in- Dependent Variable
dividuals perceive system success based on their
own goals and experiences. In the eighties and nineties, much IS research
In order to test this, we have developed a focused on the identification of factors that contrib-
model based on a literature survey that measures uted to information systems success (see Delone
and explains the (perception of) success of an in- & McLean, 1992 and Legris et al., 2003). In this
formation system. The model includes commonly study, we selected five issues that are commonly
identified measures of IS success. This model identified in literature:
is developed and described in the next section.
The resulting model was tested using empirical • user satisfaction,
data that have been derived from a survey among • system usage,
account managers of a single Dutch telecom • perceived usefulness,
company with respect to their experiences with • time effectiveness,
a single marketing information system. • decision confidence.
Most studies include user satisfaction and

MODEL DEVELOPMENT usage as most relevant measures of IS success
(e.g. Wieringa & Bruggen, 1998). User satisfac-
Table 1 above shows that while several conceptual tion is probably the most studied construct in
models exist for the explanation of IS success, none information systems research. User satisfaction
of these have been generally accepted. Based on contributes to IS success for the obvious reason
the components as described in Table 1, we de- that satisfied users would be likely to feel more
veloped a plausible but simple conceptual model. empowered, engaged, and fulfilled than unsatis-
In the conceptual model we have recognized two fied users (Woodruff & Burg, 2003). Delone and
clusters of system variables (system quality and McLean (1992) stressed that ‘satisfaction’ has
quality of implementation) to explaining the suc- a high degree of face validity, because it would
cess of a third cluster of variables: system success. be hard to deny the success of a system which
The resulting conceptual framework is depicted is appreciated highly by its users. Another com-
in Figure 2. mon factor is use or system usage. Usage had
been proposed as an IS success measure in many
frameworks for research (e.g. Delone & McLean,
27
1992; Doll & Torkzadeh, 1998; Ives et al., 1980). • ease of use,
The rationale for the application of usage as an • response time,
IS success measure is the idea, that a system does • system flexibility,
not contribute to performance if it is not used • system integration.
(Gelderman, 1998).
In addition to user satisfaction and system The threshold to use the system must be as
usage, other factors are believed to constitute IS low as possible. Ease of use therefore relates to
success. With others, Kim (1989) and Wierenga learning how to use the system as well as to actu-
en Bruggen (1998) include a third category on ally using the system. The system must be easy to
the level of individual impact on (decision) per- understand, logical and easily accessible. Also the
formance. An information system could provide necessary knowledge to use the system and the
the user with a better understanding of his or her way the system presents its output, determine its
decision context, which in its turn has a positive user friendliness (Bennett, 1977). Davis (1989)
impact on a person’s decision-making productiv- suggested a mediating role of perceived ease of use
ity (Delone & McLean, 1992). Individual impact between systems characteristics and probability
is closely related to the impact on the issue of of system use, as an indicator of system success.
decision making through an information system. Legris et al. (2003) found in their meta-analysis
In our study we have elaborated the individual of research results with the technology accep-
impact according to three related issues: tance model (TAM) a large number of empiri-
cal studies that included ‘perceived usefulness’
• perceived usefulness which refers to the and ‘perceived ease of use’ as predictors of the
degree in which someone believes that us- behavioral intention to use a technology. A short
ing a system will have a job performance response time is crucial to be able to integrate the
benefit (c.f. Davis, 1989), use of the system with the duties of its users (e.g.
• time effectiveness which refers to the de- Emery, 1971; Hamilton & Chervany, 1981). The
gree in which someone believes that using user’s waiting on the system must be prevented
a system has saved time (c.f. Wierenga en as much as possible.
Bruggen, 1998), and The system flexibility refers to the degree in
• decision confidence which refers to the which a system is focused on the user’s wishes
degree in which someone is willing to instead of the other way around. This applies
make decisions, based on information de- to the structure of the menus as well as to the
rived from the system (c.f. Wierenga en information supplied by the system. The system
Bruggen, 1998). must be easily adaptable to changes in informa-
tion needs. The ease of which the system can be
System Quality adapted to changed needs has a strong influence
on the success, measured over a longer period.
System quality is generally recognized as an im- Flexibility is the extent to which the system can
portant determinant of IS success. Many authors be made usable for new purposes without major
have developed many criteria for measuring technical intervention. Flexibility of a decision
system quality. System quality refers to the more support system has been found to be highly cor-
engineering-orientated performance characteris- related with success (Barki & Huff, 1990; Udo
tics of IT systems (Delone & McLean, 1992, p. & Davis, 1992).
64). For the ‘system quality’ construct we have The last issue to produce system quality is
selected four recurrent issues: integration with other systems. In case a system
28
is a result of various sub-systems, it is important (e.g. Alavi & Joachimsthaler, 1992); Gelderman,
to present the system to the user as if it were one 1997). The advantages of user involvement are:
integrated system. An integrated system makes it
possible to switch easily between different tasks • a better match with the user’s working
and has a universal user interface. methods,
• a better fulfillment of the user’s real needs,
Quality of Implementation • an easier surmounting of emotional and
psychological barriers, and
It is imperative that the implementation of a new • a better image of the project (the system is
information system runs as smoothly as possible. regarded as the user’s own product, instead
The less problems, conflicts or loss of time are of a system imposed by management).
encountered, the more successful the system will
be. Jong et al. (1994) found that managers, in Top management support is beyond doubt
companies which experienced a successful imple- the most frequently mentioned success factor
mentation, valued the quality of the system and the for information systems (Huizingh, 1995). It is
quality of the information higher. In addition, they generally agreed that top management support is
were more satisfied with the system, used it more critical to the successful implementation of any
frequently, and they perceived a greater impact of decision support systems (e.g. Evans & Schae-
the system on the organizational performance. In fer, 2001; Rockart & DeLong, 1998; Stroes &
this study we have selected six issues: Egberts, 1996). Within their well-known frame-
work, Rockart & DeLong (1998) show that the
• user involvement, support of the executive sponsor is related to three
• executive sponsor (top management responsibilities:
support),
• operating sponsor (IS champion), 1. initiating the development of the system,
• training of users, 2. managing and providing feedback on the
• user’s manual, proposed application and keeping a grip on
• standardization of the application. the development of the system, and
3. showing a strong and lasting interest to the
User involvement refers to the level in which people involved in the system.
users participate in the design and the maintenance
of an IS (Wierenga en Bruggen, 1998). Some 20 There are strong reasons to support the propo-
years ago, Ives and Olsen (1984) reported that sition that the support of the executive sponsor
user involvement is commonly considered a nec- greatly adds to the success of a system. Firstly, the
essary condition for successful IS development. relation between profit and loss is often uncertain.
Generally, user involvement is appropriate for In order to receive approval for the system without
unstructured problems or in situations where user the possibility to conduct a profit and loss analysis,
acceptance is important. Whatever kind of infor- it is necessary that someone of the higher manage-
mation system it concerns, early involvement of ment trusts the proposal and is willing to back it.
users seems to be an important success factor (e.g. Secondly, there are often conflicting interests. Not
Huizingh, 1995). Empirical studies have identified only different departments have diverse interests,
a (relatively strong) relation between user involve- but interests of different managers may also clash
ment and the overall satisfaction with a system within a department. In line with this, Keen (1981)
poses that the development of information systems
29
Table 2. Explanatory model for IS success

is not only a technical, but in particular a political
process. There is the natural resistance against Dependent Variable Independent Variable
organization change. With each change, resistance Construct: Constructs:
comes to light, which can only be overcome with IS success System quality
the support of the higher management. • user satisfaction • ease of use
Besides proper information of top manage- • system usage • response time
ment, a person is needed who will function as • perceived usefulness • system flexibility
the mainstay of the system, and takes care of • time effectiveness • integration
the day-to-day business related to the system. • decision confidence Quality of implementation
Rockart and DeLong (1998) labeled such a • user involvement
person as operating sponsor. Besides operating • executive sponsor
sponsor, also other terms are used for such per- • operating sponsor
sons, like internal sponsor (Little, 1975), change • user’s manual
agent (Rogers, 1983; 1995; Thierauf, 1988), and • training of users
MDSS-champion Hanssens et al, 1990). The
• standardization
presence of such a ‘champion’ has been found
to have a positive impact on IS success (Sviokla,
1989). The empirical investigation of Palvia and
In several studies, the training of users appeared
Chervany (1995) underscored that the role of the
to have a positive impact on the performance of a
project champion ‘cannot be understated’, since
system (e.g. Alavi & Joachimsthaler, 1992; Barki
it produced the largest gains in IS success.
& Huff, 1990; Sanders & Courtney, 1985; Udo &
The operating sponsor is quite often some-
Davis, 1992). In addition, the quality of a user’s
one who works as a system’s user or as a user’s
manual also contributes to the effectiveness of
manager. The operating sponsor exactly knows
information systems (Torkzadeh & Doll, 1993).
the capabilities of the system, its possibilities
Finally, we expect a positive relationship be-
and its limitations. The presence of an operating
tween the standardization of the application and
sponsor in the vicinity of the users facilitates the
the success of an information system. One has to
possibility to provide instant support in case of
decide whether a customized application (‘make’)
problems. The impact of an operating sponsor
or a standardized application (‘buy’) will be used.
has been demonstrated in empirical studies (e.g.
It is a choice between a better match with a spe-
Jong et al., 1994). They observed that there was
cific situation in case of a tailor-made application
an operating sponsor (‘an IS-champion’) in al-
on the one hand and a faster implementation,
most two thirds of the studied companies, using
combined with the advantage of working with a
a decision support system. An operating sponsor
standard in case of an ‘off the shelf’ application
contributes to:
on the other hand. Table 2 contains an overview
of the resulting model.
• the involvement of the users in the devel-
opment or purchase of an IS,
• the satisfaction with the system,
THE SURVEY
• the adaptability of the system,
• the level of information quality,
Data was collected within KPN, a Dutch telecom
• one’s own performance, and
company. At year-end 2002 KPN had more than
• the performance of the organization/
38,000 employees serving 7.9 million fixed-line
department.
30
customers, 13.4 million mobile customers, and be ‘explained’ by perceptual differences. In line
1.4 million internet subscribers. KPN is market with the aforementioned, if there would be a tight
leader in the Netherlands and offers consumers and fit between our perceptual surrogate measures
businesses a large number of telecommunication and a set of objective measures of IS success, we
services. KPN Sales is a staff department which would not be able to explain IS success by the
serves as one of the distribution channels of the explanatory variables of the conceptual model.
13 KPN business units. The KPN Sales depart- A questionnaire was developed that is prin-
ment serves three main market segments, which cipally based on the insights from the literature
are differentiated according to the sales volume study. The first draft of the questionnaire was pre-
of the accounts: the Corporate Market (top-500 tested by a small number of professionals within
customers), the Business Market Relationships the company. The pre-test group was asked to
(top 500-3,000 customers), and the Business review the survey, primarily for the clarification
Market Transactions (all other customers). of questions. Pre-testing provided the opportunity
to improve the questionnaire, and enhance its
Data Collection reliability and validity. The final questionnaire
consisted of 15 questions, measuring perceptions
This study is limited to the 114 account managers on a 5-point Likert scale. In order to minimize the
who all work for the business market relationships non-response, support was asked and received
segment (top 500-3,000 customers). The respon- from the senior management to promote the sur-
dents have identical roles (account manager), and vey. In the summer of 2003 the survey was sent
a similar education, since they all received higher by e-mail to 114 account managers who worked
vocational business education. The latter is a job within the segment Business Market Relationship.
requirement. On-the-job training is also identical All remaining respondents were phoned and asked
across the company. Finally, the selection process to complete the questionnaire. This resulted in
guarantees that persons with similar capabilities a total number of 79 completed questionnaires,
are hired. All together this indicates a fairly ho- yielding a satisfactory response rate of 69.3%.
mogeneous group of respondents.
The average number of customers per account Analysis: Construct Reliability
manager ranges between 10 and 25 customers. In
the autumn of 2002 a new marketing information The reliability of the constructs was evaluated by
system was implemented, which had to be used assessing the internal consistency of the indica-
by all of the 114 account managers for the same tors representing each construct using Cronbach’s
purpose (report, structure, and exchange customer alpha. Reliability analysis indicated that the coef-
information) and in the same way (mandatory). ficients of the Cronbach’s alpha were all higher
They received the same information, instructions, than 0.6 (range from 0.68 to 0.78). This indicated
training and manuals. Considering the fact that an acceptable internal consistency and reliability
we are dealing with a single system with users in of the measurements in this study. Table 3 shows
exactly the same setting, we expect to find identical the Cronbach’s alphas, the means and the standard
responses with these homogeneous group of users. deviations of the constructs. The constructs are
This means that differences between respondents measured on 5-point scales, indicating relatively
on their score on, for instance, the items of our low perceptual scores of the users on IS success,
system quality-construct cannot be traced back to the system quality, and the quality of implementa-
differences in corresponding objective measures tion. In addition, there is a considerable level of
of system quality. These differences could only variance in the constructs, which conflicts with
31
Table 3. Constructs and items: Reliability, means and standard deviations (n = 79)
Dependent Variables Cronbach’s alphas Means Standard

of Constructs Deviations
Dependent Variable Construct 0.781 2.77 0.731
IS success
user satisfaction
system usage
perceived usefulness
time effectiveness
decision confidence
Independent Variable Constructs
System quality 0.686 2.26 0.724
ease of use
response time
system flexibility
integration
Quality of implementation 0.703 2.74 0.698
user involvement
executive sponsor
operating sponsor
user’s manual
training of users
Standardization
prior expectations. Additional correlation analysis tion of system success construct. This finding is
showed that items that should be related, are cor- very much in line with the usual way of viewing
related, indicating convergent validity. In addition system success. Users respond positively to the
it was found that items that theoretically should not usual measures that accompany the introduction
be related, did not correlate (discriminant validity). of a new information system (user involvement,
training, support of an executive and an operational
Analysis: Fitting the Model sponsor), and system quality also has a significant
impact on the IS success. However, in this case
Multiple regression analysis has been applied in these conclusions are odd, since the data are col-
order to determine the influence of the explanatory lected from users of a single system.
variable on the success of the information system
(see Table 4). The results indicate that the data
could be fitted to the proposed model with a high CONCLUSION AND IMPLICATIONS
degree of confidence. More than 50% of the vari-
ance in the IS success construct can be explained by In this study we have reported an empirical study
the model. Indeed, both perceived system quality to assess the validity of perceived system success
and the perceived quality of implementation could as a surrogate measure for actual system success.
be shown to be the drivers of the overall percep- We developed a construct for the measurement of
32
Table 4. Model fit: Results of the multiple regression analysis
Dependent variable IS Success

Multiple R .759
R 2
.576
Adjusted R2 .565
S.E. .482
Analysis of variance d.f. Sum of squares Mean squares

Regression 2 23.411 11.706
Residual 74 17.224 .233
F=50.292 Significance F = .000
Variables Unstandardized coefficients Standardized t

Beta
(constant) .581
system quality .401 .400 4.203 a
quality of implementation .463 .447 4.698 a
Note: a indicates significance at p<0.01
IS success, including perceived user satisfaction, could fit the data, since this would demonstrate
perceived system usage, perceived usefulness, this existence of a structural component in the
perceived contribution to decision confidence, resulting measurement error.
and perceived contribution to time saving. We succeeded in fitting a model using con-
Our working hypothesis was: structs of perceived system quality and perceived
quality of implementation and conclude that we
• if perceived system success can be used as must reject our hypothesis. On the basis of these
a valid surrogate measure for actual system results perceived system success cannot appar-
success, ently be used as a valid surrogate measure for
• and we measure perceived system success actual system success. This does not mean that
of a single system where actual system the notion of perceived system success is useless.
success performance is constant, For users performing their daily duties supported
• then the resulting measurement error by a system this notion is apparently informative.
should have no structural component. The fact that we can prove that the user percep-
tion regarding system success is influenced by
In order to test this hypothesis, we carried their perceptions with regard to system and
out a survey that was aimed at a relatively large implementation quality is actually very much in
number of experienced users of a single system. line with previous research (e.g. Seddon et al.,
The survey was based on an explanatory model 1999). The resulting model apparently describes
derived from literature based on constructs of a phenomenon that is actually present in practice.
perceived system quality and perceived quality of As such, the model can be used. It suggests that
implementation. The working hypothesis would management should invest in the more practical
be rejected, if this theoretically acceptable model sides of the introduction of information systems,
33
since users respond positively to (their percep- Bailey, J. E., & Pearson, S. W. (1983). Devel-
tions of) the usual measures accompanying the opment of a tool for measuring and analyzing
introduction of a new information system (user computer user satisfaction. Management Science,
involvement, training, support of an executive 29(5), 530–545. doi:10.1287/mnsc.29.5.530
and an operation sponsor). It also suggests that
Barki, H., & Huff, S. J. (1990). Implementing
perceived system quality has a significant impact
decision support systems: Correlates of user
on perceived system success. Achieving the full
satisfaction and system usage. INFOR (Canada),
potential of information systems therefore requires
28(2), 89–101.
a thorough assessment and improvement of the
quality of the system, as perceived by its users. Bennett, J. (1977). User-oriented graphics systems
However, in this case, perceptions of reality for decision support in unstructured tasks. In Treu,
and reality itself would seem to be sufficiently dif- S. (Ed.), User oriented design of interactive graph-
ferent to warn against using perceived systems as ics systems. New York, NY: Association for Com-
a surrogate for actual system success. Apparently puting Machinery. doi:10.1145/1024273.1024275
different persons have a different perception of the
Bharadwaj, A. S., Bharadwaj, S. G., & Konsynski,
reality of system success. This would indicate that
B. R. (1999). Information Technology capability
further research is required on identifying valid
and firm performance: An empirical investigation.
measures. Another implication for further research
Management Information Systems Quarterly,
regards the sampling method in survey studies.
45(7), 1008–1024.
Most commonly, questionnaires are sent to single
representatives of a large number of companies. Caniëls, M. C. J., & Bakens, R. J. J. M. (2011).
Researchers usually start from the assumption that The effects of project management Information
variance has to be explained across companies, Systems on decision making in a multi project
not within single companies. However, our study environment. International Journal of Project
proved that nothing is further from the truth. Single Management, 30(2).
users, acting as respondents for their companies,
Davis, F. D. (1989). Perceived usefulness,
should not be seen as representative for other us-
perceived ease of use, and user acceptance of
ers in the same company. It seems that research
Information Technology. Management Infor-
reports are not always specific and explicit about
mation Systems Quarterly, 13(3), 318–340.
a perceptual sample bias and the consequences
doi:10.2307/249008
for the results and conclusions. Future and further
research cannot ignore this uncomfortable truth Davis, F. D., Bagozzi, R., & Warshaw, P. R.
in survey studies. (1989). User acceptance of computer technology:
A comparison of two theoretical models. Man-
agement Science, 35(8), 982–1003. doi:10.1287/
REFERENCES mnsc.35.8.982
Alavi, M., & Joachimsthaler, E. A. (1992). Revisit- Davis, J. D., & Kottemann, J. E. (1994). User per-
ing DSS implementation research: A meta-analysis ception of decision support effectiveness: Two pro-
of the literature and suggestions for researchers. duction planning experiments. Decision Sciences,
Management Information Systems Quarterly, 25(1), 57–78. doi:10.1111/j.1540-5915.1994.
16(1), 95–113. doi:10.2307/249703 tb00516.x
34
Davis, L., Dehning, B., & Stratopoulos, Th. (2003). Farbey, B., Land, F., & Targett, D. (1999). Mov-
Does the market recognize IT-enabled competitive ing IS evaluation forward: Learning themes and
advantage? Information & Management, 40(7), research issues. The Journal of Strategic Infor-
705–716. doi:10.1016/S0378-7206(02)00092-7 mation Systems, 8(2), 189–207. doi:10.1016/
S0963-8687(99)00021-9
De Jong, C. M., Huizingh, K. R. E., Oude Ophuis,
P. A. M., & Wierenga, B. (1994). Critical success Gallagher, C. A. (1974). Perceptions of the
factors for marketing decision support systems. value of a management Information System.
Delft, The Netherlands: Eburon. Academy of Management Journal, 17(1), 46–55.
doi:10.2307/254770
Delone, W. H., & McLean, E. R. (1992). Informa-
tion Systems success: The quest for the dependent Gelderman, M. (1997). Success of management
variable. Information Systems Research, 3(1), support systems: A literature review and an empiri-
60–95. doi:10.1287/isre.3.1.60 cal investigation. Amsterdam, The Netherlands:
Thesis Publishers Amsterdam.
Doll, W. J., & Torkzadeh, G. (1988). The mea-
surement of end-user computing satisfaction. Gelderman, M. (1998). The relation between user
Management Information Systems Quarterly, satisfaction, usage of Information Systems and
12(2), 259–276. doi:10.2307/248851 performance. Information & Management, 34(1),
11–18. doi:10.1016/S0378-7206(98)00044-5
Doll, W. J., & Torkzadeh, G. (1998). Developing
a multidimensional measure of system-use in Gelderman, M. (2002). Task difficulty, task vari-
an organizational context. Information & Man- ability and satisfaction with management support
agement, 33(4), 171–185. doi:10.1016/S0378- systems. Information & Management, 39(7),
7206(98)00028-7 593–604. doi:10.1016/S0378-7206(01)00124-0
Downing, C. E. (1999). System usage behavior Goodhue, D. L., Klein, B. D., & March, S. T.
as a proxy for user satisfaction: An empirical in- (2000). User evaluation of IS as surrogates for
vestigation. Information & Management, 35(5), objective performance. Information & Man-
203–216. doi:10.1016/S0378-7206(98)00090-1 agement, 38(2), 87–101. doi:10.1016/S0378-
7206(00)00057-4
Emery, J. C. (1971). Cost/benefit analysis of
Information Systems. SMIS Workshop report Hamilton, S., & Chervany, N. L. (1981). Evalu-
number 1, The Society for Management Informa- ating Information System effectiveness 1. Com-
tion Systems, Chicago, IL. paring evaluation approaches. Management
Etezadi-Amoli, J., & Farhoomand, A. F. (1996).
doi:10.2307/249291
A structural model of end user computing sat-
isfaction and user performance. Information & Hanssens, D. M., Parsons, L. J., & Schultz, R. L.
Management, 30(2), 65–73. doi:10.1016/0378- (1990). Market response models: Econometric
7206(95)00052-6 and time series analysis. Boston, MA: Kluwer
Academic Publishers.
Evans, J., & Schaefer, C. (2001). Ten tasks of
change: Demystifying changing organizations. Heo, J., & Han, I. (2003). Performance measure
San Francisco, CA: Jossey-Bass/Pfeiffer. of Information Systems (IS) in evolving comput-
ing environments: an empirical investigation.
Information & Management, 40(4), 243–256.
doi:10.1016/S0378-7206(02)00007-1
35
Hitt, L. M., & Brynjolffson, E. (1996). Productiv- Krishnan, M. S., & Ramaswamy, V. (1998). An
ity, business profitability, and consumer surplus: empirical analysis of customer satisfaction for
Three different measures of IT value. Management Intranet marketing systems. Decision Support
Information Systems Quarterly, 20(2), 121–142. Systems, 24(1), 45–54. doi:10.1016/S0167-
doi:10.2307/249475 9236(98)00062-1
Huizingh, K. R. E. (1995), Marketing and sales Kusters, R. J., Trienekens, J. J. M., & Hassoldt,
Information Systems. Schoonhoven, The Nether- W. (2002). On the business impact of software
lands: Academic Service (Dutch text). process improvement. In D. C. Martin (Ed.), Pro-
ceedings of the IEEE 26th International Computer
Ishman, M. D., Pegels, C. C., & Sanders, G. L.
Software & Applications Conference, (pp. 59-67).
(2001). Managerial information: System suc-
Los Alamitos, CA: IEEE Computing Society.
cess factors within the cultural context of North
America and a former Soviet Republic. The Legris, P., Ingham, J., & Collerette, P. (2003). Why
Journal of Strategic Information Systems, 10(4), do people use Information Technology? A criti-
291–312. doi:10.1016/S0963-8687(01)00055-5 cal review of the technology acceptance model.
Ives, B., Hamilton, S., & Davis, G. (1980). A
doi:10.1016/S0378-7206(01)00143-4
framework for research in computer-based man-
agement Information Systems. Management Sci- Li, E. Y. (1997). Marketing Information Systems
ence, 26(9), 910–934. doi:10.1287/mnsc.26.9.910 in small companies. Information Resources Man-
agement Journal, 10(1), 27–35.
Ives, B., & Olson, M. H. (1984). User involvement
and MIS success: A review of research. Man- Little, J. D. C. (1975). Brandaid, a marketing
agement Science, 30(5), 586–603. doi:10.1287/ mix model. Part 1: structure and Part 2, imple-
mnsc.30.5.586 mentation, calibration and case study. Operations
Research, 23(4), 628–655, 656–673. doi:10.1287/
Ives, B., Olson, M. H., & Baroudi, J. J. (1983).
opre.23.4.628
The measurement of user information satisfac-
tion. Communications of the ACM, 26, 785–793. Lüneborg, J. L., & Nielsen, J. F. (2003). Cus-
doi:10.1145/358413.358430 tomer-focused technology and performance in
small and large banks. European Management
Jiang, J. J., Klein, Roan, G., J., & Lin, J. T. M.
Journal, 21(2), 258–269. doi:10.1016/S0263-
(2001). IS service performance: Self-perceptions
2373(03)00020-3
and user perceptions. Information & Manage-
ment, 38(8), 499–506. doi:10.1016/S0378- Mata, F. J., Fuerst, W. L., & Barney, J. B. (1995).
7206(01)00072-6 Information Technology and sustained competi-
tive advantage: A resource based analysis. Man-
Keen, P. G. W. (1981). Information Systems and
agement Information Systems Quarterly, 19(4),
organizational change. Communications of the
487–505. doi:10.2307/249630
ACM, 24(1), 24–33. doi:10.1145/358527.358543
McHaney, R., Hightower, R., & Pearson, J.
Kim, K. K. (1989). User satisfaction: A synthesis
(2002). A validation of the end-user computing
of three different perspectives. The Journal of
satisfaction instrument in Taiwan. Information
Information Systems, 4(1), 1–12.
& Management, 39(6), 503–511. doi:10.1016/
S0378-7206(01)00119-7
36
Palvia, S. C., & Chervany, N. L. (1995). An Skok, W., Kophamel, A., & Richardson, I.
experimental investigation of factors influenc- (2001). Diagnosing Information Systems success:
ing predicted success in DSS implementation. Importance-performance maps in the health in-
Information & Management, 29(1), 43–53. dustry. Information & Management, 38, 409–419.
doi:10.1016/0378-7206(95)00006-I doi:10.1016/S0378-7206(00)00076-8
Poon, P., & Wagner, C. (2001). Critical success Straub, D. W., Limayem, M., & Karahanna-Evaris-
factors revisited: Success and failure cases of in- to, E. (1995). Measuring systems usage: Implica-
formation systems for senior executives. Decision tions for IS theory testing. Management Science,
Support Systems, 30(3), 393–418. doi:10.1016/ 41(8), 1328–1342. doi:10.1287/mnsc.41.8.1328
S0167-9236(00)00069-5
Stroes, H. J., & Egberts, M. E. (1996). Veranderen
Raymond, L., & Brisoux, J., J., & Azami, A. met resultaat. Deventer, The Netherlands: Kluwer
(2001). Marketing Information Systems practices BedrijfsInformatie (Dutch text).
in small manufacturing firms: antecedents and
Sviokla, J. (1989). Business implication of knowl-
consequences. Journal of Computer Information
edge-based systems. In Sprague, R. H., & Watson,
Systems, 41(3), 32–41.
H. J. (Eds.), Decision support systems: Putting
Rockart, J. F., & DeLong, D. W. (1998). Ex- theory into practice (pp. 125–151). Englewood
ecutive support systems: The emergence of top- Cliffs, NJ: Prentice Hall.
management computer use. Homewood, IL: Dow
Thierauf, R. J. (1988). User-oriented decision sup-
Jones-Irwin.
port systems. Englewood Cliffs, NJ: Prentice Hall.
Rogers, E. M. (1983, 1995). Diffusion of innova-
Torkzadeh, G., & Doll, W. J. (1993). The place
tion, (1st and 4th edition). New York, NY: The
and value of documentation in end-user comput-
Free Press.
ing. Information & Management, 24(3), 147–158.
Saarinen, T. (1996). An expanded instrument for doi:10.1016/0378-7206(93)90063-Y
evaluating information system success. Informa-
Udo, G. J., & Davis, J. S. (1992). Factors affecting
tion & Management, 31(2), 103–118. doi:10.1016/
decision support systems benefits. Information &
S0378-7206(96)01075-0
Management, 23, 359–371. doi:10.1016/0378-
Sanders, G. L., & Courtney, J. F. (1985). A field 7206(92)90017-A
study of organizational factors influencing DSS
Wierenga, B., & van Bruggen, G. H. (1998).
success. Management Information Systems Quar-
Drivers of success for marketing management
terly, 9(1), 77–93. doi:10.2307/249275
support systems: towards a theory of marketing
Saunders, C. S., & Jones, J. W. (1992). Measuring decision support. Erasmus University Rotterdam:
performance of the Information Systems function. Management Report No. 1998-8.
Journal of Management Information Systems,
Woodruf, J., & Burg, W. (2003). Satisfaction/dis-
8(4), 63–82.
satisfaction: are users predisposed? Information
Seddon, P. B., Staples, S., Patnayakuni, R., & & Management, 40(4), 317–324. doi:10.1016/
Bowtell, M. (1999). Dimensions of Information S0378-7206(02)00013-7
Systems success. Communications of the AIS,
2(20).
37
ADDITIONAL READING Urbach, N., Smolnik, S., & Riempp, G. (2008). A

methodological examination of empirical research
Petter, S., DeLone, W., & McLean, E. (2008). on Information Systems success: 2003 to 2007.
Measuring Information Systems success: Models, AMCIS 2008 Proceedings, paper 7. Retrieved
dimensions, measures, and interrelationships. from http://aisel.aisnet.org/amcis2008/7
European Journal of Information Systems, 17(3),
236–263. doi:10.1057/ejis.2008.15
38
39
Chapter 3
Information Systems
for Organizational
Effectiveness Model:
A Rationale Alignment
Govindan Marthandan
Multimedia University, Malaysia
Chun Meng Tang

UCSI University, Malaysia
ABSTRACT
Despite the proposal of various Information Systems (IS) evaluation models and approaches, IS evalu-
ation has never been straightforward. There are issues and challenges in proving the business value
of IS. Adding to the difficulty, a vast number of measures have been employed conveniently for evalua-
tion purposes without going through a rigorous validation process. Recognising the complexity for IS
researchers, IS specialists, and business managers to agree on a common model for the evaluation of
IS business value, this chapter presents an empirically validated IS evaluation model, the IS for organi-
sational effectiveness (ISOE) model, for planning, designing, implementing, and appraising IS. There
also emerges a new theory, the Information System business value (ISBV) theory, from the ISOE model
to establish that IS business value is multifaceted and are observable in the form of improvements in
organisational effectiveness.
DOI: 10.4018/978-1-4666-0170-3.ch003
Information Systems for Organizational Effectiveness Model
INTRODUCTION Koch (2002) highlight that an evaluation model

is particularly critical for business managers who
When organisations spend more on information are trying to reduce uncertainty when making a
systems (IS), it becomes more critical to keep track major IS investment decision. Gunasekaran et al.
of investment returns. The significant amount of (2006) propose that a comprehensive, integrated
resources required for such investments prompts IT evaluation approach is needed to better justify
organisations to be careful with their IS spending. IT investments2.
To justify an increase in IS spending and under- Performing an IS evaluation is not a simple
stand utilisation of limited organisational resources task. Marthandan and Tang (2010) identify eight
on IS, the correlation between IS and business IS evaluation issues and challenges: evaluation
performance has been of great interest to business scope, evaluation timing, unit of analysis, level
managers. They are keen to find out how, and to of analysis, different perspectives, different di-
what extent, IS performs to deliver the intended mensions, different measures, and underpinning
benefits (Klecun & Cornford, 2005). However, theoretical frameworks3. Cronk and Fitzgerald
conflicting findings on the business value of IS (2002) describe IS evaluation as complex, context-
reported in past studies have not helped solve the based, and multidimensional. Stefanou (2001)
problem which has been bothering IS researchers, highlights that there is a need for a multidimen-
IS specialists, and business managers for years. 1 sional and multi-perspective evaluation approach
Lederer and Mirani (1995) reckon that an un- which includes both quantitative and qualitative
derstanding of IS benefits is critical to effective perceived costs and benefits.
IS planning as this helps business managers and Having understood the need for business man-
IS specialists to align their interests and perspec- agers to have an IS evaluation model for either
tives with a common set of expected benefits in investment decision-making or post-implementa-
new IS projects. Business managers, who share tion evaluation purposes, this chapter presents an
their experiences in using IS, can help IS special- empirically validated IS evaluation model - the
ists to propose and design IS that meet business IS for organisational effectiveness (ISOE) model
priorities and requirements. Tillquist and Rodgers (Marthandan & Tang, 2009; Tang & Marthandan,
(2005) add that if business managers are not clear 2011)4. Using the balanced scorecard (BSC) as the
about how and what IT does to contribute value to underpinning framework, the ISOE model exam-
organisations, they would not be able to make the ines the role of IS on organisational effectiveness.
right IT investment decision. Love et al. (2004) The model suggests that when business managers
suggest that the ability to prescribe what benefits are able to distinguish the various dimensions of
are to be expected in an IS investment helps organisational effectiveness and detect positive
organisations make better investment decision. improvements that are enabled by IS within the
As businesses spend more on IS, the issue of IS individual dimensions of organisational effective-
business value becomes more prevalent. However, ness, the business value of IS will be evident.
IS researchers, IS specialists, and business manag- Organisations can use the ISOE model to
ers still have not agreed on a single IS evaluation study business requirements before deciding on
method, although cost-benefit analysis seems to the systems specification. The model specifies
be favoured. The decision as to which evaluation the intended business benefits as the criteria IS
method is more appropriate in a particular situation specialists and business managers should pay
is largely context dependent. Evaluators com- attention to when designing a system, helping to
monly face a difficult situation in choosing the right align business and IT strategies. Organisations
method for a particular context. Bernroider and can also use the model to do post implementation
40
evaluation. Having demonstrated that there are necessary to have a surrogate for IS so as to over-
positive organisational effectiveness improve- come the problem of having disparate computer
ments as a result of IS implementation, business systems. For a meaningful comparison across
managers can then prove the business value of IS. systems, Law and Ngai (2005) propose that the
In the sections that follow, first two sections evaluation scope should focus on just one type of
explain the background behind the original con- system instead of all types. They suggest that as
ceptual model, which eventually evolved into the ERP systems create impact at both the operational
ISOE model. A section next presents the new ISOE and strategic organisational levels, ERP systems
model and explains the emergence of a new theory provide an opportunity for a more comprehensive
based on the model. Last section provides some study of IS success.
discussion on the usefulness of the ISOE model. ERP systems are enterprise systems that are
capable of integrating cross-functional business
processes. In an IDC report, Pang (2009) high-
BACKGROUND TO THE lighted that as organisations looked to enterprise
ISOE MODEL systems like ERP systems to help improve business
efficiency and reduce operational costs, worldwide
Before introducing the ISOE model, it is impor- 2008-2013 ERP revenue was forecasted to grow
tant for this section to provide some background from US$33 billion in 2008 to US$40.4 billion in
to the model. These include a single type of IS, 2013, attaining a CAGR of 4.2% between 2008
organisational level, managerial perception, or- and 2013. In the Asia Pacific region, 2008-2013
ganisational effectiveness, and the BSC. This sec- ERP revenue was forecasted to grow from US$3.3
tion also introduces the original conceptual model billion in 2008 to US$5 billion in 2013, attaining a
from which the ISOE model eventually evolved. CAGR of 9% between 2008 and 2013. The popu-
larity of ERP systems among business sectors and
A Single Type of IS their system characteristics make them a suitable
surrogate for IS in developing the ISOE model.
To examine the role of IS on organisational ef-
fectiveness, a study about IS in general would Organisational Level
not provide valuable insight as there are disparate
computer systems being used in organisations. Evaluation of IT value can be performed at sev-
Different contexts, in terms of systems, objectives, eral levels; for example, system, individual, and
benefits, and stakeholders, would have implica- organisation. Tangpong (2008) recommends that
tions on the evaluation approaches. Some past organisational level is where IT value should be
studies attempted to develop specific measures measured. The reason is straightforward - IT is
for a particular system type. However, some did part of the wider organisational system. When
not specify the type of system under investigation, used to help improve business performance, IT
leaving questions on the reliability and validity creates value. Kohli and Grover (2008) share the
of the criteria (Grover et al., 1996). Focusing on same view. The business value of IT resides at the
a specific type of IT provides better insights into organisational level, and its presence is known
the IT payoff issue, instead of overall IT (Dasgupta in both tangible and intangible forms. Thus, an
et al., 1999). economic-focused evaluation could fail to expose
Considering that there is a possibility of dis- its true value. Martinsons and Martinsons (2002)
tortion caused by disparate IS on the dimensions reckon that as value is created from the use of IT,
of IS-enabled organisational effectiveness, it is IT value should be measured at firm level instead
41
of macroeconomic level. At macroeconomic level, resources and without placing undue strain upon
the focus is too much on sector efficiency, with- its members.” As an abstract concept about human
out recognising that IT is capable of improving judgment of effectiveness (Walton & Dawson,
more than just operational efficiency. Instead, 2001), organisational effectiveness is an impor-
they suggest that the focus should be on business tant dependent variable in organisational studies
transformation effects of IT. Grover et al. (1996) (Chelladurai, 1987; Walton & Dawson, 2001).
observe that effectiveness evaluation at the indi- In organisational performance studies, surrogate
vidual level has received heavy attention in past measures are used as indicators of organisational
studies. They comment that the individual-level effectiveness (Henri, 2004).
approach would be appropriate when assessing Researchers have been faced with a daunt-
system features or user attitude, but not suitable ing challenge in the search for valid and reliable
for studying strategic impacts of IS. Therefore, the measurement criteria, supported by underlying
focus was on the organisational level in develop- theoretical frameworks, to assess organisational
ing the ISOE model. effectiveness (Walton & Dawson, 2001). In study-
ing organisational effectiveness, a comparison of
Managerial Perception research results across studies is difficult, as it
is often compounded by such issues as different
Although it would be helpful to measure organi- measures, perspectives, and context (Cameron,
sational impact of IS with quantitative, objective 1978). Cameron (1986a) highlights four problems
measures, these measures are not easily available. faced by organisational effectiveness researchers:
Thus, an alternative is to use surrogate measures; (1) inadequacy in identifying indicators of effec-
for example, managerial perception. A subjective tiveness; (2) over-reliance on single indicators of
perceptual evaluation, coupled with a multi-item effectiveness and ignoring the relationships among
scale, can be a reliable means of evaluating IS in- multiple indicators; (3) underspecified models and
vestment (Saarinen, 1996). Ragowsky et al. (2000) ignoring the time frame of the criterion variable;
suggest that perceived benefits are an alternative and (4) over-generalisation to dissimilar organi-
way to measure IS benefits. They conclude that sations or subunits. More problems in measuring
senior managers, who are able to provide an over- organisational effectiveness have been described
view of IS benefits and whose perception reflects by Steers (1975), including construct validity,
that of their organisations, are a reliable source of criterion stability, time perspective, multiple
answers to questions on IS benefits. Tallon et al. criteria, measurement precision, generalisability,
(2000) suggest that being an IT user, as well as a theoretical relevance, and level of analysis.
decision-maker in IT investment, senior business A major concern of organisational effective-
executives are a good source of information on ness studies is identifying an evaluation model,
IT payoffs. Thus, the CEOs and MDs were the together with the relevant criteria, for use in a
target respondents in developing the ISOE model. particular context. As there is no comprehensive
conceptualisation of an effective organisation, the
Organisational Effectiveness definition and evaluation model of organisational
effectiveness change according to the organisa-
Georgopoulos and Tannenbaum (1957, p.535) tional context. Individual values and preferences,
define organisational effectiveness as “the ex- coupled with blurred scope, make the search for
tent to which an organisation as a social system, the best set of effectiveness criteria difficult as
given certain resources and means, fulfills its there are so many possibilities. The usefulness
objectives without incapacitating its means and of different evaluation models of effectiveness is
42
largely confined to objectives and scope of evalua- expectations, financial returns will improve. Using
tion in a particular context (Cameron, 1986b). The the stakeholder theory as a background, he explains
construct of organisational effectiveness suffers that meeting stakeholder expectations which are
from definition problems as a result of different clearly aligned with the business strategy brings
organisational goals, different stakeholders, and organisations long-term strategic advantage.
different measures of goals (Chelladurai, 1987). The stakeholder theory asserts that the interests
The difficulties faced by IS researchers and of all stakeholders, not just the shareholders, are
business managers searching for evidence of important to an organisation; and their interests
IS business value at the organisational level should be considered equally in decision-making
are quite similar to those faced by management (Donaldson & Preston, 1995; Hubbard, 2009;
science in organisational effectiveness studies. Jensen, 2002; Stoney & Winstanley, 2001).
Understanding the need to have multiple scale Mitchell et al. (1997) describe the three attributes
items in assessing organisational effectiveness of stakeholders: power, legitimacy, and urgency.
and that the construct of organisational effective- Power is about the influence a stakeholder has
ness is multidimensional, the BSC was adopted in over the relationship. Legitimacy is about the
developing the ISOE model. The BSC provides legitimate position a stakeholder holds within the
a strong theoretical foundation for studying the social system. Urgency is about the treatment a
dimensions of effectiveness improvements that stakeholder receives in terms of time sensitivity
are enabled by IS. and criticality. They suggest that a stakeholder
who possesses all three attributes is highly recog-
Balanced Scorecard (BSC) nisable to an organisation, moderately if two, and
weakly if one.
Introduced by Kaplan and Norton in 1992, the Although depicted as a four-perspective frame-
BSC has four perspectives: financial, customer, work, Kaplan and Norton (1996a) mention that,
business process, and learning and growth (Ka- depending on the context, the BSC could accom-
plan & Norton, 1992, 1993, 1996a). Kaplan and modate additional perspectives. Each perspective
Norton (2001) mention that although business could have four to seven measures. They reckon
organisations are keen to keep track of their intan- that it is important to have a mixture of outcome and
gible assets, they face two problems in measuring performance-driven measures. Outcome measures
them: (1) value from intangible assets is indirect, are the lagging indicators and performance-driven
and (2) value varies in different organisational measures are the leading indicators. For example,
contexts. The attempt to capture intangible assets on-time delivery is a leading indicator leading to
is important because it is impossible to manage customer satisfaction, which is a lagging indicator.
them without knowing what they are. Knowing The leading indicators are necessary to illustrate
what the intangible assets are, organisations can what leads to the lagging indicators. Involvement
then manage them better to deliver value. of senior and middle level managers is critical
Originally developed as a performance mea- in identifying the indicators because they are
surement framework, the BSC is built on qual- involved in the formulation of business strategy.
ity and lean management principles, financial As the value of IT lies in its ability to help
economics concepts, and stakeholder theory achieve business objectives and execute business
(Hubbard, 2009; Jensen, 2002; Kaplan, 2009; strategy (Luftman & Brier, 1999), the characteris-
Ulrich et al., 1999). Kaplan (2009) reckons that tics of the BSC make it a candidate model useful
when organisations manage business processes for studying the impact of IT on organisations.
efficiently and effectively to meet stakeholder With the four quadrants of the BSC each points
43
Figure 1. The conceptual model
to a dimension of organisational performance unidimensional approach to measure IS success.

(Hubbard, 2009), the BSC provides pointers to Consistent with discussions of latent model (Law
business managers on what helps create business et al., 1998), superordinate construct (Edwards,
value (Jensen, 2002).Considering the multi- 2001), reflective-reflective second-order model
criteria and multi-objective characteristics of IT (Jarvis et al., 2003), and molecular approach
investment, it is necessary to include a spectrum (Chin & Gopal, 1995); in the conceptual model,
of operational, tactical, and strategic (Dasgupta IS-enabled organisational effectiveness was con-
et al., 1999; Schniederjans & Hamaker, 2003), ceptualised as a second-order construct which
objective and subjective (Thong & Yap, 1996), was manifested by four first-order constructs:
and subjective and perceptual (Ammenwerth et financial, customer, business process, and learn-
al., 2003) measures in IT evaluation. Traditional ing and growth. 5 The first-order constructs had
accounting measures are mainly historical in reflective scale items, and they themselves were
nature. Thus, they do not provide a full descrip- treated as reflective scale items of the second-
tion of the future of strategic IT investment in a order construct.
dynamic and competitive business environment. The conceptualisation of the four first-order
When identifying IT performance measures, a constructs was based on the BSC. Each construct
shift of perspective from historical to future is represented the corresponding perspective of the
necessary. Measures that reflect the future out- BSC and was measured by a unidimensional scale
look of organisations offer more insight into IT consisting of multiple scale items. To assess pre-
performance (Bharadwaj et al., 1999). dictive validity, a criterion construct, appraisal,
was added6. As highlighted by Berghout and
The Original Conceptual Model Remenyi (2005), there is a strong need to de-
velop an evaluation model underpinned by a
Figure 1 depicts the original conceptual model theoretical framework for better evaluation of IT
from which the ISOE model eventually evolved. investment. Adopting the BSC in developing the
Rai et al. (2002) reckons that a multidimensional conceptual model satisfied the calls by IS research-
approach, which takes into account the interde- ers that the relation between a multidimensional
pendencies among the constructs, is better than a construct and its dimensions must be theoreti-
44
cally defined (Diamantopoulos & Siguaw, 2006; The construct of organisational effectiveness is
Law et al., 1998) and that the constructs concep- a perceptual reflection, which is normally captured
tualised based on theoretical frameworks (Segars, by surrogate measures, of people on different
1997), theory (Straub, 1989), and past research things (Walton & Dawson, 2001). Georgopoulos
(Churchill, 1979). and Tannenbaum (1957) suggest that identifica-
The conceptual model also answered Straub’s tion of evaluation criteria should be based on a
(1989) proposal to examine reference discipline conceptual framework. Cameron (1986b), Chel-
theories, in which one can find pointers on the ladurai (1987), and Mahoney (1967) suggest that
definition and operationalisation of constructs the construct of organisational effectiveness is
and, in addition, the relationships among the multidimensional. Mahoney (1967) adds that the
constructs. The following subsections describe variables can be grouped into different dimensions
each of the constructs in the conceptual model. of organisational effectiveness. Together these
dimensions reflect the overall organisational ef-
IS-Enabled Organisational fectiveness.
Effectiveness In the conceptual model, IS-enabled organi-
sational effectiveness was modelled as a second-
Business managers are keen to evaluate organi- order multidimensional latent construct, with each
sational effectiveness based on achievement of of the four BSC perspectives - financial, customer,
long-term and short-term organisational goals. business process, and learning and growth - as one
For example, Palvia et al. (1992) described the of its first-order constructs. Adopting the approach
contribution of a strategic computer system to suggested by Fearon and Philip (1999) to evaluat-
Federal Express Corporation’s organisational ing IS success, improvements in organisational
effectiveness in the form of strategic and specific effectiveness can be considered in three different
organisational benefits. Thus, to operationalise states: efficiency, neutrality, and deficiency. By
the construct, organisational goals must first be comparing expected improvements and realised
defined as they help identify the criteria that are improvements, depending on the extent to which
relevant to the goals (Mahoney, 1967). Without a the expected improvements are realised, an evalu-
clear definition and the relevant evaluation criteria, ation can then be made. When positive improve-
conceptualisation of organisational effectiveness ments in organisational effectiveness are evident,
is unclear (Georgopoulos & Tannenbaum, 1957). the business value of IS is proven.
There has been no common agreement on
organisational effectiveness measures. Very of- Financial
ten, the selection of measures lacks theoretical
justification. Financial measures are heavily used The financial perspective will reflect an improve-
even though these measures hardly reflect effec- ment in bottom line when organisations do well
tiveness thoroughly. As the use of inappropriate in the other BSC perspectives (Jusoh et al., 2008;
effectiveness measures can lead to inaccurate Chen & Mohamed, 2008). The financial perspec-
conclusions or decisions, the importance of reliable tive is a result of key activities in the other perspec-
and valid measures cannot be under-emphasised tives when they have been performed correctly
(Hitt, 1988). The complex nature of organisational (Enderle & Tavis, 1998) and business strategy
effectiveness does not help at all in establishing has been executed successfully (Huang, 2009).
a single theory about it as the availability of a The financial perspective consists of tangible
universal model or a set of criteria is non-existent measures of both short- and long-term financial
(Cameron, 1986b). performance (Bhagwat & Sharma, 2007) and
45
strategy implementation outcomes (Kimiloglu financial performance (Jusoh et al., 2008). Satis-
& Zarali, 2009). fied customers bring organisations continuous
The financial performance indicators, often income (Lee et al., 2008). Therefore, organisations
known as lagging indicators, depict the long-term have to be cognizant of their performance among
planning of organisational financial objectives the target customers (Chen & Mohamed, 2008)
(Cohen et al., 2008). Mooraj et al. (1999) describe by keeping track of customer and market-related
three types of financial performance indicators performance indicators (Kimiloglu & Zarali, 2009;
for the growth, sustain, and harvest stages of a Michalska, 2005).
company. Ratnasingam (2009) further explains Depending on the context in which they are
how these indicators can be used for risk assess- suggested, customer performance indicators can
ment and cost benefit analysis purposes. range from customer service improvement and
There is a heavy use of financial performance customer invoice accuracy (Velcu, 2007); market
indicators (Jusoh et al., 2008), which come in share, customer retention, and customer satisfac-
different forms. For example, revenue, operating tion (Chia et al., 2009; Cohen et al., 2008); repeat
costs, and profitability (Bryant et al., 2004); sales sales, customer profit, and customer complaint
growth, return on investment, revenue to cost, (Huang, 2008); ability to gain contracts, market
online transactions to total revenue, and online share, level of client satisfaction, ability to gain
customers to total customers (Huang, 2008); return repeat business, and reputation in delivering qual-
on investment, average profit, and turnover growth ity services (Chen & Mohamed, 2008); customer
(Chen & Mohamed, 2008); return on investment, satisfaction in terms of time, quality, price, acces-
gross revenue, profit before tax, and cost reduction sibility, reputation, and relationship (Ratnasingam,
(Chia et al., 2009); and cash flow, sales growth, 2009); and ability to solve customer complaints
operating income growth, market share, return and managing customer dissatisfaction (Kimiloglu
on equity and capital employed, profitability, and & Zarali, 2009).
sales turnover (Bhagwat & Sharma, 2007).
Business Process
Customer
To deliver the customer value proposition and meet
In an increasingly competitive market, organi- customer expectations which help promote cus-
sations are more attentive to their customers tomer satisfaction and acquire customer loyalty,
(Enderle & Tavis, 1998), knowing well that to organisations need to have in place well-designed
retain customers, one needs to be customer-centric business processes (Kimiloglu & Zarali, 2009)
(Amaratunga et al., 2002) and to satisfy customer that strengthen business competencies (Bhagwat
needs (Ratnasingam, 2009). To improve sales & Sharma, 2007), enhance operational efficiency
and build customer loyalty, organisations have to and effectiveness (Cohen et al., 2008), and build
consider matters that are important to their target business capabilities (Huang, 2009). Organisations
customers (Bhagwat & Sharma, 2007) and then learn to better align their processes with business
provide them with a value proposition (Cohen et strategies in order to fulfill expectations of share-
al., 2008). holders and customers (Chen & Mohamed, 2008)
The customer perspective provides pointers and create value for customers (Jusoh et al., 2008).
on how a company, having understood what and Ratnasingam (2009) describes two types of
how to attract customers, can design its business business processes: mission-oriented and sup-
processes so as to realise the customer value port. Mission-oriented processes are about the
proposition, which eventually contributes to good endeavour to achieve strategic business goals.
46
Support processes are about the execution of strengthen organisational development (Chen &
routine operational activities. By monitoring Mohamed, 2008) from human capital, informa-
process performance, organisations gather crucial tion capital, and organisational capital (Lee et
information to continue improving their busi- al., 2008; Huang, 2009). Human capital refers to
ness processes. Having identified processes that employee skills and expertise. Information capital
have the most business impact, organisations can refers to information sharing capacity, whereby
redesign them for better financial and customer organisational capital refers to corporate culture,
performance (Lee et al., 2008). leadership, knowledge sharing, and teamwork
Depending on the context in which they are (Park et al., 2005; Ratnasingam, 2009).
suggested, different business process performance Although learning and growth performance
indicators have been used. For example, opera- indicators have not been used as much as that of
tional excellence, effectiveness, and efficiency the other perspectives (Jusoh et al., 2008), various
(Kimiloglu & Zarali, 2009); delivery flexibility, indicators have been used to measure organisa-
urgent order responsiveness, schedule adherence, tional development. For example, team coopera-
forecast accuracy, and new product time-to-market tion, employee empowerment, and employee core
(Park et al., 2005); business process transparency, competency (Huang, 2008); guardianship, training
economies of scale, service time in accounting and participation, sustainability, and resources
tasks, access to information, process management, (Huang et al., 2006); employee satisfaction,
and cycle times (Velcu, 2007); service quality, employee turnover, employee suggestion, and
number of new services, on-time delivery, and employee training (Chia et al., 2009); and em-
waste reduction (Chia et al., 2009); and transaction ployee morale, employee motivation, employee
mechanism, transaction operations, order-taking satisfaction, employee empowerment, employee
quality, and problem-solving capability (Huang, productivity, employee skills, and system capa-
2008). bility in providing rapid, timely, and accurate
information (Chen & Mohamed, 2008).
Learning and Growth
Appraisal
The learning and growth perspective emphasises
that organisational learning and improvement is Macmillan dictionary (2003, p.57) defines ap-
critical to long-term organisational health (Jusoh praisal as “an opinion about how successful, effec-
et al., 2008). To sustain long-term organisational tive etc someone or something is.” The appraisal
growth, organisations should recognise the im- theories of emotions reckon that an appraisal is
portance of continued innovation, improvement, a cognitive evaluation of an event, which can be
and learning (Bhagwat & Sharma, 2007), which self- or third-party initiated or situational (Nerb,
is fundamental to creativity and innovation en- 2007), taking into consideration the relevance and
couragement, competence and capability building, significance of the event to the individual. The
and employee development and learning (Cohen outcomes of the appraisal subsequently trigger
et al., 2008). Otherwise organisations can face changes in emotions (Tong et al., 2007; Kappas,
competitive disadvantage in a highly competitive 2006; Richins, 1997).
marketplace (Kimiloglu & Zarali, 2009). Lazarus (1984) explains that cognitive ap-
The learning and growth perspective helps lay praisal is about a person’s valuation, commitment,
the foundations for the achievement of objectives perception, and thinking of an event. He describes
in the other BSC perspectives (Jusoh et al., 2008) two types of appraisals: primary and secondary.
as this perspective describes how organisations Primary appraisals can be of three types: irrelevant,
47
benign-positive, stressful. Irrelevant appraisals Content refers to the performance outcomes an

happen when an event has no effect as it is not appraisal is based on; for example, goals, objec-
relevant to one; benign-positive when an event tives, and competencies. By comparing the target
results in positive outcomes; and stressful when and achieved outcomes, a conclusion can then be
an event brings harm, threat, and challenge. Sec- made. If what has actually been achieved is better
ondary appraisal occurs when mitigation options than what was targeted, performance can be con-
have to be considered in response to an event that cluded as good. Otherwise, performance is poor.
brings threat and challenge. For example, Beaudry Following the appraisal theories of emotions, when
and Pinsonneault (2005), in a study of how users ERP systems are appraised to have indeed helped
cope with a significant IT event in the workplace, contribute positively to organisational effective-
suggested that a choice was made among four ness improvements, the experience can evoke
adaptation strategies after the users conducted a strong positive emotions such as satisfaction.
primary appraisal of the opportunities and threats
brought by the event and a secondary appraisal
of one’s control over the event. THE ISOE MODEL AND
In a situation of organisational change, em- A NEW THEORY
ployees will first appraise the situation by form-
ing a perception or making a valuation, before Based on the research philosophy of positivism,
deciding on their next course of action (Fugate validation of the conceptual model took the de-
et al., 2008). The appraisal of a situation can be ductive research approach (Sauders et al., 2003).
positive or negative. An event that seems to be Adopting a survey research strategy, a survey
consistent with one’s goals will result in positive questionnaire was designed for data collection
emotions such as excitement, satisfaction, and self- purposes from organisations in Malaysia that had
assurance; otherwise, events that are inconsistent implemented ERP systems. The development of
with one’s goals will arouse negative emotions a survey instrument started with a review of lit-
such as worry, anger, and frustration (Bagozzi & erature to generate an initial pool of items. After
Pieters, 1998; Kuppens et al., 2007; Nerb, 2007). initial item screenings, a pretest, and a pilot test,
Emotions change as one attaches some meaning to a survey instrument was developed and adminis-
the event (Dalakas, 2005). For instance, Hopkins tered to ERP-adopting organisations in Malaysia.
et al. (2006) surveyed 298 retirees in the US to The main study employed a single-informant
examine the impact of availability of resources approach, targeting managing directors (MDs)
such as time, finance, social support, personal and chief executive officers (CEOs) as they are
faith, and physical health on retirement style. normally the key decision-makers in major IS
They reasoned that with more of these resources purchases and are better informed about effective-
available, the appraisal would be more positive; ness improvements at the organisational level. A
otherwise, it would be negative. Positive appraisals total of 147 responses were received.
had traits such as good, enjoyable, hopeful, valu- To examine the dimensionality of individual
able, positive, and exciting. Negative appraisals constructs before a confirmatory factory analysis
had traits such as stressful, threatening, discour- (CFA), a principal components analysis (PCA) was
aging, negative, sad, difficult, more constrained, performed individually to assess each construct.
and harmful. A correlation matrix of first-order constructs
Fletcher (2001) describes that appraisal in- showed some statistically significant correla-
cludes two key elements: process and content. Pro- tions among the constructs. Thus, higher-order
cess refers to how an appraisal is being executed. factor analyses were performed to examine the
48
existence of higher-order factor structures7. The construct of IS-enabled organisational effective-

analyses revealed second-order and third-order ness correlated positively and significantly with
factor models. Statistical results provided evidence the criterion construct, that is, appraisal. Thus,
that each of the four first-order constructs in the the result provided a clear indication that ERP
conceptual model (financial, customer, business systems received positive appraisal from business
process, and learning and growth), which were managers as they attributed the improved or-
conceptualised based on the BSC, had actually ganisational effectiveness to having implemented
consisted of several sub-constructs. Altogether, ERP systems.
there were ten such sub-constructs. Subsequent As a result of model validation, the original
higher-order analyses revealed that these sub- conceptual model was redrawn (Figure 2) and
constructs could be better summarised as three was named the IS for organisational effectiveness
second-order constructs and the three second-order (ISOE) model. The ten first-order constructs, cus-
constructs as one third-order construct. tomer management, customer preferences, finan-
It is evident that the construct of IS-enabled cial returns, cost savings, production efficiency,
organisational effectiveness was indeed multi- administrative efficiency, product innovation,
dimensional; existed as a third-order construct, process innovation, organisational change, and
but not a second-order construct as originally firm competitiveness, are represented at a second-
proposed in the conceptual model; and consisted order level of abstraction via three second-order
of three second-order and ten first-order con- constructs: business sustainability, organisational
structs, but not the four first-order constructs as competence, and market responsiveness. In turn,
originally proposed in the conceptual model. 8 the three second-order constructs are represented
Analyses revealed that each of the four first-order at a third-order level of abstraction via one third-
constructs originally proposed was composed of order construct, that is, IS-enabled organisational
different numbers of sub-constructs. The second- effectiveness.
order factor model of IS-enabled organisational The ISOE model provides business managers,
effectiveness originally proposed was actually a IS researchers, and IS specialist with a proven
third-order factor model. These newly identified model for ex-ante and ex-post IS evaluation. In
first-order and second-order constructs had been ex-ante evaluation, business managers can treat
named accordingly to better reflect the meaning of the individual effectiveness improvements as
the scale items of the newly identified constructs. evaluation criteria when assessing the feasibility
9
Table 1 presents the newly identified first-order of an IS investment. In ex-post evaluation, busi-
and second-order constructs. ness managers can appraise if effectiveness im-
A CFA was then performed, applying a struc- provements have become positive or negative, or
tural modelling technique known as partial least remain neutral, sometime after an IS implementa-
squares (PLS). PLS analyses were performed in tion, and give IS a good appraisal when there are
two stages. First, the measurement model was positive improvements in the ten effectiveness
tested for internal consistency reliability, conver- dimensions.
gent validity, and discriminant validity. Next, the With the expansion of the four-perspective
structural model was examined for correlations BSC to a ten-dimensional model, the ISOE model
among the theoretical constructs by estimating has a significant implication for the development
the path coefficients and R2. CFA results pro- of a new theory. The finding is congruent with
vided evidence for internal consistency reliabil- an observation of Kaplan and Norton (1996a)
ity, convergent validity, discriminant validity, and that the BSC can have more the four perspec-
predictive validity. It was also evident that the tives. The finding also supports that the original
49
Table 1. Newly identified first-order and second-order constructs
Second-order construct First-order construct Item description

Business Sustainability Customer Management Create better value for customers
Improve customer satisfaction
Enhance relationship with customers
Improve quality of customer services
Firm Competitiveness Improve competitive position of the company
Support business growth in new markets
Create strategic advantage to enhance competitiveness
Financial Returns Improve business profit
Increase return on equity
Expand market share
Improve sales revenue
Increase return on investment
Increase return on assets
Organisational Competence Process Innovation Redesign business processes
Introduce changes in workflows and functions
Administrative Efficiency Improve operating efficiency
Enable electronic business operations
Improve productivity in administration
Promote better coordination and communication with sup-
pliers and business partners
Organisational Change Allow corporate services to be shared across different levels
of company
Support better coordination across different levels of
company
Enable different levels of company to work as an integrated
unit
Cost Savings Save costs in operational expenses
Reduce number of employees
Save costs in workforce
Market Responsiveness Product Innovation Improve quality of products
Encourage introduction of new services
Encourage introduction of new products
Customer Preferences Make possible delivery of customised, made-to-order
products
Deliver customised, interactive services to customers
Production Efficiency Improve productivity in production
Provide better production management
Provide better inventory management
Appraisal Nil Overall level of importance
Overall level of satisfaction
Overall level of success
50
Figure 2. The IS for organisational effectiveness (ISOE) model
four perspectives, financial, customer, business benefits. In view of the fact that no past studies
process, and learning and growth, are broad cat- have developed and validated an evaluation model
egories and can be further broken up. In addition, which examines the role of IS on organisational
the finding strengthens the notion that the BSC effectiveness, the ISOE model is a new contribu-
perspectives are interlinked (Kaplan & Norton, tion to ex-ante and ex-post ERP evaluation. The
1992). Analysis shows that there are statistically model can be used by IS researchers, IS specialists,
significant correlations among the ten dimen- and business managers for planning, designing,
sions. Deriving from the finding, a new theory implementing, and appraising ERP systems. The
can be proposed to explain the link between IS ISOE model has also established a new theory,
and the multi-faceted, third-order organisational the ISBV theory, which provides an explanation
effectiveness. The new theory, which is named of how IS contribute value to businesses at the
information system business value theory (ISBV organisational level.
Theory), establishes that evidence of IS business The ISOE model can be used in different
value can be found by searching for positive im- stages of systems development life cycle. First,
provements within the individual dimensions of during the feasibility stage, the model helps jus-
IS-enabled organisational effectiveness. tify an ERP investment, providing insight into
the benefits dimensions of ERP system at the
organisational level. Second, during the planning
DISCUSSION AND CONCLUSION stage, the model serves as a blueprint for strategic
IT planning and better business-ERP alignment.
Without an evaluation model, to justify the need Third, during the design stage, the model can be
for an ERP system or to appraise if the system useful for communication with technical people
has delivered its intended benefits can be dif- about system specifications. Fourth, during the
ficult (Mabert et al., 2000; Sedera et al., 2001). implementation stage, the model can be help-
In addition, an ex-post evaluation is critical to ful for understanding implementation issues or
assess the extent of realisation of the intended any changes that need to be made to the system
51
specifications or implementation process. Lastly, can serve as evaluation criteria for ex-ante and
after the system has been implemented, the model ex-post IS evaluation purposes.
can be used to appraise if the intended benefits In summary, the ISOE model has demonstrated
have been realised. If not, an investigation can that the business value of IS is exhibited in the
be initiated to understand why and then make the form of IS-enabled organisational effectiveness.
necessary corrections. A successful IS implementation brings positive
A key advantage of the ISOE model is its improvements in the ten effectiveness dimensions
generic applicability as the model points to the at the organisational level. The model has also
generic objectives of organisations implement- confirmed the specific effectiveness improve-
ing ERP systems. The model, with its validated ments within the ten effectiveness dimensions.
constructs and reliable scale items, provides IS Each specific effectiveness improvement implies
researchers, IS specialists and business managers an intended business benefit as a result of IS. The
a uniform and systematic approach to appraising intended business benefits are signs of IS busi-
ERP performance. Without an evaluation model, ness value, indicating clearly how IS help create
it can be difficult to systematically prove the busi- business value. The role of IS on organisational
ness value of ERP systems. With so many different effectiveness is clear. IS researchers, IS specialists,
evaluation criteria and approaches, performance and business managers can now measure it, dem-
evaluation can be very confusing for business onstrate it, and bust the ‘IT doesn’t matter’ myth.
managers. Thus, the model is a major step towards
comparisons across different ERP systems, mini-
mising conflicting results when different evalu- REFERENCES
ation criteria are used. In critical success factor
(CSF) studies of ERP systems, the construct of Amaratunga, D., Haigh, R., Sarshar, M., & Baldry,
IS-enabled organisational effectiveness can also D. (2002). Application of the balanced scorecard
serve as a readily-available dependent variable. concept to develop a conceptual framework to
The ISOE model shows that the construct of measure facilities management performance
IS-enabled organisational effectiveness is multi- within NHS facilities. International Journal of
dimensional. The ten first-order constructs can be Health Care Quality Assurance, 15(4), 141–151.
better represented at a higher level of abstraction doi:10.1108/09526860210431805
through three second-order constructs. And, the Ammenwerth, E., Graber, S., Herrmann, G.,
three second-order constructs through a third-order Burkle, T., & Konig, J. (2003). Evaluation of
construct. What started initially as a four-dimen- health Information Systems - Problems and
sional IS-enabled organisational effectiveness has challenges. International Journal of Medical
proven to be ten-dimensional. Thus, it is evident Informatics, 71(2/3), 125–135. doi:10.1016/
that there are multiple effectiveness dimensions S1386-5056(03)00131-X
in which IS add value at the organisational level.
In addition, The 30 scale items for each of the Bagozzi, R., & Pieters, R. (1998). Goal-directed
unidimensional first-order constructs are more emotions. Cognition and Emotion, 12(1), 1–26.
comprehensive and representative of ERP suc- doi:10.1080/026999398379754
cess, and importantly, help overcome the problem
of having just one scale item or irrelevant scale
items to measure ERP success. The surrogates
of improvements in organisational effectiveness
52
Beaudry, A., & Pinsonneault, A. (2005). Un- Cameron, K. (1986b). Effectiveness as paradox:
derstanding user responses to Information Consensus and conflict in conceptions of organisa-
Technology: A coping model of user adaptation. tional effectiveness. Management Science, 32(5),
Management Information Systems Quarterly, 539–553. doi:10.1287/mnsc.32.5.539
29(3), 493–524.
Chelladurai, P. (1987). Multidimensionality and
Berghout, E., & Remenyi, D. (2005). The eleven multiple perspectives of organisational effective-
years of the European conference on it evalua- ness. Journal of Sport Management, 1(1), 37–47.
tion: Retrospectives and perspectives for possible
Chen, L., & Mohamed, S. (2008). Contribution
future research. Electronic Journal of Information
of knowledge management activities to organi-
Systems Evaluation, 8(2), 81-98.
sational business performance. Journal of Engi-
Bernroider, E. W. N., & Koch, S. (2002). A frame- neering. Design and Technology, 6(3), 269–285.
work for the selection of ERP packages for small doi:10.1108/17260530810918289
to medium and large organisations. In Hossain,
Chia, A., Goh, M., & Hum, S. (2009). Perfor-
L., Patrick, J. D., & Rashid, M. A. (Eds.), Enter-
mance measurement in supply chain entities:
prise resource planning: Global opportunities
Balanced scorecard perspective. Benchmark-
& challenges (pp. 182–196). Hershey, PA: Idea
ing: An International Journal, 16(5), 605–620.
Group Publishing.
doi:10.1108/14635770910987832
Bhagwat, R., & Sharma, M. K. (2007). Perfor-
Chin, W. W., & Gopal, A. (1995). Adoption
mance measurement of supply chain management:
intention in GSS: Relative importance of be-
A balanced scorecard approach. Computers & In-
liefs. DATA BASE Advances, 26(2/3), 42–64.
dustrial Engineering, 53(1), 43–62. doi:10.1016/j.
doi:10.1145/217278.217285
cie.2007.04.001
Churchill, G. A. (1979). A paradigm for develop-
Bharadwaj, A., Bharadwaj, S., & Konsynski, B.
ing better measures of marketing constructs. JMR,
(1999). Information Technology effects on firm
Journal of Marketing Research, 16(1), 64–73.
performance as measured by Tobin’s q. Manage-
doi:10.2307/3150876
ment Science, 45(6), 1008–1024. doi:10.1287/
mnsc.45.7.1008 Cohen, S., Thiraios, D., & Kandilorou, M. (2008).
Performance parameters interrelations from a bal-
Bryant, L., Jones, D. A., & Widener, S. K. (2004).
anced scorecard perspective: An analysis of Greek
Managing value creation within the firm: An
companies. Managerial Auditing Journal, 23(5),
examination of multiple performance measures.
485–503. doi:10.1108/02686900810875307
Journal of Management Accounting Research,
16(1), 107–131. doi:10.2308/jmar.2004.16.1.107 Cronk, M. C., & Fitzgerald, E. P. (2002). Con-
structing a theory of IS business value from the
Cameron, K. (1978). Organisational effective-
literature. Electronic Journal of Business Research
ness: Its measurement and prediction in higher
Methods, 1(1), 11–17.
education. Administrative Science Quarterly, 32,
604–632. doi:10.2307/2392582 Dalakas, V. (2005). The effect of cognitive ap-
praisals on emotional responses during service
Cameron, K. (1986a). A study of organisational
encounters. Services Marketing Quarterly, 27(1),
effectiveness and its predictors. Management Sci-
23–41. doi:10.1300/J396v27n01_02
ence, 32(1), 87–112. doi:10.1287/mnsc.32.1.87
53
Dasgupta, S., Sarkis, J., & Talluri, S. (1999). In- Georgopoulos, B., & Tannenbaum, A. (1957).
fluence of information technology investment on A study of organisational effectiveness. Ameri-
firm productivity: A cross-sectional study. Logis- can Sociological Review, 22(5), 534–540.
tics Information Management, 12(1/2), 120–129. doi:10.2307/2089477
doi:10.1108/09576059910256493
Grover, V., Jeong, S. R., & Segars, A. H. (1996).
Diamantopoulos, A., & Siguaw, J. A. (2006). Information systems effectiveness: The construct
Formative versus reflective indicators in organi- space and patterns of application. Information
sational measure development: A comparison and & Management, 31(4), 177–191. doi:10.1016/
empirical illustration. British Journal of Man- S0378-7206(96)01079-8
agement, 17(4), 263–282. doi:10.1111/j.1467-
Gunasekaran, A., Ngai, E. W. T., & McGaughey,
8551.2006.00500.x
R. E. (2006). Information Technology and systems
Donaldson, T., & Preston, L. E. (1995). The justification: A review for research and applica-
stakeholder theory of the corporation: Concepts, tions. European Journal of Operational Research,
evidence, and implications. Academy of Manage- 173(3), 957–983. doi:10.1016/j.ejor.2005.06.002
ment Review, 20(1), 65–91.
Henri, J. F. (2004). Performance measurement
Edwards, J. R. (2001). Multidimensional con- and organisational effectiveness: Bringing
structs in organisational behavior research: the gap. Managerial Finance, 30(6), 93–123.
An integrative analytical framework. Orga- doi:10.1108/03074350410769137
nizational Research Methods, 4(2), 144–192.
Hitt, M. (1988). The measuring of organisational
doi:10.1177/109442810142004
effectiveness: Multiple domains and constituen-
Enderle, G., & Tavis, L. A. (1998). A balanced cies. Management International Review, 28(2),
concept of the firm and the measurement of 28–40.
its long-term planning and performance. Jour-
Hopkins, C. D., Roster, C. A., & Wood, C.
nal of Business Ethics, 17(11), 1129–1144.
M. (2006). Making the transition to retire-
doi:10.1023/A:1005746212024
ment: Appraisals, post-transition lifestyle,
Fearon, C., & Philip, G. (1999). An empirical and changes in consumption patterns. Jour-
study of the use of EDI in supermarket chains nal of Consumer Marketing, 23(2), 89–101.
using a new conceptual framework. Jour- doi:10.1108/07363760610655023
nal of Information Technology, 14(1), 3–21.
Huang, H. (2009). Designing a knowledge-
doi:10.1080/026839699344719
based system for strategic planning: A balanced
Fletcher, C. (2001). Performance appraisal scorecard perspective. Expert Systems with
and management: The developing research Applications, 36(1), 209–218. doi:10.1016/j.
agenda. Journal of Occupational and Or- eswa.2007.09.046
ganizational Psychology, 74(4), 473–487.
Huang, L. (2008). Strategic orientation and per-
doi:10.1348/096317901167488
formance measurement model in Taiwan’s travel
Fugate, M., Kinicki, A. J., & Prussia, G. E. (2008). agencies. Service Industries Journal, 28(10),
Employee coping with organisational change: An 1357–1383. doi:10.1080/02642060802250179
examination of alternative theoretical perspectives
and models. Personnel Psychology, 61(1), 1–36.
doi:10.1111/j.1744-6570.2008.00104.x
54
Huang, S., Lee, C., & Kao, A. (2006). Balancing Kaplan, R., & Norton, D. (2001). Transform-
performance measures for information security ing the balanced scorecard from performance
management: A balanced scorecard framework. measurement to strategic management: Part I.
Industrial Management & Data Systems, 106(2), Accounting Horizons, 15(1), 87–104. doi:10.2308/
242–255. doi:10.1108/02635570610649880 acch.2001.15.1.87
Hubbard, G. (2009). Measuring organisational Kaplan, R. S. (2009). Conceptual foundations
performance: Beyond the triple bottom line. of the balanced scorecard. In Chapman, C. S.,
Business Strategy and the Environment, 18(3), Hopwood, A. G., & Shields, M. D. (Eds.), Hand-
177–191. doi:10.1002/bse.564 book of management accounting research (pp.
1253–1271). Oxford, UK: Elsevier.
Jarvis, C. B., MacKenzie, S. B., & Podsakoff, P. M.
(2003). A critical review of construct indicators and Kappas, A. (2006). Appraisals are direct, im-
measurement model misspecification in marketing mediate, intuitive, and unwitting … and some
and consumer research. The Journal of Consumer are reflective…. Cognition and Emotion, 20(7),
Research, 30(2), 199–218. doi:10.1086/376806 952–975. doi:10.1080/02699930600616080
Jensen, M. C. (2002). Value maximization, stake- Kimiloglu, H., & Zarali, H. (2009). What
holder theory, and the corporate objective func- signifies success in e-CRM? Marketing In-
tion. Business Ethics Quarterly, 12(2), 235–256. telligence & Planning, 27(2), 246–267.
doi:10.2307/3857812 doi:10.1108/02634500910945011
Jusoh, R., Ibrahim, D. N., & Zainuddin, Y. (2008). Klecun, E., & Cornford, T. (2005). A critical
The performance consequences of multiple perfor- approach to evaluation. European Journal of In-
mance measures usage: Evidence from the Malay- formation Systems, 14(3), 229–243. doi:10.1057/
sian manufacturers. International Journal of Pro- palgrave.ejis.3000540
ductivity and Performance Management, 57(2),
Kohli, R., & Grover, V. (2008). Business value of
119–136. doi:10.1108/17410400810847393
IT: An essay on expanding research directions to
Kaplan, R., & Norton, D. (1992). The balanced keep up with the times. Journal of the Association
scorecard - Measures that drive performance. for Information Systems, 9(1), 23–39.
Harvard Business Review, 70(1), 71–79.
Kuppens, P., van Mechelen, I., Smits, D. M., De
Kaplan, R., & Norton, D. (1993). Putting the Boeck, P., & Ceulemans, E. (2007). Individual
balanced scorecard to work. Harvard Business differences in patterns of appraisal and anger ex-
Review, 71(5), 134–142. perience. Cognition and Emotion, 21(4), 689–713.
doi:10.1080/02699930600859219
Kaplan, R., & Norton, D. (1996a). The balanced
scorecard: Translating strategy into action. Har- Law, C. C. H., & Ngai, E. W. T. (2005). IT busi-
vard Business School Press. ness value research: A critical review and research
agenda. International Journal of Enterprise
Kaplan, R., & Norton, D. (1996b). Using the
Information Systems, 1(3), 35–55. doi:10.4018/
balanced scorecard as a strategic management
jeis.2005070103
system. Harvard Business Review, 74(1), 75–85.
55
Law, K. S., Wong, C. S., & Mobley, W. H. (1998). Marthandan, G., & Tang, C. M. (2009). The
Toward a taxonomy of multidimensional con- multi-faceted characteristics of IT-enabled orga-
structs. Academy of Management Review, 23(4), nizational effectiveness. Symposium on Progress
741–755. in Information and Communication Technology
(SPICT 2009), 7-8 December 2009, Kuala Lum-
Lazarus, R. S. (1984). Stress, appraisal, and cop-
pur, Malaysia.
ing. New York, NY: Springer.
Marthandan, G., & Tang, C. M. (2010). Informa-
Lederer, A. L., & Mirani, R. (1995). Anticipat-
tion Technology evaluation: Issues and challenges.
ing the benefits of proposed information sys-
Journal of Systems and Information Technology,
tems. Journal of Information Technology, 10(3),
12(1), 37–55. doi:10.1108/13287261011032643
159–169. doi:10.1057/jit.1995.20
Martinsons, M., & Martinsons, V. (2002). Rethink-
Lee, A. H., Chen, W., & Chang, C. J. (2008).
ing the value of IT, again. Communications of the
A fuzzy AHP and BSC approach for evaluating
ACM, 45(7), 25–28. doi:10.1145/514236.514256
performance of IT department in the manufac-
turing industry in Taiwan. Expert Systems with Michalska, J. (2005). The usage of the balanced
Applications, 34(1), 96–107. doi:10.1016/j. scorecard for the estimation of the enterprise’s
eswa.2006.08.022 effectiveness. Journal of Materials Processing
Technology, 162-163, 751–758. doi:10.1016/j.
Love, P. E. D., Irani, Z., & Edwards, D. J. (2004).
jmatprotec.2005.02.227
Industry-centric benchmarking of Information
Technology benefits, costs and risks for small- Mitchell, R. K., Agle, B. R., & Wood, D. J. (1997).
to-medium sized enterprises in construction. Toward a theory of stakeholder identification and
Automation in Construction, 13(4), 507–524. salience: Defining the principle of who and what
doi:10.1016/j.autcon.2004.02.002 really counts. Academy of Management Review,
22(4), 853–886.
Luftman, J., & Brier, T. (1999). Achieving and
sustaining business-IT alignment. California Mooraj, S., Oyon, D., & Hostettler, D. (1999). The
Management Review, 42(1), 109–122. balanced scorecard: A necessary good or an unnec-
essary evil. European Management Journal, 17(5),
Mabert, V. A., Soni, A., & Venkataramanan, M.
481–491. doi:10.1016/S0263-2373(99)00034-1
A. (2000). Enterprise resource planning survey
of U.S. manufacturing firms. Production and Nerb, J. (2007). Exploring the dynamics of the
Inventory Management Journal, 41(2), 52–58. appraisal-emotion relationship: A constraint
satisfaction model of the appraisal process.
Macmillan. (2003). Macmillan English diction-
Cognition and Emotion, 21(7), 1382–1413.
ary for advanced learners. Macmillan Publishers
doi:10.1080/02699930701193419
Limited.
Palvia, P. C., Perkins, J. A., & Zeltmann, S. M.
Mahoney, T. A. (1967). Managerial percep-
(1992). The PRISM system: A key to organisa-
tions of organisational effectiveness. Manage-
tional effectiveness at Federal Express Corpora-
ment Science, 14(2), B76–B91. doi:10.1287/
tion. Management Information Systems Quarterly,
mnsc.14.2.B76
16(3), 277–292. doi:10.2307/249529
56
Pang, A. (2009). WW ERP applications 2009-2013 Sedera, D., Gable, G., & Rosemann, M. (2001).
forecast and 2008 vendor shares (pp. 1–16). IDC A balanced scorecard approach to enterprise
Insights. systems performance measurement. In, G. Finnie,
D. Cecez-Kecmanovic, & B. Lo (Eds.), Proceed-
Park, J. H., Lee, J. K., & Yoo, J. S. (2005). A
ings of the Twelfth Australasian Conference on
framework for designing the balanced supply
Information Systems, Coffs Harbour Australia,
chain scorecard. European Journal of Information
5-7 December.
Systems, 14(4), 335–346. doi:10.1057/palgrave.
ejis.3000544 Segars, A. H. (1997). Assessing the unidimension-
ality of measurement: A paradigm and illustra-
Ragowsky, A., Stern, M., & Adams, D. A. (2000).
tion within the context of Information Systems
Relating benefits from using IS to an organisation’s
research. Omega, 25(1), 107–121. doi:10.1016/
operating characteristics: Interpreting results from
S0305-0483(96)00051-5
two countries. Journal of Management Informa-
tion Systems, 16(4), 175–194. Steers, R. M. (1975). Problems in the measure-
ment of organisational effectiveness. Admin-
istrative Science Quarterly, 20(4), 546–558.
doi:10.2307/2392022
tion Systems Research, 13(1), 50–69. doi:10.1287/ Stefanou, C. J. (2001). A framework for the
isre.13.1.50.96 ex-ante evaluation of ERP software. European
Journal of Information Systems, 10(4), 204–215.
Ratnasingam, P. (2009). Service quality man-
doi:10.1057/palgrave.ejis.3000407
agement applying the balanced scorecard: An
exploratory study. International Journal of Stoney, C., & Winstanley, D. (2001). Stakehold-
Commerce and Management, 19(2), 127–136. ing: Confusion or utopia? Mapping the conceptual
doi:10.1108/10569210910967888 terrain. Journal of Management Studies, 38(5),
603–626. doi:10.1111/1467-6486.00251
Richins, M. L. (1997). Measuring emotions in the
consumption experience. The Journal of Consum- Straub, D. (1989). Validating instruments in
er Research, 24(2), 127–146. doi:10.1086/209499 MIS research. Management Information Systems
Quarterly, 13(2), 147–169. doi:10.2307/248922
Saarinen, T. (1996). An expanded instrument for
evaluating information system success. Informa- Tallon, P. P., Kraemer, K. L., & Gurbaxani, V.
tion & Management, 31(2), 103–118. doi:10.1016/ (2000). Executives’ perceptions of the business
S0378-7206(96)01075-0 value of information technology: A process-orient-
ed approach. Journal of Management Information
Sauders, M., Lewis, P., & Thornhill, A. (2003).
Systems, 16(4), 145–173.
Research methods for business students (3rd ed.).
Harlow, UK: Prentice Hall. Tang, C. M., & Marthandan, G. (2011). An analyti-
cal model to measure IS-enabled organizational
Schniederjans, M. J., & Hamaker, J. L. (2003). A
effectiveness. International Journal of Enterprise
new strategic Information Technology investment
model. Management Decision, 41(1/2), 8–17.
jeis.2011040104
doi:10.1108/00251740310445527
57
Tangpong, C. (2008). IT-performance paradox Bradley, R., Pridmore, J., & Byrd, T. (2006).
revisited: Resource-based and prisoner’s dilemma Information systems success in the context of
perspectives. Journal of Applied Management and different corporate cultural types: An empirical
Entrepreneurship, 13(1), 35–49. investigation. Journal of Management Information
Systems, 23(2), 267–294. doi:10.2753/MIS0742-
Thong, J. Y. L., & Yap, C. S. (1996). Informa-
1222230211
tion Systems effectiveness: A user satisfaction
approach. Information Processing & Man- Brynjolfsson, E. (1993). The productivity paradox
agement, 32(5), 601–610. doi:10.1016/0306- of information technology. Communications of the
4573(96)00004-0 ACM, 36(12), 67–77. doi:10.1145/163298.163309
Tillquist, J., & Rodgers, W. (2005). Using asset Brynjolfsson, E., & Hitt, L. M. (2000). Beyond
specificity and asset scope to measure the value computation: Information Technology, organisa-
of IT. Communications of the ACM, 48(1), 75–80. tional transformation and business performance.
doi:10.1145/1039539.1039542 The Journal of Economic Perspectives, 14(4),
23–48. doi:10.1257/jep.14.4.23
Tong, M., Bishop, G., Enkelmann, H., Why, Y.,
Diong, S., Khader, M., & Ang, J. (2007). Emotion Chan, Y. (2000). IT value: The great divide be-
and appraisal: A study using ecological momen- tween qualitative and quantitative and individual
tary assessment. Cognition and Emotion, 21(7), and organisational measures. Journal of Manage-
1361–1381. doi:10.1080/02699930701202012 ment Information Systems, 16(4), 225–261.
Ulrich, D., Zenger, J., & Smallwood, N. (1999). Dedrick, J., Gurbaxani, V., & Kraemer, K. L.
Results-based leadership. Boston, MA: Harvard (2003). Information technology and economic
Business School Press. performance: A critical review of the empirical
evidence. ACM Computing Surveys, 35(1), 1–28.
Velcu, O. (2007). Exploring the effects of ERP
doi:10.1145/641865.641866
systems on organisational performance: Evidence
from Finnish companies. Industrial Manage- DeLone, W. H., & McLean, E. R. (1992). Informa-
ment & Data Systems, 107(9), 1316–1334. tion Systems success: The quest for the dependent
doi:10.1108/02635570710833983 variable. Information Systems Research, 3(1),
60–95. doi:10.1287/isre.3.1.60
Walton, E. J., & Dawson, S. (2001). Managers’
perceptions of criteria of organisational effec- DeLone, W. H., & McLean, E. R. (2003). The
tiveness. Journal of Management Studies, 38(2), DeLone and McLean model of Information
173–199. doi:10.1111/1467-6486.00233 Systems success: A ten-year update. Journal of
Management Information Systems, 19(4), 9–30.
Dunn, S. C., Seaker, R. F., & Waller, M. A. (1994).
ADDITIONAL READING Latent variables in business logistics research:
Scale development and validation. Journal of
Barua, A., Kriebel, C. H., & Mukhopadhyay, T. Business Logistics, 15(2), 145–172.
(1995). Information Technologies and business
value: An analytic and empirical investigation.
Information Systems Research, 6(1), 3–23.
doi:10.1287/isre.6.1.3
58
Evans, V. P. (1999). Higher-order factor analysis: Irani, Z., & Love, P. E. D. (2001). The propaga-
An introductory primer. Paper presented at the tion of technology management taxonomies for
Annual Meeting of the Southwest Educational evaluating investments in Information Systems.
Research Association, San Antonio, TX, Janu- Journal of Management Information Systems,
ary 21-23. 17(3), 161–177.
Gable, G. G., Sedera, D., & Chan, T. (2003). En- Kohli, R., & Devaraj, S. (2003). Measuring In-
terprise systems success: A measurement model. formation Technology payoff: A meta-analysis
Proceedings of the 24th International Conference of structural variables in firm-level empirical
on Information Systems, (pp. 576-591). research. Information Systems Research, 14(2),
127–145. doi:10.1287/isre.14.2.127.16019
Gable, G. G., Sedera, D., & Chan, T. (2008). Re-
conceptualising Information System success: The Lyytinen, K., & Hirschheim, R. (1987). Informa-
IS-impact measurement model. Journal of the As- tion Systems failures: A survey and classification
sociation for Information Systems, 9(7), 377–408. of the empirical literature. Oxford Surveys in
Information Technology, 4, 257–309.
Gray, B. T. (1997). Higher-order factor analysis.
Paper presented at the annual meeting of the Mirani, R., & Lederer, A. (1998). An instrument
Southwest Educational Research Association, for assessing the organisational benefits of IS
Austin, TX. January 23-25, 1997. projects. Decision Sciences, 29(4), 803–838.
doi:10.1111/j.1540-5915.1998.tb00878.x
Hair, J. F., Black, B., Babin, B., Anderson, R.
E., & Tatham, R. L. (2005). Multivariate data Nicolaou, A. I., & Bhattacharya, S. (2006).
analysis (6th ed.). Upper Saddle River, NJ: Pear- Organisational performance effects of ERP sys-
son Education. tems usage: The impact of post-implementation
changes. International Journal of Accounting
Hamilton, S., & Chervany, N. (1981a). Evaluat-
Information Systems, 7(1), 18–35. doi:10.1016/j.
ing Information System effectiveness - Part I:
accinf.2005.12.002
Comparing evaluation approaches. Management
Information Systems Quarterly, 5(3), 55–69. Poston, R., & Grabski, S. (2001). Financial impacts
doi:10.2307/249291 of enterprise resource planning implementations.
International Journal of Accounting Information
Hamilton, S., & Chervany, N. (1981b). Evaluat-
Systems, 2(4), 271–294. doi:10.1016/S1467-
ing Information System effectiveness - part II:
0895(01)00024-0
Comparing evaluator viewpoints. Management
Information Systems Quarterly, 5(4), 79–86. Seddon, P., Staples, S., Patnayakuni, R., & Bowtell,
doi:10.2307/249329 M. (1999). Dimensions of Information Systems
success. Communications of the AIS, 2(20), 1–39.
Hunton, J. E., Lippincott, B., & Reck, J. L. (2003).
Enterprise resource planning systems: Comparing Seddon, P. B., Graeser, V., & Willcocks, L. P.
firm performance of adopters and nonadopters. (2002). Measuring organisational IS effectiveness:
International Journal of Accounting Information An overview and update of senior management
Systems, 4(3), 165–184. doi:10.1016/S1467- perspectives. The Data Base for Advances in
0895(03)00008-3 Information Systems, 33(2), 11–28.
59
Segars, A. H., & Grover, V. (1998). Strategic business process, and learning and growth. The
Information Systems planning success: An in- BSC has been proposed for IT evaluation in
vestigation of the construct and its measurement. three areas: the IT department and its functions;
Management Information Systems Quarterly, systems implementation process or stages; and
22(2), 139–163. doi:10.2307/249393 IS performance.
Information Systems (IS) and Information
Stockdale, R., & Standing, C. (2006). An interpre-
Technology (IT): There is a difference between
tive approach to evaluating information systems:
the terms IS and IT. IT in general refers to the
A content, context, process framework. Euro-
technical systems, whereas IS extends the concept
pean Journal of Operational Research, 173(3),
to include the soft systems, e.g. human activity
1090–1102. doi:10.1016/j.ejor.2005.07.006
systems, in which technology is being imple-
Tanriverdi, H. (2005). Information Technology re- mented. However, these two terms are often used
latedness knowledge management capability and synonymously. In this chapter, these two terms
performance of multibusiness firms. Management are used interchangeably.
Information Systems Quarterly, 29(2), 311–334. Information System Business Value (ISBV)
Theory: A theory which establishes that evidence
Tanriverdi, H. (2006). Performance effects of
of IS business value can be found by searching
Information Technology synergies in multibusi-
for positive improvements within the individual
ness firms. Management Information Systems
dimensions of IS-enabled organisational effec-
Quarterly, 30(1), 57–77.
tiveness.
van Grembergen, W., Saull, R., & De Haes, S. Information Systems for Organisational Ef-
(2003). Linking the IT balanced scorecard to the fectiveness (ISOE) Model: As an IS evaluation
business objectives at a major Canadian financial model, the ISOE model depicts one third-order
group. Journal of Information Technology Cases construct (IS-enabled organisational effective-
and Applications, 5(1), 23–45. ness); three second-order constructs (business
sustainability, organisational competence, and
Weill, P. (1992). The relationship between invest-
market responsiveness); and ten first-order con-
ment in information technology and firm perfor-
structs (customer management, customer prefer-
mance: A study of the valve manufacturing sector.
ences, financial returns, cost savings, production
efficiency, administrative efficiency, product
doi:10.1287/isre.3.4.307
innovation, process innovation, organisational
Wetzels, M., Odekerken-Schröder, G., & van change, and firm competitiveness). The model also
Oppen, C. (2009). Using PLS path modeling for includes a criterion construct, appraisal, which is
assessing hierarchical construct models: Guide- for predictive validity assessment.
lines and empirical illustration. Management Multidimensional Construct: Suggesting
Information Systems Quarterly, 33(1), 177–195. that several interrelated dimensions can be con-
ceptualised at a higher level of abstraction in the
form of a construct, Law et al. (1998) reckon that
such a construct can be considered a multidimen-
KEY TERMS AND DEFINITIONS sional construct. A multidimensional construct is
parsimonious.
Balanced Scorecard (BSC) Framework:
Predictive Validity: Predictive validity is
Introduced by Kaplan and Norton in 1992, the
a form of criterion-related validity. Dunn et al.
BSC has four perspectives: financial, customer,
(1994) explain that criterion-related validity ex-
60
amines the correlation between a predictor variable Technology (SPICT 2009), 7-8 December
and its criterion variable. The term concurrent 2009, Kuala Lumpur, Malaysia as well as
validity is used when the criterion variable is in Tang, C.M., & Marthandan, G. (2011). An
current state. The term predictive validity is used analytical model to measure IS-enabled
when the criterion variable is in future state. organisational effectiveness. International
Superordinate Construct: A superordinate Journal of Enterprise Information Systems,
construct is manifested by several dimensions, 7(2), 50-65.
with arrows moving from the construct to its 5
The use of higher-order constructs is evi-
dimensions. In other words, the dimensions are dent in research models of past studies. For
the observed variables of the higher-order con- example, IT relatedness as a second-order
struct. A superordinate construct can be treated construct with four first-order constructs
as a second-order factor and its dimensions the (Tanriverdi, 2006); knowledge management
first-order factors. The first-order factors will have capability as a second-order construct with
the scale items as observed variables. three first-order constructs (Tanriverdi,
2005); and online experiential value as a
fourth-order construct with first, second,
ENDNOTES and third-order constructs (Wetzels et al.,
2009).
1
For examples of conflicting study findings on 6
Post-implementation evaluation to appraise
the business value of IS, refer to Marthandan, the performance of ERP systems in terms of
G., & Tang, C.M. (2010). Thirst for business organisational effectiveness improvements
value of information systems. International will only take place some time after the
Journal of Technology Diffusion, 1(1), 28-40. systems have been implemented. Thus, the
2
For detailed discussion of IS evaluation term predictive validity is used.
methods, approaches, and types, refer to 7
A higher-order factor analysis can be per-
Marthandan, G., & Tang, C.M. (2010). In- formed after the factors have been extracted
formation systems evaluation: An ongoing to explore possible higher-order factors
measure. International Journal of Business (Evans, 1999; Gray, 1997)
Information Systems, 6(3), 336-353. 8
It is common to find the number of constructs
3
For detailed discuss of IS evaluation issues originally proposed in a research model to
and challenges, refer to Marthandan, G., & change as a result of factor analysis; that is,
Tang, C.M. (2010). Information technology the number of constructs may be reduced
evaluation: Issues and challenges. Journal (Gable et al., 2003) or increased (Cohen et
of Systems and Information Technology, al., 2008).
12(1), 37-55. 9
After deciding on the number of factors to
4
For detailed description of scale develop- be retained, it is then necessary to interpret
ment, data collection, and data analysis, refer and label the factors based on the meaning
to Marthandan, G., & Tang, C.M. (2009). The of the scale items loaded on the respective
multi-faceted characteristics of IT-enabled factors (Evans, 1999; Gray, 1997; Hair et
organisational effectiveness. Symposium on al., 1998).
Progress in Information and Communication
61
62
Chapter 4
Information Systems Success:
A Review from a Bibliometric
Analysis Focus
Hugo Martinez
Universidad Industrial de Santander, Colombia
Luis Becerra
Jaime Camacho
ABSTRACT
This chapter characterizes the publication activity of Information Systems success by data collected from
the Science Citation Index and Social Science Citation Index databases from ISI Web of Science during
2001 to 2010. It provides insights into the research activities of Information Systems success research
and identifies patterns, tendencies, or regularities existing in the literature related to authors, journals,
institutes, and countries. A co-ocurrence analysis of keywords is made to indicate the core themes re-
search areas in the literature and new emerging topics. It is found that the IS success literature has a
positive growth rate, and it is likely to continue with this tendency in the future. However, the analysis
shows the need to set up a common language framework that serves as a guide to researchers to develop
a most mature body of knowledge.
INTRODUCTION fast in recent decades and has impacted a variety

of fields and contexts such as web-based and e-
Information systems success has become one of commerce (DeLone & McLean, 2004; Garrity,
the most important streams of research in the IS Glassberg, Kim, Sanders, & Shin, 2005), eGover-
literature (Urbach, Smolnik, & Riempp, 2009). ment (Wang & Liao, 2008), clinical and health
Studies on information systems success has grown (Heeks, 2006; van der Meijden, Tange, Troost, &
Hasman, 2003; Page, 2000), inter-organizational
(Lu, Huang, & Heng, 2006), project management
DOI: 10.4018/978-1-4666-0170-3.ch004
Information Systems Success
(Yen, Li, & Niehoff, 2008), developing countries comparison with more quantitative data (Moed,
(Heeks, 2002), and measurement (DeLone & De Bruin, Nederhof, & Tijssen, 1991). Then, an
McLean, 2004; Floropoulos, Spathis, Halvatzis, attempt is made to carry forward bibliometric
& Tsipouridou, 2010; Jen & Chao, 2008; Petter, analysis of the ‘IS success’ literature during the
DeLone, & McLean, 2008). years 2001 to 2010 to obtain valuable results in
Despite of a high growth rate, the assessment order to generate the interest from new researcher
of the research productivity in a quantitatively and continue supporting so far been.
and systematic way about the scientific global The Science Citation Index -SCI and Social
production is not found in the literature, at the Sciences Citation Index -SSCI databases from
best of our knowledge. The scarcity in this type of the Institute for Scientific Information (ISI) Web
information related to IS success research creates of Science were used to retrieve the data. These
the need for a systematic global analysis which databases are one of the most important and fre-
rather than individual studies, allows exploring quently used sources of research information in
the intellectual core and some properties of the all studying fields (Bayer & Folger, 1966; Braun,
research stream (Holsapple, 2008). This kind of Schubert, & Kostoff, 2000). The search equation
analysis offers support to practitioners and theo- ‘information AND system* AND success’ brought
rists giving insights about the positioning of the 4183 documents including articles, proceeding
scientific community and allowing the creation papers, reviews, editorial material, book review,
of research agendas, contact making and social letters, and biographical items; from them and
network creation at the individual, institutional after filtering by ‘document type’ option in ISI
and national levels. For this approach, the meth- WoS, only the 3250 articles were analyzed and
ods, principles and techniques related to biblio- evaluated to determine patterns and quantitative
metrics used in different disciplines of science characteristics of IS success research.
and engineering (Keiser & Utzinger, 2005; Tang The objectives of this study are: (1) to explore
& Thelwall, 2003) were used in the IS success the quantity and spread of the ‘information sys-
domain analysis. Thus, the present chapter uses tem success’ literature from 2001 to 2010; (2) to
bibliometrics to analyze literature with the purpose determine the core subjects of academic activity
of having a better understanding of the research and their linkages studied during this ten years in
in terms of trends, core themes and productivity the ‘Information Systems Success’ literature; (3)
patterns in the IS success domain. Analyzing this to identify major contributing countries, authors
patterns give us linkages among the most influ- and journals that publish ‘Information Systems
ential topics, to explore subfields, and to derive Success’ articles; and (4) to forecast potential
their potential relationships. Thus, we provide an trends for future studies in the ‘Information Sys-
assessment of the field of IS success in its evo- tems Success’ literature.
lution during the last decade, an overview of its
key topics and some insights about future trends.
Bibliometric techniques are methods based BACKGROUND
on mathematical analysis, especially statistics,
to asses and quantify attributes from a research New ways of using resources are always at the
subject such as the articles embedded in interna- forefront of managers concerns because of the
tional journals literature (Abramo, D’Angelo, & continuous changing and evolving of technology
Caprasecca, 2009) in order to evaluate quality and (Kodama, 2005). Companies apply information
efficiency. These methods are non invasive, simple systems to strengthen their abilities to enhance
to implement, allowing rapid inter-temporal organizational change and growth on their op-
63
erations (Lin, 1991) and to solve operational external and internal properties of a scientific field
weaknesses and crises (Machperson, 2005). Thus, (Estabrooks, Winther, & Derksen, 2004) and are
the efficiency and effectiveness of information helpful to overcome some obstacles of distortion
systems in different contexts is a matter of fact related to subjectivity in assessments (Aksnes &
in order to overcome difficulties and obstacles in Taxt, 2004).
fast changing environments. This could explain The area of Bibliometrics is nowadays a well
the ‘IS success studies’ growth in the later decade studied field. It emerged time ago with some im-
as an important concern for researchers and prac- portant researchers (Garfield, 1972; de Solla Price,
titioners (Urbach et al., 2009). 1963) and has been recognized on the IS domain
The Information Systems discipline has been (Straub, 2006) as a central research topic. Normally
analyzed from the normative and descriptive per- the analysis are done using information included
spective in the research literature (Neufeld, Fang, on the fields of main articles from important
& Huffs, 2007). When talking about IS success, the journals (Palvia et al., 2004; Palvia, Mao, Salam,
normative has been the most studied perspective & Soliman, 2003). This fields include mainly but
focusing on the rules, directions, and boundaries are not limited to authors name, country, affilia-
of the matter in terms of theoretical foundation, tion, and other information close to the research
object and unit of analysis, multidimensionality itself such as keywords, words in title and abstract
and measurement. On the other hand, the de- which are used and analyzed to obtain patterns
scriptive perspective is scarce, though it can give and trends (Li, Ding, Feng, Wang, & Ho, 2008)
valuable information about the state and dynamic even at the industry level (e.g. biotechnology and
of the research stream. In this study we follow nanotechnology (Murray, 2002)). Bibliometrics
the descriptive perspective using bibliometrics methods uses data to assess research performance
methods and tools. and to forecast future interest areas assuming that
Bibliometrics has been used in the IS research researchers build upon other work of scientist (Van
in variety of topics such as: text mining analysis Raan, 1996) and those papers are reflections of
of published literature databases (Kostoff, Tooth- knowledge produced by scientific research.
man, Eberhart, & Humenik, 2001); information In order to accomplish the objectives planned,
construction and information gain from the infor- the bibliometrics techniques used are differenti-
mation theory perspective (Lafouge & Michel, ated in two groups. The first one which is based
2001); and information visualization and analysis basically on frequency counting and is related
of scientific literature and patents databases for to the descriptive statistics found in the fields of
technology forecasting (Morris, DeYong, Wu, bibliographic data of articles. This descriptive part
Salman, & Yemenu, 2002). Bibliometrics is de- give us a general dimension on some indicators
fined principally as the application of mathematics that have been used in other studies at different
and statistical methods to books and other media levels of analysis such as authors (Chi & Ho,
of communication (Pritchard, 1969) or as the 2005), countries (Rahman, Haque, & Fukui, 2005),
quantitative analysis of the bibliographic features journals (Dannenberg, 1985), institutions (Ra-
of a body of literature (Hawkins, 1977). Thus, its jendram, Lewison, & Preedy, 2006), and subject
purpose is to describe literary production, pat- areas (Davis & Gonzalez, 2003) among others.
terns of influence and the cognitive structure in a The second group is related to co-occurrence
given field – e.g. authors, journals, countries, core analysis involving keywords and core themes.
themes and collaboration between authors. The These analyses make the counting and study
bibliometric methods employ reliable data and of occurrence of keywords on papers giving a
quantitative analysis to described and map some picture of topics included in the literature (Ding,
64
Chowdhury, & Foo, 2001). This kind of analysis the context to initiate research on new topics or
are based on the nature of words as key elements to known the most recognized studies, authors,
of knowledge, ideas and concepts (Callon, Cour- countries and institutions on the field. So in this
tial, & Laville, 1991) and involve relations and chapter we make a descriptive perspective of the
interactions as an indication of the core themes ‘IS success’ field in order to fill this gap.
in a specific domain (Bhattacharya, Kretschmer, On the other hand, in the IS success literature
& Meyer, 2003; Cambrosio, Limoges, Courtial, there is not a representative tool that visualize
& Laville, 1993; Ding et al., 2001). This analyses the embedded knowledge in order to highlight
have been applied on the representation of differ- core themes and features of the research stream
ent disciplines and fields in the literature (Cahlik, (Novak, 1998). The co-occurrence analysis and
2000; Ding et al., 2001; Lee & Jeong, 2008; van the mapping resulting from it give a “big picture”
Meter, Cibois, & de Saint Leger, 2004; Neff & about the knowledge structure of the IS success
Corley, 2009; Viedma-Del-Jesus, Perakakis, Mu- research stream based on scientific papers (Price,
ñoz, López-Herrera, & Vila, 2011). 1965). This kind of analysis shows influential
The purpose of the present chapter is then to terms and theories associated with those terms
analyze the SSCI and SCI literature on ‘informa- (Caleromedina & Noyons, 2008) offering a start
tion systems success’ from 2001 to 2010 using the point to develop taxonomies on specific domains
general descriptive and the co-occurrence biblio- of knowledge (Borner, Hardy, Herr, Hooloway, &
metric techniques in order to provide insights to Paley, 2007). This information was used to search
the ‘IS success’ research. for sub-domains of knowledge in ‘IS success’
research stream besides the description of cur-
rent status, and to forecast future research topics.
“INFORMATION SYSTEMS Therefore, in this chapter we make a co-ocurrence
SUCCESS” BIBLIOMETRIC analysis based on the keywords of articles pub-
ANALYSIS lished in the WoS related to ‘IS success’ field in
order to fill this gap.
Knowing the Gap
Overall Research Productivity
The IS success research stream has been studied
and used on a variety of theories and experiments During the last ten years, the IS success research
and has appeared on different journals, books and has presented a general increase on the published
conferences resulting in a growth of its body of articles as shown in the Figure 1. There is a sharp
knowledge during the last decades (Urbach et al., grow in 2001 and then there is a grow tendency
2009). However there is not a descriptive per- until 2010. It seems like there is an increase on
spective about the growth, main journals, subject growth at odd years (2001, 2003, 2005, 2007 and
categories, countries and institutions working on 2009) and then at the next even year the annual
the topic and the more productive and more cited output is maintained or even decrease (2002, 2004,
authors. This research productivity description 2006, 2008, and 2010). However, in general an
helps to identify the most important contributors existing upward trend is expected to continue in
and contributions as has been studied in some the near future. Furthermore, it is expected that
works of IS (Chua, Cao, Cousins, & Straub, 2002; the annual output in 2011 overpasses the annual
Huang & Hsu, 2005). This information can be output in 2009-2010.
used as the set up to establish cooperative work Ninety seven countries have been working on
between institutions and researchers as well as the ‘IS success’ research stream during 2001-2010.
65
Figure 1. Distribution of published articles retrieved from ISI Web of Science
The contribution of countries was estimated by tries. There is not presence of African or Latin
the location of the affiliation of each one of the American countries in this list. Only USA gives
authors of the articles. The top 15 most productive the 41% of the whole research productivity. Five
countries and institutes are ranked in Table 1 by of the seven major industrial countries (G7:
number of records (number of articles where the Canada, France, Germany, Italy, Japan, UK, and
specific country or institute appears). USA) are ranked in the top ten of most productive
Two North American countries, seven Euro- countries. Moreover, the G7 have around the 70%
pean countries, four Asian countries, and Austra- of all the production in the research stream. This
lia were ranked in top 15 most productive coun- short description is important because the number
Table 1. Top 15 most productive countries and institutes of articles published during 2001 -2010
Country # Records Institution # Records

USA 1344 University of Central Florida (USA) 11
England 311 City University of Hong Kong (PR China) 9
Germany 212 University of Colorado (USA) 9
Canada 205 Harvard University (USA) 8
Taiwan 201 Gordon Life Science Institute (USA) 6
Australia 181 National Chung Cheng University (Taiwan) 6
Peoples R China 172 Auburn University (USA) 5
Spain 115 Kookmin University (South Korea) 5
France 109 National Sun Yat Sen Univ. (Taiwan) 5
Netherlands 104 National Taiwan University of Sci & Tech 5
South Korea 96 Baylor University (USA) 4
Italy 65 Creighton University (USA) 4
Japan 57 Erasmus University (Netherlands) 4
Scotland 48 Hong Kong Polytechnic Univ. (PR China) 4
Finland 44 National Chen Kung University (Taiwan) 4
66
of research papers could reflect the activity and These subject categories are complemented by
academic level of these countries in the field other important matters such as environmental
(Arunachalam & Jinandra, 2000) and can help sciences, ecology, zoology; operations research
researchers to choose location for mobility, intern- and management science, business; industrial
ships and exchange programs in order to absorb engineering, software engineering, manufacturing
knowledge that later on can be transferred and engineering, electrical and electronic engineering
applied in their origin countries, reducing the and telecommunications; health care sciences and
knowledge gap in the IS success theory and prac- services, public health, occupational health and
tice. medical informatics; artificial intelligence, theory
On the other hand, seven of the Top 15 most and methods of computer science; and others such
productive institutes are from USA which is in as education and educational research.
line to the aforementioned productivity of this Table 2 lists the top 15 most productive journals
country, four institutes are from Taiwan, two from and the most cited journals in the different articles
People R of China and finally South Korea and analyzed between 2001-2010. Approximately
the Netherlands with one record each. The Uni- 16% of the analyzed articles reside in the top 15
versity of Central Florida is the largest contributor most productive journals.
with 11 records on the research stream. The City As can be seen from Table 2, the two most
University of Hong Kong and the University of cited journals are related to the management sci-
Colorado are second with nine records each. Har- ences but they are immediately followed by IS
vard University is third with eight publications. journals. Something to note in the Table 2 are the
The only European university appearing in the journals that appear in both sides of the table
top 15 most productive institutes is the Erasmus which are highlighted in grey color. This can be
University from Netherlands. The five countries interpreted as the most productive journals that
associated to the top 15 most productive institu- at the same time are most cited, keeping a balance
tions are included in the top 15 most productive between quantity and quality of scientific produc-
countries which is an indication of a centralized tivity. Thus, is important to have these journals
leadership of these institutions in their respec- in mind when talking and studying about IS suc-
tive countries on the IS success research stream. cess. The journals are the Information & Manage-
However some countries like England, Germany ment, MIS Quarterly, and the Journal of Manage-
and Canada have not representation in the top 15 ment of Information Systems.
most productive institutes giving the idea of a Table 3 lists the most productive and most
more distributed workload of productivity at the cited authors in the articles analyzed. The authors
institutional level. Something we can bring from Jiang, J.J and Klein, G are the most productive
the Table 1, is that although there are differences authors in the literature with almost 2% of the
in the productivity among institutes, these differ- published articles. The most cited author is Delone,
ences are not substantial and there is not a large W.H., especially for his work on definitions and
gap such as the represented at the country level. measures of IS success. However, there are other
Most of the papers belong to the following sub- cited important authors such as Davis, F.D., and
ject categories: Computer Science and Informa- Davenport, T.H. In the authors case, there are no
tion Systems (588 records), Information Science author found on both the right and left side on
and Library Science (428 records), Management Table 3.
(422 records), Computer Science- Interdisciplin- What supports the authors most cited list is
ary applications (215 records), and operations the most cited references in the literature. The
research and Management Science (213 records). work most cited was done by DeLone and McLean
67
Table 2. Top 15 list of most productive and most cited journals
Most Productive Journal # Records Most Cited Journal # Records

INFORMATION & MANAGEMENT 80 MIS QUARTERLY 754
INDUSTRIAL MANAGEMENT &
DATA SYSTEMS 56 MANAGEMENT SCIENCE 639
JOURNAL OF COMPUTER INFORMA-
TION SYSTEMS 52 COMMUNICATIONS OF THE ACM 629
EUROPEAN JOURNAL OF INFORMA-
TION SYSTEMS 47 INFORMATION SYSTEMS RESEARCH 526
EXPERT SYSTEMS WITH APPLICA-
TIONS 43 INFORMATION & MANAGEMENT 492
JOURNAL OF MANAGEMENT INFORMATION
DECISION SUPPORT SYSTEMS 36 SYSTEMS 480
INTERNATIONAL JOURNAL OF IN-
FORMATION MANAGEMENT 29 INFORMATION SYSTEMS 460
INTERNATIONAL JOURNAL OF
MEDICAL INFORMATICS 27 HARVARD BUSINESS REVIEW 424
JOURNAL OF MANAGEMENT IN-
FORMATION SYSTEMS 24 DECISION SCIENCE JOURNAL 391
INFORMATION SYSTEMS JOURNAL 22 J MANAGEMENT INFORMA 368
JOURNAL OF INFORMATION TECH-
NOLOGY 22 SCIENCE 355
JOURNAL OF THE AMERICAN MEDI-
CAL INFORMATICS ASSOCIATION 22 INFORM MANAGE-AMSTER 349
MIS QUARTERLY 22 ACADEMY OF MANAGEMENT REVIEW 348
IEEE TRANSACTIONS ON ENGI-
NEERING MANAGEMENT 21 NATURE 327
COMPUTERS IN HUMAN BEHAVIOR 20 ORGANIZATION SCIENCE 318
(1992) with 210 papers citing it; it is followed by ing author keywords has been used recently (Chiu
the same authors (DeLone & McLean, 2003) with & Ho, 2007) and their analysis offers information
176 times cited; in third place (Fornell & Larck- about trends as viewed by researchers (Garfield,
er, 1981) 136 times cited; fourth place (Davis, 1990) and intellectual structures of knowledge
1989) 113 times cited; and fifth place (Seddon, domains (Culnan, 1987).
1997). However it follows some other works We obtained 8372 author keywords from the
written by authors like Nunally, Bailey, Anderson, 3250 articles selected. 7.7% of the keywords
and Doll, to say some of them. are stated 3 or more times in the total of articles
analyzed; 4.6% are stated 4 or more times; 3%
Research Key Topics: are stated 5 or more times; and only 2% are stated
Keywords Analysis 6 or more times in the total of articles. Thus, the
topics represented by the author keywords have
Although the statistical analysis of keywords could a broad distribution of the effort dedicated to the
have been done in the last section, we included it IS success research stream although there are
in this section in order to keep coherence to the co- some topics that lead the study as is said below.
ocurrence analysis. Bibliometric analysis concern-
68
Table 3. Top 15 list of most productive and more cited authors
Most Productive Authors # Records Most Cited Authors # Records

Jiang, James J. 23 DeLone W.H. 330
Klein, Gary. 23 Davis, F. D. 208
Lederer, Albert L. 11 Davenport, T. H 204
Lin, Hsiu-Fen. 10 Markus, M.L. 171
Keil, Mark. 9 Fornell, C.R. 157
Salmeron, Jose L. 9 Ives, B. 143
Chou, Kuo-Chen. 8 Doll, W.J. 140
Hung, Shin- Yuan. 7 Nunnally, J.C. 138
Kim, Sangsoo. 7 Hair, J.F. 137
Lee, Jooryang. 7 Venkatesh, V. 128
Lee, Jae- Nam. 7 Seddon, P.B. 127
Lee, Saro. 7 Chin, W.W. 125
Love, Peter E. D. 7 Grover, V. 121
Pare, Guy. 7 Rogers, E.M. 119
Rai, Ashutosh. 7 Anderson, J.C. 118
Table 4. The most used topics by country
Knowledge Project Critical Success Information

Country Information System Management Manager Factor Technology
USA 23 19 33 10 18
England 19 5 5 9 1
Germany 2 3 1 2 3
Canada 9 1 5 6
Taiwan 8 11 10 2 1
Australia 4 3 2 4 2
Peoples R China 2 5 2 6 1
Spain 5 1 4 3
France 5 1 2 2
Netherlands 1 1 1
South Korea 2 3 3 2
Italy 1
Japan
Scotland 1 1 1
Finland 2 1 2 3 1
The most used keywords are information critical success factors (47 records) and informa-
systems (71 records), knowledge management tion technology (41 records). These most used
(53 records), project management (49 records), keywords are followed by others such as user
69
Figure 2. Knowledge structure of the IS success stream research
satisfaction, internet, evaluation, implementation prisingly, the pair USA-Taiwan has the highest
and electronic commerce to say some of them. value, followed by USA-England, South Korea-
Table 4 shows the relations between the most USA, USA-China, and USA-Canada. Some other
productive countries and the most used topics important pairs founded are Australia-England,
based on keywords. Taiwan-England, Spain-England, South Korea-
For England, Canada, Spain, and France the Taiwan, and China-Taiwan.
key research theme during this ten years is infor- We used the top 100 author keywords based on
mation systems; however for Germany, Taiwan, their frequency of appearance in articles to create
and PR China it is knowledge management. For a map that represents the knowledge structure of
USA the key theme is related to project manage- the IS success research stream. The analysis was
ment. Something interesting is that Japan, though based on the co-ocurrence of pairs of keywords
is one of the most productive countries on the by article. The fundamental assumption is that by
matter, and does not dedicate efforts to the most their nature words are the fundamental element
used themes. Japan dedicates efforts to themes of knowledge, ideas and key concepts (Callon
related to GIS (Geographical Information Sys- et al., 1991) developed in the research stream
tems), culture, adoption, and simulation among (Bhattacharya et al., 2003). Figure 2 shows the
others. main concepts or themes in the IS success on
We made the cosine cross-correlation between author keywords.
countries and keywords to obtain which countries The map is realized via the VOS -visualization
have worked on the same research topics. Sur- of similarities technique (Eck & Waltman, 2007).
70
Figure 3. Relations among the main connected keywords
This technique allows drawing a bi-dimensional gether two or more times in the different articles
graph of the proximity between words. In the map, analyzed. This brings the opportunity to establish
the larger the number of items in the neighborhood new links between different topics or to further
of a point the higher the size of the keywords, and test and support the ones which have been studied
the closer the color of the point is to black. The already. So, it seems that the scientific studies of
localization of keywords in the knowledge space the research stream are too broad and there is yet
are a representation of similarity or the number too much to investigate or that probably there is
of times the words have been used together in the a lack of continuity in the research and a wide
articles and the size of the keyword is an indica- disparity in the research focus (Chuang, Huang,
tion of the frequency of use in the research lit- & Ho, 2007). Figure 3 shows the 39 connected
erature. Thus, in this map space the words near keywords and their relations. From the figure can
to each other are similar in the research context be seen the central position of core themes such
used by the articles than words far away from as knowledge management, information quality,
each other. Then the map gives information about information system among others, that we already
the central topic (information system) of the IS mentioned.
success stream and the research around or similar The width of the lines shows the strength of
to it (critical success factor, project management, the relationship between keywords based on the
outsourcing), as well as other topics which are number of times they appeared together in the
representative on the matter (system quality, different articles. The relationship between key-
geographic information system, user satisfaction) words is an indication about the effort dedicated
From the top 100 keywords showed in the to study it and can gives an outlook to the creation
knowledge map, only 39 keywords appeared to- of new relationships and knowledge discovery.
71
Figure 4. Tendency of the five most studied themes
The stronger link obtained was among the ‘system learning, power, business models, and collabora-
quality’ and ‘information quality’ keywords (ap- tive learning. This themes can open new perspec-
peared together 15 times); these are followed by tives of study for researchers already involved in
‘service quality’, ‘user satisfaction’, and ‘informa- the IS success area or for new researchers looking
tion system success’ (appeared together 8 times); for new opportunities in the domain.
then the ‘information system’ and ‘information Finally, Figure 4 shows the tendency of the five
technology’ appeared together 7 times, and so on. most important topics in the research field from
At the end, the Figure 3 shows the formation of 2001 to 2010. The general tendency is to growth
4 connected clusters represented by different node as can be seen. However, project management has
shapes. These clusters are representations of sub- a higher tendency of growth besides knowledge
domains studied in the research stream: e-com- management and information systems which have
merce and e-business (square), information sys- decreased in the last year (2010). Critical success
tems and knowledge management (triangle), factor and information technology although have
information systems and quality (diamond), and a positive growth tendency, it is not as high as the
information systems theory and assessment three topics mentioned before.
(circle). These sub-domains have been the central
points of research during the ten years analyzed.
On the other hand, the keywords were classi- FUTURE RESEARCH DIRECTIONS
fied to obtain information about the year they have
appeared in the research steam. These give us an Future research on the descriptive perspective
understanding about relatively new topics and old of ‘IS success’ should include conference papers
ones that are studied in the IS success research. as unit of analysis and its comparison to journal
Among the relatively new topics were found: articles to check similarities and differences in this
e-business, culture, manufacturing resource plan- two contexts and see if the bibliometric tendencies
ning, information sharing, technology acceptance, and distributions fit for both of them. This can
virtual communities, GPS, social networks, online give information about the relevant conferences
72
at local or international context, and to have a The bibliometric analysis indicate that ‘IS
better understanding of emerging fields and its success’ has not developed a mature body of
acceptation by the academic community both on knowledge reflected in the great number of themes
local and internationally contexts. Future studies (keywords) isolated from each other. So it is pos-
can compare domestic tendencies by country or sible that researchers do not share yet a base of
institution and the international tendencies shown common language that allows them to understand
here. This analysis can give a “local” point of similitude and differences in the diversity of
view referred to the global research in order to research disciplines and fields studied. In other
make strategic research planning in the IS success words, the IS success research stream overlap,
research stream. Thus, something interesting to combine, interact and is used in different disci-
have into account in future studies is the cultural plines making it difficult to set down on common
factor e.g. how collective or individualistic is the terms because of the various theoretical models
behavior, and multiples levels of analyses such and tools and deviating it from its main purpose.
as authors, institutions or countries. So there is a need to coupling different views of
Future research should have into account other IS success and set up a framework that served as
variables and combinations from the analyzed a foundation for future research.
here such as the journal growth rate per year, the The main subject categories been researched
articles growth rate by country per year, and the are computer science and information systems,
institutes productivity per year in order to see the information science and library science, and
tendencies and productivity from different points management; however there are other important
of view, this information can be valuable besides subjects in other context that have been impacted
the described here to establish strategic research by the research stream such as environmental sci-
alliances in the IS success research stream. Fur- ences, engineering, health and educations subjects.
thermore, it can be quite interesting to explain The most productive and more cited journals are
why the general tendency shown in this study the Mis Quarterly, Information and Management
has an increasing “step” shape growing each two Research, and the journal of Management Infor-
years. So further quantitative studies are needed mation Systems. The most productive authors
to obtain more distinctive characteristics of IS are Jiang, Klein, and Lederer, and the most cited
success research stream. are Delone, Davis, and Davenport, the most
cited references are DeLone and McLean (1992);
(DeLone & McLean, 2003); (Fornell & Larcker,
CONCLUSION 1981); and (Davis, 1989).
The main topics used in the IS success research
There are important implications resulting from stream are information systems, knowledge man-
the present analysis. The IS success is a research agement, project management, critical success fac-
stream with a positive growth rate at a global tors, and information technology. However there
level. The number of articles has growth during are other important topics been researched such as
2001 to 2010 and it is likely that the tendency will user satisfaction, internet, evaluation, e-commerce
be maintained in the near future. Second, the IS among others. From 646 keywords used by the
success research is centralized in a few developed authors of articles, only 45 were used together in
countries e.g. the productivity of USA and Canada the literature analyzed. Some are central in the re-
results in the 50% of all the production at global search stream such as information system, system
level. Latin American and African countries are ab- quality, and knowledge management. Relative new
sent from the top research countries in IS success. topics found are culture, e-business, information
73
sharing, virtual communities, GPS, social net- Cahlik, T. (2000). Search for fundamental articles
works, online learning, power and collaborative in economics. Scientometrics, 49(3), 389–402.
learning. The result analysis by the bibliometric doi:10.1023/A:1010533506061
methods can support relevant researchers realize
Caleromedina, C., & Noyons, E. (2008). Combin-
the global landscape in IS success and establish
ing mapping and citation network analysis for a
the further research direction
better understanding of the scientific development:
The case of the absorptive capacity field. Journal
of Informatrics, 2(4), 272–279. doi:10.1016/j.
REFERENCES
joi.2008.09.005
Abramo, G., D’Angelo, C. A., & Caparasecca, A. Callon, M., Courtial, J. P., & Laville, F. (1991).
(2009). Allocative efficiency in public research Co-word analysis as a tool for describing the
funding: Can bibliometrics help? Research Policy, network of interactions between basic and techno-
38(1), 206–215. doi:10.1016/j.respol.2008.11.001 logical research: The case of polymer chemistry.
Aksnes, D. W., & Taxt, R. E. (2004). Peers reviews Scientometrics, 22(1), 155–205. doi:10.1007/
and bibliometric indicators: A comparative study BF02019280
at Norwegian University. Research Evaluation, Cambrosio, A., Limoges, C., Courtial, J. P., &
13(1), 33–41. doi:10.3152/147154404781776563 Laville, F. (1993). Historical scientometrics? Map-
Arunachalam, S., & Jinandra, D. M. (2000). Map- ping over 70 years of biological safety research
ping international collaboration in science in Asia with co-word analysis. Scientometrics, 27(2),
through coauthorship analysis. Current Science, 119–143. doi:10.1007/BF02016546
79(5), 621–628. Chiu, W. T., & Ho, Y. S. (2005). Bibliometric
Bayer, A. E., & Folger, J. (1966). Some cor- analysis of homeopathy research during the period
relates of a citation measure of productivity in of 1991 to 2003. Scientometrics, 63(1), 3–23.
science. Sociology of Education, 39(4), 381–390. doi:10.1007/s11192-005-0201-7
doi:10.2307/2111920 Chiu, W. T., & Ho, Y. S. (2007). Bibliometric
Bhattacharya, S., Kretschmer, H., & Meyer, M. analysis of tsunami research. Scientometrics, 73,
(2003). Characterizing intellectual spaces between 3–17. doi:10.1007/s11192-005-1523-1
science and technology. Scientometrics, 58(2), Chua, C., Cao, L., Cousins, K., & Straub, D.
369–390. doi:10.1023/A:1026244828759 (2002). Measuring researcher-production in In-
Borner, K., Hardy, E. F., Herr, B. W., Hooloway, formation Systems. Journal of the Association
T., & Paley, W. B. (2007). Taxonomy visualization for Information Systems, 3, 145–215.
in support of the semi-automatic validation and Chuang, K. Y., Huang, Y. L., & Ho, Y. S. (2007).
optimization of organizational schemas. Journal A bibliometric and citation analysis of stroke-
of Informatrics, 1(3), 214–225. doi:10.1016/j. related research in Taiwan. Scientometrics, 72(2),
joi.2007.03.002 201–212. doi:10.1007/s11192-007-1721-0
Braun, T., Schubert, A. P., & Kostoff, R. N. Culnan, M. J. (1987). Mapping the intellectual
(2000). Growth and trends of fullerene research structure of MIS, 1980-1985: A co-citation analy-
as reflected in its journal literature. Chemical sis. Management Information Systems Quarterly,
Reviews, 100(1), 23–37. doi:10.1021/cr990096j 11(3), 341–353. doi:10.2307/248680
74
Dannenberg, A. L. (1985). Use of epidemiology Estabrooks, C. A., Winther, C., & Derksen,
in medical specialties: An examination by cita- L. (2004). Mapping the field: A bibliometric
tion analysis. American Journal of Epidemiology, analysis of the research utilization literature in
121(1), 140–151. nursing. Nursing Research, 53(5), 293–303.
doi:10.1097/00006199-200409000-00003
perceived ease of use, and user acceptance of Floropoulos, J., Spathis, C., Halvatzis, D., &
Information Technology. Management Infor- Tsipouridou, M. (2010). Measuring the success of
mation Systems Quarterly, 13(3), 319–340. the Greek taxation Information System. Interna-
doi:10.2307/249008 tional Journal of Information Management, 30(1),
47–56. doi:10.1016/j.ijinfomgt.2009.03.013
Davis, J. C., & Gonzalez, J. G. (2003). Scholarly
journal articles about the Asian tiger economies: Fornell, C., & Larcker, D. F. (1981). Evaluat-
Authors, journals and research fields, 1986–2001. ing structural equation models with unobserv-
Asian-Pacific Economic Literature, 17(2), 51–66. able variables and measurement error. JMR,
doi:10.1046/j.1467-8411.2003.00131.x Journal of Marketing Research, 18(1), 39–50.
doi:10.2307/3151312
tion Systems success: The quest for the dependent Garfield, E. (1972). Citation analysis as a tool in
variable. Information Systems Research, 3(1), journal evaluation. Science, 178(60), 471–479.
60–95. doi:10.1287/isre.3.1.60 doi:10.1126/science.178.4060.471
DeLone, W. H., & McLean, E. R. (2003). The Garfield, E. (1990). Keywords plus-ISIS break-
DeLone and McLean model of Information through retrieval method. 1. Expanding your
Systems success: A ten-year update. Journal of searching power on current-contents on diskette.
Management Information Systems, 19(4), 9–30. Current Contents, 32, 5–9.
DeLone, W. H., & McLean, E. R. (2004). Measur- Garrity, E. J., Glassberg, B., & Kim, Y, J., Sand-
ing e-commerce success: Applying the DeLone ers, G. L., & Shin, S, K. (2005). An experimental
& McLean Information Systems success model. investigation of Web-based Information Systems
International Journal of Electronic Commerce, success in the context of electronic commerce.
9(1), 31–47. Decision Support Systems, 39(3), 485–503.
doi:10.1016/j.dss.2004.06.015
Ding, Y., Chowdhury, G. G., & Foo, S. (2001).
Bibliometric cartography of information retrieval Hawkins, D. T. (1977). Unconventional uses of
research by using co-word analysis. Information on-line information retrieval systems: On-line
Processing & Management, 37(6), 817–842. bibliometric studies. Journal of the American
doi:10.1016/S0306-4573(00)00051-0 Society for Information Science American Society
for Information Science, 28, 13–18. doi:10.1002/
Eck, N. J. P., & Waltman, L. R. (2007). VOS: A
asi.4630280103
new method for visualizing similarities between
objects. In Decker, R., & Lenz, H.-J. (Eds.), Ad- Heeks, R. (2002). Information systems and de-
vances in data analysis (pp. 299–306). Berlin, veloping countries: Failure, success, and local
Germany: Springer. doi:10.1007/978-3-540- improvisations. The Information Society, 18(2),
70981-7_34 101–112. doi:10.1080/01972240290075039
75
Heeks, R. (2006). Health Information Systems: Lu, X. H., Huang, L. H., & Heng, S. H. (2006).
Failure, success and improvisation. International Critical success factors of inter-organizational
Journal of Medical Informatics, 75(2), 125–137. information systems- A case study of Cisco and
doi:10.1016/j.ijmedinf.2005.07.024 Xiao Tong in China. Information & Management,
43(3), 395–408. doi:10.1016/j.im.2005.06.007
Holsapple, C. W. (2008). The pulse of multipartici-
pant systems. Journal of Organizational Comput- Machperson, A. (2005). Learning how to grow:
ing and Electronic Commerce, 18(4), 333–343. Resolving the crisis of knowing. Technova-
doi:10.1080/10919390802453617 tion, 10, 1129–1140. doi:10.1016/j.technova-
tion.2004.04.002
Huang, H. H., & Hsu, J. S. C. (2005). An evalu-
ation of publication productivity in Information Meijden, van der, M. J., Tange, H, J., Troost, J.,
Systems: 1999 to 2003. Communications of the & Hasman, A. (2003). Determinants of success
Association for Information Systems, 15(31), of inpatient clinical Information Systems: A lit-
555–564. erature review. Journal of the American Medical
Informatics Association, 10(3), 235-243.
Jen, Y. W., & Chao, C. C. (2008). Measuring mo-
bile patient safety Information System success: An Meter, K. v., Cibois, P., & Saint Leger (2004).
empirical study. International Journal of Medi- Correspondence & co-word analysis of ten years
cal Informatics, 77(10), 689–697. doi:10.1016/j. of BMS articles (1993-2003). Bulletin de Mé-
ijmedinf.2008.03.003 thodologie Sociologique.
Keiser, J., & Utzinger, J. (2005). Trends in the Moed, H. F., De Bruin, R. E., Nederhof, A. J., &
core literature on tropical medicine: A bibliometric Tijssen, R. J. W. (1991). International scientific
analysis from 1952-2002. Scientometrics, 62(3), cooperation and awareness within the Euro-
351–365. doi:10.1007/s11192-005-0027-3 pean community: Problems and perspectives.
Scientometrics, 21(3), 291–311. doi:10.1007/
Kodama, M. (2005). New knowledge creation
BF02093972
through leadership-based strategic community—A
case of new product development in IT and multi- Murray, F. (2002). Innovation as co-evolution of
media business fields. Technovation, 25(8), 895– scientific and technological networks: Exploring
908. doi:10.1016/j.technovation.2004.02.016 tissue engineering. Research Policy, 31, 1389–
1403. doi:10.1016/S0048-7333(02)00070-7
Lee, B., & Jeong, Y. (2008). Mapping Korea’s
national R&D domain of robot technology by Neff, M. W., & Corleyb, E. A. (2009). 35 years
using the co-word analysis. Scientometrics, 77(1), and 160,000 articles: A bibliometric exploration
3–19. doi:10.1007/s11192-007-1819-4 of the evolution of ecology. Scientometrics, 80(3),
657–682. doi:10.1007/s11192-008-2099-3
Li, L. L., Ding, G., Feng, N., Wang, M. H., & Ho,
Y. S. (2008). Global stem cell research trend: Bib- Neufeld, D., Fang, Y., & Huff, S. (2007). The IS
liometric analysis as a tool for mapping of trends identity crisis. Communications of the Association
from 1991 to 2006. Scientometrics, 80(1), 41–60. for Information Systems, 19, 447–464.
Lin, B. (1991). Health care Information System: Novak, J. D. (1998). Learning, creating, and
A management perspective. Management Impacts using knowledge: Concept maps as facilitative
of Information Technology, 370-387. tools in schools and cooperations. Mahwah, NJ:
Lawrence Erlbaum Associates.
76
Page, C. K. (2000). Critical success factors for Solla Price, D. d. (1963). Little science, big sci-
implementing a clinical information system. ence. New York, NY: Columbia University Press.
Nursing Economics, 18(5). Jannetti Publications.
Straub, D. (2006). The value of scientometric
Palvia, P., Leary, D., Mao, E., Midha, V., Pinjani, studies: An introduction to a debate on IS as a
P., & Salam, A. F. (2004). Research methodolo- reference discipline. Journal of the Association
gies in MIS: An update. Communications of the for Information Systems, 7(5), 241–245.
Association for Information Systems, 14, 526–542.
Tang, R., & Thelwall, M. (2003). US academic
Palvia, P., Mao, E., Salam, A. F., & Soliman, departmental Web-site interlinking in the United
K. (2003). Management information systems States disciplinary differences. Library & In-
research: What’s there in a methodology? Com- formation Science Research, 25(4), 437–458.
munications of the Association for Information doi:10.1016/S0740-8188(03)00053-7
Systems, 11, 289–309.
Urbach, N., Smolnik, S., & Riempp, G. (2009). The
Petter, S., DeLone, W. H., & McLean, E. R. (2008). state of research on Information Systems success.
Measuring Information Systems success: Models, Business & Information Systems Engineering,
dimensions, measures, and interrelationships. 1(4), 315–325. doi:10.1007/s12599-009-0059-y
Van Raan, A. F. J. (1996). Advanced bibliometric
236–263. doi:10.1057/ejis.2008.15
methods as quantitative core of peer review based
Price, D. J. (1965). Networks of scientific pa- evaluation and foresight exercises. Scientometrics,
pers. Science, 149, 510–515. doi:10.1126/sci- 36(3), 397–420. doi:10.1007/BF02129602
ence.149.3683.510
Viedma-Del-Jesus, M. I., Perakakis, P., Muñoz,
Pritchard, A. (1969). Statistical bibliography or M. A., López-Herrera, A. J., & Vila, J. (2011).
bibliometrics. The Journal of Documentation, Sketching the first 45 years of the journal Psy-
25(4), 348–349. chophysiology (1964–2008): A co-word-based
analysis. Psychophysiology, 48(8), 1029–1036.
Rahman, M., Haque, T. L., & Fukui, T. (2005).
doi:10.1111/j.1469-8986.2011.01171.x
Research articles published in clinical radiology
journals: Trend of contribution from different Wang, Y. S., & Liao, Y. W. (2008). Assessing
countries. Academic Radiology, 12(7), 825–829. eGovernment systems success: A validation of the
doi:10.1016/j.acra.2005.03.061 DeLone and McLean model of information sys-
tems success. Government Information Quarterly,
Rajendram, R., Lewison, G., & Preedy, V. R.
25(4), 717–733. doi:10.1016/j.giq.2007.06.002
(2006). Worldwide alcohol-related research and
the disease burden. Alcohol and Alcoholism (Ox- Yen, H. R., Li, E. Y., & Niehoff, B. P. (2008).
ford, Oxfordshire), 41(1), 99–106. doi:10.1093/ Do organizational citizenship behaviors lead to
alcalc/agh238 Information System success? Testing the me-
diation effects of integration climate and project
Seddon, P. B. (1997). A respecification and exten-
management. Information & Management, 45(6),
sion of the DeLone and McLean model of IS Suc-
394–402. doi:10.1016/j.im.2008.04.004
cess. Information Systems Research, 8, 240–253.
doi:10.1287/isre.8.3.240
77
ADDITIONAL READING Ho, Y. S. (2007). Bibliometric analysis of absorp-

tion technology in environmental science. Journal
Almeida-Filho, N., Kawachi, I., Filho, A. P., of Environmental Protection Science, 1, 1–11.
& Dachs, J. N. (2003). Research on health in-
equalities in Latin America and the Caribbean: Kaminer, N., & Braunstein, Y. M. (1998). Biblio-
Bibliometric analysis (1971-2000) and descriptive metric analysis of the impact of Internet use on
content analysis (1971-1995). American Journal scholarly productivity. Journal of the American So-
of Public Health, 93(12), 2037. doi:10.2105/ ciety for Information Science American Society for
AJPH.93.12.2037 Information Science, 49(8), 720–730. doi:10.1002/
(SICI)1097-4571(199806)49:8<720::AID-
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, ASI6>3.0.CO;2-V
H. D. (2008). Citation counts for research evalu-
ation: Standards of good practice for analyzing Koehler, W. (2001). Information Science as “Little
bibliometric data and presenting and interpreting Science”: The implications of a bibliometric
results. Ethics in Science and Environmental analysis of the Journal of the American Society
Politics, 8(1), 93–102. doi:10.3354/esep00084 for Information Science. Scientometrics, 51(1),
117–132. doi:10.1023/A:1010516712215
Chao, C., Yang, J., & Jen, W. (2007). Determin-
ing technology trends and forecasts of RFID by a Kuperman, V. (2006). Productivity in the Internet
historical review and bibliometric analysis from mailing lists: A bibliometric analysis. Journal of
1991 to 2005. Technovation, 27(5), 268–279. the American Society for Information Science and
doi:10.1016/j.technovation.2006.09.003 Technology, 57(1), 51–59. doi:10.1002/asi.20252
Dalpé, R. (2002). Bibliometric analysis of bio- Pulgarín, A., & Gil-Leiva, I. (2004). Bibliomet-
technology. Scientometrics, 55(2), 189–213. ric analysis of the automatic indexing literature:
doi:10.1023/A:1019663607103 1956-2000. Information Processing & Man-
agement, 40(2), 365–377. doi:10.1016/S0306-
De Bakker, F. G. A., Groenewegen, P., & Den 4573(02)00101-2
Hond, F. (2005). A bibliometric analysis of
30 years of research and theory on corporate
social responsibility and corporate social per-
formance. Business & Society, 44(3), 283–317. KEY TERMS AND DEFINITIONS
doi:10.1177/0007650305278086
Bibliometrics: Quantitative analysis of the
Falagas, M. E., Michalopoulos, A. S., Bliziotis, bibliographic features of a body of literature
I. A., & Soteriades, E. S. (2006). A bibliometric (Hawkins, 1977)
analysis by geographic area of published re- ISI WoS: Institute for Scientific Information
search in several biomedical fields, 1995-2003. – Web of Science.
Canadian Medical Association Journal, 175(11), Keywords analysis: Bibliometric technique
1389–1390. doi:10.1503/cmaj.060361 that uses keywords as the unit of analysis and
count their ocurrence on papers literature of a
Gu, Y. (2004). Global knowledge manage-
determined domain of knowledge.
ment research: A bibliometric analysis. Sci-
Knowledge Mapping: Methods used to draw-
entometrics, 61(2), 171–190. doi:10.1023/
ing intellectaul structures of knowledge domains
B:SCIE.0000041647.01086.f4
in bi-dimensional graphs. The most used tech-
niques are Multi Dimensional Scaling-MDS and
Visualization Of Similarities – VOS.
78
Research productivity: Publication activity SCI: Science Citation Index database from In-
that can be analyzed at differents levels such as stitute for Scientific Information – Web of Science.
author, institution, country and domain specific, SSCI: Social Sciences Citation Index database
in a research stream of knowledge. from Institute for Scientific Information – Web
of Science.
79
80
Chapter 5
Theoretical Foundations
for Information Systems
Success in Small- and
Medium-Sized Enterprises
Jan Devos
Ghent University, Belgium
Hendrik Van Landeghem

Dirk Deschoolmeester
ABSTRACT
Small and medium-sized enterprises (SMEs) adopt Information Technology (IT) and Information Systems
(IS) in order to achieve business goals and obtain net benefits. However, adopting IT/IS into an existing
organizational structure is a complex and risky task. Many investments in IT/IS, outsourced as well as
in-sourced, never fully reach the intended objectives and are therefore considered as not being successful.
In this chapter, the authors focused on IS success in small and medium sized-enterprises (SMEs) in order
to find theoretical foundations. They explain four well-known theories, often used in IS research, which
constitute the basics of their thinking. These theories are the technology acceptance model (TAM), the
theory of planned behavior (TPB), the DeLone & McLean IS success model (D&M), and the transac-
tion cost economy (TCE) model. The authors weaved the constructs of these theories into a compound
framework that delivers explanatory and predicting power for the successful adoption of IT/IS in SMEs.
This framework is also consistent with the nomological IS network established by Benbasat and Zmud
(Benbasat & Zmud, 2003). In order to validate the framework, the authors examined the extent to which
the theoretical model could provide support for the Cobit framework, often used by practitioners as an
IT governance framework, and also suitable for SMEs. Findings show that the framework offers surpris-
ing coherence and proposes a strong theoretical foundation for the normative directions of the methods
used in Cobit by IT practitioners.
DOI: 10.4018/978-1-4666-0170-3.ch005
Theoretical Foundations for Information Systems Success in Small- and Medium-Sized Enterprises
INTRODUCTION consistent with the existing IS network established

by Benbasat and Zmud (Benbasat & Zmud, 2003).
Small- and medium-sized enterprises (SMEs) This framework was empirically validated and
adopt information technology (IT) and information tested by mapping the theoretical constructs with
systems (IS) in order to achieve business goals the relevant IT processes of the Cobit-model for
and obtain net benefits. However, adopting IT/ SMEs (“Cobit Quickstart 2th Edition,” 2007; Cobit
IS into an existing organizational structure with v.4.1, 2007). Cobit is a well-known IT governance
a lack of rigid business processes is a complex model and is used by IT practitioners, even in
and risky task. Many investments in IT/IS, such SMEs (“Cobit Quickstart 2th Edition,” 2007).
as enterprise resource planning (ERP) or cus- The next section is an overview of the literature
tomer relation management (CRM), outsourced on the relationship between SMEs and IT/IS. This
as well as in sourced, never fully reach the in- analysis reviews the conceptual foundations of
tended objectives and are therefore considered small organizations and their behavior and actions
as not being successful. Despite our knowledge towards information systems. Next, we give a short
of IT/IS implementation, a lot of IT projects still overview of the concept of IT governance and its
fail (Avison, Gregor, & Wilson, 2006; Bartis relevance for SMEs. We then present an outline
& Mitev, 2008; Bharadwaj, Keil, & Mahring, of our theoretical framework. Although the use
2009; Conboy, 2010; Devos, Van Landeghem, & of well-developed theories in IS research is now
Deschoolmeester, 2008; Group, 2004). Past and common practice, we conceptualize a compound
recent research has also revealed that SMEs tend theoretical model, woven out of existing IS and
to lean strongly on external IT/IS expertise for the non-IS theories. We present the empirical findings
adoption of IT (Dibbern & Heinzl, 2009; Thong, in Section 5. Section 5 also includes our conclu-
Yap, & Raman, 1996; Yap, Soh, & Raman, 1992). sions and a summary of the main points raised
However, outsourcing IT/IS greatly increases the by this research.
complexity of governing these endeavors and
brings in new risks and burdens for IS success
(Aubert, Patry, & Rivard, 2005; Aubert, Rivard, SMES AND IT/IS
& Patry, 2004; Natovich, 2003).
In this chapter, we have focused on IS success Although small- and medium-sized enterprises
in SMEs in order to find theoretical foundations. (SMEs) are a substantial part of the business
We have explained four well-known theories, often environment and a major component of most
used in IS research, which constitute the basics of economies throughout the world, IS research has
our thinking. These theories are the technology primarily focused on large corporations (Lee,
acceptance model (TAM), the theory of planned Kim, Choi, & Lee, 2009; Snider, da Silveira, &
behavior (TPB), the DeLone & McLean IS success Balakrishnan, 2009; Wang & Ahmed, 2009). Re-
model (D&M) and the transaction cost economy search on IS success in small businesses has been
(TCE) model. We have designed a nomological largely disregarded and has mainly been conducted
network for each theory and brought the constructs in manufacturing enterprises (Harland, Caldwell,
into concert, in a compound framework that deliv- Powell, & Zheng, 2007; Olsen & Saetre, 2007).
ers much more explanatory and predicting power Although research and literature have high-
for the successful adoption of IT/IS in SMEs than lighted the definitional problems of SMEs, it has
each theory could do separately. Our framework greatly ignored the idiographic nature of a small
takes advantage of the symbiotic effect of com- business. Companies differ in size, location, own-
bining the aforementioned theories, but is also ership structure, financial performance, maturity,
81
and management style. It would be ideal to clearly different from orientations of firm-value maxi-
define a SME before starting any research. How- mization possessed by professionally-managed
ever, this is not straight forward. There are many businesses (Wang & Ahmed, 2009). A lack of
characteristics to identify a SME besides size. The readiness for networking with other enterprises
European Commission took an initiative to define and a reluctance to use advanced IT was noticed in
a SME in terms of microeconomic characteris- SMEs (Commission, 2004). SMEs perceive little
tics, such as turnover (not exceeding 50 million incentive to change business models when the
euro), annual balance sheet total (not exceeding returns are unclear (OECD, 2005). Research has
43 million euro) and headcount (fewer than 250 also shown that SMEs do not excel in knowledge
persons) (Commission, 2003). This definition is retention and in obtaining a sustainable competi-
made from a legal and economic point of view tive advantage. SMEs are slower to adopt IT than
and is not always accurate when it comes to the large enterprises (Premkumar, 2003). Also, SMEs
study of the relationship between the company cannot learn and benefit from past experiences
and IT, but this definition holds for our research. because there are not enough IS projects conducted
We have focused on entrepreneurial enter- in their organizations.
prises with a relatively small market share, that A major generic finding is that the CEO of a
are managed by its owners or part owners in a SME often has a very strong personal influence
personalized way and not by an organized manage- on the organizational decision-making (Cragg,
rial structure, and that are independent, with the 2008; Lefebvre, Mason, & Lefebvre, 1997; Levy
owners/managers having control of the activities & Powell, 2004; Thong & Yap, 1995). The CEO
of the business (Bolton, 1971). According to the is the central focus point in the organization of a
European Commission definition of SMEs, this SME and determines the direction, the strategy
constitutes the group of small enterprises with and the appetite for innovativeness (Dibrell, Da-
a headcount between ten and 100. From an IT vis, & Craig, 2008; Marcati, Guido, & Peluso,
perspective, it is often more useful to look at the 2008). His/her capabilities and preferences will
number of so called ‘white collars’ or knowledge determine the rate at which a SME changes and
workers in an organization. This definition can grows. It is the CEO who makes the decision
sometimes incorporate companies with up to 500 to adopt IT/IS. According to Rogers’ model of
employees but with a majority of ‘blue collars’ adopting an innovation process, the decision to
in their staff. Blue collars are considered as not adopt is preceded by a positive or negative atti-
using any form of IT. These companies, mainly tude towards the innovation (Rogers, 2003). The
manufacturing enterprises, fall within the afore- perceived benefits of the proposed information
mentioned definition of a SME. system have to outweigh the associated risks and
This still leaves us with a very large and hetero- costs in order to adopt the system. However, the
geneous group of organizations. However, existing CEO and employees in SMEs are overwhelmed
research has revealed some generic characteristic with day-to-day business, leaving little space for
of SMEs when it comes to the adoption of IT/IS strategic issues or other important considerations,
(Devos, Van Landeghem, & Deschoolmeester, such as process management and adopting IT/
2011). IS, not to mention process implementation and
In the years of the dotcom hype, many believed process improvement.
that IT would enable SMEs to compete with large Another important generic characteristic of
companies (Audretsch, Prince, & Thurik, 1999). SMEs is that only a limited amount of IT resources
However, due to its idiosyncratic profile, a SME and capabilities are available in a SME (Thong,
demonstrates unique strategic orientations that are 2001; Thong, et al., 1996; Verhees & Meulenberg,
82
2004). This resource deficit is a result of their Kim, 2007; Malhotra & Temponi, 2010; Nasco,
highly competitive environment, financial con- Toledo, & Mykytyn, 2008; Wang & Lin, 2009).
straints, lack of professional expertise, and their Here we consider a selection of young entrepre-
sensitivity to external forces (Welsh & White, neurial organizations, often family run businesses,
1981). The smaller the business, the less able it mostly active in a service or trading market, with a
is to hire people with specialized skills, like, for headcount of ten to 50 ‘white collars’, and which
example, IT skills (Alpar & Reeves, 1990). The outsource their IS function almost completely.
shortage of internal IT capabilities and techniques They make this decision primarily of necessity,
is the major reason why SMEs tend to outsource due to resource poverty. The academic literature on
IT and consequently depend heavily on external this specific setting is scarce and is overwhelmed
IT expertise (Dibbern & Heinzl, 2009). by practitioners’ guidelines and consultant reports,
Finally, another issue, found by research, is which lack firm theoretical foundations.
the lack of process orientation in SMEs. SMEs
still tend to be too much task-oriented, leaving
almost no space for the implementation of cross- IT GOVERNANCE FOR SMEs
functional capabilities into their rigid business
processes (Antlova, 2009; Chang, Hung, Yen, & IT governance is a term that has been evolving
Lee, 2010; Snider et al., 2009). This characteristic rapidly over the last few years, especially in practi-
is often referred as a low-capability maturity level. tioners’ communities. The IT Governance Institute
SMEs often lack the necessary skills and compe- is taking a leading role in the debate (Institute,
tencies to build and implement rigid management 2003). Many aspects of IT governance have been
systems to control their business processes. described and detailed but little work has been
However, the importance of IT/IS for SMEs is done to pull it all together. From an academic
paramount. SMEs are unable to capitalize on the perspective, research on IT governance is emerg-
economics of scale but they still have to operate ing as an important area of enquiry (Bernroider,
in a highly competitive environment with a lot 2008; Huang, Zmud, & Price, 2010; Peterson,
of uncertainties and in an increasingly complex 2004; Schwarz & Hirschheim, 2003; Weill &
business environment. Therefore, some authors Ross, 2005).
contend that small businesses should make a better The vital role of IT in enterprises has led to the
effort to adopt IT/IS more intensely (Lee et al., view that IT governance must be implemented in
2009; Malhotra & Temponi, 2010; Tan, Chong, order to sustain and enable business objectives and
Lin, & Eze, 2009). to mitigate associated risks. IT governance directly
However, the adoption and use of IT/IS in influences the benefits generated by organizational
organizations is not straightforward. IT projects IT investments (Weill & Ross, 2004). According to
differ greatly from engineering projects, because Weill, well-designed IT governance mechanisms
they are characterized by high complexity and a are likely to produce IT-related decisions that are
high likelihood of project failure (Devos et al., more firmly associated with an organization’s
2008). IS Success is certainly not guaranteed, and business processes (Weill & Ross, 2005). This
it is even difficult to define precisely (DeLone & holds true for large as well as for small businesses.
McLean, 2003; Rai, Lang, & Welker, 2002; Sed- However, the mechanisms of IT governance are
don, 1997). Recent research has highlighted the applied much more extensively in large enter-
reasons underlying the successful adoption of IT/ prises than in SMEs (Huang et al., 2010). Existing
IS in SMEs, the impediments encountered, and the mechanisms of IT governance, such as the way
organizational impact (Chang et al., 2010; Lee & critical IT processes are conducted, the creation
83
of management guidelines to accompany these IT & McLean IS success model (D&M), technol-
processes, and the assignment of responsibilities ogy acceptance model (TAM), theory of planned
and accountabilities seem to fail in SMEs, where behavior (TPB) and transaction cost economy
decision making is mostly centered round one (TCE) are amongst the most invoked theories in
person (Levy & Powell, 2008; Southern & Tilley, IS research on SMEs. D&M and TAM are two
2000). The concept of IT governance originates generic IS theories, coming from the IS discipline.
from the discussion of strategic IT planning and Since its inception in 1970, this discipline has
IT management, but its link to an overall corpo- established a new field that includes computer
rate governance structure is ‘a bridge too far’ for science, economics, engineering, organizational
most SMEs. science, managerial science, operational research,
Corporate governance is a responsibility del- and business and information science. According
egated by shareholders and the public, defined by to Gregor (2006), IS theories only gain meaning in
legislators and regulators, and shared by boards the objectivity that exists in an abstract world of
- and in some measure - with managers. However, man-made entities. We believe this is also true for
the right of ownership is not clearly separated D&M and TAM. Both of these theories are mainly
from the right to control in SMEs, certainly not explanatory, although they also contain causality
in family-owned SMEs, and therefore, an implicit and the power to make generalizations. In addi-
delegation of the managerial responsibility is not tion, both theories encompass ideas that provide
given from the shareholders to the management. ways to think about other, more-or-less related
Therefore, the true nature of IT governance can- theories. All four theories provide a solid base for
not be fully incorporated into SMEs. This calls explaining behavioral uncertainty. According to
for a rethinking of the concept of IT governance the Gregor’s classification of IS theories, TPB is
when applied in SMEs. a predicting theory (type 3), TCE is an analyzing
theory (type 1) and TAM and D&M are explaining
and predicting theories (type 4) (Gregor, 2006).
THEORETICAL FRAMEWORK On the right-hand diagram of Figure 9, we placed
the four theories in a two-dimensional graph. The
In order to achieve a better understanding of the first dimension, or Y-axis is the explanatory power
phenomenon of IS success in SMEs, a multidis- of the theory and the second dimension, or X-axis,
ciplinary and multifaceted approach is assumed. represents the predicting power of the theory.
The multidisciplinary approach stems from a Woven together, they have a synergetic effect
variety of scientific disciplines, such as econom- so that more analyzing, explaining and predicting
ics, operational research, and psychology, which power emerges. For each of the theories, we cre-
are used as sources to deliver theories to our ated a nomological network with theoretical
framework. The multifaceted approach embod- constructs that can be empirically checked. No-
ies a behavioral perspective as well as a more mological networks where designed by Cronbach
situational perspective, which is directly linked and Meehl to assure construct validity (Cronbach
to the IT artifact (e.g., ERP and CRM) and the & Meehl, 1955). Although nomological networks
organizational context, which is here, the SME. were first used to support psychological tests, the
The choice of theories stems from the literature concept can be used for any theoretical setting
on SMEs and IT (Devos, Hendrik Van Lande- that needs to be empirically validated. We discuss
ghem, & Dirk Deschoolmeester, 2010). Although each of the theories separately in the next sub-
the range of theories and theoretical foci used in sections.
publications on SMEs and IT is diverse, DeLone
84
Figure 9. Domain of knowledge based on the OISF framework (left graph) and on the compound frame-
work (right graph)
DELONE & MCCLEAN IS They state that “For each research endeavor, the
SUCCESS MODEL (D&M) selection of IS success dimensions and measures
should be contingent on the objectives and context
One of the major problems of IS success is the of the empirical investigation, but, where possible,
ambiguity of the concept and the multiplicity of tested and proven measures should be used” and
its constructs (Rai et al., 2002). DeLone and Mc- “Where possible, we advocate the application of
Clean tackled the problem by proposing a multi- existing, validated measures rather than the de-
dimensional framework based on different layers velopment of new measures” (DeLone & McLean,
(DeLone & McLean, 1992). In a semantic layer, 2003). In our work, however we used measures
the model shows how successful the transmission from the Cobit model (IT Governance Institute,
of information is, in terms of meaning. This facet 2007b). Measuring the construct ‘Use’ can be
of success is measured by the effect the informa- done by questioning the users and by objective
tion has on the receiver. The technical layer of measures performed by the system. However, the
the model defines the efficiency of the delivery two results can be conflicting.
of information and is based on the communica- The constructs, the measures and their inter-
tion theory of Shannon and Weaver (Shannon dependencies of the updated IS success model are
& Weaver, 1969). The model has six dependent shown in Figure 1. D&M is probably the most
constructs: System Quality, Information Quality, invoked IS theory in empirical studies (Ifinedo
Use, User Satisfaction, Individual Impact and & Nahar, 2009; Iivari, 2005; Petter, DeLone,
Organizational Impact. Ten years after the initial & McLean, 2008). D&M has also been used to
presentation of the model, a critical review was measure user information satisfaction in SMEs
conducted by the original authors, which resulted (Armstrong, Fogarty, Dingsdag, & Dimbleby,
in an updated model (DeLone & McLean, 2003). 2005) and to investigate the role of IT governance
The authors were aware of the difficulties that often in SMEs (Bernroider, 2008).
arise with operationalizing theoretical models.
85
Figure 1. Nomological network of the updated IS Success Model (DeLone & McLean, 2003)
Figure 2. Nomological network of theory of planned behavior (Ajzen, 1991)
Theory of Planned Behavior decision is available and that the decision itself
is strategically important. TPB is of special inter-
The central construct of the theory of planned est for decision making by an individual. This is
behavior (TPB) is the intention to adopt a certain usually the case in SMEs, where the CEO is the
behavior (Ajzen, 1991). Intention steers behavior. central decision-making authority. TPB gives an
TPB models an active decision process that can explanation of the importance of information about
be of use in making the decision to adopt IT. the decision, ex ante. The theory is often used in
There are, however, restrictions in the applica- cases in which one tries to predict the likelihood
tion of TPB for decision making in IT. Harrison of the decision options. The constructs of TPB,
et al., (1997) assume that information about the as well as the measures, are shown in Figure 2.
86
Figure 3. Nomological net of technology acceptance model (Davis, 1989; Venkatesh & Bala, 2008)
‘Behavior to adopt’ is directly proportional to person believes that using a particular system
the ‘intention to adopt’, which is positively re- would be free from effort.
lated to attitude, subject norms, and perceived In 2000, Venkatesh and Davis proposed an
behavioral control. To find measures for the con- extension for TAM (Venkatesh & Davis, 2000).
structs of TPB, subconstructs are used. For each The extension was called TAM2 and it took into
construct, there are two subconstructs that can be account social influences, such as subjective
measured easily by a set of survey questions. TPB norms, cognitive processes such as job relevance,
has already been used to explain the decision- and quality of results. These extensions had an
making process in SMEs (Harrison et al., 1997; impact on the PU construct. TAM2 was refined
Nasco et al., 2008; Riemenschneider, Harrison, again in 2008 (Venkatesh & Bala, 2008). The
& Mykytyn, 2003; Riemenschneider & McKin- focus was now on the managerial decision mak-
ney, 2001). ing from an organizational perspective. A third
version of TAM, TAM3 was proposed.
Technology Acceptance Model TAM has also had some critical remarks. Ac-
cording to Benbazat, TAM has three dysfunctional
The technology acceptance model (TAM) is gen- results: 1) there are a number of versions of TAM
erally referred to as the most influential theory in and TAM-like models so it is not always very clear
IS research (Lee, Kozar, & Larsen, 2003). TAM is what version researchers should use to compare
often used in SME research to explain the adoption findings, 2) TAM has led the attention of research-
of IT/IS (Grandon & Pearson, 2004; Igbaria, Zi- ers away from the IT artifact and the evaluation
natelli, Cragg, & Cavaye, 1997; Merono-Cerdan, of the artifact, and 3) TAM does not predict the
2008; Oh, Cruickshank, & Anderson, 2009). This effects of the adoption of IT and, therefore, it gives
model was originally designed by Davis, and a false sense of progress on knowledge building
originates from the theory of reasoned action of the real impact of IT on the individual or on the
(Davis, 1986; Davis, 1989). According to TAM, organization. (Benbasat & Barki, 2007). In a lot
only two constructs influence the behavioral in- of studies, the importance of the TAM constructs,
tention to use an information system positively. PU and PEOU, are accentuated but less effort goes
These constructs are: perceived usefulness (PU) into the usefulness of the artifact for the organiza-
and perceived ease of use (PEOU), as is shown tion. Therefore, measurement of the ‘impact’ of
in Figure 3. Perceived usefulness is the degree the construct is needed. Too many researchers
to which a person believes that using a particular have considered PU and PEOU as black boxes and
system would enhance his or her job performance. have shown little interest in their real content and
Perceived ease of use is the degree to which a meaning (Benbasat & Barki, 2007). In this work,
87
Figure 4. Transaction cost economics model for SMEs
we see TAM only as a part of larger, explanatory variety (dominance of the CEO) (Nooteboom,
model and as a supporting theory for the use of 1993). Externally, on the market side, the SME
the constructs of D&M. The constructs for TAM is confronted with the opportunistic behavior of
are directly measurable, as has been shown in a suppliers and independent software vendors (ISV).
lot of literature on the application of TAM. The decision to provide IT services internally
or externally will take into account three environ-
Transaction Cost Economics mental factors: 1) asset specific, 2) complexity
and frequency, and 3) uncertainty. According
Transaction cost economics (TCE) is one of the to Dibbern and Heinzl (2009), asset specific
cornerstones of the theory of the firm and is all factors for IT/IS are not taken into account in
about the “make-or-buy” decision (Williamson, outsourcing decisions by SMEs. The complexity
1975; Williamson, 1981). TCE is mainly used and the frequency factors are both low in SMEs.
in research on outsourcing (Aubert et al., 2004; Important or strategic decisions on adoption of
Bahli & Rivard, 2003; Carmel & Nicholson, 2005; IT/IS are not often made, not even on a yearly
Dibbern & Heinzl, 2009). According to TCE, the basis, in SMEs (Devos et al., 2011). Uncertainty
decision to provide IT services internally or exter- is an essential determinant in the “make-or-buy”
nally is based on a cost assessment which takes decision; however it is not feasible to write a
production and transaction costs into account. ‘waterproof’ contract, since it would simply cost
The latter costs are determined by contingency too much time, effort and resources to deal with
factors induced by the environmental context of all contingencies that arise during an IS project.
the organization. The economic environment does A complete contract would have to specify all
not offer perfect markets in which information is legal consequences of every possible state of the
available for all participants and where competi- project; it is clear that for IS contracts, this is not
tion is ideal. Organizations will try to compensate possible. Anderlini and Felli (2004) state: “the
for the imperfection of the market by making contracting parties may lack the necessary degree
the necessary transaction costs like monitoring, of rationality necessary to describe exactly the
controlling and governing costs. various states of nature in the ex-ante contract
A SME, like all organizations faced with the they draw up”. Therefore, it is most likely that
decision to adopt IT/IS, is confronted with parties will end up with a very incomplete contract.
bounded rationality. According to Nooteboom, Next, in the TCE model, is the cost comparison of
rationality in SMEs is more bounded along three internal and external transaction costs. Not a lot
dimensions: width (fewer functional areas in staff is known about this process in SMEs, but there is
support), depth (lower level of education), and a high expectation that resource deficits in terms
88
Figure 5. The nomological network for the TCE model for SMEs
of IT know-how, education, and governance will followed the logic of the IS Net of Benbazat and
contribute strongly to the option of externally Zmud (2003). The IS Net is presented in Figure 6.
supported IT/IS services and will outweigh the The constructs of the IS Net are:
internal sourcing of the IS function (Dibbern &
Heinzl, 2009). Finally the production costs refer • Managerial methodological and tech-
to the acquisition of the IT artifact. nological capabilities and practices
TCE is an analyzing theory with less predict- (Management)
ing and explaining power than D&M and TAM. • IT artifact
Therefore, to make a nomological network of • Use of the artifact (Usage)
TCE, we first made a conceptual scheme of TCE, • Impact on the organization and on the indi-
which is presented in Figure 3. Bounded rationality vidual (Net benefits)
and opportunism are human factors and can be
measured by observations. Complexity, frequency, For our framework, we added the construct of
asset specificity and uncertainty are environmental the market. SMEs tend to outsource their IS func-
factors which can be measured by audits and as- tion and only seldom is an information system fully
sessments. Transaction and production costs can developed with the native resources of a SME. The
be measured by a cost assessment. In Figure 5, market is therefore an important construct where
the nomological network of TCE is presented. specific forces play an important role. Devos et al
argue that the market where SMEs meet indepen-
The Compound Model dent software vendors (ISV) is actually a Lemon
market since there is information asymmetry
The aforementioned basic theories have enabled and the risks of an adverse selection. The moral
allow us to combine previous research efforts hazards loom largely (Devos, Van Landeghem, &
and to look at SMEs from a different perspective. Deschoolmeester, 2010). However, the decision
Starting from the strategic organizational perspec- to adopt IT is preceded by the intention to solve
tive of IT, a compound model or framework was a problem or to provide a solution to the existing
composed, based on the four theories. Our analysis needs of the organization. From a behavioral point
followed the plan-do-run-check cycle adopted of view, the TPB can be used to identify which
from research on quality management and well beliefs influence the decision makers to start up
used by practitioners mainly through the norma- a project to adopt IT/IS. So TPB contributes to
tive ISO 9000 standard (Tsim, Yeung, & Leung, the clarification of the construct of managerial
2002). From a more conceptual perspective, we capabilities and practices of the IS Net.
89
Figure 6. The nomological IS Net (Benbasat & Zmud, 2003)
Table 1. The contribution of the theories to the explanation of the IS Net
Once the requirements are identified, albeit information and system quality of D&M also take
embryonic, TCE helps to describe the factors the artifact into account. Thus, D&M contributes
that influence the search for the optimal form of to the explanation of the constructs of usage, IT
conducting the project. The same as in the TPB artifact, and impact. Each of the theories offer a
model, TCE contributes to the clarification of the partial clarification of the IS Net, extended with
market construct and the construct of the manage- the market construct. The contribution of each
rial capabilities and practices of the IS Net. theory to the constructs of the extended IS Net is
Since most SMEs only have access to limited summarized in Table 1.
resources, they need a clear explanation of the We also investigated how close each theory
process of acquiring IT. In the ‘use’ phase of the IS gets to an elucidation of IT governance in a SME
nomological network, TAM offers the explanatory organization. The concept of IT governance in a
constructs. The SME can only achieve success SME includes a substantial part of the IT manage-
through actual and efficient use of the IT artifact ment, because in SMEs, there is a very thin line
and TAM contributes to the explanation of this between the management and the board. Thus, IT
usage construct of the IS Net. management is actually the real starting point of
Finally, the DeLone & McLean IS success the model. It is also the starting point of the IS
model (D&M) provides the means to measure the Net. We focused on the behavioral intentions of
IS success constructs through the impact of the the management to adopt an IT artifact. This phase
IS on the SME organization. The constructs of can be explained by TPB. The transformation of
90
Figure 7. Global schema of the framework based on the IS Net
Figure 8. Framework to explain IS Success in SMEs
the intention into a real decision to adopt is pre- an ISV (Devos, Van Landeghem, & Deschool-
ceded by a make-or-buy decision, ideally modeled meester, 2009). This frameworks draws from
by TCE. The focus of the management will now principal agent theory (PAT), prospect theory
be on the IT artifact and on the use of it. The IT (PT), lemon market theory (LMT), incomplete
artifact will have a certain quality, determined by contract theory (ICT) and organizational trust
the constructs of the D&M model. The ‘use’ (OT). The explanatory and predictive power of
construct is modeled by TAM, which blends well each theory is shown in the left-hand graph in
with D&M. Here, the impact construct of the IS Figure 9. Bringing the two frameworks together
Net is called net benefits. A global schema of the in a compound model enlarges both the explana-
framework is shown in Figure 7. tory and the predictive power of the model and
The detailed framework, where all constructs increases the domain of knowledge in the area of
of the separate theories are brought together, is SMEs and IT. This is illustrated in the right-hand
shown in Figure 8. graph of Figure 9.
This framework is actually an extension of a
previous framework, which was based on out-
sourced IS failure (OISF), and which explains the
typical setting of the SME in collaboration with
91
EMPIRICAL VALIDATION find any contradictions in the way Cobit builds

OF THE FRAMEWORK up the processes.
The success of an IS greatly depends on the
In order to validate our framework, we examined amount of investment in complementary assets,
the extent to which our theoretical model can pro- such as training and documentation. Complemen-
vide support for the Cobit framework, often used tary assets do get a lot of attention in Cobit. For
by practitioners as an IT governance framework example, the AI4 process focuses completely on
and also suitable for SMEs (IT Governancen In- these assets. In the framework, we cannot deal
stitue, 2007a, 2007b). We found it useful to map directly with these kinds of investment. However,
our theoretical model with the Cobit 4.1 model, we can consider complementary assets as being
and to compare the control objectives of some part of the information quality, linking it into the
of the processes of Cobit with the explanatory, D&M model.
predicting, and analyzing authority of the newly Surprisingly, the constructs, PU and PEOU,
established theoretical foundations. which lead to user satisfaction according to TAM,
There are some major conceptual differences are not even detectable in Cobit. Although this
between our compound model and Cobit. Our is a flaw in Cobit in our opinion, it is consistent
model is deductively built up from existing with the normative character of Cobit, as a gov-
theoretical constructs. Cobit is inductively built ernance model for large public companies. For
up from observations, best practices, and the those kinds of organization, maximum control
experiences of practitioners. Although the Cobit of the business is needed. Cobit was historically
framework is strongly normative, it suffers from developed as a set of guidelines for IT auditors,
a serious lack of theoretical underpinnings. There and it largely disregards a behavioral approach
is no explicit theoretical foundation spelled out by to the IS function. This is probably one of the
the creators of Cobit, although one could derive a reasons why Cobit is not often used in smaller
strong adherence to formal control theory (Kirsch, private companies.
1997). Notwithstanding this, Cobit is probably In the findings, we have shown that our com-
the only control framework that encompasses a posed model provides a comprehensive overview
full managerial quality-cycle for the IS function. of essential steps and influences that are needed
For the purposes of our inquiry, we looked upon in IT governance. Our model also makes a clear
the most relevant Cobit processes for SMEs. distinction between activities ex ante and ex post
We focused on the processes in the acquire and of an actual implementation of an IT artifact into
implement (AI) domain of Cobit. For each process a SME organization.
in the AI domain, we made a link between the
Cobit goals for IT governance and the theoreti-
cal foundations of our framework. The mapping CONCLUSION
of the Cobit AI domain and our framework are
shown in Table 2. Our findings show that our framework offers a
The mapping shows that the framework does surprising coherence and proposes a strong theo-
indeed offer a possible theoretical foundation for retical foundation for the normative directions
some of the Cobit control objectives, although of the methods used in Cobit by IT practitioners.
we cannot explain each normative directive of The compound model provides an enhanced level
Cobit. Only the basic questions behind the control of explanation of IT/IS governance, compared
statements can be answered. However, we did not to frameworks, such as Cobit, which are often
considered as too normative, with few behavioral
92
Table 2. The mapping of the detailed control objectives of the Cobit AI domain with the framework
AI1: Identify Automated Solutions

Theoretical foundation Cobit – IT Governance goals
TPB What is needed? To ensure that business requirements are satisfied effectively
Why is it needed? and efficiently
Do we do it? Definition of needs
TCE At what cost? Analysis before acquisition
Where? Consideration of alternative sources
How? Review technological and economic feasibility
Risk and cost benefit analysis
Minimize costs
AI2: Acquire and Maintain Application Software
TCE What is the source of the cost Analysis before acquisition
(contract or control cost)? Consideration of alternative sources
Review technological and economic feasibility
Risk and cost benefit analysis
Minimize costs
AI3: Acquire and Maintain Technology Infrastructure
D&M Quality of Service Planned approach for acquisition, implementation and
upgrade of the technology infrastructure
To ensure ongoing technological support
AI4: Enable Operation and USe
D&M Quality of Service Provide training & documentation
User satisfaction Knowledge transfer and communication to users, support
staff and management
AI5: Procure IT resources
TCE What is the source of the cost Improving IT’s cost-efficiency
(contract or control cost)?
AI6: Manage Changes
D&M Quality of Service Defining and communicating change procedures
Quality of Systems Assessing, prioritising and authorising changes
Quality of Information Tracking status and reporting on changes
AI7: Install and Accredit Solutions and Changes
D&M Net benefits aligned with system Testing in a dedicated environment with relevant data
TPB requirements Release planning: definition of roll-out and migration
instructions
Post-implementation review: evaluating and approving test
results by business management
directives. The basic thoughts in existing IT gov- Our model is built up from a specific SME
ernance models are based on control theory and environment, in which certain characteristics in
they require a mature management to implement. relation to IT/IS, such as resource deficit and de-
93
pendence on external IT expertise, are taken into REFERENCES

account. Also, the egocentric role of the CEO in
SMEs is incorporated in our model. The constraint Ajzen, I. (1991). The theory of planned behavior.
of TPB, that decisions have to be made by an Organizational Behavior and Human Decision
individual, fits very well here. Our framework is Processes, 50(2), 179–211. doi:10.1016/0749-
less ‘mechanical’ and much more ‘behavioral’. 5978(91)90020-T
The practices used in SMEs to implement Alpar, P., & Reeves, S. (1990). Predictors of MS
and support information systems are not well or application in small businesses. Interfaces,
documented and are usually present only in the 20(2), 2–11. doi:10.1287/inte.20.2.2
form of human knowledge and ad hoc experience.
Measuring IS success in SMEs is not straightfor- Anderlini, L., & Felli, L. (2004). Bounded ratio-
ward, and is done (if done at all) by questioning the nality and incomplete contracts. Research in Eco-
users. This study has established new theoretical nomics, 58, 3–30. doi:10.1016/j.rie.2003.12.001
foundations for explaining IS success in SMEs.
Antlova, K. (2009). Motivation and barriers of ict
The goal of this study has been reached by
adoption in small and medium-sized enterprises.
using a ‘combine and conquer’ strategy in order
E & M Ekonomie a Management, 12(2), 140-155.
to achieve synergy between theories and common
practice. This has led to a better understanding Armstrong, B., Fogarty, G., Dingsdag, D., &
of IS success in SMEs. Future research should Dimbleby, J. (2005). Validation of a computer user
examine the possible link with other theories, in satisfaction questionnaire to measure IS success in
the context of SMEs. There are many ways of ap- small business. Journal of Research and Practice
proaching this matter, which would only enhance in Information Technology, 37(1), 27–42.
the multifaceted nature of the subject.
Aubert, B. A., Patry, M., & Rivard, S. (2005). A
The contribution of this work is bringing clari-
framework for Information Technology outsourc-
fication for practitioners and CEO’s in SMEs so
ing risk management. The Data Base for Advances
that they can keep a structured overview of the IS
in Information Systems, 36(4), 9–28.
function. The framework offers guidance from a
theoretical point of view. The study also includes Aubert, B. A., Rivard, S., & Patry, M. (2004).
a critical review of existing, normative control A transaction cost model of IT outsourcing.
frameworks, which are not always useful in SMEs. Information & Management, 41(7), 921–932.
Finally, it should be remarked that this work doi:10.1016/j.im.2003.09.001
has some limitations. First, the model has not been
Audretsch, D. B., Prince, Y. M., & Thurik, R.
validated using a large group of SMEs. The model
(1999). Do small firms compete with large firms.
was built up and was only empirically checked
Atlantic Economic Journal, 27(2), 201–209.
by the Cobit guidelines. Surveys of the basic
doi:10.1007/BF02300239
promises of the framework would be welcome.
Second, more research is needed on the adequate Avison, D., Gregor, S., & Wilson, D. (2006). Mana-
use of existing theories and the development of gerial IT unconsciousness. Communications of the
new theories to explain IS success in SMEs. The ACM, 49(7), 89–93. doi:10.1145/1139922.1139923
phenomenon is so rich in its appearance that more
perspectives can be chosen for investigation.
94
Bahli, B., & Rivard, S. (2003). The Information Chang, S. I., Hung, S. Y., Yen, D. C., & Lee, P. J.
Technology outsourcing risk: A transaction cost (2010). Critical factors of ERP adoption for small-
and agency theory-based perspective. Journal and medium- sized enterprises: An empirical study.
of Information Technology, 18(3), 211–221. Journal of Global Information Management,
doi:10.1080/0268396032000130214 18(3), 82–106. doi:10.4018/jgim.2010070104
Bartis, E., & Mitev, N. (2008). A multiple nar- Commission, E. (2003). Commission recom-
rative approach to information systems failure: mendation 2003/361/EC. Official Journal of the
A successful system that failed. European Jour- European Union, 46(L 124), 6.
nal of Information Systems, 17(2), 112–124.
Commission, E. (2004). Go digital awareness
doi:10.1057/ejis.2008.3
campaign 2001-2003: The main lessons to be
Benbasat, I., & Barki, H. (2007). Quo vadis, learnt.
TAM? Journal of the Association for Information
Conboy, K. (2010). Project failure en masse: A
Systems, 8(4), 211–218.
study of loose budgetary control in ISD projects.
Benbasat, I., & Zmud, R. W. (2003). The iden- European Journal of Information Systems, 19(3),
tity crisis within the is discipline: Defining and 273–287. doi:10.1057/ejis.2010.7
communicating the discipline’s core properties.
Cragg, P. (2008). Identifying key Information
Systems competencies in small firms. Total Qual-
27(2), 183–194.
ity Management & Business Excellence, 19(1-2),
Bernroider, E. W. N. (2008). IT governance for 29–35. doi:10.1080/14783360701601926
enterprise resource planning supported by the
Cronbach, L. J., & Meehl, P. E. (1955). Construct
DeLone-McLean model of Information Systems
vaildity in psychological tests. Psychological
success. Information & Management, 45(5),
Bulletin, 52.
257–269. doi:10.1016/j.im.2007.11.004
Davis, F. D. (1986). A technology acceptance
Bharadwaj, A., Keil, M., & Mahring, M. (2009).
model for empirically testing new end-user
Effects of Information Technology failures on the
Information Systems: Theory and results. Mas-
market value of firms. The Journal of Strategic
sachusetts Institute of Technology: Doctoral dis-
Information Systems, 18(2), 66–79. doi:10.1016/j.
sertation, Sloan School of Management.
jsis.2009.04.001
Bolton, J. E. (1971). Small firms: Report of the
perceived ease of use, and user acceptance of
committee of inquiry on small firms. London, UK:
Information Technology. Management Infor-
Department of Trade and Industry.
mation Systems Quarterly, 13(3), 319–340.
Carmel, E., & Nicholson, B. (2005). Small doi:10.2307/249008
firms and offshore software outsourcing: High
DeLone, W. H., & McLean, E. R. (1992). Infor-
transaction costs and their mitigation. Journal of
mation Systems success: The quest for the inde-
Global Information Management, 13(3), 33–54.
pendent variable. Information Systems Research,
doi:10.4018/jgim.2005070103
3(1), 60–95. doi:10.1287/isre.3.1.60
95
DeLone, W. H., & McLean, E. R. (2003). The Dibrell, C., Davis, P. S., & Craig, J. (2008).
DeLone and McLean model of Information Fueling innovation through Information Tech-
Systems success: A ten-year update. Journal of nology in SMEs. Journal of Small Business
Management Information Systems, 19(4), 9–30. Management, 46(2), 203–218. doi:10.1111/
j.1540-627X.2008.00240.x
Devos, J., Van Landeghem, H., & Deschool-
meester, D. (2008). Outsourced Information Grandon, E. E., & Pearson, J. M. (2004). Elec-
Systems Failures in SMEs: A multiple case study. tronic commerce adoption: An empirical study of
Electronic Journal of Information Systems Evalu- small and medium US businesses. Information
ation, 11(2), 73–84. & Management, 42(1), 197–216. doi:10.1016/j.
im.2003.12.010
meester, D. (2009). IT governance in SMEs: A Gregor, S. (2006). The nature of theory in informa-
Theoretical framework based on the outsourced tion systems. Management Information Systems
Information System failure. The 3rd European Quarterly, 30(3), 611–642.
Conference on Information Management and
Group, S. (2004). Third quarter research report.
Evaluation.
The Standish Group International.
Harland, C. M., Caldwell, N. D., Powell, P., &
meester, D. (2010). IT/IS and small & medium-
Zheng, J. (2007). Barriers to supply chain informa-
sized enterprises: Literature overview. Ghent
tion integration: SMEs adrift of eLands. Journal
University Association.
of Operations Management, 25(6), 1234–1254.
Devos, J., Van Landeghem, H., & Deschool- doi:10.1016/j.jom.2007.01.004
meester, D. (2010). The theory of the lemon
Harrison, D. A., Mykytyn, P. P., & Riemensch-
markets in IS research. In Dwivedi, Y. K., Wade,
neider, C. K. (1997). Executive decisions about
M., & Schneberger, S. L. (Eds.), Information
adoption of Information Technology in small
Systems theory: Explaining and predicting our
business: Theory and empirical tests. Information
digital society. Springer. doi:10.1007/978-1-
Systems Research, 8(2), 171–195. doi:10.1287/
4419-6108-2_11
isre.8.2.171
Huang, R., Zmud, R. W., & Price, R. L. (2010).
meester, D. (2011). Does size and sector matter?
Influencing the effectiveness of IT governance
A discussion of the idiographic character of the
practices through steering committees and com-
relation SMEs and IT/IS. Howest.
munication policies. European Journal of Infor-
Dibbern, J., & Heinzl, A. (2009). Outsourcing mation Systems, 19(3), 288–302. doi:10.1057/
of Information Systems functions in small and ejis.2010.16
medium sized enterprises: A test of a multi-
Ifinedo, P., & Nahar, N. (2009). Interactions
theoretical model. Business & Information Sys-
between contingency, organizational IT fac-
tems Engineering, 1(1), 101–110. doi:10.1007/
tors, and ERP success. Industrial Manage-
s12599-008-0008-1
ment & Data Systems, 109(1-2), 118–137.
doi:10.1108/02635570910926627
96
Igbaria, M., Zinatelli, N., Cragg, P., & Cavaye, Levy, M., & Powell, P. (2004). Strategies for
A. L. M. (1997). Personal computing acceptance growth in SMEs: The role of information and
factors in small firms: A structural equation model. Information Systems. Oxford, UK: Elsevier.
Levy, M., & Powell, P. (2008). Small firm trans-
21(3), 279–305. doi:10.2307/249498
formation through IS. International Journal of
Iivari, J. (2005). An empirical test of the DeLOne- Technology Management, 43(1-3), 123–141.
McLean model of Information Systems success. doi:10.1504/IJTM.2008.019411
The Data Base for Advances in Information Sys-
Malhotra, R., & Temponi, C. (2010). Critical
tems, 36(2), 8–27.
decisions for ERP integration: Small business
IT Governance Institute. (2003). Board brief- issues. International Journal of Information
ing on IT governance. Rolling Meadows, IL: IT Management, 30(1), 28–37. doi:10.1016/j.ijin-
Governance Institute. fomgt.2009.03.001
IT Governance Institute. (2007a). Cobit quickstart Marcati, A., Guido, G., & Peluso, A. M. (2008).
(2nd ed.). IT Governance Institute. The role of SME entrepreneurs’ innovativeness
and personality in the adoption of innovations.
IT Governance Institute. (2007b). Cobit v.4.1. IT
Research Policy, 37(9), 1579–1590. doi:10.1016/j.
Governance Institute.
respol.2008.06.004
Kirsch, L. J. (1997). Portfolios of control modes
Merono-Cerdan, A. L. (2008). Groupware uses
and IS project management. Information Systems
and influence on performance in SMEs. Journal
Research, 8(3), 215–239. doi:10.1287/isre.8.3.215
of Computer Information Systems, 48(4), 87–96.
Lee, S., & Kim, K. J. (2007). Factors affecting the
Nasco, S. A., Toledo, E. G., & Mykytyn, P.
implementation success of Internet-based infor-
P. (2008). Predicting electronic commerce
mation systems. Computers in Human Behavior,
adoption in Chilean SMEs. Journal of Busi-
23(4), 1853–1880. doi:10.1016/j.chb.2005.12.001
ness Research, 61(6), 697–705. doi:10.1016/j.
Lee, S. M., Kim, J., Choi, Y., & Lee, S. G. (2009). jbusres.2007.06.047
Effects of IT knowledge and media selection on op-
Natovich, J. (2003). Vendor related risks in
erational performance of small firms. Small Busi-
IT development: A chronology of an out-
ness Economics, 32(3), 241–257. doi:10.1007/
sourced project failure. Technology Analysis
s11187-007-9095-5
and Strategic Management, 15(4), 409–419.
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003). doi:10.1080/095373203000136015
The technology acceptance model: Past, present,
Nooteboom, B. (1993). Firm size effects on trans-
and the future (Vol. 12, pp. 752-780). Communi-
action costs. Small Business Economics, 5(4),
cations of Association for Information Systems.
283–295. doi:10.1007/BF01516249
Lefebvre, L. A., Mason, R., & Lefebvre, E.
OECD. (2005). OICT, E-business and
(1997). The influence prism in SMEs: The power
SMEs. OECD Digital Economy Papers, 88.
of CEOs’ perceptions on technology policy and
doi:doi:10.1787/232475651473
its organizational impacts. Management Science,
43(6), 856–878. doi:10.1287/mnsc.43.6.856
97
Oh, K. Y., Cruickshank, D., & Anderson, A. R. Schwarz, A., & Hirschheim, R. (2003). An extend-
(2009). The adoption of e-trade innovations by ed platform logic perspective of IT governance:
Korean small and medium sized firms. Techno- managing perceptions and activities of IT. The
vation, 29(2), 110–121. doi:10.1016/j.technova- Journal of Strategic Information Systems, 12(2),
tion.2008.08.001 129–166. doi:10.1016/S0963-8687(03)00021-0
Olsen, K. A., & Saetre, P. (2007). IT for niche Seddon, P. B. (1997). A respecification and ex-
companies: Is an ERP system the solution? tension of the DeLone and McLean model of IS
Information Systems Journal, 17(1), 37–58. success. Information Systems Research, 8(3),
doi:10.1111/j.1365-2575.2006.00229.x 240–253. doi:10.1287/isre.8.3.240
Peterson, R. (2004). Crafting Information Technol- Shannon, C., & Weaver, W. (1969). The math-
ogy governance. Information Systems Manage- ematical theory of communication. University
ment, 21(4), 7–22. doi:10.1201/1078/44705.21.4 of Illinois Press.
.20040901/84183.2
Snider, B., da Silveira, G. J. C., & Balakrishnan,
Petter, S., DeLone, W., & McLean, E. (2008). J. (2009). ERP implementation at SMEs: Analysis
Measuring Information Systems success: Models, of five Canadian cases. International Journal of
dimensions, measures, and interrelationships. Operations & Production Management, 29(1-2),
European Journal of Information Systems, 17(3), 4–29. doi:10.1108/01443570910925343
236–263. doi:10.1057/ejis.2008.15
Southern, A., & Tilley, F. (2000). Small firms
Premkumar, G. (2003). A meta-analysis of research and information and communication technolo-
on information technology implementation in gies (ICTs): Toward a typology of ICTs usage.
small business. Journal of Organizational Com- New Technology, Work and Employment, 15(2),
puting and Electronic Commerce, 13(2), 91–121. 138–154. doi:10.1111/1468-005X.00070
doi:10.1207/S15327744JOCE1302_2
Tan, K. S., Chong, S. C., Lin, B. S., & Eze,
Rai, A., Lang, S. S., & Welker, R. B. (2002). U. C. (2009). Internet-based ICT adoption:
Assessing the validity of IS success models: An Evidence from Malaysian SMEs. Industrial
empirical test and theoretical analysis. Informa- Management & Data Systems, 109(1-2), 224–244.
tion Systems Research, 13(1), 50–69. doi:10.1287/ doi:10.1108/02635570910930118
isre.13.1.50.96
Thong, J. Y. L. (2001). Resource constraints and
Riemenschneider, C. K., Harrison, D. A., & Information Systems implementation in Singa-
Mykytyn, P. P. (2003). Understanding it adoption porean small businesses. Omega-International
decisions in small business: Integrating current Journal of Management Science, 29(2), 143–156.
theories. Information & Management, 40(4), doi:10.1016/S0305-0483(00)00035-9
269–285. doi:10.1016/S0378-7206(02)00010-1
Thong, J. Y. L., & Yap, C. S. (1995). CEO charac-
Riemenschneider, C. K., & McKinney, V. R. teristics, organizational characteristics and Infor-
(2001). Assessing belief differences in small mation Technology adoption in small businesses.
business adopters and non-adopters of Web-based Omega-International Journal of Management
e-commerce. Journal of Computer Information Science, 23(4), 429–442. doi:10.1016/0305-
Systems, 42(2), 101–107. 0483(95)00017-I
Rogers, E. (2003). Diffusion of innovations. Free
Press; Original edition (August 16, 2003).
98
Thong, J. Y. L., Yap, C. S., & Raman, K. S. (1996). Wang, Y., & Ahmed, P. K. (2009). The moderat-
Top management support, external expertise and ing effect of the business strategic orientation
Information Systems implementation in small on eCommerce adoption: Evidence from UK
businesses. Information Systems Research, 7(2), family run SMEs. The Journal of Strategic In-
248–267. doi:10.1287/isre.7.2.248 formation Systems, 18(1), 16–30. doi:10.1016/j.
jsis.2008.11.001
Tsim, Y. C., Yeung, V. W. S., & Leung, E. T. C.
(2002). An adaptation to ISO 9001:2000 for certi- Weill, P., & Ross, J. (2004). IT governance: How
fied organisations. Managerial Auditing Journal, top performers manage it decision rights for su-
17(5), 245–250. doi:10.1108/02686900210429669 perior results. Harvard Business Press.
Venkatesh, V., & Bala, H. (2008). Technology Weill, P., & Ross, J. (2005). A matrixed approach
acceptance model 3 and a research agenda on to designing IT governance. MIT Sloan Manage-
interventions. Decision Sciences, 39(2), 273–315. ment Review, 46(2), 26.
doi:10.1111/j.1540-5915.2008.00192.x
Welsh, J. A., & White, J. F. (1981). A small busi-
Venkatesh, V., & Davis, F. D. (2000). A theo- ness is not a little big business. Harvard Business
retical extension of the technology acceptance Review, 59(4), 18.
model: Four longitudinal field studies. Manage-
Williamson, O. E. (1975). Markets and hierar-
ment Science, 46(2), 186–204. doi:10.1287/
chies: Analysis and antitrust implications. New
mnsc.46.2.186.11926
York, NY: Free Press.
Verhees, F., & Meulenberg, M. T. G. (2004).
Williamson, O. E. (1981). The economics of
Market orientation, innovativeness, product in-
organization - The transaction cost approach.
novation, and performance in small firms. Journal
American Journal of Sociology, 87(3), 548–577.
of Small Business Management, 42(2), 134–154.
doi:10.1086/227496
doi:10.1111/j.1540-627X.2004.00102.x
Yap, C. S., Soh, C. P. P., & Raman, K. S.
Wang, T. C., & Lin, Y. L. (2009). Accurately pre-
(1992). Information Systems success factors in
dicting the success of B2B e-commerce in small
small business. Omega-International Journal
and medium enterprises. Expert Systems with
of Management Science, 20(5-6), 597–609.
Applications, 36(2), 2750–2758. doi:10.1016/j.
doi:10.1016/0305-0483(92)90005-R
eswa.2008.01.033
99
APPENDIX: ACRONYMS USED
CRM: Customer Relation Management

Cobit: Control Objectives for Information and related Technologies
D&M: DeLone and McLean IS Success Model
ERP: Enterprise Resource Planning
IT: Information Technology
IS: Information Systems
SME: Small and medium-sized enterprise
TAM: Technology Acceptance Model
TCE: Transaction Cost Economy
TPB: Theory of Planned Behavior
100
Section 2
IT Projects, Service Management
and Performance
102
Chapter 6
Measuring IT Service
Management Performance:
A Model Development
Francis Gacenga
University of Southern Queensland, Australia
Aileen Cater-Steel
Mark Toleman
Wui-Gee Tan
ABSTRACT
Prompted by the realisation that IT is now seen as a service, with a customer focus and process orienta-
tion, the authors propose a model to measure IT service management (ITSM) performance. Measuring
ITSM performance will enable organisations to demonstrate the benefit from their investment. The model
is based on a systematic literature review that progressed from considering the general areas of organi-
sation performance measurement to examining commonly used performance metrics. Although there
are a number of studies on ITSM implementation, only a few considered the performance measurement
of ITSM. A structured method for the design of the model was adopted through a three-level analysis. A
comparison of existing performance measurement frameworks was first made to identify those that are
suitable for ITSM and that would facilitate communication between the business and IT function. This
was done using appropriate dimensions from past work of various performance measurement research-
ers. The frameworks were then classified along these dimensions to identify their completeness, eliminate
unnecessary dimensions, and identify the natural dimensions for ITSM.
DOI: 10.4018/978-1-4666-0170-3.ch006
Measuring IT Service Management Performance
INTRODUCTION (Dedrick, Gurbaxani, & Kraemer, 2003; Weill,

1992), IS quality (Chang & King, 2005; Pitt,
We have taken a holistic view in development of the Watson, & Kavan, 1995), IS effectiveness (Scott,
model, in integrating approaches such as the bal- 1995; Seddon, Graeser, & Willcocks, 2002) and
anced scorecard (BSC), broad economic perspec- IS performance (Marchand & Raymond, 2008;
tives and service oriented ITSM. The integrated Saunders & Jones, 1992; Son, Weitzel, & Laurent,
approach caters for the wide range of variables 2005; van der Zee & de Jong, 1999).
of interest and the large number of stakeholders We will focus on the performance measurement
involved in the performance of ITSM. It is envis- of IT service management (ITSM). Performance
aged that the proposed model will provide a basis measurement of ITSM concerns the effectiveness
of standardisation and performance comparison and efficiency of the IT capabilities and use of IT
for organisations implementing ITSM. In the next resources in terms of productivity, profitability and
stage of the research the proposed model will be quality. We apply the recommendation of Tangen
evaluated by ITSM practitioners for its relevance (2005, p. 40) that performance measurement
and usefulness. should be understood as a broad term that “covers
Information systems (IS) academics and IS both overall economic and operational aspects.”
practitioners have grappled with the measurement The objective of this chapter is to propose a
of effectiveness and performance of IT/IS func- model that can be used by academics and practi-
tion. The motivation for this interest is explained tioners to measure the performance of ITSM. Our
by DeLone and McLean (2003) who state “the model is developed from reviewing management
measurement of IS success or effectiveness is and information systems literature.
critical to our understanding of the value and
efficacy of IS management actions and IS invest-
ments.” More broadly Richard, Devinney, Yip and BACKGROUND
Johnson (2009, p. 719) state that “organisational
performance is the ultimate dependent variable of The growth of the service economy has resulted in
interest for researchers concerned with just about service oriented thinking as organisations attempt
any area of management.” to increase value delivered to their customers.
At the organisational level the challenge of This shift in service orientation is manifested in
measuring performance has been recognised and IT/IS as information technology service manage-
many performance measurement frameworks and ment (ITSM) (Galup, Dattero, Quan, & Conger,
models have been proposed such as SERVQUAL 2009). Organisations are faced with the challenge
(Parasuraman, Zeithaml, & Berry, 1985), Sink and of managing IT as a collection of services deliv-
Tuttle model (Sink & Tuttle, 1989), results and ered to the customer, by administering a complex
determinants framework (Fitzgerald, Johnston, portfolio of capabilities, resources and offerings.
Brignall, Silvestro, & Voss, 1994), balanced IT/IS organisations can no longer solely focus on
scorecard (BSC) (Kaplan & Norton, 1992), the composite technologies and systems but need
performance pyramid (Lynch & Cross, 1993) to manage the resultant IT services. IT services
and the performance prism (Neely, Adams, & are provided by IT/IS departments internal to the
Kennerley, 2002). organisation as well as managed by service provid-
Approaches to resolving the challenge of per- ers external to the organisation. An example of
formance measurement of the IS function have managed services is observed in cloud computing
also taken many forms, for example, IS success which has three service models: cloud software as
(DeLone & McLean, 2003), IS productivity
103
a service, cloud platform as a service and cloud Weitzel and Laurent (2005) who state that the
infrastructure as a service (Mell & Grance, 2009). control and governance of the IT function is be-
Previously information systems management coming quite critical in organisations due to the
was technology oriented and placed emphasis significant investment in IT function expenditure.
on systems. The focus has shifted with the wide Measuring IT investment has been problematic
adoption of IT service management resulting in for many years and been described as the “IT
a customer focus and process orientation. The productivity paradox” – the inability to “validate
focus is on the output of IT departments: IT a consistent relationship between IT investments
services which are viewed as the culmination of and firm performance” (Barua & Lee, 1997, p.
end-to-end IS processes generated through IS/ 146). Although it has been claimed that the IT
IT capabilities and resources. Given this shift productivity paradox has been resolved (Bryn-
in focus to IT services and the new practices in jolfsson & Hitt, 1998; Dedrick, et al., 2003), a
the form of IT service management (ITSM) how similar challenge of performance measurement
should we measure organisational performance? of ITSM is evident. Aligning IT and business was
Few studies document the problem of mea- recently ranked in the top five key management
suring and reporting the performance of IS (e.g. concerns based on 243 responses, and has been
Chang & King, 2005; DeLone & McLean, 1992; the top concern for IT managers for almost thirty
Pitt, et al., 1995; Scott, 1995; Seddon, et al., 2002) years (Luftman & Ben-Zvi, 2010). Therefore it is
and identify the area of performance measurement worthwhile to develop a performance measure-
of the IT function as under researched (Chang ment framework for ITSM to help business and
& King, 2005). This lack of research in the area IT managers.
is also noted by Dehning and Richardson who This chapter develops an ITSM performance
report that “never before has IT played such an measurement framework that is important to
important role in the existence of companies, yet organisations for a number of reasons. Though
the overall impact of IT on performance remains organisations report benefits from their ITSM
largely an unexplained puzzle” (2002, p. 27). They implementation few studies have shown sup-
also state that the least developed area within the porting metrics used to derive reported benefits.
strategic use of IT, is the relation between IT and ITSM performance measurement is also impor-
performance. tant to organisations because of the size of the
It is reported that IT operations get the larg- investment of IT expenditure and competitive
est share of IT expenditure with estimates of 70 advantage that IT management can deliver to an
percent (Tracy, Guevara, Harcourt, & Stegman, organisation. It is beneficial for organisations to
2006). Gartner predicted that worldwide IT spend- measure the performance of their ITSM and it has
ing would reach “$3.6 trillion in 2011, a 5.1 per- been found that investments in IT management
cent increase from 2010. In 2010, worldwide IT are value relevant, that is they have an impact
spending totalled $3.4 trillion, up 5.4 percent from on the organisation’s market value (Dehning &
2009 levels” (Gartner Inc., 2011). IS productivity Richardson, 2002).
measurement studies show that IS management is In a paper discussing the importance of non-
a key factor that gives an organisation competi- financial measures, Ittner and Lacker (2003, p. 2)
tive advantage, for instance, Weill showed that report that “companies that adopted non-financial
the quality of the firm-wide management and measures with a causal link between those mea-
commitment to IT was found to be a significant sures and financial outcomes produced signifi-
moderator between strategic IT investment and cantly higher returns on assets and returns on equity
firm performance (1992). This is echoed by Son, over a five-year period than those that did not.”
104
An ITSM performance measurement framework ITSM frameworks and increasingly the ISO/
that includes quantitative and non-quantitative IEC 20000(ISO/IEC, 2005) standard for IT
measures linked with the organisation’s strategy Service Management have been readily adopted
may lead to potential gains for an organisation. by organisations in both the public and private
sectors worldwide and increasingly in Australia
as a means to provide effective management and
CURRENT PRACTICE control of IT service (Cater-Steel & Tan, 2005;
Cater-Steel, Tan, & Toleman, 2007, 2008, 2009).
A systematic literature review is undertaken to IT departments are under pressure to deliver IT
aggregate empirical evidence using a variety of service that facilitates achievement of organisa-
techniques in differing contexts (Kitchenham et tion’s strategic goals.
al., 2009). We chose to use a systematic literature The extensive adoption of ITSM frameworks
review to employ a transparent and repeatable by organisations may point to the acceptance by
method. This method has its origins in evidence- IT service managers that frameworks such as
based research conducted in medical science but ITIL and standards such as ISO/IEC 20000 can
has been recommended for management studies deliver real operational efficiencies, ultimately
(Tranfield et al., 2003), software engineering translating into revenue-increasing and cost-
and IS (Kitchenham et al., 2009). The literature reducing benefits. Organisations implementing
review progresses from general areas of organisa- ITSM initiatives report realisation of benefits
tion performance measurement and ITSM to the in cost savings and standardisations in delivery
specific area of ITSM performance measurement. of IT service and support. However, despite the
As more organisations embrace ITSM as an appeal and the tremendous potential to realise
approach to manage the operations of their IT benefits, the implementation of ITSM initiatives
organisations a number of ITSM frameworks, is slowed not only by the size of the investment
standards and models have been developed. Some required but also by the difficulty in quantifying
of the common frameworks and models include benefits and linking operational and financial
IT Infrastructure Library (ITIL®), IBM® Service benefits together. The study of six German firms
Management Reference Model, Microsoft® Op- by Hochstein et al. (2005) revealed that none of
erations Framework and HP® IT Service manage- these large organisations produced a business case
ment Reference Model. ITIL versions 2 and 3 are for ITIL investment, due to the difficulty in de-
the ITSM frameworks adopted or implemented termining benefits. A similar situation is reported
the most. According to Cater-Steel (2009) “Many in relation to Australian (Cater-Steel, Toleman,
IT service departments are adopting IT service & Tan, 2006) and US organisations (Pollard &
management best practice frameworks such as the Cater-Steel, 2009).
IT Infrastructure Library (ITIL) to improve the There are a number of studies on adoption or
quality of service to customers” (p.1). Championed implementation of ITSM but little research has
by the internationally active IT Service Manage- been undertaken. Few empirical studies to date
ment Forum (itSMF), ITIL is widely adopted and show the benefits of implementation underpinned
recognised for providing effective management by the performance measurement of ITSM. In
and control of IT service delivery and support addition to this, in a paper reviewing literature
(Barafort, Di Renzo, & Merlan, 2002; Hochstein, on modern performance measurement systems
Tamm, & Brenner, 2005; Hochstein, Zarnekow, Tangen (2004, p. 736) conclude that “the modern
& Brenner, 2005b). frameworks show what to measure, but give little
105
guidance when it comes to the question of how identified by De Toni and Tonchia (2001) to
to measure it.” compare performance measurement systems.
Table 1 compares commonly used performance
Performance Measurement measurement frameworks along the dimensions
Frameworks and ITSM proposed by De Toni and Tonchia. This compari-
son is done to identify frameworks suitable for
At the organisational level performance measure- an ITSM performance measurement framework
ment frameworks have been designed and used that would facilitate communication between the
to monitor productivity, profitability, quality, business and IT.
speed, delivery and flexibility. The performance The six columns in Table 1 represent some of
measurement frameworks used in IS are adopted the performance measurement architecture dimen-
from organisational level performance measure- sions identified by De Toni and Tonchia (2001).
ment frameworks. The current context of ITSM We have added the second column which identi-
performance measurement is that there is no stan- fies the purpose of the performance measurement
dardised ITSM performance measurement frame- framework. The third column is the balanced
work. Organisations measure the performance of scorecards dimension where several separate
their ITSM using a wide variety of means ranging performances are considered independently cor-
from not using any measure to using a variety of responding to diverse perspectives. The fourth
metrics, to applying well organised performance column relates to a synthesis of low-level measures
frameworks. into more aggregated indicators. The fifth column
A structured and systematic basis for design- checks frameworks which distinguish between
ing a performance measurement framework for internal and external performances. The sixth
ITSM is attempted by performing the three levels dimension considers frameworks that distinguish
of analysis: the internal relationship of customer/supplier (De
Toni & Tonchia, 2001).
1. Performance Measurement Framework Some of the frameworks shown in Table 1 have
Architecture (Table 1), been used in the performance measurement of IS
2. Performance Measurement Framework as well as the performance measurement of ITSM.
Perspectives (Table 2) and The Sink and Tuttle model (Rolstadås, 1998)
3. Performance Metrics (Table 3). provides seven performance criteria and includes
most of those found in the BSC, performance
According to Tangen (2005, pp. 39-40) “Per- pyramid and performance prism. This chapter
formance measurement is a complex issue that compares a number of performance measure-
normally incorporates at least three different dis- ment frameworks, some of which have already
ciplines: economics, management and accounting. been applied to ITSM, to identify appropriate
It includes almost any objective of competition performance measurement dimensions that can be
and manufacturing excellence whether it is related used to structure the proposed ITSM performance
to cost, flexibility, speed, dependability or qual- measurement framework.
ity.” Tangen describes a number of performance Performance measurement frameworks will be
measurement frameworks which are considered classified along their perspectives to identify their
in this chapter in developing the dimensions of completeness, eliminate unnecessary perspectives
ITSM performance metrics. We use the concep- and identify the natural dimensions for ITSM.
tual dimensions and the constructive variables This is shown in Table 2 and the consideration
of modern performance measurement systems of perspectives handled by the performance mea-
106
Table 1. Comparison of performance measurement frameworks
Performance Measurement Purpose of Balanced Frustum Internal and Value Chain

Framework Framework. scorecards (Architecture External Based
Measures the (Architecture Vertical and (Architecture (Architecture
performance Balanced) Balanced) Balanced and Horizontal)
of? Horizontal by
process)
Balanced Scorecard Organisation ✓ ✓
(Kaplan & Norton, 1992)
Tableau de Bord Organisation ✓
(Chiapello & Lebas, 1996)
IS Balanced Scorecard IS Function ✓
(Martinsons, Davison, & Tse, 1999)
IS Functional Scorecard (ISFS) IS Function ✓ ✓
(Chang & King, 2005)
Performance Prism Organisation ✓
(Neely, Adams, & Crowe, 2001)
Performance Pyramid Organisation ✓
(Lynch & Cross, 1993)
Sink and Tuttle model Organisation ✓
(Sink & Tuttle, 1989)
Theory of Constraints Organisation ✓
(Goldratt, 1990)
Intangible Asset Monitor Organisation ✓
(Sveiby, 1997)
Skandia Navigator Intellectual ✓
(Edvinsson, 1997) Capital of the
Organisation
Goal Question Metrics (GQM) Software ✓
(Basili, Caldiera, Rombach, & Mar-
ciniak, 1994)
SERVQUAL Service Organi- ✓
(Parasuraman, 2004) sation
Results and Determinants frame- Organisation ✓
work.
(Fitzgerald, et al., 1994)
Medori and Steeple Organisation ✓
(Medori & Steeple, 2000)
Note: The shaded performance measurement frameworks are closely related.
surement framework is similar to the analysis by The sample catalogue presented in Table 4 clas-
Abran and Buglione (2003). Similar comparisons sifies commonly used ITSM performance mea-
of performance measurement frameworks have surement metrics along four dimensions. The
been done by other researchers (e.g. Anderson ITSM performance metrics used are derived from
& McAdam, 2004). the literature review and the findings from a sur-
In classifying ITSM performance metrics four vey of ITSM benefits conducted in 2009 (Ga-
dimensions of quality, time, cost and flexibility cenga, Cater-Steel, & Toleman, 2010). The
proposed by Neely, Gregory and Platts (2005) as measurement perspectives in Table 2 are derived
well as De Toni and Tonchia (2001) will be used. from the performance measurement frameworks
107
Table 2. Comparison of performance measurement frameworks along their perspectives
Performance measurement Financial Non-financial Results and Determinants;

framework / Dimensions Inputs, Process, Output, Outcomes
and Goals
Performance measurement Cost external Non-cost external;
matrix Non-cost internal
(Keegan, Eiler, & Jones, 1989)
Balanced scorecard Financial perspec- Customer perspective;
(Kaplan & Norton, 1992) tive Innovation and Learning perspec-
tive;
Internal business Perspective
Results and determinants Financial perfor- Quality of service; Results: Financial performance, Com-
model mance Flexibility; petitiveness
(Fitzgerald, et al., 1994) Resource utilisation; Determinants: Quality of service, Flex-
Innovation ibility, Resource utilisation, Innovation
Measures for time based Internal configura-
Competition tion
(Azzone, Masella, & Bertelè,
1991)
Performance Pyramid Objectives and mea- External Effectiveness;
(SMART) sures, Financial Internal Efficiency;
(Lynch & Cross, 1993) Objectives and Measures: Market
Skandia navigator Skandia Financial focus Customer focus;
AFS Human focus;
(Edvinsson, 1997) Process focus;
Renewal and development focus
Performance measurement Core elements, Core elements,
Framework Non-core elements, Non-core elements
(Brignall & Ballantine, 1996)
Brown’s inputs, processes, Outputs, Financial Inputs: Employees, Customer needs,
outputs, outcomes framework results raw materials, capital
(Brown, 1996) Process: Processing systems
Output: Products, services
Outcome: Customer response
Goals: Repeat business
Performance prism Stakeholder satis- Stakeholder satisfaction;
(Neely, et al., 2001) faction Strategies;
Processes;
Capabilities;
Stakeholder contribution
Sink and Tuttle model Budget ability/ Quality;
(Sink & Tuttle, 1989) profitability Quality of Work Life;
Innovation;
Productivity
IS Functional Scorecard Inputs: Resources
(ISFS) (Chang & King, 2005) Process: IS function performance
Output: IS Function Outputs
Outcome: Business process effective-
ness
Goals: Organisational performance
Theory of Constraints Net Profit, ROI and
(Goldratt, 1990) Cash flow
Continued on following page
108
Table 2. Continued
Performance measurement Financial Non-financial Results and Determinants;
framework / Dimensions Inputs, Process, Output, Outcomes
and Goals
Intangible Asset Monitor Peoples competence; Results: Risk reduction/ stability
(Sveiby, 1997) Internal Structure;
External Structure;
Value Creation Modes: growth
renewal;
Utilisation efficiency
SERVQUAL Tangibles Reliability; Responsiveness; As-
(Parasuraman, 2004) surance; Empathy
Medori and Steeple Competitive Priority Competitive priorities e.g Quality,
(Medori & Steeple, 2000) e.g. Cost Flexibility, Time, Delivery;
Future Growth
Table 3. Evaluation of ITSM performance metrics for sample
Performance Dimensions
ITSM Process
Quality Productivity Profitability
Metrics
Efficiency Effectiveness Efficiency Effectiveness Efficiency Effectiveness
Time to resolve ✓ ✓
incidents
Increased cus- ✓
tomer service
Resolution per ✓ ✓
priority
Reduction in ✓ ✓
number of recur-
ring incidents
Number of ✓
permanent fixes
implemented
Percentage of ✓
calls completed
within service
levels
Number of inci- ✓
dents caused by
changes
Assets out of ✓
service
Percentage of ✓ ✓
satisfied custom-
ers as per quick
survey responses
109
examined. The four dimensions used in Table 3 sions that can be used to structure the proposed
are extracted from a widely cited extensive review ITSM performance measurement framework.
of performance measurement literature performed A number of studies report the significance of
by Neely et al. (2005). Sub-categorisations of the performance measurement at the organisational
four dimensions are adopted from De Toni and level. Underlining this significance “organisa-
Tonchia (2001). tional effectiveness is the ultimate question in
Table 3 provides sample metrics related to the any form of organisational analysis” (Hall, 1980).
incident, problem and service asset and configu- Henri (2004, p. 94) introduces an enhanced
ration management processes. Each metric is perspective on performance measurement: “In
evaluated along the performance dimensions of essence, organisational effectiveness represents
quality, productivity and profitability, specifi- the outcome of organisational activities while
cally in terms of its contribution to efficiency or performance measurement consists of an assess-
effectiveness. Some of the ITSM process metrics ment tool to measure effectiveness.” This refine-
currently in use cover multiple areas of quality ment is important as performance measurement
and productivity as well as measuring efficiency is the perspective given emphasis in this chapter.
and effectiveness, however most metrics only Both Henri and Hall recognise the significance
address a single aspect as shown by the entries in and challenge of performance measurement to
Table 3. researchers and practitioners.
A classification of approaches to organisational
Proposed Model: Solutions effectiveness shows:
and Recommendations
1. a goal model which defines effectiveness as
The performance measurement of ITSM is gaining the degree to which an organisation realises
interest, with a number of studies and publications its goals and
proposing ITIL performance metrics (Barafort, Di 2. a resource acquisition model which defines
Renzo, Lejeune, Prime, & Simon, 2005; Brooks, effectiveness of an organisation as the ability
2006; Steinberg, 2006; van Grembergen, De to exploit its environment in the acquisition
Haes, & Amelinckx, 2003), IT service perfor- of scarce and valued resources to sustain its
mance and quality measures (Hochstein, 2004; functioning (Hall, 1980).
Praeg & Schnabel, 2006), business value of ITIL
(itSMF Germany, 2008; Moura, Sauve, Jornada, An extended classification offers five organi-
& Radziuk, 2006; Šimková & Basl, 2006; Yixin sational effectiveness categories (Henri, 2004):
& Bhattacharya, 2008), ITIL process capability
and maturity assessment (itSMF International, 1. Goal models – focused on the achievement
2008; Valdés et al., 2009), software for measur- of the organisations goals;
ing ITIL process performance (Lahtela, Jäntti, & 2. System models – consider the resources and
Kaukola, 2010) and evaluation frameworks for processes necessary to attain goals;
ITIL (Hochstein, Zarnekow, & Brenner, 2005a; 3. Strategic constituencies model – the pow-
McNaughton, Ray, & Lewis, 2010). erful constituencies gravitating around the
The proposed model is based on a comparison organisation;
of existing performance measurement frameworks 4. Competing values model – the values on
some of which have already been applied to ITSM. which the evaluation of effectiveness are
Based on the literature reviewed we have identified grounded;
appropriate performance measurement dimen-
110
5. Ineffectiveness model – the absence of to both IT applications and the IS department or

ineffectiveness factors as a source of function as a whole.” The study concluded that the
effectiveness. IS/IT BSC developed evaluates the overall IT/IS
function in terms of efficiency and effectiveness.
In a associated classification, performance This chapter extends the findings of the study by
measurement models are grouped into two views: including aspects of the study in service oriented
cybernetic views – whereby performance mea- approach to the management of the IT/IS function.
surement is based mainly on financial measures The model developed in this chapter recognises
and considered a component of the planning and the distinctions made between the short-term and
control cycle; and holistic views – based on mul- long-term performance evaluation and proposes
tiple non-financial measures where performance using ITSM performance metrics as a foundation
measurement acts as an independent process inte- for benefits with a short term as well as long term
grated in a broader set of activities (Henri, 2004). perspective. The BSC ensures the ITSM metrics
In this view performance measurement plays a used are easily linked to higher level organisation
key role in the development of strategic plans objectives by applying common terminology for
and evaluation of the achievement of organisa- metrics used by IT and business (Huchendorf,
tional objectives. The organisational effectiveness 2005; van der Zee & de Jong, 1999). Use of the
models and performance measurement models are BSC approach has the added benefit for IT man-
useful in classifying the performance measures agement of integrating planning and evaluation
and frameworks used in ITSM. cycles (van der Zee & de Jong, 1999, p. 144).
In this chapter we focus more on a holistic view The BSC has been applied in previous theoretical
of performance measurement models. ITSM lays studies by other ITSM researchers such as Donko
emphasis on customer orientation and the delivery and Traljic (2009), Huchendorf (2005), Moura et
of value in process oriented systems (van Bon al. (2006), Praeg et al. (2006), van der Zee and de
& van Selm, 2008). The effectiveness of ITSM Jong (1999). and Grembergen et al. (2003, p. 2).
closely aligns with the goal and system models The model shown in Figure 1 can be used
of organisational effectiveness. This is the case by organisations to measure the performance of
because the ITSM approach is customer focused their ITSM as well as evaluate the metrics they
with IT services ultimately aimed at achieving or- are currently using for completeness in service
ganisational goals. ITSM frameworks such as ITIL orientation, financial and non-financial perspec-
are process oriented and consider the resources tives and broad economic terms. The model can
and activities required to attain the process goals. also be used to organise the existing metrics and
The BSC which is extensively applied in ITSM develop exhaustive performance metric registers.
performance measurement falls in the domain of The performance measurement model can be used
holistic performance measurement models. as a basis for developing meaningful reports for
A frequently cited study developed a BSC for the business that can be used for reporting input
information systems that measures and evaluates into planning, monitoring and evaluation. The
IS activities from four perspectives: business performance model would be useful in laying
value, user orientation, internal process, and fu- the foundation for ITSM business cases. At the
ture readiness (Martinsons, et al., 1999, p. 6). It industry level, wide adoption of the performance
distinguishes between two categories of IT/IS per- measurement model would lead to standardisation
formance evaluation: “the short-term cost-benefit of ITSM performance measurement and enhance
evaluation that is commonly applied to individual the ability of organisations implementing ITSM to
projects, and the longer-term perspective relevant benchmark on other organisations using like terms.
111
Figure 1. Model to measure the performance of ITSM
FUTURE RESEARCH DIRECTIONS CONCLUSION
The model proposed in this chapter will be tested The chapter undertook a multi-discipline review
by ITSM practitioners to evaluate its relevance of the literature on performance measurement.
and usefulness. Further research on the model IT/IS is now seen as a service, with a customer
will involve creating a catalogue of ITSM met- focus and process orientation, and measuring that
rics by populating the model with performance service to show its productivity, profitability and
metrics for ITSM services, functions, processes quality to those using the service is critical. The
and technology. The catalogue developed will investment in ITSM is significant, so showing the
be evaluated for completeness and where a lack benefit of that investment is important to corporate
of metrics is identified new measures will be and IT management in organisations.
proposed. The catalogue will be evaluated by a The main contribution of this chapter is the
panel of academic and industry experts before it development of a model that can be used to
is made widely available to the ITSM practitioner measure the performance of ITSM. The model is
community for application and improvement. distilled from the literature available and widely
Future research will entail developing guidance used performance metrics. We have taken a holistic
on how to measure the performance of ITSM by view in development of the model, in integrating
providing the factors influencing the organisations approaches such as the BSC, broad economic
selection of ITSM performance metrics. This perspectives and service oriented ITSM. The
guidance will assist organisation contextualise integrated approach caters for the wide range
the ITSM performance measurement model to of variables of interest and the large number of
address their specific needs. stakeholders involved in the performance of ITSM.
112
The performance measurement model devel- Barua, A., & Lee, B. (1997). The Information
oped addresses an area that has been identified Technology productivity paradox revisited: A
as important and lacking in research. The ITSM theoretical and empirical investigation in the
performance measurement framework provides manufacturing sector. International Journal of
a basis of standardisation and performance com- Flexible Manufacturing Systems, 9, 145–166.
parison for organisations implementing ITSM. It doi:10.1023/A:1007967718214
addresses an area that has been identified as chal-
Basili, V., Caldiera, G., Rombach, D., & Marcin-
lenging to organisations implementing ITSM by
iak, J. (1994). The goal question metric approach.
providing a means to show a relationship between
In Marciniak, J. J. (Ed.), Encyclopedia of software
ITSM investment and benefit from this investment.
engineering. Wiley.
Black, J., Draper, C., Lococo, T., Matar, F., & Ward,
REFERENCES C. (2007). An integration model for organizing
IT service management. IBM Systems Journal,
Abran, A., & Buglione, L. (2003). A multidi- 46(3), 405–422. doi:10.1147/sj.463.0405
mensional performance model for consolidating
balanced scorecards. Advances in Engineering Brignall, S., & Ballantine, J. (1996). Performance
Software, 34(6), 339–349. doi:10.1016/S0965- measurement in service businesses revisited. Inter-
9978(03)00033-4 national Journal of Service Industry Management,
7(1), 6–31. doi:10.1108/09564239610109393
Anderson, K., & McAdam, R. (2004). A critique
of benchmarking and performance: Lead or Lag. Brooks, P. (2006). Metrics for IT Service Man-
Benchmarking: An International Journal, 11(5), agement (1st ed.). Zaltbommel, The Netherlands:
465–483. doi:10.1108/14635770410557708 Van Haren Pub.
Azzone, G., Masella, C., & Bertelè, U. (1991). Brown, M. G. (1996). Keeping score using the right
Design of performance measures for time-based metrics to drive world-class performance (p. ix).
companies. International Journal of Opera- Brynjolfsson, E., & Hitt, L. M. (1998). Beyond
tions & Production Management, 11(3), 77–85. the productivity paradox. Communications of the
doi:10.1108/01443579110143412 ACM, 41(8), 49–55. doi:10.1145/280324.280332
Barafort, B., Di Renzo, B., Lejeune, V., Prime, Cater-Steel, A. (2009). IT service departments
S., & Simon, J.-M. (2005). ITIL based service struggle to adopt a service-oriented philosophy.
management measurement and ISO/IEC 15504 International Journal of Information Systems
process assessment: A win-win opportunity. in the Service Sector, 1(2), 69–77. doi:10.4018/
SPICE 2005 Conference. Austria. jisss.2009040105
Barafort, B., Di Renzo, B., & Merlan, O. (2002). Cater-Steel, A., & Tan, W.-G. (2005). Imple-
Benefits resulting from the combined use of ISO/ mentation of IT infrastructure library (ITIL) in
IEC 15504 with the Information Technology Australia: Progress and success factors. 2005 IT
infrastructure library (ITIL). 4th International Governance International Conference, Auckland,
Conference on Product Focussed Software De- New Zealand.
velopment and Process Improvement (PROFES).
113
Cater-Steel, A., Tan, W.-G., & Toleman, M. (2007). Dedrick, J., Gurbaxani, V., & Kraemer, K. L.
itSMF Australia 2006 Conference: Summary of (2003). Information Technology and economic
ITIL adoption survey responses. Toowoomba, performance: A critical review of the empirical
Australia: University of Southern Queensland. evidence. ACM Computing Surveys, 35(1), 1–28.
doi:10.1145/641865.641866
Cater-Steel, A., Tan, W.-G., & Toleman, M.
(2008). itSMF Australia 2007 Conference: Sum- Dehning, B., & Richardson, V. J. (2002). Returns
mary of ITSM standards and frameworks survey on investments in Information Technology: A re-
responses. Toowoomba, Australia: University of search synthesis. Journal of Information Systems,
Southern Queensland. 16(1), 7–30. doi:10.2308/jis.2002.16.1.7
Cater-Steel, A., Tan, W.-G., & Toleman, M. DeLone, W. H., & McLean, E. R. (1992). Informa-
(2009). itSMF Australia 2008 Conference: Sum- tion Systems success: The quest for the dependent
mary of ITSM standards and frameworks survey variable. Information Systems Research, 3(1),
responses. Toowoomba, Australia: University of 60–95. doi:10.1287/isre.3.1.60
Southern Queensland.
DeLone, W. H., & McLean, E. R. (2003). The
Cater-Steel, A., Toleman, M., & Tan, W.-G. (2006). DeLone and McLean Model of Information
Transforming IT service management the ITIL Systems success: A ten-year update. Journal of
impact. ACIS 2006 17th Australasian Conference Management Information Systems, 19(4), 9–30.
on Information Systems. Adelaide, Australia:
Donko, D., & Traljic, I. (2009). Measurement
Australasian Association for Information Systems.
of service effectiveness and establishment of
Chang, J., & King, W. (2005). Measuring the baselines. WSEAS Transactions on Information
performance of Information Systems: A functional Science and Applications, 6(8), 1310–1319.
scorecard. Journal of Management Information
Edvinsson, L. (1997). Developing intellectual
Systems, 22(1), 85–115.
capital at Skandia. Long Range Planning, 30(3),
Chiapello, E., & Lebas, M. (1996). The Tableau 366–373. doi:10.1016/S0024-6301(97)90248-X
de Bord, A French approach to management
Fitzgerald, L., Johnston, R., Brignall, S., Silvestro,
information. Paper presented at the 19th Annual
R., & Voss, C. (1994). Performance measurement
Meeting of the European Accounting Association.
in service businesses. Cambridge, UK: CIMA.
Conger, S., Winniford, M., & Erickson-Harris,
Gacenga, F., Cater-Steel, A., & Toleman, M.
L. (2008, 14-17 August). Service management
(2010). An international analysis of IT service
in operations. Paper presented at the Fourteenth
management benefits and performance measure-
Americas Conference on Information Systems,
ment. Journal of Global Information Technology
Toronto.
Management, 13(4), 28–63.
De Toni, A., & Tonchia, S. (2001). Performance
Galup, S., Dattero, R., Quan, J., & Conger, S.
measurement systems - Models, characteristics
(2009). An overview of IT service management.
and measures. International Journal of Opera-
Communications of the ACM, 52(5), 124–127.
tions & Production Management, 21(1/2), 46–71.
doi:10.1145/1506409.1506439
doi:10.1108/01443570110358459
114
Gartner Inc. (2011). Quarterly IT spending fore- Hochstein, A., Zarnekow, R., & Brenner, W.
cast. Retrieved 31 January, 2011, from http:// (2005b). ITIL as common practice reference
www.gartner.com/technology/research/quarterly- model for IT service management: formal assess-
it-forecast/ ment and implications for practice. Proceedings
of the 2005 IEEE International Conference on
Goldratt, E. M. (1990). What is this thing
e-Technology, e-Commerce and e-Service (pp.
called theory of constraints and how should it
704-710). Hong Kong.
be implemented?Great Barrington, MA: North
River Press. Huchendorf, S. C. (2005). Information Technology
infrastructure library (ITIL) performance metrics.
Gronroos, C. (1994). From scientific management
University of Minnesota, College of Continuing
to service management. International Journal
Education 2005.
of Service Industry Management, 5(1), 5–20.
doi:10.1108/09564239410051885 ISO/IEC. (2005). ISO/IEC 20000-1:2005 Infor-
mation technology service management, part 1:
Hall, R. H. (1980). Effectiveness theory and
Specification (pp. 16). Sydney, Australia: Stan-
organisational effectiveness. The Journal of
dards Australia. itSMF Germany. (2008). Total
Applied Behavioral Science, 16(4), 536–545.
value of IT (TVITEM). Retrieved 07 February,
doi:10.1177/002188638001600408
2011, from http://www.itsmf.de/ak_tv_it.html
Henri, J.-F. (2004). Performance measurement
itSMF International. (2008). Self assessment (ITIL
and organisational effectiveness: Bridging
V2). Retrieved 07 February, 2011, from http://
the gap. Managerial Finance, 30(6), 93–123.
www.itsmfi.org/content/self-assessment-itil-v2
doi:10.1108/03074350410769137
Ittner, C. D., & Lacker, D. F. (2003). Coming up
Hochstein, A. (2004). Managing IT service qual-
short on nonfinancial performance measurement.
ity as perceived by the customer: The service
Harvard Business Review, (November): 1–9.
oriented IT SERVQUAL. Paper presented at the
International Telecommunications Society (ITS) Kaplan, R., & Norton, D. (1992). The balanced
15th Biennial Conference. scorecard: Measures that drive performance.
Harvard Business Review, 70(1), 71–79.
Hochstein, A., Tamm, G., & Brenner, W. (2005,
May 26 - 28). Service-oriented IT management: Katzan, H. (2008). Service science concepts,
Benefit, cost and success factors. Paper presented technology, management. Bloomington, IN:
at the Thirteenth European Conference on Infor- iUniverse, Inc.
mation Systems, Regensburg, Germany.
Keegan, D., Eiler, R. G., & Jones, C. R. (1989).
Hochstein, A., Zarnekow, R., & Brenner, W. Are your performance measures obsolete? Man-
(2005a). Evaluation of service-oriented IT management Accounting, 70(12), 45–60.
agement in practice. Proceedings of the Interna-
Keel, A. J., Orr, M. A., Hernandez, R. R., Pa-
tional Conference on Services Systems and Ser-
trocinio, E. A., & Bouchard, J. (2007). From a
vices Management (ICSSSM) (Vol. 1, pp. 80-84).
technology-oriented to a service-oriented ap-
proach to IT management. IBM Systems Journal,
46(3), 549–564. doi:10.1147/sj.463.0549
115
Lahtela, A., Jäntti, M., & Kaukola, J. (2010). Neely, A., Adams, C., & Crowe, P. (2001). The
Implementing an ITIL-based IT service manage- performance prism in practice. Measuring Busi-
ment measurement system. Paper presented at ness Excellence, 5(2), 6-13.
the Fourth International Conference on Digital
Neely, A., Adams, C., & Kennerley, M. (2002).
Society, St. Maarten, Netherlands Antilles.
The performance prism: The scorecard for mea-
Luftman, J., & Ben-Zvi, T. (2010). Key issues for suring and managing business success. London,
IT executives 2009: Difficult economy’s impact UK: Prentice Hall Financial Times.
on IT. MIS Quarterly Executive, 9(1), 49–59.
Neely, A., Gregory, M., & Platts, K. (2005). Per-
Lynch, R. L., & Cross, K. F. (1993). Measure up! formance measurement system design: A literature
Yardsticks for continuous improvement. Cam- review and research agenda. International Journal
bridge, Mass: Brackwell Business. of Operations & Production Management, 25(12),
1228–1263. doi:10.1108/01443570510633639
Marchand, M., & Raymond, L. (2008). Research-
ing performance measurement systems: An Infor- OGC. (2007). Service strategy. London, UK: TSO
mation Systems perspective. International Journal for the Office of Government Commerce.
of Operations & Production Management, 28(7),
OUP. (2000). The Oxford English dictionary
663–686. doi:10.1108/01443570810881802
online. Retrieved from http://gateway.library.
Martinsons, M., Davison, R., & Tse, D. (1999). The qut.edu.au/login?url=http://dictionary.oed.com/
balanced scorecard: A foundation for the strategic entrance.dtl
management of Information Systems. Decision
Parasuraman, A. (2004). Assessing and improv-
Support Systems, 25(1), 71–88. doi:10.1016/
ing service performance for maximum impact:
S0167-9236(98)00086-4
Insights from a two-decade-long research journey.
McNaughton, B., Ray, P., & Lewis, L. (2010). Performance Measurement and Metrics, 5(2),
Designing an evaluation framework for IT service 45–52. doi:10.1108/14678040410546064
management. Information & Management, 47(4),
Parasuraman, A., Zeithaml, V. A., & Berry, L. L.
219–225. doi:10.1016/j.im.2010.02.003
(1985). A conceptual model of service quality
Medori, D., & Steeple, D. (2000). A framework and its implications for future research. Journal
for auditing and enhancing performance measure- of Marketing, 49(4), 41–50. doi:10.2307/1251430
ment systems. International Journal of Operations
Pitt, L., Watson, R., & Kavan, C. (1995). Service
& Production Management, 20(5), 520–533.
quality: A measure of Information Systems ef-
doi:10.1108/01443570010318896
fectiveness. Management Information Systems
Mell, P., & Grance, T. (2009). Effectively and Quarterly, 19(2), 173–187. doi:10.2307/249687
securely using the cloud computing paradigm.
Pollard, C., & Cater-Steel, A. (2009). Justifica-
National Institute of Standards and Technology.
tions, strategies, and critical success factors in
Moura, A., Sauve, J., Jornada, J., & Radziuk, E. successful ITIL implementations in U.S. and
(2006). A quantitative approach to IT investment Australian companies: An exploratory study.
allocation to improve business results. Proceed- Information Systems Management, 26, 164–175.
ings of the Seventh IEEE International Workshop doi:10.1080/10580530902797540
on Policies for Distributed Systems and Networks
(pp. 87-95).
116
Praeg, C., & Schnabel, U. (2006). IT-service cachet Son, S., Weitzel, T., & Laurent, F. (2005). De-
- Managing IT-service performance and IT-service signing a process-oriented framework for IT
quality. Proceedings of the 39th Annual Hawaii performance management systems. The Electronic
International Conference on System Sciences, Journal of Information Systems Evaluation, 8(3),
Vol. 2. IEEE Computer Society. 219–228.
Richard, P. J., Devinney, T. M., Yip, G. S., & Steinberg, R. A. (2006). Measuring ITIL: Mea-
Johnson, G. (2009). Measuring organizational suring, reporting and modeling the IT service
performance: Towards methodological best prac- management metrics that matter most to IT senior
tice. Journal of Management, 35(3), 718–804. executives. Victoria, Canada: Trafford.
doi:10.1177/0149206308330560
Sveiby, K. E. (1997). The new organizational
Rolstadås, A. (1998). Enterprise performance wealth: Managing & measuring knowledge-based
measurement. International Journal of Operations assets (1st ed.). San Francisco, CA: Berrett-
& Production Management, 18(9/10), 989–999. Koehler Publishers.
doi:10.1108/01443579810225577
Tangen, S. (2004). Performance measurement
Salle, M. (2004). IT service management and IT from philosophy to practice. International Journal
governance: Review, comparative analysis and of Productivity and Performance Management,
their impact on utility computing. HP Research 53(8), 726–737. doi:10.1108/17410400410569134
Labs.
Tangen, S. (2005). Demystifying productivity and
Saunders, C. S., & Jones, J. W. (1992). Measuring performance. International Journal of Productiv-
performance of the Information Systems function. ity and Performance Management, 54(1), 34–46.
Journal of Management Information Systems, 8, doi:10.1108/17410400510571437
63–82.
Tracy, L., Guevara, J., Harcourt, O., & Stegman,
Scott, J. E. (1995). The measurement of informa- E. (2006). IT key metrics data: Key industry mea-
tion systems effectiveness: evaluating a measur- sures: Sample measures. Gartner 2006.
ing instrument. SIGMIS Database, 26(1), 43–61.
Valdés, O., St-Jean, M., Renault, A., Picard, M.,
doi:10.1145/206476.206484
Cortina, S., & Betry, V. (2009). ITSM process
Seddon, P., Graeser, V., & Willcocks, L. (2002). assessment supporting ITIL. Amersfoot, The
Measuring organizational IS effectiveness: An Netherlands: Van Haren Publishing.
overview and update of senior management
van Bon, J., & van Selm, L. (2008). ISO/IEC
perspectives. SIGMIS Database, 33(2), 11–28.
20000: An introduction (1st ed.). Zaltbommel,
doi:10.1145/513264.513270
The Netherlands: Van Haren Publishing.
Šimková, E., & Basl, J. (2006). Business value
van der Zee, J. T. M., & de Jong, B. (1999).
of IT. Systems Integration Conference, Prague.
Alignment is not enough: Integrating business
Sink, D. S., & Tuttle, T. C. (1989). Planning and and Information Technology management with
measurement in your organization of the future. the balanced business scorecard. Journal of Man-
Norcross, GA: Industrial Engineering and Man- agement Information Systems, 16(2), 137–156.
agement Press.
117
van Grembergen, W., De Haes, W., & Amelinckx, the customer. It is superimposed on management
S. I. (2003). Using COBIT and the balanced score- domains such as systems management, network
card as instruments for service level management. management, systems development, and on many
Information Systems Control Journal, 4, 56–62. process domains like change management, asset
management and problem management.” A broad
Vargo, S., & Lusch, R. (2004). Evolving to a new
definition offering a focus on the organisation as
dominant logic for marketing. Journal of Market-
the customer of IT services is seen in Winniford
ing, 68(1), 1–17. doi:10.1509/jmkg.68.1.1.24036
et al.’s (2009, p. 153) definition: “Information
Weill, P. (1992). The relationship between in- Technology Service Management (ITSM) focuses
vestment in Information Technology and firm on defining, managing, and delivering IT services
performance: A study of the valve manufactur- to support business goals and customer needs”.
ing sector. Information Systems Research, 3(4), A definition emphasising process and customer
307–333. doi:10.1287/isre.3.4.307 orientation is offered by Bon and Selm (2008,
p. 20): “IT Service Management is the manage-
Winniford, M., Conger, S., & Erickson-Harris,
ment of all processes that co-operate to ensure
L. (2009). Confusion in the ranks: IT service
the quality of live IT services, according to the
management practice and terminology. Infor-
levels of service agreed by with the customer.” A
mation Systems Management, 26(2), 153–163.
broad definition of ITSM which gives a customer
doi:10.1080/10580530902797532
orientation without committing to the mechanics
Yixin, D., & Bhattacharya, K. (2008). Estimat- of the organisation is provided by Keel (2007,
ing business value of IT services through process p. 549): “Information technology service man-
complexity analysis. IEEE Network Operations agement (ITSM) is a discipline for managing
and Management Symposium (pp. 208-215). organisations providing information technology
(IT) services from a customer’s perspective.” In
an attempt to provide an overall integration of
ITSM models, standards and frameworks Black
KEY TERMS AND DEFINITIONS (2007, p. 408) defines ITSM as being “about the
definition and delivery of IT services and the
Effectiveness: Effectiveness, which involves
management of the organisation that provides the
doing the right things, at the right time, with the
service.” This is similar to “Information Technol-
right quality, can be defined as the ratio between
ogy Service Management (ITSM) focuses on
actual output and expected output” (Sink & Tuttle,
defining, managing, and delivering IT services
1989).
to support business goals and customer needs,
Efficiency: Efficiency simply means doing
usually in IT Operations” (Conger, Winniford,
things right, defined as the ratio between resources
& Erickson-Harris, 2008, p. 1). Based on these
expected to be consumed and actually consumed”
definitions this chapter defines IT service manage-
(Sink & Tuttle, 1989).
ment as an approach to managing IT operations
IT Service Management: The following are
organised around services by an organisation that
some of the definitions of IT service management
delivers IT services to customers. This definition
from the literature review. A process oriented
is similar to the broad definition offered by Keel
definition of ITSM is offered by Young cited in
et al. (2007) but describes ITSM as an approach
Salle (2004, p. 9): “a set of processes that coop-
rather than a discipline. This chapter identifies
erate to ensure the quality of live IT services,
Information Systems as the discipline.
according to the levels of service agreed to by
118
Performance Measurement: A performance is a provider/client interaction that creates and

measurement definition that is widely used in pro- captures value” (Katzan, 2008, p. 2). A more de-
duction and operations management, accounting tailed definition of service is “the application of
and information systems journal articles and used specialised competences (knowledge and skills)
in this chapter is: “the process of quantifying the through deeds, processes, and performances for
efficiency and effectiveness of action” (Neely, the benefit of another entity or the entity itself”
et al., 2005, p. 1229). Performance measurement (Vargo & Lusch, 2004, p. 2). This chapter adopts
should be understood as a broad term that “covers the definition by UKs’ Office of Government
both overall economic and operational aspects” Commerce (OGC) (2007) and Bon et al. (2008,
(Tangen, 2005, p. 40) including measures of p. 15) “a service is a means of delivering value
productivity, profitability and quality. to customers by facilitating the outcomes that
Performance Measurement Framework: customers want to achieve without the owner-
The definition is derived from the performance ship of specific costs and risks.” This definition
measurement system definition by Neely, Greg- of service is adopted as it is consistent with the
ory and Platts (2005, p. 1229), “a performance other literature reviewed and is widely adopted
measurement system can be defined as the set by the ITSM community.
of metrics used to quantify both the efficiency Service Management: Service Management
and effectiveness of actions.” A framework is is defined in different disciplines. We considered
defined as “a structure composed of parts framed a definition from IT service management and
together, especially one designed for enclosing or another from Service Marketing. According to
supporting anything; a frame or skeleton” (2000). OGC (2007): “Service Management is a set of spe-
A framework is desirable as it goes beyond ad- cialised organisational capabilities for providing
dressing underlying performance measurement value to customers in the form of services.” The
and addresses the practicalities of measurement chapter adopts the broader definition of service
making the performance measurement system management by used in Gronroos (1994, p. 6)
meaningful to practitioners (Tangen, 2004). “Service management is a total organisational ap-
Service: There is a variety of definitions of proach that makes quality of service, as perceived
service used in the literature reviewed and the by the customer, the number one driving force for
following are a few that were appraised for use the operations of the business.”
in the chapter. One definition states “a service
119
120
Chapter 7
Fuzzy Approach for
Monitoring Projects Success
in the IT/IS Industry
Jose L. Salmeron
University Pablo of Olavide, Spain
Cristina Lopez
University Pablo of Olavide, Spain
ABSTRACT
There are many uncertainties that can influence the success of Information Technology (IT) and Information
Systems (IS) projects. These are characterized to be highly complex and risky, among other issues. These
features explain the high rate of failures in this kind of projects. So, if practitioners want to prevent un-
desired outcomes in their IT/IS projects, they have to continuously manage the risks existing in them. In
this way, practitioners should monitor risks impacts on IT/IS projects success. However, current methods
used for it, have several limitations that can be overcome by employing artificial intelligence techniques.
Based on the fuzzy theory, this chapter proposes the use of fuzzy approaches to model risks effects on
IT/IS projects success measures. Its applicability is presented through an illustrative case. The findings
highlight that the method proposed give project managers insights into the causes of failure or delay of
their IT/IS projects, in order to develop effective strategies.
DOI: 10.4018/978-1-4666-0170-3.ch007
Fuzzy Approach for Monitoring Projects Success in the IT/IS Industry
INTRODUCTION uct innovation development are extremely

complex, risky and expensive endeavour
Worldwide companies develop information tech- (Cormican & O’Sullivan, 2004).
nology (IT) projects to maintain and incorporate
the most innovative technologies in their infor- The main characteristics are classified in seven
mation systems (IS). These projects have certain categories (Peffers & Gengler, 2003; Rodriguez-
characteristics that make them different from Repiso, et al., 2007; Salmeron & Herrero, 2005):
the rest of engineering projects and increase the abstract constraints, difficulty of visualization,
chances of their failure. These characteristics must excessive perception of flexibility, hidden com-
be considered when developing and managing plexity, uncertainty, tendency to software failure,
any IT/IS project. However, the management of and goal to change existing business processes.
complex IT/IS projects is challenging even when From the late 1970s to the late 1990s, research
measures of success are known and understood. and practice were dominated by three classical
The practical management of IT/IS projects methodologies for managing success in infor-
beyond the theories for success finds significant mation systems: critical success factors (CSF)
difficulties as follows (Rodriguez-Repiso et al., (Rockart, 1979), technology acceptance model
2007; Working Group from the Royal Academy (TAM) (Davis, 1989), and DeLone and McLean
of Engineering and the British Computer Society, success model (DeLone & McLean, 1992), among
2004): others. In the last few years, new methodologies
for identifying, classifying and evaluating success
• IT/IS projects are often poorly defined, factors in IT/IS projects emerged. These are critical
codes of practice are frequently ignored, success chains (CSC) (Peffers & Gengler, 2003),
and in some cases not many lessons are analytic hierarchy process (AHP) (Salmeron &
learned from past experience. Herrero, 2005) and delphi method (Nasir et al.,
• Market pressures demand delivery in the 2011), to name a few.
shortest time frame even if it may result in However, when analyzing failures of complex
a lower quality product. IT/IS and engineering projects, it seems clear
• The rapid pace of technological progress in that the widely accepted assessment criteria for
IT/IS hinders the expertise in a particular measuring projects success cannot guarantee suc-
technique and creates a culture where the cess when IT/IS projects are concerned. In fact,
use of tools not completely tested is ac- as indicated in Kwak and Stoddard (2004), the
ceptable and commonplace. nature of software projects creates many risks that
• The tendency to write new software code must be managed diligently. Practitioners should
to perform well-established functions de- handle risks factors effectively to avoid failures
creases reliability. and increase the likelihood of project success
• IT/IS projects involve numerous iterations (Aloini et al., 2007). A proper risks management
and continuous interaction between every- helps to practitioners to be aware of the real situ-
one involved in design and implementa- ation of the project, their problematic aspects and
tion. Their work is highly interdependent potential causes existing of IT/IS project failure
which necessitates efficient communica- (Iversen et al., 2004). They will thus resolve more
tion within the project team. efficiently the threats’ IT/IS projects. Otherwise,
• IT/IS projects contain a greater degree of improper risks management lead the carrying out
novelty than other engineering projects. In of evitable error and the appearance of potential
particular, IT/IS projects related to prod- problems, which make the achievement of IT/
121
Figure 1. Fuzzy cognitive map example

IS project success difficult (Bannerman, 2008;
Wallace et al., 2004). Therefore, active risks
management is necessary to reach IT/IS projects
success (Bakker et al., 2010).
In order to support practitioners in this task,
this chapter focuses on the development of a new
method based on using fuzzy dynamic systems.
This allows to model the risks factors influences
on measures of IT/IS projects success. In reality,
success is a complex concept, and its perception
is complicated, unstructured and not readily
quantifiable. This method seems to be the most
adequate to deal with this ambiguity. Moreover,
fuzzy dynamic systems are flexible and can be
customized in order to consider the characteristics
of different IT/IS projects.
The next section of the chapter introduces
fuzzy cognitive maps (FCM). Specifically, their
fundamentals, as well as the development and
inference process are exposed. Then, we present
an illustrative case to show the applicability of
the fuzzy dynamic systems for managing IT/IS variables X = (x 1, x 2 , , x n ) , n being the num-
project success and risks. Finally, we outline the ber of FCM nodes. For any context, the nodes are
conclusions and indicate some future research a set of entities that are relevant to the domain of
directions. study. Because FCMs are hybrid methods (Xiro-
giannis & Glykas, 2007) that mix fuzzy logic
(Bellman & Zadeh, 1970, Zadeh, 1965) and neu-
FUZZY COGNITIVE MAPS ral networks (Kosko, 1996), each causal connec-
tion is assessed by its intensity.
An FCM is a technique which represents systems FCM edges indicate the intensity of causal
using a joint of finite variables and signed fuzzy connections between nodes. These are repre-
cause-effect connections between them. In other sented by directed edges (ei → j ). Each ei → j indi-
words, this is a fuzzy signed directed graph with
cates the influence of the causal variable ( x i ) on
feedback. Figure 1 shows an example of an FCM
model. These are composed of dynamic nodes and the effect variable ( x j ).The ei → j take values in
edges between them. FCM nodes represent the the fuzzy causal interval [-1, 1]. There are three
characteristics of the phenomenon studied. These possible types of causal connections between
are the most relevant dynamic variables or factors nodes:
describing a target real-world dynamic system.
FCM nodes may represent great variety of • ei → j > 0 . This indicates positive causality
variables, such as failures, goals, controls, poli- between nodes x i and x j . In other words,
cies, diseases, or events, among others. These are
an increase (decrease) in the value of x i
labelled by such variables forming the set of
122
leads to an increase (decrease) in the value of specific domains and people. The CM study
of x j . perceptions about the world and the way they act
• ei → j < 0 . This indicates negative causality to reach human desires within their world. The
foundation for this theory was given by Kelly
between nodes x i and x j . In other words, (1970), based on a particular cognitive psychologi-
an increase (decrease) in the value of x i cal body of knowledge called personal construct
leads to an decrease (increase) in the value theory. Specifically, a CM shows a representation
of x j . of how humans think about a particular issue, by
• ei → j = 0 . This indicates no causal rela- analyzing, arranging the problems and graphically
mapping concepts that are interconnected (Eden &
tionship exist between x i and x j . In other
Ackermann, 2004). In addition, it identifies causes
words, an increase (decrease) in the value and effects and explains causal links.
of x i does not alter the value of x j . CMs were used for the first time by Axelrod
(1976) to represent the causal relationships among
These values are represented in the n × n factors in order to outline a decision-making pro-
matrix called adjacency matrix ( E ) (1). cess. These have been considered an especially
useful technique in problem solving (Axelrod,
    e1→n  1976; Eden & Ackermann, 2004) where many
 
       decisional variables are causally interrelated (Kim
 
E =    ei → j    (1) & Lee, 1998). In the same manner, CM can also
  help decision-makers in analyzing the hidden
      
  causal relationships that can contribute to reaching
en →1      more relevant and significant solutions.
A CM is a formal model with construction
The adjacency matrix corresponding to the rules, which is characterized by defining a hier-
FCM example (Figure 1) is the following: archical structure for a decisional process. This
is composed of nodes which represent the most
 e e1→3  e1→5    relevant factors of a decisional environment
 
 
1→2
 (Axelrod, 1976). Furthermore, a CM allows the
     e 2→ 7  identifying of the type of relationship by means
    e 3→ 5    of incorporating plus (+) and minus (-) signs
 
E =       e4→7  (Dikerson & Kosko, 1994). A positive relation-
 
 e5→2    e5→   ship between two factors should be considered as
 →6

 e6→2  e6→4     stimulating, while a negative relationship should
  be considered as an inhibiting one between two
       
nodes. With these rules, a CM can be represented
through an adjacency matrix which shows the sign
of the relationship, while keeping in mind that in
the case there being an absence of relationship
Origins of FCM between these two factors, the corresponding entry
will be empty. The path between two factors is the
Cognitive Maps (CM) (Axelrod, 1976) and re-
sequence of all the nodes connected by arrows,
cently, FCM (Kosko, 1986), have emerged as
but without considering their signs (Kardaras &
alternative tools for modelling the behaviour
Karakostas, 1999).
123
CM possesses, as their main limitation, the sign ei → j values. When an expert assigns a
impossibility of quantifying relationships among ei → j value, three issues exist and must be
variables. With the purpose of offering a solution
kept in mind (Schneider et al., 1998). In the
to this weakness and enhancing CM, fuzzy num-
bers have been conjugated with cognitive maps. first place, they should be aware of ei → j
In this context, FCM were introduced by Kosko weight indicate how strong the x i node is
(1986), and allow the providing of fuzzy causation on x j node. Secondly, the strength of the
measures to the CM proposed by Axelrod (1976). relationship is given by a fuzzy weight pre-
FCM substitute the signs (+) and (-) for a fuzzy ceded by a positive or negative sign indi-
value between - 1 and 1. Through this range of cating whether that relationship is direct or
value FCM allow the making of a wider interpreta- inverse respectively. Lastly, the causality
tion of complex problems (Lee et al., 2002; Lee relationship needs to be indicated to estab-
et al., 2004; Liu and Miao, 1999; Schneider et al., lish if the x i node is a cause of x j node or
1998), due to the possibility of applying weights
vice versa. In third place, experts should
to the relationships. Indeed, this soft computing
take into account the data domain.
technique helps to take critical decisions in areas
• STEP 3. Adjacency matrix is generated
such as manufacturing, medicine, industrial engi-
on the basis of the information obtained in
neering, marketing, finance, and other domains
STEPS 1 and 2.
require multiple and interrelated time constrained
• STEP 4. Graph-based FCM model is
decisions within strongly uncertain and complex
drawn on the basis of the information ob-
environments.
tained in STEPS 1 and 2.
FCM Development
Different methods can be used to build FCMs
(Kang et al., 2004; Lee & Ahn, 2009; Salmeron,
FCMs constitute neuro-fuzzy systems (Kosko,
2010; Schneider et al., 1998). However, if it is
1986; Lee et al., 2004) which are able to incor-
foreseeable that the map may feature a large num-
porate human knowledge (Stylos & Groumpos,
ber of variables and causal relationship between
2000). These are normally built by experts who
them, it can become necessary to use methodolo-
have experience and sound knowledge in this re-
gies, as Augmented FCM, to reach a consensual
gard. They offer specific knowledge in designing
result. The Augmented approach (Dikerson &
the FCM model (nodes, intensity and signs of the
Kosko, 1994; Salmeron, 2009), allows to reach
edges). FCMs are normally built by experts, who
a consensual FCM. In this way, this method re-
draw each one individually their FCM model,
quires that experts do not change their judgment
which represents his/her knowledge in the phe-
for consensus purposes (Salmeron, 2009), as is
nomena studied. The main steps in this process
required by the Delphi methodology. Moreover,
are as follows:
participants’ answers are not constrained by strict
nodes. This allows us to construct a complete
• STEP 1. Experts identify the relevant
FCM that represents all experts’ opinions. It is
characteristic, factors or variables which
possible because the Augmented FCM approach
represent the issue studied better. That is,
is an additive method. This approach consists
they indentify which nodes should be put
in adding the FCM generated by each expert.
together in the FCM.
Combining the FCMs together, conflicting rela-
• STEP 2. Experts identify the causal rela-
tionship are canceled out or minimized whereas
tionships among nodes. That is, they as-
124
agreement reinforces causal connections (Özesmi For example, let us consider two FCMs with
& Özesmi, 2004). some common nodes. Starting from each adja-
In this sense, the augmented adjacency matrix cency matrix:
(E AUG
) is built adding the adjacency matrix of
each expert. The resulting augmented matrix in- A B C
cludes the union of the causal nodes for all of the A 0 0.8 −0.4
experts. If an expert’s FCM does not include a E FCM 1 =  
B 0 0 0.1 
specific concept, then those rows and columns in  
the adjacency matrix are all zero. The resulting C 0 0 0 
augmented matrix is computed as follows:
A C D
n
E AUG = ∑ Ei (2) A  0 0.2 0
E FCM 2 =  
C  0 0
i =1
0
 
D 0.7 0.1 0
where n is the number of experts, Ei and is the

adjacency FCM matrix for expert i . The augmented adjacency matrix would be
Let us define two FCMs with no common nodes: built as follows:
FCM A with x iA as nodes, FCM A = {x iA } , and
FCM B with x jB as nodes, FCM B = {x jB }. The A B C D
adjacency matrix of FCM A is E A = wiA→ j ; and A  0 0.4 −0.1 0
 
the adjacency matrix of FCM B is E B = wiB→ j . E AUG = B  0 0 0.05 0
 
C  0 0 0 0
FCM A and FCM B are drawn by the expert A  
D 0.35 0 0.05 0
and B , respectively. The augmented adjacency
matrix is:
w A
 0  FCM Inference
E AUG
=  i → j  (3)
 0 wiB→ j 
FCMs are dynamic systems involving feedback
whereby change in a node may affect other nodes,
If there are common nodes, then the element which in turn can affect the node initiating the
w AUG
i→j
in the augmented matrix is computed by change. The analysis begins with the design of
(4). the initial vector state (C 0 ), which represents the
initial value of each variable or concept (node) at
the instant 0. The initial vector state with n nodes
∑
n
wik→ j
w AUG
i→j
= k =1
(4) is denoted as:
n
(
C 0 = C 10 C 20  C n0 ) (5)
where n is the number of FCMs added, one from
each expert, k is the identifier for each FCM, and
i and j are identifiers of the relationships.
125
where C i0 is the initial value of node x i at the gree of fuzzification in the [0,1] interval (Bueno
instant 0. & Salmeron, 2009; Grant & Osei-Bryson, 2005).
The new values of the nodes are computed in The FCM inference process finishes when
an iterative vector-matrix multiplication process stability is reached. The final vector state shows
with an activation function, which is used to map the effect of the change in the value of each node
monotonically the node value into a normalized on the FCM. After the inference process, the FCM
range [0, 1]. The value of each concept C it at the reaches one of three states following a number of
instant t would be computed as: iterations. It settles down to a fixed pattern of node
values, the so-called hidden pattern or fixed-point
  attractor (Papageorgiou, 2011).
C it +1 = f C it + ∑ C jt ⋅ e j →i  (6) Alternatively, the state can continue cycling be-
 
i≠j tween several fixed states, known as a limit cycle.
With a continuous function, a third possibility is
a chaotic attractor. This occurs when, instead of
where f(x) is the transformation function. stabilizing, the FCM continues to produce different
The state changes along the process. Usually, results (state vector values) for each cycle. In this
two kinds of transformation functions are used in case, the technique fails to offer a useful outcome
FCM dynamics (Bueno & Salmeron, 2009). The for IT/IS project success analysis.
first one is the unipolar sigmoid (7). The sigmoid
is the most common function (Bueno & Salmeron,
2009) when the concept (node) value maps onto AN ILLUSTRATIVE CASE
the range [0, 1]. OF FCM FOR MANAGING IT/
IS PROJECT SUCCESS
1
f (x ) = (7) IS/IT projects success may be threatened by many
1 + e −λ⋅x
factors. Hence, to avoid undesired outcomes,
practitioners have to monitor risks influences on
The second is the hyperbolic tangent (8). IT/IS project success. Given this professional
This is the most common function (Feyzioglu et need, we propose to use an fuzzy approach for
al., 2007) when the concept value maps onto the monitoring the progress of them. This method
range [-1, 1]. is explained through an illustrative case. This is
focused on enterprise systems projects.
e λ⋅x − e −λ⋅x The model proposed allows to predict the ef-
f (x ) = tanh(x ) = (8)
e λ⋅x + e −λ⋅x fects of specific risks scenarios on measures of
IT/IS projects success through simulations. With
Both functions use λ as a constant for function its development in mind, experts were consulted.
slope (degree of fuzzification). The FCM designer The building process and application of the final
has to specify the lambda value. For large values model are described in the following subsections.
of lambda (e.g., λ≥10) the sigmoid approximates
a discrete function that maps its results to interval Developing the FCM Model
(0,1); for smaller values of lambda (e.g., λ≤1) the
sigmoid approximates a linear function; while FCMs are normally developed by experts who
values of lambda closer to 5 provide a good de- have experience and sound knowledge in this
regard. They offer specific knowledge in design-
126
Figure 2. FCM of risks influences on IT/IS projects success
ing the FCM model (nodes, intensity and signs of The final FCM consists of eighteen risk nodes
the edges). The Augmented approach allows to (R) and four measures of success (S). This is
reach a consensual FCM. In this way, this method showed in Figure 2. Table 1 summarizes their
requires that experts do not change their judgment nodes and descriptions. In addition, the FCM
for consensus purposes, as is required by the Del- contains 28 edges, represented by the 22 × 22
phi methodology (Salmeron, 2009). Moreover, augmented matrix in Box 1.
participants’ answers are not constrained by strict
nodes. This allows us to construct a complete Dynamic Behaviour of FCM Model
FCM that represents all experts’ opinions. For
these reasons, we decided to use this approach FCM technique allows to predict the behaviour
to construct the FCM. of the simulated model over time, using different
The experts specifically identified the nodes initial vector states. For this purpose, firstly, we
and the relationships between them. Thus, we designed what-if scenarios at the instant 0. With
obtained an adjacency matrix describing each the intention of observing the evolution of several
expert’s opinion. We eventually created the initial scenarios, each analysis begins with the
augmented matrix by aggregating the adjacency definition of an initial vector (Co), which repre-
matrix of each expert (Kosko, 1996). This ag- sents the risks scenario proposed. This means that
gregation process depends on whether there are it would start a specific IT/IS project, but one o
common nodes between the FCMs. Because our several risks are known before. Each initial vec-
experts’ adjacency matrices had common nodes, tor state interacts with the adjacency matrix (E)
we computed the elements for the augmented as detailed before.
matrix using (4). With this in mind, we defined the value of
each node in each initial state vector (5), for each
127
Table 1. FCM nodes

The application of hyperbolic tangent function
ID NODES requires that FCM designers establish the value of
R1 IT/IS project requirements are not properly defined
λ. This is a parameter that determines the grade
R2 Be short of detailed of plan of the IT/IS project
of fuzzification of (8).
R3 Cannot meet IT/IS project milestones
• For larger values of λ (e.g. λ=10) the hy-
R4 Continuing stream of requirement modifications
perbolic tangent function approximates a
R5 Developing the wrong system functions
discrete function that maps its results to
R6 Developing the wrong users interfaces
[-1, 1].
R7 Inadequate assessment of performance requirements
• For smaller values of λ (e.g. λ= 1) the hy-
R8 Excessively-complex requirements
perbolic tangent function approximates a
R9 Failure to manage end user expectations
linear function.
R10 Gold plating
• For values of λ closer to 5, it provides a
R11 High turnover within IT/IS project team
good degree of fuzzification in [-1, 1] in-
R12 Immaturity of the new technology terval (Stylios & Groumpos, 1999).
Wrong estimation of hardware and software capabili-
R13
ties
In our study, both Simulation 1 and Simulation
R14 Simulations tools are inadequate
2 reached a stability threshold. Table 2 depicts
R15 Incompatible artifacts
the findings obtained simulating each Scenario.
R16 Insufficient personnel within IT/IS project
In the first Simulation, we wanted to know
R17 Inadequately-trained IT/IS project personnel
how those elements, which can be classified into
IT/IS project team personnel lack necessary skills/
R18
knowledge
requirements risk dimensions (Wallace et al.,
S1 Users satisfactions
2004), impact on the measures of IT/IS projects
success. This refers to issues associated with the
S2 IT/IS project time
IT/IS projects requirements demanded. Hence,
S3 IT/IS project cost
we compute R1 (IT/IS project requirements are
S4 IT/IS project requirements
not properly defined), R4 (Continuing stream of
requirement modifications), R7 (Inadequate as-
sessment of performance requirements), R8
simulated scenario. In this proposed case, we
(Excessively-complex requirements) and R10
established a couple of initial vectors state, each
(Gold plating), with a value of one in (5). The
one had some risks activated. The Scenarios were
initial vector for the concept values is shown in
transformed according to (6). To do so, we previ-
Box 2.
ously had to decide which transformation function
The results express how the changes in risks
use in our study. We selected the hyperbolic tangent
derived from issues associated with the require-
function (8) because the values of the nodes can
ments demanded may affect other risks, as well
fall within the range [-1, 1] (Feyzioglu et al., 2007).
as the measures of IT/IS projects success. In
That is, the vector nodes could acquire negative
Table 2, we can see that the impacts of the require-
values, unlike other transformation functions.
ments’ risks on the rest of the risks was from 0
In addition, hyperbolic tangent function need a
(null effect) to + 0.3836 (slightly moderate effect).
lower number of interactions to reach a stable
The three most highly affected risks were R5, R6
scenario, compared to other transformation func-
and R9. These risks represent very diverse issues
tions (Bueno & Salmeron, 2009).
such as the projects management and technical
risks. R9 was the highest impacted risk (+0.3836).
128
Box 1.
0.0 0.2 0.0 0.0 0.3 0.3 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0 0.0 0.0 0.0 −0.8 
 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 .0 0.0 0.0 0.0 0.0 −0.6 −0.2 0 
 
0.0 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0. 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
 
0.0 0.0 0.0 0.0 0.0 0.0 0.25 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.4 0.0 
 
0.0 0.0 0.0 0.0 0 .0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.4 0.0 0.0 0.0 
 
0.0 0. 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.4 0.0 0.0 0.0 
0.0 
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.5 0.0 0.0 −0.3

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 .0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.1
 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 
E AUG =  
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 

0.0
 0.0 0.0 0.0 0 .2 0 .2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 

0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.2 0.0 0.0 
 
0.0 0.0 0.0 0.0 0 .0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.1 0.0 0.0 
 
0.0 0. 0 0.0 0.0 0.0 0.0 0.0 0.0 0. 4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 −0.2 −0.2 −0.1
0.0 
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 .0 0.0 0.0 0.0 0.0 0.0 

0.0
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0..0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0. 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0..0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
This highlights that the requirements’ risks exert majority of the measures of success. In fact, the
slightly moderate influence on the proper manage- values reached by them in this simulation were
ment of end users expectation. That is, when within the interval [-0.8618, -0.5489]. This means
projects requirements are excessively-complex that when one of above risks change, it causes a
and/or continuously changing, and/or these are change in all the measures of success in the op-
not properly defined and/or evaluated, it makes posite direction. In other words, if the probability
the management of end users expectation difficult. and/or impact (Boehm, 1991) of risks activated
Moreover, the activated risks also affect the de- in Scenario 1 increase, then the measures of IT/
velopment of system functions (0.3082) and users IS project success will be, at least, moderately
interfaces (0.3082). Therefore, to avoid these impacted.
failures, IT project managers should take measures S1 (Users satisfactions) and S2 (IT/IS project
addressed at monitoring the requirements’ risks. time) are the most strongly impacted measures of
With this in mind, they should intensify their ef- success. The first is S1 with a value of -0.8618.
forts to establish standard processes, procedures This demonstrates that if the risks derived from
and/or methodologies. These will indicate person- issues associated with the requirements demanded
nel what the best way to identify, define and assess are not properly monitored, the satisfaction of
the requirements demanded is. system users will be likely affected. This is very
Simulation 1 also makes it evident that the critical because, if system users become dissatis-
requirements’ risks negatively affect the measures fied, they will not use the system properly. They
of IT/IS projects success. Moreover, the findings can even sabotage the application. In such cases,
show that these risks exert a high influence on the the adopting organization will not benefit from
129
Table 2. Simulations results

the expected results of the system adoption. The
second measure most strongly impacted was S2 ID SCENARIO 1 SCENARIO 2
with a value of -0.6948. This shows that the oc- R1 0.0344 0.0000
currence of the risks activated may highly impact R2 0.2706 0.0000
the possibility of IT/IS projects surpass the sched- R3 0.0000 0.0000
uled time. S3 (IT/IS project cost) and S4 (IT/IS
R4 0.0344 0.0000
project requirements) also were impacted, with a
R5 0.3082 0.0000
value of -0.5489 and -0.6764, respectively. This
R6 0.3082 0.0000
highlights that the risks activated in Scenario 1
R7 0.2907 0.0000
have a slightly high influence on the fulfillment
R8 0.0344 0.0000
of the requirements demanded. Moreover, these
R9 0.3836 0.6658
risks cause cost overruns in IT/IS projects.
R10 0.0344 0.0000
Finally, in the second Simulation, we wanted
R11 0.0000 0.0362
to know how those elements, which can be clas-
sified into team risk dimensions (Wallace et al., R12 0.0000 0.0000
2004), affect the measures of IT/IS project success. R13 0.0000 0.0000
This dimension refers to issues associated with R14 0.0000 0.0000
the personnel of IT/IS projects teams. Hence, we R15 0.0000 0.0000
computed R11 (High turnover within IT/IS project R16 0.0000 0.0362
team), R16 (Insufficient personnel within IT/IS R17 0.0000 0.0362
project), R17 (Inadequately-trained IT/IS project R18 0.0000 0.3433
personnel), R18 (IT/IS project team personnel lack S1 -0.8618 -0.8181
necessary skills/knowledge), with a value of one S2 -0.6948 -0.5742
in (5). The initial vector for the concept values is S3 -0.5489 -0.5507
shown in Box 3. S4 -0.6764 -0.7584
The second Simulation express how the
changes in risks derived from issues associated
Box 2.
0
C Scenario _1 (
= 1 0 0 1 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 )
Box 3.
0
C Scenario _2 (
= 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 1 0 0 0 0 )
130
with the personnel of IT/IS projects teams may also were affected, with a value of -0.5742 and
affect other risks, as well as IT/IS projects success. -0.5507, respectively. Hence, the risks activated
The findings obtained show that the impacts of in Scenario 2 may moderately provoke delays in
the activated risks on the rest of the risks was IT/IS project and the appearance of unforeseen
from 0 (null effect) to +0.6658 (high effect). R9 expenses.
was again the highest affected risk (+0.6658).
This shows that the team’ risks have high and
positive influence on the proper management of FUTURE RESEARCH DIRECTIONS
end users expectation. Therefore, the managers AND CONCLUSION
should continuously train and carefully select the
members of their teams, as well as to avoid the The successful development of IT/IS projects
high turnover of personnel. The results also reveal is a cost, complex and risky activity. In fact, the
that the rest of most strongly impacted risks were progress and outcomes of the final projects are
the same activated risks. This proves that the risks usually uncertain. Moreover, these projects require
derived from issues associated with the personnel facing many unforeseen risks. Poor risk manage-
of IT/IS projects teams are closely related. There- ment often leads to failure, which may adversely
fore, IT project managers should carefully moni- affect to a greater or lesser extent on the measures
tor the risks activated, in order to avoid undesired of IT/IS project success. In addition, if an IT/
events. IS project fails, this may negatively affects the
The results also show that the risks activated company performance. Hence, an effective risks
in Scenario 2 may threaten IT/IS projects success. management will be a critical issue. In this way,
Furthermore, the influence on the majority of the practitioners have to monitor the critical risks
measures of success is very high. In fact, the aver- influencing this kind of projects.
age impact was -0.67541. This means that when For this purpose, practitioners require tools
one of above risks change, it causes a change in and methodologies which support their work.
the measures of success in the opposite direc- In this sense, this chapter proposes the use of an
tion. Therefore, if the probability and/or impact artificial intelligence technique for monitoring
(Boehm, 1991) of risks activated in Scenario 2 risks impacts on the measures of IT/IS projects
increase, then the users’ satisfaction, the pos- success. Moreover, we provide an illustrative case
sibility of IT/IS projects meets the requirements to show the usefulness of the approach proposed.
demanded and does not surpass the scheduled time With this in mind, we defined two initial scenarios
and budgeted cost limits may be highly affected. and simulated them. Each simulation begins with
S1 (Users satisfactions) and S4 (IT/IS project the definition of the initial risk scenario. It means
requirements) were the most strongly impacted that it would start a specific IT/IS project, but
measures of success. The first was the same that one o several risks are known before. The FCM
in Scenario 1 (S1), with a value of -0.8181. This evolution generates the final IT/IS project success
highlights how the action of risks derived from situation for each risks scenario.
issues associated with the personnel of IT/IS proj- The author’ proposal offers some advantages
ects teams may highly impact on the satisfaction in comparison with others similar tools. This
of system users. The second was S4 with a value chapter proposes an innovative and flexible
of -0.7584. This reveals that the occurrence of the technique called Fuzzy Cognitive Maps to IT/IS
team’ risks may highly impact the possibility of risks scenarios. It can be adapted to a wide range
IT/IS projects meets the requirements demanded. of problems, especially in knowledge intensive
S2 (IT/IS project time) and S3 (IT/IS project cost)
131
environments. This flexibility allows the model scenarios, especially “what-if” analysis. For these
to be turned into a useful and innovative tool. reasons, we think that FCMs is a valuable tool
Firstly, FCM technique allows the defining to monitor risks effects on the measures of IT/IS
of relationships between concepts. Through this project success.
characteristic, decisional models that are more Notwithstanding, FCM are not a close research
reliable for interrelated environments are defined. topic yet. Further research about FCM develop-
Secondly, FCM is able to quantify the influence of ment and validation are needed. Our future work
the relationships between concepts. Through this will be directed towards the building of methods
attribute, a better support in complex decisions for ranking FCM based scenarios. Specifically, we
can be reached. Finally, with this FCM model it expect to propose approaches addressed at using
is possible to develop a what-if analysis with the the scenarios’ built, assessment and ranking as
purpose of describing possible scenarios. a whole. It could bring methodological support
FCM application was used to analyze IT/ to scenario-based decision making in scenario
IS projects risks and interesting findings were analysis.
extracted. Two risks initial scenarios were simu-
lated and their impacts over the model have been
detailed. Through this proposal, one can observe REFERENCES
which the most relevant risks are, and, above all,
which have a greater impact on the measures of Aloini, D., Dulmin, R., & Mininno, V. (2007). Risk
IT/IS projects success. It allows using FCM as a management in ERP project introduction: Review
simulation tool, where the initial risk scenarios of the literature. Information & Management,
suggests the future problems in IT/IS projects. 44(6), 547–567. doi:10.1016/j.im.2007.05.004
However, FCMs technique suffers some limita- Axelrod, R. (1976). Structure of decision: The
tions. The resulting FCM is highly dependent on cognitive maps of political elites. Princeton, NJ:
the data source. In fact, a number of experts are Princeton University Press.
required to ensure objective and globally valid
results. However, the complexity of data analysis Bakker, K., Boonstra, A., & Wortmann, H. (2010).
and manipulation increases with the number of Does risk management contribute to IT project suc-
experts interviewed. To avoid this, the number of cess? A meta-analysis of empirical evidence. Inter-
participating experts was selected according to a national Journal of Project Management, 28(5),
recommended range. In addition, the model has 493–503. doi:10.1016/j.ijproman.2009.07.002
to be as accurate as possible, because an incon-
Bannerman, P. L. (2008). Risk and risk manage-
sistent model eliminates the results’ validity. For
ment in software projects: A reassessment. Journal
this reason, we carefully made up heterogeneous
of Systems and Software, 81(12), 2118–2133.
panels of experts.
doi:10.1016/j.jss.2008.03.059
Other limitation of FCM is its high number of
degrees of freedom in the design of the experi- Bellman, R. E., & Zadeh, L. A. (1970). Deci-
ment. In fact, there are not rules which guide to sion-making in a fuzzy environment. Manage-
the FCM designers in the choice of the transforma- ment Science, 17(4), 141–164. doi:10.1287/
tion functions, assignation of parameters’ values, mnsc.17.4.B141
among others. In spite of these limitations, FCMs
Boehm, B. W. (1991). Software risk management:
has been successfully applied for structuring and
Principles and practices. IEEE Software, 8(1),
supporting decisional processes. Also, FCMs pro-
32–41. doi:10.1109/52.62930
vide excellent mechanisms to develop forecasting
132
Bueno, S., & Salmeron, J. L. (2009). Benchmark- Iversen, J. H., Mathiassen, L., & Nielsen, P.
ing main activation functions in fuzzy cognitive A. (2004). Managing risk in software process
maps. Expert Systems with Applications, 36(3), improvement: An action research approach.
5221–5229. doi:10.1016/j.eswa.2008.06.072 Management Information Systems Quarterly,
28(3), 395–433.
Cormican, K., & O’Sullivan, D. (2004). Audit-
ing best practice for effective product innovation Kang, I., Lee, S., & Choi, J. (2004). Using fuzzy
management. Technovation, 24(10), 819–829. cognitive map for the relationship management in
doi:10.1016/S0166-4972(03)00013-0 airline service. Expert Systems with Applications,
26(4), 545–555. doi:10.1016/j.eswa.2003.10.012
perceived ease-of-use, and user acceptance of Kardaras, D., & Karakostas, B. (1999). The use
Information Technology. Management Infor- of fuzzy cognitive maps to stimulate the informa-
mation Systems Quarterly, 13(3), 319–340. tion systems strategic planning process. Informa-
doi:10.2307/249008 tion and Software Technology, 41(4), 197–210.
doi:10.1016/S0950-5849(98)00125-6
tion Systems success: The quest for the dependent Kelly, G. (1970). A brief introduction to personal
variable. Information Systems Research, 3(1), construct theory. London, UK: Academic Press.
60–95. doi:10.1287/isre.3.1.60
Kim, H. S., & Lee, K. C. (1998). Fuzzy impli-
Dikerson, J., & Kosko, B. (1994). Virtual worlds cations of fuzzy cognitive map with emphasis
as fuzzy cognitive maps. Proceedings of IEEE on fuzzy causal relationship and fuzzy partially
Virtual Reality Symposium, 3(2), 173-189. causal relationship. Fuzzy Sets and Systems, 97(3),
303–313. doi:10.1016/S0165-0114(96)00349-1
Eden, C., & Ackermann, F. (2004). Analyz-
ing cognitive maps to help structure issues or Kosko, B. (1986). Fuzzy cognitive maps. Interna-
problems. European Journal of Operational tional Journal of Man-Machine Studies, 24(24),
Research, 159(3), 673–686. doi:10.1016/S0377- 65–75. doi:10.1016/S0020-7373(86)80040-2
2217(03)00431-4
Kosko, B. (1996). Fuzzy engineering. Prentice
Feyzioglu, O., Buyukozkan, G., & Ersoy, M. S. Hall.
(2007). Supply chain risk analysis with fuzzy
Kwak, Y. H., & Stoddard, J. (2004). Project risk
cognitive maps. IEEE International Conference
management: lessons learned from software de-
on Industrial Engineering and Engineering Man-
velopment environment. Technovation, 24(11),
agement, (pp. 1447 –1451).
915–920. doi:10.1016/S0166-4972(03)00033-6
Grant, D., & Osei-Bryson, K. (2005). Using fuzzy
Lee, K. C., Kim, J. S., Chung, H. N., & Kwon, S.
cognitive maps to assess MIS organizational
J. (2002). Fuzzy cognitive map approach to Web-
change impact., Proceedings of the 38th Annual
mining inference amplification. Expert Systems
Hawaii International Conference on System Sci-
with Applications, 22(3), 197–211. doi:10.1016/
ences, (pp. 1-11).
S0957-4174(01)00054-9
133
Lee, S., & Ahn, H. (2009). Fuzzy cognitive map Rodriguez-Repiso, L., Setchi, R., & Salmeron, J. L.
based on structural equation modeling for the (2007). Modelling IT projects success with fuzzy
design of controls in business-to-consumer e-com- cognitive maps. Expert Systems with Applications,
merce web-based systems. Expert Systems with 32(2), 543–559. doi:10.1016/j.eswa.2006.01.032
Applications, 36(7), 10447–10460. doi:10.1016/j.
Salmeron, J. L. (2009). Augmented fuzzy cogni-
eswa.2009.01.070
tive maps for modelling LMS critical success fac-
Lee, S., Kim, B. G., & Lee, K. (2004). Fuzzy cog- tors. Knowledge-Based Systems, 22(4), 275–278.
nitive map-based approach to evaluate EDI per- doi:10.1016/j.knosys.2009.01.002
formance: A test of causal model. Expert Systems
Salmeron, J. L. (2010). Modelling grey uncer-
with Applications, 27(2), 287–299. doi:10.1016/j.
tainty with fuzzy grey cognitive maps. Expert
eswa.2004.02.003
Systems with Applications, 37(12), 7581–7588.
Liu, Z., & Miao, Y. (1999). Fuzzy cognitive map doi:10.1016/j.eswa.2010.04.085
and its causal inferences. IEEE International
Salmeron, J. L., & Herrero, I. (2005). An AHP-
Fuzzy Systems Conference Proceedings, 3, (pp.
based methodology to rank critical success factors
1540-1545).
of executive information systems. Computer Stan-
Nasir, M. H. N. M., & Sahibuddin, S. (2011). dards & Interfaces, 28(1), 1–12. doi:10.1016/j.
Addressing a critical success factor for software csi.2004.09.002
projects: A multi-round Delphi study of TSP.
Schneider, M., Shnaider, E., Kandel, A., & Chew,
International Journal of the Physical Sciences,
G. (1998). Automatic construction of FCMs. Fuzzy
6(5), 1213–1232.
Sets and Systems, 93(2), 161–172. doi:10.1016/
Özesmi, U., & Özesmi, S. L. (2004). Ecological S0165-0114(96)00218-7
models based on people’s knowledge: A multi-step
Stylos, C. D., & Groumpos, P. P. (1999). Math-
fuzzy cognitive mapping approach. Ecological
ematical formulation of fuzzy cognitive maps.
Modelling, 176(1-2), 43–64. doi:10.1016/j.ecol-
Proceedings of the 7th Mediterranean Conference
model.2003.10.027
on Control and Automation, (pp. 2251-2261).
Papageorgiou, E. I. (2011). A new methodology
Stylos, C. D., & Groumpos, P. P. (2000). Fuzzy
for decisions in medical informatics using fuzzy
cognitive maps in modeling supervisory control
cognitive maps based on fuzzy rule-extraction
systems. Journal of Intelligent and Fuzzy Systems,
techniques. Applied Soft Computing, 11(1),
8, 83–98.
500–513. doi:10.1016/j.asoc.2009.12.010
Wallace, L., Keil, M., & Rai, A. (2004). Under-
Peffers, K., Gengler, C. E., & Tuunanen, T. (2003).
standing software project risk: A cluster analysis.
Extending critical success factors methodology
to facilitate broadly participative Information
doi:10.1016/j.im.2003.12.007
Systems planning. Journal of Management In-
formation Systems, 20(1), 51–85. Wallace, L., Keil, M., & Rai, A. (2004). How
software project risk affects project performance:
Rockart, J. F. (1979). Chief executives define
An investigation of the dimensions of risk and
their own data needs. Harvard Business Review,
an exploratory model. Decision Sciences, 35(2),
52(2), 81–93.
289–321. doi:10.1111/j.00117315.2004.02059.x
134
Working Group from the Royal Academy of IT/IS Projects: Projects developed to maintain
Engineering and the British Computer Society. and incorporate the most innovative technologies
(2004). The challenges of complex IT projects. in the enterprise information systems.
London, UK: The Royal Academy of Engineering. IT/IS Projects Decision Making: Fuzzy
dynamic systems support practitioners in IT/IS
Xirogiannis, G., & Glykas, M. (2007). Intelligent
projects decision making.
modeling of e-business maturity. Expert Systems
Measures of Success: Conditions must be
with Applications, 32(2), 687–702. doi:10.1016/j.
satisfied to support successful IT/IS projects.
eswa.2006.01.042
Risks Factors: Events that cause dangerous
Zadeh, L. A. (1965). Fuzzy sets. Information failures in IT/IS projects.
and Control, 8, 338–353. doi:10.1016/S0019-
9958(65)90241-X
ENDNOTE
1
The mean was calculated by summing the
KEY TERMS AND DEFINITIONS values of all the measures of success and
dividing by the number of measures of suc-
Decision Support: Fuzzy dynamic systems
cess in the list.
are considered decision support technique.
Fuzzy Dynamic Systems: Are fuzzy systems
involving feedback over time.
135
Section 3
Information Systems
Maintenance and Development
137
Chapter 8
A Model to Assist the
Maintenance vs. Replacement
Decision in Information Systems
O. Tolga Pusatli
Cankaya University, Turkey
Brian Regan
University of Newcastle, Australia
ABSTRACT
An extensive literature review shows that Information Systems (IS) are changed and eventually replaced
by substitutes under the influence of productivity, popularity, and specialisation of IS along with as-
sociated available support facilities, maintenance activities, failures, and user feedback. This chapter
compiles those factors emerged from software engineering, IS, software quality assurance measurements,
and computer science literature. A final product is a proposed model bringing those factors together as
they are suspected to raise a need for taking the decision of evaluating change requests that may lead
to a further maintenance or replace the IS. While keeping in mind that limitations on advanced testing
exist, an expected service of such a model may help decision makers to explain maintenance/replacement
decision of IS/component in a more itemized manner, hence diminish overburden pressure of experience
responsibility on them.
DOI: 10.4018/978-1-4666-0170-3.ch008
A Model to Assist the Maintenance vs. Replacement Decision in Information Systems
INTRODUCTION AND BACKGROUND The decision to replace or maintain IS or major

components is presented in the literature as an
While much research and discussion has oc- organisational decision, made by some group of
curred around the best processes for developing stakeholders. An early model of when to replace
new information systems (IS), there has been rather than maintain a system was provided in
less analysis of the dynamics of making the Gupta and Raghunathan (1988), based on relative
important decision of whether to move from the costs associated with maintenance and replace-
maintenance mode for an existing system into ment. Other factors such as availability of skills and
developing or purchasing a new replacement business goals were noted but not incorporated in
software system. Throughout their lifecycles, IS that model. Pragmatically, the vision or otherwise
evolve with continual maintenance, which fixes of top management leadership and organisational
errors and extends functionality, but eventually politics also play a role in the replacement deci-
most systems become unstable and unsustain- sion, as reported in a case study (Rada & Finley,
able. Making the call on when that point has been 2004) on whether or not to maintain legacy IS.
reached is an important and challenging decision Part of the problem for management however
for many managers. The main objective of this is the difficulty in predicting life expectancy of IS
chapter is to propose a model that may be useful so as to plan for replacement, according to (Glass,
to explain a path leading to a decision point of 1998). That is, it is difficult to predict when main-
whether the IS/component should be replaced or tenance will not be enough and replacement will
further maintained. While we investigate literature be needed, although these authors say generally
on IS including technology acceptance, success that as the effort involved to maintain the system
models, user feedback, software functionality, increases, time for replacement approaches. An-
bespoke versus off-the-shelf systems, maturity other example is reported by Salmeron and Lopez
models, life cycle analyses, software evolution (2010), where maintenance is an “obligation” in
measurement and quality assurance, we find it ERP systems despite such a task is risky for its
beneficial to mention that the objective of this complexity and cost. Because an organisation’s IS
chapter excludes social, economical and account- can be composed of subsystems such as revenue,
ing associated factors. administration, e-mail and operations support, the
Under the influence of factors such as changes decision to replace whole systems is not only dif-
in functional requirements, external constraints ficult to make, but it also has significantly higher
and new hardware environments, software systems risk when compared to replacing a portion. New
or components are finally abandoned and a sub- releases hence usually concentrate on improving a
stitute is usually sought. Heales (2002) contrasts small number of subsystems (Sommerville, 2004).
“evolution” brought by maintenance activities One of the “laws of program evolution” (Meir
such as updates, error corrections and changes Manny Lehman, Ramil, Wernick, Perry, & Turski,
to improve functionality with “revolution”, the 1997) is that, as systems evolve, their complexities
realisation that a system needs to be replaced with increase unless work is done to maintain them. All
implications of associated upheaval. In Heales’ software systems reach a state of internal interac-
language, we are interested in the decision whether tion such that any effort to fix one error is expected
to evolve or to go to revolution. to introduce more new errors, with a resulting
In practice, decisions to maintain or replace cascade of system failures and errors (Yourdon,
systems are often based on a set of guidelines 1989). Thus, the quality of systems will decline
tailored to an organisation’s needs through system until they are eventually replaced. Addressing this
reviews and feasibility studies (Horch, 2003). issue, we can recall Heales’s research (Heales,
138
2002), where the term “volatility index” has been to prevent future failures. Hence, maintenance
coined to describe the increasing instability of IS activities do not lie wholly in any one of the types
as they approach the stage when evolution (being corrective, perfective, adaptive or preventive.
maintainable) is no longer practicable and they Lehman and colleagues (Lehman, 1980;
are replaced. The higher the volatility index is, Lehman & Belady, 1985; Lehman & Ramil, 2000;
the higher the probability is that systems have to Lehman & Ramil, 2003; Lehman et al., 1997)
be replaced. claim that systems must be continually adapted or
An early study put by Gupta and Raghunathan they become progressively less satisfactory; the
(1988) investigating costs associated with main- functional content of systems must be continually
tenance and replacement concluded that as the increased to maintain user satisfaction over their
number of change requests rose, the organisation lifetime. Changes in the training and expectations
should consider replacement, rather than evolu- of users make subsystems appear “old” even
tion. Increased modularity of a system increases when there are no software aging issues (analysed
its ability to adapt to external changes and thus later in this chapter) or maintenance problems
extends its life span. Similarly, the flexibility of (Jazayeri, 2005). In studying requirements over
a system to adapt to changing data formats used time, Anderson and Felici (2002) identify user
in an industry sector has been shown to increase feedback as a driver for the evolution of systems.
its longevity, by the definition given in (IEEE, Evolution occurs with rising expectations of the
1990) (flexibility or adaptability). users as well as with increasing knowledge and
experience of the developers, but these factors are
hard to measure objectively (Jazayeri, 2005). For
DRIVERS OF CHANGE REQUESTS this reason, Jazayeri argues that any measurements
AND MAINTENANCE would yield only an incomplete picture of the true
evolutionary phenomena. Meanwhile, acceptance
The various reasons for evolving a system – per- of a system by the users (and manager) secures
forming maintenance – have been divided into four its place as the system may be – in an extreme
categories (IEEE, 1990): to correct faults (correc- case – rejected if those people are excluded in the
tive maintenance); to improve performance and/ maintenance process (Salmeron & Lopez, 2010).
or add functionality (perfective maintenance); to In the literature however, new features and
adapt to a changing environment (adaptive main- emerging technologies are also reported as factors
tenance); and to be ready for a potential problem in system evolution (Jazayeri, 2005). Additional
(preventive maintenance). Although maintenance capabilities can be added when new technology
activities are classified and recognised in this allows for new functionality. Changes in IS brought
manner, a maintenance activity can be of more about by new technology may be driven by a range
than one type. For example, modifying a system to of stakeholders including customers, competitors,
accommodate more than one server is an adaptive suppliers, vendors, government regulation, mass
maintenance activity but it also can be considered media promotion beside the users, IT staff and
as perfective as performance is enhanced. Simi- top management (Bharati & Chaudhury, 2006).
larly, research such as that reported in (Li et al., In the technology adoption literature on the
2006) into better debugging techniques and stud- other hand, IS innovations are often described as
ies reported in (Qin, Tucek, Zhou, & Sundaresan, “diffusing” through organisations (White, 2008).
2007) into quick recovery from software failures, In fact, popularity (i.e. the extent of adoption) is a
is motivated by a mix of needs: to identify and fundamental part of technology adoption models,
fix software faults, to increase productivity, and which typically show uptake as a bell-shaped
139
curve over time (Rogers, 1962) (Rogers’ works dency among them. Lack of compatibility of new
on diffusion of innovation represents this curve, technologies with the current IS slows the rate of
where innovators and early adopters are at the adoption, as does distrust of new products, which
leading edge of the curve forming a relatively may be poorly understood and largely untested
small percentage of the total number of adopters). (White, 2008). System architectures that do not
A sample case study of adoption highlights some allow changes, rigid original design principles
of the factors that affect the variability in adoption and imprecise requirements are also obstacles
(Venkatesh & Morris, 2000). to change, along with resource limitations of the
Changing business environments is also con- software development team and the organisational
sidered as another direct driver of evolution (Jaza- environment (Eick, Graves, Karr, Marron, &
yeri, 2005). Large scale maintenance, requiring Mockus, 2001). “Decay” is a term used in (Eick
many small changes to be made in many places, et al., 2001) to describe the difficulty of modify-
may be required when there are new business re- ing a component due to accumulation of changes.
quirements.Veerman (2006) illustrates large scale Successful maintenance activities have been
maintenance effects by referring to Y2K and the found to depend not only on how maintainable the
introduction of the Euro currency. The problems software is, but also on the quality of the techni-
are increased when programs are highly interre- cal support team, and on attitudes and activities
lated or when new requirements force deployment of management and the user (Bhatt, Shroff, &
of new hardware and software technologies and Misra, 2004). Service quality has been identified
migration to different platforms; for example as an item influencing IS success (DeLone &
when legacy systems are migrated to web-based McLean, 2002), and service quality in turn has
systems (Colosimo, Lucia, Scanniello, & Tortora, been shown to depend on personnel (the IT team)
2009). Portability across platforms is hence seen as well as physical facilities, equipment, and the
as one of the key factors in enabling IS evolution quality of communication (Roses, Hoppen, &
(Veerman, 2006; Colosimo et al., 2009). Another Henrique, 2009). The availability and quality
point associated with the business environment of on-going support is a factor in replacement
is that excessive workload may degrade perfor- decisions (Shailer, 2001) and the quality of the
mance through degradation of system resources support offered have also been shown to be fac-
(Li, Vaidyanathan, & Trivedi, 2002). Preventive tors in off-the-shelf products obtaining a share in
maintenance activities are consequently motivated the software market (Smolin, 2008). Maintaining
by increased system usage, which is a product of software products through patches (updates that
an organisation’s operations (Singh & Goel, 2007). are supplemented by the developer companies)
In contrast to system evolution, the term is a common way to provide technical support.
“software evolution” focuses on the versions User feedback, business change and techno-
of software, rather than on whole systems or on logical change are also reported as reasons for
individual software products, where software change requests for maintenance. However, at
products, as with other goods sold in the market, some point, a system or a component of it becomes
are considered to have limited lifetime, which “too hard” to maintain and is replaced. Indicators
may be extended as developers release further of approaching end-of-life have been identified
versions, update patches, or merely guidelines as the time since the IS were first deployed (a
for work-arounds which are all integrated into coarsely quantified age) and the period between
the software package (Jazayeri, 2005). maintenance requests (Heales, 2002).
An obstacle to independent evolution of soft- An increasing number of maintenance activi-
ware, components or subsystems is the interdepen- ties is a sign of potential need for a replacement.
140
System age is known to significantly increase As a bespoke system would be developed ac-
maintenance time (Dekleva, 1992); while this cording to the organisation’s requirements, the
argument is open to discussion (e.g. (Heales, degree of specialisation is naturally high. System
2002) suggests that an organisation’s IT staff specialisation is a term used to describe the fit
can become more efficient when they maintain of a system with the specific requirements of
the same system over time) system age can in- the organisation. It can also be achieved through
fluence replacement decisions (Dekleva, 1992). selection of generic components (off-the-shelf)
Other characteristics of IS that are associated followed by customisation of the system (Haines,
with approaching end-of-life are the generation 2003). Disadvantages of customising an off-the-
of language used to develop the systems and their shelf product are that upgrades are not always
size, in terms of both the numbers of users and the possible and further modifications are more
executable lines of code (Heales, 2002); where difficult and hence more costly (Shailer, 2001).
the author suggests that size increases volatility. Another potential problem with both bespoke
and customised systems is dependency on the
technical personnel who created the specialisa-
ORGANISATIONAL REQUIREMENTS tions. However, purchasing components from
AND IS DEVELOPMENT different vendors may bring incompatibility issues
(Bucchiarone, Polini, Pelliccione, & Tivoli, 2006)
System specialisation, whether by bespoke sys- with errors created during integration because
tems or customised off-the-shelf systems, comes the output of one component may not match the
at a cost of ongoing maintenance complexities. input of another. This can force further modifi-
Future maintenance demands are reduced by cation in the whole system (Bucchiarone et al.,
ensuring compatibility and portability across 2006). So when IS are composed of a variety of
platforms, by modularity and by good IT support platforms, it is desirable that tailored programs
(including use of popular products), amongst other be able to be transferred with as little effort as
factors. The decision to replace or maintain IS is possible. That is, portability across platforms is
obviously connected with understanding the cur- important (Vicente, Alonso, & Amador, 1991).
rent requirements that the organisation has of its Flexibility (strongly associated with adaptability
IS. There is extensive literature on the elicitation and sometimes seen as synonym for portability,
of IS requirements. Knowing the requirements, however, we specifically take portability across
steps can be taken to upgrade IS to meet the or- platforms in this work) is “the ease with which a
ganisation’s needs, subject to resource constraints. system or component can be modified for use in
Meeting requirements through system develop- applications or environments other than those for
ment has been presented as a decision between which it was specifically designed” (IEEE, 1990).
four basic options: bespoke system built by the Adaptability is encouraged by the use of platform
organisation’s own IS team; bespoke system and vendor-free services, and systems that have
developed externally; buying an off-the-shelf been created to be portable to new environments
package; and buying an off-the-shelf package then (Tilley et al., 2002), (Peng, Li, & Mili, 2007)
customising it to suit the workplace (Shailer, 2001). and by designing for interoperability (Cayir &
Here, a bespoke system is one written especially Basoglu, 2008). Organisations benefit more from
for a customer’s particular requirements and the systems with more interoperability, according to
process of customisation is an “activity that leads (Cayir & Basoglu, 2008). Languages that sup-
to a change of a process or data definition in a port applications in distributed environments are
generic system” (Haines, 2003). designed to enable interoperability and reusability
141
of software components (e.g. NET Visual Basic, turn mainly induced by frequent requests from
Java and PHP). the end-users. The following quote is an example
To improve maintainability, organisations are that underlines the effect of the end user for a
also moving to more modular systems (Szyperski, technological change, in this case toward Internet
Gruntz, & Murer, 2002). Approaches such as technologies:
component based development (CBD) and ser-
vice oriented architecture (SOA) make IS more ...it is the mostly the end users, who suggested the
maintainable through increased flexibility and management of the company migrated the origi-
scalability in structure and increasing efficiency nal legacy systems towards the web. (Colosimo
in development. Here, CBD is “an approach to et al., 2009)
the development of software systems that builds
applications from components, or prewritten When it comes to collecting user feedback,
building blocks….” (Hodges, 2007) and SOA is there are many suggestions in the literature on how
“essentially a collection of services that commu- to do it; examples include basic mechanism such
nicate with each other... A service is a program as suggestion boxes, phone calls used since the
that has a specific function, typically a business mid 1970’s (Trotter, 1976), to the 2000’s implicit
function, such as analysing an individual’s credit user feedback collections and evaluations over the
history….” (Hodges, 2007). Internet (Sharma & Jansen, 2005; Tauritz, Kok,
Component based approaches are not without & Sprinkhuizen-Kuyper, 2000).
risk, for example, there may be problems with As we infer so far, feedback from users is
connectivity of components if there are inappro- important; however, they need to be treated cau-
priate choices of language abstractions and data tiously, as expectations from IS may be sometimes
type constructors (Soares, 1992). Large systems unrealistically high. According to Staples, Wong
are becoming more and more modular with CBD and Seddon, (2002), it was found to be particularly
and SOA, and a market has been created for shar- important to generate realistic expectations about
ing components among developers and reuse of the ease of use, usefulness and the quality of the
services (Boertien, Steen, & Jonkers, 2005; Marks information that would be held by the changed
& Bell, 2006; Gill & Tomar, 2010). Popularity of system.
the components in this market is one of the sig- The importance of managing expectations
nificant elements in developers’ decisions when about the first two characteristics is supported by
choosing IS components, according to (Hong & Davis and colleagues’ the technology acceptance
Lerch, 2002). model (TAM) (Davis, Bagozzi, & Warshaw, 1989;
Davis, 1993), which explains users’ acceptance
of technology and can be applied to explain
THE CRUCIAL ROLE OF THE USER computer acceptance in work practices. In the
TAM, intention to use a technology depends on
Since the users are the ultimate elements of IS perceived ease of use of the technology and its
and their feedback is important within this aspect perceived usefulness to the user’s task. Intention to
of the study, this section touches on technology use drives the usage behaviour, so that decreased
acceptance models and the role of user satisfac- intention linked with user dissatisfaction will
tion with the systems in IS success models. In a affect the adoption of new technlogies, and in
case study on changing business work practices, particular, new IS.
(Colosimo et al., 2009) observed that this change The role of user satisfaction in IS development
was caused by changes in the IS, which were in and success is also well established. As well as
142
Figure 1. DeLone-McLean model for IS success (DeLone & McLean, 1992)
ease of use, satisfaction depends on functionality and acceptance, it is worth noting that tailored
being available when required (availability), and systems are not guaranteed to be accepted by the
on system performance (Shye et al., 2008; Mar- users (Kyhlbäck & Sutter, 2007). However, off-
kovic & Wood, 2004). Usage of IT systems and the-shelf IS may have difficulties responding to
user satisfaction are emphasised in the framework the needs of the work place.
developed by DeLone and McLean (DeLone &
McLean, 1992; DeLone & McLean, 2002) to
discuss and model IS success. As illustrated in THE AGING OF IS PRODUCTS
Figure 1, quality of the system and quality of
the information in it provides affect use of the As discussed earlier, while the system as a whole
system and user satisfaction with it. Use and evolves, components of an IS age. This is because
user satisfaction influence each other, as well as systems are composed of integrated subsystems
the individual user of the IS in the organisation. and each subsystem, when it is believed to be
Finally, impact on the users creates an impact out-dated, is replaced. This relates to the suspicion
on the organisation, i.e. an impact on the way of that the aging of software and hardware may lower
delivering a work of an individual may cause an productivity, thus we must focus on IS compo-
impact on the organisation. nents, including both software and hardware.
DeLone and McLean’s model has contributed Parnas (1994) claims that software aging has
to a great deal of research, some of which supports two causes: not maintaining the software correctly,
the relationships and some of which challenges and ongoing modifications due to maintenance
elements of the model. For example, Seddon needs. The “phenomenon of software aging” is
(1997) questions the role of IS use in the model often defined to be the accumulation of errors
so that it should be considered as a behaviour during the runtime of the software, driving it
rather than an IS success measure, and Iivari to failure through crashing, hanging or freezing
(2005) acknowledges that system and information (Garg, Moorsel, Vaidyanathan, & Trivedi, 1998;
quality are significant predictors of user satisfac- Jing, Jian, Xue-long, & Feng-yu, 2005). Such
tion with the system, but argues they are not problems emerge when software applications
predictors for system use. Nevertheless, the execute continuously for long periods of time
model is widely accepted in the IS literature and and the performance of software degrades. For
has often been adapted, for example with research example, scientific and analytical applications
into the impact of social factors in virtual com- or modelling and simulation applications may
munities (Lin, 2008), or applied as an overall run for days or weeks, equally web servers are
framework while measuring the dimensions of not expected to be interrupted for long periods of
global IS success. In considering user satisfaction time and can witness their performance degrade.
143
Manifestations of software aging include slow or deleted versus the total number of functions,
degradation and exhaustion of a system resource where a function is defined as “a software module
(such as memory leaking), unreleased file locks, that performs a specific action, is invoked by the
data corruption and numerical error accumulation. appearance of its name in an expression, may
Slow performance degradation occurs resulting in receive input values, and returns a single value”
limiting effective use of system resources. (IEEE, 1990). The rationale is to determine the
Related to aging, software rejuvenation refers relative impact of the changes, keeping in mind
to techniques to counter aging. Various methods that the more modifications that take place, the
have been developed for different IT sub-domain more unstable the software product is.
applications (Vaidyanathan & Trivedi, 2005; Liu, The specific SMI formula is
Trivedi, Ma, Han, & Levendel, 2002; Okamura,
Miyahara, & Dohi, 2005; Bao, Sun, & Trivedi, SMI = (MT – (Fa + Fc + Fdel))/MT
2005). Typically, rejuvenation involves stopping
the running software, doing garbage collection,
flushing operating system kernel tables, and reini- where:
tializing internal data structures (Jing et al., 2005).
Such re-initialization has become the caricature MT: # of software functions in the current delivery
of first responses by many help desk operators. Fa: # of software functions in the current delivery
Hardware is another item associated with that are additions to the previous delivery
aging. Many organisations have replacement of Fc: # of software functions in the current delivery
hardware policies defined by time. This is partly that include internal changes from a previ-
related to the warranty (e.g. three years warranty ous delivery
for a server) given by the manufacturers and or- Fdel: # of software functions in the previous deliv-
ganisational accounting practices. Organisations ery that are deleted in the current delivery
may find it risky not to replace older hardware
and the new hardware may need new software SMI has been criticised (IEEE, 2006) for
and may not function properly with the legacy not being a measure of maturity but a measure
components. An example is given in (Grudin, of module change rate and not a good one even
2004) which reports the need to upgrade from of that, because the value could be negative.
Windows 95 to 98 when replacing single monitors Neverthless, the SMI has led to variants such as
with multi-monitors because Windows 95 could the requirement maturity index (RMI), which
not recognise the latter. Assessing the reliability considers the number of requirements, number of
of systems in terms of age can be useful for changes in requirements, cumulative and average
hardware (Xu, 2007). We found several examples of those numbers for historical analysis (Anderson
distributed by IT companies showing associated & Felici, 2002).
risks involved when a server is out of warranty. After the addition of the age to the literature
The concept of “maturity”is also related to survey, we have seen that there is a potential in
age, where maturity is associated with stable the measurements used in software engineering
functionality. An important measure of software to reconsider what is shaping change requests.
maturity is software maturity index (SMI), which
is included in a IEEE standard used to quantify
the readiness of a software product. SMI is a for-
mula that provides a number based on the count
of functions that have been added, changed and/
144
METRICS FOR CHANGE a given attribute” (IEEE, 1990). A “measurement”

MANAGEMENT is then a task which computes a metric from the
attributes of the entities within a given domain,
Knowing the attributes that have been found worth using clearly defined rules (Bourque et al., 2004).
measuring is relevant to our model building even Metrics must be purpose oriented (Berander &
though metrics are often controversial. Software Jönsson, 2006) and have clear objectives, usually
engineering provides metrics, such as reliability, generated from requirement lists. In practice such
which predict future maintainability of systems a list may not always possible to follow, however
and which are related to modularity as well as size. requirements should be implemented; despite
IS metrics include information and system quality the difficulties in the implementation, there are
measures, as well as measures of productivity in works such as Berander and Svahnberg (2009)
terms of organisational outputs. In addition to discussing prioritization of reqirements through
these examples, the subject of metrics is one of measurement and use of metrics.
the richest literature in providing knowledge on However, the role of measurement in software
not only what to measure but also on software engineering is essential to understanding software
quality factors. For instance, a recent study by processes. Bourque and colleagues (2004) argue
Hansen, Jonasson and Neukirchen (2011) on that software engineering without measurement
product quality reveals that there is a strong evi- would be hard to interpret, they argue that without
dence that software defects can be explained by measurement, management would be difficult.
architectural quality; the researchers find such a However, other researchers (e.g. Purao & Vaish-
pattern by the help of analyzing product metrics navi, 2003; Berander & Jönsson, 2006)) point
and architecture metrics. out that there is a lack of consensus in software
There is an engineering principle that mea- measurements. In fact, many defined metrics
surement is a mandatory task (Bourque, Wolff, are not used (Berander & Jönsson, 2006). While
Dupuis, Sellami, & Abran, 2004). This principle metrics need to be goal or purpose oriented, a
would imply that measurement is necessary for goal must first be determined along with a way
monitoring IS evolution and replacement. There of measuring the degree of attainment of the goal,
have been many metrics proposed for different and both tasks may be subjective. For example,
purposes in the software engineering domain; counting the “lines of code”, as a metric, may
works such as Purao and Vaishnavi (2003) and serve as an indication of the complexity of a
Herlocker, Konstan, Terveen and Riedl (2004) system. However, line count is not a measure that
bring many of them together. Unfortunately, it is provides any insight into the activity of a system,
not straightforward to identify what metrics can as not every line of code has the same relevance
indicate the need to replace system components at run-time. Counting of lines of code has also
from amongst the factors that have been measured been claimed to be inappropriate for component
in software engineering. However, while the based systems, rather, complexity metrics for such
metrics do not generally address or describe the systems should be based on number of components
evolution of a system after it has been released and interactions among them (Gill & Balkishan,
into the market, the literature directly or indirectly 2008; Mahmood & Lai, 2006). Specifications of
identifies attributes, which affect quality, suitabil- the rules for the process of quantification may also
ity and performance of systems, and so is relevant be ambiguous (Berander & Jönsson, 2006). For
to the replacement decision. A metric is formally instance, implementations of the same metrics in
defined as “a quantitative measure of the degree to different software tools to support assessment of
which a system, component, or process possesses
145
software design have been found to give different butes such as how flexible or reliable a system is.
results (Lincke, Lundberg, & Löwe, 2008). Recalling software decay from previous sections, it
Other factors make measurement a challenge in can be measured through code decay, which is the
IT domains at various levels. For example, the ele- decrease of the quality of the code due to further
ments in the IEEE standard concerning evaluation modifications (Parnas, 1994), but degradation of
of productivity (IEEE, 1993) are broad-ranging systems need to be measured through observations
and dynamic, such as documents per person per of activities required to add new functionalities
hour or lines of code produced per day. To our or new hardware, or repairing faults. An indirect
knowledge, there are no generally accepted met- measure of system decay proposed in (Eick et al.,
rics for many qualities, such as class cohesion in 2001) involves the relative effort, time to complete,
software development, which address software and quality of modifications. To quantify the ef-
quality when new features are added (Mäkelä & fect of aging, for example, in operating system
Leppänen, 2007). It has also been argued in the resources, various metrics such as “estimated
case of software complexity that measures are not time to exhaustion” have been proposed in an
only subjective, but they do not satisfy a theory attempt to develop proactive techniques to limit
of measurement (Misra & Kilic, 2007) and this system crashes. The “time to exhaustion” metric
charge can be laid against many IT measures. suggested in (Garg et al., 1998) is based on slope
In the end, not every useful quality of software estimation of the resource usage on UNIX variants
can be objectively measured. Attributes have been and can be applied to different system resources
divided into categories of external and internal, including free memory and file table size. Another
according to whether they are indirectly or directly approach (L. Li et al., 2002) focuses on estimating
measurable, respectively (Vliet, 2008). External resource exhaustion through time series analysis,
metrics are most likely to be subjective. More where they test the system by generating artifi-
explicitly, internal attributes, such as defects, can cial workload to the web server and monitor the
be measured, for example by counting, while an resources for applications involving web servers.
external attribute, such as maintainability, can In relation to the system failures, there is reli-
be measured only with the help of internal attri- ability, which is the ability of a system or compo-
butes, which act as surrogates, such as measuring nent to perform its required functions under stated
modularity with a count of components. Another conditions for a specified period of time (IEEE,
internal metric is “lines of code”, which is simple to 1990). The literature on reliability measures is
implement by counting, whereas “effort” required not new. One of the earlier works on software
to produce those lines is difficult to determine reliability measures identifies mean time to a
and so is an external attribute, which can only potential failure and cumulative execution time
be approximated by surrogate measures such as as surrogate measures (Yamada, Osaki, & Tanio,
“development time”. 1992) proposes assessment techniques based on
There is a great deal of literature on measure- errors remaining after the testing phase, as well
ment of external attributes in software develop- as on failure and hazard rates. Those errors may
ment, software quality and software maintenance be captured later via user feedback. Errors and
(Vliet, 2008). Many of the metrics proposed in failures remain the main elements in measuring
the literature are directly or indirectly related to reliability (e.g. (IEEE, 2006)) although inclusion
structural connections between number of classes, of measures of software complexity, test effective-
number of times a class in invoked, and class size, ness and consideration of the complete operating
which are all internal attributes. These measures environment have been recommended to make
are used as surrogates in measuring external attri- reliability assessment more accurate (Xu, 2007).
146
There are many metrics developed to measure DEFINING THE FACTORS

aspects of software development. Modularity
and a system’s resulting flexibility are important As we have covered in the previous sections, the
for further maintenance. As with other software topic of change in IS crosses many disciplines,
metrics, the objectives of the software develop- and has been the subject of a great deal of research
ment project shape the criteria for their evaluation. as well as being a topic of practical importance.
In some environments speed of development is This section defines the factors came out from
critical, in which case a low number of classes the literature survey that we have reported so far.
may be desirable because the development team In this chapter, we define maintenance to be
is rushing to produce software within a tight time any activity that brings change to a software-based
frame. A related adverse by-product may be that component or system but does not replace it. Such
the production rate for lines of code per day is maintenance activities are described as part of
high because of the duplication of code elements. the life cycle. Life cycles of IS are captured in
Metrics connected with the productivity of software aging, software evolution and maturity
IS have been controversial (Du, Yang, & Wang, models as explained earlier.
2008; Hitt & Brynjoifsson, 1996). Such measures IS or its components’ replacement and main-
of productivity concentrate on efficiency and ef- tainenance are partially covered in the software
fectiveness of the systems. Overall performance aging literature. Aging refers to two different is-
measures include operational performances, sues in software: basically and more commonly,
especially system availability and throughput it refers to software crashes due to runtime ac-
(that is, producing the output within specified cumulation of problems such as data corruption;
time boundaries and the quality of the content secondly, age represents the duration of the IS or
of output). However, benefits of IT may not al- software component since initial deployment in
ways be easy to measure as they can be in forms the organisation. We use age in the latter sense.
such as customer service or convenience, which Over time, environmental conditions, including
may be intangible. Hitt (Hitt & Brynjoifsson, business requirements, change IS. Maintenance
1996) underlines the contribution of IT to busi- activities try to evolve IS so as to preserve
ness productivity but claims that there has been compatibility with current business and user de-
mismeasurements of output; for instance, where mands and with current technologies. Software
customer service or convenience are the output, evolution, occurs because an IS or component
there are difficulties as well as subjectivity that is reconfigured to fix bugs, to improve existing
may lead to mismeasurement. functionality, to meet new demands or to prevent
For the purposes of our research, the attributes potential problems that may happen later. In this
which are deemed to be of interest in the literature way, the system evolves. The organisation’s IS
on metrics are more important than the form of tend to become more complex with time (hence
the metric and measurements, and whether such harder to maintain) due to the number of changes.
measurements can be objectively quantified or Additionally, systems may become incompatible
not. The attributes that are measured provide with the current technology. Studies on software
general evaluation criteria for IS suitability to an maturity models highlight changes in a system
organisation. that drive it away from the original specifications
So far, we have reviewed selected works re- and which may make it unstable, that is, have an
ported in the literature to understand what can be increased risk of failure. At some point, the life
included in a change request in IS or in a part of it. cycle terminates with a decision to replace the
system or component.
147
This process of evolution has been monitored decision makers to consider when comparing
so commonly in the systems that it has been con- bespoke systems with off-the-shelf systems.
sidered as a phenomenon, as we have mentioned, These factors include the availability of people
earlier. Lehman and his colleagues have studied with technical skills to maintain the system, and
software systems for over 30 years to fortify and after sales support. Support from the vendor of
generalise their proposed laws of system evolu- off-the-shelf systems is not limited to providing
tion (an early example: (Meir Manny Lehman, some maintenance activities, but also do this with
1980)). According to the Lehman laws, a software a quality and on time.
system must evolve continuously for satisfaction Studies subsequent to Lehman’s laws on re-
within the user community; otherwise it becomes placement and maintenance show that predicting
less useful through time. These laws also say the the life expectancy of systems is not an easy task
change or decay process continues until the system (Glass, 1998). However, it is generally accepted
is judged by those responsible for deciding on that frequent maintenance activities is a warning
maintaining or replacing it, that replacement is the of a potential replacement.
more cost effective option. Works on component A topic related to the IS life cycle is reliability
based development (CBD) and service oriented research, which includes measurements to predict
architecture (SOA) show that systems become failure time and frequency of failures of software
more maintainable with modularity. As most IS systems or components. Those studies are basi-
adopt modular structures, partial upgrades are cally concerned with the errors that occur with
possible, that is, components may be replaced on-going changes in IS as well as hidden ones
instead of whole IS. Software complexity, which that appear after testing and before release of the
is related to the degree of dependency and inter- IS when they are installed in the organisations.
action of components, has been suggested as an Errors may lead to degradation in performance
issue when making change decisions concerning and hence reduce IS productivity. However, IS
a partial replacement and maintenance in such an productivity is a complex concept subject to much
environment. A less complex system delivering discussion in the literature. Studies touching this
the same functionality as a more complex one is issue underline the challenges in measuring ef-
preferable; e.g. (Meir Manny Lehman & Ramil, ficiency and effectiveness of IS.
2000) discusses problems such as serious delays The end-user has a prominent position in the
when it becomes difficult to understand where, employment of IS, hence cannot be ignored in
why and how the maintenance is needed and can change (replacement or maintenance) decisions.
be implemented. Apart from complexity ratings, Research into the effect of IS replacements on
metrics have been developed to measure a range the users, hence on the organisation, has been
of factors including performance, usability, ef- reported by Davis’ TAM (Davis et al., 1989; Da-
ficiency and reliability that help determine how vis, 1993), which shows that usefulness is linked
well the systems are performing from a techni- to intention and behaviour of use and therefore,
cal perspective. One of the aims of formulating feedback of the user about systems cannot be ig-
such metrics it to provide a quantitative evalua- nored. The technology acceptance literature also
tion of the benefits of keeping a system as it is, addresses the reaction of users and the organisa-
compared with enhancing or even replacing it. tions to changes in their IS. Acceptance of new
To develop the metrics, critical features that are technologies, new systems or new approaches
to be measured have to be identified and units is known to vary with the characteristics of the
for measurements decided. Additional research user or organisation, with so-called early adopters
(e.g. (Shailer, 2001)) has identified factors for leading the way; over time, others follow. User
148
satisfaction with existing systems is a key factor & Goel, 2007) and aging systems (Parnas, 1994;
in IS success models such as that of Delone and Eick et al., 2001; Heales, 2002), by connectivity/
Mclean (1992). IS success models are relevant to interoperability problems (Vicente et al., 1991;
the change decision because successful systems Tilley et al., 2002; Cayir & Basoglu, 2008) and
are arguably less likely to be changed. by complaints and negative feedback from users
While users’views are important, user feedback (Bharati & Chaudhury, 2006). Against this the
should be managed as their recommendations may decision to maintain the existing IS seems more
not be always “suitable” for the organisation or likely to be made when there is good technical
may not be possible to implement (Staples et al., support for the systems (Smolin, 2008), when IS
2002). However, studies such as Saarelainen et al., are compatible with other (sub)systems and with
(2006) conclude that organisations may not make changing technologies (Gill & Balkishan, 2008;
replacement decisions as rationally as traditional Bucchiarone et al., 2006), when they or their
decision making models depict: intuition may play components are popular (Venkatesh & Morris,
an important role. Hence, such decisions may not 2000; Hong & Lerch, 2002; DeLone & McLean,
always be understood or explained easily, as they 1992; Iivari, 2005) and provide specialised func-
may derive from a body of experience particular tionality (Haines, 2003; Du et al., 2008; Hitt &
to the person or personnel involved. Brynjoifsson, 1996).
On a technical level, replacement decisions
have to take into account the need for compat-
ibility and interoperability amongst existing sys- THE MODEL
tems. Within the context of IS, interoperability is
sometimes taken as synonym of compatibility (e.g. Drawing on the lessons learned from the above
(IEEE, 1990)). In this chapter we define compat- literature, the factors we have identified were
ibility as “the ability of two or more systems or reviewed in interviews with industry partici-
components to perform their required functions pants and their responses were used to guide the
while sharing the same hardware or software design of the following model of system replace-
environment” and interoperability as the ability ment. Once the model was compiled then it was
of IS to exchange information; however, we have confirmed by further interviews with industry
chosen to exclude interoperability in that sense participants. Following the early IT history of
in this work; this is not because we ignore its developing bespoke systems, this was followed
importance but we have chosen to further study by the growth of the multinational vendors with
interoperability as a future extension to this work. turnkey systems in a range of domains. Still, they
Drawing these various threads together we can may be specialised on specific operating systems
deduce that the decision to enact change in IS (or and hardware, and they have specific functions for
a part of it) – what we call the the change request business. An IS is preferred to fit into the work
(Jazayeri, 2005; Anderson & Felici, 2002; Rada environment by the help of its configurability.
& Finley, 2004; Staples et al., 2002) - is made by With all those specialisations, it is produced for
the organisation’s nominated decision makers; a specific user community with similar skills;
however, this literature review suggests we can for this reason market share would be accord-
isolate some items to explain the source of the ingly. Hence, the system specialisation defines
requests. We see themes of system replacement popularity of the information system. Over time,
being driven by increasing systems errors and fail- the specific attributes of the information system
ure (IEEE, 2006; Li et al., 2006; Qin et al., 2007), may not be enough for the ever changing business.
by high systems workloads (Li et al., 2002; Singh The performance, usability and efficiency of the
149
Figure 2. The suggested model
system are locked with the system specialisation; the user communities. The user feedback plays
in this sense the system specialisation limits the an important role in this whole process, because
productivity. An information system may produce it is one of the informants of the IS in all aspects.
errors and those errors may cause failures. The Thus, user feedback informs the change requests.
impact of the failures may differ from minor print Towards the end, the decision makers elimi-
out problems to whole system down. Thus, errors/ nate, modify and add change requests for the IS.
failures can limit productivity. Consecutively, a Those requests are shaped and/or filtered by the
change (potentially a drop) in productivity may policy, cost and (sometimes) intuition of the deci-
cause a request for maintenance activity. Although sion makers. Finally, a change request can cause
the maintenance activities need to be approved (corrective, adaptive, perfective or preventive)
by the organisation’s higher executive officers, maintenance activity. Although maintenance
available support facilities (through training, activities are changing systems specialisation for
documentation, 24/7 help desk and forums) of- “good”, they may increase risk of new errors and
fer possible maintenance activities to request failures by causing instability in the long term.
changes. Additionally and cautiously, popularity A representation of the suggested model can be
of a system (may) increase availability of support seen on Figure 2.
facilities in terms of exchange idea platforms of
150
The model is centred on the change requests dressed by maintenance changes, which do
made to maintain or modify part or all of an IS. not include complete replacement?
The change request is driven by combinations of • Are the errors and direct failures attribut-
three key factors: user feedback, changes to sup- able to the system’s behaviour impacting
port facilities and productivity impacts of the organisations productivity beyond the cost
system. However, each of these form distinct of replacement?
pathways to generate forces for change. The
other factors lying behind these are the extent of Finally, there is the question that if the system
system specialisation, that is, how the system is is to be replaced, are the resources available for
tailored to satisfy organisational requirements; such a move? This will be both in terms of rel-
popularity, the degree to which an IS or a com- evant funding requirements and also the human
ponent is liked or supported by its user commu- resources needed to support the change, which
nity; and the prevalence and severity of errors and is typically for at least the transition period more
failures in the systems. The threshold for system than those required for the maintenance phase. The
replacement will always be driven by change responses to the above questions may provide a
requests, whether as the result of an explicit for- decision path for resolving the replacement deci-
mal procedure or an implicit verbal complaint sion; however, the model is too young to provide
process. a normative guideline. A related issue in the re-
placement decision has been interoperability with
other systems. Whilst this is an area for further
IMPLICATIONS exploration, it’s impact on the replacement deci-
sion is already embedded in the factors contained
The model is a new and open to discussion in- without this model.
strument; however, it may serve the industry for
it itemizes factors behind change requests. As
seen from Figure 2, a critical tipping point can be CONCLUSION
from user feedback, changes to support facilities
or productivity impacts of the system directly and In this chapter, we have drawn on the literature to
can be identified by responses to the following examine the forces that drive the need for change
questions related to each factor: in information systems and in particular the crite-
ria that drive a system to the tipping point where
• Is support being withdrawn either ongoing maintenance changes to major revision
from external suppliers or internal IT or replacement of a system. We have discussed
infrastructure? the issues causing a decline in user perceptions
• Is support becoming more expensive such of a system over the passage of time and the fac-
that the return on investment is less than tors (system specialisation, popularity, avalilable
that from the cost of a replacement system? support facilities, productivity, errors and failures,
• Is the system impeding productivity such user feedback and maintenance) that may affect
that the costs or losses incurred by system those perceptions. There is a variety of metrics
performance are more than the cost of a re- that have been used to quantify these factors and
placement system? the others within organizations that may affect the
• Are problems and issues raised in user decision to replace a system. The model that has
feedback on the system unable to be ad- resulted brings all these factors together to provide
a framework for managing the replacement deci-
151
sion. It effectively provides the key questions that Boertien, N., Steen, M. W. A., & Jonkers, H.
must be addressed by management in deciding (2005). Evaluation of component-based devel-
about software replacement. It is still too early opment methods. In Krogstie, J., Halpin, T., &
to claim that this model is a guideline decision Siau, K. (Eds.), Information modeling methods
tree; however, it brings together factors that may and methodologies. Hershey, PA: Idea Group
address the question of “replace or maintain”, it Publishing.
promises a future in factoring company and/or
Bourque, P., Wolff, S., Dupuis, R., Sellami, A.,
product based decision tree.
& Abran, A. (2004). Lack of consensus on mea-
surement in software engineering: Investigation
of related ıssues. Paper presented at the 14th In-
REFERENCES
ternational Workshop on Software Measurement
Anderson, S., & Felici, M. (2002). Quantitative IWSM/MetriKon, Magdeburg, Germany.
aspects of requirements evolution. Paper presented Bucchiarone, A., Polini, A., Pelliccione, P., &
at the 26th Annual International Computer Soft- Tivoli, M. (2006). Towards an architectural ap-
ware and Applications Conference (COMPSAC). proach for the dynamic and automatic composition
Bao, Y., Sun, X., & Trivedi, K. S. (2005). A of software components. Paper presented at the
workload-based analysis of software aging, and ROSATEA ‘06: ISSTA 2006 Workshop on Role
rejuvenation. IEEE Transactions on Reliability, of Software Architecture for Testing and Analysis.
54(3), 541–548. doi:10.1109/TR.2005.853442 Cayir, S., & Basoglu, A. N. (2008, 27-31.
Berander, P., & Jönsson, P. (2006). A goal ques- July.2008). Information Technology interoper-
tion metric based approach for efficient measure- ability awareness: A taxonomy model based on
ment framework definition. Paper presented at information requirements and business needs.
the ISESE ‘06: 2006 ACM/IEEE İnternational Paper presented at the Portland International
Symposium on Empirical Software Engineering, Conference on Management of Engineering &
Rio de Janeiro, Brazil. Technology (PICMET), Cape Town, South Africa.
Berander, P., & Svahnberg, M. (2009). Evaluating Colosimo, M., Lucia, A. D., Scanniello, G., &
two ways of calculating priorities in requirements Tortora, G. (2009). Evaluating legacy system
hierarchies – An experiment on hierarchical cu- migration technologies through empirical stud-
mulative voting. Journal of Systems and Software, ies. Information and Software Technology, 51,
82, 836–850. doi:10.1016/j.jss.2008.11.841 433–447. doi:10.1016/j.infsof.2008.05.012
Bharati, P., & Chaudhury, A. (2006). Study- Davis, F. D. (1993). User acceptance of infor-
ing the current status of technology adoption. mation technology: System characteristics, user
Communications of the ACM, 49(10), 88–93. perceptions and behavioral impacts. Interna-
doi:10.1145/1164394.1164400 tional Journal of Man-Machine Studies, 38(3).
doi:10.1006/imms.1993.1022
Bhatt, P., Shroff, G., & Misra, A. K. (2004).
Dynamics of software maintenance. ACM SIG- Davis, F. D., Bagozzi, R. P., & Warshaw, P. R.
SOFT Software Engineering Notes, 29(5), 1–5. (1989). User acceptance of computer technology:
doi:10.1145/1022494.1022513 A comparison of two theoretical models. Man-
mnsc.35.8.982
152
Dekleva, S. M. (1992). The ınfluence of the In- Grudin, J. (2004, 6-10.November.2004). Return
formation Systems development approach and on ınvestment and organizational adoption. Paper
maintenance. Management Information Systems presented at the ACM conference on Computer
Quarterly, 16(3), 355–372. doi:10.2307/249533 supported cooperative work, Chicago, Illinois.
DeLone, W. H., & McLean, E. R. (1992). Informa- Gupta, Y. P., & Raghunathan, T. S. (1988). A pre-
tion Systems success: The quest for the dependent liminary model for İnformation System replace-
variable. Information Systems Research, 3(1), ment. Omega, 16(4), 289–296. doi:10.1016/0305-
60–95. doi:10.1287/isre.3.1.60 0483(88)90065-5
DeLone, W. H., & McLean, E. R. (2002). Infor- Haines, M. N. (2003). Customization, configura-
mation Systems success revisited. Paper presented tion, or modification? A taxonomy for Informa-
at the 35th Hawaii International Conference on tion System specialization. In Khosrow-Pour, M.
System Sciences, Hawaii. (Ed.), Information Technology and organizations:
Trends, ıssues, challenges and solutions (pp.
Du, J., Yang, X., & Wang, Z. (2008). Effective
899–900). Hershey, PA: IGI Publishing.
runtime scalability metric to measure produc-
tivity in high performance computing systems. Hansen, K. M., Jonasson, K., & Neukirchen, H.
Paper presented at the Conference on Computing (2011). An empirical study of software architec-
Frontiers, Ischia, Italy. tures’ effect on product quality. Journal of Systems
and Software, 84, 1233–1243. doi:10.1016/j.
Eick, S. G., Graves, T. L., Karr, A. F., Marron, J.
jss.2011.02.037
S., & Mockus, A. (2001). Does code decay? As-
sessing the evidence from change management Heales, J. (2002). A model of factors affecting an
data. IEEE Transactions on Software Engineering, information system’s change in state. Journal of
27(1). doi:10.1109/32.895984 Software Maintenance and Evolution: Research
and Practice, 14, 409–427. doi:10.1002/smr.271
Garg, S., Moorsel, A. v., Vaidyanathan, K., &
Trivedi, K. S. (1998, 4-7 November). A methodol- Herlocker, J. L., Konstan, J. A., Terveen, L. G.,
ogy for detection and estimation of software aging. & Riedl, J. T. (2004). Evaluating collaborative
Paper presented at the The Ninth International filtering recommender systems. ACM Trans-
Symposium on Software Reliability Engineering, actions on Information Systems, 22(1), 5–53.
Paderborn, Germany. doi:10.1145/963770.963772
Gill, N. S., & Balkishan. (2008). Dependency Hitt, L. M., & Brynjoifsson, E. (1996). Productiv-
and ınteraction oriented complexity metrics of ity, business profitability, and consumer surplus:
component-based systems. ACM SIGSOFT Soft- Three different measures of Information Tech-
ware Engineering Notes, 33(2). nology value. Management Information Systems
Quarterly, 20(2), 121–121. doi:10.2307/249475
Gill, N. S., & Tomar, P. (2010). Modified devel-
opment process of component-based software Hodges, M. S. (2007). Computers: Systems, terms
engineering. ACM SIGSOFT Software Engineer- and acronyms (17th ed.). SemCo.
ing Notes, 35(2).
Hong, S.-J., & Lerch, F. J. (2002). A laboratory
Glass, R. L. (1998). Editor’s corner: How long study of consumers’ preferences and purchas-
do information systems last? Journal of Systems ing behavior with regards to software compo-
and Software, 40(2), 91. doi:10.1016/S0164- nents. ACM SIGMIS Database, 33(3), 23–37.
1212(97)00002-2 doi:10.1145/569905.569909
153
Horch, J. W. (2003). Practical guide to software Lehman, M. M., & Ramil, J. F. (2000). Software
quality management (2nd ed.). Artech House. evolution in the age of component-based software
engineering. IEE Proceedings. Software, 147,
IEEE. (1990). IEEE standard computer diction-
249–255. doi:10.1049/ip-sen:20000922
ary: A compilation of IEEE standard computer
glossaries - 610. IEEE. Lehman, M. M., & Ramil, J. F. (2003). Soft-
ware evolution—Background, theory, practice.
IEEE. (1993). IEEE standard for software pro-
Information Processing Letters, 88, 33–44.
ductivity metrics. (IEEE Std 1045-1992).
doi:10.1016/S0020-0190(03)00382-X
IEEE. (2006). IEEE Std 982.1 - 2005 IEEE
Lehman, M. M., Ramil, J. F., Wernick, P. D., Perry,
standard dictionary of measures of the software
D. E., & Turski, W. M. (1997, 5-7 November). Met-
aspects of dependability. (Revision of IEEE Std
rics and laws of software evolution-The nineties
982.1-1988: IEEE).
view. Paper presented at the Fourth International
Iivari, J. (2005). An empirical test of the De- Software Metrics Symposium, Albuquerque, New
Lone-McLean model of İnformation System Mexico, USA.
success. ACM SIGMIS Database, 36(2), 8–27.
Li, L., Vaidyanathan, K., & Trivedi, K. S. (2002,
doi:10.1145/1066149.1066152
3-4.October.2002). An approach for estimation of
Jazayeri, M. (2005, 5-6.September. 2005). Spe- software aging in a Web server. Paper presented
cies evolve, individuals age. Paper presented at at the International Symposium on Empirical
the Eighth International Workshop on Principles Software Engineering.
of Software Evolution (IWPSE’05), Lisbon,
Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., &
Portugal.
Zhai, C. (2006). Have things changed now? Paper
Jing, Y., Jian, X., Xue-Long, Z., & Feng-Yu, L. presented at the 1st Workshop on Architectural
(2005, 10-12.October.2005). Modeling and avail- and System Support for İmproving Software
ability analysis of nested software rejuvenation Dependability, San Jose, California.
policy. Paper presented at the IEEE International
Lin, H.-F. (2008). Determinants of successful
Conference on Systems, Man and Cybernetics.
virtual communities: Contributions from system
Kyhlbäck, H., & Sutter, B. (2007). What does it characteristics and social factors. Information
take to replace an old functioning information & Management, 45, 522–527. doi:10.1016/j.
system with a new one? A case study. International im.2008.08.002
Journal of Medical Informatics, 76(Supplement 1),
Lincke, R., Lundberg, J., & Löwe, W. (2008).
S149–S158. doi:10.1016/j.ijmedinf.2006.05.047
Comparing software metrics tools. Paper pre-
Lehman, M. M. (1980, September). Programs, sented at the 2008 İnternational Symposium on
life cycles, and laws of software evolution. Paper Software Testing and Analysis.
presented at the IEEE.
Liu, Y., Trivedi, K. S., Ma, Y., Han, J. J., & Lev-
Lehman, M. M., & Belady, L. A. (1985). Program endel, H. (2002, 12-15.Nov.2002). Modeling and
evolution: Processes of software change. analysis of software rejuvenation in cable modem
termination systems. Paper presented at the 13th
International Symposium on Software Reliability
Engineering, ISSRE.
154
Mahmood, S., & Lai, R. (2006). A complex- Qin, F., Tucek, J., Zhou, Y., & Sundaresan, J.
ity measure for UML component-based system (2007). Rx: Treating bugs as allergies—A safe
specification. Software, Practice & Experience, method to survive software failures. ACM Trans-
38(2), 117–134. doi:10.1002/spe.769 actions on Computer Systems, 25(3).
Mäkelä, S., & Leppänen, V. (2007). A software Rada, R., & Finley, S. (2004). The aging of a
metric for coherence of class roles in Java pro- clinical İnformation System. Journal of Biomedi-
grams. Paper presented at the 5th İnternational cal Informatics, 37(5), 319–324. doi:10.1016/j.
Symposium on Principles and Practice of Pro- jbi.2004.06.003
gramming in Java, Lisboa, Portugal.
Rogers, E. M. (1962). Diffusion of ınnovations.
Markovic, B., & Wood, J. (2004). User satisfaction Free Press.
with CBA computer LABS. Journal of Computing
Roses, L. K., Hoppen, N., & Henrique, J. L.
Sciences in Colleges, 19(4), 232–239.
(2009). Management of perceptions of İnformation
Marks, E. A., & Bell, M. (2006). Service-oriented Technology service quality. Journal of Busi-
architecture: A planning and ımplementation ness Research, 62(9), 876–882. doi:10.1016/j.
guide for business and technology. John Wiley jbusres.2008.10.005
& Sons.
Saarelainen, M.-M., Ahonen, J. J., Lintinen, H.,
Misra, S., & Kilic, H. (2007). Measurement theory Koskinen, J., Kankaanpää, I., Sivula, H., et al.
and validation criteria for software complexity (2006). Software modernization and replacement
measures. ACM SIGSOFT Software Engineering decision making in ındustry: A qualitative study.
Notes, 32(2), 1–3. doi:10.1145/1234741.1234760 Paper presented at the 10th International Confer-
ence on Evaluation and Assessment in Software
Okamura, H., Miyahara, S., & Dohi, T. (2005,
Engineering (EASE), UK.
4-8 April). Effect of preventive rejuvenation in
communication network system with burst arrival. Salmeron, J. L., & Lopez, C. (2010). A multicriteria
Paper presented at the Autonomous Decentralized approach for risks assessment in ERP maintenance.
Systems, ISADS. Journal of Systems and Software, 83, 1941–1953.
doi:10.1016/j.jss.2010.05.073
Parnas, D. L. (1994, 16-21 May). Software ag-
ing. Paper presented at the 16th International Seddon, P. B. (1997). A respecification and ex-
Conference on Software Engineering (ICSE-16), tension of the DeLone and McLean model of IS
Sorrento, Italy. success. Information Systems Research, 8(3),
240–253. doi:10.1287/isre.8.3.240
Peng, Y., Li, F., & Mili, A. (2007). Modeling
the evolution of operating systems: An emprical Shailer, D. M. (2001). Project manager’s toolkit:
study. Journal of Systems and Software, 80, 1–15. Practical checklists for systems development.
doi:10.1016/j.jss.2006.03.049 Butterworth-Heinemann.
Purao, S., & Vaishnavi, V. (2003). Product metrics Sharma, H., & Jansen, B. J. (2005). Automated
for object-oriented systems. ACM Computing Sur- evaluation of search engine performance via
veys, 35(2), 191–221. doi:10.1145/857076.857090 implicit user feedback. Paper presented at the 8th
Annual İnternational ACM SIGIR Conference
on Research and Development in İnformation
Retrieval, Salvador.
155
Shye, A., Ozisikyilmaz, B., Mallik, A., Memik, Vaidyanathan, K., & Trivedi, K. S. (2005). A com-
G., Dinda, P. A., Dick, R. P., et al. (2008). Learn- prehensive model for software rejuvenation. IEEE
ing and leveraging the relationship between Transactions on Dependable and Secure Comput-
architecture-level measurements and ındividual ing, 2(2), 124–137. doi:10.1109/TDSC.2005.15
user satisfaction. Paper presented at the ISCA
Veerman, N. (2006). Automated mass mainte-
‘08: 35th International Symposium on Computer
nance of a software portfolio. Science of Com-
Architecture.
puter Programming, 62, 287–317. doi:10.1016/j.
Singh, Y., & Goel, B. (2007). A step towards scico.2006.04.006
software preventive maintenance. ACM SIGSOFT
Venkatesh, V., & Morris, M. G. (2000). Why
Software Engineering Notes, 32(4).
don’t men ever stop to ask for directions? Gen-
Smolin, R. (2008). From program to product: der, social ınfluence, and their role in technology
Turning your code into a saleable product. Apress. acceptance and usage behavior. Management
Soares, P. G. (1992). On remote procedure call.
doi:10.2307/3250981
Paper presented at the Conference of the Centre
for Advanced Studies on Collaborative research, Vicente, B. d., Alonso, A., & Amador, J. (1991).
Toronto, Ontario, Canada. Dynamic software replacement model and its Ada
implementation. Paper presented at the Annual
Sommerville, I. (2004). Software engineering (7th
International Conference on Ada, TRI-Ada ‘91:
ed.). Pearson Addison-Wesley.
Today’s Accomplishments; Tomorrow’s Expecta-
Staples, D. S., Wong, I., & Seddon, P. B. (2002). tions, San Jose, California.
Having expectations of İnformation Systems
Vliet, H. v. (2008). Software engineering: Prin-
benefits that match received benefits: Does İT
ciples and practice (3rd ed.). John Wiley&Sons.
really matter? Information & Management, 40,
115–131. doi:10.1016/S0378-7206(01)00138-0 White, M. (2008). Permeating innovative technol-
ogies throughout the institution. Paper presented
Szyperski, C., Gruntz, D., & Murer, S. (2002).
at the 36th Annual ACM SIGUCCS Conference
Component software: Beyond object-oriented
on User Services Conference, Portland, OR.
programming (2nd ed.). Addison-Wesley.
Xu, S. (2007, 17-19 December). An accurate
Tauritz, D. R., Kok, J. N., & Sprinkhuizen-Kuyper,
model of software reliability. Paper presented at
I. G. (2000). Adaptive ınformation filtering
the 13th IEEE International Symposium on Pacific
using evolutionary computation. Information
Rim Dependable Computing.
Sciences, 122(2-4), 121–140. doi:10.1016/S0020-
0255(99)00123-1 Yamada, S., Osaki, S., & Tanio, Y. (1992). Software
reliability measurement and assessment methods
Tilley, S., Gerdes, J., Hamilton, T., Huang, S., Mül-
during operation phase and their comparisons.
ler, H., & Wong, K. (2002). Adoption challenges
Systems and Computers in Japan, 23(7), 23–34.
in migrating to Web services. Paper presented at
doi:10.1002/scj.4690230703
the Fourth International Workshop on Web Site
Evolution. Yourdon, E. (1989). Modern structured analysis.
Englewood Cliffs, NJ: Prentice-Hall.
Trotter, P. (1976). User feedback and how to get it.
Paper presented at the 4th Annual ACM SIGUCCS
Conference on User Services, Tucson.
156
KEY TERMS AND DEFINITIONS Popularity: Defines how much the IS is ad-
opted in the industry as well as in the organisation
Avaliable Support Facilities: Support pro- that it is employed
vided by the developer company and user com- Productivity: Rate of producing output hence
munity such as forums on the web. performance related usability
Change Request: A request for maintenance Replacement and Maintenance Decision:
or replacement of IS/component to the decision The action of determining whether the IS/com-
makers ponent should be replaced or further maintained.
Error and Failure: Any defect in the system User Feedback: Any information associated
and malfunctionality, respectively with the IS gathered from the user
Information Systems Specialisation: The
fit of an IS with the specific requirements of the
organisation
157
158
Chapter 9
A Steady-State Framework for
Integrated Business Change
and Information Systems
Development and Maintenance
Simon McGinnes
Trinity College Dublin, Ireland
ABSTRACT
Success models often treat Information Systems (IS) as static. Yet most IS evolve continuously, and most
development effort occurs during the so-called maintenance phase. For an IS to succeed, its evolutionary
process must also remain successful. Unfortunately many IS projects fail, particularly when outsourced.
This chapter argues that the practice of managing IS work in project form may itself be implicated in
IS failure. The project model is critically examined, identifying mismatches with the reality of IS work
as a component of business change. The results suggest that merely trying harder to make projects suc-
ceed may be ineffective. An alternative framework for “steady state” development is proposed, which
characterises IS work as evolutionary and inseparable from its context of business change, providing
a blueprint for IS development without the need for projects, and offering improved chances of success
when “big bang” project management would otherwise be the only option.
DOI: 10.4018/978-1-4666-0170-3.ch009
A Steady-State Framework for Integrated Business Change and IS Development and Maintenance
INTRODUCTION multiple systems and/or projects, and objectives

often shift as work proceeds. These factors suggest
This chapter examines factors that contribute to that the project model may be less appropriate
success in information systems (IS) work. Suc- for IS development. The chapter is structured
cess models for IS focus on the use and benefits as follows. After some historical background on
of IS, often treating them as static entities (Delone competing views of IS development, the nature
& McLean, 2003; Salmerona & Smarandacheb, of contemporary IS work is critically analysed to
2006; Seddon & Kiew, 2007). This accords with the assess its fit with project structures. The chapter
traditional view of IS development as episodic and builds on this analysis to present a framework
project-based. According to that view, IS emerge of principles for “steady-state” IS development
through a relatively brief development process without the use of projects. The chapter concludes
and remain essentially unchanging thereafter. with suggestions for further research.
Yet, in reality, contemporary IS are dynamic; they
constitute an important component of business
change and evolve over long periods. In fact, the BACKGROUND: COMPETING
large majority of effort expended on IS develop- VIEWS OF IS DEVELOPMENT
ment occurs during the so-called maintenance
phase, typically spanning many years (Grubb & The engineering project is today the dominant
Takang, 2003). Therefore, for an IS to provide structural metaphor for thinking about IS devel-
benefits, the continuous process through which opment. Metaphors shape perception, particularly
it comes into being and evolves must also remain in IT which has re-purposed many existing ideas
successful. This process is often constituted as including record, file and even computer (Ezh-
a series of interlinked projects. Unfortunately, kova, 2005; Light, 1999). “Good” engineering
many IS projects fail, despite substantial research is a scientific and formal process rather than a
efforts to determine the causes of failure (Janssen fuzzy, intuitive one, and its product is technology.
& Klievink, 2010; Keider, 1984; Warne & Hart, The engineering project metaphor indicates that
1996). Outsourced IS developments are especially IS requirements must be specified clearly before
at risk (Iacovou & Nakatsu, 2008). construction commences, and design must proceed
The cost of IS project failure is high. Advice on according to established principles. IS develop-
how to avoid it often points to project management ment is characterised as bursts of focused activity
inadequacies. But this research explores the propo- (IT projects) in which technology is manipulated
sition that the use of project management may itself (system design) in a planned manner (project man-
be implicated. Project-based work structures are agement) by experts (developers) on behalf of non-
often applied routinely, with little consideration of experts (end users) using analytical techniques (IS
alternatives. Projects can produce quick results; a methodologies) to derive solutions (IS) to known
focus on limited goals allows the team to ignore problems (requirements). This engineering project
wider complicating factors. When there is little view of IS development falls into a wider set of
organisation in work, packaging tasks into project approaches to problem-solving characterised by
form can provide helpful structure. But it can be the top-down application of external expertise and
difficult to fit work into a project when the work is analytical thinking; Operational Research (OR)
continuous, or tasks have complex dependencies, is one example. Its goal is to build an IT system;
or the goals are hazy or changing. This is the case any associated business change is considered a
in IS, most of which evolve continually and are separate concern, to be managed independently.
integrated with other systems. Projects impact on Messy real-life issues, such as poor communica-
159
tion, fuzzy requirements and shifting priorities, are program specifications. Work proceeds in short
treated as aberrations; despite the high incidence iterations; examples include sprints (Sutherland,
of these problems, it is assumed that they can be 2001) and timeboxes (Stapleton, 2002). Agile
avoided in well-run projects (Liu et al., 2010). methods do not fit the engineering project model
Although dominant, the engineering view of well and are less amenable to traditional project
IS development has not gone unchallenged. One management. At the extreme, each agile project
alternative sees IS development not as mechanis- collapses into a uniform period of continuous,
tic engineering or manufacturing, but as ad hoc, iterative development; the project is merely a
political and subject to conflict; IT professionals container for this process. The project manager
are “change agents” (Ciborra, 1993). In Socio- cannot rely on conventional “objective” measures
Technical Design, IS development is viewed as of cost, time, and conformance to specification.
participative and social issues are given explicit Instead, progress and quality may be measured
attention, helping to moderate the technological only through subject assessment of client satisfac-
imperative (Mumford & Henshall, 1978). In Soft tion and business utility.
Systems Methodology the analyst avoids dwell- Despite widespread project failure, and a
ing on technological concerns, applying “systems variety of alternative views of IS work, the
thinking” in a holistic approach to organisational convention of packaging IS work into projects
problem-solving (Jacobs, 2004). remains part of the mental furniture and is rarely
Many such alternative perspectives charac- questioned. Notions of agility and iterative work
terise IS development as an evolutionary, col- have found support, but are typically used in the
laborative process. Arguments for this view often context of projects rather than instead of projects.
focus on the uncertainty of IS development and its For example, organisations wishing to outsource
intertwined relationship with business concerns, IT tasks are urged to parcel work out in discrete
which set it apart from other procurable services packages, with requirements defined explicitly
and artefacts. Iterative, participative approaches in advance and with fixed-price, fixed-duration
are said to be more effective in complex envi- contracts (Ambler & Constantine, 2000; Will-
ronments than traditional “big bang” projects. cocks et al., 2004). This advice hardly seems to
Agile methods exemplify this perspective; they represent a prescription for flexibility or iterative,
emphasise the dynamic nature of IS develop- participatory work.
ment, aim to reduce bureaucracy, and play down
planning and requirements analysis in favour of
adaptability (Mutton & Golbeck, 2003). The goal ANALYSIS
is to help the developer respond to the client’s
developing knowledge of needs and possibilities Below we analyse the characteristics of IS work
(Erickson, Lyytinen, & Siau, 2005; Highsmith & from various perspectives, which have been chosen
Cockburn, 2001). to highlight key differences between project-based
Agile methods assume that objectives, require- and non-project-based approaches.
ments and priorities will shift. No clear distinction
is drawn between the different development tasks a. Underlying management philosophy (control
of requirements, design and programming, since vs. flexibility);
the developer may engage in all of these activities b. Scope of work (IS vs. business change);
as the need arises. In agile development there is c. Allocation of tasks (outsourcing vs.
less of the documentation traditionally used to con- in-sourcing);
trol projects such as requirements statements and d. Form of contract (protection vs. partnership);
160
e. Characterisation of work (development vs. expense of planning (Fowler & Highsmith, 2001;
maintenance); Lindvall et al., 2002). In theory, if a project begins
f. Phasing of delivery (evolutionary vs. revo- with only sketchy requirements, a high-quality
lutionary change); result will be obtained provided there is flexibility
g. Relationships between projects (standalone to change tack as necessary during the project.
vs. overlapping projects). Conventional project management sees shifting
requirements and priorities as evidence of project
Underlying Management Philosophy failure. But requirements and priorities naturally
(Control vs. Flexibility) shift, as experienced IS developers know (Lormans
et al., 2004). In this sense there is a fundamental
In IS development work, it is not always obvious mismatch between the conventional teachings of
how to balance control with flexibility (Boehm & project management and the reality of IS work.
Turner, 2005). Conventional project management According to the agile view, if requirements
values control, but many conventionally-managed cannot be known accurately in advance, there is
IS projects fail to deliver expected benefits (Jones, little hope of accurately predicting the time and
2003). Agile methods promise better results money needed to satisfy them (Hickey, Davis,
but demand flexibility and remove many of the & Kaiser, 2003). Instead, developer and client
conventional “levers” for controlling projects should collaborate to discover requirements and
(Erickson, et al., 2005). develop suitable designs, working in partnership
Quality is an important driver in this trade-off. to maximise business utility and thereby obtain
In engineering projects, quality is equated with value for money (Abrahamsson et al., 2002). Qual-
conformance to specification and seen as the ity is then equated with subjective measures such
outcome of a formalised, documented “quality as business utility and client satisfaction. Agile
process”. Objectives and priorities must be set methods thus naturally complement IS success
unambiguously. “Scientific” laws and rules are ap- models, which use similar measures (Delone &
plied; technical skills and measurement are valued McLean, 2003; Salmerona & Smarandacheb,
and production is held to well-defined tolerances. 2006; Seddon & Kiew, 2007).
Perceived flaws are blamed on issues like poor
specification and lack of testing. The “obvious” Suggestion: Be flexible over scope, cost and
solution is an emphasis on top-down control, duration, as far as possible; minimise advance
procedures, standards and quality assurance, with planning and top-down control. Avoid advance
inspections (formal reviews) and sampling (formal requirements specification; expect requirements
testing) to ensure conformance to documented and priorities to evolve as work proceeds, and
requirements (Mellor, Clark, & Futagami, 2003). respond accordingly.
One result is greater bureaucracy and increasing
cost. Although it is widely considered self-evident Scope of Work (Technology
that greater control and conformance will lead vs. Business Change)
to better-quality IS, it is not clear that evidence
supports this view (Agrawal, Jain, & Jain, 2010; Business change and IS work are closely related:
Dybå & Dingsøyr, 2008). “the final outcome of a system project is in fact
Agile methods look at quality differently. They a real organisational change” (Ciborra, 1993).
assume that IS development is unpredictable and Business change is itself a prime cause of IS work,
that quality is best attained through flexibility and it is said that organisations must transform their
over requirements, priorities and designs, at the processes to exploit the possibilities of IT. If IS are
161
comprised of people, technology and processes, tweaking”—making selective changes to systems

then organisations must be prepared to adapt all in the hope that operations will be affected in a
of these aspects in order to realise benefits. This desired way. But it is unrealistic for management
may include change to work practices, alteration to to assume that new systems will somehow cause
or automation of processes, installing equipment, the desired changes to occur without associated
hiring and transferring staff, training, and so on. business change efforts.
The intimate connection between IS and busi- If IS work and business change are two sides
ness change is one reason why IS projects fail. of the same coin, a failure to reflect this connec-
Business change means changing people and tion in the organisation of work is likely to cause
what they do, and this is rarely easy. Individuals problems. The naive IT professional may think that
may be reluctant to make the necessary changes IS can be introduced without significantly affect-
or even to accept that change is necessary. Alter- ing the organisation, but in reality it is paradoxical
ing established practices can be painful and may to define a project as only the IT-related aspects
involve loss of employment, threats to personal of organisational change; “there is no such thing
power and destabilising upheaval. Change im- as an IT project in isolation from its business
posed in a top-down way, by management or IT change programme” (West-Knights, 2000). The
experts, rarely succeeds without at least teething idea of an IT-only project is dangerous; it tends
troubles. Sometimes it is the affected end users who to institutionalise compartmentalised thinking
see most clearly the pitfalls in proposed change and reinforces the illusion that IS and organisa-
programmes. A case study of a large outsourced tion are distinct. If this thinking is translated into
government IT project illustrates typical problems action and technology is introduced without due
of this nature (Khalfan, 2003). Top management consideration for organisational implications, the
were reluctant to become involved, and the end results are likely to be poor.
users who would be most affected by the new IS As an example, one common mistake when
were excluded from contract negotiations with the planning for new systems is to simplify business
vendor, a multinational IT firm. There was a lack activity into a process flowchart, and then to design
of support for the process restructuring necessary systems to support the idealised, logical process
for the new IS to work properly; management and rather than the messy real-life process, “misrep-
end users resisted change to jobs and procedures. resenting collective, interactive work as a linear,
After three years, the vendor withdrew from the clear cut, and predictable workflow” (Ash, Berg,
contract, and the project ultimately failed. & Coiera, 2004). It is easier to design a system
This experience is not atypical in the public for a simplified business process than a real one.
sector; many other instances of project failure But, when implemented, the system will not match
have been reported (Kristensen, 2001). After reality and will be liable to fail. The problem here
a series of high-profile IT project failures, the is compartmentalised thinking, brought about by
UK National Audit Office concluded that many artificial separation of IT work from business
problems in IT projects “are not about the tech- change. Nevertheless, much IS work continues
nology itself, but about the determination of the to be managed separately from business change
business processes that it is designed to serve” initiatives. To have the best chance of succeeding,
(NAO, 2004). Although private sector organisa- IS projects ought to be constituted as “business
tions are typically reluctant to publicise failure, change plus IT” projects, with IT work and busi-
they are thought to experience similar problems ness change managed as a single process rather
(Loucopulos et al., 2004). IS implementation is than distinct pursuits (Reich & Benbasat, 2000).
often used as a kind of top-down “organisational
162
Suggestion: Avoid IT-only projects; manage IS (NAO, 2004). This procurement method creates a
work as an integral part of business change and risk that collaboration will become peripheral and
business process evolution. off-hand, rather than a structural and central to the
work. The vendor’s profit motive and the fixed
Allocation of Tasks (Outsourcing duration of the contract encourage the vendor to
vs. In-Sourcing) engage with the client only as much as needed to
meet the project’s explicit goals. Important user
The same problem occurs in outsourcing practice. interactions during requirements specification,
It is usual for outsourced IS projects to be framed prototyping and design may be rushed or even
explicitly as IT work, excluding business change. omitted (Symon & Clegg, 2005). The vendor may
Vendors submit tenders to perform the IT tasks, rely on written communication in the mistaken
and the associated business change is implicitly belief that it can capture the nuances of require-
left to the client organisation. The “IS” part, which ments, resulting in misunderstandings.
could normally be expected to connect organisa- In most IS development, close collaboration
tion with technology, can therefore fall through between developer and end user important. But
the cracks. Obviously, business change cannot be the conventional outsourcing model interferes
outsourced; paying a third party to change on one’s with this relationship. It assumes that IT projects
behalf is nonsensical, unless the third party also are conducted by experts (developers) for non-
takes over responsibility for the relevant business experts (end users)—not by developers and end
tasks. However, partitioning projects between users acting in close collaboration. In outsourc-
“distinct” in-house business and outsourced IT ing contracts, these roles can become fixed by
spheres is also unrealistic, since it encourages the legal provisions, further limiting opportunities
outsourcing vendor to proceed with inadequate for collaboration. In extreme cases, communica-
business input and creates ambiguity about who tion between the two groups must be funnelled
is responsible for all-important business change. through single individuals on each side, seriously
Apart from the constraints imposed by geographi- hindering agile development (McGinnes, 2011).
cal separation, there can be a significant gulf in IS It is important when IT work is outsourced to
outsourcing between client and vendor in culture, ensure that developers and end users collaborate ef-
background, goals and even language, making it fectively, despite the barriers imposed by contracts,
difficult to achieve success (McGinnes, 2011). geographical separation, time zone differences,
Outsourcing offers the promise of reduced cost, language and culture. Full outsourcing makes this
but the evidence is that it is risky and subject to difficult, but intermediate degrees exist between
hidden costs, offering less flexibility than in-house complete in-sourcing and full outsourcing, some
development and requiring very careful manage- of which provide more scope for collaboration.
ment (Carmel & Nicholson, 2005; Willcocks & One is the practice of contracting-in: recruiting
Lacity, 2006). individual external experts alongside business
Outsourced projects are generally conducted end users into an in-house group, so that both
according to established procurement practice, in IT and business knowledge are encapsulated in
which vendors tender for work and are selected a single team. This arrangement can also help to
on a per-project basis. This allows services to be circumvent other negative side-effects of full-
purchased in productised form. Long engagements blown outsourcing, such as the client’s loss of
may be handled as a series of projects, for which capability and “winner’s curse” (Jørgensen &
the vendor may be required to tender repeatedly, Grimstad, 2005).
perhaps in the context of a framework contract
163
Suggestion: Avoid contracting-out whole proj- ner’s curse) and then finds it necessary to adopt a
ects; instead, contract-in technical expertise to defensive attitude with the client to avoid losing
form joint in-house teams, so that IS work and their profit margin altogether (Jørgensen & Grim-
business change can be achieved collectively by stad, 2005). It is easy to see how dissatisfaction
a unified group. can result for both sides. Nevertheless, both clients
and vendors often overlook their own contribu-
Form of Contract (Protection tions to negative outcomes and blame the other
vs. Partnership) party. Legal remedies may be invoked, but they
are often unsatisfactory once a project has failed.
When outsourced IS development is procured by Legal action is slow, and no amount of financial
fixed-price (FP) tender, requirements are normally compensation can retrospectively provide the
fixed at the time the contract is agreed. This may system that was needed, at the time it was needed.
give clients a sense of security, particularly if there Mainstream thinking about IS outsourcing is
is a belief that risk has been offloaded. However, dominated by notions of formality and control;
the protection afforded by FP contracts is far from the unintended effects of procurement methods
certain, for several reasons (Avison, Gregor, & and contract forms are rarely considered (Wang,
Wilson, 2006). At root there is a mismatch between 2002; Willcocks & Lacity, 2006). However, the
the legal demand for detailed requirements and the IS outsourcing literature has begun to address is-
practical need for flexibility. Agile methods treat sues of flexibility, trust and partnership (Kinnula
requirements only as a starting point; end user and et al., 2007; Sakthivel, 2007; Sauer, 2006). Under
developer collaborate in an atmosphere of mutual the shared risk model, emphasis is placed on part-
trust. But traditional project management relies on nership which, like marriage, requires “diligent
well-defined requirements so that projects can be work and commitment from both parties” and
planned in detail, even though they are unlikely to “the ability to change and grow” (Gong, Tate,
be well-understood before the work commences. & Alborz, 2007). In outsourcing, true partnership
The result is premature decision-making. Once a requires genuine alignment of goals, something
contract is signed, the vendor is under pressure that FP contracts signally fail to achieve (Lac-
to resist change to requirements, because of the ity, 2008). Alternative contract forms can give
potential impact on costs and timescales (Jamie- a better trade-off between legal protection and
son, Vinsen, & Callender, 2005). Although flex- partnership; they include time and materials,
ibility is needed to ensure success as the project cost-plus, optional scope and shared cost/reward
progresses, it is impossible for the vendor to be contracts (Beck & Cleal, 1999; Gopal et al., 2003;
flexible under a contract that stipulates precise Lichtenstein, 2004; Verhoef, 2005). Under these
adherence to stated requirements and timescales. arrangements the focus can shift from a reliance
These opposing views are usually irreconcil- on legal obligations for the duration of a contract,
able in FP procurement. Perverse outcomes can towards trust, flexibility and the quality of the
result, as in the case where the vendor strives continuing vendor-client relationship (Brady &
to achieve out-of-date project goals or to meet Davies, 2010).
superseded requirements in order to be paid. The
result is a lose-lose scenario. The client hopes that Suggestion: Ensure that vendor-client agreements
a tight contract will maximise value and transfer emphasise trust and partnership, rather than legal
risk, but is disappointed when the vendor cannot protection, and the quality of the ongoing rela-
deliver as promised. The vendor feels obliged to tionship rather than obligations for the duration
sign up to terms that may be unachievable (win- of a project. Replace fixed-price/duration/scope
164
contracts with flexible contracts (e.g. time and naturally lends itself to a continuous or iterative
materials, cost-plus, shared risk/reward). process (Van Deursen, Visser, & Warmer, 2007)
(Bhatt, Shroff, & Misra, 2006). The eagerness
Characterisation of Work to cast IS work as inherently project-based may
(Development vs. Maintenance) stem from the fact that work packaged into project
form can be easier to sell to clients, management
IS work has historically been thought of in two and team members. Projects are inherently aspira-
ostensibly separate areas of concern: the develop- tional and optimistic, while maintenance can have
ment of new IS and the maintenance of existing negative connotations: clients express frustration
IS. Typical IS development tasks include systems over maintenance costs, management blame poor
analysis and design, programming and imple- progress on maintenance needs and developers
mentation. Maintenance tasks include bug-fixing, regard maintenance as dull work. Nevertheless,
enhancement and support. This dual conception if the bulk of IS development occurs during the
of IS work may have been an accurate view when continuous process of maintenance, it makes
systems tended to be relatively limited, static and sense to acknowledge this reality by characteris-
discrete. However, IS are now typically large and ing development as a continuous process rather
overlapping, and both types of work occur in both than as a series of projects.
contexts, as do specialist tasks such as database ad-
ministration, quality assurance and infrastructure Suggestion: Characterise and manage IS devel-
management. Therefore the distinction between opment and maintenance as a single continuous
development and maintenance is probably not process.
very relevant to contemporary IS work.
Although maintenance is sometimes consid- Phasing of Delivery (Evolutionary
ered a low-status or less important activity, it vs. Revolutionary Change)
accounts for the majority (at least 75%) of effort
and value creation in IS development (Banker & IS projects tend to compress business change into
Slaughter, 1997; Grubb & Takang, 2003). Main- short durations, necessitating large and discontinu-
tenance involves work on a system that is already ous adjustment at implementation time. This is
operating within a specific environment. Many IS often problematic and is itself a prime cause of
have complex relationships with systems, projects project failure. After substantive analysis of fail-
and the organisation, and during maintenance the ing, expensive government IS projects, the UK
developer cannot neglect these relationships by National Audit Office advised government depart-
concentrating on limited or short-term objectives. ments to “mandate no big-bang implementations
In maintenance, wider and longer-term conse- … unless approved” and to announce no govern-
quences for the organisation must be taken into ment initiatives that depend on new IS “before
account. This is very different from the situation analysis of risks and implementation options has
in new development projects, which traditionally been undertaken” (NAO, 2004).
have not had to contend with pre-existing systems A more pernicious reason for project failure is
and often deliberately narrow the focus to consider the emergence of unintended outcomes. Projects
only explicitly-stated, immediate goals (Grubb & are invariably planned with expectation of busi-
Takang, 2003). ness benefit, but unanticipated effects can occur.
IS development has historically been charac- This is not simply due to inept design and by
terised as project-based activity. Maintenance can no means peculiar to IS; in many spheres, well-
also be managed as a series of projects, but it more intentioned interventions can result in perverse
165
consequences (Boudon, 1982). For example, that create the need for large and discontinuous
building new roads can worsen traffic (Arnott & business change.
Small, 1994); introducing better drugs can make
disease harder to treat (Cohen, 1992); legislation Suggestion: Use a process of managed parallel
designed to reduce poverty can create “poverty evolution for business processes and IS; deliver
traps” (Villanueva & Caston, 1981). Similarly, technology and business changes in small incre-
new IS can harm organisations, despite the best ments; review outcomes and priorities progres-
of intentions (Beynon-Davies, 1999). sively.
Systems theory sheds some light on the
mechanisms behind perverse or unintended Relationships between
consequences. Well-intentioned and apparently Projects (Standalone vs.
sensible change is liable to backfire because Overlapping Projects)
of second-degree effects such as feedback and
self-correction (Binbasioglu & Winston, 2004; Classically, IS project management has been
Waddington, 1977). Organisations are complex, concerned primarily with the new development
dynamic systems and changing them is rarely of new, standalone systems. This was probably
straightforward, especially when IS are involved. appropriate for early computer-based IS, which
In IS, some specific types of behaviour often were relatively limited in scope; most projects
result in unwanted outcomes. For example, end were discrete, green-field developments in isolated
users may continue to work in old ways; they may application areas. Projects typically did not im-
subvert new systems or use them in inappropriate pact much on one another and interfaces between
ways to circumvent perceived shortcomings. New, systems were infrequently required. Most projects
unexpected patterns of behaviour may arise, with could proceed without too much concern for other
unpredictable consequences. projects or systems (see Figure 1).
Despite the promise of radical or rapid or- The idea of a project was useful in early IS
ganisational transformation, more incremental development precisely because it was a good fit
business change can offer good results. Effective with reality. But the contemporary situation is
process change is often best achieved using a different. Today’s systems are larger and more
learning approach, especially where IT is involved integrated. The growth and increasing integration
(Hughes & Golden, 2004). The field of Change of IS mean that most IS projects now involve
Management evolved specifically to study and modification to a portfolio of systems which form
help organisations deal with this process (Burnes, part of a complex IS infrastructure. As a result
2010). Total Quality Management promotes the many projects affect the same systems concur-
idea of incremental change, specifically because rently, or in sequence, or both (see Figure 2). This
it is an easier process for participants to con- makes the project model less realistic and more
trol. In the context of IS work, the evidence is of an abstraction: an artificial view of how work
that evolutionary change is generally safer and is actually done.
more effective than revolutionary change, since There is a relationship between system scale
it allows organisations to test new approaches and project dependencies. As systems grow and
with less risk. More frequent and less ambitious proliferate, their dependencies increase in a non-
implementations, managed in tandem with evolu- linear fashion, and so does the likelihood that
tionary business change, are to be preferred over work involving them will have additional impacts.
infrequent, ambitious “big bang” implementations The people, processes and systems affected by
one project are disproportionately more likely to
166
Figure 1. Isolated, limited systems: Little overlap between projects
be affected by other projects. In this context it is change and associated IS development. Taking
increasingly difficult to maintain the view of this trend to its logical conclusion, each organisa-
projects as separate from one another, and com- tion should have a single programme of change
partmentalised thinking is required to insulate the consisting of a sequence of small projects, each
project team from being distracted by wider con- containing an increment of business change with
sequences. This type of thinking was helpful when its associated IS implementation.
projects were truly distinct, but in today’s environ-
ment it can be counterproductive. For example, Suggestion: Avoid projects constituted with short-
considering organisational change only within a term or limited-scope objectives; assume that IS
single project is typically detrimental if many work will affect multiple systems and multiple parts
parts of the organisation share systems. Unfortu- of the organisation, and manage work accordingly.
nately, the prior experience of developers and
managers may tend to maintain the illusion of Framework
separation between projects. If IT specialists
extrapolate their past experience of developing This chapter has considered the pros and cons
relatively self-contained systems to predict the of packaging IS work into projects from several
outcome of future projects, operating on more perspectives, which are summarised in Table 1.
extensive and interconnected systems, they may Suggestions are summarised in the rightmost
tend to underestimate the impacts. column as “desirable alternatives”. Together, they
Programme management is one response to provide a simple framework for IS work as an
the need to coordinate multiple related streams of alternative to the use of projects. Alluding to the
work. It involves the orchestration of a portfolio cosmological argument between steady-state and
of projects, helping to mitigate the problems big-bang models of the universe, the framework
caused when each project considers its own goals could be described as a “steady state” model of
in isolation. It demonstrates the benefit in apply- IS work, in contrast to the “big bang” project-
ing pan-project, strategic thinking (Somanchi & based model (Kragh, 1996). According to the
Dwivedula, 2010; Thiry, 2010). Because of the steady-state framework, IS work is best viewed
complex, interrelated nature of modern IS, strong as an evolutionary and incremental process, and
programme management is essential in business it may be helpful in some contexts to abandon
167
Figure 2. Pervasive, integrated systems: Increased project overlap
the use of projects altogether, instead managing a single unified team. This does not neces-
a continuous process of IT and business change. sarily imply geographical co-location, but
To summarise, in steady-state development: steps are taken to ease collaboration across
the whole team.
a. IS tasks are not managed in projects. Work e. Fixed scope/price/duration contracts are
is instead organised as a continuous produc- avoided. Contracts or agreements are made
tion process in which delivery is made in as flexible as possible; suitable forms may
small increments and outcomes are evalu- include time and materials, cost-plus, and
ated progressively. Management decision- shared risk/reward.
making takes into account the organisation’s f. In drafting contracts or other agreements,
long- and short-term objectives rather than the parties resist the impulse to set up legal
merely considering short-term, project- protections and to attempt to offload risk.
related goals. Instead the emphasis is on trust, partnership
b. Detailed requirements are not documented and the quality of the long-term client-vendor
in advance. Flexibility over requirements relationship. Communication and collabora-
allows developers and end users to decide tion are valued over contractual compliance.
together what requirements should be met,
in what order. Some of these suggestions (and particularly the
c. “Pure IT” projects are avoided. Instead, work second) will be familiar to users of agile methods,
involving IT is included in larger business and the steady-state framework is broadly com-
change initiatives. IT work and business patible with the use of agile methods. However,
change are achieved jointly by a team of other suggestions represent a more significant
business and IT professionals through a departure—particularly the arguments against
process of parallel incremental change. It is projects, contracting-out and conventional con-
assumed that any aspect of the organisation tract forms. The framework would be suitable
and its systems may be affected. for most contemporary IS development work,
d. Work is not contracted out to an external particularly where at least one existing system
team. Instead, IT professionals are con- is involved, or a new system will integrate with
tracted-in individually as required to create existing systems, or where multiple developments
168
Table 1. Steady-state framework: analysis of outsourced IS work from different perspectives
Perspective Dimensions Conventional Risks associated with Desirable alternative in

practice in conventional outsourced IS work
outsourced IS work practice in outsourced IS work
Underlying Control vs. High control; fixed Fixed scope, price and duration • Be flexible over scope, cost and dura-
manage- flexibility scope, price and may be unachievable. tion, as far as possible; minimise advance
ment duration planning and top-down control.
philosophy • Avoid advance requirements specifica-
Requirements docu- Requirements may be unknown
tion; expect requirements and priorities
mented in advance at start (so documents likely to be
to evolve as work proceeds, and respond
incorrect).
accordingly.
Priorities set in Priorities may shift (e.g. due to
advance business change or as knowledge
grows).
Scope of IS work vs. Varies, but often proj- Systems may be designed without Avoid IT-only projects; manage IS work
work organisation- ects are constituted as due consideration of business as an integral part of business change and
al change ‘IT only’ issues; required organisational business process evolution.
change may not happen.
Allocation Outsourcing Work is split: IT Communication and collaboration Avoid contracting-out whole projects;
of tasks vs. parts are outsourced; may be poor/absent; mismatches instead, contract-in technical expertise to
in-sourcing business change is are likely between system imple- form joint in-house teams, so that IS work
in-sourced mentation and process change. and business change can be achieved col-
lectively by a unified group.
Form of Protection vs. Standard procurement Plans/systems may be fragmented; • Ensure that vendor-client agreements
contract partnership practice (discrete poor continuity; short-term emphasise trust and partnership, rather
projects) decision-making. than legal protection, and the quality of
the ongoing relationship rather than obli-
Emphasis on legal Focus on contractual protections
gations for the duration of a project.
protection and focus may set parties against one another
• Replace fixed-price/duration/scope
on duration of project and create mutual mistrust.
contracts with flexible contracts (e.g.
only
time and materials, cost-plus, shared risk/
Tight contracts e.g. Vendor and client may adopt rigid, reward).
fixed-price/scope/ defensive attitudes (to avoid losses
duration and achieve value for money).
Charac- Development Varies, but work is Project model may be unrealistic Characterise and manage IS development
terisation vs. mainte- often outsourced in since most IS work is continuous. and maintenance as a single continuous
of work nance project form process.
Phasing of Evolution- ‘Big bang’; imple- Unexpected outcomes may arise Use a process of managed parallel evolu-
delivery ary vs. mentations, limited when substantial deliveries are tion for business processes and IS; deliver
revolutionary deliveries per project implemented and abrupt business technology changes in small increments;
change change is needed. review outcomes and priorities progres-
sively.
Relation- Standalone Projects (and systems) Knock-on effects may be ignored; Avoid projects constituted with short-term
ships vs. overlap- overlap but are man- project’s limited goals may over- or limited-scope objectives; assume that
between ping projects aged as if they were ride client’s wider/longer-term IS work will affect multiple systems and
projects separate interests. multiple parts of the organisation, and
manage work accordingly.
affect one or more systems. Figure 3 summarises FUTURE RESEARCH DIRECTIONS

the key differences between project-based de-
velopment (a) and steady-state development (b). The steady-state framework could be seen as a
bare-bones IS methodology, or a philosophy that
could drive the construction of methodologies.
But IS methodologies are generally applied within
169
Figure 3. Project-based development vs. steady-state development
projects; the steady-state framework is an alterna- inseparable; change is something organisations

tive to projects. For this reason it is perhaps better can achieve, but not something they can buy.
described as a theory about work organisation. As Although discontinuous change is sometimes
a theory it can be tested through empirical research inevitable, IS development and business change
to determine its utility in practice. It would be fit naturally into a continuous, evolutionary cycle.
valuable to evaluate the impact, when steady-state A project is a discrete, non-continuous task. Why
development is used, of varying environmental then has the use of projects been so popular in IS
factors such as project size, extent of outsourcing, development? This may simply be a case of inertia;
type of vendor, type of client, application domain, projects were a good match with early IS work
platform technology, national/organisational and the same practices have continued despite a
culture, and so on. It would be helpful to look changing environment. Or perhaps projects are
at organisations already practising steady-state easier to sell to stakeholders than continuous work.
development, or something akin to it. A qualita- Whatever the cause, continued use of the project
tive research approach would be appropriate, with model does not match the reality of present-day
the framework applied in practice and the results IS development. IS evolve continuously; change
evaluated in an interpretive fashion. and uncertainty are constants. The “engineering
project” abstraction of IS work is a metaphor
which likens IS development to project engineer-
CONCLUSION ing—building bridges and the like. But perhaps
other metaphors would be more beneficial. If IS
The steady-state framework offers a blueprint for development were thought of as the organisational
IS development without projects. Steady-state equivalent of personal growth, for example, then
development is characterised as a continuous change and uncertainty might be seen quite differ-
production process undertaken by developers and ently: not as aberrations but as welcome aspects
end users working in partnership. Steady-state of the process.
development does away with the conventional What does the steady-state framework say
idea of outsourcing as procurement. This makes about IS success? First, it characterises success
sense if we accept that IS and business change are as subjective; IS work is successful if the impor-
tant stakeholders deem it so—the most important
170
stakeholder being the client or end user. In this REFERENCES

regard the steady-state framework accords with
conventional success models which evaluate IS Abrahamsson, P., Salo, O., Ronkainen, J., &
according to measures of usefulness, satisfaction Warsta, J. (2002). Agile software development
and quality, all of which are ultimately subjective methods: Review and analysis (No. 478). Espoo,
(Delone & McLean, 2003). Like these models, Finland: Technical Research Centre of Finland
the steady-state framework judges success in the (VTT Publications.
client’s terms, not those of the vendor or according Agrawal, E., Jain, P., & Jain, V. (2010). A com-
to any supposedly-objective yardstick. Notably, parative study of ISO 9001, CMMI and Six Sigma
it does not try to relate success to conventional with reference to software process quality. Global
project management measures of cost, schedule Journal of Enterprise Information System, 1(1).
and conformance to specification, because the
evidence shows that these measures can produce Ambler, S. W., & Constantine, L. L. (2000). The
quite unsuccessful outcomes in IS work from unified process inception phase. Lawrence, KS:
the client’s subjective perspective (Wu, 2005). CMP Books.
Secondly, the steady-state framework views IS
Arnott, R., & Small, K. (1994). The economics
as dynamic, shifting entities which evolve and
of traffic congestion. American Scientist, 82,
coalesce over periods of years. It is no longer
446–446.
sufficient to pretend that IS come into being at a
point in time and remain roughly the same for long Ash, J. S., Berg, M., & Coiera, E. (2004). Some
periods thereafter. Many organisations either have unintended consequences of Information Tech-
or are creating integrated enterprise systems; rather nology in health care: The nature of patient care
than talking in any particular organisation about Information System-related errors. Journal of the
“an IS”, we should talk about “the IS”: the sum American Medical Informatics Association, 11(2),
total of the organisation’s portfolio of integrated 104. doi:10.1197/jamia.M1471
and non-integrated systems, a super-system which
Avison, D., Gregor, S., & Wilson, D. (2006). Mana-
changes constantly as it is modified, extended
gerial IT unconsciousness. Communications of the
and integrated. The steady-state framework ac-
ACM, 49(7), 88–93. doi:10.1145/1139922.1139923
knowledges the existence of the super-system and
provides a way of working that complements it. Banker, R. D., & Slaughter, S. A. (1997). A field
Finally, the steady-state framework acknowledges study of scale economies in software mainte-
that, for dynamic and changing information sys- nance. Management Science, 43(12), 1709–1725.
tems to be successful, the dynamic and changing doi:10.1287/mnsc.43.12.1709
process by which they come into being and evolve
Beck, K., & Cleal, D. (1999). Optional
must also be successful. Success in this process
scope contracts. Retrieved 11th July, 2011,
is facilitated by principles such as continuous
from http://www.xprogramming.com/ftp/
production, requirements flexibility and the inte-
Optional+scope+contracts.pdf
gration of IT and business change. In particular,
steady-state development contributes to IS success Beynon-Davies, P. (1999). Human error and
by discouraging projects in favour of continuous Information Systems failure: The case of the Lon-
work carried out by integrated developer-user don ambulance service computer-aided despatch
teams, using flexible contracting-in arrangements system project. Interacting with Computers, 11(6),
that promote trust, partnership and relationship 699–720. doi:10.1016/S0953-5438(98)00050-2
quality rather than legal protections.
171
Bhatt, P., Shroff, G., & Misra, A. K. (2006). Dybå, T., & Dingsøyr, T. (2008). Strength of
Influencing factors in outsourced software main- evidence in systematic reviews in software engi-
tenance. ACM SIGSOFT Software Engineering neering. Paper presented at the 2nd ACM-IEEE
Notes, 31(3), 1–6. doi:10.1145/1127878.1127883 International Symposium on Empirical Software
Engineering and Measurement.
Binbasioglu, M., & Winston, E. (2004). Systems
thinking for identifying unintended consequences Erickson, J., Lyytinen, K., & Siau, K. (2005).
of IT: Packaged software implementation in small Agile modeling, agile software development,
businesses. Journal of Computer Information and extreme programming: The state of research.
Systems, 45(1), 86–93. Journal of Database Management, 16(4), 88–100.
doi:10.4018/jdm.2005100105
Boehm, B., & Turner, R. (2005). Management
challenges to implementing agile processes in Ezhkova, I. (2005). Self-organizing representa-
traditional development organizations. IEEE tions. Cybernetics and Systems, 36(8), 861–875.
Software, 22(5), 30–39. doi:10.1109/MS.2005.129 doi:10.1080/01969720500306345
Boudon, R. (1982). The unintended consequences Fowler, M., & Highsmith, J. (2001). The agile
of social action. Macmillan. manifesto. Software Development, 9(8), 28–32.
Brady, T., & Davies, A. (2010). 9 learning to de- Gong, H., Tate, M., & Alborz, S. (2007). Manag-
liver a mega-project. Procuring Complex Perfor- ing the outsourcing marriage to achieve success.
mance: Studies of Innovation in Product-Service System Sciences, 2007. HICSS 2007. 40th Annual
Management, 174. Hawaii International Conference on, 239c-239c.
Burnes, B. (2010). Call for papers: Jour- Gopal, A., Sivaramakrishnan, K., Krishnan, M.
nal of Change Management. Journal of S., & Mukhopadhyay, T. (2003). Contracts in off-
Change Management, 10(2), 241–242. shore software development: An empirical analy-
doi:10.1080/14697011003798085 sis. Management Science, 49(12), 1671–1683.
doi:10.1287/mnsc.49.12.1671.25120
Carmel, E., & Nicholson, B. (2005). Small
firms and offshore software outsourcing: High Grubb, P., & Takang, A. A. (2003). Soft-
transaction costs and their mitigation. Journal of ware maintenance: Concepts and prac-
Global Information Management, 13(3), 33–54. tice. World Scientific Publishing Company.
doi:10.4018/jgim.2005070103 doi:10.1142/9789812564429
Ciborra, C. U. (1993). Teams, markets and systems: Hickey, A. M., Davis, A. M., & Kaiser, D. (2003).
Business innovation and Information Technology. Requirements elicitation techniques. Comparative
Cambridge University Press. Technology Transfer and Society, 1(3), 279–304.
doi:10.1353/ctt.2003.0026
Cohen, M. (1992). Epidemiology of drug re-
sistance: Implications for a post-antimicrobial Highsmith, J., & Cockburn, A. (2001). Agile
era. Science, 257(5073), 1050. doi:10.1126/sci- software development: the business of innovation.
ence.257.5073.1050 Computer, 34(9), 120–127. doi:10.1109/2.947100
Delone, W. H., & McLean, E. R. (2003). The Hughes, M., & Golden, W. (2004). Achieving
DeLone and McLean model of Information business process change in the public sector:
Systems success: A ten-year update. Journal of Revolution or evolution? Electronic Government,
Management Information Systems, 19(4), 9–30. an International Journal, 1(2), 152-165.
172
Iacovou, C. L., & Nakatsu, R. (2008). A risk Kinnula, M., Seppanen, V., Warsta, J., & Vilm-
profile of offshore-outsourced development proj- inko, S. (2007). The formation and management
ects. Communications of the ACM, 51(6), 89–94. of a software outsourcing partnership process.
doi:10.1145/1349026.1349044 40th Annual Hawaii International Conference on
System Sciences, HICSS 2007, (p. 240b).
Jacobs, B. (2004). Using soft systems methodology
for performance improvement and organisational Kragh, H. (1996). Cosmology and controversy:
change in the English national health service. The historical development of two theories of the
Journal of Contingencies and Crisis Manage- universe. Princeton University Press.
ment, 12(4), 138–149. doi:10.1111/j.0966-
Kristensen, J. K. (2001). The hidden threat to
0879.2004.00444.x
e-government–Avoiding large government IT
Jamieson, D., Vinsen, K., & Callender, G. (2005). failures. OECD Public Management Policy Brief.
Agile procurement: New acquisition approach to
Lacity, M. (2008). Managing relationships across
agile software development. 31st EUROMICRO
the life cycle of an outsourcing arrangement. In
Conference on Software Engineering and Ad-
Lacity, M. C., & Willcocks, L. P. (Eds.), Informa-
vanced Applications, 2005, (pp. 266-273).
tion Systems and outsourcing: Studies in theory
Janssen, M., & Klievink, B. (2010). ICT-project and practice (pp. 303–340). Palgrave MacMillan.
failure in public administration: the need to in- doi:10.1057/9780230594838
clude risk management in enterprise architectures.
Lichtenstein, Y. (2004). Puzzles in software de-
Paper presented at the 11th Annual International
velopment contracting. Communications of the
Digital Government Research Conference on
ACM, 47(2), 61–65. doi:10.1145/966389.966391
Public Administration Online: Challenges and
Opportunities, Puebla, Mexico. Light, J. S. (1999). When computers were women.
Technology and Culture, 40(3), 455–483.
Jones, C. (2003). Variations in software develop-
ment practices. Software, IEEE, 20(6), 22–27. Lindvall, M., Basili, V., Boehm, B., Costa, P.,
doi:10.1109/MS.2003.1241362 Dangle, K., Shull, F., et al. (2002). Empirical
findings in agile methods. Proceedings of the 2nd
Jørgensen, M., & Grimstad, S. (2005). Over-
Annual Conference on XP/Agile Universe, (pp.
optimism in software development projects: “The
197-207).
winner’s curse”. Proceedings of IEEE Coniele-
Comp, (pp. 280–285). Liu, J., Chen, H., Chen, C., & Sheu, T. (2010).
Relationships among interpersonal conflict,
Keider, S. P. (1984). Why systems devel-
requirements uncertainty, and software project
opment projects fail. Journal of Informa-
performance. International Journal of Project
tion Systems Management, 1(3), 33–33.
Management, 29(5).
doi:10.1080/07399019408963043
Lormans, M., van Dijk, H., van Deursen, A.,
Khalfan, A. (2003). A case analysis of busi-
Nocker, E., & de Zeeuw, A. (2004). Managing
ness process outsourcing project failure pro-
evolving requirements in an outsourcing context:
file and implementation problems in a large
An industrial experience report. Proceedings of
organisation of a developing nation. Business
the 7th International Workshop on Principles of
Process Management Journal, 9(6), 745–759.
Software Evolution, (pp. 149-158).
doi:10.1108/14637150310506675
173
Loucopulos, P., Lyytinen, K., Liu, K., Gilb, T., Sauer, J. (2006). Agile practices in offshore out-
& Maciaszek, L. A. (2004). Project failures: sourcing–An analysis of published experiences.
Continuing challenges for sustainable Informa- Paper presented at the 29th Information Systems
tion Systems. ICEIS’2004 Sixth International Research Seminar in Scandinavia, IRIS.
Conference on Enterprise Information Systems.
Seddon, P., & Kiew, M. Y. (2007). A partial test
McGinnes, S. (2011). Barriers to client col- and development of DeLone and McLean’s model
laboration in agile offshore Information Systems of IS success. Australasian Journal of Informa-
development. Paper presented at the International tion Systems, 4(1).
Conference on Information Systems Development
Somanchi, V., & Dwivedula, R. (2010). Program
(ISD2011), Edinburgh.
management approach for large-scale Informa-
Mellor, S. J., Clark, A. N., & Futagami, T. tion Technology development projects: A case
(2003). Model-driven development-Guest edi- study. Journal of Project. Program & Portfolio
tor’s introduction. Software, IEEE, 20(5), 14–18. Management, 1(1), 41–54.
doi:10.1109/MS.2003.1231145
Stapleton, J. (2002). DSDM: A framework for
Mumford, E., & Henshall, D. (1978). Participa- business centred development. Addison-Wesley.
tive approach to computer systems design: A
Sutherland, J. (2001). Agile can scale: Inventing
case study of the introduction of a new computer
and reinventing SCRUM in five companies. Cutter
system. New York, NY: Halsted Press.
IT Journal, 14(12), 5–11.
Mutton, P., & Golbeck, J. (2003). Visualization of
Symon, G., & Clegg, C. (2005). Construct-
semantic metadata and ontologies. Proceedings of
ing identity and participation during techno-
the Seventh International Conference on Informa-
logical change. Human Relations, 58(9), 1141.
tion Visualization (IV’03) (pp. 300-305). IEEE.
doi:10.1177/0018726705058941
NAO. (2004). Improving IT procurement: The
Thiry, M. (2010). Program management: Funda-
impact of the Office of Government Commerce’s
mentals of project management. Gower.
initiatives on departments and suppliers in the
delivery of major IT-enabled projects (5 November Van Deursen, A., Visser, E., & Warmer, J. (2007).
2004: National Audit Office. Reich, B. H., & Ben- Model-driven software evolution: A research
basat, I. (2000). Factors that influence the social agenda. Proceedings of the International Work-
dimension of alignment between business and shop on Model-Driven Software Evolution held
Information Technology objectives. Management with the ECSMR’07.
Verhoef, C. (2005). Quantitative aspects of
Sakthivel, S. (2007). Managing risk in offshore sys- outsourcing deals. Science of Computer Pro-
tems development. Communications of the ACM, gramming, 56(3), 275–313. doi:10.1016/j.
50(4), 69–75. doi:10.1145/1232743.1232750 scico.2004.08.003
Salmerona, J. L., & Smarandacheb, F. (2006). Villanueva, T., & Caston, R. (1981). Welfare leg-
Processing uncertainty and indeterminacy in islation and American poverty traps: Ironies and
Information Systems projects success mapping. characteristics. Journal of Social and & Societal
Computational Modeling in Applied Problems: Welfare, 8, 733.
Collected Papers on Econometrics (p. 94). Op-
erations Research, Game Theory and Simulation.
174
Waddington, C. H. (1977). Tools for thought about Aubert, B. A., Patry, M., & Rivard, S. (2005). A
complex systems. St. Albans, UK: Palladin Press. framework for information technology outsourc-
ing risk management. ACM SIGMIS Database,
Wang, E. T. G. (2002). Transaction attributes and
36(4), 9–28. doi:10.1145/1104004.1104007
software outsourcing success: an empirical inves-
tigation of transaction cost theory. Information Bajari, P., & Tadelis, S. (2001). Incentives versus
Systems Journal, 12(2), 153–181. doi:10.1046/ transaction costs: A theory of procurement con-
j.1365-2575.2002.00120.x tracts. The Rand Journal of Economics, 32(3),
387–407. doi:10.2307/2696361
Warne, L., & Hart, D. (1996). The impact of
organizational politics on Information Systems Beulen, E., Ribbers, P., Origin, A., & Hoofddorp,
project failure: A case study. Paper presented N. (2003). IT outsourcing contracts: practical
at the 29th Hawaii International Conference on implications of the incomplete contract theory.
System Sciences (HICSS), Maui, Hawaii. Proceedings of the 36th Annual Hawaii Interna-
tional Conference System Sciences, (p. 10).
West-Knights, L. (2000). Getting IT right for
government: A review of public sector IT proj- Boehm, B., & Turner, R. (2003). Using risk to
ects. Retrieved from http://www.intellectuk.org/ balance agile and plan-driven methods. Computer,
component/content/article/139/4439-getting-it- 36(6), 57–66. doi:10.1109/MC.2003.1204376
right-for-government-report
Cavana, R. Y., & Mares, E. D. (2004). Integrating
Willcocks, L., Hindle, J., Feeny, D., & Lacity, critical thinking and systems thinking: From prem-
M. (2004). IT and business process outsourcing: ises to causal loops. System Dynamics Review,
The knowledge potential. Information Systems 20(3), 223–235. doi:10.1002/sdr.294
Management, 21(3), 7–15. doi:10.1201/1078/44
Chan, F. K. Y., & Thong, J. Y. L. (2009). Accep-
432.21.3.20040601/82471.2
tance of agile methodologies: A critical review and
Willcocks, L., & Lacity, M. C. (2006). Global conceptual framework. Decision Support Systems,
sourcing of business and IT services. Palgrave 46(4), 803–814. doi:10.1016/j.dss.2008.11.009
Macmillan. doi:10.1057/9780230288034
Cottmeyer, M. (2008). The good and bad of agile
Wu, W. (2005). Carrying out contract successfully offshore development. Paper presented at the Agile
ends up with an IT outsourcing failure: Relation- 2008 Conference.
ship management cases of IT outsourcing in a
Ewusi-Mensah, K. (2003). Software development
cross-cultural context. ACIS 2005 Proceedings,
failures. MIT Press.
(p. 23).
Gonzalez, R., Gasco, J., & Llopis, J. (2010).
Information Systems outsourcing reasons
and risks: A new assessment. Industrial Man-
ADDITIONAL READING
agement & Data Systems, 110(2), 284–303.
Alter, S. (2004). Desperately seeking systems doi:10.1108/02635571011020359
thinking in the Information Systems discipline. Goodpasture, J. C. (2009). Project management
Proceedings of the Twenty-Fifth International the agile way: Making IT work in the enterprise.
Conference on Information Systems, (pp. 757- J. Ross Publishing.
769).
175
Grinyer, A. R. (2007). Investigating the adoption Miranda, S. M., & Saunders, C. S. (2003). The
of agile software development methodologies in social construction of meaning: An alternative
organisations. Milton Keynes, UK: Department perspective on information sharing. Information
of Computing, Faculty of Mathematics and Com- Systems Research, 14(1), 87–106. doi:10.1287/
puting, The Open University. isre.14.1.87.14765
Hoda, R., Noble, J., & Marshall, S. (2008). A for Natovich, J. (2003). Vendor related risks in
agile: Issues with awareness and adoption. Paper IT development: A chronology of an out-
presented at the Agile 2008 Conference. sourced project failure. Technology Analysis
and Strategic Management, 15(4), 409–419.
Jones, C. (1996). Patterns of software systems
doi:10.1080/095373203000136015
failure and success. International Thomson
Computer Press. Nelson, M. R. (2005). Understanding large-scale
IT project failure: Escalating and de-escalating
Jørgensen, M., & Moløkken, K. (2004). How large
commitment. Public Administration and Public
are software cost overruns? Critical comments
Policy, 111, 93.
on the Standish Group’s CHAOS reports. Simula
Research Laboratories. Nerur, S., Mahapatra, R. K., & Mangalaraj, G.
(2005). Challenges of migrating to agile meth-
Kendall, K. E., Kong, S., & Kendall, J. E. (2010).
odologies. Communications of the ACM, 48(5),
The impact of agile methodologies on the quality
72–78. doi:10.1145/1060710.1060712
of Information Systems: Factors shaping strategic
adoption of agile practices. International Jour- Plösch, R. (2004). Contracts, scenarios and pro-
nal of Strategic Decision Sciences, 1(1), 41–56. totypes: An integrated approach to high quality
doi:10.4018/jsds.2010103003 software. New York, NY: Springer-Verlag Inc.
Kliem, R. (2004). Managing the risks of offshore Snir, E. M., & Hitt, L. M. (2004). Vendor screening
IT development projects. Information Systems in Information Technology contracting with a pilot
Management, 21(3), 22–27. doi:10.1201/1078/4 project. Journal of Organizational Computing and
4432.21.3.20040601/82473.4 Electronic Commerce, 14(1), 61–88. doi:10.1207/
s15327744joce1401_4
Kontio, J., Hoglund, M., Ryden, J., & Abrahams-
son, P. (2004). Managing commitments and risks: Spekman, R. E., & Davis, E. W. (2004). Risky busi-
Challenges in distributed agile development. ness: Expanding the discussion on risk and the ex-
Proceedings 26th International Conference on tended enterprise. International Journal of Physi-
Software Engineering, ICSE 2004 (pp. 732-733). cal Distribution & Logistics Management, 34(5),
414–433. doi:10.1108/09600030410545454
Lyytinen, K., & Hirschheim, R. (1988). Informa-
tion systems failures—A survey and classification Tafti, M. H. A. (2005). Risks factors associ-
of the empirical literature. Oxford Surveys in ated with offshore IT outsourcing. Industrial
Information Technology, 257-309. Management & Data Systems, 105(5), 549–560.
doi:10.1108/02635570510599940
McCracken, D. D., & Jackson, M. A. (1982). Life
cycle concept considered harmful. ACM SIG-
SOFT Software Engineering Notes, 7(2), 29–32.
doi:10.1145/1005937.1005943
176
Van Der Vyver, G., Lane, M. S., & Koronios, A.

(2010). Facilitators and inhibitors for the adop-
tion of agile methods. In Siau, K., Chiang, R., &
Hardgrave, B. C. (Eds.), Systems analysis and
design: People, processes, and projects (p. 18).
ME Sharpe Inc.
Agile Methods: Approaches to IS work that use

incremental, iterative development, where require-
ments and solutions evolve through collaboration
between self-organising, cross-functional teams.
IS Project: Selected IS work, packaged into
project form.
IS Work: Tasks pertaining to the development,
maintenance, enhancement and support of IS.
IT Procurement: The practice of purchas-
ing IT-related goods and services from external
vendors.
Project Management: An approach to the
management of work which focuses on the
achievement of a selected set of tasks with lim-
ited goals.
Steady-State Development: Continuous,
integrated business change and IS development
conducted by a joint developer-end user team
and relying on trust and partnership rather than
contractual protections.
Steady-State Framework: A set of principles
which describe how steady-state development
can occur.
177
Section 4
Enterprise Resource Planning
Systems
179
Chapter 10
Enterprise Resource Planning
Acceptance Model (ERPAM):
Extended TAM for ERP Systems in
Operational Phase of ERP Lifecycle
Simona Sternad
University of Maribor, Slovenia
Samo Bobek
University of Maribor, Slovenia
ABSTRACT
Enterprise resource planning (ERP) systems have been implemented in most organizations for few
years. But most of the organizations cannot really expose promised benefits of ERP systems. One of
the reasons might be ERP users who do not accepted and use ERP system properly. In IT/IS literature
organizational users have been exposed as important factor, which has influence on IT/IS acceptance
and usage. Technology acceptance model (TAM) proposed by Davis (1989) has been most widely used
model for researching user acceptance and usage of IT/IS. While this research is not the first attempt
to apply TAM to ERP context, the authors of this chapter aim to make more contributions to the topic.
First, they focus on the ERP system use in routine (mature) stage, and because of that, they use construct
extended use instead of actual use. In the latest research of ERP system usage, the relationship between
work compatibility and usefulness has been examined. New relationships between work compatibility and
attitude toward using ERP system have been added. In all TAM studies regarding ERP context, a small
number of external factors have been researched. The groups of external factors that have influence on
ERP extended usage have been researched. The proposed model has been empirically tested using data
collected from a survey of 293 ERP users in 44 organizations across country.
DOI: 10.4018/978-1-4666-0170-3.ch010
Enterprise Resource Planning Acceptance Model (ERPAM)
INTRODUCTION Most organizations have had implemented ERP

solutions for some time. But competition, global-
Enterprise resource planning (ERP) systems are ization and crises have compelled organizations to
systems that typically consist of an enterprise-wide rethink about their information technologies (IT)
set of management tools that balance demand and especially ERP systems which maintenance and
supply; contain the ability to link customers and licensing is quite huge expense of organizations.
suppliers into a complete supply chain; employ ERP systems could be implemented successfully
proven business processes for decision-making; from a technical perspective, but success may
provide high degree of cross-functional integra- depend on ERP users being willing to use the
tion; enable people to run their business with high delivered system (Kwahk & Lee, 2008). Most
levels of customer service and productivity, and technical issues can be fixed, but people resistance
simultaneously lower costs and inventories etc. or poor use of ERP system is more difficult to
(Wallace & Kremzar, 2001). Their characteris- fix. As Langenwalter et al. (2000) pointed out,
tics place them as integrated, all-encompassing, organization can have customers, products, plants
complex mega packages designed to support the and equipment, but without skilled, dedicated,
key functional areas of an organization (Adam knowledgeable people, it cannot function well.
& Sammon, 2004). They solve the critical prob- We can say that ERP system only gives organiza-
lems of integrating information from various tions real benefits if ERP users accept and use it
sources inside and outside of an organization’s extensively during daily tasks.
environment and make it available, in real-time, As every information system (IS) ERP solu-
to all employees and partners of those organizations also go through three phases of lifecycle:
tions (Motiwalla & Thompson, 2009). The most selection, implementation and operation phase;
important contributions of ERP systems are that which consists of stabilization stage and routine
they significantly reduce the time to complete stage. Much of the success of ERP implemen-
business processes and help organizations to share tation lies in operation phase of ERP lifecycle
information (Lee et al., 2010) and that organiza- (Motiwalla & Thomson, 2009; Bradford, 2008).
tions usually offer a better work environment for Stabilization stage is the time from Go-live to
their employees as they are given more efficient about 30 to 90 days after, or until the number of
system to work with. ERP implementations almost issues and problems has been reduced to a small,
always require business process reengineering, manageable number. After that, companies enter
because of the need to adopt the organizational to routine stage, in which they have to put more
processes to match the capabilities of the ERP effort into people and process improvements
system (Amonako-Guampah & Salam, 2004). (Bradford, 2008). In routine stage users accept
They allow separate business processes to be the system and the usage becomes a regular day-
put together into one compact system what the to-day activity (Rajagopal, 2002). It often takes
vendor consider “best practice” (Bradly & Lee, many months or even years for experienced users
2007). Organizations turn to ERP systems to im- to get comfortable with the ERP system because,
prove efficiency and become more responsive to early in an ERP system’s life, these users tend to
customer needs. Because of their characteristics resist using it for their work. They already have
they have huge impact on organizational workers a set of processes and a comfort level in getting
especially on ERP users. Impact of ERP systems their work done, the complex ERP systems may
on their users and their acceptance and/or refusing appear threatening and intrusive (Musaji, 2002).
has been recognized as one of key factors of ERP Eventually the users’ turn-on their ERP systems
implementation success. and they begin to work using ERP systems, instead
180
of having to navigate many different systems. (proposed by Nah et al., 2005; Sun et al., 2009;
However, at some point in the ERP system’s life Calisir et al., 2009) and to examine groups of
users begin to see the advantages of the ERP external factors which have impact on actual ERP
system and then they carefully begin to explore system use based on TAM. Survey data has been
its functions, gradually reaching success. Finally, collected from ERP users where ERP system has
having mastered the system, the users begin to get operated for more than one year. The proposed
creative. This shows that the ERP users have ac- research model is analyzed by using the PLS
cepted the system and are putting it to extended use. approach. The rest of this chapter is organized
Moreover, numerous factors influence IS suc- as follows: literature review, enterprise resource
cess, especially individual acceptance or resistance acceptance model, methodology, results, discus-
(Amoako-Gyampah & Salam, 2004). To improve sion, and conclusion.
the efficiency and effectiveness of ERP system
use, organizations need to research factors that
impact user satisfaction. In this area, technologi- TECHNOLOGICAL ACCEPTANCE
cal acceptance model (TAM) is one of the most MODEL (TAM) AND ERP SYSTEMS
widely used models for explaining the behavioral
intention and actual usage, and can improve our Several theoretical models have been used to
understanding of how influence on actual usage investigate the determinants of acceptance and
could help increase efficiency and effectiveness the use of new information technology (IT), such
of ERP system use (Shih & Huang, 2009). Review as the theory of reasoned action (TRA) (Fishbein
of the literature shows that in past few years a few & Ajzen, 1975), the theory of planned behaviour
studies that have examined the users’ adoption of (TPB) (Ajzen, 1991), the theory of the technology
ERP systems through TAM have been published acceptance model (TAM) (Davis et al., 1989), etc.
after 2009 (Shih & Huang, 2009; Calisir et al., Compared to competing models, TAM is believed
2009; Youngberg et al., 2009; Sun et al., 2009; to be more parsimonious, predicative, and robust
Lee et al., 2010). But all of them examine few (Venkatesh & Davis, 2000; Lu et al., 2003; Liu &
contextual factors that have influence on intention Ma, 2006), and so among the theoretical models
to use ERP system or ERP use in stabilization stage is most widely used by IS/IT researchers (Davis,
or earlier stages of ERP lifecycle. In addition very 1989; Davis et al., 1989; Amoako-Gyampah &
few studies have been conducted regarding tech- Salam, 2004; Lee et al., 2010). TAM posits that
nology acceptance of ERP systems. Through their two beliefs − perceived usefulness (PU) and
scientific work, researchers have recognized that perceived ease of use (PEOU) − are of primary
the generality of TAM and researching of small relevance for computer acceptance behaviour
numbers additional factors that have impact on (Davis et al., 1989). PU is defined as “the degree
TAM fails to supply more meaningful informa- to which a person believes that using a particular
tion on users’ opinions about a specific system system would enhance his or her job performance”
especially of ERP system which is considered as (Davis 1989, p. 320). PEOU in contrast, refers to
a strategic IS in organizations. Therefore there “the degree to which a person believes that us-
is the need to incorporate additional factors or ing a particular system would be free of effort”
integrate it with other IT acceptance models for (Davis 1989, p. 320). The two central hypotheses
improvement of its specificity and explanatory in TAM state that PU and PEOU positively influ-
utility (Agarwal & Prasad, 1998; Lu et al., 2003). ence an individual’s attitude towards using a new
The purpose of this chapter is to extend original technology, which in turn influences his or her
TAM with cognitive construct work compatibility behavioural intention to use it. Finally, intention
181
is positively related to the actual use. TAM also on attitude toward using ERP system and
predicts that PEOU influences PU, as Davis et al., symbolic adoption in the post-implemen-
(1989, p. 987) put it, “effort saved due to the im- tation phase (stabilization stage).
proved perceived ease of use may be redeployed, • Amoako-Gyampah and Salam (2004).
enabling a person to accomplish more work for Their study evaluated the impact of one be-
the same effort”. The key purpose of TAM is to lief construct (shared beliefs in the benefits
provide a basis for tracing the impact of external of a technology) and two technology suc-
factors on internal beliefs, attitudes, and intentions cess factors (training and communications)
(Davis et al., 1989). on PU and PEOU in one global organiza-
The original TAM is well established and tested tion in the implementation phase.
and furthermore, a variety of extensions regarding • Shivers-Blackwell and Charles (2006)
external factors by examining the antecedents of have researched student readiness for
PU and PEOU has been developed. Venkatesh change (through gender, computer self-
and Davis (2000) proposed an extension of TAM efficacy, and perceived benefits of ERP) on
– TAM2 by identifying general determinants of behavioural intention regarding the ERP
PU, which are subjective norm, image, job rel- implementation.
evance, output quality, results demonstrability, and • Bradley and Lee (2007) have investigated
PEOU. Venkatesh (2000) developed a model of the via case studies the relationship between
determinants of PEOU, which are computer self- training satisfaction and PEOU, PU, effec-
efficacy, computer anxiety, computer playfulness, tiveness and efficiency when implementing
and perceptions of external control (or facilitating an ERP system at a mid-sized university.
conditions). Venkatesh and Bala (2008) combined • Hsieh and Wang (2007) have researched
TAM2 and the model of the determinants of PEOU the impact of PU and PEOU on extended
and developed an integrated model of technology use in the post-implementation phase (rou-
acceptance – TAM3. tine stage).
One of commonly cited reasons for lack of • Kwahk and Lee (2008) have examined
sophisticated use of ERP systems is the lack of the formation of readiness for change
user acceptance. As several studies (i.e. Umble et (enhanced by two factors: organizational
al., 2002; Nah et al., 2004) have revealed, a com- commitment and perceived personal com-
mon reason for ERP failures can be attributed to petence) and its effect on the perceived
users’ reluctance and unwillingness to adopt and technological value of an ERP system
to use the implemented ERP system. A literature leading to its use in post-implementation
review of past ERP studies regarding TAM indi- (stabilization stage).
cates that few studies have investigated ERP user • Bueno and Salmeron (2008) have devel-
acceptance and usage. Research shows that small oped a research model based on TAM for
number of researches has been published and all testing the influence of the critical suc-
of them expose small numbers of determinants cess factors (top management support,
(external factors) or cognitive factors which can communication, cooperation, training,
have influence on ERP acceptance and usage in and technological complexity) on ERP
different phases of an ERP system lifecycle: implementation.
• Uzoka et al., (2008) have extended TAM
• Nah et al., (2004) tested the impact of four to research the selection of ERP system
cognitive constructors (PU, PEOU, per- using factors: impact of system quality,
ceived compatibility, and perceived fit) information quality, service quality, and
182
support quality as key determinants of wal & Prasad, 1998; Lu et al., 2003). Because
cognitive response, to which ERP system of the fact that ERP systems are complex and a
to purchase/use. complex systems decrease usefulness and ease of
• Sun et al., (2009) have extended IT usage use (Momoh et al., 2010), a better understanding
models to include the role of ERP’s per- of the factors leading ERP users’ acceptance of
ceived work compatibility in users’ ERP ERP systems is necessary to facilitate successful
usage intention, usage, and performance in ERP usage (Nah et al., 2004).
work settings.
• Shih and Huang (2009) study attempts to
explain behavioural intention and actual ERP ACCEPTANCE MODEL
use through incorporated additional be-
havioural constructs: top management sup- To examine ERP users’ use of ERP systems, we
port, computer self-efficacy, and computer need to extend the TAM. Synthesizing prior re-
anxiety in the post-implementation (rou- searches on TAM and researches on ERP systems,
tine stage). a conceptual model that represents the cumulative
• Calisir et al., (2009) have examined factors body of knowledge accumulated over the years
(subjective norms, compatibility, gender, from TAM and ERP research has been developed
experience, and education level) that affect (Figure 1). Because our research is focused on
users’ behavioural intention to use an ERP current usage of ERP system in routine stage,
system based on potential ERP users at one measured by degree of actual use, there is no need
manufacturing organization. to examine the behavioural intention on actual use,
• Youngberg et al., (2009) have researched behavioural intention was dropped from purposed
impact of PEOU, results demonstrability, research model, the same as researchers Simon
and subjective norm on PU and impact of it and Paper (2007) and Pijpers and Montfort (2006)
on usage behaviour in the post-implemen- did. Empirical researches have found support for
tation phase (stabilization stage). original relationships of TAM in ERP settings in
• Lee et al., (2010) have examined fac- routine stage (Lee et al., 2010; Shih and Huang,
tor organizational support (formal and 2009; Sun et al., 2009; Hsieh and Wang, 2007).
informal) on original TAM factors in the Because we have examined ERP use after one year
post-implementation. of operation we cannot talk about PU and PEOU
but about ERP usefulness (U) and ERP ease of
In summary, because of high rate of ERP use (EOU). The grey area within the dotted line
implementation failure more research in the area denotes the original TAM.
of technological acceptance is needed (Youngberg
et al., 2009; Kwahk & Lee, 2008). The original Extended Use
TAM is well established and tested and further-
more, a variety of extensions has been developed TAM was originally conceptualized in the context
in different IT environments. More researches on of personal use and ignored the role of organiza-
TAM stress out that if we want to explain higher tional work on IT usage or its predictors (Sun et
proportion of the variance (typically about 40%) al., 2009). Firstly, organizational users use ERP
we have to include more external factors which systems to perform specific tasks and speed orga-
have impact on cognitive factors (Davis et al., nizational work, because of that it is important to
1989; Venkatesh & Davis, 2000; Legris et al., examine the role of organizational work in ERP
2003) or integrate other cognitive factors (Agar- usage. Secondly, ERP systems are mandatory to
183
Figure 1. Conceptual model
use by organizational workers where one user’s Work Compatibility

tasks on the ERP system are tightly integrated
with other users’ tasks (Nah et al., 2004). In In case of ERP implementations, other cognitive
other words, ERP users generally do not have a considerations besides usefulness and ease of use
choice not to use the ERP system, regardless of may become relevant (Nah et al., 2004). In the
their attitude. On the other hand, organizations ERP context, organizations have to adopt busi-
that implement ERP systems want to use their ness processes of an implemented ERP system.
systems to the fullest potential and realize the Although one of the major benefits of ERP systems
promised benefits. Because of all that for orga- is that they offer to organizations solution with best
nizations it is important that ERP users not just business practices it is not necessary that this is the
use basic functionality of ERP systems but to use best options for ERP users. Inherent business rules
extended functionality of them. Hsieh and Wang behind the processes gives them little choice but
(2007) have defined extended use (EU) as the to follow strict business processes of ERP system,
use behavior that goes beyond typical usage and unlike the old systems, which allowed them to
can lead to better results and returns. EU captures carry out many different process variations. So
the breath and frequency of using different ERP organizations deploy ERP systems to facilitate
features and functions. If the users find the ERP organizational work rather than to match users’
system to be useful, he/she will be more inclined personal preferences or habits. At this presume
to fully examine and use its features and functions we view work compatibility (WC) strictly as the
(Saeed & Abdinnour-Helm, 2008). Original TAM fit of ERP system to organizational work, and not
includes factor actual usage, but because we have to personal preferences or work habits. WC, like
examined usage of ERP systems in routine stage U and EOU, is more a perceptual construct as it
of ERP lifecycle we have changed this factor to is the perception of fit between IT and work that
EU, which measures the degree of extended use motivates employees to use the system, irrespec-
of ERP system. tive of the actual extent of fit (Sun et al., 2009).
WC refers to the degree to which can ERP user
do most of their tasks in ERP system. WC influ-
ences U and so it demonstrates the importance of
incorporating work compatibility in models of IT
184
usage as exposed Sun et al. (2009) and Scott and Thompson et al., 2006; Shivers-Blackwell
Walczak (2009). In a context of ERP usage it is & Charles, 2006; Venkatesh & Bala, 2008;
expected that relationship between WC and U Shih & Huaug, 2009) is an individual judg-
as the more work compatible ERP system is, the ment regarding users’ computer skills (Shin
more useful it is for ERP users. We also presume and Huang, 2009) and is defined as the de-
that WC has strong direct effect on attitude toward gree to which an individual believes that
using ERP system (AT) and on EU, not just indirect he/she has the ability to perform a specific
effect through U as if ERP users believe that ERP task/job using the computer (Venkatesh &
system is more work compatible with their daily Bala, 2008; Shih & Huang, 2009).
tasks, they will have more positive attitude toward • Personal innovativeness toward IT
using that system and also toward extended use (Agarwal & Prasad, 1999; Roges, 2003;
of that system. Yi et al., 2006; Thompson et al., 2006).
According to the innovation diffusion the-
External Factors ory (Rogers, 2003; Yi et al., 2006), people
react differently to a new idea, practice, or
Even though TAM is a model applicable to a object due to their differences in individual
variety of technologies, the constructs of TAM innovativeness, a predisposed tendency
need to be extended by incorporating additional toward adaptation an innovation. Personal
factors regarding particular IS (Calisir et al., 2009). innovativeness toward IT represents the
Schwarz (2003) review of identified antecedents of degree to which an individual is willing to
cognitive factors (PEOU and PU) puts factors into try out a new IT (Agarwal & Prasad, 1998).
three groups, which are: individual variable (such • Computer anxiety (Venkatesh et al., 2003;
as computer experience, self-efficacy, and prior Liu & Ma, 2006; Venkatesh & Bala, 2008;
experiences), organizational influences (such as Shih & Huang, 2009) represents degree of
management and external support, and perceived an individual’s apprehension, or even fear,
resources) and technology characteristics (such as when he/she is faced with the possibil-
accessibility of the medium and interface type). ity of using computers (Venkatesh, 2000).
When researching individual variables the Individuals with lower anxiety are much
four external factors were exposed. We name more likely to interact with computers than
this group of factors personal characteristics and people with higher anxiety (Shih &Huang,
information literacy (PCIL) which includes per- 2009).
sonality characteristics that can have influence
on individuals’ perceptions of IS acceptance and Based on prior research regarding ERP systems
usage, and are not dependent on a particular IT. we expose external factors into two groups: (1) a
PCIL includes factors: group of factors which represent system and tech-
nological characteristics (STC) and (2) a group of
• Experience with computer (Davis et al., factors which represents organizational-process
1998; Venkatesh et al., 2003; Thompson et characteristics (OPC). STC and OPC groups in-
al., 2006; Venkatesh & Bala, 2008; Calisir clude some critical success factors (CSF) of ERP
et al., 2009) has been found to be important implementations exposed by Akkermans and van
factor for the acceptance of a technology Helden (2002), Somers and Nelson (2004), Bobek
(Calisir et al., 2009). and Sternad (2010), which have influence on ERP
• Computer self-efficiency (Venkatesh users in the operation phase.
& Davis, 2000; Venkatesh et al., 2003;
185
STC are those silent features of ERP systems et al., 2006;Bradford, 2008) refer to the
that can help individuals develop favorable/ degree to which an ERP users’ views in-
unfavorable perceptions regarding U, EOU and/ adequate users manuals and help as the
or WC. In contrast to most IT implementation reason for one’s unsuccessful ERP perfor-
research, the fact that ERP implementation re- mance (Kelley, 2001).
search is focused on one technology has enabled
the effect of specific technological characteristics OPC capture various social processes, mecha-
to be examined. Cooper and Zmud (1990) pointed nisms and support organizations that guide indi-
out that lack of attention to system and techno- viduals to facilitate the use of an ERP system.
logical characteristics is a serious deficiency in OPC includes:
most IT implementation research. We have not
found a research which has examined system and • Social influence (Venkatesh, 1998;
technology characteristics upon the ERP system Venkatesh et al., 2003; Thomson et al.,
user acceptance. Surveying different researches 2003; Bradford, 2008; Calisir et al., 2009)
following external factors have been exposed: which joins two factors: subjective norm
and social factors. Subjective norm is de-
• Data quality (Venkatesh, 1998; Venkatesh fined as a person’s perception that most
& Davis, 2000; Gattiker & Goodhue, 2005; people who are important to him/her think
Kositnurit et al., 2006; Insiti, 2007). ERP that he/she should or should not perform
system provides easy access to corporate the behaviour in question (Venkatesh,
data, but if that data is inaccurate or irrel- 1998). Social factors are defined as indi-
evant to the business processes in the sub- vudual‘s internalization of the reference
unit, there will be few benefits. So without group’s subjective culture, and specific in-
accurate and relevant data, an organization terpersonal agreements that the individual
is severely constrained in the coordination has made with others in specific social situ-
and task efficiency benefits it can achieve ations (Thompson et al., 2006; Venkatesh
from its ERP system (Gattiker & Goodhue, et al., 2003).
2005). • Fit with business processes (Amoako-
• ERP system functionality (Musaji, 2002; Gyampah & Salam, 2004; Nah et al., 2004;
Somers et al., 2003; Lu et al., 2003; Bradley & Lee, 2007; Bradford, 2008;
Kositnurit et al., 2006; Insiti, 2007). Bobek & Sternad, 2010). ERP packages
System functions are used to measure the are built around best practices in specific
rapid response, stability, easy usage and industries. But the software may not nec-
flexibility of the ERP system. essarily fit the operating practices of an
• ERP system performance (Boudreau, adopting organization. Nah et al. (2004)
2002; Musaji, 2002; Venkatesh et al., define perceived fit from an end-user’s
2003; Somers et al., 2003; Kositnurit et al., perspective as the degree to which the ERP
2006; Liu & Ma, 2006; Insiti, 2007) refers system perceived by a user to meet his/her
to the degree to which person believes that organization’s needs.
a system is reliable and responsive during • Training and education on ERP system
a normal course of operations (Liu & Ma, (Amonko-Gyampah & Salam, 2004;
2006). Bradley & Lee, 2007; Bueno & Salmeron,
• User manuals help (Kelley, 2001; 2008; Bobek & Sternad, 2010) is an im-
Boudreau, 2002; Musaji, 2002; Kositnurit portant component in ERP implementa-
186
tion projects and is recommended before, METHODOLOGY

during and after implementation (Bueno &
Salmeron, 2008). Training and education Sample and Procedure
on ERP system is defined as a degree to
which user thinks that he/she had enough Our sample consists of companies which use SAP
formal and informal training after ERP or Microsoft Dynamics solutions for more than one
implementation. year. So they can be considered as ERP systems
• ERP support (Boudreau, 2002; Lee et al., in the routine stage. A total of 122 companies the
2010). In an ERP system environment, if initial e-mail has been send to each organization
the organization provides sufficient sup- to verify if they matched our selection criteria and
port to organizational workers for their to explain the purpose of the study. 44 organiza-
tasks, they are more likely to enjoy their tions agreed to participate in the survey and they
work and improve their performance were asked to distribute the survey questionnaire
through usage of the ERP system (Lee et to their ERP users. All respondents were required
al., 2010). We define ERP support as the to have used ERP system in their daily work. 293
degree to which an individual view ad- questionnaires were properly filled by respondents
equate ERP support as the reason for one’s and used for the purpose of analysis.
successful ERP usage.
• ERP communication (Kelley, 2001; Musaji, Measures
2002; Boudreau, 2002; Amonko-Gyampah
& Salam, 2004; Bueno & Salmeron, 2008; The constructs of purposed model are: ERP
Bobek & Sternad, 2010) is viewed as hav- usefulness (U), ERP ease of use (EOU), work
ing a high impact on initiation to system compatibility (WC), attitude toward ERP use (AT)
acceptance, as it helps to minimize pos- and extended use of ERP system (EU) for basic
sible user resistance (Somers & Nelson, TAM of ERP systems, influence by constructs
2004). Communication problems refer to of external variables. The constructs of external
the lack of communication regarding the variables are distributed among three second-level
ERP applications and their modifications constructs which are personal characteristics and
(Kelley, 2001). Because of that ERP com- information literacy (PCIL), system and techno-
munication has been defined as the degree logical characteristics (STC) and organizational-
to which an individual view sufficient process characteristics (OPC). PCIL includes
communication regarding the ERP as the factors: experience with computer, computer
reason for one’s successful ERP usage. self-efficiency, personal innovativeness toward
IT and computer anxiety. STC includes factors:
The problem of TAM researches are that most ERP data quality, ERP system functionality, ERP
researchers investigate small numbers of external system performance and user manuals (help).
factors that have influence on user acceptance and And OPC includes factors: social influence, fit
usage. In context of organizations IS, especially with business processes, training and education
ERP systems, there are more external factors that on ERP, ERP support and ERP communication.
can have influence on users acceptance and usage. Our model includes 18 first-order factors and three
Because of that, conceptualization of higher-order second-order factors. All the items of factors were
factors (in our case second-order factors), in which measured on a 7-point Likert scale, ranging from
more external factors jointly have to be observed. ‘strongly disagree‘to ‘strongly agree’ taken from
relevant prior researches and adapted to relate to
187
the context of ERP usage. In addition, demographic is a predicative technique suitable for situations
information has been collected. with less theory development; it places minimal
The instrument in multiple stages has been de- demands on measurement scales; it avoids factor
veloped as suggested by Straub (1989). First, based indeterminacy problems and inadmissible solu-
on an extensive literature review, the initial instru- tions; it avoids identification problems of recursive
ment was developed. The research instrument was models; it makes no assumptions about the data;
discussed with ten experts of ERP systems. The it requires no specific distributions for measured
purpose of these discussions was to refine the variables; it assumes the errors are uncorrelated;
first-order constructs and the items measuring for it works well with small samples and it is better
each construct. Based on the problems identified suited for analyzing complex relationships and
by the respondents, minor improvements to the models.
questionnaire have been made. Models, which include second-order factors,
Although all items had been validated by past consist of a higher-order factors that are modeled
researches, the adopted instrument was examined as causally impacting a number of first-order fac-
to ensure content, construct validity and reliability tors (i.e. standard factors with measured indicators;
within the ERP context. As Amoako-Gyampah and Chin, 1998). Therefore, these second-order factors
Salam (2004) pointed out, instrument validation or are not directly connected to any measurement
re-validation was necessary because the validity items. A higher-order factor can thus be created
of the instrument may not have been persistent by specifying a latent variable which represents
across different technologies and user groups. all the manifest variables of the underlying lower-
The instrument was pilot tested with a group of order factors (i.e. if a second-order construct
30 ERP users in one organization. The instru- consists of four underlying first-order factors
ment’s reliability was evaluated, the Cronbach’s each with three indicators, then the second order
alpha values ranged from 0.58 to 0.91, indicating factor can be specified using all indicators (i.e.
a satisfactory level of reliability exceeding value twelve indicators) of the underlying first order
0.5 (Hinton et al., 2004). As part of the pre-test indicators). And tests if validity for second-order
comments and suggestions on questionnaire items factors should, by analogy, follow the same
and items wording have also been taken into ac- procedure that is used to examine the validity of
count. Based on the results of the pilot testing, first-order factors (Chin, 1998). One then needs
revisions and additions were made to the instru- to demonstrate the convergent validity of the
ment. Pilot participants were included in the main first-order factors by examining the strength of
data gathering effort since they were part of the the paths connecting the second-order factors to
population of interest. Final scales and items are the first-order factors (Chin, 1998). He added that
listed in Appendix A. we should expect a large percentage of these paths
to be at 0.70 and above as well as adequate model
Model Estimation fit. PLS approach allows the conceptualization of
higher-order factors by repeated use of manifest
Hierarchical models using latent variables can variables (Tenenhaus et al., 2005).
be estimated using structural equation model- Due to all exposed benefits, we decided to use
ing (SEM). Both covariance-based SEM and PLS approach. The empirical data was analyzed
component-based SEM, or partial least squares in two stages involving a PLS technique, using
(PLS) approach can be employed to estimate the Smart PLS 2.0 M3 (Ringle & Will, 2005). In the
parameters in a hierarchical model. According to first stage, all measurement scales have been ex-
Chin (1998) PLS has several major strengths: it amined for their psychometric properties, while
188
Table 1. Descriptive statistics of respondents’ characteristics
Measure Items Frequency Percent

Age <19 0 0
20-29 22 7.5
30-39 111 37.9
40-49 103 35.2
50-59 56 19.1
60+ 1 0.3
Educational level Lower education 96 32.8
High school education 54 18.4
College education or more 143 48.8
the second stage focused on hypothesis testing (6) have marked corporate government and/or
and analysis. Path significance has been estimated top management. Average total working years is
using bootstrapping resampling techniques with 15.4 years (min = 1, max = 43, SD = 10.2) and
500 sub-samples. While analyzing data, we fol- average working years on these working place is
lowed the guidelines specified in Tenenhaus et 7.6 years (min = 1, max = 37, SD =7.4).
al., (2005) and Henseler et al., (2009). The average respondent had used ERP system
for 4.7 years (min = 1, max = 18) in their daily
work. Respondents estimated their intensity of
RESULTS AND ANALYSIS ERP usage via the following statement: “I would
rate the intensity of my job-related ERP system
Descriptive Statistics use to be…” on a 7-point Likert scale, ranging
from ‘not important’ to ‘very important’. The
293 questionnaires were properly filled by re- average value is 5, which means ‘little bigger
spondents from 44 organizations and used for the importance’ than average. The most of respondents
purpose of analysis. Survey respondents repre- selected ‘6’, or ‘much bigger importance’. Further,
sented different groups of industries, including IT respondents estimated their frequency of ERP use
and telecommunications (44.0%), manufacturing via three statements proposed by Schwarz (2003),
(35.2%), professional, scientific and technical ac- also on 7-point Likert scale. Responses to these
tivities (10.2%), wholesale and retail trade (4.1%) questions are presented in Table 2. For all three
and others (6.5%). Respondents were 51.5% (151) statements, an average value is a little above 4,
male and 48.5% (142) female. Most of them which represents a moderate degree of use.
(67.2%) had high school education or more. De-
scriptive statistics of respondents’ characteristics Measurement Model
is presented in Table 1. ERP system has been used
for 4.73 years in average (min = 1, max = 18, SD Although all the scales were derived from previ-
= 3.0). 53.6% (157) have marked working place ously developed and validated measures and the
as worker (experts and other employees), 31.7% instrument’s reliability was evaluated through the
(93) have marked low management (e.g. manager pilot testing, psychometric properties (measure-
of group or organization unit), 12.6% (37) have ment model) of these scales were assessed via
marked middle management (e.g. CIO) and 2%
189
Table 2. Degree of use
In a typical one-month period, what is the likelihood of you … Average Mediana Modusa
… using most of features of the ERP solution? 4.01 4 4
… using more features than other users of the ERP solutions? 4.22 4 4
… using more obscure aspects of the ERP solutions? 4.09 4 4
Scale 1-7, ‘not important’ to ‘very important’.
a
evaluation of reliability, convergent validity and higher than recommended level of 0.70 and also
discriminant validity, each of measurement scale. all values AVE exceeded 0.50. Remainder of our
For external factors second-order factor proce- measurement scales shows strong evidence for
dure has been used. Second-order factors can be ap- convergent validity. In Table 6 the CR and AVE
proximated using various procedures. The method of the measures in the second-order model are
of repeated indicators known as the hierarchical included, demonstrating that CRs exceeding 0.80
component model suggested by Wold (1989) is and AVE is equal or exceeding 0.50 (except for
the easiest to implement. The second-order factor PCIL where it is 0.49) which provides evidence
is directly measured by observed variables for for reliable measurements.
all the first-order factors that are measured with Discriminant validity between constructs has
reflective indicators. While this approach repeats been assessed following Fornell and Larcker’s
the number of manifest variables used, the model recommendation that the square root of AVE for
can be estimated by the standard PLS algorithm each construct should exceed the bivariate cor-
(Hensler et al., 2009). Because all of external relations between that construct and all other
factors did not meet assessment requirements of constructs (Fornell & Larcker, 1981). The inter-
the measurement model, we excluded them from construct correlation matrix (Table 4) shows that
further analysis. These external factors are: com- the principal diagonal elements (square root AVE)
puter self-efficacy and experience with computer exceed non-diagonals elements in the same row
from PCIL group, ERP functionality from STC or columns (bivariate correlations), demonstrating
group and ERP support, ERP communications that the discriminate validity of all scales is also
and ERP training and education from OPC group. adequate.
The final version of the model is presented. We Overall, these measurement results are satis-
examined two measures of reliability: Cronbach’s factory and suggest that it is appropriate to proceed
alpha (α) and composite reliability (CR). As shown with the evaluation of the structural model.
it Table 3 each of our 43 scales had Cronbach’s
alpha exceeding 0.70 and composite reliability Structural Model
exceeding 0.80, assuring adequate reliability for
our measurement scales. The next step in analysis was to examine the
For convergent validity Fornell and Larcker’s path significance and magnitude of each of our
assessment criteria has been adopted: all item relationship effect and the overall explanatory
factor loadings should be significant and exceed power of the proposed model. The relationships
0.70, and the average variance extracted (AVE) testing results are based on bootstrapping (with
for each construct should exceed 0.50 (Fornell & 500 subsamples) to test the statistical significance
Larcker, 1981). Table 3 lists item factor loadings, of each path coefficient using t-tests, as recom-
all of which were significant at p < 0.01 and were mended by Chin (1998). Results of this analysis
190
Table 3. Psychometric properties of the instrument
Item Item
Construct Indicators Loadings α CR AVE
mean S.D.
Personal innovativeness toward IT pcil_pi1 5.40 1.35 0.86 0.85 0.91 0.77
pcil_pi2 4.57 1.68 0.86
pcil_pi3 5.14 1.59 0.92
Computer anxiety pcil_ca1 a 1.57 1.00 0.87 0.76 0.87 0.69
pcil_ca2 a
1.42 0.95 0.87
pcil_ca3 6.37 1.00 0.73
Data quality stc_dq1 5.41 1.03 0.82 0.91 0.93 0.69
stc_dq2 5.43 1.25 0.85
stc_dq3 4.90 1.41 0.82
stc_dq4 5.41 1.24 0.88
stc_dq5 4.96 1.44 0.85
stc_dq6 5.21 1.35 0.74
System performance stc_sp1 5.14 1.43 0.79 0.86 0.90 0.64
stc_sp2 5.14 1.50 0.72
stc_sp3 5.45 1.23 0.87
stc_sp4 5.33 1.35 0.73
stc_sp5 5.23 1.30 0.88
User manuals stc_um1 5.00 1.51 0.88 0.84 0.91 0.76
stc_um2 5.00 1.59 0.84
stc_um3 4.54 1.61 0.90
Business processes fit opc_bpf1 5.61 1.27 0.91 0.91 0.94 0.84
opc_bpf2 5.56 1.28 0.89
opc_bpf3 5.39 1.29 0.89
Social influence opc_si1 5.94 1.16 0.76 0.84 0.90 0.68
opc_si2 6.01 1.03 0.79
opc_si3 5.98 1.15 0.88
opc_si4 6.03 1.09 0.87
U u1 5.51 1.32 0.91 0.96 0.97 0.89
u2 5.44 1.32 0.96
u3 5.49 1.33 0.97
u4 5.40 1.40 0.93
EOU eou1 5.25 1.34 0.94 0.85 0.93 0.87
eou2 4.90 1.41 0.93
WC wc1 4.78 1.57 0.89 0.90 0.94 0.84
wc2 5.10 1.42 0.95
wc3 5.18 1.42 0.90
191
Table 3. Continued
Item Item
Construct Indicators Loadings α CR AVE
mean S.D.
AT at1 5.93 1.18 0.94 0.90 0.95 0.91
at2 5.68 1.32 0.96
EU eu1 4.98 1.42 0.84 0.89 0.92 0.75
eu2 4.01 1.72 0.83
eu3 4.22 1.68 0.91
eu4 4.09 1.66 0.87
a
Items have been inverted before processing of statistical data in Smart PLS.
Table 6. Path coefficients external variables in second-order model a
PCIL STC OPC

α = 0.79 α = 0.92 α = 0.86
CR = 0.85 CR = 0.93 CR = 0.89
First-order external factors AVE = 0.49 AVE = 0.50 AVE = 0.54
Personal innovativeness 0.85 (t = 43.20)
Computer anxiety 0.74 (t = 14.65)
Business process fit 0.83 (t = 37.3)
Social influence 0.72 (t = 15.05)
ERP data quality 0.90 (t = 67.96)
ERP system performance 0.85 (t = 38.18)
ERP user manuals (help) 0.69 (t = 16.78)
a
All t-values are bigger then 3.31 and are significant at p < 0.01.
Table 4. Intercorrelations of the latent variables a
1 2 3 4 5 6 7 8 9 10 11 12
1. Personal innovativeness toward IT 0.88
2. Computer anxiety 0.34 0.83
3. ERP data quality 0.15 0.19 0.83
4. ERP system performance 0.10 0.19 0.64 0.80
5. User manuals (help) 0.11 0.18 0.52 0.47 0.87
6. Business processes fit 0.16 0.18 0.68 0.57 0.40 0.92
7. Social influence 0.07 0.16 0.25 0.29 0.26 0.45 0.83
8. WC 0.14 0.12 0.64 0.58 0.38 0.72 0.36 0.91
9. U 0.16 0.17 0.58 0.57 0.35 0.67 0.41 0.74 0.94
10. EOU 0.19 0.22 0.55 0.58 0.47 0.49 0.22 0.61 0.63 0.93
11. AT 0.11 0.14 0.52 0.54 0.33 0.58 0.32 0.70 0.74 0.59 0.95
12. EU 0.25 0.02 0.27 0.35 0.16 0.39 0.26 0.45 0.41 0.31 0.41 0.87
a
Square root of AVE on diagonal.
192
Table 5. Parameter estimation of the PLS model by the bootstrap method and effect size
Standardized path
Dependent variable Independent variable t-values f2
coefficient
WC OPC 0.39 6.83** 0.21b
STC 0.42 7.32** 0.25b
U OPC 0.27 5.28** 0.12a
WC 0.40 6.34** 0.19b
EOU 0.27 4.64** 0.12a
EOU PCIL 0.11 2.12* 0.02a
SCT 0.61 13.17** 0.61c
AT WC 0.30 3.56** 0.09a
U 0.44 5.92** 0.21b
EOU 0.14 2.28* 0.03a
EU WC 0.32 4.41** 0.06a
AT 0.19 2.59** 0.03a
Notes: **p<0.01, *p<0.05.
are shown in Table 5 and Figure 2. EOU has a other relationships between PCIL, STC and OPC
positive and significant effect on U (β = 0.27, on EOU, U and WC, but none of the relationships
p<0.01) and a weaker, but significant effect on AT was significant.
(β = 0.14, p<0.05). U has a strong positive effect Effect size (f2) can be viewed as a gauge for
on AT (β = 0.44, p<0.01). WC has a strong and whether a predictor latent variable has a weak,
significant positive effect on U (β = 0.40, p<0.01). medium or large effect at the structural level
WC also has a relatively weaker but significant (Hensler et al., 2009). In our model average f2
positive effect on AT (β = 0.30, p<0.01) and EU is 0.16 which represent medium effect predictor
(β = 0.32, p<0.01). latent variables at the structural level with none
Effect size (f2) measures impact of independent insignificant impact of independent variables on
variables on depended variables and can be ns not depended variables (Table 5).
significant effect, a small: 0.02, b medium: 0.15 The structural model demonstrates predictive
or c large: 0.35 effect on dependent variable at the power as the variance explained (R2) in key endog-
structural level (Henseler et al., 2009). enous constructs and is 0.54 for WC, 0.64 for U,
As it can be seen from Table 6 and Figure 0.42 for EOU, 0.62 for AT and 0.22 for EU. All of
2, the loadings of the first-order factors on the R2 can be described as “moderate” by Chin (1998),
second-order factors exceed 0.70 and second-order except EU which can be described as “weak”. The
factors have significant positive effect on WC, U findings show that our model explains large part
and/or EOU. PCIL has weak but significant posi- of variance in the endogenous variables, with an
tive effect on EOU (β = 0.11, p<0.05). STC has average R2 of 0.49. Communality and redundancy
strong positive effect on EOU (β = 0.61, p<0.01) coefficients are also presented in Table 7. They
and a little less strong effect on WC (β = 0.42, can be used essentially in the same way as the R2,
p<0.01). And OPC has strong positive effect since they reflect the relative amount of explained
on WC (β = 0.39, p<0.01) and on U (β = 0.27, variance for latent and manifest variables.
p<0.01). Furthermore, we have checked also all
193
Figure 2. Results of structural model analysis. (path significance: **p < 0.01, * p < 0.05, n.s. = not
significant (shapes are marked dotted)).
Table 7. Explained variance (R2), communality, redundancy and blindfolding results of cv-communality
(H2) and cv-redundancy (F2)
cv-communality cv-redundancy
R2 Communality Redundancy
(H2) (F2)
OPC 0.54 0.39 0.39
STC 0.49 0.42 0.42
PCIL 0.49 0.28 0.28
WC 0.54 0.84 0.30 0.63 0.43
U 0.64 0.89 0.25 0.79 0.55
EOU 0.42 0.87 0.35 0.51 0.34
AT 0.62 0.90 0.30 0.59 0.54
EU 0.22 0.75 0.14 0.57 0.16
Average 0.49 0.63 a 0.270 0.52 0.39
a
Computed as a weighted average of the different communalities with the weights being the number of manifest variables per each
construct (Tenenhaus et al., 2005).
194
As important part of model evaluation is the on AT (Amoako-Gyampah & Salam, 2004; Nah
examination of fit indexes reflecting the predictive et al., 2004; Shivers-Blackwell & Charles, 2006,
power of estimated inner and outer model relation- Bueno & Salmeron, 2008; Calisir et al., 2009). In
ships, which can be measured by evaluating particular, the study also shows that U serves as a
goodness-of-fit (GoF) coefficient (Tenenhaus et mediating role which enhances the positive effect
al., 2005; Guenzi et al., 2009). The general crite- of EOU on AT of the ERP users in the organization.
rion for evaluating GoF is to calculate the geo- Perceptual construct work compatibility (WC)
metric mean of the average communality and the was presented as degree to which can ERP user do
average R2 of endogenous variables (Tenenhaus most of their tasks in ERP system. In our research
et al., 2005). According to the results in Table WC influences U, which supports prior research-
7GoF is 0.55, which can be considered as satisfac- ers (Sun et al., 2009; Scott & Walczak, 2009).
tory (Tenenhaus et al., 2005). Then, the blindfold- Even more, WC directly and indirectly (through
ing approach proposed by Wold (1982) has been U) influences AT and can be seen as if ERP users
followed to calculate the cv-communality and can do most of their tasks in ERP systems, they
cv-redundancy indexes. The cv-communality have better attitude toward using ERP solutions
index (H2) measures the global quality of the and because of that, they also use ERP system in
measurement model, if they are positive for all extended use. In the routine stage organizations
blocks of variables (Guenzi et al., 2009). Where should put more effort into people and process
the cv-redundancy index i.e. Stone-Geisser’s Q2 improvements (Bradford, 2008). Many people
(which Tenenhaus et al., (2005) calls F2), measures who work with the system only master their
the global quality of the structural model. The particular path and do not attempt to understand
model has predicative relevance if F2>0 (Guenzi the technology used (Boudreau, 2002). In this
et al., 2009). As shown in Table 7, the measure- stage ERP users should accept the system and
ment model (average H 2 is 0.52) shows a little the usage should become a regular day-to-day
better quality than the structural one (average F2 activity. From the survey it can be seen that WC
is 0.39). importantly contributes to EU and because of that
this relationship should be further researched.
The aim of our research was to expose and
SOLUTIONS AND research which factors have influence on the
RECOMMENDATIONS ERP usage in the operation phase. Based on the
analytical results, this study has found that it is
From Figure 2, it can be first noted that the EOU possible to observe more external factors through
has a positive direct influence on the U and on AT. the second-order factors. Research shows that
Second, U has a direct influence on the AT. Third, PCIL has weak but significant positive effect on
the results also show that the indirect effect of EOU. STC has a strong positive effect on EOU
EOU on AT (through U) is greater than the direct and on WC. On the other hand, OPC has a strong
effect of EOU on AT. The results of current study positive effect on U and on WC. Furthermore, we
support the findings of prior researches concern- have checked also all other relationships between
ing EOU influence on U (Amoako-Gyampah & PCIL, STC and OPC on EOU, U and WC, but
Salam, 2004; Hsieh & Wang, 2007; Bueno & none of the relationships was significant.
Salmeron, 2008; Sun et al., 2009; Youngberg et In addition, analytical results of the model
al., 2009; Calisir et al., 2009; Lee et al., 2010), reveal that among four factors within PCIL, which
the influence of EOU on AT (Nah et al., 2004; include experience with a computer, computer self-
Bueno & Salmeron, 2008) and the influence of U efficiency, personal innovativeness toward IT and
195
computer anxiety, only personal innovativeness The impact of OPC on U supports the findings of
toward IT and computer anxiety, appeared to be prior studies (Venkatesh & Davis, 2000; Venkatesh
important for ERP users. These factors appear to & Bala, 2008). Furthermore, statistical analysis
be important during implementation (Davis et reveals that the business process fit is the most
al., 1998; Venkatesh et al., 2003; Thompson et contributing factor in the group OPC. Bancroft
al., 2006; Venkatesh & Bala, 2008; Calisir et al., et al., (2001) and Somers and Nelson (2004),
2009; Shivers-Blackwell & Charles, 2006; Shih pointed out that business process reengineering
& Huang, 2009), but according to our research plays a particular crucial role in the early stages
in the routine stage they become less important. of implementation; it is moderately important
This is consistent with the arguments of Venkatesh in the acceptance stage; and tends to be less im-
and Davis (2000) and Thomson et al., (2006), portant once the technology becomes the part of
who suggest that through experience, the ease of the routine stage. Our study confirms studies of
use perceptions become more rooted at specific authors who argue that business process fit and
features of the software and less influenced by reengineering is a continuous process and is also
general personal traits. included in post-implementation phase (Bradford,
The study also concludes that STC factors (data 2008; Welti, 1999). The factor social influence
quality, system performance, and user manuals) can be described with CSF factors top manage-
are significant in determining EOU and WC, ment support and sponsor which shows that top
while the ERP functionality has not been found management support and sponsorship were critical
as a statistically significant factor of STC. The in all phases of the ERP lifecycle. Nearly all of
quality of data has been cited as an important the research included top management support
factor in the successful ERP implementation and involvement of CSF for the successful ERP
(Gattiker & CFPIM, 2002; Ngai et al., 2008). Our implementation and their inclusion is needed
research shows that for ERP users data quality is throughout the implementation project (Somers
the most important external factor of STC. Exist- & Nelson, 2004; Umble et al., 2002; Gattiker &
ing literature provides evidence in support of our CFPIM, 2002; Bobek & Sternad, 2010). Spon-
findings that for ERP users it is important to get sor plays a critical role in the acceptance of the
real and accurate data at the moment that they technology and he/she is usually at the senior
need it (Gattiker & Goodhue, 2005; Venkatesh, management level so they have the authority to
1998; Venkatesh & Davis, 2000; Kositnurit et al., make substantial organizational changes occur
2006). System performance is concerned with the (Akkermans & Helden, 2002; Ngai et al., 2008).
performance characteristics of the ERP system, Despite the fact that factors, the ERP support (Lee
including reliability, flexibility and response et al., 2010; Bobek and Sternad, 2010), the ERP
time and is an important CSF in the operation communication (Al-Mashari et al., 2003; Somers
phase (Bobek & Sternad, 2010). It was found as & Nelson, 2004; Bobek & Sternad, 2010) and
important factor of STC in the routine stage. Our the ERP training and educations (Motiwalla &
study also exposes that complete and up to date Thompson, 2009; Boudreau, 2002; Akkermans &
user manuals (help) is an important factor of STC Helden, 2002; Al-Mashari et al., 2003; Bancroft
in the routine stage. et al., 2001; Bradford & Florin, 2003; Umble et
The third group of researched external factors al., 2002, Bobek & Sternad, 2010) are often men-
were factors included in OPC. The results of cur- tioned CSF, they have not been found statistically
rent study support that external factors business significant in our study.
process fit and social influence through factor OPC In the routine stage, users should accept the
have significant influence on the ERP usefulness. system and the usage should become a regular
196
activity. It often takes many months or even years of factors which influence the acceptance and use
for experienced users to get comfortable with an of ERP systems. The aim of our chapter was to
ERP system. Eventually, users begin to see the include factors WC and EU in TAM, and to ex-
advantages of the ERP system and they begin tend the number of observed factors which have
to explore its functions, gradually reaching suc- influence on the user acceptance and use in the
cess. This shows that ERP users have accepted routine (mature) stage. Because of observation of
the ERP system and are putting it into extended a large number of external factors, the concept of
use. Although ERP users use ERP systems in the second-order factors was used. The use of the
average almost 5 years in their daily work and concept of second-order factors, together with the
their intensity of job-related ERP system use is use of PLS approach to test our model, allow us to
little bigger than average intensity, their degree use a small sample size. TAM was used because
of ERP system use is moderate. The aim of each it is the most widely used and empirically tested
organization, which has implemented ERP system, model for explaining actual use of IS (Davis, 1989;
should be that ERP users use their ERP solution in Davis et al., 1989; Amoako-Gyampah & Salam,
the extended way. Through the researched model 2004; Lee et al., 2010), despite our research being
we propose the organization to put more effort in focused on external factors and their influence
the exposed external factors which have impact on the actual use of ERP system based on the
through the second-order factors on ERP accep- published research about this issue in the paper.
tance and usage. For more detail interventions in The research improved the understanding
each organization further research (interviews of of how the influence of several external factors
ERP users) should be carried out. can increase the degree of attitude of ERP users
toward the ERP system in the routine stage by
incorporating three groups of external factors:
CONCLUSION personal characteristics and information literacy
(PCIL), system and technological characteristics
The most important contributions of the ERP (STC) and organizational-process characteristics
systems are that they significantly reduce the (OPC). Conceptual factor PCIL includes exter-
time to complete business processes and help nal factors: personal innovativeness toward IT,
organizations to share information (Lee et al., computer anxiety, computer self-efficacy, and
2010), and that organizations usually offer a better computer experience. Data quality, system per-
work environment for their employees as they are formance, user manuals, and ERP functionality
given more efficient system to work with. In the are included in conceptual factor STC. Business
routine stage, ERP systems could be implemented processes fit, social influence, ERP support, ERP
successfully from a technical perspective, but communication, and ERP training are included
success may depend on ERP users being willing in conceptual factor OPC. PCIL, STC and OPC
to use the delivered system (Boudreau, 2002; have been mentioned in several studies published
Kwahk & Lee, 2008). within the context of external factors research in
There are some studies on TAM regarding the field of IS acceptance (some of the authors
ERP systems among which majority are focused related their research eventually to TAM, but
in the selection and implementation phase. Stud- not all). The research shows that PCIL is not so
ies focused in the post-implementation phase are important concerning the use of ERP systems in
scarce and have been published recently after 2009 routine (mature) stage, despite the fact that it was
(Sun et al., 2009; Shin & Huang, 2009; Lee et al., mentioned in some other studies not related to
2010). Majority of them consider limited number ERP systems (Venkatesh et al., 2003; Venkatesh &
197
Bala, 2008). STC and OPC are somehow equally Amoako-Gyampah, K., & Salam, A. F. (2004). An
important but they have impact on different vari- extension of the technology acceptance model in
ables of TAM concerning ERP use. STC external an ERP implementation environment. Information
factors data quality, system performance and user & Management, 41(6), 731–745. doi:10.1016/j.
manuals are significant in determining EOU and im.2003.08.010
WC, while external factors business process fit
Bancroft, N., Seip, H., & Sprengel, A. (2001).
and social influence through factor OPC have
Implementing SAP R/3 (2nd ed.). Greenwich, CT:
significant influence on the U and WC.
Manning Publications Co.
This research has potential for a practical ap-
plication in the degree of ERP system usage. By Bobek, S., & Sternad, S. (2010). Management
confirming external factors, organizations should of ERP solutions. In Vaish, A. (Ed.), Recent ad-
take into consideration exposed and researched vances in management and information security
factors, if they want to improve the degree of the (pp. 31–47). New Delhi, India: Shree Publishers
ERP system usage. & Distributors.
Boudreau, M. C. (2002). Learning to use ERP tech-
nology: A causal model. The 36th Hawaii Interna-
REFERENCES
tional Conference on System Sciences. Retrieved
Adam, F., & Sammon, D. (2004). The enterprise July 7, 2008, from http://csdl2.computer.org/comp/
resource planning decade: Lesson learned and proceedings/hicss/2003/1874/08/187480235b.pdf
issues for the future. Hershey, PA: Idea Group Bradford, M. (2008). Modern ERP – Select,
Publishing. implement & use today’s advanced business sys-
Agarwal, R., & Prasad, J. (1999). Are individual tems. North Carolina State University: College
differences germane to the acceptance of new of Management.
Information Technologies? Decision Sciences, Bradford, M., & Florin, J. (2003). Examining
30(2), 361–391. doi:10.1111/j.1540-5915.1999. the role of innovation diffusion factors on the
tb01614.x implementation success of enterprise resource
Ajzen, I. (1991). The theory of planned behavior. planning systems. International Journal of Ac-
Organizational Behavior and Human Decision counting Information Systems, 4(3), 205–225.
Processes, 50(2), 179–211. doi:10.1016/0749- doi:10.1016/S1467-0895(03)00026-5
5978(91)90020-T Bradley, J., & Lee, C. C. (2007). ERP training
Akkermans, H., & Helder, K. (2002). Vicious and user satisfaction: a case study. International
and virtuous cycles in ERP implementation: A Journal of Enterprise Information Systems, 3(4),
case study of interrelations between CSF. Euro- 33–55. doi:10.4018/jeis.2007100103
pean Journal of Information Systems, 11, 35–46. Bueno, S., & Salmeron, J. L. (2008). TAM-
doi:10.1057/palgrave/ejis/3000418 based success modelling in ERP. Interacting
Al-Mashari, M., Al-Mudimigh, A., & Zairi, M. with Computers, 20(6), 515–523. doi:10.1016/j.
(2003). Enterprise resource planning: Taxonomy intcom.2008.08.003
of critical factors. European Journal of Opera-
tional Research, 146(2), 352–364. doi:10.1016/
S0377-2217(02)00554-4
198
Calisir, F., Gumussoy, C. A., & Bayram, A. (2009). Guenzi, P., Georges, L., & Pardo, C. (2009). The
Predicting the behavioural intention to use enter- impact of strategic account managers’ behaviors
prise resource planning systems – An exploratory on relational outcomes: An empirical study. In-
extension of the technology acceptance model. dustrial Marketing Management, 38(3), 300–311.
Management Research News, 32(7), 597–613. doi:10.1016/j.indmarman.2007.09.011
doi:10.1108/01409170910965215
Henseler, J., Ringle, C. M., & Sinkovics, R. (2009).
Chin, W. W. (1998). Issues and opinion on struc- The use of partial least squares path modelling in
tural equation modelling. Management Informa- international marketing. Advances in International
tion Systems Quarterly, 22(1), 7–16. Marketing, 20, 277–319.
Cooper, R. B., & Zmud, R. W. (1990). Information Hinton, P. R., Brownlow, C., Mcmurray, I., &
Technology implementation research: A techno- Cozens, B. (2004). SPSS explained. London, UK:
logical diffusion approach. Management Science, Routledge Taylor & Francis Group.
36(2), 123–139. doi:10.1287/mnsc.36.2.123
Hsieh, J. J. P. A., & Wang, W. (2007). Explaining
Davis, F. D. (1989). Perceived usefulness, employees’ extended use of complex Informa-
perceived ease of use, and user acceptance of tion Systems. European Journal of Information
Information Technology. Management Infor- Systems, 16(3), 216–227. doi:10.1057/palgrave.
mation Systems Quarterly, 13(3), 319–340. ejis.3000663
doi:10.2307/249008
Insiti, M. (2007). ERP end-users business produc-
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. tivity: A field study of SAP & Microsoft. Keystone
(1989). User acceptance of computer technology: Strategy Research 2007. Retrieved June 12, 2010,
A comparison of two theoretical models. Man- from http://download.microsoft.com
Kelley, H. (2001). Attributional analysis of com-
mnsc.35.8.982
puter self-efficacy. Unpublished doctoral disserta-
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, tion, Richard Ivey School of Business.
intention, and behavior: An introduction to theory
Kositanuri, B., Nqwenyama, O., & Osei-Bryson,
and research. MA: Addison-Wesley Reading.
K. M. (2006). An exploration of factors that impact
Fornell, C., & Larcker, D. F. (1981). Evaluat- individual performance in an ERP environment:
ing structural equation models with unobserv- An analysis using multiple analytical techniques.
able variables and measurements errors. JMR, European Journal of Information Systems, 15(6),
Journal of Marketing Research, 18(1), 39–50. 556–568. doi:10.1057/palgrave.ejis.3000654
doi:10.2307/3151312
Kwahk, K. Y., & Lee, J. N. (2008). The role of
Gattiker, T. F., & CFPIM. (2002). Anatomy of an readiness for change in ERP implementation:
ERP implementation gone awry. Production and Theoretical bases and empirical validation.
Inventory Management Journal, 43(3/4), 96–105. Information & Management, 45(7), 474–481.
doi:10.1016/j.im.2008.07.002
Gattiker, T. F., & Goodhue, D. L. (2005). What hap-
pens after ERP implementation: Understanding the Langenwalter, G. (2000). Enterprise resources
impact of interdependence and differentiation on planning and beyond: Integrating your entire
plant-level outcomes. Management Information organization. Boca Raton, FL: St. Lucie Press.
Systems Quarterly, 29(3), 559–585.
199
Lee, D. H., Lee, S. M., Olson, D. L., & Chung, Pijpers, G. G. M., & Montfort, K. (2006). An
S. H. (2010). The effect of organizational sup- investigation of factors that influence senior
port on ERP implementation. Industrial Man- executives to accept innovations in Information
agement & Data Systems, 110(1-2), 269–283. Technology. International Journal of Manage-
doi:10.1108/02635571011020340 ment, 23(1), 11–23.
Legris, P., Ingham, J., & Collerette, P. (2003). Why Rajagopal, P. (2002). An innovation–diffusion
do people use Information Technology? A criti- view of implementation of enterprise resource
cal review of the technology acceptance model. planning (ERP) systems and development of re-
Information & Management, 40(3), 191–204. search model. Information & Management, 40(2),
doi:10.1016/S0378-7206(01)00143-4 87–114. doi:10.1016/S0378-7206(01)00135-5
Liu, L., & Ma, Q. (2006). Perceived system Ringle, C. M., & Will, A. (2005). SmartPLS 2.0
performance: A test of an extended technology (M3). Retrieved June 1, 2009, from: http://www.
acceptance model. Journal of Organizational and smartpls.de
End User Computing, 18(3), 1–24.
Rogers, E. V. (2003). Diffusion of innovation (4th
Lu, J., Chun-Sheng, Y., Liu, C., & Yao, J. E. ed.). New York, NY: The Free Press.
(2003). Technology acceptance model for wireless
Saeed, K. A., & Abdinnour-Helm, S. (2008).
Internet. Internet Research: Electronic Network-
Examining the effects of Information System char-
ing Applications and Policy, 13(3), 206–222.
acteristics and perceived usefulness on post adop-
doi:10.1108/10662240310478222
tion usage of Information Systems. Information
Momoh, A. (2010). Challenges in enterprise re- & Management, 45(6), 376–386. doi:10.1016/j.
source planning implementation: State-of-the-art. im.2008.06.002
Business Process Management Journal, 16(4),
Schwarz, A. (2003). Defining Information Tech-
537–565. doi:10.1108/14637151011065919
nology acceptance: A human-centred, manage-
Motiwalla, L. F., & Thompson, J. (2009). En- ment-oriented perspective. Unpublished doctoral
terprise systems for management. Upper Saddle dissertation, University of Huston.
River, NJ: Pearson Prentice Hall.
Scott, J. E., & Walczak, S. (2009). Cognitive en-
Musaji, Y. F. (2003). Integrated auditing of ERP gagement with a multimedia ERP training tool:
systems. New York, NY: John Wiley & Sons, inc. Assessing computer self-efficacy and technology
acceptance. Information & Management, 46(4),
Nah, F. F., Tan, X., & Teh, S. H. (2004). An em-
221–232. doi:10.1016/j.im.2008.10.003
pirical investigation on end-users’ acceptance
of enterprise systems. Information Resources Shih, Y. Y., & Huang, S. S. (2009). The actual
Management Journal, 17(3), 32–53. doi:10.4018/ usage of ERP systems: An extended technology
irmj.2004070103 acceptance perspective. Journal of Research
and Practice in Information Technology, 41(3),
Ngai, E. W. T., Law, C. C. H., & Wat, F. K. T.
263–276.
(2008). Examining the critical success factors in
the adoption of enterprise resource planning. Com-
puters in Industry, 59(6), 548–564. doi:10.1016/j.
compind.2007.12.001
200
Shivers-Blackwell, S. L., & Charles, A. C. Umble, E. J., Haft, R. R., & Umble, M. M. (2002).
(2006). Ready, set, go: Examining student Enterprise resource planning: implementation
readiness to use ERP technology. Journal of procedures and CSF. European Journal of Opera-
Management Development, 25(8), 795–805. tional Research, 146(2), 241–257. doi:10.1016/
doi:10.1108/02621710610684268 S0377-2217(02)00547-7
Simon, S. J., & Paper, D. (2007). User acceptance Uzoka, F. M. E., Abiola, R. O., & Nyangeresi, R.
of voice recognition technology: An empirical (2008). Influence of product and organizational
extension of the technology acceptance model. constructs on ERP acquisition using an extended
Journal of Organizational and End User Comput- technology acceptance model. International
ing, 19(1), 24–50. doi:10.4018/joeuc.2007010102 Journal of Enterprise Information Systems, 4(2),
67–83. doi:10.4018/jeis.2008040105
Somers, M. T., Nelson, K., & Karimi, J. (2003).
Confirmatory factor analysis of the end-user Venkatesh, V. (1998). User acceptance of infor-
computing satisfaction instrument: Replication mation technology: A unified view. Unpublished
within an ERP domain. Decision Sciences, 34(3), doctoral dissertation, University of Minnesota.
595–621. doi:10.1111/j.1540-5414.2003.02428.x
Venkatesh, V. (2000). Determinants of perceived
Somers, T. M., & Neslon, K. G. (2004). A taxono- ease of use: Integrating perceived behavioural
my of players and activities across the ERP project control, computer anxiety and enjoyment into
life cycle. Information & Management, 41(3), the technology acceptance model. Information
257–278. doi:10.1016/S0378-7206(03)00023-5 Systems Research, 11(4), 342–365. doi:10.1287/
isre.11.4.342.11872
Straub, D. (1989). Validating instruments in
MIS research. Management Information Systems Venkatesh, V., & Bala, H. (2008). Technology
Quarterly, 13(2), 147–169. doi:10.2307/248922 acceptance model 3 and a research agenda on
interventions. Decision Sciences, 39(2), 273–315.
Sun, Y., Bhattacherjee, A., & Ma, Q. (2009).
doi:10.1111/j.1540-5915.2008.00192.x
Extending technology usage to work settings:
The role of perceived work compatibility in ERP Venkatesh, V., & Davis, F. D. (2000). A theo-
implementation. Information & Management, retical extension of the technology acceptance
46(6), 351–356. doi:10.1016/j.im.2009.06.003 model: Four longitudinal field studies. Manage-
ment Science, 46(2), 186–205. doi:10.1287/
Tenenhaus, M., Vinzi, V. E., Chatelin, Y. M., &
mnsc.46.2.186.11926
Lauro, C. (2005). PLS path modelling. Computa-
tional Statistics & Data Analysis, 48(1), 159–205. Venkatesh, V., Morris, M. G., Davis, G. B., & Da-
doi:10.1016/j.csda.2004.03.005 vis, F. D. (2003). User acceptance of Information
Technology: toward a unified view. Management
Thompson, R., Compeau, D., & Higgins, C.
(2006). Intentions to use Information Technolo-
gies: An integrative model. Journal of Organiza- Wallace, T. F., & Kremzar, M. H. (2001). ERP:
tional and End User Computing, 18(3), 25–46. making it happen – The implementer’s guide to
doi:10.4018/joeuc.2006070102 success with enterprise resource planning. New
York, NY: John Wiley & Sons.
201
Wang, E. T. G., Shih, S.-P., Jiang, J. J., & Klein, functional departments, from product planning,
G. (2007). The consistency among facilitating fac- parts purchasing, inventory control, product dis-
tors and ERP implementation success: A holistic tribution, to order tracking. They usually include
view of fit. Journal of Systems and Software, application modules for the finance, accounting
81(9), 1609–1621. doi:10.1016/j.jss.2007.11.722 and human resources aspects of a business.
Extended Use (EU): Is the use behavior where
Welti, N. (1999). Successful SAP R/3 implemen-
ERP user uses more functionality of ERP system
tation – Practical management of ERP project.
than typical users.
Addison-Wesley.
Organizational-Process Characteristics
Wold, H. (1982). Soft modelling: The basic de- (OPC): Capture various social processes, mecha-
sign and some extensions. Systems under indirect nisms and support organizations that guide indi-
observations (Vol. 2, pp. 1–52). Elsevier Science. viduals to facilitate the use of an ERP system.
Partial Least Squares (PLS) Approach: Is
Yi, Y. M., Fiedler, K. D., & Park, J. S. (2006).
one of structural equation modeling (SEM) which
Understanding the role of individual innovative-
can be used for situations with less theory develop-
ness in the acceptance of IT-based innovativeness:
ment, smaller sample size and it is better suited
Comparative analyses of models and measures.
for analyzing complex relationships and models.
Decision Sciences, 37(3), 393–426. doi:10.1111/
Personal Characteristics and Information
j.1540-5414.2006.00132.x
Literacy (PCIL): Includes personality charac-
Youngberg, E., Olsen, D., & Hauser, K. (2009). teristics that can have influence on individuals’
Determinants of professionally autonomous end perceptions of IS acceptance and usage.
user acceptance in an enterprise resource plan- System and Technological Characteristics
ning system environment. International Journal (STC): Are silent features of ERP systems that
of Information Management, 29(2), 138–144. can help individuals develop favorable/unfavor-
doi:10.1016/j.ijinfomgt.2008.06.001 able perceptions regarding IT/IS.
Technology Acceptance Model (TAM):
Is most widely used model for explaining the
behavioral intention and actual usage of IT/IS
KEY TERMS AND DEFINITIONS proposed by Davis (1989).
Work Compatibility (WC): Is the degree
Enterprise Resource Planning (ERP) Sys-
to which can ERP user do most of their tasks in
tems: Are modular solutions based on principles
ERP system.
of best practices which integrate activities across
202
APPENDIX A: QUESTIONNAIRE ITEMS
Variable Source Scale Measurement items

Personal innovative- Yi et al., 2006; pcil_pi1 If I hear of a new IT, I look for ways to experiment with it.
ness Thompson et al.,
pcil_pi2 Among my peers I am usually the first to try out new IT.
2006.
pcil_pi3 I like to experiment with new IT.
Computer a Venkatesh, 1989; pcil_ca1 a
Working with a computer makes me nervous.
Venkatesh et al.,
pcil_ca2a I get a sinking felling when I think of trying to use a computer.
2003.
pcil_ca3 I feel comfortable working with a computer.
Computer self- Venkatesh et al., pcil_cse1 I can complete the job using any program if I have only software manu-
efficacy 2003; als or/and build-in help for assistance.
Thompson et al.,
pcil_cse1 I can complete the job using any program if I can call someone for help
2006.
if I am stuck.
ERP data quality Gattiker & Goodhue, Stc_dq1 The ERP system provides the precise information I need.
2005;
Stc_dq2 The information contents provided by the ERP system meet my needs.
Venkatesh, 1998;
Kositanuri et al., Stc_dq3 The ERP system provides reports that seem to be exactly what I need.
2006.
Stc_dq4 The ERP system provides sufficient information to my needs.
Stc_dq5 The ERP system provides all features I need.
Stc_dq6 I am satisfied with the speed of interacting with the system.
ERP system perfor- Venktesh et al., 2003; stc_sp1 It is fast to search data in the ERP system.
mance Kositanuri et al.,
stc_sp2 The ERP system loads quickly.
2006;
Liu and Ma, 2006. stc_sp3 I was able to retrieve data quickly.
stc_sp4 It is fast to create a new record in this system.
stc_sp5 It is fast to use this ERP system.
User manuals (help) Kelley, 2001. stc_um1 The content and index of the user manuals are useful.
stc_um2 The user manuals are current (up to date).
stc_um3 The user manuals are complete.
ERP functionality Kositanuri et al., stc_fu1 It is easy to detect possible errors in the ERP system.
2006.
stc_fu2 It is easy to correct errors that happen in the ERP system.
stc_fu3 It is easy to change the output format.
ERP business pro- Amoako-Gyampah & opc_bpf1 The ERP system fits well with my business needs.
cesses fit Salam, 2004; Nah et
opc_bpf2 The ERP system fits well with the business needs of my department.
al., 2004.
opc_bpf3 All in all, the ERP system is satisfactory in meeting my needs.
Social influence Venkatesh et al., opc_si1 My supervisor is very supportive at the use of the ERP system for my
2003; job.
Venkatesh, 1998.
opc_si2 In general, the organization has supported the use of the ERP system.
opc_si3 People who influence my behaviour think that I should use the ERP
system.
opc_si4 People who are important to me think that I should use the ERP system.
ERP support Kelley, 2001. opc_su1 I do not know who to phone for support of ERP system.
opc_su2 The support people talk in terms that I do not understand.
opc_su3 I ask other users for help with this application rather than the support
staff.
203
Variable Source Scale Measurement items

ERP communication Kelley, 2001. opc_cu1 The ERP team does not inform me about the current situation of ERP
system.
opc_cu2 The ERP team does not explain how application modifications will
have impact on my job.
ERP training Amoako-Gyampah & opc_tr1 I have received additional formal training for ERP since the conclusion
Salam, 2004; Kosi- of the initial training.
tanuri et al., 2006;
opc_tr1 I have received informal training (e.g. half hour of support from a peer
Kelley, 2001.
or training officer) for ERP system.
ERP usefulness (U) Davis, 1989; us1 Using ERP system in my job enables me to accomplish tasks more
Venkatesh & Davis, quickly.
1996.
us2 Using ERP system improves my job performance.
us3 Using ERP system enhances my effectiveness at the job.
us4 Using ERP system makes it easier to do my job.
ERP ease of use Davis, 1989; eou1 My interaction with ERP system is clear and understandable.
(EOU) Venkatesh & Davis,
eou2 I find ERP system easy to use.
1996.
Work compatibility Nah et al., 2004; wc1 Using ERP system is compatible with all aspects of my work.
(WC) Yi et al., 2006.
wc2 Using ERP system fits well with the way I like to work.
wc3 Using ERP system fits with my work style.
Attitude toward using Venkatesh et al., at1 Using the ERP system is a good idea.
ERP system (AT) 2003;
at2 I like the idea of using the ERP system to perform my job.
Nah et al., 2004;
Amoako-Gyampah
&Salam, 2004.
Extended use (EU) Nah et al., 2004; au1 I would rate the intensity of my job-related ERP system use to be…
Schwarz, 2003.
au2 In a typical one-month period, what is the likelihood of you using most
of the features of the ERP system?
au3 In a typical one-month period, what is the likelihood of you using more
features than the other users of the ERP system?
au4 In a typical one-month period, what is the likelihood of you, using more
obscure aspects of the ERP system?
204
205
Chapter 11
Specific Factors for ERP-Success
Measurement in Healthcare
Stephan Kronbichler
UMIT, Austria
Herwig Ostermann
UMIT, Austria
ABSTRACT
There are different success factors that influence the outcome of ERP-projects. According to Bullen and
Rockart (1981), these success factors depend on five main influencing variables. One of these variables is
industry. This shows that each industry has certain requirements on success measurement other industries,
in general, do not. In healthcare for example the legal constraints are different from other branches.
Each success measurement model has a specific aim and covers specific needs. It measures different
success dimensions and can be used at different stages of ERP-projects. Some of the models consider
different phases within ERP-projects others can be used to determine overall ERP-success after the
implementation of an IS. One important factor for the use of such models is if the models covers branch
specific needs in a way that allows the researchers to allocate meaningful results and recommendations
for actions. This chapter investigates the specific needs on ERP-success measurement of the healthcare
industry and the fit of the existing success measurement models for this use case.
INTRODUCTION measurement models which were identified

through literature review and practical experi-
The target of this chapter is the conceptual design ences. ERP projects usually deal with managerial/
of possibilities how ERP (Enterprise Resource economic information systems development. In
Planning)-projects can be more successful when the healthcare context the focus is more on the
considering critical success factors and success integration of clinical systems like the HIS (health
information systems), the EHR (electronical
DOI: 10.4018/978-1-4666-0170-3.ch011 health record), the PACS (picture archiving and
Specific Factors for ERP-Success Measurement in Healthcare
communication system), the MPI (master patient the business change which took place over the
index), the RIS (radiology information systems last years. Costs are more important for healthcare
and the LIMS (laboratory information manage- institutions than 20 years ago. The healthcare
ment systems. IS (Information system) projects are sector is now challenging with similar issues as
often very difficult to handle and many companies the private sector companies.
spend a lot of money in implementing new, state Another objective of this chapter is the field
of the field solutions to stay competitive or get of critical success factors which were identified
competitive advantage (Gronau, 2001; Nah, et. by literature review and to figure out if there are
al., 2001; Esteves, et. al., 2002). Another reason specific success factors within the healthcare
for starting new risky IS-projects is the fact that industry or not. As a fist implication of this the
a lot of legacy systems run out of support and success factors which seem to occur just during
these systems cannot handle the business needs implementations or projects in healthcare are de-
of nowadays. One problem with legacy system scribed and an explanation for this is demonstrated.
is the compatibility to the new operating systems In the section with the success measurement
which do not allow running the legacy system models the author lists the possibilities of how
frontend anymore. Therefore old hard- and soft- success of ERP systems can be measured through
ware must be used. success measurement approaches which were
There is no ERP system which reflects all identified in the literature. The different areas of
corporate processes, it is possible however, to application, the strengths and the weaknesses of
reconsider the process landscape in the company the models are shown. At the end of the chapter
and adapt the processes to the software. This the success measurement models of the health
results in a standard solution or a solution which informatics are investigated and compared to the
is close to the standard (“vanilla ERP”). Vanilla more general models identified before. Interesting
ERP offers advantages compared with modified similarities were found but there are also some
solutions in the maintenance, upgrades, the abil- differences existing.
ity of manufacturer support and the support by In general success measurement projects are
external consultants. Adapted ERP systems offer usually divided in the following three categories
the advantage that the adjustments help to support (Auer, 2004):
the existing processes and that no or only slight
Business Process Reengineering (BPR) has to be • Ex-Post measurement: What benefits or
carried out. This is particularly important for the harms were provided or caused by ICT
replacement of legacy systems because the users systems in the past?
are used to do things in a certain way and some • Ex-ante measurement: What benefit or
companies don’t have the willingness to change. harm will provided or caused by ICT sys-
There are also industry-specific solutions (specific tems in the future?
ERP solutions which support specific processes • Real-time observation: When, where and
of various industries, e.g. retail, automotive or the how must be intervened to realize the po-
meat industry), which allows a solution close to the tential benefits and prevent potential dam-
standard. These branch solutions are sometimes age caused by a non working system?
offered by the main software vendor itself and
sometimes by 3rd party suppliers. This chapter focuses on models that can be
One objective of this chapter is to show the used for ex-post analysis and models for ex-ante
increasing importance of ERP-implementations analysis. Real-time considerations are providing
in healthcare institutions. One reason for that is more ongoing monitoring of systems to identify
206
bottlenecks or potential for improvement. The DRG as the new costing and budgeting standard
models in this chapter were selected because has changed the industry. According to Münch
they follow either a very specific approach and (2002) the DRG is a patient classification system
are different from other models or represent a which can be used for systematical assignment
very important publication in this field, a kind of of analogue cases to homogeneous cost groups
reference literature which is basis for many other considering medical grouping criteria.
researches. The identified success measurement DRG systems try to put the economic and
models for this investigation are therefore: clinical-medical perspective on a common de-
nominator. The patients and their treatment is
• The I/S success model of DeLone and roughly divided into two groups (Fischer, 2000):
McLean (1992, 2002, 2003)
• The Gable et al. (2003) model • clinically defined groups (usually after di-
• The Extended ERP Systems Success agnosis and treatment)
Measurement Model of Ifinedo (2006) • Groups with similar treatment costs
• Enterprise system experience cycle by
Markus and Tanis (2000) The aim of the clinical side is optimal treat-
• Ex-Ante Evaluation of Stefanou (2001) ment and will therefore try to group the patients in
• The Task Technology Fit (TTF) published similar cases based on problems and goals in the
by Smyth (2001) treatment. This is achieved using so-called care
• The FITT model of Ammenwerth (2003) paths. The economic side, usually represented by
• The benchmark approach of Hübner- the hospital administration or cost carrier, tries to
Bloder and Ammenwerth (2009) group the patients on the basis of incurred costs.
This side wants to define production and sales
units respectively cost objects based on the indi-
BACKGROUND vidual costs of a patient. At company level, it’s
really about balancing and planning of costs and
Business Changes in Healthcare results. From a political and external point of view
in the Course of Time the most important issue is the cost containment
which was not always considered before DRG
From a business point of view the increasing (Fischer, 2000).
importance of state of the art information systems DRG causes a change from per diem allow-
in healthcare facilities can be explained with ance to case allowances and that leads to a more
fundamental changes in the healthcare institu- economically thinking within healthcare institu-
tions. Years ago the hospitals had a global budget tions. Healthcare institutions have to reconsider
which was the main concern within the industry. their range of services and try to find new sources
The hospitals got a budget which was based on of finance. From business site of view this is
historical data. The legal constraints disposed the reason for the importance of ERP-systems
that supererogation and underperformance were in healthcare. ERP-systems help to handle cost
accordingly considered in the budget. Nowadays transparency and implement standardized pro-
the discussion moved to a more fair allocation cesses to do the daily business in an effective
of money in relation to the performance of the way. Integrated data with standardized reports
hospital (Mansky, 2001). This was caused by and real time financial data within the system
the implementation of the DRG (Diagnosis Re- help the management to get appropriate data for
lated Groups) in a lot of countries. Implementing their analysis of the cost structures.
207
The increasing importance of the computer During the 1990s, ERP systems in healthcare
science in the healthcare industry from a techni- were just best-of-breed solutions which had little
cal point of view is primarily due to the fact that connection to each other. Between systems was
the trend seen for a long time is the change from hardly any interaction and each system was run
isolated applications to fully integrated informa- by different departments and each department
tion systems. These fully integrated systems help maintained their own data. The management of
to fulfill the nowadays needs. The digital storage the healthcare institutions was dealing with lim-
of documents and images in a central system, with ited information, regarding the scope, the timely
centralized authorization management points to an procurement and the availability of information
increasing integration of the systems. However, (Wyatt, 2002). Many decisions were made with-
this results in an increased dependence regarding out complete and timely reporting data. This was
the availability of the systems and their correct the case in other industries too. Companies have
functioning. Because of the demographic trend chosen for all the different sections the suppos-
in industrialized nations, healthcare systems are edly best solution and this solution was integrated
facing new challenges. This primarily results in into the system environment. Today, however,
the increase of chronic diseases, increasing expen- companies primarily want to implement systems
diture for the healthcare and the right of people with a uniform database. Often special software
to modern medical care which assumes ethical is indeed still used for certain areas. But this
dimensions (Haas, 2004). Lenz and Kuhn (2010) software is linked to the central system and can
have illuminated the development of information access the data to avoid redundancy. In addition,
systems and approaches in healthcare. Therefore all data usually can be achieved through a central
the authors analyzed the available standards and system with central user management.
tools. In principle there are two basic approaches: To understand the needs of the healthcare and
afterwards the specific needs regarding ERP-
• Best-of-breed approaches success measurement in the healthcare industry
• ERP approaches the system landscape in a typical healthcare en-
vironment needs to be explained. In healthcare
The advantage of best-of-breed approaches is the following systems are usually implemented:
that for each part (part means functionality which
could be covered by one single system too) of the • Radiological information system (RIS):
system the best part can be selected. However, The main task of a RIS is usually the docu-
many interfaces between the systems are necessary mentation and management of data and
to maintain the necessary data exchange. In ERP processes such as the creation of radiologi-
systems a more integrated approach is chosen. The cal findings or the scheduling of radiologi-
master data is available only one time centrally cal investigations (Frodl, 2010)
and motion data is updated directly to other mod- • Master patient index (MPI): The MPI con-
ules. That means that there are no redundancies tains information on all patients of a partic-
and duplication of data is no longer necessary. If ular hospital (or hospital affiliation). That
systems need to access the data which are in the means the name, date of birth, sex, address,
ERP then the ERP-is usually the “leading system” treatments, according dates and number of
in which the data are maintained and from which the clinical record (Claeys, 1996)
data are sent to the other systems (or a connection • Electronic health record (EHR): The EHR
via real time interfaces is implemented). provides clinicians of any profession or
specialty with an integrated view of the
208
complete health and healthcare history of ERP / HEALTH INFORMATION

each patient under their care (Kalra and SYSTEM IMPLEMENTATIONS IN
Ingram, 2006) THE HEALTHCARE INDUSTRY
• Picture archiving and communication
system (PACS): PACS combines imaging The low number of publications in the healthcare
systems such as for example X-ray, mag- related ERP field required the usage of publica-
netic resonance imaging, computer tomog- tions that deal with critical success factors during
raphy, ultrasound or endoscopes and pro- projects which investigated the implementation
vides therefore a database-driven archive of clinical health information systems.
(Bärwolff et al., 2004). Catholic Healthcare West began in 1996 with
• Laboratory information management sys- the SAP implementation project and invested 120
tem (LIMS): The LIMS has a database that million U.S. dollars until they decided to stop the
contains the test plans from the laboratory project in 2002. SAP is an enterprise resource
which have to be processed. The test re- planning system which can support healthcare
sults are then stored in a database and can institutions in their daily business when e.g. order-
be compared with the test specifications. ing medical equipment or pharmaceuticals. The
If the comparison is positive the sample is industry solution IS-H provides hospitals with the
released for further processing (Früh et al., specific functionality needed in the healthcare,
2008). e.g. patient management, the medical or care
• Document management system (DMS): documentation and the settlement of the patients.
Due to legal reasons and requisite evi- In 2002, the company decided to introduce the
dence needs in hospitals there are amounts software from Lawson in 40 sites as SAP did not
of data occurring. Its handling must be fast, meet the expectations. A company internal study
efficient and carried out by systematic ac- found that SAP is more suitable for manufactur-
cess. The systematic retrieval of informa- ing companies and not for the healthcare sector.
tion from different sources is an essen- The new ERP system saved the company $ 94
tial foundation for every time accessible million over 8 years – this was the result of a
knowledge (Gulbins et. Al., 2002). calculation done by the management. The main
• Clinical information system: A clinical problem with the second ERP implementation
information system provides support for was the acceptance and the resistance of users
nursing and medical activities. These ac- due to the unsuccessful implementation of SAP.
tivities include the medical documentation, By working with key users and the definition of
meal planning, vital signs documentation, “Change Champions” within the second project
medical reports and the service planning this project was completed successfully in spite
(Frodl, 2010) of all reservations. Often change management
• Interfaces: Due to the fact that the systems in large ERP projects is more important than the
have to exchange data with each other (e.g. software to be implemented (Havenstein, 2005).
patient data or findings), it is also necessary Huq and Martin (2006) examined business
to provide appropriate interfaces. These process reorganization (BPR) in a hospital. In the
interfaces can have various forms such as investigation there were two kind of BPR causes
direct database access or remote function which were investigated. The first one was the
calls which return the necessary data. BPR which was initiated by the management.
The second one was the BPR which was neces-
sary since the hospital decided to implement SAP.
209
It was necessary to align the business processes found out that the effectiveness and efficiency in
according to SAP to match best practices for the terms of ordering processes was increased by the
healthcare sector. The reorganization was car- introduction of SAP IS-H. The new solution did
ried out by the IT department in collaboration include the following benefits:
with business managers. The introduction in the
hospital went through the following phases: the • Inventory reduction for drugs and medical
preparation phase, the analysis phase, design phase materials
and final phase. The realignment of the business • Increased productivity in the planning
processes already took place in the analysis phase. processes
The implementation team documented the exist- • Improvement of the quality of information:
ing processes and put this knowledge to SAP real time information about the availabil-
consultants. The consultants tried to implement ity of inventories and the status of orders;
these processes within the SAP system. Identified visibility, better quality and more timely
gaps between existing processes and the possible information which can lead to better deci-
processes in the system delayed the go-live for sion-making in healthcare
6 months. This project seems to contradict with • Common data and practices in all of the
the studies made by Soh et al. (2003). During a different departments which led to less
case study in a hospital in Singapore the authors confusion about orders and improved the
found out that the users in most cases can live communication between departments
with gaps in the functionality of the ERP-product • Automatically generated list of require-
and tolerate this. ments which result from clinical orders
Monteiro (2003) investigated the integra- (bill of material)
tion possibilities of information systems in the • Accurate accounting (no loss of income)
healthcare industry. Furthermore he extended • Real-time updates of patient data
his research to ERP systems too. The author was • The existence of data on the ordered / ap-
referring to the Norwegian hospitals and the local proved drugs for patients
system environment. A high degree of integration • Traceability of closed orders
of individual systems in the system landscape is
hard to implement due to the fact that each of the Boostra and Govers (2009) developed a theo-
various installations has different data storage. retical model which can be used to investigate how
To access these data storages is often very much the participation and the behavior of stakeholders
effort and a problem for the integration efforts.. influence the outcome of the ERP implementation
A similar study was done by Stefanou and Re- in hospitals. During the field test of the model
vanoglou (2006) for an ERP project in a hospital they included interests, problems and possible
in Greece. At the beginning of their publication, solutions recorded and analyzed (regarding the
the integration of ERP and EAI (Enterprise Ap- individual stakeholders). The authors found out
plication Integration) in the hospital was analyzed. that it is within the discretion of the hospital doctors
The authors found that the system infrastructure in to decide whether they own their own practices
healthcare (as shown in Chapter 4.1) has special- and processes or not. ERP systems, however,
ized systems and the seamless integration between were created to implement standardized business
these systems and the ERP systems is required for processes. Because of this, the processes repre-
successful integration. sented in the system are sometimes contrary to
As a result of the investigation examined by the interests and the views of doctors. The doctors
Revanoglou and Stefanou (2006) the authors have the power to decide whether an adjustment
210
is needed to the existing processes or the system of privacy increase. In the United States the HIPAA
is not appropriate. In the worst case, this can lead (Health Insurance Portability and Accountability
to a rejection of the system. Act) is situated. This institution was adopted to
According to Peabody (2002) a hospital in simplify transactions in healthcare and protect
Michigan has chosen a completely different personal data. Pumphrey et al. (2007) examined
way. The managers decided not to customize the ERP systems and their HIPAA compliance. ERP
system and to do a complete redesign of business systems allow the health facilities to pass a lot
processes within the hospital. The main reasons of responsibility regarding HIPAA to the ERP
were the costs and the greatly shortened introduc- vendor. According to Pumphrey et al. (2007) a
tion phase of the ERP system if the system was a comprehensive authorization concept within the
standard system. In the first year by standardizing meaning of HIPAA can be very expensive for the
the product master data, purchasing processes healthcare institution. Data protection guidelines
and the transition to ERS (Electronic Receipt must be developed, implemented, monitored and
Settlement) the hospital has already achieved cost continuously maintained and expanded.
savings of approximately $ 4 million. Heeks et al. (1999) examined why Information
According to Lillis (2002), the main problem systems are used in healthcare on one hand suc-
of ERP implementations in the healthcare sector is cessful and on the other hand the implementation
the integration of the ERP system into the existing fails. The authors justified the increased risk of
system landscape. ERP-systems are compared to failure the following way:
clinical information systems not designed for a
specific industry and the needs of this industry. • Thinking - reality gap: the thinking behind
The clinical information systems are only used in the development of the hospital informa-
healthcare and for this area of application those tion systems does not necessarily coincide
systems are often more advanced. ERP-systems with the processes of the hospital
have a broader range of application. The legacy • Private - public sector gaps: Caused by the
systems are often for a decade or longer in use development of applications for the private
and therefore it is not always guaranteed that the sector which are then used in public areas
systems can communicate. • Country-specific issues: The systems are
But the processes in the industry-specific ERP not deployed in the country in which they
systems have now been strongly adapted to the are developed
best practices in healthcare industry. This was
shown by Kontio et al. (2009) in an investigation Success Factors and
which dealt with the key factors of successful Success Measurement
maintenance processes of cardiac patients in
the emergency rooms. The authors came to the In the first part of this section the importance of
conclusion that there are numerous events in the critical success factors for IS implementations in
emergency room, which can be supported by general is shown. Afterwards the success factors
ERP systems. Furthermore the system can help within ERP-projects in general are compared to
in decision making. CSF in the healthcare specific projects. This should
Another characteristic of the healthcare sector demonstrate that the CSF are in generally similar,
is the sensitivity of the stored data. As there is more but there are some CSF which are very specific
and more healthcare information captured and for this branch and these are not considered in
distributed via various interfaces electronically other industries. At the end of this section the
worldwide so does the concern for the protection most important success measurement models are
211
demonstrated and the difference between the com- author did this with a generalist approach, i.e. he
mon models and the healthcare specific models limited the investigation of success factors not to
is shown. This should demonstrate if there are specific systems but expanded his investigation to
specific needs or “common” models can be used. the entire healthcare sector. Using various stories
with the storytelling approach the following criti-
Critical Success Factors cal success factors (for the introduction of new
in ERP Projects technologies): dislike of change / experience in
the industry, training, stakeholder engagement,
According to Bullen and Rockart (1981), the communication of results during and after the
critical success factors of 5 areas are dependent: introduction, professional handling of “Forced
breaks”.
• The industry: Every industry has its own Many companies in the healthcare sector are
CSF which are characterized by the pecu- using ERP systems. The critical success factors
liarities of the industry within the field of clinical information systems
• The competitive strategy and position in differ little from the CSF occurring during ERP
the industry every business is in a unique projects. Paré et al. (2008) investigated CSF in
situation, which is due to the development the area of clinical information systems and cre-
of the company in the past and the com- ated afterwards a collection. This collection was
petitive strategy. Moreover, the position in then validated by a Delphi study with 21 experts.
the industry is of importance. Sellito and Carbone (2007) studied the success
• environmental factors: These factors can factors in a project at a local hospital in Australia.
not be influenced by the company (the Trimmer et al. (2002) investigated critical suc-
economy, politics, legal framework, ...) cess factors during ERP implementation projects
• Temporary factors: These factors have an in hospitals. The investigation of the authors was
effect for a given period to the company if based on publications in the field of Critical success
something extraordinary happened (e.g. a factors (CSF), which were adapted for the health-
plane crash in which members of manage- care sector. Among other things the CSF model
ment lost their lives) of Rockart (1979, 1981), who is considered to be
• Leadership: critical success factors are also the founder of the CSF method, was used. Rockart
dependent on the leadership and the man- identified CSF for the healthcare industry already
agement. Different managers (e.g. produc- in 1979 which are due to technological change not
tion company or CRM-related companies) valid anymore. The CSF method concentrates on
have different CSF, which are important to the few key areas of activity in which favorable
them. results are necessary to reach the goals. Because
these areas of activity are crititcal, the manager
This demonstrates that for CSF influencing should have the appropriate information to allow
IS-projects the industry and the environmental him to determine if the events in each area are pro-
factors play an important role. That means that ceeding sufficiently or not (Bullen, Rockart; 1981).
each industry has specific factors which are in- Trimmer et al. (2002) investigated a clinic and a
fluencing the outcome of IS-projects. Some are hospital regarding critical success factors during
very similar and some are more different. the ERP implementation. The authors analyzed
Leonard (2004) investigated why the number 10 critical issues on their importance and then
of successful implementations of information tech- they lined them up according to their importance.
nology in the healthcare industry was so low. The
212
The “critical aspects” published by Trimmer et al. A comparison of the healthcare specific in-
(2002) are (ranked by their importance): vestigations with the results of Kronbichler et al.
(2009) shows that there are healthcare specific
• Selecting the Right People CSF which cannot be assigned to one of the CSF
• staff morale identified by Kronbichler et al. (2009).
• approval of top management Paré et al. (2008) have identified a total of 3 risk
• reengineering factors, which do not fit into the classification of
• Integration CSF provided by Kronbichler et al. (2009). “Legal
• Training of staff and ethical constraints” probably relates mainly
• implementation costs to the healthcare sector and is for IS-projects in
• Time of introduction general not of importance. The legal framework
• ERP Consultant regarding accounting or accounting data consis-
• ERP Vendor tency must be guaranteed by the manufacturer
and ethical aspects don’t play a role in this area.
Kronbichler et al. (2009) investigated CSF in The legal constraint in the healthcare sector is
ERP-projects in general and did a extended review the duty to preserve records for a very long time
of the literature. The identified CSF which were (depending on the country). 30 years are not un-
more then 70 were afterwards grouped together. commonly defined by law. Considering this, the
That means that similar or identical CSF for which retention period and the level of access seems
the different authors used different names were to be very important. The other two risk factors
grouped to one CSF within one common term. The identified for the healthcare by Paré et al. (2008)
result of this investigation are the following CSF: are more a result of a success measurement than
risk factors which can influence an IS-project.
• Top Management support Sellito and Carbone (2007) identified a success
• Change management factor that has not occurred in the investigation
• Project Management of Kronbichler et al. (2009) and that was “Gov-
• Team composition & Teamwork ernmental Influence “. This success factor is due
• Business process reorganisation to the fact that hospitals are often owned by the
• Business plan and vision public sector and this sector has to provide finan-
• Architecture choices, technical implemen- cial resources in order to develop or implement
tation, technological infrastructure the information system. In traditional companies
• (Interdepartmental) cooperation and in the private sector the governmental influence
communication plays a very minor role in the field of informa-
• Project Champion / Empowered decision tion systems. The influence occurs only, if legal
makers changes need to be mapped into a system (e.g. in
• User involvement / training finance or in the human resource area).
• Software development, testing, The risk factors Trimmer et al. (2002) examined
troubleshooting during ERP projects in healthcare all fall under
• Legacy systems knowledge (data analysis the classified section of Kronbichler et al. (2009).
& conversion) Except the “implementation costs”. The success
• Partnership factor “morale of the employees” can be assigned
• Deliverable dates / smaller scope to the critical success factor “involvement of the
• Vendor support user / training”, because it has already be shown by
Havenstein (2005) that the moral can be improved
213
Figure 1. Updated IS Success Model (DeLone and McLean, 2003)
by user involvement. The implementation costs DeLone and McLean (2002) published a 10
are not a critical success factor in the traditional year update of their IS-success model (see Figure
sense, since implementation costs should be clear 1) which offered the addition of service quality
if the CSF project management and business plan and the collapsing of individual impact and or-
are considered before the project starts. If the ganizational impact on net benefits (DeLone and
project management CSF and the effort estimation McLean, 2002; Ding and Straub, 2007). The suc-
failed, however, estimates do not reflect the actual cess measurement dimension “use” was replaced
expenses, it may be that projects fail because of by “Intention to use”. With this change of the
high costs. model the authors wanted to resolve some of the
process versus causal concerns that Seddon (1997)
Success Measurement Models raised. To all intends and purposes the adoption
was based on alterations in the role and manage-
The DeLone and McLean Information ment of information systems and on publications
Systems Success Model which were done since publishing their paper.
The model published by DeLone and McLean in
The model which seems to occur in every pub- 2002 shows that “use” must precede “user satis-
lication regarding success measurement in the faction” in a process sense. Positive experience
field of information systems is the DeLone and with “use” leads to greater “user satisfaction” in
McLean (DeLone and McLean 1992; DeLone a causal sense. All the “Use” related constructs
and McLean, 2002; DeLone and McLean, 2003) lead to “net benefits. “Net benefits” consists of
model. The DeLone and McLean model includes Individual and Organisational Impact which were
six interdependent dimensions of success mea- mentioned in the original model of 1992 (DeLone
surement which are: System quality, information and McLean, 2003; Wu and Wang, 2006).
quality, use, user satisfaction, individual impact
and organisational. The model provides a com- The Gable et al. (2003) Model
prehensive taxonomy on IS success and identified
over 100 IS success measures during the review Gable et al. (2003) made an exploratory inventory
of the collected articles (Elpez and Fink, 2006). survey which was afterwards used as the basis for
214
Figure 2. The revised model of (Gable et al., 2003)
success measurement model building. The model • it is a measurement model (not a causal/
was called the “A Priori Model”. The “A Priori process model of success)
Model” was using five constructs and forty-two • no “use” in the model
sub-constructs. • satisfaction is not an explicit dimension
For the success measures the authors synthe- (overall)
sized the Delone and McLean constructs and mea- • new / additional measures were added
sures with the associated measures from Sedera (Gable et al., 2003)
et al. (2003). The main difference to the DeLone
& Mc Lean (2002) model is that the construct use The dimensions and the meaning of the dif-
is not part of the a priori model. The model build ferent dimensions:
by Gable et al. (2003) was tested for its validity
and the validity of the four model dimensions. • Individual impact: impact of the system on
During the validation process the authors found the individual working with the system
out that there is a convergence to a higher-order • Organisational impact: impact of the sys-
phenomenon, enterprise system success. Gable tem on the organisation
et al. (2003) revised their model afterwards (see • System quality: consists of measures like
Figure 2) and the resulting model has four quad- ease of use, flexibility or data accuracy
rants: individual impact, organisational impact, • Information quality: consists of measures
information quality and system quality. These like timeliness, relevance or importance of
four quadrants are related dimensions of the the information worked up
multidimensional phenomenon: enterprise system
success (Gable et al., 2003). The main differences The Gable et al. model application possibili-
to the DeLone and McLean model are: ties are success measurements at a certain point
215
Figure 3. The Extended ERP systems success measurement model (Ifinedo, 2006)
of time. Together, the four dimensions reflect a in the model. Some measures provided for
complete view of the enterprise system and the this success dimension are technical sup-
success level within the enterprise (Gable et al. port provided, relationship with the organ-
2003). isation or credibility and trustworthiness
(Ifinedo, 2005).
The Extended ERP Systems • Workgroup impact: This dimension means
Success Measurement Model that “workgroup” encompasses sub-units
and/or functional departments of an organ-
Ifinedo (2006) who did a lot of research in the field isation. Workgroups like teams or groups
of IS success measurement extended the dimen- can contribute a lot to the success of an
sions of success proposed by Gable et al. (2003). ERP-project. In his publication Ifinedo
During research the author found through literature (2006) refers to CSF research, which
review and interviews that ERP systems success showed that workgroup impact is one of the
measurement models might be limited because most important success factors. Measures
two important dimensions may not be considered. for this dimension are the improvement of
The dimensions which were missing were: interdepartmental communication or orga-
nizational-wide communication.
• Vendor/Consultant Quality: This dimen-
sion was added because firms tend to as- During the validation of the model an ad-
sociate the role and quality of the providers ditional finding was that System Quality and
of the implemented software with its over- Organizational Impact are perhaps the two most
all success of the organization (Ifinedo, important dimensions for ERP systems success
2005; Ifinedo and Nahar, 2006). The di- (see Figure 3).
mension vendor / consultant quality should
measure the influence of external quality Ex-ante Evaluation of ERP Software
on the ERP-systems success. Vendor and
consultant are grouped together because The main focus of the research of Stefanou (2001)
they represent an external view considered is the ex-ante evaluation of ERP-system. Further-
216
Figure 4. Major phases of ERP-lifecycle (Stefanou, 2001)
more the ex-ante evaluation supports the selection ness needs, the company’s capabilities
process of ERP systems. According to Stefanou and various constraints according to the
(2001) an ex-ante evaluation is very important ERPs functionality has to be made dur-
because of the fact that selecting an ERP is a long ing this phase. Therefore, a list of business
time commitment to a certain vendor and all the processes and the required technological
general conditions which is very costly too. The changes to cover these processes has to
difference to most of the other models is the fact be made. There are 5 categories or con-
that the model of Stefanou (2001) does not provide straints: Technical, organisational, human,
the possibility of a ex-post evaluation which con- financial and time constraints.
centrates on an evaluation of an existing system. • ERP selection: During this phase the se-
Explanation of the different phases (see Figure lection of needed ERP modules and ad-
4) within the model: ditional software which is necessary to
handle the daily business has to be done.
• Clarification of the business vision: The An ERP product, vendor and support ser-
business vision is a starting point for ERP vices evaluation is done during this phase
initiation/acquisition. For the evaluation too. Sometimes companies have to decide
and selection of an appropriate system, a if they prefer a all-in-one solution or they
clear business vision is necessary. The prefer best-of-breed solutions.
business vision could be to increase the • ERP implementation: costs and benefits
competitive advantage. arising from the ERP implementation proj-
• Comparing needs vs. Capabilities: The de- ect are estimated.
tailed examination and definition of busi-
217
• Operation, maintenance and evolution: every outcome of a phase is influencing the next
After finishing the implementation project phase. At the end of their article Markus and
there is a continuous check necessary if the Tanis (2000) provide “A Process Theory of En-
solution fulfils the needs of the business. terprise System Success” with different phases,
This phase includes estimation of the costs the successful outcome, necessary conditions,
and benefits which will arise in the future probabilistic processes and a recipe for success.
from operating, maintaining and extending Compared to other models, this model is the only
the ERP system with additional functional- one which provides a theoretical framework for
ity (Stefanou, 2001). analyzing retrospectively and prospectively the
business value of a system.
The framework of Stefanou (2001) guides the
evaluator through all the important stages which Task-Technology Fit (TTF) Construct
must be considered when evaluating ERP systems. as an Indicator of ERP Success
A simple evaluation based on ease of use, useful-
ness and involvement of end users, as it has been The Task-technology fit (TTF) measures the ac-
suggested by Montazemi et al. (1996) cannot be ceptance of the system with the 3 main influence
used for such complex and risky project as the factors: task, ERP (technology) and user. ERP is
projects are in the ERP-sector. viewed as a tool used by individuals carrying out
their tasks and if those tasks can not be performed
Markus and Tanis in a accurate way, the fit is not good. Users use the
information system to support them in perform-
Markus and Tanis (2000) mentioned that there ing of their tasks. Task-technology fit measures
are different phases characterized by key play- the degree to which a technology supports an
ers, typical activities, characteristic problems, individual in performing his or her portfolio of
appropriate performance metrics and a range of tasks (Goodhue, 1995). The model Smyth (2001)
possible outcomes. Each experience made with presented (see Figure 6) includes compared to
ERP is unique, and experiences may differ from the original model of Goodhue and Thompson
company to company and from the specific point (1995) 2 other accepted success indicators, Per-
of view (that means different project member ceived usefulness, what Ives and Olson (1984)
roles, roles within the company, external roles...). call “aggregate organizational benefit” and “user
In the publication of Markus and Tanis (2000) an satisfaction” what DeLone and McLean (1992)
enterprise system experience cycle with different reported as a further important indicator of IS
phases is shown (see Figure 5). Each phase men- implementation success. According to Smyth
tioned in the publication includes a description, (2001) perceived usefulness and user satisfaction
key actors, typical activities, common errors or are the most important indicators of ERP success.
problems, typical performance metrics and pos- If the users cannot fulfil their tasks with the
sible outcomes. ERP in the way they are used to do it, a low
The field of application of the Markus & Tanis level of Task-Technology Fit is reached. Poor TTF
(2000) model is very wide. It can be used for causes low user satisfaction and this would con-
multiple success measurement approaches at dif- tribute to the lack of success of the ERP package.
ferent stages of an ERP-project. It provides the The perceived usefulness of the system is influ-
possibility to make plans and take actions if a enced by organisational factors and that’s influ-
result is not corresponding to the project plan and encing the user satisfaction in a direct and an
to increase the quality in the next phase because indirect way.
218
Figure 5. Adopted enterprise system experience cycle (Markus & Tanis, 2000)
Figure 6. ERP Success Model (Smyth, 2001)
219
Figure 7. Indirect influence of the IS on the patient care (Ammenwerth, 2003)
Other Approaches faulty transplantation of blood bags incorrectly

coded to overdose which lead to deaths. The re-
There are other models in the field of success search question of Ammenwerth and Shaw (2005)
measurement which were due to various reasons was if it is possible to implement and design a
not considered in this chapter. The focus of this system that is free of errors, easy to use and tolerant
chapter is to show the most important approaches with user errors. A possible improvement could
of success measurement for IS / ERP systems and be achieved through ongoing evaluation as this
compare those models with the models which serves to improve knowledge and to prevent and
arised within the healthcare sector. Rosemann discover side effects. This would in turn lead to
and Wiese (1999) and Martinsons et al. (1999) better systems in future and may prevent errors that
published the Balanced Scorecard (BSC) approach would have brought far-reaching consequences
which can be used for evaluation of tasks within without evaluation.
a BSC and afterwards for the strategic planning Figure 7 demonstrates that in a lot of the
of the future development of the system based on reported cases the negative impact is not caused
the results of the evaluation. There are models by the quality of the system but the use of the
which are very similar to existing models such system by the employees. Ammenwerth (2003)
as the research of Seddon (1997) or very specific mentioned the electronical file. This file provides
models which review just one aspect of IS suc- functionality like alarms and warning but these
cess, like the research of Sedera et al. (2003) who make only sense if these signals are interpreted
studied the relation of the size of the organisa- correctly by the clinical staff.
tion and the success achieved. Another example
is the publication of Wu and Wang (2007) who The FITT Model
investigated the key user’s viewpoint in success
measurement approaches. Ammenwerth et al. (2006) designed a new model
called the FITT (“Fit between Individuals, Task
Healthcare Information Systems and Technology”) framework. As a basis for this
Success Measurement Models model various models were identified which are
used as reference works for success measurement
Ammenwerth and Shaw (2005) have examined in general (see section above):
whether poor quality which has a massive impact
on health can often be avoided with doing evalua- • The IS success model of DeLone and
tion. Information and communication technology McLean (1992)
can be specified inadequate, have functional errors, • The Technology Acceptance Model (TAM)
be unreliable or work false. Such system errors by Davis (1993)
can disturb the working process and influence the • The task-technology fit model (TTF) by
decisions of the staff and can cause harm to the Goodhue et al. (2000) - which in turn is the
health or the patients. These harms range from basis for the TTF by Smyth (2001)
220
Figure 8. The FITT (Ammenwerth, 2003)
This shows that the different systems cannot Figure 8 shows the 3 parties which have to
be separated as strictly and if there is a specific work together to reach a high level of success.
system (e.g. e-commerce application, information Individual can mean single person or a group of
system in healthcare) the evaluation methods are people who implies the need to work together to
based on very similar approaches. Present ERP accomplish a task. Technology is a set of tools,
systems which are more or less specific informa- such as Hardware and software and the integration
tion systems have to be evaluated in a different of different application systems. Task involves
way in some areas (due to the often company-wide the totality of tasks and work processes to be
presence of the system). fulfilled. The arrangement of the elements shows
The FITT model distinguishes between the that people and technology form the foundation
characteristics of user, task and technology. The that is necessary to solve the overlying tasks (Am-
properties itself play only a minor role, more menwerth, 2003). The FITT model focuses on the
important is the fit between the properties, as interaction of users with the technology and as a
this is necessary for a high user acceptance and result of this an evaluation influences the users
successful implementation. The basis for this and the IT (if user and technology do not “fit”).
approach is the FIT, the fit between technology, This can also lead to process improvement because
people and task. Because of the rigidity of the fit information systems handle processes in a certain
which was due to dynamic changes in technol- way and processes are handled through the users
ogy, people and task according to Ammenwerth in their daily work with the system. Often the
(2004) like in reality of the healthcare industry cause for a lack of fit is a non-matching process
not working well the author did some changes within the company.
to the model. A goal of successful information
management should be that technology; people The Benchmark Approach
and task fit as closely as possible and thus result in
a high user acceptance. A good integration of the Another interesting approach to measure the
user in the system selection and comprehensive success of hospital information systems is the
support during the implementation phase, e.g. to Delphi study by Hübner-Bloder and Ammen-
increase the fit between people and technology werth (2009). The Delphi study was conducted
- indirectly increases the user acceptance (Am- in several rounds. In round one, the focus was
menwerth, 2003) to collect the performance indicators which are
relevant for hospital information systems. In the
221
second round the importance of benchmarking tion, future requirements will not be considered
indicators by the use of a Likert-scale was raised. in the evaluation.
In the third round, the results from Round 2 were
analyzed and then returned to the experts from
round 2 in order to give them the opportunity to SOLUTIONS AND
reconsider their scores yet. Using the Delphi study RECOMMENDATIONS
the authors identified 77 performance indicators,
divided into 8 categories: As part of this chapter the characteristics of the
healthcare sector such as the system landscape
• Technical quality which consists of many specialized applications
• Software Quality were addressed. In addition special requirements
• Architecture and Interface Quality of the healthcare industry have been shown and
• IT manufacturer’s quality the implications for ERP and IS projects. The low
• Quality of IT support and IT Departments number of publications in the healthcare related
• Support of workflows ERP field required the usage of publications that
• IT result quality deal with critical success factors during health
• IT costs information systems implementation. The critical
success factors which were identified by Kronbi-
Hübner-Bloder and Ammenwerth (2009) chler et al. (2009) were used as the ERP-specific
have consciously not focused on purely financial success factors and afterwards compared to critical
aspects, as many benchmarking studies do, but success factors identified in healthcare informa-
they focused on other things such as workflow tion system (HIS) implementation projects and
support, user satisfaction and on clinical aspects. one ERP implementation project in which took
In addition, benchmarking approaches in the IT place in the healthcare environment. The find-
business are considered from a purely technical ings of the healthcare specific publications were
point of view as these technical aspects can often be then assigned to the umbrella terms defined by
measured objectively and easily (e.g. performance Kronbichler et al. (2009). The CSFs which could
measurement). Using the benchmarking indicators not be assigned to an umbrella term were the ones
that were defined by the authors a comprehensive which are healthcare specific.. Based on these
evaluation of an IT system can be undertaken. publications it could be shown that between the
The benchmark approach of Hübner-Bloder healthcare industry and the studies which inves-
and Ammenwerth (2009) covers almost all areas. tigate ERP systems in general is little difference
IT-related measures are in the “Technical Quality” regarding the success factors identified during the
and “Architecture and Interface Quality” available. case studies. That implies that the studies on ERP
User-specific measures are in the “Software Qual- implementations in other areas may also be used
ity” and “IT Support and IT Department Quality”. for healthcare. The peculiarities of the healthcare
Management-related measures can especially sector (based on the ERP implementation) are
be found in the “IT Outcome Quality” category, limited primarily to the following areas:
such as “Contribution of the HIS systems to the
strategic goals of the medical, nursing and admin- • Legal constraints (Mansky, 2001, Paré et
istrative management”. The external perspective al. 2008)
is integrated in the “IT Vendor Quality”. As a ◦◦ Data migration from legacy systems
benchmark normally evaluates the actual situa- ◦◦ Access permissions
222
• A variety of systems that need to be con- FUTURE RESEARCH DIRECTIONS

nected to the ERP system (interfaces)
(Pumphrey et al., 2007) The factors influencing success or success mea-
• The competitive structures are in some surement can change over time due to change in
countries due to the industry structure not technical, cultural or legal constraints. Because
as strong as in other industries (for non- of this, it is necessary to conduct an ongoing
privatized hospitals) (Busse et al., 2006) research project in the field of success in ERP-
projects. The performance measurement models
The identified models were compared with were, however, over time (since 1992), adapted
the models in the healthcare sector to identify the only slightly.
differences. The models which were developed to New results on the above areas may also occur
measure the success of the systems in healthcare, in extending existing models with new knowledge
however, differ only slightly from the models in (e.g. technological change). ERP systems will play
the ERP context. All of the models use similar an important role for companies in the future too
success measures and similar approaches for doing because the economy is a very dynamic environ-
the success measurement. An evaluation of ERP ment that requires ongoing adjustments and a
systems in the healthcare industry does not need system which can handle the business needs of
the use of a specialized model, a conventional nowadays.
model can used for this too. In this research a comparison of success mea-
The impact of this research is the link between surement models and critical success factors was
success measurement approaches in healthcare and made without conducting case studies. Through
other industries. It was demonstrated that a large case studies the feasibility of the models could
part of the critical success factors does not depend be identified and the models could be compared
on the industry. The importance of high-quality on the basis of that. Interesting in this context
systems in the healthcare sector is crucial therefore would be the creation of an anonymous database
models to measure success are very important in which contains a “success comparison” (results
this industry. It has been shown that these models of success measurements of various companies
are based on the same models as the ERP-specific on the basis of defined measures). This could be
models and there were just small adoptions neces- carried out to the participating companies and they
sary for the healthcare environment. This implies would have a comparison with other companies.
that the success measurement approaches seem Another interesting future research could
to be almost branch independent and each of the be done comparing companies which are doing
models can be used for a success measurement success measurement and companies which are
approach in the healthcare industry. Maybe the not doing it. A possible measure for this could
measures provided by some of the models must be be the user acceptance and satisfaction or other
extended but this could be caused by investigating stakeholders which are using the system.
a specific system (e.g. CRM) too. ERP projects Furthermore different branches and their
in the healthcare field can therefore easily be specific needs could be considered in feature
compared with the ERP projects in other sectors. research. Another branch with very specific
Every industry has its own specific features, the needs is the meat industry. One example which
healthcare sector shows no major deviations than differs from other branches is the meat industry.
other industries. The complexity within the production is high be-
cause on every point at the value-added chain the
meat must be weighted. Another example is the
223
livestock procurement which is totally different ethical constraints and governmental influence.
to the purchase of common goods, every animal These minor differences are caused by access
has a different weight and different condemns permissions because of privacy of patient data
and as a result of this a different price. If the input and the long time period in which the data have
material is pork the butchers in the production to be accessible in the system.
have to separate it according to the production In the area of IS / HIS / ERP success measure-
planning. This cannot be handled with a bill of ment a selection of relevant models for the study
material. This demonstrates that the production was carried out by the author too. Care was taken
requirements for an ERP-system is different to that most of the selected models represented a
other industries too, the bill of material cannot specific focus and do not coincide with other
handle the requirements occurring. models considered. Some models (such as the
model of Ifinedo) is similar to another model
but extended the base model with new insights.
CONCLUSION It cannot be guaranteed that all relevant models
for the ERP field have been considered, as this
In the previous sections it was demonstrated that is due to the large number of publications in this
many projects have failed due to various reasons. area very difficult. Success measurement models
Often the willingness of companies (or their em- identified in healthcare industry are similar to the
ployees) for change was not present (Boonstra success measurement models which were not
and Govers, 2009). However, other factors such developed for a certain branch.
as non-matching systems, processes which do The objective of this chapter was to make ERP
not correspond to the processes of the healthcare projects more transparent and to demonstrate that
facility led to a failure. Particularly impressive considering branch specific measurement and suc-
is the Case-Study of Havenstein (2005) in which cess factors can lead to a more successful project
a hospital has replaced SAP with the system of outcome. Using critical success factors and provide
Lawson because SAP did not meet expectations. a methodology for the application of the CSF helps
This would not have happened if the existing to identify the main problems and to take actions
processes, as in the investigation of Huq and early enough to avoid problems within later phases
Martin (2006), would have been compared with or in future. The success measurement models
the possible processes in SAP. That means that are used ultimately to guarantee a high level of
the evaluation of the processes should have been satisfaction and success to the users. This can be
done in an appropriate way (Verville, 2002). done in various stages of ERP projects to meet
At least then the company would have known the quality requirements.
already in the validation period that the system Especially the healthcare industry is constantly
did not fit to the processes within the company. changing due to the demographic development of
The findings made by the authors demonstrate the civilization. Nowadays quality management
that the healthcare industry deals with the same concepts such as Six Sigma (with the 0-fault
issues as other industries. In this chapter it was tolerance) are implemented in the healthcare
demonstrated that the critical success factors in sector, these concepts can cause changes to the
healthcare HIS / ERP-projects which are important information systems within the healthcare too.
for the outcome of a project are very similar to The success factors in turn can be used by consul-
the ones identified in ERP-projects which took tants and other project members to ensure project
place in other industries. The CSFs which just success with a higher probability. A continuous
occurred in healthcare projects were legal and
224
monitoring of CSF in practical projects should Claeys, T. (1996). Medical filling (2nd ed.). Cen-
lead to better results. gage Learning.
Davis, F. D. (1993). User acceptance of Informa-
tion Technology: System characteristics, user per-
REFERENCES
ceptions and behavioural impacts. International
Ammenwerth, E. (2003). Die Bewertung von Journal of Man-Machine Studies, 38, 475–487.
Informationssystemen des Gesundheitswesens doi:10.1006/imms.1993.1022
– Beiträge für ein umfassendes Informations- DeLone, W. H., & McLean, E. R. (1992). Informa-
management. Habilitationsschrift zur Erlangung tion Systems success: The quest for the dependent
der Venia docendi für das Fach Medizinische variable. Information Systems Research, 3(1),
Informatik. 60–95. doi:10.1287/isre.3.1.60
Ammenwerth, E., Iller, C., & Mahler, C. (2006). DeLone, W. H., & McLean, E. R. (2002). Infor-
IT-adoption and the interaction of task, technol- mation Systems success revisited. Conference
ogy and individuals: A fit framework and a case Proceedings, 35th Hawaii International Confer-
study. BMC Journal of Medical Informatics and ence on System Sciences, Los Alamitos.
Decision Making, 6(3).
Ammenwerth, E., & Shaw, N. T. (2005). Bad health DeLone and McLean model of Information
informatics can kill – Is evaluation the answer? Systems success: A ten-year update. Journal of
Methods of Information in Medicine, 44(1), 1–3. Management Information Systems, 19(4), 9–30.
Auer, C. (2004). Performance Mesurement für das Ding, Y., & Straub, D. (2007). Using marketing
Customer Relationship Management – Controlling exchange theory to conceptualize IS quality and
des IKT-basierten Kundenbeziehungsmanage- re-conceptualize the IS success model. Twenty
ments. Deutscher Universitäts-Verlag, 2004. Eighth International Conference on Information
Bärwolff, H., Victor, F., & Hüsken, V. (2006). Systems, Montréal.
IT-Systeme in der Medizin. Vieweg Verlag. Esteves, J., Pastor, J., & Casanovas, J. (2002).
Boonstra, A., & Govers, M. J. G. (2009). Un- Monitoring business process redesign in ERP
derstanding ERP system implementation in a implementation projects. Americas Conference
hospital by analysing stakeholders. New Tech- on Information Systems 2002, Proceedings Paper
nology, Work and Employment, 24(2), 177–193. 125.
doi:10.1111/j.1468-005X.2009.00227.x Fischer, W. (2000). Grundzüge von DRG-Syste-
Bullen, C. V., & Rockart, J. F. (1981). A primer on men. In KH-Report (pp. 13–31).
critical success factors. Working paper 1220-81. Frodl, A. (2010). Gesundheitsbetriebslehre: Be-
Report. Massachusetts Institute of Technology triebswirtschaftslehre des Gesundheitswesens.
(MIT), Sloan Scool of Management, Center for Gabler Verlag.
Information System Research.
Früh, K. F., Maier, U., & Schaudel, D. (2008).
Busse, R., Schreyögg, J. A., & Gericke, C. (2006). Handbuch der Prozessautomatisierung: Proz-
Management im Gesundheitswesen (2nd ed.). essleittechnik für verfahrenstechnische Anlagen.
Berlin, Germany: Springer. doi:10.1007/978-3- Oldenbourg Industrieverlag.
540-29465-8
225
Gable, G., Sedera, D., & Chan, T. (2003). Enter- Ifinedo, P. (2006). Extending the Gable et al.
prise systems success: A measurement model. enterprise systems success measurement model:
Twenty Fourth International Conference on In- A preliminary study. Journal of Information
formation Systems, Seattle. Technology Management,17(1), 14 -33.
Goodhue, D. L., & Thompson, R. L. (1995). Ifinedo, P., & Nahar, N. (2006). Quality, impact
Task-technology fit and individual performance. and success of ERP-systems: A study involving
Management Information Systems Quarterly, some firms in the Nordic-Baltic region. Journal
19(2), 213–235. doi:10.2307/249689 of Information Technology Impact, 6(1), 1–28.
Gronau, N. (2001). Industrielle Standardsoft- Ives, B., & Olson, M. H. (1984). User involvement
ware – Auswahl und Einführung: keine Angabe. and MIS success: A review of research. Man-
München, Germany: Oldenbourg. agement Science, 30(5), 586–603. doi:10.1287/
mnsc.30.5.586
Gulbins, J., Seyfried, M., & Strack-Zimmermann,
H. (2002). Dokumenten Management (3rd ed.). Kalra, D., & Ingram, D. (2006). Information
Berlin, Germany: Springer. Technology solutions for healthcare. Elsevier.
Haas, P. (2004). Medizinische Informationssys- Kontio, E., Korvenranta, H., Lundgren-Laine, H.,
teme und Elektronische Krankenakten. Springer. & Salanterä, S. (2009). Key elements of success-
full care process of patients with heart symptoms
Havenstein, H. (2005). Healthcare provider nears
in an emergency care – Could an ERP system
end of 10-Year ERP journey. Computerworld,
help? In Saranto, K. (Eds.), Connecting health
39(51), 52–53.
and humans. IOS Press.
Heeks, R., Mundy, D., & Salazar, A. (1999). Why
Kronbichler, S. A., Ostermann, H., & Staudinger,
healthcare Information Systems succeed or fail.
R. (2009). A review of critical success factors for
Information Systems for public sector manage-
ERP projects. The Open Information Systems Jour-
ment. Working Paper Series, Paper No. 9, 1999.
nal, 3, 14–25. doi:10.2174/1874133900903010014
Hübner-Bloder, G., & Ammenwerth, E. (2009).
Lenz, R., & Kuhn, K. A. (2003). Zur Architektur
Benchmark hospital Information Systems – A
und Evolution von Krankenhausinformations-
Delphi study. Methods of Information in Medicine,
systemen, Innovative Informatikanwendungen.
48, 508–518. doi:10.3414/ME09-01-0044
Band 2, Beiträge der 33. Jahrestagung der Ge-
Huq, Z., & Martin, T. N. (2006). The recovery sellschaft für Informatik e.V. (GI), 29. September
of BPR implementation through an ERP ap- – 2. October.
proach – A hospital case study. Business Pro-
Leonard, K. J. (2004). Critical success factors
cess Management Journal, 12(5), 576–587.
relating to healthcare’s adoption of new technol-
doi:10.1108/14637150610691000
ogy: A guide to increasing the likelihood of suc-
Ifinedo, P. (2005). Do organisational technologi- cessful implementation. Electronic Healthcare,
cal contingency factors influence the perception 2(4), 72–81.
of ERP systems success? An exploratory study
Lillis, K. (2002). Despite advances, software
in the Baltic-Nordic Region of Europe. Confer-
makers and hospitals still face formidable chal-
ence Proceedings, The 4th International Business
lenges. Healthcare Purchasing News, November,
Information Management Association (IBIMA)
(pp. 16 – 20).
Conference, Lisbon, pp. 427 – 437.
226
Mansky, T. (2001). Asuwirkungen der Reform Pumphrey, L. D., Trimmer, K., & Beachboard,
des Krankenhausentgeltsystems auf die stationäre J. (2007). Enterprise resource planning systems
Versorgung des Krankenhausmanagement. In and HIPAA. Research in Healthcare Financial
Ackermann, T. (Ed.), Schwerpunkt: Vergütung- Management, 11(1), 57–75.
sreform mit DRGs (pp. 173–194). Schattauer.
Rockart, J. F. (1979). Chief executives define
Markus, M. L., & Tanis, C. (2000). The enterprise their own data needs. Harvard Business Review.
system experience—From adoption to success. March-April.
In Zmud, R. W. (Ed.), Framing the domains of
Rosemann, M., & Wiese, J. (1999). Measuring
IT management: Projecting the future through
the performance of ERP software – A Balanced
the past. Cincinnati, OH: Pinnaflex Educational
Scorecard approach. Conference Proceeding,
Resources, Inc.
The 10th Australasian Conference on Information
Martinsons, M., Davison, R., & Tse, D. (1999). Systems, 8(4).
The balanced scorecard: A foundation for the
Seddon, P. (1997). A respecification and extension
strategic management of information systems.
of the DeLone and McLean model of IS success.
Journal of Decision Support Systems, 25, 71–88.
doi:10.1016/S0167-9236(98)00086-4
doi:10.1287/isre.8.3.240
Montazemi, A., Cameron, D., & Gupta, K. (1996).
Sedera, D., Gable, G., & Chan, T. (2003). ERP
An empirical study of factors affecting software
success: Does organisation size matter? 7th Pa-
package selection. Journal of Management Infor-
cific Asia Conference on Information Systems,
mation Systems, 13(1), 89–105.
Adelaide, (pp. 1075-1088).
Monteiro, E. (2003). Integrating health Informa-
Sellitto, C., & Carbone, D. (2007). Success fac-
tion Systems: A critical appraisal. Methods of
tors associated with health Information Systems
Information in Medicine, 42(4), 428–432.
implementation: A study of an Australian regional
Münch, E. (2002). Strukturwandel im Kranken- hospital. Journal of Business Systems, Governance
haus-Markt – Investitionsfinanzierung im Um- and Ethics, 2(4).
bruch. Handelsblatt Gesundheitskongress.
Smyth, R. W. (2001). Challenges to successful ERP
Nah, F. F.-H., & Lau, J. L.-S. (2001). Critical fac- use. The 9th European Conference on Information
tors for successful implementation of enterprise Systems, Bled.
systems. Business Process Management Journal,
Soh, C., Sia, K. S., Boh, W. F., & Tang, M.
7(3), 285–296. doi:10.1108/14637150110392782
(2003). Misalignments in ERP implementation:
Paré, G., Sicotte, C., Jaana, M., & Girouard, D. A dialectic perspective. International Journal of
(2008). Prioritizing clinical Information System Human-Computer Interaction, 16(1), 81–100.
project risk factors: A Delphi study. Proceedings doi:10.1207/S15327590IJHC1601_6
of the 41st Hawaii International Conference on
Stefanou, C. J. (2001). A framework for the ex-ante
System Sciences.
evaluation of ERP software. European Journal of
Peabody, P. (2002). Setting a new standard. Health Information Systems, 10, 204–215. doi:10.1057/
Management Technology, November. palgrave.ejis.3000407
227
Stefanou, C. J., & Revanoglou, A. (2006). ERP in- Al-Mudimigh, A., Zairi, M., & Al-Mashari, M.
tegration in a healthcare environment: A case study. (2001). ERP implementation: An integrative
Journal of Enterprise Information Management, methodology. I3E Zürich.
19(1), 115–130. doi:10.1108/17410390610636913
Aladwani, A. M. (2001). Change management
Trimmer, K. J., Pumphrey, L. D., & Wiggins, C. strategies for successful ERP implementation.
(2002). ERP implementation in rural healthcare. Business Process Management Journal, 7(3),
Journal of Management in Medicine, 16(2/3), 266–275. doi:10.1108/14637150110392764
113–132. doi:10.1108/02689230210434871
Behrens, S., Jamieson, K., Jones, D., & Cranston,
Verville, J. (2002). Evaluating ERP Software in M. (2005). Predicting system success using the
the acuqisition process: A three phase activity. technology acceptance model: A case study. 16th
International Association for Computer Informa- Australasian Conference on Information Systems,
tion Systems Conference, (pp. 625 – 631). Sydney.
Wu, J. H., & Wang, Y. M. (2006). Measuring KMS Bhatti, T. R. (2005). Critical success factors for the
success: A respecification of the DeLone and implementation of enterprise resource planning
McLean’s model. Information & Management, (ERP): Empirical validation. The Second Inter-
43(6), 728–739. doi:10.1016/j.im.2006.05.002 national Conference on Innovation in Information
Technology, IIT’05.
Wu, J. H., & Wang, Y. M. (2007). Measuring
ERP success: The key-users’ viewpoint of the Bingi, P., Sharma, M., & Godla, J. (1999). Critical
ERP to produce a viable IS in the organization. issues affecting an ERP implementation. Informa-
Computers in Human Behavior, 23(3), 1582–1596. tion Systems Management, 16(3), 7–15. doi:10.1
doi:10.1016/j.chb.2005.07.005 201/1078/43197.16.3.19990601/31310.2
Wyatt, J. (2002). ERP in the consumer-driven Bosilj-Vuksic, V., & Spremic, M. (2005). ERP sys-
marketplace. Health Management Technology, tem implementation and business process change:
23(7), 24. Case Study of a pharmaceutical company. Journal
of Computing and Information Technology, 13,
11–24. doi:10.2498/cit.2005.01.02
ADDITIONAL READING Buckhout, S., Frey, E., & Nemec, J. (1999). Mak-
ing ERP succeed: Turning fear into promise. IEEE
Al-Mashari, M., Al-Mudimigh, A., & Zairi, M. Engineering Management Review, 19, 116–123.
(2003). Enterprise resource planning: A taxonomy
of critical factors. European Journal of Opera- Cameron, K. S., & Whetten, D. A. (1983). Some
tional Research, 146(2), 352–364. doi:10.1016/ conclusions about organizational effectiveness.
S0377-2217(02)00554-4 In Cameron, K. S., & Whetten, D. A. (Eds.),
Organizational effectiveness: A comparison of
Al-Mashari, M., Ghani, S. K., & Al-Rashid, W. multiple models. New York, NY: Academic Press.
(2006). A study of the critical success factors of
ERP implementation in developing countries. Chien, S.-W., & Tsaur, S. M. (2007). Investigat-
International Journal of Internet and Enter- ing the success of ERP systems: Case studies in
prise Management, 4(1), 68–95. doi:10.1504/ three Taiwanese high-tech industries. Computers
IJIEM.2006.008866 in Industry, 58(8-9), 783–793. doi:10.1016/j.
compind.2007.02.001
228
Cooke, D., Gelman, L., & Peterson, W. J. (2001). Fosser, E., Leister, O. H., & Moe, C. E. (2008).
ERP trends, (pp. 1-28). The Conference Board, Inc. Organisations and vanilla software: What do we
know about ERP systems and competitive ad-
Craig, D., & Tinaikar, R. (2006). Divide and
vantage? European Conference on Information
conquer: Rethinking IT strategy. McKinsey &
Systems, ECIS 2008 Proceedings.
Company. Retrieved from http://www.mckinsey.
de/downloads/publikation/mck_on_bt/2006/ Ghosh, S. (2002). Challenges on a global imple-
MoIT9_ITStrategy_VF.pdf mentation of ERP software. Engineering Manage-
ment Conference, IEMC ’02.
Curlee, T. R., & Tonn, B. T. (1987). The success
or failure of management Information Systems: A Guba, E. G., & Lincoln, Y. S. (1989). Fourth
theoretical approach. Oak Ridge National Labo- generation evaluation. SAGE.
ratory for the U.S. Department of Energy under
Haines, M. N., & Goodhue, D. L. (2003). Imple-
Contract no. DE-AC05-840R21400.
mentation partner involvement and knowledge
Darshana, S., Gable, G., & Rosemann, M. (2001). transfer in the context of ERP implementations.
A balanced scorecard approach to enterprise International Journal of Human-Computer Inter-
systems performance measurement. Conference action, 16(1), 23–38. doi:10.1207/S15327590I-
Proceedings, Twelfth Australasian Conference on JHC1601_3
Information Systems, Coffs Harbour.
Hedman, J., & Borell, A. (2005). Broadening In-
Davenport, T. H. (1998). Putting the enterprise into formation Systems evaluation through narratives.
the enterprise system. Harvard Business Review, The Electronic Journal of Information Systems
76(4), 121–131. Evaluation, 8(2), 115–122.
Drucker, P. F. (2004). The Daily Drucker. New Holland, C. P., & Light, B. (1999). A critical suc-
York, NY: Harper Collins. cess factors model for ERP implementation. IEEE
Computer Society, 16(3), 30–35.
Elpez, I., & Fink, D. (2006). Information Systems
success in the public sector: Stakeholders’ Per- Jarrar, Y. F., Al-Mudimigh, A., & Zairi, M. (2000).
spectives and emerging alignment model. Issues ERP implementation critical success factors – The
in Informing Science and Information Technology, role and impact of business process management.
3, 219–231. ICMIT, Proceedings of the 2000 IEEE Interna-
tional Conference on Information Technology.
Esteves, J., & Pastor, J. (2001). Analysis of critical
success factors relevance along SAP implemen- Kaplan, R. S., & Norton, D. P. (1997). Using the
tation phases. Seventh Americas Conference on balanced scorecard as a strategic management
Information Systems. system. Harvard Business Review OnPoint, 1-13.
Esteves-Sousa, J., & Pastor-Collado, J. (2000). Kemppainen, I. (2004). Change management
Towards the unification of critical success factors perspectives in an ERP-implementation. European
for ERP-implementations. 10th Annual Busi- Conference on Information Systems.
ness Information Technology Conference (BIT),
King, W. R. (2005). Ensuring ERP implementa-
Manchester.
tion success. Information Systems Management,
22(3), 83–84. doi:10.1201/1078/45317.22.3.200
50601/88749.11
229
Magnusson, J., Nilsson, A., & Carlsson, F. (2004). Nah, F. F.-H., & Lau, J. L.-S. (2001). Critical fac-
Forecasting ERP implementation success – To- tors for successful implementation of enterprise
wards a grounded framework. European Confer- systems. Business Process Management Journal,
ence on Information Systems. 7(3), 285–296. doi:10.1108/14637150110392782
Markus, M. L., Axline, S., Petrie, D., & Tanis, C. Nah, F. F.-H., Zuckweiler, K. M., & Lau, J. L.-S.
(2000). Leraning from adopters’ experiences with (2003). ERP implementation: Chief information
ERP: Problems encountered and success achieved. officers perceptions of critical success factors.
Journal of Information Technology, 15(4), 245– International Journal of Human-Computer In-
265. doi:10.1080/02683960010008944 teraction, 16(1), 5–22. doi:10.1207/S15327590I-
JHC1601_2
May, M. (2003). Business process management –
Integration in a Web-enabled environment. Great Nicolaou, A. I. (2004). ERP systems implemen-
Britain: Pearson Education. tation: Drivers of post-implementation success.
DSS2004 Conference Proceedings.
Murray, M., & Coffin, G. (2001). A case study
analysis of factors for success in ERP system Nicolau, A. I. (2004). Firm performance effects
implementations. Proceedings of the Seventh in relation to the implementation and use of en-
American Conference on Information Systems. terprise resource planning systems. Journal of
Myers, B. L., Kappelman, L. A., & Prybutok,
jis.2004.18.2.79
V. R. (1996). A care for including work group
productivity measures in a comprehensive IS as- Parr, A., & Shanks, G. (2000). A model of
sessment model. The 27th Annual Meeting of the ERP project implementation. Journal of
Decision Science Institute, Orlando. Information Technology, 15(4), 289–303.
doi:10.1080/02683960010009051
Myers, B. L., Kappelman, L. A., & Prybutok, V.
R. (1997). A comprehensive model for assessing Ranzhe, J., & Xun, Q. (2007). A study on critical
the quality and productivity of the Information success factors in ERP-Systems implementation.
Systems function: Toward a theory for Information International Conference on Information Systems.
Systems assessment. In Garrity, E. J., & Sanders,
Shanks, G., Parr, A., Hu, B., Corbitt, B. J., Tha-
G. L. (Eds.), Information Systems success mea-
nasankit, T., & Seddon, P. (2000). Differences in
surement. Hershey, PA: Idea Group Publishing.
critical success factors in ERP systems implemen-
Nah, F. F.-H., & Delgado, S. (2006). Critical tation in Australia and China: A cultural analysis.
success factors for enterprise resource planning European Conference on Information Systems.
implementation and upgrade. Journal of Computer
Sherer, S. A., & Alter, S. (2004). Information Sys-
tem risk and risk factors: Are they mostly about
Nah, F. F.-H., & Delgado, S. (2006). Critical Information Systems? Communications of the
success factors for enterprise resource planning Association for Information Systems, 14, 29–64.
implementation and upgrade. Journal of Computer
Somers, T. M., & Nelson, K. (2001). The impact
of critical success factors across the stages of
enterprise resource planning implementations.
Proceedings of the 34th Hawaii International
Conference on System Sciences.
230
Sumner, M. (1999). Critical success factors in Willcocks, L. P., & Sykes, R. (2000). The role of the
enterprise wide information systems projects. CIO and IT function in ERP. Communications of the
Conference Proceedings, the American Confer- ACM, 43(4), 32–38. doi:10.1145/332051.332065
ence on Information Systems, New Orleans.
Woodside, J. M. (2007). EDI and ERP: A real-time
Sumner, M. (2000). Risk factors in enterprise-wide/ framework for healthcare data exchange. Journal
ERP projects. Journal of Information Technology, of Medical Systems, 31(3), 178–184. doi:10.1007/
15, 317–327. doi:10.1080/02683960010009079 s10916-007-9053-4
Umble, E. J., Haft, R. R., & Umble, M. M. (2003). Yusuf, Y., Gunasekaran, A., & Abthorpe, M. S.
Enterprise resource planning: Implementation (2004). Enterprise Information Systems project
procedures and critical success factors. Euro- implementation: A case study of ERP in Rolls-
pean Journal of Operational Research, 146(2), Royce. International Journal of Production
241–257. doi:10.1016/S0377-2217(02)00547-7 Economics, 87(3), 251–266. doi:10.1016/j.
ijpe.2003.10.004
Van Stijn, E., & Wensley, A. (2005). ERP’s best
practices and change: An organizational memory Zhang, L. L., Matthew, K. O., Zhang, Z., & Ba-
mismatch approach. European Conference on nerjee, P. (2003). Critical success factors of enter-
Information Systems. prise resource planning systems implementation
success in China. Proceedings of the 36th Hawaii
Wee, S. (2000). Juggling toward ERP success:
International Conference on System Sciences.
Keep key success factors high (pp. 1–5). ERP
News.
231
Section 5
Websites and E-Commerce
Systems
233
Chapter 12
Issues Facing Website
Evaluation:
Identifying a Gap
Ahmad Ghandour
University of Otago, New Zealand
Kenneth R. Deans
George L. Benwell
ABSTRACT
As business organisations have become more reliant on Information Technology in achieving success,
Information Systems have become essential. Business organisations now use websites as part of their
Information Systems as a medium for communication and transactions between the business and their
customers. A better understanding of how to evaluate a website is necessary. This chapter explores web-
site evaluation and recognises the current challenges facing website evaluation. It begins by identifying
the type of website the current study is focussed on, namely the e-commerce website. This is followed
by reviewing the literature on website evaluative approaches and anecdotally described issues with
the existing performance measures. Three perspectives were identified when evaluating websites, user,
designer, and owner perspectives. While the user and the designer perspectives are well advanced in the
literature, there is a relative dearth of scholarly studies that address the owners’ needs. The provision
of such a perspective may enhance an owner’s ability to increase returns and benefits from their online
activity. Such gap need to be filled, a call from the authors.
DOI: 10.4018/978-1-4666-0170-3.ch012
Issues Facing Website Evaluation
INTRODUCTION chapter begins with e-commerce evaluation fol-

lowed by a description of e-commerce websites
Many business organisations have responded that the current research focussed on. The next
to the media hype surrounding the growth of e- section reviews the literature on website evalua-
commerce and invested in its potential. A website tive approaches, followed by frameworks used to
is then built as a first step. Businesses however, measure website performance. Finally, we discuss
need to be able to evaluate their websites as part a number of issues associated with the existing
of their management practices. Clear, useful evaluative approaches identifying a research gap
measurements that capture website performance before concluding with a summary.
have enabled businesses to evaluate the results and
returns of their website. Good website evaluation
offers strategic benefits such as positive return E-COMMERCE EVALUATION
on investment and improved operations. The
evaluation of a website could be improved when Researchers have been interested in e-commerce
performance data is known. Many measures have for almost two decades. Some studies proposed
been proposed to capture website performance. A and investigated different business models for
website is very often a sales channel (sometimes trading on the Internet. Others investigated the
a company’s sole interface) between the business, benefits of the Internet to businesses (Cockburn
its customers and the world at large. With grow- & Wilson, 1996; Ng & Wilson, 1998), and the
ing demands to see returns on internet-related benefits from implementing e-commerce (Piris,
investments, a stronger focus on performance is Fitzgerald, & Serrano, 2004; Zhuang & Lederer,
becoming critical. 2003). This has led to approaches that are used to
Measuring the performance of a website has conduct evaluations of e-commerce at all levels of
been researched in many ways and various con- implementation. Two distinct research streams are
texts over the past decade. However, as website identified: (a) the uptake and, (b) the success of
performance is neither simple nor straight forward, such initiatives. Researchers follow either stream
its measurement is multidimensional in nature. to identify factors that impact the effectiveness
The different perspectives, website user, owner of an e-commerce initiative in an organisation.
and designer add another layer of complexity to In the uptake approach, e-commerce is con-
the construct measurements. Clearly, establishing sidered an innovation. Researchers have extended
accurate website performance measurements is works such as Rogers’ six stages of the diffusion of
an interesting problem for website owners and innovation model (Rogers, 1983, 2003) to acquire
researchers. An effective framework for the evalu- an underlying knowledge of how e-commerce is
ation of a website generates improved capabilities adopted through the five stages (initiation, adop-
to modify and predict future successful website tion, implementation, evaluation, and integration).
behaviour. While there are evaluative studies that Rogers also classified adopters into six categories:
propose performance measurements for websites, non-adopters, innovators, early adopters, early
a number of observations can be drawn. majority, late majority and laggards. Further,
In this chapter, three key points are discussed. researchers (Al-Qirim, 2005; Eastin, 2002; Gran-
First, website evaluation techniques are traced don & Pearson, 2004; Hong & Zhu, 2006; Pease
and evaluated. Second, current evaluation frame- & Rowe, 2005; Samiaji & Zowghi, 2005; Teo
works in the light of these evaluation techniques & Pian, 2004; Zhu & Kraemer, 2003) explored
are presented. Third, flaws in these evaluation potential factors impact on e-commerce adoption
techniques and frameworks are explored. This as shown in Figure 1.
234
Figure 1. Generic model in the adoption research
While this approach (the uptake) has been This approach may be used to identify factors
useful for predicting inhibitors and barriers to related to the success of implementing e-commerce
adopting e-commerce, it is of little use to owners in businesses and provide guidelines to overcome
seeking to evaluate their return on investment. challenges and exploit opportunities (Chakraborty,
Another useful and informative research stream Lala, & Warren, 2002; Feindt, Jeffcoate, & Chap-
is e-commerce success. Researchers are seeking pell, 2002; Sung, 2006; Vijayaraman & Bhatia,
to identify critical success factors (CSF) of e- 2002). In the same vein, Han and Noh (1999) used
commerce. The term (CSF) was first coined by critical failure factors (CFFs) to explore the fac-
Rockart (1979). He defined CSF as the limited tors that inhibit the growth of e-commerce.
number of areas in which results, if they are sat- Nowadays e-commerce is not a novelty. It is a
isfactory, will ensure successful competitive reality and presents most businesses with a great
performance for the business. The basic method- opportunity as well as a great challenge. It has
ology is to decompose an objective into its con- become widely adopted and the pressure on busi-
stituent factors. Activities supporting the factors nesses to account for investment in their website
are then entered into a matrix and their relation- has increased. Hence the need for businesses to
ship to the objective is validated. Finally, the be able to assess their e-commerce activities as
activities are performed and measured. Research- part of their overall performance becomes neces-
ers have extended the above technique to measure sary.
success in terms of these CSF according to the
generic framework as illustrated in Figure 2.
Figure 2. Generic model for critical success factor
235
E-COMMERCE WEBSITE ing in traditional business planning techniques.

This kind of action has caused disillusionment
Over the past two decades, business practice has in businesses after setting up their web presence
been enabled by information technology (IT), in and their subsequent difficulty to clearly articu-
particular the Internet, and e-commerce has ex- late the real benefits from being online (Bloch,
ploded. Indeed, the evolution of IT has provided Pigneur, & Segev, 1996). Successfully integrat-
for alternative systems supporting organisations to ing websites into the fabric of their business and
become more efficient and effective than before. their sales strategies is critical for the success of
Organisations now can conduct business over the businesses that sell products or services online.
internet manifested by a web presence. “one highly Effective web-based selling requires businesses
visible aspect of e-commerce activity that is often to have adequate knowledge of how to measure
seen as the first step towards online trading is the the performance of their website (Stockdale, Lin,
launch of a Website” (Stockdale & Lin, 2008, p. & Stoney, 2005). In addition, the proliferation of
72). In a business organisation, the website is the websites, and the many calls from businesses for
medium for communication and transactions be- their use, has prompted researchers to investigate
tween the business and its customers. The website, the effectiveness of such initiatives.
the basic element of conducting business online,
is a collection of pages residing on servers that
are connected to the World Wide Web. It is part of WEBSITE EVALUATION
information systems in the business organisation
and it is written in a special language enabling As defined by Hung and McQueen (2004), website
different functionalities allowing the access to evaluation is the measurement and assessment of a
anyone with an internet connection (Molla & website’s performance. The measurement of web-
Licker, 2001). site performance, however, has proved difficult
The functionality of the website is reflected as it depends on which stakeholder perspective
by the business’s online model. Websites that (the user, the designer or the business) is taken. It
offer goods and services for sale (henceforth is also difficult because it is a multidimensional
“website”) have become the way to trade with concept (Palmer, 2002) that can be assessed at
customers. A business model that employs such different levels (individual, organisational) using
a website serves as a communication channel for different interrelated criteria (Molla & Licker,
bi-directional information transfer, a platform for 2001). Multiple, interrelated success dimensions
transacting, an interface for providing customer are more likely to capture changes in performance
service (Quelch & Klein, 1996; Udo & Marquis, than one single item or even a set of financial
2001) and facilitates marketing activities (Deans measures (Segars & Grover, 1998). Financial
& Adam, 2000; Schubert & Selz, 2001; Udo & measures only portray one facet of performance
Marquis, 2001). (Barua, Konana, Whinston, & Fang, 2001; Barua,
However, according to Epstein (2005), busi- Kriebel, & Mukhopadhyay, 1995).
nesses “took the plunge” without thinking seri- However, many website evaluation measures
ously about what drives e-commerce success. and contexts have been proposed over the past
While Ghandour, Benwell and Deans (2007) decade. Although somewhat different in their ap-
contend that planning strategically is a pre req- proaches, the underlying principles of these studies
uisite for e-commerce to succeed, Foster & Lin are very similar in providing evaluation criteria
(2004) posit that many e-commerce ventures failed (Hong & Kim, 2004). Such criteria are used to
because of the lack of planning and proper ground- diagnose current websites and also to guide the
236
Table 1. Previous evaluation studies on websites
Author (s) Website typology/criteria

Hoffman & Novak, 1995 Online store front, Internet presence sites (flat ads, Image, and Information), and Con-
tent (fee-based, sponsored, and searchable database)
Forrester Research, 1996 Promotional, content, and transactional sites
Rice, 1997 Content, enjoyable, uniqueness, layout, and ease of finding
Ho, 1997 Informational, transactional, and promotional
Griffith & Krampf, 1998 Online sales, communication, and customer service
Deans & Adams, 2000 Communicational, transactional, and relationships
Young & Benamati, 2000 Customer Sales, Customer Service, Communication, and Informational
Hong & Kim, 2004 Architectural criteria for website evaluation–conceptual framework and empirical
validation
optimisation and development of future websites. determine the extent of the technology used for
Past studies have shown that issues pertaining industries, countries, and regions. Websites can
to website evaluation and management are of a also be analysed to identify value-adding features
great importance to managers all over the world and then classified using a suitable framework.
(Deans & Adam, 2000; Ho, 1997; Kim, Shaw, & In another series of studies, Deans and Adam
Schneider, 2003). Ultimately, earlier evaluation (2000) identified three criteria: communication,
studies attempted to classify websites into typolo- transaction, and relationship, to analyse marketing
gies driven by the scholarly desire to identify and websites in Australian and New Zealand online
categorise website elements for each typology businesses. Businesses carry out different online
(Bauer & Scharl, 2000). These studies resulted functions depending on their overall website objec-
in identifying e-commerce business models (Goi, tives. Table 1 summarises the previous evaluation
2007; Hoffman, Novak, & Chatterjee, 1995; Wen, studies on websites.
Chen, & Hwang, 2000). These types of studies categorise websites
Other evaluations have sought to identify and according to their content; for example, a trans-
categorise website elements and link them with actional site contains content that allows for the
the purpose of the site. Forrester Research (1996) ordering of products/services or the execution of
defined three types of website: a promotional site financial transactions. If a website falls into that
that advertises a company’s products and ser- category then suggestions for the content and
vices; a content site that provides updated news, structure are made. These types of studies have
weather, or entertainment; and a transactional contributed to the field of website evaluation but
site that provides interactive shopping, banking, are broad based and serve only to advance the
or customer service. Ho (1997) classified the development of websites in terms of their structure
business purpose(s) of a commercial website into and contents. They are of little use, however, for
three categories: informational, transactional, and website owners to understand payoffs from certain
promotional; and that sites create value for their features. The recent literature on evaluating web-
visitors in four ways: timely, custom, logistic, and sites reports that the user’s perception of a website
sensational. These give a framework resulting in can strongly affect its usage and that the user is
12 possible features (purpose–value combina- affected by the positive experience with the in-
tions) a site could offer. Data were aggregated to terface. These studies address the topic of website
237
quality and user satisfaction with the website as affects the number of people who visit the site
a means of website evaluation. Several instruments and subsequently conduct business on it (Udo &
have been developed to define, measure, and Marquis, 2001). These studies have also served
manage the quality of websites (Aladwani & to improve website designs in terms of “what to
Palvia, 2002; Barnes, 2003; Katerattanakul & include” but offer little to inform businesses on
Siau, 1999; Loiacono, Watson, & Goodhue, 2002; success or profitability. Table 2 shows evaluation
Mich, Franch, & Gaio, 2003; Sharkey, Scott, studies on websites from various perspectives.
Galway, & Acton, 2007; Webb & Webb, 2004; P.
Zhang & Von Dran, 2002). Similarly, instruments
have been developed to define and measure user WEBSITE SUCCESS
satisfaction with the website (Bailey & Pearson, MEASUREMENT MODELS
1983; Baroudi & Orlikowski, 1988; Bharati &
Chaudhury, 2004; Doll & Torkzadeh, 1988; McK- Other researchers have emphasised the importance
inney, Yoon, & Zahedi, 2002; Muylle, Moenaert, of combining the different aspects of websites
& Despontin, 2004; Wang, Tang, & Tang, 2001; such as website quality and user satisfaction in a
Zviran, Chanan, & Itay, 2006). Other evaluation framework. This is to acknowledge the multidi-
frameworks classify websites according to various mensionality of website performance concept that
functionalities through the exposition of the dif- cannot be captured in a single measure (DeLone
ferences in “score” (Elliot, Moّrup-Petersen, & & McLean, 2004; Molla & Licker, 2001). Based
Bjoّrn-Andersen, 2000). These particular evalu- on the work of Shannon & Weaver (1949); Mason
ation efforts restrict themselves to ratings of (1978), DeLone & McLean (1992) have developed
features and do not link evaluations to outcome a taxonomy of information system success. The
measures. classic model of communication, source-channel-
Another school of thought on evaluation takes receiver, by Shannon & Weaver (1949) offers a
the designer perspective and reveals features general theory for sending and receiving mes-
that are imperative for websites (Auger, 2005; sages through a channel and a way of analysing
Day, 1997; Drèze & Zufryden, 1997; Kim, et al., communication problems. The model consisted
2003; Kohavi & Parekh, 2003; Mich, et al., 2003; of a sender (a source of information), a transmis-
Olsina, Lafuente, & Rossi, 2000; Palmer, 2002; sion medium (with noise and distortion), and a
Spiliopoulou, 2000; Turban & Gehrke, 2000; Udo receiver (whose seeks to reconstruct the sender’s
& Marquis, 2001; P. Zhang, Small, Von Dran, & message). Communication is successful when the
Barcellos, 1999). The information gained from receiver accurately reconstructs the message and
these studies is important to designers who face is more effective when the meaning conveyed to
the task of planning future website designs. Others the receiver leads to the desired outcome expected
have emphasised the effectiveness of websites in by the sender.
terms of its visitors (Alpar, 2001; Huizingh, 2002; Mason (1978) used the communication model
Quaddus & Achjari, 2005). of Shannon & Weaver (1949) to conceptualise the
The underlying premise of these studies is performance of a communication channel and
that the success of the website is measured by suggested a framework exhibiting three criteria
how satisfied the users are with it and then how of success, each of which is necessary to capture
willing they would be to revisit the site. The changes in performance:
more frequently visitors return to a website, the
greater the probability they will make a purchase. • Technical: how accurately the message is
For example, maintaining a high quality website transferred to the receiver.
238
Table 2. Concept map and prior literature (Webster & Watson, 2002)
AUTHORS Perspective Concepts

Customer Owner Designer Website Website Website Financial Owner
Features Quality Usage Returns Satisfaction
Hoffman &Novak, 1996 X X
Zhang et al., 1999 X X X
Olsina et al., 2000 X X X
Spiliopoulou 2000 X X X
Elliot et al., 2000 X X
Thelwall, 2000 X X X
Turban & Gehrke, 2000 X X X
Udo & Marquis, 2001 X X
Barua et al., 2001 X X X X
Palmer, 2002 X X
Chakraborty et al., 2002 X X X
Molla & Licker 2002 X X X X
Barnes & Vidgen, 2003 X X
Olson & Boyer, 2003 X X
Kim et al., 2003 X X X
Kohavi & Parekh, 2003 X X
Lu, 2003 X X X
Mich et al., 2003 X X
Zhu & Kraemer, 2003 X X X
Teo & Pian 2004 X X
Gonza`lez et al., 2004 X
DeLone & MacLean, 2004 X X X X
Webb, 2004 X X
Quddas & Achjari, 2006 X X X
Auger, 2005 X X X X
Pujani & Xu 2006 X X X X X
Katerattanakul & Siau, 2006 X
Huizingh et al., 2007 X X
Fisher, 2007 X X X
Hong, 2007 X X X X X
Tang & Huang, 2008 X X
Harison & Boonstra, 2008 X X X
Ramanathan, 2010 X X X
239
Figure 3. Initial research model and its creation as applied to IS (DeLone & McLean, 2003)
• Semantic: how precisely the receiver re- this model and validated the links between its
ceives the intended message. constructs. This model has been used extensively
• Effectiveness: how effective is the channel in information systems research.
in terms of the benefit accrued to the stake- Molla and Licker (2001) extended the original
holders (sender or receiver) information systems success model of DeLone
and McLean (1992) to measure e-commerce
DeLone and McLean (1992) interpreted the system success. As noted by Molla & Licker
three success dimensions identified by Mason, (2001), a website has additional attributes (e.g.
(1978) into a high level research model as shown 24/7 availability, page loading speed, accessibility,
in Figure 3. Along with the empirical management up-to-date accuracy, accessibility and so forth)
information systems literature, the high level which distinguish it from a traditional informa-
model has guided them to a multidimensional tion system. It may, therefore require additional
lower level model of information success (Figure constructs to capture these business functions as
4) to provide guidance to future researchers. shown in Figure 5.
In the model showed in Figure 4, “systems DeLone and McLean’s work is still contribut-
quality” measures technical success; “information ing towards a universal model (Petter & McLean,
quality” measures semantic success; and “use,” 2009), which many have employed when looking
“user satisfaction,” “individual impact,” and at information system performance (Ballantine et
“organisational impacts” measure effectiveness. al., 1996; Pitt et al., 1995; Rai et al. 2002; Seddon,
Since 1992, the DeLone and McLean informa- 1997). Although Seddon (1997) argued to exclude
tion system success model has been central for “usage” from the model and used ‘net benefit’ in
research addressing information systems success his characterisation of the outcome, These modi-
(Ballantine, Bonner, Levy, & Martin, 1996; Pitt, fications were endorsed by Delone and McLean
Watson, & Kavan, 1995; Rai, Lang, & Welker, in 2002 with an updated information systems
2002; Seddon, 1997). Many studies have tested success model (DeLone & McLean, 2003; DeLone
Figure 4. DeLone and McLean IS success model (DeLone & McLean, 1992)
240
Figure 5. E-commerce system success model (Molla &Licker, 2001)
& McLean, 2002) as shown in Figure 6. The The DeLone and McLean Model exhibits the
updated model identified six interrelated dimen- three criteria of success identified by Mason (1978)
sions of information systems success. It sug- discussed above: system quality (technical), in-
gested that the quality of the content, system, and formation quality (semantic), use, user satisfac-
service of the information systems, determine the tion, and net benefit (effectiveness). It has also
users’ intention to use and their actual use and been extended to measure website effectiveness.
satisfaction with the information systems. The DeLone and McLean (2004) asserted that the
more satisfied they are with the system, the more model can be adapted to the measurement chal-
they will use it, and this determines the benefits lenges of e-commerce.
that they obtain from it. The benefits then reinforce
the users’ intention to use, their actual use, and
their satisfaction with the information systems
(DeLone & McLean, 2003).
Figure 6. Updated DeLone and McLean IS success model (DeLone & McLean, 2003)
241
CONCLUDING ISSUES is creating value; they are the ones who need to
AND RESEARCH GAP know the return for their IT investment (Tallon
& Kraemer, 2002). The objective of a website
While each of the evaluative studies and the web- that sells products and services is to maximise
site success models made significant contributions, profit. This, according to Krishnamurthy (2003),
a number of observations can be drawn. can only be achieved through maximising value
to the customer. Hence the need to articulate a
Perspective Web-unique value proposition, communicate it
and deliver it to the customers. This may lead to
Stakeholder perspective on measurement is an a future transaction. The effectiveness of such
important issue in the evaluation process. Sed- value, according to Schaupp, Fan, & Belanger
don, Staples, Patnayakuni and Bowtell (1999) (2006), is determined by gathering clickstream
argued that success should be investigated from data to make inferences regarding the website’s
whose perspective. Other researchers also called success. On the other hand, financial returns
for stakeholder consideration in the evaluation remain the tangible benefit the business accrues
process and pointed out that whose perspective from investing in such a website (Barua et al.,
is considered critical in the determination of suc- 2001; Barua et al., 1995). Other studies, however,
cess. Other researchers also called for stakeholder indicated that owner satisfaction is the ultimate
consideration in the evaluation process and pointed website performance measure that reflects both
out that whose perspective is considered critical tangible and intangible benefits (Huizingh et al.,
in the determination of success (Belanger et al., 2007). Therefore, website performance measure-
2006; Mich, et al., 2003). Indeed, the evaluation ments need to encapsulate value creation, usage,
perspective reflects the stakeholder experience financial returns and owner satisfaction. Most
point of view (Lagsten, 2011). For example, us- importantly, these should be viewed from the
ers hold a central stake when user satisfaction is business’ perspective.
under consideration under consideration (Stock- There have been a number of attempts to
dale & Lin, 2008); but website statistics may have include the owner perspective (Huizingh et al.,
greater influence on owner’s satisfaction with the 2007; E. Huizingh, 2002; Lu, 2003; Quaddus &
website. In both cases, users and owners are both Achjari, 2005; Stockdale, Lin, & Liu, 2006; Teo
stakeholders in the website. & Pian, 2004; Zhu & Kraemer, 2003). However,
Table 2 indicates that most evaluative ap- these are either incomplete Lu, (2003) for example,
proaches to analyse the success of a website are included only one item, company satisfaction, to
based on the user perspective as opposed to the measure website performance) or used the already
owner perspective. Other studies considered fac- established user perspective tools to evaluate
tors used by designers as a basis for creating and websites. For example, Stokedale et al., (2006),
improving websites (Liu & Arnett, 2000; Olsina, guided by the instrument “eQual,” developed by
Lafuente, & Rossi, 2000); Turban & Gehrke, 2000; Barnes & Vidgen (2003), tried to identify critical
Udo & Marquis, 2001). While these insights of success factors for effective website usage by
understanding customer behaviours and designers’ small businesses from the owner’s perspective.
attitudes on the web are important, it is critical While “eQual” was developed from a “customer
to identify website effectiveness from the busi- voice” perspective, owners were able to see where
ness (owner) perspective (Ghandour, Benwell, & they would like to progress in the future, given
Deans, 2011). The owners’ perceptions help to pin- the resources. In summary, a tool is needed which
point areas within the business where e-commerce is based on the owner’s experience, rather than
242
customer experience, to enable owners to realise Integration

the benefits in order to progress or reinvest.
As apparent in the website evaluation section
Multidimensionality above, past studies are fragmented and lack com-
prehensiveness in capturing the full functionality
Many studies have argued that evaluation across of a website, e.g. integrating technical and human
one dimension has led to mixed results being perspectives into financial-perspective outputs.
reported in the literature (Gable, 1996; Myers et While the technical perspective represents the
al., 1998; Ramanathan, 2010; Segars & Grover, investment in the website, the customer interaction
1998). Previous studies that have taken the user represents the human perspective and the payoffs.
perspective have demonstrated the need to evalu- The technical perspective demonstrates the com-
ate across multi-dimensions and synthesised the munication effect of linking the customers with
different website evaluation criteria in one frame- the business. Also, it represents IT investment
work (DeLone & McLean, 1992; Molla & Licker, to present a value in the website. The customer
2001). On the other hand, studies that have taken interaction with the website is just as important
the owner perspective have based their evaluation as, with no customers visiting the website, the
on analysing website (features) functionalities system will have no value. Stakeholders need to
(Elliot et al., 2000), click stream data for website recognise the financial benefits of their investment.
improvements (Alpar, 2001; Quaddus & Achjari, Epstein (2004) suggested that the outputs which
2005), financial returns (Zhu, 2004), or owner lead to profitability and an increase in return on
satisfaction (Huizingh, 2002). While each of these investment (ROI), sales, and market share are:
studies has recognised the multidimensionality of channel optimisation, cost saving, customer
evaluation, none have combined these dimensions acquisition, customer loyalty and retention, and
into a single framework as a means of evaluation. value. In the case of a website, these benefits are
Finding the relationships between these dimen- the target output measures of performance that
sions will provide additional insights into the way are congruent with the predetermined business
evaluation is conducted. objectives and will determine the ultimate out-
come that the business will have in the long run,
Rigour of Context namely, profit (DeLone & McLean, 2004). Finally,
as noted by Epstein (2004), it is only by making a
Past research on success lacks rigor when it comes “business case” for e-commerce expenditure that
to measuring success dimensions. In some cases, owners can truly integrate e-commerce impacts
ad hoc success determinants were suggested into their business. A clear business case can be
(Campbell, 2001). DeLone and McLean (1992) presented by identifying metrics (with indicators)
suggest that dimensions and measures are devel- of e-commerce performance and their impact on
oped for a particular context. Yet, most studies do profitability. These indicators empower owners
not elaborate on the rationale for the inclusions with the information to evaluate whether the e-
of their success dimensions. The conceptualisa- commerce program is achieving its stated objec-
tion and the process of the instrument design are tives and is contributing to profitability.
confusing with regard to the contextual domains
employed. However, the conceptual context for the Website Type
present study is that the website is a business issue
rather than a technical one (Ghandour, Benwell, Evaluation criteria differ between different types
& Deans, 2010; Thelwall, 2001). of websites as discussed by Kim and Lee (2002)
243
and Hong and Kim (2004). Zhang, von Dran, SUMMARY

Blake and Pipithsuksunt (2001) investigated the
most important features across different website This chapter has argued that e-commerce has
types and concluded that the website type was become a reality for most businesses and offers
important when dealing with evaluation. For ex- companies significant gains, opportunities and
ample, privacy/security is important for websites challenges. As it has become more widely adopted,
that transactional capability but not important for there has been increased pressure on businesses
education websites. On the other hand, there are to account for the investment in their website. At
features that are important for all type of websites the same time, interest in the evaluation of these
such as navigation. websites has grown. However, whereas general
A number of studies have investigated evalu- website effectiveness measures have been devel-
ation for different types of websites, health infor- oped and widely discussed in the literature, these
matics websites (Gaertner & Smith, 2001; Maryati, methods have typically been established from the
Ray, & Lampros, 2006; Yusof, Papazafeiropoulou, customer’s point of view. This resulted in website
Paul, & Stergioulas, 2008), online communities performance being measured in terms of design,
(Hsiu-Fen & Gwo-Guang, 2006), and tourism structure, and content. Although valuable in their
(Baloglu & Pekcan, 2006; Benckendorff & Black, own right, these measures give little insight to
2000; Ip, Law, & Lee, 2010; Morrison, Taylor, returns resulting from website investment. The
& Douglas, 2005; Park & Gretzel, 2007; Scharl, dilemma presented in this chapter is the absence
Wober, & Bauer, 2004). The evaluation criteria of a framework to capture website functionality
therefore, should consider the type of website. The from the owner’s perspective. A number of is-
type of website which is the subject of this chapter, sues were discussed that need to be included in
however, is a website that sells products/services. an owner perspective framework, which will be
In conclusion, and within the confines of the focus of the following chapter.
these issues, this section has identified a gap in
website evaluation research. This gap is a failure
to approach website evaluation from an owner’s REFERENCES
perspective that captures the functionalities of
the website as a business issue. Until now, there Al-Qirim, N. (2005). An empirical investigation of
have been no evaluation tools to diagnose the an e-commerce adoption-capability model in small
performance of a website as a business task or businesses in New Zealand. Electronic Markets,
a sales channel. This could enable businesses to 15(4), 418–437. doi:10.1080/10196780500303136
realise returns for their investment and help them Aladwani, A., & Palvia, P. (2002). Develop-
progress to a higher level of e-commerce activ- ing and validating an instrument for measuring
ity. This could also enable owners to benchmark user-perceived Web quality. Information & Man-
their own web experience against competitors. agement, 39(6), 467–476. doi:10.1016/S0378-
This chapter has addressed this research gap by 7206(01)00113-6
considering e-commerce success from the owner
perspective and sought to provide businesses with Alpar, P. (2001). Measuring the efficiency of
a guiding principle to evaluate their own websites. website traffic generation. International Journal
of Electronic Commerce, 6(1), 53–74.
244
Auger, P. (2005). The impact of interactivity and Belanger, F., Fan, W., Schaupp, L. C., Krishen,
design sophistication on the performance of com- A., Everhart, J., & Poteet, D. (2006). Website
mercial websites for small businesses. Journal success metrics: Addressing the duality of goals.
of Small Business Management, 43(2), 119–137. Communications of the ACM, 49(12), 114–116.
doi:10.1111/j.1540-627x.2005.00129.x doi:10.1145/1183236.1183256
Bailey, J., & Pearson, S. (1983). Development Benckendorff, P., & Black, N. (2000). Destination
of a tool for measuring and analyzing computer marketing on the Internet. The Journal of Tourism
user satisfaction. Management Science, 29(5), Studies, 11(1), 11–21.
530–545. doi:10.1287/mnsc.29.5.530
Bharati, P., & Chaudhury, A. (2004). An empirical
Ballantine, J., Bonner, M., Levy, M., & Martin, A. investigation of decision-making satisfaction in
(1996). The 3-D model of Information Systems Web-based decision support systems. Decision
success: The search for the dependent variable Support Systems, 37(2), 187–197. doi:10.1016/
continues. Information Resources Management S0167-9236(03)00006-X
Journal, 9(4), 5–14.
Bloch, M., Pigneur, Y., & Segev, A. (1996). On
Baloglu, S., & Pekcan, Y. (2006). The website the road of electronic commerce-A business value
design and internet site marketing practices of framework, gaining competitive advantage and
upscale and luxury hotels in Turkey. Tourism some research issues.
Management, 27(1), 171–176. doi:10.1016/j.
Campbell, K. (2001). Seven steps to success:
tourman.2004.07.003
Creating the perfect e-commerce system. Journal
Barnes, S., & Vidgen, R. (2003). Interactive e- of Convergence, 1(2), 11–18.
government: Evaluating the website of the UK
Chakraborty, G., Lala, V., & Warren, D. (2002).
inland revenue. Journal of Electronic Commerce
An empirical investigation of antecedents of
in Organizations, 2(1), 22.
B2B websites’ effectiveness. Journal of Internet
Baroudi, J., & Orlikowski, W. (1988). A short- Marketing, 16(4), 51–72. doi:10.1002/dir.10044
form measure of user information satisfaction: A
Cockburn, C., & Wilson, T. (1996). Business use
psychometric evaluation and notes on use. Journal
of the World Wide Web. International Journal
of Management Information Systems, 4(4), 44–59.
of Information Management, 16(2), 83–102.
Barua, Kriebel, C., & Mukhopadhyay, T. (1995). doi:10.1016/0268-4012(95)00071-2
Information technologies and business value: An
Day, A. (1997). A model for monitoring website
analytic and empirical investigation. Informa-
effectiveness. Internet Research: Electronic Net-
tion Systems Research, 6(1), 3–23. doi:10.1287/
working Applications and Policy, 7(2), 109–115.
isre.6.1.3
doi:10.1108/10662249710165244
Barua, Konana, P., Whinston, A., & Fang, Y.
Deans, K. R., & Adam, S. (2000). Internet survey
(2001). Driving e-business excellence. MIT Sloan
data collection: The case of WebQUAL. Paper
Management Review, 43(1), 36-44.
presented at the ANZMAC Conference, University
Bauer, C., & Scharl, A. (2000). Quanti- of New South Wales, Sydney.
tive evaluation of Web site content and
structure. Internet Research, 10(1), 31–44.
doi:10.1108/10662240010312138
245
DeLone, & McLean, E. R. (2003). The DeLone Feindt, S., Jeffcoate, J., & Chappell, C. (2002).
and McLean model of Information Systems suc- Identifying success factors for rapid growth in
cess: A ten-year update. Journal of Management SME e-commerce. Small Business Economics,
Information Systems, 19(4), 9–30. 19(1), 51–62. doi:10.1023/A:1016165825476
DeLone, W. H., & McLean, E. R. (1992). Informa- Foster, J., & Lin, A. (2004). Collaborative e-
tion Systems success: The quest for the dependent business planning: Developing an enterprise
variable. Information Systems Research, 3(1), learning tool for information management and
60–95. doi:10.1287/isre.3.1.60 Information Systems curricula. Journal of Elec-
tronic Commerce in Organizations, 2(2), 28–43.
DeLone, W. H., & McLean, E. R. (2002). Infor-
doi:10.4018/jeco.2004040103
mation Systems success revisited. Paper presented
at the The 35th Hawaii International Conference Gable, G. (1996). A multidimensional model of
on System Sciences-HICSS-35, Hilton Waikoloa client success when engaging external consul-
Village/Island of Hawaii. tants. Management Science, 42(8), 1175–1198.
doi:10.1287/mnsc.42.8.1175
DeLone, W. H., & McLean, E. R. (2004). Measur-
ing e-commerce success: Applying the DeLone Gaertner, N., & Smith, M. (2001). E-commerce
& McLean Information Systems success model. in a Web-based environment: Auditing rela-
International Journal of Electronic Commerce, tive advantages in the Australian health sector.
9(1), 31–47. Managerial Auditing Journal, 16(6), 347–365.
doi:10.1108/02686900110395497
Doll, W., & Torkzadeh, G. (1988). The measure-
ment of end-user computing satisfaction. Man- Ghandour, A., Benwell, G., & Deans, K. (2010).
agement Information Systems Quarterly, 12(2), Measuring ecommerce website success. Interdis-
259–274. doi:10.2307/248851 ciplinary Journal of Contemporary Research In
Business (IJCRB), 1(12).
Drèze, X., & Zufryden, F. (1997). Testing web-
site design and promotional content. Journal of Ghandour, A., Benwell, G., & Deans, K. (2011).
Advertising Research, 37(2), 77–91. Measuring the performance of ecommerce web-
sites–An owner’s perspective. Pacific Asia Jour-
Eastin, M. S. (2002). Diffusion of e-commerce:
nal of the Association for Information Systems,
An analysis of the adoption of four e-commerce
3(1), Article 2.
activities. Telematics and Informatics, 19(3),
251–267. doi:10.1016/S0736-5853(01)00005-3 Ghandour, A., Benwell, G., & Deans, K. R. (2007).
The contribution of goal setting to the success of
Elliot, S. R. Moّrup-Petersen, A. S., & Bjoّrn-
ecommerce systems among small and medium
Andersen, N. (2000). Towards a framework for
enterprises in New Zealand. Paper presented at the
evaluation of commercial websites. Paper pre-
Pan-Pacific Conference XXIV of the Pan-Pacific
sented at the 13th International Bled Electronic
Business Association.
Commerce Conference, Bled, Slovania.
Goi, C. L. (2007). A review of existing website
Epstein, M. J. (2005). Implementing successful
models for e-commerce. Journal of Internet Bank-
e-commerce initiatives. Strategic Finance, 86(9),
ing and Commerce, 12(1), 1–17.
22–29.
246
Grandon, E., & Pearson, J. M. (2004). Electronic Huizingh, K. A., Bijmolt, T., & Hoekstra, J. (2007).
commerce adoption: An empirical study of small How important are transactional or informational
and medium US businesses. Information & functions for website success? Paper presented
Management, 42(1), 197–216. doi:10.1016/j. at the ANZMAC 2007, Dunedin, New Zealand.
im.2003.12.010
Hung, W., & McQueen, R. (2004). Developing an
Han, K., & Noh, M. (1999). Critical failure fac- evaluation instrument for e-commerce websites
tors that discourage the growth of electronic from the first-time buyer’s viewpoint. The Elec-
commerce. International Journal of Electronic tronic Journal of Information Systems Evaluation,
Commerce, 4(2), 25–43. 7(1), 31–42.
Ho, J. (1997). Evaluating the World Wide Web: Ip, C., Law, R., & Lee, H. A. (2010). A review
A global study of commercial sites. Journal of of website evaluation studies in the tourism and
Computer-Mediated Communication, 3(1). Re- hospitality fields from 1996 to 2009. International
trieved from http://www.ascusc.org/jcmc/vol3/ Journal of Tourism Research, Early View (Online
issue1/ho.html Version of Record published before inclusion in
an issue).
Hoffman, D., Novak, T., & Chatterjee, P. (1995).
Commercial scenarios for the Web: Opportunities Katerattanakul, P., & Siau, K. (1999). Measuring
and challenges. Journal of Computer-Mediated information quality of websites: Development of an
Communication, 1(3), 1–20. instrument. Paper presented at the The Twentieth
International Conference on Information Systems.
Hong, W., & Zhu, K. (2006). Migrating to
Internet-based e-commerce: Factors affecting e- Kim, S., Shaw, T., & Schneider, H. (2003).
commerce adoption and migration at the firm level. Website design benchmarking within indus-
Information & Management, 43(2), 204–221. try groups. Internet Research, 13(1), 17–26.
doi:10.1016/j.im.2005.06.003 doi:10.1108/10662240310458341
Hong, & Kim. (2004). Architectural criteria for Kohavi, R., & Parekh, R. (2003). Ten supplemen-
website evaluation–Conceptual framework and tary analyses to improve e-commerce websites.
empirical validation. Behaviour & Information Paper presented at the Fifth WEBKDD Workshop,
Technology, 23(5), 337-357. Washington DC, USA.
Hsiu-Fen, L., & Gwo-Guang, L. (2006). Krishnamurthy, S. (2003). E-commerce manage-
Determinants of success for online comment: Text and cases. Mason, OH: Thomson/
munities: An empirical study. Behaviour & South-Western.
Information Technology, 25(6), 479–488.
Lagsten, J. (2011). Evaluating Information Sys-
doi:10.1080/01449290500330422
tems according to stakeholders: A pragmatic
Huizingh, E. (2002). The antecedents of perspective and method. Electronic Journal
website performance. European Jour- Information Systems Evaluation, 14(1), 73–88.
nal of Marketing, 36(11/12), 1225–1247.
Liu, C., & Arnett, K. (2000). Exploring the
doi:10.1108/03090560210445155
factors associated with website success in the
context of electronic commerce. Information &
Management, 38(1), 23–33. doi:10.1016/S0378-
7206(00)00049-5
247
Loiacono, E., Watson, R., & Goodhue, D. (2002). Muylle, S., Moenaert, R., & Despontin, M. (2004).
WebQual: A measure of website quality. In Evans, The conceptualization and empirical valida-
K., & Scheer, L. (Eds.), Marketing theory and tion of website user satisfaction. Information
applications (Vol. 13, pp. 432–438). Chicago, IL: & Management, 41(5), 543–560. doi:10.1016/
American Marketing Association. S0378-7206(03)00089-2
Lu, J. (2003). A model for evaluating e-commerce Myers, B., Kappelman, L., & Prybutok, V. (1998).
based on cost/benefit and customer satisfaction. A comprehensive model for assessing the quality
Information Systems Frontiers, 5(3), 265–277. and productivity of the Information Systems func-
doi:10.1023/A:1025645210350 tion: Toward a theory for Information Systems
assessment. In Garritty, E., & Sanders, G. (Eds.),
Maryati, M., Ray, J., & Lampros, K. (2006). To-
Information Systems success measurement (pp.
wards a framework for health Information Systems
94–121). Hershey, PA: Idea Group.
evaluation. Paper presented at the 39th Annual
Hawaii International Conference on System Sci- Ng, H., & Wilson, T. (1998). Business use of
ences (HICSS’06) Track 5. the World Wide Web: A report on further inves-
tigations. International Journal of Information
Mason, R. (1978). Measuring information output:
Management, 18(5), 291–314. doi:10.1016/
A communication systems approach. Information
S0268-4012(98)00021-8
& Management, 1(4), 219–234. doi:10.1016/0378-
7206(78)90028-9 Olsina, L., Lafuente, G., & Rossi, G. (2000).
E-commerce site evaluation: A case study. Paper
McKinney, V., Yoon, K., & Zahedi, F. (2002).
presented at the 1st International Conference on
The measurement of Web-customer satisfaction:
Electronic Commerce and Web Technologies
An expectation and disconfirmation approach.
(EC-Web 2000).
doi:10.1287/isre.13.3.296.76 Palmer, W. (2002). Website usability, design,
and performance metrics. Information Sys-
Mich, L., Franch, M., & Gaio, L. (2003). Evalu-
tems Research, 13(2), 151–167. doi:10.1287/
ating and designing the quality of websites: The
isre.13.2.151.88
2QCV3Q metamodel. IEEE MultiMedia, 10(1),
34–43. doi:10.1109/MMUL.2003.1167920 Park, Y., & Gretzel, U. (2007). Success factors
for destination marketing websites: A qualita-
Molla, A., & Licker, P. (2001). E-commerce system
tive meta-analysis. Journal of Travel Research,
success: An attempt to extend and respecify the
46-63(1), 46.
DeLone and McLean model of success. Journal
of Electronic Commerce research, 2(4), 131-141. Pease, W., & Rowe, M. (2005). Diffusion of in-
novation - The adoption of electronic commerce
Morrison, A., Taylor, J., & Douglas, A. (2005).
by small and medium enterprises (SMEs) - A
Website evaluation in tourism and hospitality.
comparative analysis. Australasian Journal of
Journal of Travel & Tourism Marketing, 17(2),
233–251. doi:10.1300/J073v17n02_18
Petter, S., & McLean, E. R. (2009). A meta-analytic
assessment of the DeLone and McLean IS suc-
cess model: An examination of IS success at the
individual level. Information & Management,
46(3), 159–166. doi:10.1016/j.im.2008.12.006
248
Piris, L., Fitzgerald, G., & Serrano, A. (2004). Scharl, A., Wober, K., & Bauer, C. (2004). An
Strategic motivators and expected benefits from integrated approach to measure website effec-
e-commerce in traditional organisations. Interna- tiveness in the European hotel industry. Infor-
tional Journal of Information Management, 24(6), mation Technology and Tourism, 6(4), 257–271.
489–506. doi:10.1016/j.ijinfomgt.2004.08.008 doi:10.3727/1098305032781157
Pitt, L., Watson, R., & Kavan, C. (1995). Service Schaupp, L., Fan, W., & Belanger, F. (2006).
quality: A measure of Information Systems ef- Determining success for different website goals.
fectiveness. Management Information Systems Paper presented at the The 39th Annual Hawaii
Quarterly, 19(2), 173–187. doi:10.2307/249687 International Conference on System Sciences
(HICSS’06) Track 6.
Quaddus, M., & Achjari, D. (2005). A model for
electronic commerce success. Telecommunica- Schubert, P., & Selz, D. (2001). Measuring the ef-
tions Policy, 29(2-3), 127–152. doi:10.1016/j. fectiveness of e-commerce websites. In Barnes, S.,
telpol.2004.11.009 & Hunt, B. (Eds.), E-commerce & v-business (pp.
83–102). Oxford, UK: Butterworth-Heinemann.
Quelch, J., & Klein, L. (1996). The Internet and
international marketing. Sloan Management Re- Seddon, P. (1997). A respecification and extension
view, 37(3), 60–75. of the DeLone and McLean model of IS success.
Rai, A., Lang, S., & Welker, R. (2002). Assessing
doi:10.1287/isre.8.3.240
the validity of IS success models: An empirical test
and theoretical analysis. Information Systems Re- Seddon, P., Staples, S., Patnayakuni, R., & Bowtell,
search, 13(1), 50–69. doi:10.1287/isre.13.1.50.96 M. (1999). Dimensions of Information Systems
success. Communications of AIS, 2(20), 1–61.
Ramanathan, R. (2010). E-commerce success
criteria: determining which criteria count most. Segars, A., & Grover, V. (1998). Strategic Informa-
Electronic Commerce Research, 10(2), 191–208. tion Systems planning success: An investigation of
doi:10.1007/s10660-010-9051-3 the construct and its measurement. Management
Rockart, J. (1979). Chief executives define their
doi:10.2307/249393
own data needs. Harvard Business Review, 57(2),
81–93. Shannon, C., & Weaver, W. (1949). The math-
ematical theory of communication. Urbana, IL:
Rogers, E. (1983). Diffusion of innovation (3rd
University of Illinois Press.
ed.). New York, NY: Free Press.
Sharkey, U., Scott, M., Galway, I., & Acton, T.
Rogers, E. (2003). Diffusion of innovations (5th
(2007). The influence of quality on the success
ed.). New York, NY: Free Press.
of e-commerce systems. National University of
Samiaji, S., & Zowghi, D. (2005). Information Ireland.
Technology adoption process within Indonesian
Spiliopoulou, M. (2000). Web usage mining for
SMEs: An emperical study. Paper presented at
website evaluation. Communications of the ACM,
the 16th Australian Conference on Information
43(8), 127–134. doi:10.1145/345124.345167
Systems.
249
Stockdale, R., & Lin, C. (2008). Evaluation of Vijayaraman, B., & Bhatia, G. (2002). A frame-
the effectiveness of small and medium sized work for determining success factors of an
businesses websites in a business to business e-commerce initiative. Journal of Internet Com-
context. In Calero, C., Muñoz, C., Moraga, A., merce, 1(2), 63–75. doi:10.1300/J179v01n02_05
& Piattini, M. (Eds.), Handbook of research on
Wang, Y., Tang, T., & Tang, J. (2001). An instru-
Web information systems quality (pp. 71–85).
ment for measuring customer satisfaction toward
Hershey, PA: Information Science Reference.
websites that market digital products and services.
doi:10.4018/978-1-59904-847-5.ch004
Journal of Electronic Commerce Research, 2(3),
Stockdale, R., Lin, C., & Liu, Y. (2006). The 89–102.
perception of website effectiveness by West Aus-
Webb, H., & Webb, L. (2004). SiteQual: An in-
tralian small business owners in a B2B context.
tegrated measure of website quality. The Journal
Paper presented at the Pacific Asia Conference
of Enterprise Information Management, 17(6),
on Information Systems (PACIS06).
430–440. doi:10.1108/17410390410566724
Stockdale, R., Lin, C., & Stoney, S. (2005). The
Webster, J., & Watson, R. (2002). Analyzing the
effectiveness of SME websites in a business to
past to prepare for the future: Writing a literature
business context. Paper presented at the The IADIS
review. Management Information Systems Quar-
E-Commerce Conference.
terly, 26(2), 13–23.
Sung, T. (2006). E-commerce critical success fac-
Wen, H., Chen, H., & Hwang, H. (2000). E-
tors: East vs. West. Technological Forecasting and
commerce website design: Strategies and models.
Social Change, 73(9), 1161–1177. doi:10.1016/j.
Information Management & Computer Security,
techfore.2004.09.002
9(1), 5–12. doi:10.1108/09685220110366713
Tallon, P., & Kraemer, K. (2002). Executives’
Yusof, M., Papazafeiropoulou, A., Paul, R., &
perspectives on IT: Unraveling the link between
Stergioulas, L. (2008). Investigating evaluation
business strategy, management practices and IT
frameworks for health Information Systems. In-
business value. Paper presented at the America’s
ternational Journal of Medical Informatics, 77(6),
Conference on Information Systems (ACIS2002).
377–385. doi:10.1016/j.ijmedinf.2007.08.004
Teo, T., & Pian, Y. (2004). A model for Web adop-
Zhang, P., Small, R., Von Dran, G., & Barcellos,
tion. Information & Management, 41(4), 457–468.
S. (1999). Websites that satisfy users: A Theoreti-
doi:10.1016/S0378-7206(03)00084-3
cal framework for Web user interface design and
Thelwall, M. (2001). Effective websites for small evaluation. Paper presented at the The 32nd An-
and medium-sized enterprises. Journal of Small nual Hawaii International Conference on System
Business and Enterprise Development, 7(2), Sciences, HICSS-32.
149–159. doi:10.1108/EUM0000000006836
Zhang, P., & Von Dran, G. (2002). User expecta-
Turban, E., & Gehrke, D. (2000). Determinants of tions and rankings of quality factors in different
e-commerce website. Human Systems Manage- website domains. International Journal of Elec-
ment, 19, 111–120. tronic Commerce, 6(2), 9–33.
Udo, G., & Marquis, G. (2001). Factors affect-
ing e-commerce website effectiveness. Journal
250
Zhang, P., von Dran, G. M., Blake, P., & Pipith- Chiou, W. C., Perng, C., Tsai, J. T., & Lin, C.
suksunt, V. (2001). Important design features in C. (2008). The review of the website evaluation
different website domains. e-Service Journal, framework in Information Systems and marketing
1(1), 77. journals. International Journal of Information
Systems and Change Management, 3(1), 81–104.
Zhu, K., & Kraemer, K. (2003). E-commerce
doi:10.1504/IJISCM.2008.019290
metrics for net-enhanced organizations: As-
sessing the value of e-commerce to firm perfor- Epstein, M. J. (2004). Implementing e-commerce
mance in the manufacturing sector. Information strategies: A guide to corporate success after
Systems Research, 13(3), 275–295. doi:10.1287/ the dot.com bust (1st ed.). Westport, CT: Praeger
isre.13.3.275.82 Publishers.
Zhuang, Y., & Lederer, A. (2003). An instrument Harison, E., & Boonstra, A. (2008). Reaching new
for measuring the business benefits of e-commerce altitudes in e-commerce: Assessing the perfor-
retailing. International Journal of Electronic mance of airline websites. Journal of Air Trans-
Commerce, 7(3), 65–99. port Management, 14(2), 92–98. doi:10.1016/j.
jairtraman.2008.02.002
Zviran, M., Chanan, G., & Itay, A. (2006). User
satisfaction from commercial websites: The effect Hinton, M., & Barnes, D. (2009). Discover-
of design and use. Information & Management, ing effective performance measurement for e-
43(2), 157–178. doi:10.1016/j.im.2005.04.002 business. International Journal of Productivity
and Performance Management, 58(4), 329–345.
doi:10.1108/17410400910951008
ADDITIONAL READING Kleist, V. F. (2003). An approach to evaluat-
ing e-business Information Systems projects.
Barnes, D., & Hinton, M. (2007). Searching for Information Systems Frontiers, 5(3), 249–263.
e-business performance measurement systems. doi:10.1023/A:1025693109442
The Electronic Journal Information Systems
Evaluation, 10(1), 1–8. Law, R., Qi, S., & Buhalis, D. (2010). Progress
in tourism management: A review of website
Bremser, W., & Chung, Q. (2005). A framework evaluation in tourism research. Tourism Man-
for performance measurement in the e-business agement, 31(3), 297–313. doi:10.1016/j.tour-
environment. Electronic Commerce Research man.2009.11.007
and Applications, 4, 395–412. doi:10.1016/j.
elerap.2005.07.001 Mohini, S., & John, B. (2005). Performance evalu-
ation of e-business in Australia. The Electronic
Brown, I., & Jayakody, R. (2009). B2C e-Com- Journal of Information Systems Evaluation, 8(1),
merce Success: A test and validation of a revised 71–80.
conceptual model. The Electronic Journal of
Information Systems Evaluation, 12(2), 129–148. Van der Heijden, H. (2003). Factors influenc-
ing the usage of websites: The case of a generic
portal in The Netherlands. Information & Man-
agement, 40(6), 541–549. doi:10.1016/S0378-
7206(02)00079-4
251
KEY TERMS AND DEFINITIONS services, through which customers interact with
for bidirectional information transfer, a platform
Critical Success Factor (CSF): CSF was first for transacting, and an interface for services.
coined by Rockart (1979). He defined CSF as the Diffusion of Innovations: Diffusion of Inno-
limited number of areas in which results, if they vations is a theory developed by Roger 1983 that
are satisfactory, will ensure successful competitive seeks to explain how an innovation can be adopted
performance for the business. Perspective: From which the view is taken
DeLone and McLean Information Systems including the experience. For example, owner’s
Success Model: D&M IS Success Model is a perspective is related to the owner’s experience
framework developed by DeLone and McLean with their own website.
in 1992 to conceptualise the performance of Website Evaluation: Website evaluation is
information systems from the user perspective. the assessment and measurement of websites
The framework was based on a study by Mason performance.
(1978) that was informed by the communication Website Features: Website features are the
theory of Shannon (1949). messages communicated to the users for the
E-Commerce: E-commerce in the present purpose of exchanging value. They are functions
study is a means of selling products/service related to informational, transactional, service,
through a website and promotional capabilities of a website.
E-Commerce Website: E-commerce website
refers to the website that involves selling goods and
252
253
Chapter 13
Website Evaluation Criteria:
An Owner’s Perspective
Ahmad Ghandour
George L. Benwell
Kenneth R. Deans
ABSTRACT
There is often a need for business organisations to evaluate their current and potential website in order
to maximize the payoffs from website investments. Current evaluative approaches for the performance
of e-commerce websites do not adequately address owners’ concerns regarding the payoffs from their
e-commerce investment. This chapter establishes criteria to evaluate e-commerce websites based on an
owner’s perception rather than the customer’s perception. Drawing upon theories of communication,
resource based view and process oriented approach, an evaluation framework of three dimensions is
developed. The three dimensions are: website offer, usage, and payoff. These three dimensions are used
to explain the performance of a website, culminating in a website evaluation model. Each dimension is
a business process of the website that the organisation needs to monitor.
INTRODUCTION Despite the increased attempts to trade online and

the subsequent investments, managers still strug-
Many business organisations are motivated to gle to measure the success of their websites. These
invest in doing business online and have created attempts can results in ineffective e-commerce
websites for e-commerce transactions. Indeed, e- activities and the consequent disappointment in
commerce websites have proliferated as managers recognisable benefits. As a consequence, there is
are told that trading online is already leading to a need for businesses to evaluate their current and
reshaping of customer and supplier relationships. potential site functionalities in order to maximize
the payoffs from website investments. Current
DOI: 10.4018/978-1-4666-0170-3.ch013 evaluative approaches for the performance of
Website Evaluation Criteria
e-commerce websites fail to adequately address BACKGROUND AND

owners’ concerns about the payoffs from their MOTIVATION OF THE STUDY
e-commerce investment. An extensive search of
related literature established that no multi-item Business organisations that have a website as a
scale is available to measure the owners’ perceived sales channel need to be engaged in an evaluation
e-commerce website success. A provision of such effort as part of their management practices. Such
a scale would not only further enhance manage- evaluations promise strategic benefits such as a
ment’s ability to understand the payoffs from positive return on investment (Park & Gretzel,
their investment but also act as an indicator of the 2007; Stockdale & Lin, 2008) and help the pro-
weaknesses and strengths occurring in the web- gression to higher levels of e-commerce activity
site. An e-commerce website, however, is a sales (Ally, Cater-Steel, & Toleman, 2007).
channel (sometimes a business’s sole interface) Throughout the literature, e-commerce
between the business, customers and the world website evaluation has been studied from the
at large. In the contemporary competitive busi- user perspective or the technical aspect, while
ness environment, an innovative, well designed there are fewer studies that deal with the owner
and managed website can provide the advantage perspective (Barnes & Vidgen, 2003; DeLone
a business needs to conduct its e-commerce ac- & McLean, 2004; Elliot, Moّrup-Petersen, &
tivities successfully. However, and according to Bjoّrn-Andersen, 2000; Molla & Licker, 2001;
Epstein, (2004), making the commitment without Petre, Minocha, & Roberts, 2006; Quaddus &
thinking seriously about what drives a successful Achjari, 2005; Schubert & Selz, 2001; Torkzadeh
e-commerce website leads to inefficiency and to & Dhillon, 2003; Turban & Gehrke, 2000; Udo &
managerial dissatisfaction. Marquis, 2001; Van der Merwe & Bekker, 2003).
The remainder of this chapter is organised into Nevertheless, the owner perspective presents an
six sections. The first section examines theoretical important supplement to the user perspective as-
perspectives for measuring the performance of a pects, as they may provide answers to the impact
website. The primary purpose of this section is to of e-commerce investment (payoffs resulting from
build a rationale and theoretical basis for defining investment). The reasons for such shortcomings
website performance. The second section builds a are mainly due to the emphasis given to the user
high level model of inquiry based on the theoreti- and the need to understand what is suitable for
cal perspectives of section one. It identifies three them. Another reason is that owners are usually
dimensions of website performance that need to be not willing to divulge financial data or even user
measured. In this section the model is also submit- statistics on their websites, making the investment
ted to seven questions proposed by Cameron & payoff analysis more difficult.
Whetten, (1983) when measuring performance. Since the crash of the dotcoms in 2001, e-
The third section discusses the first dimension of commerce has progressed into another phase
website performance, website investment. The and investors are no longer interested in secur-
fourth section examines website usage which is ing a virtual space but rather conducting online
the second dimension of website performance. operations justified by an adequate return on
The fifth section discusses the third dimension of investment (Epstein, 2004). According to Ep-
website performance, the website payoff. Section stein (2004), businesses that failed to survive the
six concludes this chapter. crash were those with little to no managerial and
performance guidelines. As e-commerce begins
to mature, many researchers and owners have
recognised the need to identify and measure the
254
impact of their e-commerce expenditure. With online space (Amit & Zott, 2002; Constantinides,
measurement, businesses may generate more ef- 2002; Krishnamurthy, 2003). Krishnamurthy
fective e-commerce websites, and provide owners (2003) proposed an internet toolkit with six-Cs
with a better understanding of what may be of to deliver value. These are: Commerce (placing
practical value (Palmer, 2002). An evaluation and order and payment online), Communication, Con-
assessment of an e-commerce website against a nectivity (reaching any user that is connected to
framework will determine the level at which the the Internet, users among themselves and with
business currently stands (Ally, et al., 2007). How- employees can also be connected), Community
ever, the appropriate metrics have not been well (through social media networks), Content (de-
developed (Epstein, 2004). According to Segars tailed information), and Computing (such as track
& Grover (1998), multiple, interrelated success shipping). Zott, Amit, & Donlevy, (2000) also
dimensions are more likely to capture changes in identified four key value drivers: efficiency (for
performance than one single item or even a set of example reduce information asymmetries between
financial measures. Providing a framework, with buyers and owners), complementarities (value
the appropriate metrics, to evaluate e-commerce creation can be leveraged when bundled with
websites from the owners’ perspective is the theme complementary products from other suppliers),
of the present study. This study is motivated by lock-in (the ability to prompt users to engage in
value delivery, website usage metrics, and payoffs. repeat transactions), and novelty (the introduction
of novel processes).
Value Delivery In the online space, however, users interact with
a commercial website for informational, transac-
The popularity of online trading with the low en- tional and/or services purposes (Deans & Adam,
try cost has motivated many business to invest in 2000; Molla & Licker, 2001). Hence businesses
websites to sell goods and services (Stockdale, Lin, can create value when delivering these functions
& Stoney, 2005). This has also led many to prema- in terms of technology delivery, product delivery,
turely launch websites which then fail to deliver and customer delivery process (Ramanathan,
any value (Stockdale & Lin, 2008). This may be 2010; Sharma, Krishnan, & Grewal, 2001). Porter
a self-reinforcing cycle in that many owners have (2001) described how the internet creates value for
little knowledge of how to create a web-unique businesses by providing a standardised infrastruc-
proposition appealing to the target group(s). Amit ture, an intuitive browser interface for information
& Zott (2002) asserted that companies struggle access and delivery, bidirectional communication,
to survive if they fail to communicate a value. and ease of connectivity. Krishnamurthy (2003)
Further they suggested that the value creation suggested a five-step process to deliver value to
requirements for e-commerce are different from the user: the first step is to identify how different
that of traditional business. consumers perceive value. Step 2 is to choose
In the physical space (traditional business), the which value element will be delivered. Step 3 is
drivers of the value are the well-known four-Ps to provide the value and tell how value is being
of marketing: Product (that meet the needs of the delivered. Step 4 is to help users learn about the
customer), Price (great value for the product), nature of the value. Step 5 is to assess how users
Place (available at the right place, at the right time, perceive the value being delivered.
in the right quantity), and Promotion (how the Based on such requirements and suggestions,
targeted groups are informed about the product). the current study introduces the concept of “site
Further, researchers have suggested replacing the offer” as the value to the user.
traditional marketing mix four-Ps with one for the
255
Website Usage Metrics (Giaglis, Paul, & Doukidis, 1999; Lin, Huang,
& Tseng, 2007; Segars & Grover, 1998). Often
Online technology is able to collect large amounts e-commerce initiatives fail to deliver what is ex-
of detailed data on visitor traffic and activities on pected because businesses focus on technology
websites. Such data offer a plethora of metrics and do not measure the intangibles (Giaglis, et al.,
which businesses must carefully choose from 1999; Lin, et al., 2007). In addition, businesses
for different purposes (Clifton, 2010; Phippen, are failing to understand the ways in which e-
Sheppard, & Furnell, 2004). Otherwise, the sheer commerce investment leads to a payoff (Kohli,
amount of data available can be overwhelming, et al., 2003). These challenges along with the
as can the multitude of ways to compare it. Cur- emphasis on the owner’s perspective (that is, the
rently, there are tools that provide various statis- owner’s experience) have generated new interest
tics about website usage (for example, Google and research on approaches to success measure-
analytics, LiveSTATS.XSP, and CMS400.NET). ment of e-commerce websites.
Such tools only provide raw metrics potentially
of little benefit to businesses using them (Clif-
ton, 2010; Hong, 2007; Phippen et al., 2004; M. THE PROBLEM
Spiliopoulou, Pohle, & Faulstich, 2000). In addi-
tion, even though web metrics may be capable of E-commerce has become a reality for most busi-
extracting interesting website usage patterns and nesses and simultaneously constitutes a great
could indicate how users are actually using the opportunity and a great challenge for these busi-
website, the link to business performance is still nesses. It has become more widely adopted and
very tenuous (Hahn & Kauffman, 2001). Website pressure on businesses to account for investment
owners would benefit from converting these raw in their website has increased; consequently inter-
metrics into meaningful information that can be est in the evaluation of these websites has grown.
used to evaluate the performance of e-commerce However, whereas general website effectiveness
websites. measures have been developed and widely dis-
cussed in the literature, the development of website
Payoffs evaluation methods has been typically taken from
the user point of view leading to measurement
The difficulty in articulating the benefits derived of the performance of websites in terms of their
from being online has been a major impediment design, structure, and content. Although valuable
to e-commerce developments. Many researchers to a great extent, these measures give little direct
and owners have recognised the need to measure insight to payoffs resulting from investment. This
the benefits and value resulting from e-commerce study is focused on evaluating the websites from
initiatives (Epstein, 2004; Kohli, Sherer, & the owner’s point of view. Specifically, this study
Baron, 2003). The challenge is to quantify the aims to address the following questions:
link between investment and payoffs. Careful
identification and measurement of the payoffs (1) What are the appropriate dimensions and
permit businesses to demonstrate the impact measures for evaluating the performance of a
on profitability and value creation. They also website based on the owner point of view?
provide information for resource allocation deci-
sions. However, the payoffs from implementing (2) Can the dimensions and measures be combined
e-commerce are not only calculated in monetary to yield a testable model that determines website
terms but other, intangible benefits are essential effectiveness?
256
THEORETICAL PERSPECTIVES transactions between a business and its custom-

ers (i.e. B2B and B2C). Through the website,
In order to develop a framework for website per- businesses communicate, transact, promote and
formance evaluation, a theoretical background is provide services to their customers. This has led
needed (Kim & Lee, 2002). Existing studies have to the view that a website is a (an):
emerged from different theoretical backgrounds
and have been informed by somewhat dissimilar 1. IT investment that can create competitive
goals. The objective for this study is to provide value which impacts the performance of a
owners with a set of guiding principles to evalu- business in terms of IT payoffs (Bharadwaj,
ate their website. This would enable businesses to 2000; Mata, Fuerst, & Barney, 1995;
generate insights into how performance evaluation Melville, Kenneth, & Vijay, 2004). This
could be conducted and what tools and techniques resource-based view has been used to ex-
need to be developed. While the interdisciplin- plain website effectiveness determined by
ary nature of e-commerce (Al-Qirim, 2003) its capabilities (Zhu, 2004; Zhu & Kraemer,
adds further layers of complexity into a website 2003).
performance, it requires the integration of the 2. Process model of creation where the
relevant theoretical perspectives from the different benefits of the website are accrued to the
disciplines. The different theoretical perspectives business, accumulated through users. This
will help to explain the multiple aspects related process-oriented view has been used to
to the performance of the website. explain the change of website performance
On one hand, the website is a technical issue in relation to its features (Barua, Konana,
and the user perspective is the basis of evaluation Whinston, & Fang, 2001; Barua, Kriebel,
of the effectiveness of the website’s performance. & Mukhopadhyay, 1995).
This perspective views the website as:
The following subsections discuss only the
1. An information system (IS) that has a large resource based view and process oriented ap-
number of users who are not confined by proach as the current chapter takes the owner’s
its organisational context and for whom perspective.
web use is volitional. From this stems the
justification to use IS theories to explain Resource Based Theory
website performance (DeLone & McLean,
2004; Hao, Duo-lin, & Zhi-jie, 2010; Molla Unlike the approach of looking at businesses
& Licker, 2001). in terms of their products, the Resource-Based
2. A marketing channel with two-way commu- View (RBV) looks at businesses in terms of their
nication channel between the business and resources. This approach focuses on the benefits
its customers (Kent, 1998; Detmar. Straub, accrued to the business from business-specific
Hoffman, Weber, & Steinfield, 2002). The resources rather than the economic profits from
communication theory can then be applied product market. RBV links business performance
to explain website performance (DeLone & to organisational resources and capabilities.
McLean, 2004; Molla & Licker, 2001). Resources such as managerial skills, IT exper-
tise, organisational skills, and intangible assets,
However, the owner perspective views the singularly or in combination, create sustainable
website as a business task (resource). The website advantages by enabling distinctive abilities within
is the medium of business communications and
257
a business (Barney, 1991; Teece, Pisano, & Shuen, tional performance measures, that define a broader
1997; Wernerfelt, 1984). conceptualisation of organisational performance
Information Technology (IT) attributes have by focusing on factors that ultimately lead to
been studied as resources that can create com- financial performance (Hofer & Sandberg, 1987;
petitive value which impact on the performance Kaplan, 1983). Performance measurement could
of a business in terms of IT payoffs (Bharadwaj, be improved by examining both operational and
2000; Mata, et al., 1995; Melville, et al., 2004; financial measures (Venkatraman & Ramanujam,
D. Straub & Klein, 2001). Straub & Klein (2001) 1986). The extensive literature review by Mur-
asserted that resources introduced by the internet phy, Trailer, & Hill, (1996) concluded that it is
lead to sustainable competitive advantage for the important to use not only multiple dimensions of
businesses engaged in Internet activities. Teece, performance but also multiple measures for each
et al., (1997) used RBV to address the realities dimension used.
of high-velocity markets and rapid technological Barua et al., (1995) outlined a generalised
change from the Dynamic Capabilities Perspec- process oriented approach and empirically tested a
tive (DCP). According to Teece et al., (1997), two-stage approach to assessing the impacts of IT
this perspective explains how businesses achieve investments. IT investments showed a significant
new forms of competitive advantage by renewing positive effect on intermediate operational per-
technological, organisational, and managerial re- formance, which in turn may affect higher levels
sources to achieve congruence with the changing of financial performance. The process oriented
business environment. approach has stimulated in a stream of research
Zhu & Kraemer (2003), extend DCP to ad- by shedding light on the value creation process
dress e-commerce as a dynamic capability that that IT initiatives have on businesses and on how
represents the level of integration with customers that value could be measured. Mukhopadhyay,
and suppliers and investigated how businesses Kekre, & Kalathur (1995) managed to quantify
leverage their investment to create unique Internet- financial gains arising from improved informa-
enabled resources. They developed a framework tion exchanges as a result of using EDI over a
for assessing the value of e-commerce to business decade. Tallon & Kraemer (2002) introduced
performance grounded in resource-based theory. a process oriented model of IT business value.
It was to provide a mechanism for representing
Process Oriented Approach goals for IT in the sense that different business
strategies emphasize different parts of the value
Unlike the variance approach that explains the chain within a corporation. They concluded that
variability of a dependent variable based on its cor- executives’ perspectives are an important factor
relation with one or more independent variables, in evaluating payoffs from IT investment. Also
the process oriented approach attempts to explain their IT management practices contribute to better
the occurrence of an outcome by identifying the IT investment decisions.
sequence of events preceding it. Venkatraman & In the context of online business, Barua et al.,
Ramanujam (1986) provide a framework classify- (2001) developed and empirically tested a model
ing performance measures as either financial or concluding that the achievement operational
operational. Financial performance is at the core excellence will lead to improved financial per-
of the organisational effectiveness domain. Such formance. The central idea is that owners need to
performance measures are considered necessary, understand what drives operational performance
but not sufficient to define overall effectiveness and then to commit the necessary resources to the
(Chakravarthy, 1986). Beyond this core, lie opera- development of the drivers (Barua et al., 2001).
258
The operational excellence and the subsequent 1. From whose perspective is effectiveness is
financial gains in the process oriented approach being judged?
are driven by the investment in IT.
The stakeholder perspective of measurement
is an important issue in the evaluation process.
A THEORETICAL DOMAIN OF Seddon, et al., (1999) argued that success should
WEBSITE PERFORMANCE be investigated from whose perspective. Other
researchers also called for stakeholder consider-
DeLone & McLean (1992) synthesised commu- ation in the evaluation process and pointed out
nication theory and previous empirical research. that whose perspective is considered critical in
They developed a model known as DeLone and the determination of success (Belanger et al.,
McLean Information System (D&M IS) Success 2006; Mich, Franch, & Gaio, 2003). Indeed, the
Model to identify dimensions and measures for evaluation perspective reflects the stakeholder
information systems success based on the user experience point of view. For example, users hold
perspective. “The creation of the D&M IS Success a central stake when user satisfaction is under
Model was driven by process understanding of consideration (Stockdale & Lin, 2008); but website
IS and their impacts. This process model has just statistics may have greater influence on owner’s
three components: the creation of the system, the satisfaction with the website. In both cases, users
use of the system, and the consequences of this and owners are both stakeholders in the website.
system use. Each of these steps is necessary, but not Previous work however, evaluated websites
sufficient, condition for the resultant outcome(s)” based on one of three perspectives: users, owners,
(DeLone & McLean 2003, p16). and designers. From the perspective of the users,
Similar to their approach, the current study the website is considered to be successful when
has built on the DeLone & McLean (1992) high their expectations are met and their interaction with
level model. This, however, warrants further the website is a positive experience (Schaupp, Fan,
discussion. First, since the model is based on the & Belanger, 2006). The quality of users’ experi-
user experience of the website (user perspective), ences and predominantly users’ satisfaction with
some changes are necessary in order to make it the website, have been used in recent research
relevant to the owner perspective. Second, with the initiatives as determinants of success (Aladwani
website is a special type of IS, where the primary & Palvia, 2002; Loiacono, Watson, & Goodhue,
system users are visitors to the website. Visitors 2002; Ranganathan & Ganapathy, 2002).
interact with the website for a purpose and their From the designer perspective, according to
interaction is paying off the organisation. Molla Turban & Gehrke (2000) success is a measure of
& Licker (2001) and DeLone & McLean (2004) user experience in relation to the technical aspect
extended the D&M IS original model to measure of the site.
e-commerce success. Thirdly, Seddon, Staples, From the owner perspective, success is mea-
Patnayakuni, & Bowtel (1999) recommended sured by the website’s ability to attract qualified
that anyone seeking to evaluate an IT application users who will aid the business to achieve its
should have a clear answer to each of the follow- stated goal (Belanger, et al., 2006; Schaupp, et
ing seven questions that have been suggested by al., 2006). Ways to make inferences regarding
Cameron & Whetten (1983) on organisational website effectiveness include: analysing the click
performance measurement: stream data from the website traffic (Alpar, 2001;
Belanger, et al., 2006; Quaddus & Achjari, 2005;
Schaupp, et al., 2006), financial returns (Auger,
259
2005; Barua, et al., 2001; Zhu, 2004), and owner The level of analysis is dependent on the
satisfaction (Huizingh, Krawczyk, Bijmolt, & domain being focused on. If the domain is the
Hoekstra, 2007; E. Huizingh, 2002). organisation as a whole then the unit of analysis is
the organisation. Or, if the domain is the websites
2. What is the domain of activity? then the unit of analysis will be the website in the
organisation and so on.
Since it is a process model and as asserted by
DeLone & McLean (2003 p16) the application of 4. What is the purpose of the evaluation?
a process model to empirical research requires a
contextual variance specification of the model. In Identifying the purpose for judging effective-
contextualising the high level model mentioned ness will help determine the appropriate constitu-
above to identify measures for website success, a encies, domain and level of analysis.
website is first created and made available on the
Web. Next, online users voluntarily visit the site. 5. What is the time frame employed?
Their experience with various features contained
in the website will either satisfy or dissatisfy them The outcomes may occur in a short time, or
according to the system, information and service incrementally over long period of time. Hence, the
quality exhibited to them, which will impact time frame whether it is short-term or long-term
the conduct of their work (DeLone & McLean, should be made explicit. Long-term effectiveness
2003). This user perspective is not the focus of may not be compatible with short-term effective-
the present research. ness, and sometimes effect cannot be detected
From the owner’s perspective (which is the using the wrong time frame.
focus of the present study), businesses invest in
their website by creating a value to be exchanged 6. What types of data are to be used?
with its visitors. Investors (businesses) realise pay-
offs for their investment if they see their website Data being used to measure success should
is creating awareness, generating traffic or driv- be identified whether it is objective, perceptual
ing sales. Therefore the website is created with a or empirical and so on.
set of features to attract users for the purpose of
exchanging value. Online users voluntarily visit 7. Against which referent is effectiveness being
the site and the business responds to their queries judged?
and communicates a set of features with quality
sound for a positive user experience (Jensen, User satisfaction is the significant success
2003). Consequently, this will not only impact on measure when user experience with the website
the business but also determine what metrics need is being measured. If the owner perspective is
to be used to understand users’ behavior in the site. taken, the owner satisfaction is the important
Both user interaction with the site and benefits variable(s) to measure.
accrued to the business determine owner’s satis- Applying those recommendations, utilising
faction with the site. Their satisfaction enhances the DCP in the RBV and the process oriented
the system by either reinvesting or understanding approach as theoretical underpinnings, creates a
the payoff from their initial expenditure. theoretical structure as shown in Figure 1 which
presents the high-level research model that guides
3. What is the level of analysis? the general inquiry of the current study.
260
Figure 1. Initial research model based on RBV and process oriented approach
As stated by DeLone & McLean (2004), the 2002). When a business decides to be online, its
selection of e-commerce performance dimensions presence needs to be appropriate to the needs of
and measures should be contingent on the objec- the business and should focus on supporting its
tives and the context of the empirical investigation. business goals. Websites for e-commerce need
The objective of the present research is to con- to include functionalities to share information,
tribute to the theoretical and practical understand- facilitate transaction, and improve customer ser-
ing of how to develop a means of evaluating vice (Zhu & Kraemer 2003). Zott & Amit (2000)
websites based on owners’ perceptions of their identified two strategies for value creation in e-
websites’ performance. commerce – the efficiency that e-commerce busi-
In the subsections that follow, an extensive ness models exhibit and the degree to which they
review of literature was conducted for each of create “stickiness” (stickiness is the effectiveness
the three dimensions to identify their underlying of the website in holding the visitor’s attention).
factors. Creating a website of value may be achieved by a
combination of what a website offers and how it
Website Investment offers it in order to effectively meet expectations
of both users and the owner of the site (Hong,
The creation of a website represents an investment 2007). Website offer (what and how), however,
for the organisation. According to Zhu & Kraemer from the users’ perspective refers to their experi-
(2003), businesses invest in their websites to sup- ence with the system (DeLone & McLean 2004;
port their strategic initiative to use the web to deal Molla & Licker 2001), the effectiveness of which
with customers. A website for e-commerce is a is represented by users’ satisfaction. By contrast,
transactional one as it sells its products/services from the owner perspective such constructs are
online. In general, selling involves creating value, intended to communicate an organisational image
the driver of which determines a competitive and product branding, to inform visitors to the site,
advantage. A website serves as a communication to support access to information and knowledge,
channel for bidirectional information transfer and to support sales and to enhance customer service
communication, a platform for transacting, an (Fitzpatrick, 2000). The effectiveness of these may
interface for providing customer service, and a be captured by analysing clickstream flowing to
facilitation for marketing initiatives. Creating a the site (Schaupp et al 2006) or financial returns
website, according to Thelwal (2001), is a busi- (Zhu 2004; Auger 2005) or owner satisfaction
ness task rather than a technical issue. The website (Huizingh et al 2007).
is therefore created with a web unique selling The current research refers to what the website
proposition appealing to the target group(s) by offers (site offer) as its features. These are business
assigning a mix of these features (informational, functions related to informational, transactional,
transactional, service, and promotional) in order service, and promotional capabilities, of a website
to create awareness, generate traffic, and drive (Kim & Lee, 2002; Pujani & Xu, 2005; Sellitto,
sales (Turban & Gehrke 2000; Constantinides Wenn, & Burgess, 2003). Once the website has
261
been fully implemented, owners should be inter- enhance the users’ knowledge. According to Song
ested in determining if their site meets its intended & Zahedi (2001), it should facilitate resources
goal and consequently if it is performing to their to users seeking more information, customise a
expectation (Belanger et al 2006; Schaupp et al response according to their needs, and provide
2006). Owners perceive their success in part in payment and receiving options. These features
the value their website creates for their custom- exhibited in the website are the messages com-
ers. Online businesses are now able process large municated to the users. How precisely the user
amounts of information that can then be available receives these intended messages represents the
in the form of assistance for consumers’ purchases semantic criteria in the communication theory of
of goods, as well as information services. Online Shannon & Weaver, (1949) as identified by Mason,
technologies, at least conceptually, improve a (1978). However, from the owner’s perspective
business’s immediacy and responsiveness which these messages represent goals to be achieved and
strengthen customer relationships while broaden- the performance of these goals to the business’s
ing its coverage in the marketplace (Kauffman expectation defines the success measure of the
& Walden, 2001; Ngai & Wat, 2002). Treacy & semantic criteria. According to the resource-
Wiersema (1997) have suggested that e-commerce based view the semantic criteria is how a business
transforms businesses into a customer intimacy weaves website goals (functionalities) to create
discipline delivering not what the market wants, capabilities that contributes to a business’s overall
but what a specific customer wants. The customer- e-commerce effectiveness (Zhu, 2004). Analys-
intimate company makes it a business of knowing ing the effectiveness of these goals/capabilities
the people it sells to and the product and services provides some insight into the business’s online
they need. The online user, however, cannot yet offering. The business, however, needs to deal
interact directly with the product. This requires with whether the aforementioned feature goals
a substitute to enhance the product knowledge. and capabilities (informational, transactional,
The provision of services on the website replaces customer service and promotional) appeals to
the sales person in the traditional business. Us- visitors. If it is not optimal, an adjustment may be
ers, therefore, need not only to know who they needed, such as adding or deleting some of their
are dealing with, but also need help with product online offering elements. However, poor sales
selection and all the information they would have may result from other problems which deal with
if they were shopping physically (Lohse & Spiller, the online offering’s effectiveness. Complete un-
1998). Service quality is one of the key factors in derstanding requires looking at other areas which
determining the success or failure of e-commerce. increase users’ experiences that could influence
A high standard of the service quality is the means purchase intention. According to the communi-
by which the potential benefits of e-commerce are cation theory of Shannon & Weaver, (1949) as
realised (Yang, 2001). identified by Mason, (1978), this is achieved by
Site visits and sales are generated by promo- the technical criteria “how accurately the message
tion (Lohse & Spiller 1998). Promoting products is received by the recipient”. The technical aspect
or services within the website is another aspect of the website is covered in the “how” element
that can be communicated to users. A website of the website and is referred to as the website
that features price-based promotions (discounts, quality. It is defined as the effort of the business to
special offers and rebates) and non-price promo- make their website visible, usable and maintained
tions (what’s new) attracts visitors’ attention (Fitzpatrick, 2000). According to the resource-
and increases their intention to transact (Song & based view this is how businesses leverage their
Zahedi, 2001). Moreover, a website is intended to
262
investment in their capabilities to create unique To this end, it can be concluded that the ability
online resources (Zhu & Kraemer, 2003). of the website to communicate a value and differ-
The issue of visibility according to Thewall entiate its offer (proposition) from its competitor
(2001) “is one that is easy for the inexperienced is viewed as one of the predictors of website’s
to ignore. An otherwise excellent website may effectiveness. The website proposition (offering)
be completely ignored because few potential consists of the various website business features
customers ever find it”. Awareness of the web- (capabilities) along with the effort of the business
site according to Hoffman, Novak, & Chatterjee, in making their website functional and up-to-date.
(1995) generates visits and repeat visits leading
to a successful website. “Visibility is defined Website Usage
as the extent to which a user is likely to come
across a reference to a company’s Web site in his According to the process oriented approach the
or her online or offline environment” (Drèze & impact of IT investment occurs through an inter-
Zufryden 2004, p22). Visitors can be directed to mediate stage (Barua et al., 1995). Information
a website through offline advertising (Aldridge, systems (IS) researchers have demonstrated that
Forcht, & Pierson, 1997; Drèze & Zufryden, usage is a key variable in explaining the perfor-
2004; Hodges, 1997), by following a link from mance impact of information technology. Seddon
another site (Aldridge, et al., 1997; Dholakia & et al., (1999) pointed out that system use is a good
Rego, 1998; Drèze & Zufryden, 2004; Fitzpatrick, proxy for IS success when the use is not manda-
2000; Hodges, 1997; Turban & Gehrke, 2000), tory. DeLone & McLean, (2003) posited that IS
or by modern social interaction media such as quality affects subsequent “use” which will in turn
Facebook, Twitter, YouTube etc. determine the benefits that accrue to the business.
The other effort needed from the business is In e-commerce, website users are customers;
to make their website customisable to visitors their use is most often voluntary. The nature of
(Chakraborty, Lala, & Warren, 2002; Lii, Hyung, & the system’s use and the amount of the usage are
Tseng, 2004; Zhu, 2004; Zhu & Kraemer, 2003), ef- both important indicators of effectiveness. This
ficient navigation (Elliot, et al., 2000; Fitzpatrick, will not only impact the business but also will
2000; Gonza’lez & Palacios, 2004; S. Kim, Shaw, assist the business in improving the quality of its
& Schneider, 2003; Lii, et al., 2004; McCarthy & website (DeLone & McLean, 2003). Therefore,
Aronson, 2000; Nielsen, 1999), which is the flow user traffic measures should be determined with
of the site due to its professional design. Users need reference to the number of new or repeated visitors,
to be comfortable to move around the site easily. the number of conversion rates and the pattern
Every page must have consistent navigation links of their navigation (DeLone & McLean, 2004).
to move around, and safety in terms of security A common method of measuring website us-
and privacy (Barua, et al., 2001; Belanger, Hiller, age is by conducting market research and asking
& Smith, 2002; Chakraborty, et al., 2002; Elliot, users of their experience with the website. Such
et al., 2000; Lii, et al., 2004; Turner, Zavod, & an approach is often costly and time consuming
Yurcik, 2001). Another issue of “how” is to keep (Myra. Spiliopoulou & Pohle, 2001; Weischedel
the site maintained and up-to-date. Visitors can & Huizingh, 2006). Alternatively, data can be
easily leave the website if they find no maintenance automatically collected about visits to the site
(Dholakia & Rego, 1998; Fisher, C. Annemieke, which allow owners to aggregate data and possibly
& Bentley, 2007; Lii et al., 2004; Nielsen, 1995; evaluate their website effectiveness (Schonberg,
Thelwall, 2001). Cofino, Hoch, Podlaseck, & Spraragen, 2000;
Myra. Spiliopoulou & Pohle, 2001). Online tech-
263
nology is able to collect large amounts of detailed the cause of the problem and ultimately seek a
data on visitor traffic and activities on websites. remedy (for example, search engine optimisation).
Such data offer a plethora of metrics to which
businesses must carefully choose measures for Website Payoff
different purposes (Phippen et al., 2004).
For the present study, usage is captured by the While website usage represents the success at the
different metrics available to owners who utilise site level (operational), payoffs represent suc-
clickstream data. These metrics are critical to as- cess at both individual and organisational level
sess website activities and user behaviour. From (DeLone & McLean 2003). DeLone & McLean
the owner perspective, such metrics may suggest (2003) replaced both individual and organisational
where improvements can be made with regard to impacts in their original model with the “net
design, layout, and navigation issues (Schonberg benefit” construct for the sake of parsimony. It
et al., 2000). Despite the limitations of clickstream is the net benefit construct that sets the level of
data (Weischedel & Huizingh, 2006), detailed analysis and captures the impact of e-commerce
and concrete data on users’ behaviour can be to the stakeholder. Net benefit is determined by
collected to indicate trends rather than provide context and objectives for e-commerce invest-
definitive data/statistics on website usage. Indeed, ment (DeLone & McLean 2004). While there are
a reasonable measure could be determined by as- other forms of benefits the financial returns are
sessing whether the full functionality of a website of special interest. Giaglis et al. (1999) observed
is being used for its intended purposes (Welling that the most common methods of evaluating
& White, 2006). information technology investments is by way of
The traffic on a website can be measured by a established accounting techniques, such as Return
number of metrics; among these is traffic volume on Investment (ROI).
flowing to the site. Traffic remains a valid sur- Huizingh (2007) defines e-commerce benefits
rogate measure of effectiveness as without traffic in terms of owner satisfaction – a proxy measure
no revenues could be generated. Even with heavy of the financial payoff to justify e-commerce
traffic, there may be no sales. Achieving high traf- expenditure. Hong (2007) defines satisfaction as
fic volumes is still a prerequisite for higher level conforming to individual’s expectations. In their
goals in most websites (Alpar, 2001; Kim & Lee, study, Zhang and von Dran (2000) used Herzberg’s
2002). Other types of measures include page hits, hygiene-motivator theory and propose a model of
page views, unique visitors, and viewing time two factors to identify essential design features
(Alpar, 2001; Quaddus & Achjari, 2005). Quad- (hygiene) and other features that increase users’
dus & Achjari, (2005) used page view, stickiness, satisfaction (motivators). It is preferred to separate
conversion rate and the extent of the contribution owner satisfaction from website financial returns
of e-commerce to meet the business goals for their on the grounds that effectiveness is derived from
definition of website effectiveness. The metrics both financial and non-financial criteria (Dess
available to owners can still be misleading and & Robinson, 1984). Owners determine their IT
their interpretation needs to be accurate in order expenditure according to their satisfaction but
to be effective. Serving as an indicative measure, not according to their financial return from the
they can help to identify weaknesses that need to website. Financial benefits and owner satisfaction
be considered. Upon discovering a problematic are not substitutes for one another.
situation (e.g. the number of visitors is below When the website drives traffic, communicates
expectation), a business could further investigate certain features that enhance users’ experience,
generates trust and strengthens the competitive
264
position of the business, then owners are inclined oriented approach, these metrics are the measures
to be satisfied as they feel that their web presence that lead to the website payoffs.
is paying off. The extent to which this has been The payoff concept was divided into two
realised is either in monetary terms (sales increased constructs: financial returns and owner satisfac-
or cost reduction) or in the form of intangible tion, which includes the non-financial measures
benefits (such as strengthening competitive posi- of the website. Since the owner perspective is
tion) (Huizingh 2002). The main reason businesses adopted, the ultimate dependent variable is the
establish a presence on the web is to capture owner satisfaction as a contextual variance of the
these tangible and intangible benefits (Kim et al., model. Changes in website performance occur
2003). It is concluded that owner satisfaction and along the resulting five factors: features, quality,
financial returns are two different concepts when usage, financial returns, and owner satisfaction.
measuring the payoffs of the website. The resulting model shown in Figure 2 represents
the taxonomy and typology for evaluation. Fig-
ure 2 shows that these five factors are related in
A FRAMEWORK OF SUCCESS a certain manner. Website offer (investment in
features and quality) is related to owner satisfac-
Drawing upon D&M IS Success Model, applying tion (ultimate payoff) through website usage and
the performance measure of Cameron & Whetten the financial realization.
(1983) as suggested by Seddon et al., (1999), a The website is evaluated against the effective-
high level framework of three constructs can be ness of each of the five constructs. While the
established. These constructs are website invest- effectiveness of each of the construct is largely
ment, website usage, and website payoffs. In episodic, the overall effectiveness of the website
setting up the website, the business has the task should be the aggregate effectiveness of the three
of creating a web unique selling proposition ap- dimensions. In other words, the three associated
pealing to the target group(s). The conceptual dimensions (five constructs) may be governed by
model for this (website investment) is built upon a common latent phenomenon i.e. website ef-
the theory of communication (the web as a com- fectiveness (success) as illustrated in Figure 3.
munication channel) and the resource based view The structure posits a second order factor govern-
approach (web capabilities are resources in which ing the correlations among website offer, usage,
a business can create a value through their invest- and payoff explaining the covariance between
ment which can be used to sustain a competitive them. The theoretical interpretation of this
advantage). The communication theory is used higher order factor is an overall trait of website
to explain the value (message) proposed to us- success. The efficacy of the model is yet to be
ers. The resource-based view is used to link the assessed by empirical data.
investment in creating the value to the payoffs.
Website features (informational, transactional,
service, and promotional capabilities) and quality DISCUSSION
(visibility, usability, and maintainability) were
identified as the messages and resources of the This study centers on a gap in the literature and
website respectively. the need for an evaluative approach from the
Usage refers to the information about users as owner’s perspective. In particular, the purpose
it is available to the business through metrics that of this study is to provide a guiding principle for
report website activities. According to the process evaluating a website. The present study presents
the development of an owner evaluation based
265
Figure 2. A framework of evaluation
Figure 3. A framework of success
framework for evaluating a websites’performance. Despite the importance, as well as the need for
A framework (taxonomy) for evaluation was such a framework, there is no practical evaluation
proposed. It complements existing frameworks framework that considers the owner’s experience
in that it evaluates a different aspect of website when evaluating websites. The previous evalua-
performance, that of evaluation from the owner’s tion approaches by others are based on the users’
perspective. This evaluation framework, however, experience and do not address explicitly and
differs in terms of generality, specificity, theory directly owner concerns. In order to bridge this
underpinning and provides explicit evaluation gap, two research objectives are posed:
criteria for a website as a business task.
266
1. What are the appropriate dimensions and rised, success is a higher-order phenomenon that
measure for evaluating the performance of is evidenced through high performance across
a website based on the owner point of view? multiple dimensions.
2. Can the dimensions and measures be Since there appears to be no theory that ad-
combined to yield to a testable model that equately discusses the selection of variables for
determines website effectiveness? measuring website success, the identification of
measures and criteria to evaluate websites are one
The first objective is to determine what the of the biggest challenges and a key contribution
appropriate concepts are for measuring the perfor- of the present study. In many instances, owners
mance of a website based on the owner point of have no structured set of criteria upon which to
view. The present study attempts to achieve this gauge the activities associated with their website.
goal by forming a base model that captures the Such identification therefore is much needed for
changes in the website performance. The model is accurately assessing the value and performance
based upon the following: the idea of DeLone & of websites for management.
Mclean (1992) information system success model Another key contribution brought by this study
(that success is a multidimensional and process is that it relies on business owner responses rather
concept), applying Cameron & Whetten (1983) than those of consumers. Websites are becoming
performance measures as suggested by Seddon et the business channels for many owners, investment
al (1999), using the resource based view (RBV) rival payoffs. In practice website investments are
and the process model approach. Three broad di- too seldom systematically evaluated often for the
mensions of performance were deemed adequate lack of tools. From evaluation perspective such
in providing a structure for website evaluation. a framework proposed by this study enables
These dimensions are termed investment, usage, businesses to monitor the performance of their
and payoff. An extensive review of the literature website and provides only a guiding principle of
was conducted to identify the underlying factors evaluation. Businesses can still use the framework
of each dimension and its measurement items. to develop relevant business strategies and tactics
Five factors were identified; these are website and set clear policies along the guiding principle
feature, website quality, website usage, financial provided by this study, as follows:
returns, and owner satisfaction. If owners are not satisfied and their perceived
The second objective was to investigate the success on website offer is less than expected, this
relationships amongst the identified dimensions is, of course, cause for concern; the value driver
and propose a model of website evaluation. Nine then has to be looked at closely. When owners
links were identified leading to an a-priori model. are not satisfied but the perceived success on
The resulting model has a solid theoretical basis their website offer is at or more than expected,
and provides a framework that relates the five this suggests potential to be exploited and more
constructs with each other. The relation between advantage can be gained. If owners are satisfied
the three constructs (dimensions) has also resulted and perceived success of the website offer is less
in another interesting model, the website success than expected, this will be for a short term and
model. While the model of evaluation provides more investment is needed if future gains are to
a means to evaluate the website across the three be realised. Finally, if owners are satisfied and
constructs, the change of the overall website per- their perceived success on their offering is at or
formance is due to the change in any one of them. more than expectation, then the ultimate goal is
In other words the three constructs contribute to achieved and the objectives are to maintain such
the overall performance of the website. As theo- a position to continue reaping benefits from being
267
Figure 4. Applying the success model
online. Figure 4 provides a graphical view of the 3. Payoff accrued to the business due to the
above scenario. use of its website as realised by the owner
in terms of the website financial returns and
the owner’s satisfaction.
CONCLUSION
Along these three dimensions a website
Website evaluation has become a critical process evaluation can be conducted. Also, these three
in the online business performance measurement dimensions explain the overall performance of the
effort. However, website evaluation research to website, yielding a website success model. Each
date has been focused on the user point of view dimension is a business process of the website
leading to what to include in the website. An owner that its owner needs to monitor.
perception based on a set of evaluation criteria To date, there appears to have been no owner-
is required for management to justify its invest- based evaluation framework to assess websites in
ment. This research was aimed at developing an the e-commerce environment. In this study, five
evaluation framework to address these manage- evaluation factors were presented. While owners
rial concerns. Although many researchers have should find these factors useful to evaluate their
suggested that a variety of factors could be used website, academics interested in performing
to evaluate website performance, few systematic research relative to the evaluation or success of
studies have yet addressed the specific topic of fac- websites can use these factors as the basis for
tors which relate investment to payoffs. This study their research.
has developed a framework of three dimensions
that can be used to evaluate websites. These are:
REFERENCES
1. Website offer factor capturing the value
proposed to the customer. This is identified Al-Qirim, N. (2003). Electronic commerce in
in terms of website features and website small to medium-sized enterprises: Frameworks,
quality. Issues and implications. Hershey, PA: Idea Group
2. Usage (Visitors’ information captured by Publishing. doi:10.4018/978-1-59140-146-9
the website measured by Web analytics).
268
Aladwani, A., & Palvia, P. (2002). Develop- Barua, Kriebel, C., & Mukhopadhyay, T. (1995).
ing and validating an instrument for measuring Information Technologies and business value: An
user-perceived Web quality. Information & Man- analytic and empirical investigation. Informa-
agement, 39(6), 467–476. doi:10.1016/S0378- tion Systems Research, 6(1), 3–23. doi:10.1287/
7206(01)00113-6 isre.6.1.3
Aldridge, A., Forcht, K., & Pierson, J. (1997). Barua, Konana, P., Whinston, A., & Fang, Y.
Get linked or get lost: Marketing strategy for (2001). Driving e-business excellence. MIT Sloan
the Internet. Internet Research: Electronic Net- Management Review, 43(1), 36-44.
Belanger, F., Fan, W., Schaupp, L. C., Krishen,
doi:10.1108/10662249710171805
A., Everhart, J., & Poteet, D. (2006). Website
Ally, M., Cater-Steel, A., & Toleman, M. (2007, success metrics: Addressing the duality of goals.
5-7 Dec 2007). A website sophistication model Communications of the ACM, 49(12), 114–116.
based on value-added technology solutions and doi:10.1145/1183236.1183256
services. Paper presented at the 18th Australasian
Belanger, F., Hiller, J. S., & Smith, W. J. (2002).
Conference on Information Systems (ACIS),
Trustworthiness in electronic commerce: The
Toowoomba, Australia.
role of privacy, security, and site attributes. The
Alpar, P. (2001). Measuring the efficiency of Journal of Strategic Information Systems, 11(3-4),
website traffic generation. International Journal 245–270. doi:10.1016/S0963-8687(02)00018-5
of Electronic Commerce, 6(1), 53–74.
Bharadwaj, A. S. (2000). A resource-based per-
Amit, R., & Zott, C. (2002). Value drivers of e- spective on Information Technology capability
commerce business models. In Hitt, M. A., Amit, and firm performance: An empirical investigation.
R., Lucier, C., & Nexon, R. (Eds.), Creating value: Management Information Systems Quarterly,
Winners in the new business environment (pp. 24(1), 169–196. doi:10.2307/3250983
15–47). Oxford, UK: Blackwell.
Cameron, K. S., & Whetten, D. A. (1983). Orga-
Auger, P. (2005). The impact of interactivity and nizational effectiveness: A comparison of multiple
design sophistication on the performance of com- models. New York, NY: Academic Press.
mercial websites for small businesses. Journal
Chakraborty, G., Lala, V., & Warren, D. (2002).
of Small Business Management, 43(2), 119–137.
An empirical investigation of antecedents of
doi:10.1111/j.1540-627x.2005.00129.x
B2B websites’ effectiveness. Journal of Internet
Barnes, S., & Vidgen, R. (2003). Measuring website Marketing, 16(4), 51–72. doi:10.1002/dir.10044
quality improvements: A case study of the forum
Chakravarthy, B. S. (1986). Measuring strategic
on strategic management knowledge exchange.
performance. Strategic Management Journal,
Industrial Management & Data Systems, 103(5),
7(September/October), 437–458. doi:10.1002/
297–309. doi:10.1108/02635570310477352
smj.4250070505
Barney, J. B. (1991). Firm resources and sustained
Clifton, B. (2010). Advanced Web metrics with
competitive advantage. Journal of Management,
Google analytics (2nd ed.). Indianapolis, IN:
17(1), 99–120. doi:10.1177/014920639101700108
Sybex.
269
Constantinides, E. (2002). The 4S Web-marketing Elliot, S. R. Moّrup-Petersen, A. S., & Bjoّrn-

mix model. Electronic Commerce Research and Andersen, N. (2000, 19-21, 2000). Towards a
Applications, 1(1), 57–76. doi:10.1016/S1567- framework for evaluation of commercial websites.
4223(02)00006-6 Paper presented at the 13th International Bled
Electronic Commerce Conference, Bled, Slovania.
Deans, K. R., & Adam, S. (2000). Internet survey
data collection: The case of WebQUAL. Paper Epstein, M. J. (2004). Implementing e-commerce
presented at the ANZMAC Conference, University strategies: A guide to corporate success after
of New South Wales, Sydney. the dot.com bust (1st ed.). Westport, CT: Praeger
Publishers.
DeLone, & McLean, E. (2003). The DeLone and
McLean Model of Information Systems success: Fisher, J., Annemieke, C., & Bentley, J. (2007).
A ten-year update. Journal of Management Infor- Moving from a Web presence to e-commerce:
mation Systems, 19(4), 9–30. The importance of a business—Web strategy for
small-business owners. Electronic Markets, 17(4),
DeLone, & McLean, E. (2004). Measuring e-
253–262. doi:10.1080/10196780701635864
commerce success: Applying the DeLone &
McLean Information Systems success model. Fitzpatrick, R. (2000). Additional quality factors
International Journal of Electronic Commerce, for the World Wide Web. Paper presented at the
9(1), 31–47. The Second World Congress for Software Quality
Yokohama, Japan.
DeLone, W., & McLean, E. (1992). Information
Systems success: The quest for the dependent Giaglis, G., Paul, R., & Doukidis, G. (1999).
variable. Information Systems Research, 3(1), Dynamic modeling to assess the business value
60–95. doi:10.1287/isre.3.1.60 of electronic commerce. International Journal of
Electronic Commerce, 3(3), 35–52.
Dess, G., & Robinson, R. (1984). Measuring orga-
nizational performance in the absence of objective Gonza’lez, M., & Palacios, B. (2004). Quantitative
measures: The case of the privately-held firm evaluation of commercial websites: An empirical
and conglomerate business unit. Strategic Man- study of Spanish firms. International Journal
agement Journal, 5(3), 265–273. doi:10.1002/ of Information Management, 24(4), 313–328.
smj.4250050306 doi:10.1016/j.ijinfomgt.2004.04.009
Dholakia, U., & Rego, L. (1998). What Hahn, J., & Kauffman, R. (2001). Evaluating web-
makes commercial Web pages popular? Euro- site performance in Internet-based selling from
pean Journal of Marketing, 32(7/8), 724–736. a business value perspective. Paper presented at
doi:10.1108/03090569810224119 the International Conference on Electronic Com-
merce, Vienna, Austria.
Drèze, X., & Zufryden, F. (2004). Measurement of
online visibility and its impact on Internet traffic. Hao, X., Duo-Lin, L., & Zhi-Jie, L. (2010). The
Journal of Interactive Marketing, 18(1), 20–37. research on e-commerce website success mode.
doi:10.1002/dir.10072
Hodges, M. (1997). Preventing culture clash on
the World Wide Web. Technology Review, 100(8),
18–19.
270
Hofer, C., & Sandberg, W. (1987). Improving Kim, & Lee. (2002). Critical design factors for
new venture performance: Some guidelines for successful e-commerce systems. Behaviour &
success. American Journal of Small Business, Information Technology, 21(3), 185-199.
12(Summer), 11–25.
Kohli, R., Sherer, S., & Baron, A. (2003).
Hoffman, D., Novak, T., & Chatterjee, P. (1995). Editorial—IT investment payoff in e-busi-
Commercial scenarios for the Web: Opportunities ness environments: Research issues. Infor-
and challenges. Journal of Computer-Mediated mation Systems Frontiers, 5(3), 239–247.
Communication, 1(3), 1–20. doi:10.1023/A:1025632525371
Hong, I. (2007). A survey of website success Krishnamurthy, S. (2003). E-commerce manage-
metrics used by Internet-dependent organizations ment: Text and cases. Mason, OH: Thomson/
in Korea. Internet Research, 17(3), 272–290. South-Western.
doi:10.1108/10662240710758920
Lii, Y., Hyung, J., & Tseng, D. (2004). The ef-
Huizingh, E. (2002). The antecedents of fects of Web operational factors on marketing
website performance. European Jour- performance. Journal of American Academy of
nal of Marketing, 36(11/12), 1225–1247. Business, 5(1/2), 486–494.
doi:10.1108/03090560210445155
Lin, C., Huang, Y., & Tseng, S. (2007). A study of
Huizingh, K. A., Bijmolt, T., & Hoekstra, J. (2007). planning and implementation stages in electronic
How important are transactional or informational commerce adoption and evaluation: The case of
functions for website success? Paper presented Australian SMEs. Contemporary Management
at the ANZMAC 2007, Dunedin, New Zealand. Research, 3(1), 83–100.
Jensen, J. (2003). Issues facing SMEs in their Lohse, G., & Spiller, P. (1998). Electronic Shop-
adoption of electronic commerce. Retrieved Au- ping. Communications of the ACM, 41(7), 81–88.
gust 22, 2006, from http://www.ica.ogit.gov.au/ doi:10.1145/278476.278491
papers03/jensenpaper12final.pdf
Loiacono, E., Watson, R., & Goodhue, D. (2002).
Kaplan, R. (1983). Measuring manufacturing WebQual: A measure of website quality. In Evans,
performance: A new challenge for managerial K., & Scheer, L. (Eds.), Marketing theory and
accounting research. Accounting Review, 58(Oc- applications (Vol. 13, pp. 432–438). Chicago, IL:
tober), 686–705. American Marketing Association.
Kauffman, R., & Walden, E. (2001). Economics Mason, R. (1978). Measuring information output:
and electronic commerce: Survey and directions A communication systems approach. Information
for research. International Journal of Electronic & Management, 1(4), 219–234. doi:10.1016/0378-
Commerce, 5(4), 5–116. 7206(78)90028-9
Kent, M. L. (1998). Does your website attract or Mata, F., Fuerst, W., & Barney, J. (1995). Infor-
repel customers? Three tests of website effective- mation Technology and sustained competitive ad-
ness. Public Relations Quarterly, 43(4), 31–34. vantage: A resource-based analysis. Management
Kim, S., Shaw, T., & Schneider, H. (2003).
doi:10.2307/249630
Website design benchmarking within indus-
try groups. Internet Research, 13(1), 17–26.
doi:10.1108/10662240310458341
271
McCarthy, R., & Aronson, J. (2000). Competing in Palmer, W. (2002). Website usability, design,
the virtual world: E-commerce factors that affect and performance metrics. Information Sys-
the property-casualty insurance industry. Paper tems Research, 13(2), 151–167. doi:10.1287/
presented at the Sixth Americas Conference on isre.13.2.151.88
Information Systems.
Park, Y., & Gretzel, U. (2007). Success factors
Melville, N., Kenneth, K., & Vijay, G. (2004). for destination marketing websites: A qualita-
Review: Information Technology and organiza- tive meta-analysis. Journal of Travel Research,
tional performance: An integrative model of IT 46-63(1), 46.
business value. Management Information Systems
Petre, M., Minocha, S., & Roberts, D. (2006).
Quarterly, 28(2), 283–322.
Usability beyond the website: An empirically-
Mich, L., Franch, M., & Gaio, L. (2003). Evalu- grounded e-commerce evaluation instrument
ating and designing the quality of websites: The for the total customer experience. Behaviour
2QCV3Q metamodel. IEEE MultiMedia, 10(1), & Information Technology, 25(2), 189–203.
34–43. doi:10.1109/MMUL.2003.1167920 doi:10.1080/01449290500331198
Molla, A., & Licker, P. (2001). E-commerce system Phippen, A., Sheppard, L., & Furnell, S.
success: An attempt to extend and respecify the (2004). A practical evaluation of Web analyt-
DeLone and McLean model of success. Journal ics. Internet Research: Electronic Network-
of Electronic Commerce research, 2(4), 131-141. ing Applications and Policy, 14(4), 284–293.
doi:10.1108/10662240410555306
Mukhopadhyay, T., Kekre, S., & Kalathur, S.
(1995). Business value of Information Technol- Porter, M. (2001). Strategy and the Internet. Har-
ogy: A study of electronic data interchange. Man- vard Business Review, 79(3), 62–78.
Pujani, V., & Xu, J. (2005). E-commerce in Indo-
137–156. doi:10.2307/249685
nesian SMEs: Towards a research model of web-
Murphy, G. B., Trailer, J. W., & Hill, R. C. (1996). site success. Paper presented at the International
Measuring performance in entrepreneurship Conference on Services Systems and Services
research. Journal of Business Research, 36(1), Management (ICSSSM05).
15–23. doi:10.1016/0148-2963(95)00159-X
Quaddus, M., & Achjari, D. (2005). A model for
Ngai, E., & Wat, F. (2002). A literature review and electronic commerce success. Telecommunica-
classification of electronic commerce research. tions Policy, 29(2-3), 127–152. doi:10.1016/j.
Information & Management, 39(5), 415–429. telpol.2004.11.009
doi:10.1016/S0378-7206(01)00107-0
Nielsen, J. (1995). A home-page overhaul using criteria: Determining which criteria count most.
other web sites. IEEE Software, 12(3), 75–78. Electronic Commerce Research, 10(2), 191–208.
doi:10.1109/52.382190 doi:10.1007/s10660-010-9051-3
Nielsen, J. (1999). Differences between print Ranganathan, C., & Ganapathy, S. (2002). Key
design and Web design. Alertbox. Retrieved from dimensions of business-to-consumer websites.
http://useit.com/alertbox/990124.html (accecced Information & Management, 39(6), 457–465.
on January19, 2009). doi:10.1016/S0378-7206(01)00112-4
272
Schaupp, L., Fan, W., & Belanger, F. (2006). Spiliopoulou, M., & Pohle, C. (2001). Data min-
Determining success for different website goals. ing for measuring and improving the success of
Paper presented at the The 39th Annual Hawaii websites. Data Mining and Knowledge Discovery,
International Conference on System Sciences 5(1-2), 85–114. doi:10.1023/A:1009800113571
(HICSS’06) Track 6.
Spiliopoulou, M., Pohle, C., & Faulstich, L.
Schonberg, E., Cofino, T., Hoch, R., Podlaseck, (2000). Improving the effectiveness of a web-
M., & Spraragen, S. (2000). Measuring success. site with Web usage mining. In Masand, B., &
Communications of AIS, 43(8), 53–57. Spilliopoulou, M. (Eds.), WEBKDD, LNAI (pp.
142–162). Berlin, Germany: Springer-Verlag.
Schubert, P., & Selz, D. (2001). Measuring the ef-
fectiveness of e-commerce websites. In Barnes, S., Stockdale, R., & Lin, C. (2008). Evaluation of
& Hunt, B. (Eds.), E-commerce & v-business (pp. the effectiveness of small and medium sized
83–102). Oxford, UK: Butterworth-Heinemann. businesses websites in a business to business
context. In Calero, C., Muñoz, C., Moraga, A., &
Seddon, P., Staples, S., Patnayakuni, R., & Bowtell,
Piattini, M. (Eds.), Handbook of research on Web
M. (1999). Dimensions of Information Systems
Information Systems quality (Vol. 4, pp. 71–85).
success. Communications of AIS, 2(20), 1–61.
Hershey, PA: Information Science Reference.
Segars, A., & Grover, V. (1998). Strategic Informa- doi:10.4018/978-1-59904-847-5.ch004
tion Systems planning success: An investigation of
Stockdale, R., Lin, C., & Stoney, S. (2005). The
the construct and its measurement. Management
effectiveness of SME websites in a business to
business context. Paper presented at the The IADIS
doi:10.2307/249393
E-Commerce Conference.
Sellitto, C., Wenn, A., & Burgess, S. (2003). A
Straub, D., Hoffman, D., Weber, B., & Steinfield,
review of the websites of small Australian win-
C. (2002). Toward new metrics for Net-enhanced
eries: Motivations, goals and success. Informa-
organizations. Information Systems Research,
tion Technology Management, 4(2), 215–232.
13(3), 227. doi:10.1287/isre.13.3.227.80
doi:10.1023/A:1022954429432
Straub, D., & Klein, R. (2001). E-competitive
Shannon, C., & Weaver, W. (1949). The math-
transformations. Business Horizons, 44(3), 3–12.
ematical theory of communication. Urbana, IL:
doi:10.1016/S0007-6813(01)80029-X
University of Illinois Press.
Teece, D., Pisano, G., & Shuen, A. (1997). Dynam-
Sharma, A., Krishnan, R., & Grewal, D. (2001).
ic capabilities and strategic management. Strategic
Value creation in markets: A critical area of focus
ManagementJournal, 18(7),509–533.doi:10.1002/
for business-to-business markets. Industrial Mar-
(SICI)1097-0266(199708)18:7<509::AID-
keting Management, 30(4), 391–402. doi:10.1016/
SMJ882>3.0.CO;2-Z
S0019-8501(01)00153-5
Thelwall, M. (2001). Effective websites for small
Song, J., & Zahedi, F. (2001). Web design in
and medium-sized enterprises. Journal of Small
e-commerce: A theory and empirical analysis.
Business and Enterprise Development, 7(2),
Paper presented at the International Conference
149–159. doi:10.1108/EUM0000000006836
on Information Systems, New Orleans, LA.
273
Torkzadeh, G., & Dhillon, G. (2003). Measur- Yang, Z. (2001). Consumer perceptions of service
ing factors that influence the success of Internet quality in Internet-based electronic commerce.
commerce. Information Systems Research, 13(2), Paper presented at the The 30th EMAC Confer-
187–204. doi:10.1287/isre.13.2.187.87 ence, Bergen.
Treacy, M., & Wiersma, F. (1997). The discipline Zhang, P., & Von Dran, G. (2000). Satisfiers and
of market leaders: Choose your customers, nar- dissatisfiers: A two-factor model for website
row your focus, dominate your market. London, design and evaluation. Journal of the American
UK: HarperCollins. Society for Information Science American Soci-
ety for Information Science, 51(14), 1253–1268.
Turban, E., & Gehrke, D. (2000). Determinants of
doi:10.1002/1097-4571(2000)9999:9999<::AID-
e-commerce website. Human Systems Manage-
ASI1039>3.0.CO;2-O
ment, 19, 111–120.
Zhu, K. (2004). The complementarity of Informa-
Turner, C., Zavod, M., & Yurcik, W. (2001).
tion Technology infrastructure and e-commerce
Factors that affect the perception of security and
capability: A resource-based assessment of their
privacy of e-commerce websites. Paper presented
business value. Journal of Management Informa-
at the Fourth International Conference on Elec-
tion Systems, 21(1), 167–202.
tronic Commerce Research.
Zhu, K., & Kraemer, K. (2003). E-commerce
Udo, G., & Marquis, G. (2001). Factors affect-
metrics for Net-enhanced organizations: As-
ing e-commerce website effectiveness. Journal
sessing the value of e-commerce to firm perfor-
mance in the manufacturing sector. Information
Van der Merwe, R., & Bekker, J. (2003). A frame- Systems Research, 13(3), 275–295. doi:10.1287/
work and methodology for evaluating e-commerce isre.13.3.275.82
websites. Internet Research: Electronic Network-
Zott, C., Amit, R., & Donlevy, J. (2000). Strategies
ing. Applications and Policy, 13(5), 330–341.
for value creation in e-commerce: Best practice in
Venkatraman, N., & Ramanujam, V. (1986). Europe. European Management Journal, 18(5),
Measurement of business performance in strategy 463–475. doi:10.1016/S0263-2373(00)00036-0
research: A comparison of approaches. Academy
of Management Review, 11(4), 801–814.
Weischedel, B., & Huizingh, E. (2006). Website ADDITIONAL READING
optimization with Web metrics: A case study.
Paper presented at the 8th International Confer- Brown, I., & Jayakody, R. (2009). B2C e-com-
ence on Electronic Commerce, Fredericton, New merce success: A test and validation of a revised
Brunswick, Canada. conceptual model. The Electronic Journal of
Information Systems Evaluation, 12(2), 129–148.
Welling, R., & White, L. (2006). Website
performance measurement: Promise and real- Ghandour, A., Benwell, G., & Deans, K. (2011).
ity. Managing Service Quality, 16(6), 654–670. Measuring the performance of ecommerce web-
doi:10.1108/09604520610711954 sites–An owner’s perspective. Pacific Asia Jour-
nal of the Association for Information Systems,
Wernerfelt, B. (1984). A resource-based view of 3(1), Article 2.
the firm. Strategic Management Journal, 5(2),
171–180. doi:10.1002/smj.4250050207
274
Ghandour, A., Benwell, G. L., & Deans, K. R. Yusof, M., Papazafeiropoulou, A., Paul, R., &
(2010). Measuring e-commerce website success. Stergioulas, L. (2008). Investigating evaluation
Interdisciplinary Journal Of Contemporary Re- frameworks for health Information Systems. In-
search In Business (IJCRB), 1(12). ternational Journal of Medical Informatics, 77(6),
377–385. doi:10.1016/j.ijmedinf.2007.08.004
Ghandour, A., Deans, K., Benwell, G., & Pillai,
P. (2008). Measuring ecommerce website success.
Paper presented at the 19th Australasian Confer-
ence on Information Systems, Christchurch, New KEY TERMS AND DEFINITIONS
Zealand.
Features: Business functions related to infor-
Hinton, M., & Barnes, D. (2009). Discover- mational, transactional, service, and promotional
ing effective performance measurement for e- capabilities, which should all be contained in a
business. International Journal of Productivity website for e-commerce
and Performance Management, 58(4), 329–345. Financial Benefits: The return accrued to the
doi:10.1108/17410400910951008 business due to the use of website which can be
Ip, C., Law, R., & Lee, H. A. (2010). A review realised in terms of return on investment and other
of website evaluation studies in the tourism and measures that reflect performance in market, such
hospitality fields from 1996 to 2009. International as market share and market growth. It is one facet
Journal of Tourism Research, Early View (Online of website performance.
Version of Record published before inclusion in Owner Satisfaction: Owner satisfaction is
an issue). what conforms to owner’s expectation. When
the website drives traffic, communicates certain
Lagsten, J. (2011). Evaluating Information Sys- features that enhance users’ experience, generates
tems according to stakeholders: A pragmatic trust and strengthens the competitive position of
perspective and method. Electronic Journal the business, then owners are inclined to be satis-
Information Systems Evaluation, 14(1), 73–88. fied as they feel that their website is paying off.
Law, R., Qi, S., & Buhalis, D. (2010). Progress Payoff: The impact resulting from investment
in tourism management: A review of website which can be measured in terms of website finan-
evaluation in tourism research. Tourism Man- cial return and its owner satisfaction.
agement, 31(3), 297–313. doi:10.1016/j.tour- Perspective: From which the view is taken
man.2009.11.007 including the experience. For example, owner’s
perspective is related to the owners experience
Petter, S., & McLean, E. R. (2009). A meta-analytic with their own website.
assessment of the DeLone and McLean IS suc- Quality: Defined as the effort of the business to
cess model: An examination of IS success at the make their website visible, usable and maintained.
individual level. Information & Management, Usage: Refers to the information about users as
46(3), 159–166. doi:10.1016/j.im.2008.12.006 it is available to the business through metrics that
Stacie, P., William, D., & Ephraim, M. (2008). report website activities. It reflects how customers
Measuring Information Systems success: Models, are using the website.
dimensions, measures, and interrelationships. Website Offer: The mix of attributes that cre-
European Journal of Information Systems, 17(3), ate a Web unique selling proposition appealing to
236. doi:10.1057/ejis.2008.15 the target group(s). It is merely explaining what
the site has to offer in a more parsimonious way.
Offer is captured by website features and quality.
275
276
Chapter 14
Antecedents of Children’s
Satisfaction with Company
Websites and their Links
with Brand Awareness
Lucie Sermanet
IESEG School of Management Lille-Paris, France
Frank Goethals
IESEG School of Management (LEM-CNRS) Lille-Paris, France
Andrea Carugati
Aurélie Leclercq-Vandelannoitte
ABSTRACT
This chapter focuses on children’s satisfaction with the use of commercial websites. The authors address
two relevant gaps in extant literature: children as users of Information Technology and the concept of
Information System success itself. Children’s use of IT needs focus since this age group has been largely
neglected by extant IS research. The concept of success needs further research because the research
results on this dependent variable have been controversial, and there is still no full agreement on the
antecedents of success. In this chapter, the authors therefore contextualize the classic Information
Systems success model of DeLone and McLean to IS success with children. They carried out a survey of
a sample of 76 children aged 11 to 14, located in various cities in France. The results show that children
get higher brand awareness if they are more satisfied with the kids website of the brand. The chapter
finds the satisfaction with the site depends upon the personalization that is possible and the presence
of dynamic elements on the site. Characters and animations that help navigating the site are also ap-
preciated. A good presentation of the site’s rules could not be shown to be a significant antecedent of
satisfaction. Qualitative data that was gathered suggests that children are not really aware of the link
between symbols they see on the site and products that are sold.
DOI: 10.4018/978-1-4666-0170-3.ch014
Antecedents of Children’s Satisfaction with Company Websites and their Links with Brand Awareness
INTRODUCTION is still under debate. Despite the ongoing interest

in information system implementations and the
The research presented in this chapter focuses resulting focus on the assessment of information
on children’s use of online commercial websites. system success, the question of the evaluation of
With the development of the Internet, more and information systems remains a challenge. Faced
more people, from different backgrounds, age, and with the difficulty of obtaining objective measures
gender have access to online websites provided for the success of IS, user satisfaction appeared
by companies. Companies have to make sure the as a substitutive measure of IS success (DeLone
websites fit the evolution of customers’ needs. & McLean, 1992). The DeLone and McLean in-
Company websites allow people to search for formation systems success model has served for a
information, which helps them in their purchase long time as the basis for measuring the complex
decisions, or to directly buy products online. dependent variable in IS research and evaluating
Among the different age groups, children are a system’s effectiveness. Since its introduction in
among the fastest growing user groups and, as 1992, the model has been widely used, evaluated
technology becomes more and more user friendly, and extended. A growing number of studies have
they are increasingly impacted by technological measured the success of information systems in
evolutions. For instance, they access the Internet different organizations and contexts. Most of
to chat online, to play online games or to make these studies have attempted to analyse the role
school assignments. Some companies have un- of various factors that influence the information
derstood the importance of this market segment systems’effectiveness, or explored how to measure
as children influence the purchasing behaviour of information system success. The contributions
their parents) and have developed virtual gaming of these various researches carried out in the last
markets. In comparison to classic media such as decade have not only applied and validated the
television, online media have the ‘advantage’ there original model but have also challenged and pro-
are no restrictions on advertisements to children. posed enhancements to it. The updated DeLone and
As an example of a successful child website, we McLean IS success model (DeLone & McLean,
note that Disney has launched a specific website 2003), which results from these contributions,
(i.e. “club penguin”) allowing children to interact has refined the initial model and has consider-
with other children in a virtual world. Given the ably improved the measurement of IS success.
significance of children’s growing habits on the However, the application of this model by prolific
Internet and the potential business value of this IS literature has created little consensus on the as-
market segment, understanding the main deter- sessment of IS success. Additional research about
minants of children’s satisfaction with websites its main constructs and antecedents, as well as ap-
and evaluating their success is becoming an plications to the last technological evolutions and
interesting area of research. This research area users’ habits, along with continued research into
has been targeted in marketing research but it is organizational effectiveness and user satisfaction
still quite underdeveloped in information systems are needed. One way to get around this limitation
(IS) research. If we look beyond the effect of is by contextualizing the IS success model to a
children in influencing the buying behaviour of specific situation. In this way the model becomes
parents and we want to focus on the children’s less general but the validity for the specific target
actual satisfaction with the IT artefact, we find group becomes higher. Our study takes the offset
very scant literature. in children as user group and investigates the
Not only the specific user group of children has importance of two main constructs of this model,
been neglected but the concept of success of IS system quality and information quality, and their
277
Figure 1. The original model of DeLone and McLean (1992)
relation to children satisfaction. These have been quality, portability, integration and importance
recognized by IS researchers as key components (DeLone & McLean, 1992). As it can be evinced
of an information system’s success but rarely from these measures of success, the key assump-
they have been contextualized to user groups tion of this model is that the organization that
beyond working adults. In specific, the relevance implemented the system was also employing the
to children (age 11 to 14) is investigated, looking people that used the system and that the organiza-
at specific practices such as personalizability and tion would benefit not only from use but espe-
animations as attributes of information quality and cially from satisfaction and constant prolonged
system quality. The research model is applied to use.
children’s use of commercial websites, which has The limitation to the specific organizational
been scantly analyzed by IS research until now. context was highlighted by the same authors
(DeLone & McLean, 1992) that stated that
Literature Review researchers should systematically combine in-
dividual measures from IS success categories to
The most commonly used model to assess informa- create a comprehensive measurement instrument.
tion systems success is the model of DeLone and Researchers have argued that the DeLone and
McLean (1992). This model builds on the idea that McLean model should be contextualized in sev-
system quality and information quality will lead eral situations before it could be used as a global
to use and user satisfaction, which will impact model to assess the satisfaction with a website.
user performance and ultimately organizational In particular, previous research (e.g. Schaupp et
performance. The model is shown in Figure 1. al., 2009) has found that there are important dif-
The association between system use and indi- ferences in assessing a website success from the
vidual impact has been shown to be significant. individual user’s viewpoint or from the original
The measures used to determine system use are: organisation’s employee viewpoint.
frequency of use, time of use, number of ac- In the literature the issue of website success has
cesses, usage pattern, and dependency. Individu- been investigated quite broadly. Using the database
al impacts were measured in terms of job perfor- business source complete with the search string
mance and decision making performance. The “website AND success” and zooming in on peer
relation between system quality and individual review articles we found 55 hits. Among these hits
impact has also been found to be significant. The we find 39 dealing with website success in general
system quality has been measured in terms of ease and most articles cite or directly extend DeLone
of use, functionality, reliability, flexibility, data and McLean’s success model to the context of
278
Figure 2. Updated model of DeLone and McLean (2003)
websites. The major line of critique to the original the behavioral dynamics are not only happening
success model is in line with the critique on context online. For example, Razzouk et al. (2005) found
expressed above. This issue has been addressed that website success is highly linked to off line
bringing variations and additions to the original marketing. Rababah et al. (2011) recognize that
model. The most important changes with respect the evolution of websites and web services is so
to the original model concern the meaning of fast that a measure of success needs to be likewise
success. Since websites are targeted to individual dynamic. All in all, from the literature it can be
users, the individual impact and organisational evinced that, over time, the relation between
impact have been merged into one variable that online initiatives and success is becoming increas-
shows success of the website with individuals. ingly complex.
Literature (e.g. Molla & Licker, 2001) has used For our context of children use and success,
a variable called ‘net benefits’ (or variations on the literature is more sparse. Only few articles
the term) that represented the combination of use from our broad search have addressed directly
and user satisfaction. In an updated version of children as users. For example, Livingstone et
the success model, DeLone and McLean (2003) al. (2005) investigate the motivations behind
move away from the variable “organizational the use of online systems by a young population
impact” and adopt the same route by indicating (high school students). The research shows that
“net benefits” as dependent variable (see Figure interactive and creative uses of the Internet are
2 for the updated information systems success encouraged by the previous experiences in using
model of DeLone and McLean, 2003). the Internet. However this type of population
The literature is quite consistent regarding the and their behaviour is far from the ability and
definition and the measurement of success. In experience of elementary school kids. In order to
most cases, and in line with the model above find the antecedents of success we needed to find
(Figure 2), the independent variables are con- specific constructs that make sense to the specific
nected to technical characteristics of the website population through questions pertinent to them.
(speed, colours, etc.) which therefore are consid- In order to make the measure of the satis-
ered main drivers of success (Younghwa & Kozar, faction with the website the simplest possible
2006; Chuan-Chuan & Hsipeng 2000). In other we have grouped impacts into a single variable
cases, however, the relation is not so simple and called “satisfaction”. This is in line with other
279
research (e.g. on work satisfaction or satisfac- evaluate the success of children websites and to
tion with training) that state that satisfaction has understand the determinants of children’s satis-
to be measured directly and not through other faction with the websites. We focus on the two
constructs. User satisfaction has been more and dimensions information quality and system qual-
more regarded as a substitutive measure of the ity. More specifically, we considered four issues
success of information systems. The benefits and which appear as key elements in the analysis of
success of an IS are indeed extremely difficult to the success of up-to-date websites. Our research
quantify, insofar as they are in large part intangible model is shown in the Figure 3.
(Brynjolffson & Hitt, 1996). The success of IS is The system quality is defined in the literature
connected to the unfolding of processes inside as the degree to which the website is performing
the organizational structure and depends on the as planned. Measures of system quality typically
behavior of individuals (Zviran & Erlich, 2003). focus on performance characteristics of the system
An IS can be qualified as effective only when it under study. Some research has looked at resource
is regarded as such by the users. The investments utilization and investment utilization, hardware
in information technology will only produce an utilization efficiency, reliability, response time,
operational, financial or competitive value for the ease of terminal use, content of the database, ag-
firm if they transform themselves into systems gregation of details, human factors, and system
or applications that individuals perceive favor- accuracy. Hamilton and Chervany’s list of system
ably and use effectively (Rowe, 2002; Leclercq, quality measures (Hamilton & Chervany, 1981)
2007). User satisfaction thus appears as the most includes: data currency, response time, turnaround
pertinent criteria for the evaluation of an IS, as time, data accuracy, reliability, completeness,
guarantee of its success (Powers & Dickson, 1973; system flexibility and ease of use. More recently,
Galletta & Lederer, 1989). It explains why it is the Seddon (1997) considered system quality to be
most used measure of success of an IS (DeLone linked to reliability, user interface consistency,
& McLean). To extend the DeLone and McLean ease of use, documentation quality, and quality
(2003) model to the use that children make of and maintainability of the program code. We will
websites we need however to contextualize the focus especially on the ease of use construct,
two key antecedents of satisfaction: information which may deserve particular attention in a kid’s
quality and system quality. Furthermore, given context. One important element that may influence
the specific leisure characteristics of the websites, the ease of use, is the way navigation on the site
we want to investigate how trendy practices like is enabled. For example, navigation on a kids site
personalization and animations affect satisfaction can be enabled by having some spatial represen-
in a children population. It is, then, important to tation of places to visit in the virtual world, or via
have a good definition of these concepts. In the a more classic textual menu. Also, dynamic ele-
following section we will detail the research model ments such as animations or characters can give
and the hypothesis. First, we will turn to system ad-hoc advice on what one can do. These two
quality and then to information quality. elements, the appreciation of the navigation and
the dynamic nature of interactions will be in-
cluded in our model as system quality components.
RESEARCH MODEL In the updated DeLone and McLean IS success
AND HYPOTHESES model, information quality is measured in terms
of accuracy, timeliness, completeness, relevance,
We start from the updated DeLone and McLean and consistency (Etezadi-Amoli & Farhoomand,
IS Success Model (DeLone & McLean, 2003) to 1996; Seddon & Kiew, 1994; Teo & Wong, 1998;
280
Figure 3. The research model
Wixom & Watson, 2001). Applied to our research relationship between user satisfaction and brand
question, this dimension is defined as the measure awareness has also been assessed, in order to get
of the semantic success, and it is expressed through insight into the way a website affects the percep-
personalised content, complete and relevant infor- tion children have of the brand providing the
mation, ease of understanding of content, and ad- website. The user satisfaction with commercial
equation of the format. Another definition is given websites may indeed increase their brand aware-
by Wikipedia: Information quality is a measure of ness, which moreover appears as a key perception
the value, which the information provides to the that any company would use to measure their
user of that information. Personalizing content is brand strength. Two control variables are used.
thus one aspect of information quality that needs Prior research showed that gender plays a role
to be assessed and, therefore is included in our in information systems acceptance by adults.
model. The place where information is provided Our research needs to investigate whether this is
to users of the kids website is the ‘game regula- also the case in a younger population. The sec-
tions’ section. Therefore, we will include the ond control variable concerns the website under
child’s appreciation of the game presentation as consideration (as children in the sample had to
a second aspect of information quality that could evaluate two websites, see below).
lead to satisfaction with the website. The hypotheses in the model are then as fol-
Two constructs, the personalization of the lows:
web site and the game presentation, are related
to the ‘information quality’ construct. The way H1: Children will be more satisfied with the site
navigation is enabled and the level of dynamism if the site is perceived as offering sufficient
are related to the ‘system quality’ construct. The personalisation.
281
Figure 4. Screenshot of the Rik & Rok website
H2: Children will be more satisfied with the site the user group we needed to use websites in the
if the site offers a good presentation of the mother language of the children. The study was
game. carried out in France with French children and
H3: Children will be more satisfied with the therefore both websites are French. Both websites
site if the way navigation is enabled on the are oriented towards children and both websites are
site helps children more to find what is of dealing with the same type of product (food), this
interest to them. in order to avoid any bias linked to more attrac-
H4: Children will be more satisfied with the site tive markets (e.g. toys seem to be more attractive
if there are dynamic animations and charac- for children than food). This would also avoid a
ters that help them to better understand the bias of different target group oriented websites.
website. No purchases can be made on those websites. We
H5: The satisfaction with the site depends on the decided to take advantage of the chance to have
child’s gender. (all) children use two websites (instead of one) in
H6: The more children are satisfied with the site, order to get more variety in responses.
the higher is their increase in brand aware- Figure 4 shows an example of a question that
ness. appears on the Rik et Rok website. Rik et Rok is
the Auchan brand dedicated to children (Auchan
is one of the major supermarket chains in France).
RESEARCH SETTING AND By making some healthy product quiz they can
RESEARCH DESIGN also place the Logo of the brand (in the upper
left corner) and the brand also appears through
In our study, we have chosen to focus on children’s the characters that also appear on the package of
use of two websites: “Rik et Rok,” and “les petits the products in the supermarket.
repères”. Given the sensitivity and young age of
282
Getting data directly from children is very hard method design: a qualitative analysis to define
and was seldom done in prior research in this issues and concepts, a quantitative analysis to do
field. We dealt with this problem in a pragmatic the survey and test the hypotheses and at the end
way in the sampling method. We have chosen to a qualitative approach again to get more insight
gather data from school children in three cities into the findings.
situated in France where we could have access to We provided our potential respondents with the
children in the age group that we wanted to assess: website address and a guideline of the different
Orleans, Paris and Brest. In these cities we se- parts they had to assess. Children were not grouped
lected a mix of private and public schools where in a classroom. This last point is important because
the survey could take place. It is difficult to create we consider it important that children have no
a contact between researchers and children. contact with others when answering the question-
Therefore, we used two methods to contact our naire because they are easily influenced by others’
target group to get responses. First, we asked comments or believes. Moreover, we wanted to
teachers to ask their students to visit the two assess how children perceived the website at home
websites and to fill out the questionnaire. This (i.e. in their normal surfing environment), thus it is
method is particularly relevant because children important to provide the same environment during
may feel that the teacher is not a stranger and if the research as the environment in real situations.
he says that a questionnaire is safe, children and Data was collected through questionnaires that
their parents will believe it. Thus it made it easi- were distributed directly to children and answered
er to get children to answer our questionnaire. by the children alone, with no support from adults
This returned 57 replies. Then, in order to have in order to avoid bias. This questionnaire method
other participants, we used the snowballing tech- provides the respondent with anonymity. This is
nique. Children contacted their friends in order particularly relevant as we are dealing with chil-
to make them answer the questionnaire that was dren’s responses because most of the parents do
provided online. This returned 19 replies, given not want to let their children answer a survey if
a total of 76 replies. they have to provide their name by fear of loss of
Children had to answer the questionnaire by privacy or harassment. Such questionnaires also
ticking a box on a 7-point Likert scale to show avoid interviewer bias. Again, this is especially
they totally agree (=1) or totally disagree (=7) on a important with children because children are af-
statement. The questionnaire was composed of 20 fected by the way people are talking around them
questions about the websites and 4 demographic and they may answer differently if they are in front
questions. Gender was coded as 1 for boys and of somebody and they may feel uncomfortable
2 for girls. A downside of this research is that with a stranger. At home, they can answer easily
most constructs were measured through only one to the questionnaire provided.
question. Still, this is the only way sufficient re- The questionnaire was preceded and followed
sponses could be received form a child population. by in depth interviews. The aim of the pre-ques-
To deal with this shortcoming, we also gathered tionnaire interviews was to gather information on
qualitative data. the children’s general attitude towards the Internet.
Indeed, as is the case of much research that Another aim was to find out what expectations
deals with the DeLone and McLean model, we children have with respect to questionnaires and to
used the mixed method, which couples quantita- find an appropriate wording for the questionnaire.
tive and qualitative method. Such a design allows A problem with the fact that children would have
us to have a wider view of the research problem. to respond to the questionnaire at home is that
More specifically, we used the sequential mixed nobody would be able to provide them an explana-
283
tion of the wording if they would not understand. of 12 years). They like to create new personages
Thus, it is important to design the questionnaire and develop their dreamed personality: “I am
with appropriate wording and we had to reduce playing in an online game and I am a well known
the number of questions to the minimal level, be- warrior, but in real life I am not good at fighting
cause children may not agree to spend a lot of time at all” (Boy of 14 years).
answering it. Therefore, 8 pre-survey interviews Most of the children interviewed from 11 to 13
have been conducted in face-to-face mode. These do not have a computer in their bedroom; the girl
were aimed primarily at improving the accuracy aged 14 that we interviewed had a computer and
of the wording, the design and the length of the Internet access in her bedroom, but the Internet
survey. The interviews were conducted by two access is restricted by her parents. The computer
adults with one kid. One adult was someone the they use to go online are either their parents’ work-
interviewee knew in order to avoid bias linked to ing laptop, where they have no right to download
the risk of talking to strangers (the researcher). games and applications without permission, or
Moreover, the interviews were conducted at the a “family computer” placed in a specific room
participants’ home in order to let them navigate called the “bureau”. Access to that computer is
on the tool they used to use in daily life, in order very easy, but parents are often passing by in order
to avoid the bias linked to unknown hardware. to check what children are doing on the computer.
Two interviews were made per age (11, 12, 13, These findings are similar to what we have seen
14), one for each gender, to prevent any bias to in literature.
define the possibility of gender correlation with Internet use seems to depend upon gender.
some of our questionnaire parts. Boys prefer to play online games. Those can
The idea of the after-questionnaire interviews be either small and easy-to-understand games
was to gain insight in the answers provided by (somehow like the old packman or Tetris games)
the questionnaire responses. Two children were or crusade games that allow multiplayer strate-
selected in each classroom for this, leading to gies. The second category is more for boys from
six interviews. Moreover, another interview was 13 years onwards, because younger boys are not
conducted with the designer of one of the websites allowed to spend much time on Internet and this
(les petits repères), in order to get insight into type of game is time consuming. Girls are more
what they think about their website and how they oriented to websites such as msn, Caramail and
assess the success of it. other types of chat. Until the age of 12 they are
using it to talk to their friends; “I am on msn to
talk with my friends or to do some homework
RESEARCH RESULTS using their help” (Girl of 12 years). After the age
AND DISCUSSION of 12 they start to use it to have fun with friends
that are physically in the room by chatting with
Part 1: General Internet Behavior strangers: “One day we were talking on Caramail,
and my friend made a mistake by saying ‘sorry
The eight pre-survey interviews gave insight on I have to go to the Nirvana concert’. Of course
the general attitude towards internet. In general all the singer was dead, it was so funny!” (Girl of 14
children were open to new ideas and new things to years). Given this difference in Internet use, we
discover, especially on the Internet because they expect the statistical test to confirm Hypothesis
find the Internet more attractive than the real world: 5 (“The satisfaction with the site depends on the
“Internet is some kind of a magic world, we can child’s gender.”).
find everything without leaving the house” (Girl
284
Table 1. Results of the statistical tests
Satisfaction Know Brand Know Brand

Personalisation .177* -.066
Game Presentation .03 .059
Navigation interesting .281** .059
Dynamism .332** .044
Website -.024 .014
Gender -.127* -.016
Satisfaction .233* .285**
* p <=.05; ** < .001
The other important thing about the way site is perceived as helping better to find what is
children are using Internet is that they globally of interest, the child is also more satisfied with
believe that their skills on computer and Internet the site, confirming Hypothesis 3. Dynamism
are better than their parent’s skills. Hence, they do is also a significant antecedent of children’s
not search for advice from their parents. This was satisfaction with the website: the more dynamic
even true for the child whose parents are working animations and characters are present that help the
in informatics. “We have had a school course on child to understand the website, the higher is the
Internet, but I know everything, if I need something satisfaction with the site. Gender is a significant
I search it in the Google search box” (Boy of 13 control variable: boys were more satisfied with the
years). We wanted to know more about how they websites than girls. All in all, the model explains
are navigating online and the problem that we 44% (R²) of the variation in the satisfaction level.
saw is that children do not make the difference of As the second column with figures in Table 1
information seen on a chat and information seen shows, none of the variables mentioned earlier
on online encyclopedia. have a significant direct relation with the knowl-
edge of the brand. Still, the website satisfaction
Part 2: Statistical Test Results has a significant, positive relation with brand
knowledge. The effects of personalization, an
The results of the regression tests are shown in interesting navigation and dynamism upon brand
Table 1. The figures are the standardized betas, knowledge are thus mediated by the satisfaction
whereas an asterisk (*) shows the statistical sig- variable. The brand-knowledge predicting model
nificance. The figures in the left column show is statistically significant at p=0.001, but the R²
the results of a test to explain the satisfaction is only 15%.
with the website on the basis of the four concepts
mentioned earlier (personalization, game pre- Part 3: Post-Survey Interviews
sentation, interesting navigation and dynamism)
together with the two control variables. A website A group of 6 students have been interviewed after
that allows for sufficient personalization leads to completing the questionnaire. Two children in
more satisfaction for the child, confirming Hy- each class, but not necessary one boy and one
pothesis 1. A better game presentation does not girl since the selection has been made randomly.
lead to a higher satisfaction. Hypothesis 2 is thus Two children at two different places said (inde-
rejected. If the way navigation is enabled on the pendently of each other) that “R&R is more fun
285
than PR” [Paris - child 1, Orlean – child 2]. The product. One of the children even said that even if
fun criteria is clearly important to them, as is il- they see the characters they do not (consciously)
lustrated by the fact that more dynamic elements make the link between the brand characters and
lead to more satisfaction. The criteria “colours” the product itself. Such affirmations assume the
has been noticed four times in interviews, thus contrary of the information provided in a research
for further research including children we may paper (Youngjin & Alavi, 2001).
incorporate these criteria in the model of assess- We also had a semi-structured interview with
ing children websites. the website designer of “les petits repères”. The
Answers provided with the choice made on the web designer had developed the web structure
part of the questionnaire that deals with the presen- according to the advice of a teaching expert.
tation of the website explains the importance of a They had no information directly coming from
spatial presentation. The Petits Repères website is the children they are targeting. The only informa-
considered as “too well organized” (Brest – child tion they collect is the connection frequency on
2), the explanation of this “too well organized” each of the website’s parts, and globally there are
sentence has been explained by the child by the no specific parts that seem more attractive to the
fact that he likes to play and let his toys on the children. The games are updated every 3 months
ground and things that are too organized are not to provide new games to keep attracting children.
fun for him. The design proposed by the Rik et Globally they say they reach 30 000 visitors per
Rok website is preferred by the children but they month. The company that designs the website did
are unhappy with the speed of the defilement of not agree to put product images online, thus Petits
the isles, thus the design suffers from this speed, Repères has no product images on the website.
even if they find it more interesting than the design There is only some reminder of the brand through
proposed by Petits Repères. the logo and the name. They have also created a
The isles system proposed on the Rik et Rok bonus that involves access to hidden games by
website allows the children to have a global view entering a code that is presented on the packaging
of the website and what they can do on it. “The of the product “Les petits repères” but this is not
isles allow me to choose the theme that I want to working very well with children. Our contact says
work with” (Paris - child 2). This is not the case that it may be linked to the fact that the brand “les
with Petits Repères website because we have to petits repères” was not directly linked to children,
follow the arrow system in order to reach the and the brand was created to fit with the needs
game and one game can lead to others. “I have of people who wanted to buy low price products.
not understood the Petits Repères website. I did (At the contrary the Rik et Rok brand has been
not know which game I could play with” (Brest created in the aim of targeting directly children.)
– child 1). The designer concluded the interview by saying
Game presentation is not a significant anteced- that the website of “les petits repères” is searching
ent of children’s satisfaction with the website. The more to do indirect marketing through telling the
reason for this became evident when talking to the message of an environmental-friendly and healthy
children: they don’t read the game guidelines and way of living. This interview shows us that the
the game presentation does not matter to them. website of the brand is part of the brand market-
Finally, and most important, many of the 6 ing, but the web designer has no real feedback
children interviewed said that the connection be- on the aspects of the website that are profitable
tween the product and the brand is not clear, even (in terms of brand image and in terms of usage
if the Rik et Rok website presents the product of intention) and other parts that can be erased from
the brand on the website through pictures of the
286
the website of the company, in order to improve If we consider the fact that the parents are
the ease of use or the speed of the connection. keen on developing their children’s mind with
Our survey allows such companies to have an healthy consumption and protecting them from
overview of how their website is doing and what publicity, we may test the impact of the websites
kind of improvements they can make. The DeLone on their satisfaction with the brand. Because if
and McLean framework is used as a guideline children make no link between the website and
to provide an audit of the website faculties and the products, parents who are consuming general
even if the results that we have found could not food products proposed by both companies may
be generalized to every website it gives a guide have a better image of the brand after seeing their
to other managers to assess their website. children playing on their websites, due to the fact
The website satisfaction is impacting brand that games are oriented to healthy consumption
awareness, thus companies need to take care of and environmental issues.
the user satisfaction with their website. Otherwise
they will not be able to get the benefits of their
website. But we need to remind the fact that only FUTURE RESEARCH DIRECTIONS
a small percentage of the brand awareness is ex-
plained by user satisfaction, so managers should This paper could only investigate a small number
not use the company website as their biggest tool of antecedents of children’s satisfaction with com-
to increase brand awareness. pany websites. Unlike past studies which focused
One main aspect that can be taken into account on children’s parents or teachers as their main
by the web designer is the fact that children are not respondents, our study uses data collected from
looking at the explanations of the game. Hence, children themselves. Although such answers were
there is no need to spend much time on develop- not easy to collect, and the survey instrument has
ing game explanations and the games should be to be limited because of the respondents character-
intuitively clear. Moreover game explanations istics, our study provides unique data, which gives
may affect the speed of the navigation and lead insight into the children’s perceptions of websites,
to a negative impact. Other aspects of the website and not what parents or teachers think about what
should be developed like the personalization of a children think. Still, this leaves much space for
part of the web site or the dynamism of the web site. further research. Indeed, the model of DeLone
The petit repères designer had no intention to and McLean (2003) can be used to identify many
put the product image of the brand on the web site. other elements that may be relevant to assess. For
In contrast, the Rik et Rok web site had chosen example, other system quality elements such as
to put product information on each part of the the response time of the system seem to play a
site. The children were not aware of any relation role (as pointed out in the post-survey interviews).
between the product presented and the web site Similarly, other information quality elements,
games. Games proposed by both websites are such as the colours that are used, are also likely
more related to general issues (such as environ- to be significant antecedents of the satisfaction (as
ment and nutrition) than to the brand itself. This also pointed out by the post-survey interviews).
information also explains why despite the fact Other research could investigate the link between
that the majority of the children interviewed are a child’s satisfaction with the website and the
satisfied by the website, they only say to have a website design on the one hand and the intent of
small intention to purchase a product from that parents to buy products on the other hand.
brand (26% for Rik et Rok, and 23% for Petits
Repères; data not used in the statistical tests).
287
CONCLUSION REFERENCES
This paper studied children’s satisfaction with Brynjolfsson, E., & Hitt, L. M. (1996). Paradox
the use of commercial websites. We addressed lost? Firm-level evidence of the returns to Infor-
two relevant gaps in extant literature: children as mation Systems spending. Management Science,
users of information technology and the concept 42(4), 541–558. doi:10.1287/mnsc.42.4.541
of information system success itself. Children’s Chuan-Chuan, J. L., & Hsipeng, L. (2000).
use of IT needed focus since this age group had Towards an understanding of the behavioural
been largely neglected by extent IS research. intention to use a web site. International Journal
The concept of success needed further research of Information Management, 20(3), 197–208.
because the research results on this dependent doi:10.1016/S0268-4012(00)00005-0
variable have been controversial and there is still
no full agreement on the antecedents of success. DeLone, W. H., & McLean, E. R. (1992). Informa-
In this chapter we therefore contextualized the tion system success: The quest for the dependent
classic information systems success model of variable. Information Systems Research, 3(1),
DeLone and McLean (1992, 2003) to IS success 60–95. doi:10.1287/isre.3.1.60
with children. We carried out a survey of a sample
DeLone, W. H., & Mclean, E. R. (2003). The De-
of 76 children aged 11 to 14, located in various
Lone and McLean model of Information System
cities in France. This study not only allowed for
success: A ten year update. Journal of Management
confirming the role played by certain variables
Information Systems, 19, 9–30.
identified in the research literature as determinants
of user satisfaction, considered as an appropriate Etezadi-Amoli, J., & Farhoomand, A. F. (1996).
substitute for the IS success. These results are A structural model of end user computing sat-
also particularly insightful for companies who isfaction and user performance. Information &
wish to better address the needs of children and Management, 30(2), 65–73. doi:10.1016/0378-
increase the strength of their brand, though the 7206(95)00052-6
deployment of an appropriate commercial website.
Galletta, D. F., & Lederer, A. L. (1989). Some
Our results show that children get a higher brand
cautions on the measurement of user information
awareness if they are more satisfied with the kids
satisfaction. Decisions Science Journal, 20(2),
website of the brand. Still, qualitative data that
168–187.
was gathered suggests that children are not really
aware of the link between symbols they see on Hamilton, S., & Chervany, N. L. (1981). Evalu-
the site and products that are sold. We also found ating Information System effectiveness. Part I.
the satisfaction with the site depends upon the Comparing evaluation approaches. Management
personalization that is possible and the presence Information Systems Quarterly, 5(3), 55–69.
of dynamic elements on the site. Characters and doi:10.2307/249291
animations that help navigating the site are also
Leclercq, A. (2007). The perceptual evaluation of
appreciated. A good presentation of the site’s rules
Information Systems using the construct of user
could not be shown to be a significant antecedent
satisfaction: Case study of a large French group.
of satisfaction. Several important consequences
The Data Base for Advances in Information Sys-
can thus be drawn from this research, which may
tems, 38(2), 27–60.
help companies to (re)design their commercial
website in a successful manner.
288
Livingstone, S., Bober, M., & Helsper, E. J. (2005). Teo, T. S. H., & Wong, P. K. (1998). An empirical
Active participation or just more information? study of the performance impact of computeriza-
Information Communication and Society, 8(3), tion in the retail industry. The International Journal
287–314. doi:10.1080/13691180500259103 of Management Science, 26(5), 611–621.
Molla, A., & Licker, P. S. (2001). E-commerce Wixom, B. H., & Watson, H. J. (2001). An
systems success: An attempt to extend and re- empirical investigation of the factors affect-
specify the Delone and McLean model of IS suc- ing data warehousing success. Management
cess. Journal of Electronic Commerce Research, Information Systems Quarterly, 25(1), 17–41.
2(4), 131–141. doi:10.2307/3250957
Powers, R. F., & Dickson, G. W. (1973). MIS Yoo, Y., & Alavi, M. (2001). Media and group
project management: Myths, opinion and reality. cohesion: Relative influences on social presence,
California Management Review, 15(3), 147–156. task, participation, and group consensus. Man-
Rababah, O., Al-Shboul, M., & Al-Sayyed, R.
371–390. doi:10.2307/3250922
(2011). A new vision for evaluating the quality
of e-commerce websites. International Journal Younghwa, L., & Kozar, K. A. (2006). Investi-
of Advanced Corporate Learning, 4(1), 32–41. gating the effect of website quality on e-business
success: An analytic hierarchy process (AHP)
Razzouk, N., Seitz, V., Lamude, K., & Kepekci,
approach. Decision Support Systems, 42(3),
A. C. (2005). A content analysis of value-based
1383–1401. doi:10.1016/j.dss.2005.11.005
advertising on the Internet: An empirical investiga-
tion. Journal of Website Promotion, 1(3), 61–73. Zviran, M., & Erlich, Z. (2003). Measuring IS
doi:10.1300/J238v01n03_05 user satisfaction: Review and implications. Com-
munications of the AIS, 12(5), 81–103.
Rowe, F. (2003). Faire de la Recherche en Sys-
tèmes d’Information. Vuibert, Fnege.
Schaupp, L. C., Bélanger, F., & Weiguo, F. (2009).
ADDITIONAL READING
Examining the success of websites beyond e-
commerce: An extension of the IS success model. Cenfetelli, R. T., Benbasat, I., & Al-Natour, S.
Journal of Computer Information Systems, 49(4), (2008). Addressing the what and how of online ser-
42–52. vices: Positioning supporting-services functional-
Seddon, P. B. (1997). A respecification and ex- ity and service quality for business-to-consumer
tension of the DeLone and McLean model of IS success. Information Systems Research, 19(2),
success. Information Systems Research, 8(3), 161–181. doi:10.1287/isre.1070.0163
240–253. doi:10.1287/isre.8.3.240 Cheolho, Y., & Sanghoon, K. (2009). Devel-
Seddon, P. B., & Kiew, M.-Y. (1994). A partial oping the causal model of online store suc-
test and development of the DeLone and McLean cess. Journal of Organizational Computing
model of IS success. Proceedings of the Inter- and Electronic Commerce, 19(4), 265–284.
national Conference on Information Systems, doi:10.1080/10919390903262644
Atlanta, GA, (pp. 99–110). Children’s Online Privacy Protection Act of 1998
(COPPA). (2000). Journal of Internet Law, 3(7),
22.
289
Ghandour, A., Benwell, G., & Deans, K. (2010). Lin, J. (2007). Online stickiness: Its antecedents
Measuring ecommerce website success. Interdis- and effect on purchasing intention. Behaviour
ciplinary Journal of Contemporary Research in & Information Technology, 26(6), 507–516.
Business, 1(12), 21–41. doi:10.1080/01449290600740843
Hart, T. A., Chaparro, B. S., & Halcomb, Mangiaracina, R., Brugnoli, G., & Perego, A.
C. G. (2008). Evaluating websites for older (2009). The ecommerce customer journey: A
adults: Adherence to ‘senior-friendly’ guide- model to assess and compare the user experience
lines and end-user performance. Behaviour of the ecommerce websites. Journal of Internet
& Information Technology, 27(3), 191–199. Banking & Commerce, 14(3), 1–11.
doi:10.1080/01449290600802031
Ming-Hsien, Y., Chandlrees, N., Binshan, L., &
Hernández, B., Jiménez, J., & Martín, M. (2009). Hung-Yi, C. (2009). The effect of perceived ethical
Key website factors in e-business strategy. Interna- performance of shopping websites on consumer
tional Journal of Information Management, 29(5), trust. Journal of Computer Information Systems,
362–371. doi:10.1016/j.ijinfomgt.2008.12.006 50(1), 15–24.
Hong, W., Thong, J. Y. L., & Tam, K. Y. (2004). O’Connor,H.,&Madge,C.(2004).‘Mymum’sthirty
Does animation attract online users’ attention? The years out of date’. Community Work & Family, 7(3),
effects of flash on information search performance 351–369. doi:10.1080/1366880042000295754
and perceptions. Information Systems Research,
15(1), 60–86. doi:10.1287/isre.1040.0017
criteria: determining which criteria count most.
Jackson, L. A., Samona, R., Moomaw, J., Ramsay, Electronic Commerce Research, 10(2), 191–208.
L., Murray, C., Smith, A., & Murray, L. (2007). doi:10.1007/s10660-010-9051-3
What children do on the Internet: Domains vis-
Reuter, K. (2005). Weaving a library Web: A guide
ited and their relationship to socio-demographic
to developing children’s websites. Journal of the
characteristics and academic performance.
American Society for Information Science and
Cyberpsychology & Behavior, 10(2), 182–190.
Technology, 56(14), 1555–1556. doi:10.1002/
doi:10.1089/cpb.2006.9970
asi.20211
Jones, S. C., Wiese, E., & Fabrianesi, B. (2008).
Schaupp, L., Bélanger, F., & Weiguo, F. (2009).
Following the links: Food advertising and promo-
Examining the success of websites beyond e-
tion on children’s magazine websites. Journal
commerce: An extension of the IS success model.
of Nonprofit & Public Sector Marketing, 20(2),
Journal of Computer Information Systems, 49(4),
165–190. doi:10.1080/10495140802224803
42–52.
Lee, Y., & Kozar, K. A. (2006). Investigating the
Stol, W. h., Kaspersen, H. W., Kerstens, J. J.,
effect of website quality on e-business success:
Leukfeldt, E. R., & Lodder, A. R. (2009). Gov-
An analytic hierarchy process (AHP) approach.
ernmental filtering of websites: The Dutch case.
Decision Support Systems, 42(3), 1383–1401.
Computer Law & Security Report, 25(3), 251–262.
doi:10.1016/j.dss.2005.11.005
doi:10.1016/j.clsr.2009.03.002
Liau, A., Khoo, A., & Ang, P. (2008). Parental
awareness and monitoring of adolescent Internet
use. Current Psychology (New Brunswick, N.J.),
27(4), 217–233. doi:10.1007/s12144-008-9038-6
290
Tam, K. Y., & Ho, S. Y. (2005). Web personal- Waiguny, M., Nelson, M. R., & Terlutter, R.
ization as a persuasion strategy: An elaboration (2010). Persuading playfully? The effects of
likelihood model perspective. Information Sys- persuasion knowledge and positive affect on
tems Research, 16(3), 271–291. doi:10.1287/ children’s attitudes, brand beliefs and behaviors.
isre.1050.0058 American Academy of Advertising Conference
Proceedings, 67-70.
Van der Heijden, H., Verhagen, T., & Creemers, M.
(2003). Understanding online purchase intentions: Wonsun, S. (2010). Consumer socialization on-
Contributions from technology and trust perspec- line: Antecedents and consequences of children’s
tives. European Journal of Information Systems, skepticism toward online advertising. American
12(1), 41–48. doi:10.1057/palgrave.ejis.3000445 Academy of Advertising Conference Proceedings.
291
Section 6
Applications
293
Chapter 15
A Model to Measure
E-Learning Systems Success
Ahmed Younis Alsabawy
Aileen Cater-Steel
Jeffrey Soar
ABSTRACT
E-learning involves adopting and exploiting the potential of new, advanced Information Technology
in development and delivery of education. In spite of a rapid growth in the e-learning field there still
exists a range of issues facing the stakeholders of e-learning systems. One of the key issues is how to
measure e-learning system success. Although considerable attention has been paid to the Information
Systems success issue, there remain arguments about the factors which are most effective for measur-
ing Information System success. The issue of measuring Information System success has an impact on
evaluating e-learning systems success. This chapter aims to fill this void by proposing an evaluation
methodology model to assess e-learning systems success. The contribution of this study is the proposed
model to evaluate the success of e-learning systems. The model is based on a thorough review of the
e-learning success literature and existing Information Systems success models.
INTRODUCTION 2008). Ahmed et al. (2007) consider technology

as the main aspect of the college today and it is
The education sector has been radically impacted believed to be the essential cause behind students’
by developments in information technology. In the and communities’ success. Rogers (2000) argues
education arena, substantial amounts of money that the role of technology in generating rapid
have been invested in the systematic development change in higher education cannot be ignored and
of technology infrastructure (Georgina & Olson, education has received substantial benefits from
the use of IT. The introduction of IT has supported
DOI: 10.4018/978-1-4666-0170-3.ch015 communication between teachers and students,
A Model to Measure E-Learning Systems Success
and communication between different universi- individual and organisational goals. The introduc-
ties around the world. Using library resources has tion of IT is no longer limited to back-office busi-
become easier because of IT. The introduction of ness functions but has grown to include the core
IT in higher education is deemed to be the main processes in health, education, transport, bank-
key to quality improvements (Turoff, 1999) and ing, and other fields. The advantages, which are
one of the most crucial IT enabled developments generated by using technologies such as internet,
in the last decade (McGill & Klobas, 2009). hardware, and software, have pushed organisations
In the context of knowledge management, e- to employ IT to facilitate more and more of their
learning has become a key driver in establishing activities. Shannak (1999) argues that performance
virtual communities (Hardaker & Smith, 2002). of primary and supportive activities is considered
New technologies such as mobile and iPod have impracticable in organisations without informa-
been employed in e-learning. The main purpose tion technology. In addition, impacts of using IT
of using these technologies is “to enhance active are extended to include the macro economy of
learning methods and assess students’ understand- different countries. Doig (2002) states that “the
ing” (Stav et al. 2010, p. 179). Because of the inter- information revolution is sweeping through our
net revolution, it is now possible to exchange and economy. No company can escape its effects”
disseminate information and knowledge between (p. 1).
members of virtual communities without consid- E-learning is believed to be the main outcome
ering hierarchical channels (Koh & Kim, 2004). of adopting and using the new and more advanced
E-learning systems are knowledge management IT in the education sector. These e-learning sys-
systems; knowledge management tools such as tems have also been adopted by non-educational
collaboration and community tools, and software organisations (Wang & Wang, 2009). Daneshgar
enable the learners to acquire knowledge (Lau & and Toorn (2009) suggest that “In order to sustain
Tsui, 2009). Institutions of higher education are a competitive advantages in today’s economy, char-
main source for creating, transferring, exchanging, acterised by rapid change, knowledge-intensive
and saving knowledge. These institutions attempt and technological-orientation, organizations are
to find effective approaches to enhance the process adopting e-learning to facilitate the achievement of
of knowledge and to achieve their objectives; e- lifelong learning, and to retain employees” (p.16).
learning systems have provided an effective way The introduction of e-learning systems enables
for achieving this. There still exists a range of is- non-educational organisations to receive valuable
sues facing the stakeholders of e-learning systems; benefits. For example, IBM saved USD200 mil-
one of the key issues is measuring e-learning lion in 1999, providing five times the learning at
system success. Although considerable attention one-third the cost of their previous methods and
has been paid to the information systems success Rockwell Collins reduced training expenditure by
issue, there remain arguments about the factors 40% with only a 25% conversion rate to web-based
which are most effective in measuring informa- training (Strother, 2002). Furthermore, according
tion system success (Rai, Lang, & Welker, 2002). to Giga Information Group, use of an e-learning
system to train employees rose from 21% in 2002
to 75% in 2005 (Wang & Wang, 2009).
BACKGROUND In the USA, 90% of 2-year and 89% of 4-year
public education institutions offered distance
Information technology (IT) has become an es- education courses in 2000-2001 with enrolments
sential factor in organisational success due to the of 1,472,000 and 945,000 respectively out of
critical role of IT in enabling the achievement of total enrolment of 3,077,000 (Holsapple & Lee
294
Post, 2006). Furthermore, learning management use the term differently” (p.23). This lack of evalu-
systems (LMSs) have been adopted by 95% of ation of e-learning systems success is believed
all higher education institutions in the United to be a central concern for the researchers and
Kingdom (McGill & Klobas, 2009). It is worth the stakeholders of these systems. MacDonald
mentioning that transnational courses are de- and Thompson (2005) state that “Educators and
livered by most Australian universities through researchers have voiced concern over the lack
using educational software (Shurville, O’Grady, of rigorous evaluation studies of e-learning pro-
& Mayall, 2008). According to the Australian grams” (p. 1). According to Ardito et al. (2006)
Flexible Learning Framework (2009), a survey of an effective methodology to evaluate e-learning
registered training organizations (RTOs) indicated system success is still unavailable. Furthermore,
that 39% of all vocational education and training the issue of lack of an effective methodology to
(VET) activities were based on e-learning systems. evaluate e-learning system success is no longer
Because e-learning systems are considered to restricted to higher education field but now ex-
be a special kind of information system (Wang, tends to the non-educational organisations. Wang
Wang, & Shee, 2007) the issue of measuring et al. (2007) state that “ Little research has been
information system success has an impact on conducted to assess the success and/or effective-
evaluating e-learning systems success as well. ness of e-learning systems in an organizational
In the context of e-learning systems, this issue context” (p.1792).
is considered more complicated because the
e-learning term is used with different points of
view. In addition, e-learning systems deal with MOTIVATION
different types of customers, for instance, inter-
nal customers (services provider, ICT staff, and E-learning systems are considered to be multidis-
academics); and external customers (students and ciplinary so the evaluation of these systems will
trainers). The variety of stakeholders is deemed to be from different points of view. The literature
be an essential challenge in delivering e-learning on e-learning emerged from two main fields:
services to the customers and in evaluating the information systems and education. In this terms,
success of these systems. The main justification Ozkan and Koseler (2009) state that “the success
of this claim is that different customers have dif- of an e-learning system may be considered as an
ferent needs and different criteria to evaluate this emerging concept of ‘social issues’ and ‘technical
system. In this regard Zeithaml (2002) states that issues’ and depends on numerous circumstance,
“Too many companies are performing poorly in rather than a black-and-white formula” (p. 1286).
delivering service on the Web, and a large part of Before 1990, the evaluation of e-learning systems
this problem is the lack of complete understanding success received little attention from researchers.
of what customers want in this medium” (p. 135). According to McGorry (2003), the main direction
This challenge becomes more complex, especially of research was the differences between traditional
when the members of the same stakeholder group and distance education. After 1990, the direc-
have different aspects. This challenge is clearly tion of research in this field started to focus on
apparent with students because the backgrounds the issue of the quality of e-learning (McGorry,
of students who take the courses via the internet 2003) (MacDonald & Thompson, 2005). This
may be significantly different from those in full- direction was supported after attention was paid
time, mainstream courses (Roffe, 2002). Cohen to this issue by some educational institutions such
and Nycz (2006) state that “E-learning can be as the western interstate commission for higher
difficult to understand because different authors
295
education (WICHE), and the institute for higher in the last decade. However, these applications are
education policy (Frydenberg, 2002). still facing the challenge of failure. The initiatives
Quality is considered to be an essential factor of e-commerce still encounter problems. Pather
in assessing e-learning system success but there (2006) states that “There is little understanding of
are other factors that should be considered in the the crucial importance of managing the technol-
evaluation process. In spite of the importance of ogy through which the internet and Web delivers
this construct in evaluating the whole system or e-Commerce opportunities”.
information system department, it is considered to E-learning systems also encounter the prob-
be not enough to measure the success of a single lem of failure. Despite the considerable amounts
system. The quality of other constructs should of money which have been spent on e-learning
be taken into account such as system quality and systems, some organisations continue to fail to
information quality (DeLone & McLean, 2003). meet the targeted advantages (Strother, 2002).
The stakeholders are considered an essential According to Rovai and Downey (2009), the
component in organisations’ activities. In the British Government spent $113 million in 2000 to
information systems field, the stakeholders are establish an e-learning project called The United
believed to be a significant factor in evaluating Kingdom e-University (UKeU). In February
information systems success (Shee & Wang, 2004, the British Government announced that
2008). “In the same sense, the stakeholders of UKeU had failed because it did not meet recruit-
e-learning are those that are affected by it” (Wag- ing targets. In another example, the New York
ner, Hassanein, & Head, p. 28). In the context of University online (NYU) closed due to economic
e-learning systems, studies continue to ignore the conditions. The lack of evaluation is believed to
issue of multiple stakeholders because most of be a significant reason for the failure e-learning
the research has focused on single stakeholders, systems. According to McGorry (2003) many
such as students. This direction of research led to educational institutions have not considered this
scant attention to establishing a comprehensive important issue of evaluating e-learning systems.
measurement that can evaluate e-learning systems As Ozkan and Koseler (2009) state “However,
success and ensure the stakeholders achieved their the development, management, and continuous
goals. This study aims to fill this void by propos- improvement of e-learning system are quite
ing an evaluation methodology model to assess challenging both for the educational institutions
e-learning systems success. and for industry. In that, assessment has become
A critical issue facing IT projects is their high an essential requirement of a feedback loop for
rate of failure. The major concern of the organisa- continuous improvement” (p. 1286). Therefore,
tions’ senior management is the lack of success these systems need to be assessed continuously
of the new information systems and their failure to make sure that the outputs meet users’ needs.
in achieving the expected goals (Saleh, 2002). However, there are some dilemmas in measuring
Furthermore, Saleh (2002) states that “In 1995 the success of e-learning systems and in determin-
in the United State alone, 31.1% of projects are ing the most effective technique to undertake this
cancelled before they finish, with a cost of 81 process (McGorry, 2003; Wang, et al., 2007;Ardito
billion. Only 52.75% of projects are completed et al., 2006). Thus, the research problem addressed
with 189% of their original estimated cost, out of in this chapter is: What are the main factors con-
those, only 42% of the originally proposed features sidered to be important in measuring e-learning
and functions are fulfilled” (p.1). The electronic system success?
applications such as e-commerce, e-learning, e- E-learning systems have become essential in
health, and e-banking have become commonplace universities and substantial amounts of money are
296
spent on these systems yearly (Georgina & Olson, Measuring e-learning system success is the
2008). Hence, assessing e-learning systems suc- main objective of this study. To achieve this
cess is a crucial issue. The process of evaluating objective, a proposed model has been designed.
e- system success is significant because it assists in More specifically, the objective of this study is
managing, maintaining, and developing these sys- to identify those factors which affect e-learning
tems and to diagnose the problems that need to be systems success and place them in a holistic model.
solved. The main purpose of the evaluation process
is to ensure that the objectives of implementing
the systems are achieved. The differences in goals LITERATURE REVIEW
of stakeholders create a difficulty in assessing the
success of e-learning systems. Furthermore, most The prior studies are believed to be the essential
of the previous research that has dealt with the basis to establish the theoretical model. Informa-
e-learning systems success issue was limited to tion system success has received considerable
one type of stakeholder i.e. students, and ignored attention by researchers in this arena. Previous
the other types of stakeholders. The diversity of contributions in this field were in different di-
stakeholders who evaluate e-learning systems will rections because of differences in views held by
provide a holistic picture about these systems and authors. The review of the literature is undertaken
their outputs. The significance of this study is to in two directions: information system success and
identify the factors impacting on the success of e-learning system success.
e-learning systems and place these factors in a
proposed model. Four views have been considered Information System Success
in designing the study model and as follows:
Success of information systems is believed to be
1. Technical view: this model includes factors the central issue in the information systems field.
to measure the technical performance of this Because of considerable amounts of money and
system and to identify the efficiency and the efforts being spent on IT/IS projects, the evalua-
quality of this system. tion of their success has became a critical issue.
2. Attitude view: this view includes the factors So, substantial research has been performed and
which deal with user behavior in using e- considerable attention paid to this issue. The
learning systems and the satisfaction gained diversity in approaches of researchers has led to
from the use of this system. the development of different methodologies and
3. Marketing view: includes the factors relat- models to measure information system success.
ing to the product of e-learning systems The contributions to measuring information sys-
(e-learning service). These factors relate to tem success can be classified in four approaches.
service quality delivery and IT infrastructure
service. Technology Acceptance Approach
4. Organisational view: this view relates to net
benefits generated from using e-learning The technology acceptance model (TAM) is
systems. The net benefits block has been considered to be one of the most important
prepared based on organizational effective- contributions in the information systems arena.
ness literature especially through the contri- “TAM continues to be the most widely applied
butions of Cameron (1981; 1986; 1978). theoretical model in the IS field” (Lee, Kozar, &
Larsen, 2003, p. 752). Davis (1986) introduced
TAM. This model was established based on the
297
Figure 1. Theory of reasoned action (Ajzen & Fishbein, 1969)
theory of reasoned action (TRA) developed by uptake and acceptance of these systems (Smart,
Ajzen and Fishbein (1969). Figure 1 shows TRA. 2009). Figure 2 shows TAM.
The focus of TRA is to identify the determinants Ongoing maintenance and enhancement of
of consciously intended behavior (Davis, Bagozzi, this model has been continued by researchers.
& Warshaw, 1989). According to TRA, actual Venkatesh and Davis (2000) developed TAM 2
behavior is believed to be the outcome of the ef- by adding the social influence process (subjective
fects of two categories of significant beliefs: norms, voluntariness, and image) and the cogni-
behavioral and normative (Sagar, 2006). Based tive instrumental process (job relevance, output
on TRA, Davis (1986) introduced the TAM. The quality, result demonstrability, and perceived ease
goal of TAM is to explain the determinants of of use).
computer acceptance incorporating user behavior Based on eight previous models of technology
across a broad range of technologies and popula- acceptance, Venkatesh et al. (2003) developed a
tions, while at the same time being both “parsi- new model called Unified Theory of Acceptance
monious and theoretically justified” (Davis, et and Use of Technology (UTAUT). The determi-
al., 1989, p. 985). The main use of TAM is to nants of usage behavior according to UTAUT
measure success of information systems through are performance expectancy, effort expectancy,
social influence, and facilitating conditions. The
Figure 2. Technology acceptance model (TAM) (Davis, 1989)
298
mediation variables were gender, age, experience, User Involvement Approach

and voluntariness. UTAUT is believed to be a
useful tool for managers to evaluate the success Much research has been conducted on user
of new technology (Venkatesh et al., 2003). An- involvement in the information systems field.
other contribution to the body of knowledge of Some of these researchers have considered user
TAM is TAM 3. This model was developed by involvement as a critical factor in implementation,
Venkatesh and Bala (2008). The factors affecting development, and success of information systems.
the acceptance of technology according to TAM3 Baroudi et al. (1986) state that “User involvement
are grouped into three categories, namely control in information system development is generally
beliefs, intrinsic motivation, and emotion. considered an important mechanism for improv-
ing system quality and ensuring successfully
User Satisfaction Approach systems implementation” (p. 232). Furthermore,
Tait and Vessey (1988) considered user involve-
User satisfaction is considered one of the most ment to be one of the most important factors in
important factors in assessing success of infor- the design and implementation processes. There
mation systems (Sedera & Tan, 2010). Since the are many studies conducted on the role of user
1980s, user satisfaction has received consider- involvement in information system success. Har-
able attention as a measurement of information ries and Weistroffer (2009) selected 28 studies
system success. Bailey and Pearson (1983) are about user involvement published from 1996 to
believed to be the earliest researchers who used 2009 in scholarly journals. The finding of these
user satisfaction to measure information systems 28 studies is that the user involvement is believed
success. Bailey and Pearson (1983) state that “If to be an essential factor in creating information
a formal information system exists, its success system success.
at meeting those needs either reinforces on frus-
trates the user’s sense of satisfaction with that DeLone and McLean Approach
source” (p. 530). Ives et al. (1983) also adopted
user information satisfaction (UIS) to evaluate There have been many attempts to measure
information systems success. UIS is considered information systems success. The DeLone and
an essential measurement to evaluate the success McLean model (1992;2003) is believed to be
of information systems. Ives et al. (1983) state the most commonly used. This model was based
that “UIS is a perceptual or subjective measure on a review of 180 articles. After analysing the
of system success; it serves as a substitute for variables in these articles the outcomes were clas-
objective determinants of information system ef- sified within six dimensions of information system
fectiveness which are frequently not available” (p. success. These dimensions and the relationships
785). The contribution to the body of knowledge between them are shown in Figure 3.
of user satisfaction still continues. It is used as This model is considered to be a comprehen-
an essential construct in the models which deal sive model because it does not depend on a single
with the information system success issue. This construct; also it is an attempt to create an inte-
approach has received considerable support for grated framework to measure information system
measurement of information success by research- success. Many studies have been conducted to
ers in the information system discipline such test the validity of this model, for example, Sed-
as Doll and Torkzadeh (1988), Melone (1990), don (1997), Rai et al. (2002), Iivari (2005), and
Gelderman (1998), and Wang and Liao (2007). Wang (2008). In 2003 DeLone and McLean pub-
lished the updated model of information system
299
Figure 3. DeLone and McLean model of Information System success (DeLone & McLean, 1992)
success. The updated model was based on sug- system success have been in different directions.
gestions offered by other researchers, criticisms Different criteria in evaluating e-learning system
directed to the original model, and the empirical success have appeared because of differences in
studies which have adopted this model. Service approaches adopted by various authors as to the
quality and intention to use have been added to term e-learning (Ozkan & Koseler, 2009). The
the updated model, and the net benefits have been studies which have dealt with this issue can be
used instead of individual and organizational classified into four approaches.
impacts. Figure 4 shows the updated DeLone and
McLean model. Technology Acceptance Approach
E-Learning System Success This approach is considered to be a common ap-

plication in the information system field. The main
E-learning systems are believed to be a recent purpose of using this approach is to measure the
application of information technology in the acceptance of using technology and the success
education sector. Therefore, there is still a lack of of these technologies. The studies which adopted
techniques which are used to evaluate success of TAM in the e-learning systems field have changed
these kinds of systems. The attempts of researchers some constructs of this model and extended the
to create suitable instruments to assess e-learning model. Roca et al. (2006) combined Expectancy
Figure 4. Updated DeLone and McLean model (DeLone & McLean, 2003)
300
Disconfirmation theory and TAM to create a new on the quality of the e-learning system as a whole
model to measure e-learning continuous inten- and are not limited to service quality. MacDonald
tion. Martinez-Torres et al. (2008) adopted TAM et al. (2001) have proposed a model called the
and made essential changes to the constructs of demand-driven learning model (DDLM). This
this model. The main purpose of these changes model was established relying on five factors
on TAM items is to make them relevant to the e- which were considered to be essential to creating
learning system usage context. The studies which e-learning quality in higher education: structure,
are considered supportive of this approach are content, delivery, service, and outcome. Studies
conducted by Selim (2007), Abbad et al. (2009), by McGorry (2003), MacDonald and Thompson
and Ngai et al. (2007). (2005), and Lee and Lee (2008) are believed to
be supportive of this approach.
User Satisfaction Approach
DeLone and McLean Approach
User satisfaction has received considerable at-
tention from the researchers in the information E-learning systems are considered to be the
system field. This attention included e-learning most important IT projects in universities (Lee,
systems. Sun et al. (2008) classified the critical Yoon, & Lee, 2009). However, the evaluation of
factors which drive successful e-learning in six these systems is still facing problems as there
dimensions which are learner, instructor, course, is a lack of measurements to evaluate the suc-
technology, design, and environmental. Based on cess of these projects. The DeLone and McLean
the organizational perspective, Naveh et al. (2010) model is believed to be one of the most important
investigated students’ use and satisfaction of measurements which can be used to address this
learning management systems (LMS). According issue in the e-learning field. Some studies have
to this perspective, three organizational variables adopted this model without changes, for example
lead to student use and satisfaction of LMS. The Lin (2007). Other studies have considered this
organizational variables are the role definition model as a basis to creating new models to assess
and the departmental division as derived from e-learning system success, for example, studies
orientation of the academic institution according conducted by Holsapple and Lee-Post (2006), and
to discipline goal, the commitment to hierarchi- Lee-Post (2009).
cal degree structures and accreditation process
as derived from coordinator mechanisms of the
academic institution, and the regularized aspects PROPOSED MODEL
of relationships among roles. Studies conducted
by Shee and Wang (2008), and Wu et al. (2010) E-learning systems are facing the critical problem
are considered to be supportive of this approach. of measuring success of these systems. Based on
that, this study raises a number of research ques-
E-Learning Quality Approach tions to investigate the factors which are affecting
e-learning systems success. A model is proposed
Studies have and still pay considerable attention based on these factors. The proposed model is a
to e-learning quality. Also, the quality issue has contribution to the field of information systems.
received attention from educational institutions According to James et al. (1982) “Theory means
such as WICHE, and the Institute for Higher a set (or sets) of interrelated causal hypotheses
Education Policy (Frydenberg, 2002). The con- that attempts to explain the occurrence of phe-
tributions which adopted this approach focused nomena, physical, biological, social, cultural, or
301
Figure 5. Proposed model to measure e-learning system success
psychological” (p. 27). Based on the definition of 2. Causal connection among the variables.
theory above, the causality approach is adopted 3. A theoretical rationale for each causal
in this study. hypothesis.
Causality or ‘cause and effect’ approach is 4. Boundaries, which specify the context.
considered to be an important tool to discover 5. Stability, which implies that the hypoth-
the phenomena. The introduction of the causality esized structure of causal connections will
approach has received considerable attention in be consistent over specified time intervals.
social science and is frequently used in the infor-
mation systems field. The main justification to Figure 5 shows the model developed from the
use this approach is that it provides the ability to literature review which is summarised in Table 1.
show causal relationships among the factors of the The proposed model was established based on
phenomena occurring in a physical system (Atoji, the epistemological and ontological concepts.
Koiso, & Nishida, 2002). To establish a theoreti- Epistemology can be classified as positivist, in-
cal model there are five basic components that terpretive, and critical. Positivist studies are used
should be considered (James, et al., 1982, p. 27): with structured instrumentation and are based on
a priori fixed relationships. Interpretive studies
1. Phenomena or the variables that act as causes explicitly adopt a nondeterministic perspective
and effect. and attempt to explore the phenomena of interest
302
Table 1. List of studies supportive of model
Factors Information System References E-learning System References

Information Bailey and Pearson (1983), Miller and Doyle (1987), DeLone and Holsapple and Lee Post (2006), Roca et
Quality McLean (1992) (2003), Seddon (1997), Skok et al. (2001), Rai et al. (2006), Wang et al. (2007), Ozkan and
al. (2002), Kahn et al. (2002), Lee et al. (2002), McKinney et al. Koseler (2009),Wang and Wang (2009),
(2002), Bharati and Berg (2005), Iivari (2005), Byrd et al. (2006), Ramayah et al. (2010).
Ifinedo (2006), Nicolaou and McKnight (2006), Stvilia et al.(2007),
Stvilia et al. (2008), Price et al.(2008), Gable et al. (2008), Wang
(2008), Zhi-yong et al. (2009), Gorla et al. (2010), Gorla and Lin
(2010), Landrum et al. (2010).
System Quality Bailey and Pearson (1983), Mahmood (1987), DeLone and McLean Holsapple and Lee Post (2006), Roca et al.
(1992) (2003), Wang and Strong (1996), Seddon (1997), Skok et (2006), Wang et al. (2007), Liaw (2008),
al. (2001), Rai et al. (2002), McKinney et al. (2002), Iivari (2005), Ozkan and Koseler (2009),Wang and Wang
Bharati and Berg (2005), Byrd et al. (2006), Ifinedo (2006),Wang (2009), Ramayah et al. (2010), McGill and
(2008), Zhi-yong et al. (2009), Gable et al. (2008), Landrum et al. Klobas (2009).
(2010), Gorla et al. (2010), Gorla and Lin (2010).
Usefulness Davis (1989), Seddon (1997), Venkatesh and Davis (2000), Rai et Arbaugh (2000), Pituch and Lee (2006),
al. (2002), Hung (2003), Yang (2005), Byrd et al. (2006), Sabherwal Roca et al. (2006), Liaw (2007), Martinez-
et al. (2006), Landrum et al. (2007), Venkatesh and Bala (2008), Torres et al. (2008), Lee-Post (2009), Wang
Larsen et al. (2009), Landrum et al. (2010), Yeh and Teng (2011). and Wang (2009), Abbad et al. (2009), Joo
et al. (2011), Cho (2011).
User Satisfaction Bailey and Pearson (1983), Ives et al. (1983), Baroudi et al. (1986), Arbaugh (2000), Roca et al. (2006),
Lehman (1996), Doll and Torkzadeh (1988), DeLone and McLean Holsapple and Lee Post (2006), Wang et
(1992) (2003), Amoli and Farhoomand (1996), Seddon (1997), al. (2007), Shee and Wang (2008), Sun et
Skok et al. (2001), Rai et al. (2002), Xiao and Dasgupta (2002), al.(2008), Adeyinka and Mutula (2010), Wu
McKinney et al. (2002), Xiao and Dasgupta (2005), Ong and Lai et al. (2010), Naveh et al. (2010).
(2007), Wixom and Todd (2005), Iivari (2005), Sabherwal et al.
(2006), McGill and Klobas (2008), Wang (2008), Gable et al.
(2008), Landrum et al. (2010), Udo et al. (2010), McNamar and
Kirakowski (2011).
Customer Value Zmud (1983), Snitkin and King (1986), Aldag and Power (1986), Wang et al. (2007), McGill and Klobas
(Internal cus- Skok et al. (2001), Iivari (2005), Ifinedo (2006), Davern and Wilkin (2008).
tomer) (2010).
Customer Value Hitt and Brynjolfsson (1996), Shun and Yunjie (2006), Wang Chiu et al. (2005), Holsapple and Lee
(External) (2008), Chang et al. (2009), Kuo et al.(2009). Post (2006), Adeyinka and Mutula (2010),
Martinez-Torres et al. (2008).
Organizational Benbasat and Dexter (1986), Miller and Doyle (1987), Hitt and Wang et al. (2007).
Value Brynjolfsson (1996), Seddon (1997), Mirani and Lederer (1998),
Amit and Zott (2001), Skok et al. (2001), Shang and Seddon (2002),
Gable et al. (2008), Tzeng et al. (2008), Gorla and Wong (2010),
Gorla and Lin (2010).
Society Value Seddon (1997), Ryan et al. (2002), Tallon et al. (2000). ----------
Service Quality Pitt et al. (1995), Dyke et al. (1997), Berry and Parasuraman (1997), Brigham (2001), McLoughlin and Luca
Delivery Watson et al. (1998), Zeithaml et al. (2000), Liu and Arnett (2000), (2001), Frydenber (2002), Mcgorry (2003),
Cox and Dale (2001), Yoo and Douthu (2001), Zeithaml et al. Chiu et al.(2005), Reid (2005), Oliver
(2002), Zeithaml (2002), Wolfinbarger and Gilly (2003), Wilkin and (2005), MacDonald and Thompson (2005),
Castleman (2003), Landrum and Prybutok (2004), Yang and Fang Roca et al. (2006), Holsapple and Lee Post
(2004), Parasuraman et al. (2005), Kettinger and Lee (2005), Yang (2006), Wang et al. (2007), Lee and Lee
et al. (2005), Lai (2006), Lee and Kozar (2006), Bauer et al. (2006), (2008), Wang and Wang (2009), Ozkan and
Fassnacht and Koese (2006), Hwang and Kim (2007), Cristobal Koseler (2009), Ramayah et al. (2010).
et al. (2007), Loiacono et al. (2007), Rauyruen and Miller (2007),
Roses et al. (2009), McManus (2009), Park and Gretzel (2007),
Ding et al.(2010), Udo et al.(2010).
IT infrastructure Broadbent and Weill (1997),Weill et al. (2002), Weill and Vitale --------------
Service (2002), Hwang et al. (2002), Murakami et al. (2007), Fink and
Neumann (2007), King and Flor (2008), Bekkers (2009), Fink
and Neumann (2009), Sobol and Klein (2009), Bhatt et al.(2010),
Ramirez et al. (2010), Hicks et al. (2010) Jehangir (2011).
303
Table 2. List of hypotheses of study
H1 Hypotheses of Quality variables.

H1a Information quality directly affects usefulness.
H1b Information quality directly affects user satisfaction.
H1c Information quality indirectly affects net benefits via usefulness.
H1d Information quality indirectly affects indirectly net benefits via user satisfaction.
H1e System quality directly affects usefulness.
H1f System quality directly affects user satisfaction.
H1g System quality indirectly affects net benefits via usefulness.
H1h System quality indirectly affects net benefits via user satisfaction.
H2 Hypotheses of Usefulness and user satisfaction.
H2a Usefulness directly affects user satisfaction.
H2b User satisfaction directly affects net benefits.
H3 Hypotheses of Service delivery and IT infrastructure services.
H3a IT infrastructure directly affects information quality.
H3b IT infrastructure directly affects system quality.
H3c IT infrastructure directly affects usefulness.
H3d IT infrastructure directly affects user satisfaction.
H3e IT infrastructure directly affects service quality delivery.
H3f IT infrastructure indirectly affects net benefits via service quality delivery.
H3g Service quality delivery directly affects usefulness.
H3h Service quality delivery directly affects user satisfaction.
H3i Service quality delivery directly affects net benefits.
H3j Service quality delivery indirectly affects net benefits via usefulness.
H3k Service quality delivery indirectly affects net benefits via user satisfaction.
in its natural setting without imposing any a ering various stakeholders’ points of view about
priori understanding on it. Critical studies aim to the effect of factors on the success of e-learning
critique the status quo and remove contradictions systems. The model proposed in this chapter has
from organisations and society through the expo- been developed based on the relationships between
sure of deep-seated structural problems (Or- the constructs; therefore, there are 21 relationships
likowski & Baroudi, 1991). For this study, the to be tested. These relationships are formulated as
positivist paradigm is adopted to identify the fac- hypotheses. The hypotheses are listed in Table 2.
tors affecting e-learning system success and to
evaluate the e-learning system success in higher Study Contribution
education.
The degree of subjectivity versus objectivity is This study attempts to enhance the information
believed to be the central to the concept of ontology system and information technology field. More
(Cater-Steel, 2004). An objective view is taken in specifically, the contribution of this chapter is
this study by investigating the pertinent factors the development a model to assess e-learning
of e-learning system success, as well as consid- systems success. This new model will be ho-
304
listic because different perspectives have been investigated and addressed in this arena. The
considered in relation to technical, user attitude, lack of measurements to evaluate the success of
marketing and organisational. Another contribu- e-learning systems is considered to be the critical
tion is through the inclusion of the net benefits issues. The model proposed in this chapter has
factor. Perez-Mira (2010) states that “Individual been derived based on an extensive literature
impact per se is the most ambiguous to define … review. However, the validity and reliability of
Organizational impact does not have a clear and the proposed model should be tested. A set of
defined measurement variable” (p. 25). Differ- questionnaires have been developed and a survey
ent views of value are employed to measure the is planned at an Australian University. Three
net benefits of e-learningsystems dealing with stakeholder groups involved in e-learning will be
customer value, organizational value, and society surveyed: students, academic staff and ICT staff.
value. These three types of value provide a com- The survey data will be analysed to examine the
prehensive picture about the benefits generated relationships among the model constructs. Another
by e-learning systems. IT infrastructure has been issue related to e-learning systems is that of the
used in this model. This construct is deemed to role of e-learning systems in knowledge manage-
be substantial in creating competitive advantage, ment. More specifically, studies should identify the
decision making, and achieving organisational role of e-learning systems in creating, formatting,
goals. However, scant attention has been paid distributing, applying, and saving the knowledge.
to this construct in the e-learning arena. To this Moreover, different groups of stakeholders deal
author’s knowledge, this factor has not been used with e-learning systems. Therefore, those groups
previously as a construct to measure the success of stakeholders should be considered in the future
of e-learning systems. studies in the e-learning system field. Finally, tests
Service quality delivery is a key factor in evalu- of the proposed model should be conducted in
ating the success of an individual system and the non-educational organisations that use e-learning
performance of the information system department systems to identify the problems and shortfalls
(DeLone & McLean, 2003). The proposed model in the success of e-learning systems. The scope
has taken into account this important construct. of this research is limited to deal with e-learning
This factor is placed as a base of usefulness, user systems as a kind of information system. The
satisfaction and net benefits. In addition, it is a pedagogy perspective is not considered in this
mediation factor between IT infrastructure and chapter. The model proposed in this study has
net benefits. Another contribution of this model not been empirically tested. However, it is hoped
is that two kinds of effects have been considered that the proposed model will be tested and further
in the model: direct effect and indirect effect. The refined in the future with different institutions and
indirect effects are useful to identify the path of with different platforms of e-learning systems.
each factor reaching to the net benefits through
the other constructs.
CONCLUSION
FUTURE RESEARCH DIRECTIONS E-learning systems are believed to be the most

common recent IT applications in higher education
The e-learning systems field has received exten- institutions. Also, non-educational organisations
sive attention from the researchers, educational have been adopting these systems to train their
institutions, and non-educational institutions. employees. However, measuring information
However, there are some issues that need to be systems success is considered to be the main issue
305
in this field. In the context of e-learning systems, Aldag, R., & Power, D. (1986). An empiri-
this issue is believed to be more complicated be- cal assessment of computer-assisted decision
cause the e-learning term is used with different analysis. Decision Sciences, 17(4), 572–588.
points of view and from different stakeholders. doi:10.1111/j.1540-5915.1986.tb00243.x
The previous studies which dealt with this issue
Amit, R., & Zott, C. (2001). Value creation in
used four approaches: TAM, user satisfaction,
e-business. Strategic Management Journal, 22(6
e-learning quality, and DeLone and McLean
7), 493-520.
model. In spite of these attempts, there remain
arguments about the factors which are most ef- Arbaugh, J. (2000). How classroom environ-
fective in measuring e-learning system success. ment and student engagement affect learn-
Based on prior work, a proposed model has been ing in Internet-based MBA courses. Busi-
designed to evaluate e-learning system success. ness Communication Quarterly, 63(4), 9–26.
Four views have been considered in designing doi:10.1177/108056990006300402
this model: technical, attitude, marketing, and
Ardito, C., Costabile, M., Marsico, M., Lanzilotti,
organisational. The causality approach has been
R., Levialdi, S., & Roselli, T. (2006). An approach
adopted to show the causal relationships among
to usability evaluation of e-learning applications.
the constructs of the model.
Universal Access in the Information Society, 4(3),
270–283. doi:10.1007/s10209-005-0008-6
REFERENCES Atoji, Y., Koiso, T., & Nishida, S. (2002). Informa-
tion filtering for emergency management.
Abbad, M., Morris, D., & de Nahlik, C. (2009).
Looking under the bonnet: Factors affecting stu- Australian Flexible Learning Framework. (2009).
dent adoption of e-learning systems in Jordan. 2009 E-learning benchmarking survey. Brisban,
International Review of Research in Open and Australia: Author.
Distance Learning, 10(2), 1–25. Bailey, J., & Pearson, S. (1983). Development
Adeyinka, T., & Mutula, S. (2010). A proposed of a tool for measuring and analyzing computer
model for evaluating the success of WebCT course user satisfaction. Management Science, 29(5),
content management system. Computers in Hu- 530–545. doi:10.1287/mnsc.29.5.530
man Behavior, 26(6), 1795–1805. doi:10.1016/j. Baroudi, J., Olson, M., & Ives, B. (1986). An
chb.2010.07.007 empirical study of the impact of user involvement
Ahmed, H., Daim, T., & Basoglu, N. (2007). on system usage and information satisfaction.
Information Technology diffusion in higher edu- Communications of the ACM, 29(3), 232–238.
cation. Technology in Society, 29(4), 469–482. doi:10.1145/5666.5669
doi:10.1016/j.techsoc.2007.08.011 Bauer, H. H., Falk, T., & Hammerschmidt, M.
Ajzen, I., & Fishbein, M. (1969). The prediction (2006). E-transQual: A transaction process-based
of behavioral intentions in a choice situation. approach for capturing service quality in online
Journal of Experimental Social Psychology, 5(4), shopping. Journal of Business Research, 59,
400–416. doi:10.1016/0022-1031(69)90033-X 866–875. doi:10.1016/j.jbusres.2006.01.021
306
Bekkers, V. (2009). Flexible information infra- Cameron, K. (1978). Organizational effective-

structures in Dutch e-government collaboration ness: Its measuement and prediction in higher
arrangements: Experiences and policy implica- education. Doctoral dissertation, Yale.
tions. Government Information Quarterly, 26(1),
Cameron, K. (1981). Domains of organizational
60–68. doi:10.1016/j.giq.2007.09.010
effectiveness in colleges and universities. Acad-
Benbasat, I., & Dexter, A. (1986). An investiga- emy of Management Journal, 24(1), 25–47.
tion of the effectiveness of color and graphical doi:10.2307/255822
information presentation under varying time
Cameron, K. (1986). A study of organizational
constraints. Management Information Systems
effectiveness and its predictors. Management
Quarterly, 10(1), 59–83. doi:10.2307/248881
Science, 33(1), 87–112. doi:10.1287/mnsc.32.1.87
Berry, L., & Parasuraman, A. (1997). Listening
Cater-Steel, A. (2004). An evaluation of software
to the customer-the concept of a service-quality
development practice and assessment-based pro-
Information System. Sloan Management Review,
cess improvement in small software development
38(3), 65–76.
firms. Doctoral Dissertation, Griffith University.
Bharati, P., & Berg, D. (2005). Service quality
Chang, H. H., Wang, Y.-H., & Yang, W.-Y. (2009).
from the other side: Information Systems manage-
The impact of e-service quality, customer satisfac-
ment at Duquesne Light. International Journal
tion and loyalty on e-mrketing: Moderating effect
of Information Management, 25(4), 367–380.
of perceived value. Total Quality Management,
doi:10.1016/j.ijinfomgt.2005.04.008
20(4), 423–443. doi:10.1080/14783360902781923
Bhatt, G., Emdad, A., Roberts, N., & Grover, V.
Chiu, C., Hsu, M., Sun, S., Lin, T., & Sun, P. (2005).
(2010). Building and leveraging information in dy-
Usability, quality, value and e-learning continu-
namic environments: The role of IT infrastructure
ance decisions. Computers & Education, 45(4),
flexibility as enabler of organizational responsive-
399–416. doi:10.1016/j.compedu.2004.06.001
ness and competitive advantage. Information &
Management, 47(7-8), 314–349. doi:10.1016/j. Cho, P.-A. J. H. V. (2011). Comparing e-learning
im.2010.08.001 tools’ success: The case of instructor–student
interactive vs. self-paced tools. Computers &
Brigham, D. (2001). Converting student support
Education, 57(3), 2025–2038. doi:10.1016/j.
services to online delivery. International Review
compedu.2011.05.002
of Research in Open and Distance Learning,
1(2), 1–16. Cohen, E., & Nycz, M. (2006). Learning objects
and e-learning: An informing science perspec-
Broadbent, M., & Weill, P. (1997). Management
tive. Interdisciplinary Journal of Knowledge and
by maxim: How business and IT managers can
Learning Objects, 2, 23–34.
create IT infrastructures. Sloan Management
Review, 38, 77–92. Cox, J., & Dale, B. G. (2001). Service qual-
ity and e-commerce: An exploratory analy-
Byrd, T., Thrasher, E., Lang, T., & Davidson,
sis. Managing Service Quality, 11, 121–131.
N. (2006). A process-oriented perspective of IS
doi:10.1108/09604520110387257
success: Examining the impact of IS on opera-
tional cost. Omega, 34(5), 448–460. doi:10.1016/j.
omega.2005.01.012
307
Cristobal, E., Flavian, C., & Guinaliu, M. (2007). Doll, W., & Torkzadeh, G. (1988). The measure-
Perceived e-service quality (PeSQ) measurement ment of end-user computing satisfaction. Man-
validation and effects on consumer satisfaction and agement Information Systems Quarterly, 12(2),
website loyalty. Managing Service Quality, 17(3), 259–274. doi:10.2307/248851
317–340. doi:10.1108/09604520710744326
Etezadi-Amoli, J., & Farhoomand, A. (1996).
Daneshgar, F., & Toorn, V. (2009). E-learning in A structural model of end user computing sat-
workplace versus e-learning in higher education. isfaction and user performance. Information &
Australian Education Computing, 1(24), 16–22. Management, 30(2), 65–73. doi:10.1016/0378-
7206(95)00052-6
Davern, M., & Wilkin, C. (2010). Towards an
integrated view of IT value measurement. Interna- Fassancht, M., & Koese, I. (2006). Quality of
tional Journal of Accounting Information Systems, electronic service: Conceptualizing and testing a
11(1), 42–60. doi:10.1016/j.accinf.2009.12.005 hierarchical model. Journal of Service Research,
9(1), 19–37. doi:10.1177/1094670506289531
Davis, F., Bagozzi, R., & Warshaw, P. (1989). User
acceptance of computer technology: A comparison Fink, L., & Neumann, S. (2007). Gaining agility
of two theoretical models. Management Science, through IT personnel capabilities: The mediating
35(8), 982–1003. doi:10.1287/mnsc.35.8.982 role of IT infrastructure capabilities. Journal of
the Association for Information Systems, 8(8),
Davis, F. D. (1986). A technology acceptance
440–462.
model for empirically testing new end-user Infor-
mation System: Theory and result. Doctoral Dis- Fink, L., & Neumann, S. (2009). Exploring
sertation Massachusetts Institute of Technology. the perceived business value of the flexibility
enabled by Information Technology infrastruc-
ture. Information & Management, 46(2), 90–99.
tion System success: The quest for the dependent
doi:10.1016/j.im.2008.11.007
variable. Information Systems Research, 3(1),
60–95. doi:10.1287/isre.3.1.60 Frydenberg, J. (2002). Quality standards in
e-learning: A matrix of analysis. International
Review of Research in Open and Distance Learn-
DeLone and McLean model of Information System
ing, 3(2), 1–15.
success: A ten-year update. Journal of Manage-
ment Information Systems, 19(4), 9–30. Gable, G., Sedera, D., & Chan, T. (2008). Re-
conceptualizing Information System success: The
Ding, D. X., Hu, P. J.-H., & Sheng, O. R. L.
IS-impact measurement model. Journal of the As-
(2010). E-SELFQUAL: A scale for measuring
sociation for Information Systems, 9(7), 377–408.
online self-service quality. Journal of Business
Research, 64(5). Gelderman, M. (1998). The relation between user
satisfaction, usage of Information Systems and
Doig, G. (2002). A study of the factors that influ-
performance. Information & Management, 34(1),
ence the perception of strategic information value
11–18. doi:10.1016/S0378-7206(98)00044-5
in major UK based organisations. Doctoral Thesis
Loughborough University. Georgina, D., & Olson, M. (2008). Integration of
technology in higher education: A review of faculty
self-perceptions. The Internet and Higher Educa-
tion, 11(1), 1–8. doi:10.1016/j.iheduc.2007.11.002
308
Gorla, N., & Lin, S. (2010). Determinants of Hwang, H., Yeh, R., Chen, H., Jiang, J., & Klein,
software quality: A survey of Information Sys- G. (2002). IT investment strategy and IT infrastruc-
tems project managers. Information and Software ture services. The Review of Business Information
Technology, 52(6), 602–610. doi:10.1016/j. Systems, 6(2), 55–63.
infsof.2009.11.012
Hwang, Y., & Kim, D. J. (2007). Customer self-ser-
Gorla, N., Somers, T., & Wong, B. (2010). Or- vice system: The effect of perceived Web quality
ganizational impact of system quality, informa- with service contents on enjoyment, anxiety, and
tion quality, and service quality. The Journal of e-trust. Decision Support Systems, 43, 446–760.
Strategic Information Systems, 19(3), 207–228. doi:10.1016/j.dss.2006.12.008
doi:10.1016/j.jsis.2010.05.001
Ifinedo, P. (2006). Extending the Gable et al.
Hardaker, G., & Smith, D. (2002). E-learning enterprise systems success measurement model:
communities, virtual markets and knowledge cre- A preliminary study. Journal of Information
ation. European Business Review, 14(5), 342–350. Technology Management,17(1), 14-33.
doi:10.1108/09555340210444194
Iivari, J. (2005). An empirical test of the
Harris, M., & Weistroffer, H. (2009). A new look DeLone-McLean model of information system
at the relationship between user involvement in success. ACM SIGMIS Database, 36(2), 8–27.
systems development and system success. Com- doi:10.1145/1066149.1066152
munications of the Association for Information
Ives, B., Olson, M., & Baroudi, J. (1983). The
Systems, 24(1), 42.
measurement of user information satisfaction.
Hicks, B., Culley, S., McMahon, C., & Powell, Communications of the ACM, 26(10), 785–793.
P. (2010). Understanding Information Systems doi:10.1145/358413.358430
infrastructure in engineering SMEs: A case study.
James, L., Mulaik, S., & Brett, J. (1982). Causal
Journal of Engineering and Technology Manage-
analysis: Assumptions, models, and data. Beverly
ment, 27(1 2),57-73.
Hills, CA: Sage.
Hitt, L., & Brynjolfsson, E. (1996). Productiv-
Jehangir, M., Dominic, P., Naseebullah, N., &
ity, business profitability, and consumer surplus:
Khan, A. (2011). Towards digital economy: The
Three different measures of Information Tech-
development of ICT and e-commerce in Ma-
nology value. Management Information Systems
laysia. Modern Applied Science, 5(2), 171–178.
Quarterly, 20(2), 121–142. doi:10.2307/249475
doi:10.5539/mas.v5n2p171
Holsapple, C., & Lee Post, A. (2006). Defining,
Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online
assessing, and promoting e learning success:
university students’ satisfaction and persistence:
An Information Systems perspective. Decision
Examining perceived level of presence, useful-
Sciences Journal of Innovative Education, 4(1),
ness and ease of use as predictors in a structural
67–85. doi:10.1111/j.1540-4609.2006.00102.x
model. Computers & Education, 57, 1654–1664.
Hung, S. (2003). Expert versus novice use of the doi:10.1016/j.compedu.2011.02.008
executive support systems: An empirical study.
Kahn, B., Strong, D., & Wang, R. (2002). Infor-
mation quality benchmarks: Product and service
doi:10.1016/S0378-7206(02)00003-4
performance. Communications of the ACM, 45(4),
184–192. doi:10.1145/505248.506007
309
Kettinger, W., & Lee, C. (2005). Zones of toler- Larsen, T., Sørebø, A., & Sørebø, Ø. (2009). The
ance: Alternative scales for measuring Information role of task-technology fit as users’ motivation to
Systems service quality. Management Information continue information system use. Computers in
Systems Quarterly, 29(4), 607–623. Human Behavior, 25(3), 778–784. doi:10.1016/j.
chb.2009.02.006
King, W., & Flor, P. (2008). The development of
global IT infrastructure. Omega, 36(3), 486–504. Lau, A., & Tsui, E. (2009). Knowledge manage-
doi:10.1016/j.omega.2006.02.004 ment perspective on e-learning effectiveness.
Knowledge-Based Systems, 22(4), 324–325.
Koh, J., & Kim, Y. (2004). Knowledge sharing in
doi:10.1016/j.knosys.2009.02.014
virtual communities: an e-business perspective.
Expert Systems with Applications, 26(2), 155–166. Lee, B.-C., Yoon, J.-O., & Lee, I. (2009). Learn-
doi:10.1016/S0957-4174(03)00116-7 ers’ acceptance of e-learning in South Korea:
Theories and results. Computers & Education, 53,
Kuo, Y.-F., Wu, C.-M., & Deng, W.-J. (2009).
1320–1329. doi:10.1016/j.compedu.2009.06.014
The relationship amongservice quality, perceived
value, customer satisfaction, and post-purchase in- Lee, J., & Lee, W. (2008). The relationship of e-
tention in mobile value-added services. Computers learner’s self-regulatory efficacy and perception
in Human Behavior, 25, 887–896. doi:10.1016/j. of e-Learning environmental quality. Computers
chb.2009.03.003 in Human Behavior, 24(1), 32–47. doi:10.1016/j.
chb.2006.12.001
Lai, J. (2006). Assessment of employees’ per-
ceptions of service quality and satisfaction with Lee, Y., & Kozar, K. (2006). Investigating the
e-business. International Journal of Human- effect of website quality on e-business success:
Computer Studies, 64(9), 926–938. doi:10.1016/j. An analytic hierarchy process (AHP) approach.
ijhcs.2006.04.007 Decision Support Systems, 42(3), 1383–1401.
doi:10.1016/j.dss.2005.11.005
Landrum, H., & Prybutok, V. (2004). A service
quality and success model for the information Lee, Y., Kozar, K., & Larsen, K. (2003). The
service industry. European Journal of Operational technology acceptance model: Past, present, and
Research, 156(3), 628–642. doi:10.1016/S0377- future. Communications of the Association for
2217(03)00125-5 Information Systems (Volume 12, Article 50),
12(50), 725-780.
Landrum, H., Prybutok, V., & Zhang, X. (2007). A
comparison of Magal’s service quality instrument Lee, Y., Strong, D., Kahn, B., & Wang, R. (2002).
with SERVPERF. Information & Management, AIMQ: A methodology for information quality
44(1), 104–113. doi:10.1016/j.im.2006.11.002 assessment. Information & Management, 40(2),
133–146. doi:10.1016/S0378-7206(02)00043-5
Landrum, H., Prybutok, V. R., & Zhang, X.
(2010). The moderating effect of occupation on Lee-Post, A. (2009). E-learning success model:
the perception of information services quality and An Information Systems perspective. Electronic
success. Computers & Industrial Engineering, Journal of e-Learning, 7(1), 61-70.
58(1), 133–142.
Lehman, M. (1996). Laws of software evolution
revisited. Software Process Technology, 114,
108–124. doi:10.1007/BFb0017737
310
Liaw, S. (2008). Investigating students’ perceived Martinez-Torres, M., Toral Marín, S., Garcia, F.,
satisfaction, behavioral intention, and effective- Vazquez, S., Oliva, M., & Torres, T. (2008). A
ness of e-learning: A case study of the Blackboard technological acceptance of e-learning tools used
system. Computers & Education, 51(2), 864–873. in practical and laboratory teaching, according to
doi:10.1016/j.compedu.2007.09.005 the European higher education area. Behaviour
& Information Technology, 27(6), 495–505.
Liaw, S., Huang, H., & Chen, G. (2007). Surveying
doi:10.1080/01449290600958965
instructor and learner attitudes toward e-learning.
Computers & Education, 49(4), 1066–1080. McGill, T., & Klobas, J. (2009). A task-technology
doi:10.1016/j.compedu.2006.01.001 fit view of learning management system im-
pact. Computers & Education, 52(2), 496–508.
Lin, H. (2007). Measuring online learning sys-
doi:10.1016/j.compedu.2008.10.002
tems success: Applying the updated DeLone and
McLean model. Cyberpsychology & Behavior, McGorry, S. (2003). Measuring quality in
10(6), 817–820. doi:10.1089/cpb.2007.9948 online programs. The Internet and Higher
Education, 6(2), 159–177. doi:10.1016/S1096-
Liu, C., & Arnett, K. (2000). Exploring the
7516(03)00022-8
factors associated with website success in the
context of electronic commerce. Information & McKinney, V., & Yoon, K. (2002). The measure-
Management, 38(1), 23–33. doi:10.1016/S0378- ment of Web-customer satisfaction: An expecta-
7206(00)00049-5 tion and disconfirmation approach. Information
Systems Research, 13(3), 296–315. doi:10.1287/
Loiacono., e. T., Watson., R. T., & Goodhue., D.
isre.13.3.296.76
L. (2007). WebQual: An instrument for consumer
evaluation of websites. International Journal of McLoughlin, C., & Luca, J. (2001). Quality in
Electronic Commerce, 11(3), 51-87. online delivery: What does it mean for assessment
in e-learning environments. Paper presented at the
MacDonald, C., Stodel, E., Farres, L., Breithaupt,
Annual Conference of the Australasian Society
K., & Gabriel, M. (2001). The demand-driven
for Computer in Learning in Tertiary Education,
learning model: A framework for Web-based
Melbourne, Australia
learning. The Internet and Higher Education,
4(1), 9–30. doi:10.1016/S1096-7516(01)00045-8 McManus, J. (2009). A perspective om service
delivery. Management Services, Winter, 8-11.
MacDonald, C., & Thompson, T. (2005). Structure,
content, delivery, service, and outcomes: Quality McNamara, N., & Kirakowski, J. (2011).
e-learning in higher education. International Re- Measuring user-satisfaction with electronic
view of Research in Open and Distance Learning, consumer products: The consumer products
6(2), 1–25. questionnaire. International Journal of Human-
Computer Studies, 69(6), 375–386. doi:10.1016/j.
Mahmood, M. (1987). System development meth-
ijhcs.2011.01.005
ods-A comparative investigation. Management
Information Systems Quarterly, 11(3), 293–311. Melone, N. (1990). A theoretical assessment of
doi:10.2307/248674 the user-satisfaction construct in Information
Systems research. Management Science, 36(1),
76–91. doi:10.1287/mnsc.36.1.76
311
Miller, J., & Doyle, B. (1987). Measuring the Orlikowski, W. J., & Baroudi, J. J. (1991). Study-
effectiveness of computer-based Information Sys- ing Information Technology in organizations:
tems in the financial services sector. Management Research approaches and assumptions. Informa-
Information Systems Quarterly, 11(1), 107–124. tion Systems Research, 2(1), 1–28. doi:10.1287/
doi:10.2307/248832 isre.2.1.1
Mirani, R., & Lederer, A. (1998). An instrument Ozkan, S., & Koseler, R. (2009). Multi-dimen-
for assessing the organizational benefits of IS sional students’ evaluation of e-learning systems
projects. Decision Sciences, 29(4), 803–838. in the higher education context: An empirical
doi:10.1111/j.1540-5915.1998.tb00878.x investigation. Computers & Education, 53(4),
1285–1296. doi:10.1016/j.compedu.2009.06.011
Murakami, E., Saraiva, A., Ribeiro Junior, L.,
Cugnasca, C., Hirakawa, A., & Correa, P. (2007). Parasuraman, A., Zeithaml, V. A., & Berry, L. L.
An infrastructure for the development of distrib- (1988). SERVQUAL: A. Journal of Retailing,
uted service-oriented Information Systems for 64, 1.
precision agriculture. Computers and Electron-
Parasuraman, A., Zeithaml, V. A., & Mal-
ics in Agriculture, 58(1), 37–48. doi:10.1016/j.
hotra, A. (2005). E-S-QUAL: A multiple-item
compag.2006.12.010
scale for assessing electronic service quality.
Naveh, G., Tubin, D., & Pliskin, N. (2010). Student Journal of Service Research, 7(3), 213–233.
LMS use and satisfaction in academic institutions: doi:10.1177/1094670504271156
The organizational perspective. The Internet and
Park, Y. A., & Gretzel, U. (2007). Success factor
Higher Education, 13(3), 127–133. doi:10.1016/j.
for destination marketing websites: A qualitative
iheduc.2010.02.004
meta-analysis. Journal of Travel Research, 46,
Ngai, E., Poon, J., & Chan, Y. (2007). Empirical 46–63. doi:10.1177/0047287507302381
examination of the adoption of WebCT using
Pather, S. (2006). E-commerce Information
TAM. Computers & Education, 48(2), 250–267.
Systems (ECIS) success: A South African study.
doi:10.1016/j.compedu.2004.11.007
Doctral Dissertation, Cape Peninsula University
Nicolaou, A., & McKnight, D. (2006). Perceived of Technology
information quality in data exchanges: Effects
Pérez-Mira, B. (2010). Validity of DeLone and
on risk, trust, and intention to use. Information
McLean’s Model of Information Systems success
Systems Research, 17(4), 332. doi:10.1287/
at the website level of analysis. Doctral Disserta-
isre.1060.0103
tion, Lousiana State University.
Oliver, R. (2005). Quality assurance and e-
Pitt, L., Watson, R., & Kavan, C. (1995). Service
learning: Blue skies and pragmatism. Research
quality: A measure of Information Systems ef-
in Learning Technology, 13(3), 173–187.
fectiveness. Management Information Systems
doi:10.1080/09687760500376389
Quarterly, 19(2), 173–187. doi:10.2307/249687
Ong, C., & Lai, J. (2007). Measuring user satis-
Pituch, K., & Lee, Y. (2006). The influence of
faction with knowledge management systems:
system characteristics on e-learning use. Comput-
Scale development, purification, and initial test.
ers & Education, 47(2), 222–244. doi:10.1016/j.
Computers in Human Behavior, 23(3), 1329–1346.
compedu.2004.10.007
doi:10.1016/j.chb.2004.12.012
312
Price, R., Neiger, D., & Shanks, G. (2008). De- Roses, L., Hoppen, N., & Henrique, J. (2009).
veloping a measurement instrument for subjective Management of perceptions of Information
aspects of information quality. Communications Technology service quality. Journal of Busi-
of the Association for Information Systems, 22(1), ness Research, 62(9), 876–882. doi:10.1016/j.
49–74. jbusres.2008.10.005
Rai, A., Lang, S., & Welker, R. (2002). Assessing Rovai, A., & Downey, J. (2009). Why some dis-
the validity of IS success models: An empirical test tance education programs fail while others suc-
and theoretical analysis. Information Systems Re- ceed in a global environment. The Internet and
search, 13(1), 50–69. doi:10.1287/isre.13.1.50.96 Higher Education, 13(3), 141–147. doi:10.1016/j.
iheduc.2009.07.001
Ramayah, T., Ahmad, N., & Lo, M. (2010). The
role of quality factors in intention to continue Ryan, S., Harrison, D., & Schkade, L. (2002).
using an e-learning system in Malaysia. Procedia- Information-Technology investment decisions:
Social and Behavioral Sciences, 2(2), 5422–5426. When do costs and benefits in the social subsys-
doi:10.1016/j.sbspro.2010.03.885 tem matter? Journal of Management Information
Systems, 19(2), 85–127.
Ramirez, R., Melville, N., & Lawler, E. (2010).
Information Technology infrastructure, organiza- Sabherwal, R., Jeyaraj, A., & Chowa, C. (2006).
tional process redesign, and business value: An Information System success: Individual and or-
empirical analysis. Decision Support Systems, ganizational determinants. Management Science,
49(4), 417–429. doi:10.1016/j.dss.2010.05.003 52(12), 1849–1864. doi:10.1287/mnsc.1060.0583
Rauyruen, P., & Miller, K. (2007). Relationship Sagar, V. (2006). A digital library success model
quality as a predictor of B2B customer loyalty. for computer science student use of a meta-search
Journal of Business Research, 60(1), 21–31. system. Masters Thesis, Faculty of Virginia Poly-
doi:10.1016/j.jbusres.2005.11.006 technic Institue and State.
Reid, I. (2005). Quality assurance, open and Saleh, Y. M. (2002). Information System/Infor-
distance learning, and Australian universities. mation Technology success and evaluation: An
International Review of Research in Open and evaluation framework and general pracitioner
Distance Learning, 6(1), 1–11. model. PhD, University of Salford, Uk.
Roca, J., Chiu, C., & Martínez, F. (2006). Un- Seddon, P. (1997). A respecification and extension
derstanding e-learning continuance intention: An of the DeLone and McLean model of IS success.
extension of the technology acceptance model. In- Information Systems Research, 8(3), 240–253.
ternational Journal of Human-Computer Studies, doi:10.1287/isre.8.3.240
64(8), 683–696. doi:10.1016/j.ijhcs.2006.01.003
Sedera, D., & Tan, F. (2010). User satisfaction:
Roffe, I. (2002). E-learning: engage- An overarching measure of enterprise system
ment, enhancement and execution. Qual- success. Pacific Asia Conference on Information
ity Assurance in Education, 10(1), 40–50. System, 7-10 July, Bangkok, Thailand.
doi:10.1108/09684880210416102
Selim, H. (2007). Critical success factors for
Rogers, D. (2000). A paradigm shift: Technology e-learning acceptance: Confirmatory factor mod-
integration for higher education in the new mil- els. Computers & Education, 49(2), 396–413.
lennium. Educational Technology Review, 19–27. doi:10.1016/j.compedu.2005.09.004
313
Shang, S., & Seddon, P. (2002). Assessing and Sobol, M., & Klein, G. (2009). Relation of CIO
managing the benefits of enterprise systems: The background, IT infrastructure, and economic
business manager’s perspective. Information Sys- performance. Information & Management, 46(5),
tems Journal, 12(4), 271–299. doi:10.1046/j.1365- 271–278. doi:10.1016/j.im.2009.05.001
2575.2002.00132.x
Stav, J., Nielsen, K., Hansen-Nygård, G., &
Shannak, R. O. (1999). An interpretive approach Thorseth, T. (2010). Experiences obtained with
to analysing the role and value of Information integration of students response systems for iPod
Systems in Jordanian finacial organisations. Touch and iPhone into e-learning environments.
Doctor of philosophy. Aston University. Electronic Journal of e-Learning, 8(2), 179-90.
Shee, D., & Wang, Y. (2008). Multi-criteria Strother, J. (2002). An assessment of the effective-
evaluation of the Web-based e-learning system: ness of e-learning in corporate training programs.
A methodology based on learner satisfaction and International Review of Research in Open and
its applications. Computers & Education, 50(3), Distance Learning, 3(1), 1–17.
894–905. doi:10.1016/j.compedu.2006.09.005
Stvilia, B., Gasser, L., Twidale, M., & Smith,
Shun, C., & Yunjie, X. (2006). Effects of outcome, L. (2007). A framework for information quality
process and shopping enjoyment on online con- assessment. Journal of the American Society for
sumer behaviour. Electronic Commerce Research Information Science and Technology, 58(12),
and Applications, 5(4), 272–281. doi:10.1016/j. 1720–1733. doi:10.1002/asi.20652
elerap.2006.04.004
Stvilia, B., Twidale, M., Smith, L., & Gasser, L.
Shurville, S., O’Grady, T., & Mayall, P. (2008). Information quality work organization
(2008). Educational and institutional flex- in Wikipedia. Journal of the American Society
ibility of Australian educational software. for Information Science and Technology, 59(6),
Campus-Wide Information Systems, 25(2), 74–84. 983–1001. doi:10.1002/asi.20813
doi:10.1108/10650740810866576
Sun, P., Tsai, R., Finger, G., Chen, Y., & Yeh,
Skok, W., Kophamel, A., & Richardson, I. D. (2008). What drives a successful e-learning?
(2001). Diagnosing Information Systems success: An empirical investigation of the critical factors
Importance-performance maps in the health club influencing learner satisfaction. Computers &
industry. Information & Management, 38(7), Education, 50(4), 1183–1202. doi:10.1016/j.
409–419. doi:10.1016/S0378-7206(00)00076-8 compedu.2006.11.007
Smart, W. (2009). Information system success: Tait, P., & Vessey, I. (1988). The effect of user
Evaluation of a carbon accounting and seques- involvement on system success: A contingency
tration system. Doctoral Dissertation, Southern approach. Management Information Systems
Cross University. Quarterly, 12(1), 91–108. doi:10.2307/248809
Snitkin, S., & King, W. (1986). Determinants of Tallon, P., Kraemer, K., & Gurbaxani, V. (2000).
the effectiveness of personal decision support sys- Executives’ perceptions of the business value
tems. Information & Management, 10(2), 83–89. of information technology: A process-oriented
doi:10.1016/0378-7206(86)90050-9 approach. Journal of Management Information
Systems, 16(4), 145–173.
314
Turoff, M. (1999). Education, commerce, com- Wang, W., & Wang, C. (2009). An empirical study
munications: The era of competition. WebNet of instructor adoption of Web-based learning sys-
Journal, 1(1), 22–31. tems. Computers & Education, 53(3), 761–774.
doi:10.1016/j.compedu.2009.02.021
Tzeng, S., Chen, W., & Pai, F. (2008). Evaluating
the business value of RFID: Evidence from five Wang, Y. (2008). Assessing e commerce sys-
case studies. International Journal of Produc- tems success: A respecification and validation of
tion Economics, 112(2), 601–613. doi:10.1016/j. the DeLone and McLean model of IS success.
ijpe.2007.05.009 Information Systems Journal, 18(5), 529–557.
doi:10.1111/j.1365-2575.2007.00268.x
Udo, G., Bagchi, K., & Kirs, P. (2010). An assess-
ment of customers’e-service quality perception, Wang, Y., & Liao, Y. (2007). The conceptualiza-
satisfaction and intention. International Journal tion and measurement of m-commerce user sat-
of Information Management, 30(6), 481–492. isfaction. Computers in Human Behavior, 23(1),
doi:10.1016/j.ijinfomgt.2010.03.005 381–398. doi:10.1016/j.chb.2004.10.017
Van Dyke, T., Kappelman, L., & Prybutok, V. Wang, Y., Wang, H., & Shee, D. (2007). Measuring
(1997). Measuring Information Systems service e-learning systems success in an organizational
quality: Concerns on the use of the SERVQUAL context: Scale development and validation. Com-
questionnaire. Management Information Systems puters in Human Behavior, 23(4), 1792–1808.
Quarterly, 21(2), 195–208. doi:10.2307/249419 doi:10.1016/j.chb.2005.10.006
Venkatesh, V., & Bala, H. (2008). Technology Watson, R., Pitt, L., & Kavan, C. (1998). Measur-
acceptance model 3 and a research agenda on ing information systems service quality: Lessons
interventions. Decision Sciences, 39(2), 273–315. from two longitudinal case studies. Management
doi:10.1111/j.1540-5915.2008.00192.x Information Systems Quarterly, 22(1), 61–79.
doi:10.2307/249678
Venkatesh, V., & Davis, F. (2000). A theo-
retical extension of the technology acceptance Weill, P., Subramani, M., & Broadbent, M. (2002).
model: Four longitudinal field studies. Manage- Building IT infrastructure for strategic agility.
ment Science, 46(2), 186–204. doi:10.1287/ MIT Sloan School of Managment, Working Paper
mnsc.46.2.186.11926
Weill, P., & Vitale, M. (2002). What IT infra-
Venkatesh, V., Morris, M., & Davis, G., Davis. structure capabilities are needed to implement
(2003). User acceptance of Information Tech- e-business models. MIS Quarterly Executive,
nology: Toward a unified view. Management 1(1), 17–34.
Wilkin, C., & Castleman, T. (2003). Develop-
Wagner, N., Hassanein, K., & Head, M. (2008). ment of an instrument to evaluate the quality of
Who is responsible for e-learning success in higher delivered systems. Paper presented at the 36th
education? A stakeholders’ analysis. Analysis, Annual Hawaii International Conference, Big
11(3), 26–36. Island, Hawaii.
Wang, R., & Strong, D. (1996). Beyond accuracy: Wixom, B., & Todd, P. (2005). A theoretical
What data quality means to data consumers. integration of user satisfaction and technology
Journal of Management Information Systems, acceptance. Information Systems Research, 16(1),
12(4), 33. 85–102. doi:10.1287/isre.1050.0042
315
Wolfinbarger, M., & Gilly, M. C. (2003). eTailQ: Zeithaml, V., Parasuraman, A., & Malhotra, A.
dimensionlizing, measuring and predicting (2000). A conceptual framework for understanding
etail quality. Journal of Retailing, 79, 183–198. e-service quality: Implications for future reseach
doi:10.1016/S0022-4359(03)00034-4 and managerial practice. Cambridge, MA: Mar-
keting Science Institute.
Wu, J., Tennyson, R., & Hsia, T. (2010). A study of
student satisfaction in a blended e-learning system Zeithaml, V., Parasuraman, A., & Malhotra, A.
environment. Computers & Education, 55(1), (2002). An empirical examination of the service
155–164. doi:10.1016/j.compedu.2009.12.012 quality-value-loyalty chain in an electronic chan-
nel. Chapel Hill, NC: University of North Caroline.
Xiao, L., & Dasgupta, S. (2002). Measurement
of user satisfaction with Web-based Information Zeithaml, V. A. (2002). Service excellence in elec-
Systems: An empirical study. Paper presented at tronic channels. Managing Service Quality, 12(3),
the 2002 Americas Conference on Information 135–138. doi:10.1108/09604520210429187
Systems, Dallas, TX.
Zhi-Yong, G., Jian-Min, G., Fu-Min, C., & Lin,
Xiao, L., & Dasgupta, S. (Eds.). (2005). User G. (2009). Estimation method for manufacturing
satisfaction with Web portals: An empirical study. information quality based on material constraint
Hershey, PA: Idea Group Publishing. relationship. Computer Integrated Manufacturing
Systems, 1, 57–72.
Yang, Z., Cai, S., Zhou, Z., & Zhou, N. (2005).
Development and validation of an instrument to Zmud, R. (1983). The effectiveness of external
measure user perceived service quality of infor- information channels in facilitating innovation
mation presenting Web portals. Information & within software development groups. Manage-
Management, 42(4), 575–589. ment Information Systems Quarterly, 7(2), 43–58.
doi:10.2307/248912
Yang, Z., & Fang, X. (2004). Online service quality
dimensions and their relationship with satisfaction
A content analysis of customers reviews of securi-
ties brokerage services. International Journal of KEY TERMS AND DEFINITIONS
Service Industry Management, 15(3), 302–326.
doi:10.1108/09564230410540953 E-Learning Systems: Perform the teaching,
learning, and training activities by using the elec-
Yeh, R. K. J., & Teng, J. T. C. (2011). Extended tronic media to create value for different groups
conceptualization of perceived usefulness: Em- of stakeholders such as students, academic staff,
pirical test in the context of information system and ICT staff.
use continuance. Behaviour & Information E-Learning System Success: The ability of
Technology, 30(2), 1–16. doi:10.1080/014492 this system to provide the different stakeholders
9X.2010.517272 (students, academic staff, and ICT staff) with
Yoo, B., & Donthu, N. (2001). Developing a scale their needs to perform the required activities and
to measure the perceived quality of an Internet produce outputs.
shopping site (SITEQUAL). Quarterly Journal Information Quality: A main aspect used
of Electronic Commerce, 2(1), 31–47. to assess the value of information as delivered
to users. Quality includes characteristics such as
confidentiality, integrity, and access.
316
IT Infrastructure: The physical assets, hu- System Success: The ability of a system to
man resources, knowledge, hardware, software deliver the required outputs which can be used
and telecommunications networks which are to achieve the organisational activities and meet
employed to deliver IT services and applications. the users’ needs.
Service Quality: Delivering the services to User Satisfaction: The attitudes of information
customers taking into account matching these systems users which are generated as a result of
services to the customers’ needs and expectations. using IT systems in performing work.
317
318
Chapter 16
End-User Participation in
Health IT Development:
The EUPHIT Method
Anna Marie Balling Høstgaard

Aalborg University, Denmark
ABSTRACT
Despite there being extensive cumulative knowledge and many experiences about factors that contribute
to health Information Technology (HIT) success, lessons are yet to be learned as many HIT developments
still face a number of problems - many of them of an organizational nature. This chapter presents a new
method - the EUPHIT method – for studying and understanding one of the most crucial organizational
success factors in HIT development: end-user participation. The method was developed and used for
the first time throughout a research study of an EHR planning process in a Danish region. It has proved
effective in disclosing the interactions that occur between the different social groups involved in HIT
development, and in understanding the underlying reasons for these. This allows HIT project manage-
ment to explore new avenues during the development process in order to support, facilitate, and improve
real end-user participation.
DOI: 10.4018/978-1-4666-0170-3.ch016
End-User Participation in Health IT Development
INTRODUCTION developed and used for the first time throughout

a research study of an EHR planning process in
Today an extensive body of knowledge and many a Danish region. Thus, it was developed within
research experiences about factors that contribute and specifically for the healthcare sector.
to health information technology (HIT) success Initially, the chapter will provide the back-
exist (Ash, Stavre, & Kuperman, 2003; Berg, 2001; ground for developing the method. Subsequently,
Kaplan & Harris-Salamone, 2009; Kaye et al., the EUPHIT method will be described - including
2010; Van der Meijden, Tange, Troost, & Hasman, the theoretical framework on which it is based,
2003). Yet, lessons are still to be learned as many its target groups, and when and how to use it.
HIT implementations continue to face a number Next, a more detailed review of the method will
of problems, many of them of an organizational be provided through a thorough description of the
nature (Ash et al., 2003; Berg, 1999; Edmond- research study throughout which it was developed
son, 2003; Høstgaard & Nøhr, 2004; Lorenzi et and used for the first time. Finally, the strengths
al.,1997; Van der Meijden et al., 2001). One of and the limitations of the method will be discussed
the most crucial organizational success factors in as well as its relation to other approaches.
HIT development is end-user participation (Berg
& Winthereik, 2004; Høstgaard, 2009; Kensing et
al., 1996; Kensing & Blomberg, 1998; Kushniruk BACKGROUND
& Turner, 2011; Lorenzi & Riley, 1995; Avison &
Fitzgerald, 1995a). In this chapter, the concept is The fulfillment of the key objectives in health-
used in the sense of enabling end-users to exert care - patient safety and quality of treatment and
real influence in decision-making throughout the care - is totally dependent on an optimally func-
technological development process. Most methods tioning health information and communication
developed to support and facilitate this well-known (IC) infrastructure within the healthcare sector.
success factor have been developed for organiza- During recent years, many healthcare providers
tions in general and have focused on the design have made great efforts to replace the current
stage (Bødker et al., 2004; Kensing et al., 1996; health IC infrastructure - the paper-based health
Mumford & Weir, 1979) – so far, no method has record - with an electronic health record (EHR),
been developed for the healthcare sector specifi- because concurrent advances in information
cally or for the entire IT development process. technology indicate that this can resolve many of
This chapter presents a new method – the the problems, e.g. accessibility and data validity,
“EUPHIT method” (End-User Participation in associated with the paper-based health record.
HIT development) - for studying and understand- However, studies have shown that the intro-
ing end-user participation in HIT development. It duction of HIT – e.g. the EHR – besides solving
is aimed at HIT project management at all levels some problems, often brings with it a number of
– and others working in the field of health infor- new problems, including some of an organizational
mation technology success. The method presents nature (Ash et al., 2003; Berg, 1999; Edmondson,
a new methodological approach to collect more 2003; Høstgaard & Nøhr, 2004; Lorenzi et al.,
detailed information on social groups participating 1997; Van der Meijden et al., 2001). Research has
in HIT development and their interaction during also revealed not only pitfalls to be aware of but
the development. This allows HIT management also success factors to be met in order for an EHR
to explore new avenues during the HIT develop- implementation to be successful (Ash et al., 2003;
ment process in order to support and facilitate real Berg, 2001; Kaplan & Harris-Salamone, 2009;
end-user participation. The EUPHIT method was Kaye et al., 2010; Van der Meijden et al., 2003).
319
Some crucial – and well-known – organizational the early stages, they must possess both technical
success factors are that the clinicians must obtain and clinical knowledge. Otherwise they will be
positive clinical benefits (Ash et al., 2003; Rob- subject to the influence from other more powerful
ertson et al., 2010; Simonsen & Hertzum, 2006) groups (Avison et al., 1995b). Berg argues that
as a result of an EHR implementation and joint clinicians generally are incompetent as regards
ownership (Bødker et al., 2004; Johnson, 2006; speaking the language of “specifications” and
Lorenzi et al., 1997; Lorenzi et al.,1995; Pare, “configurations” and of expressing their “needs”
Sicotte & Jacques, 2006;) of the decisions made in the work situation. The skills needed for this
during the development process. A prerequisite for can only be developed over time by taking them
both is real participation in the EHR development on board early in the process and - according to
process – not pretended or symbolic – enabling cli- Avison et al. - by giving them the necessary skills
nicians to exert real influence in decision-making before they enter into the process (Berg, 2001).
(Berg, 2001; Bødker et al., 2004; Høstgaard, 2009;
Kensing & Blomberg, 1998; Kushniruk & Turner, Representation
2011; Lorenzi & Riley,1995; Lorenzi et al., 1997;
Pare, Sicotte & Jacques, 2006). Experience and Representation of all groups of clinicians is impor-
research have shown the following prerequisites tant in order to achieve joint ownership from all
for achieving real end-user participation: relevant participant-groups (Avison et al., 1998;
Mumford, 1983). Avison and Fitzgerald (1995b)
Early Involvement argue that in order to achieve real participation
of the end-users, “all major interests should be
Mumford classifies end-user participation in in- represented, including each section and function,
formation technology (IT) development into three grade, age group and so on” (p.357). However,
different types according to depth: consultative, Berg describes how trying to accommodate “ev-
representative and consensus participation. Ac- erybody’s” requests and “needs” can result in an
cording to Mumford, real influence in IT-devel- EHR process losing direction and momentum
opment is best achieved by using the consensus (Berg, 2001, p.149). Thus, for HIT working groups
type of participation, where end-users are involved to be able to reach agreements, not all wards and
throughout the technological development process all medical specialties can be represented. The goal
(Mumford & Weir, 1979; Mumford, 1983). In line is for upper management to set frames allowing
with this, some researchers state that early involve- end-users the best possible representation.
ment of clinicians (end-users) in HIT development
is essential because, as the technological develop- Workload Reduction
ment process proceeds, the scope of influence of
the clinicians and the possibility of changing the Kensing and colleagues have found that workload
decisions already made are progressively limited. reduction – meaning that colleagues take over part/
Therefore, by involving the clinicians already in all of the clinical duties for the clinicians involved
the planning stage, their influence on decision- (it does not involve extra payment) - is a precon-
making increases considerably. (Johnson, 2006; dition for real end-user participation at certain
Lorentzen, 1994; Müller, Kjær-Rasmussen & stages in HIT development. The authors argue
Nøhr, 1989; Protti, 2002). However, participation that “it is the responsibility of the management
in the early stages of a technological development to provide users with the time and possibilities to
process requires more than clinical knowledge participate in this process” (Kensing, Simonsen
Thus, for clinicians to make informed decisions in & Bødker, 1996, p.132).
320
Despite the existence of an extensive body THE EUPHIT METHOD

of knowledge about best practices and success
factors, HIT development still faces a number of Theoretical Framework
organizational problems (Black al., 2011; Robert-
son et al., 2010; Wears. & Berg, 2005). However, The EUPHIT method is based on the combina-
a search through the literature has revealed that tion and further development of the following
even though end-user participation is known to two theories:
be a crucial organizational success factor in HIT
development, no empirical studies of end-user • The Socio-Technical-Carrier-of-
involvement in the health information field have Technology theory (STCT)
been conducted in recent years, whereas a number • The Social Construction of Technology
of studies focusing on this issue have been com- theory (SCOT)
pleted in the medical device field (Money et al.,
2011; Nies & Pelayo, 2010; Shah & Robinson, The Socio-Technical-Carrier-of-
2006; Shah & Robinson, 2009; Shah, Farrow & Technology Theory (STCT)
Robinson, 2009). This calls for future HIT de-
velopment research to address how the existing The STCT theory was developed by researchers
knowledge within the health informatics field can at Aalborg University in the 1990s by combin-
be applied in practice. This view is supported by ing and further developing two theories: “the
Kaplan et al. who - based on a literature review socio-technical theory” and “the social-carrier-
and the results of an AMIA workshop - report that of-technology theory”.
many HIT systems are not successful “despite an In the socio-technical theory a broad concept
accumulation of best practices research identify- of technology is introduced, focusing on the mi-
ing success factors” (Kaplan & Harris-Salamone, cro level and the actor as opposed to the macro
2009, p. 291). These authors therefore call for fur- level. This technology concept is open-ended to
ther research into the development of new methods enable an understanding of the relation between
to support successful HIT development. In line technological and social change. According to
with this, Black et al. have found, in a recently the concept, technology embraces a combina-
published study, that the empirical evidence for tion of four constituents: technique (meaning the
the beneficial impact on the quality and safety technological object in question), knowledge,
of healthcare of most eHealth technologies often organization and product. These four constituents
is absent or modest, and the authors state that if are inseparable components of any technology.
the expected benefits of eHealth are to be real- A qualitative change in any one of the compo-
ized, substantial future research resources will be nents will eventually result in supplementary,
required (Black et al., 2011). compensatory, and/or retaliatory change in the
In this perspective, a new method for collecting other components. For a technology to be con-
more detailed information on end-user partici- sidered as such, it has to be applied and result in
pation in HIT development in order to support, a product, and for this to happen, actors have to
facilitate and improve real end-user participa- be active within each of the four components.
tion – and thus improving the adoption of HIT in The socio-technical theory is process-oriented
healthcare environments - will fill a void in the and focuses on the technological development
current body of knowledge about success factors process. As illustrated in Figure 1, this process
and best practices and, not least of all, on how to can be described in five stages. Within each of
apply these in practice. these stages, the actors make their selections
321
Figure 1. The five stages in the technological development process
based on possibilities and interests. This means the development process and, in this way, are
that at every stage a selection takes place (black sequentially linked to each other, they are named
circles) which leaves out many potential actions linked social-carriers-of-technology. In reality,
not chosen (grey circles) (Kuanda, 2003; Müller, the combined and linked social-carriers-of-
1991; Müller et al.,1988). technology are most often intertwined (Edquist
The focus of the social-carrier-of-technology & Edquist, 1979).
theory is the actors. It was developed to identify According to the STST theory - developed by
and study the relevant “social-carriers-of-tech- combining and further developing the two above
nology” involved in the development of new theories - the choices made by the actors include
technology. The concept refers to a group of ac- not only the technique but all four components in
tors, or a social entity, who chooses the new the socio-technological technology concept: tech-
technology and carry it forward towards the next nique, knowledge, organization and product. The
stage in a technological development process. STCT theory states that every qualitative change
According to this theory, this will happen only if – and the final outcome/product – in a technologi-
the “social-carriers-of-technology” have the nec- cal development process can be traced back to a
essary resources (interest, power, organization, change in the composition of the social carriers
information, access and knowledge) to change of technology and in the conditions necessary
their status from potential to actual carriers of the to achieve the status of an actual social-carrier-
technology in question. Often two or more social- of-technology (Høstgaard, 2009; Müller, 1990;
carrier-of-technology groups choose and imple- Müller et al., 1988; Nøhr & Bernstein, 1992).
ment the technology together. These groups are In the EUPHIT method, the STCT theory is
named combined social-carriers-of-technology. used as a framework to shed light on the interac-
When, on the other hand, the social groups in- tions that occur between the relevant social groups
volved carry the technology in different stages of
322
in Health IT development within a relatively short and the amount of power of each group involved
time frame by examining: (Bijker et al., 1989; Bruun Jensen et al., 2007).
The various social groups operate from dif-
• the vested interests of the different relevant ferent technological frames. A technological
social carrier groups of technology in rela- frame represents the purposes, the goals and the
tion to the EHR (interest); problem-solving processes attached to the tech-
• their options to carry through these inter- nological object – historically and nowadays – for
ests (power); the different social groups. The contents of this
• the degree of support from their profes- frame determine which meaning a particular social
sional association (organization); group attaches to the technological object, and it
• their opportunities to see and test the sys- also structures the interaction between the actors
tem in practice (access); and within each group. However, not all actors in a
• the amount of knowledge and informa- group might necessarily feel an equally strong
tion they had acquired about EHR both in sense of belonging to the group and hence to the
general and particularly regarding the dif- technological frame. Some may even be members
ferent options available (knowledge and of more than one social group. Actors who feel
information) a strong sense of belonging to one technological
frame are said to have a high degree of “inclu-
The Social Construction of sion” in the group attached to this frame. This
Technology Theory (SCOT) can be problematic in terms of innovation since
these actors may be inclined to act according to
According to the SCOT theory, every stage in experiences and methods to which they are ac-
a technological development process involves customed. Actors involved in several social groups
choices between various options. These choices most often experience a lower degree of inclusion
are made by the different social groups involved in the groups - and they are therefore often able
based on the different meaning (purpose, goal, to present new and different interpretations of
interests) they attach to the technological object and solutions to problems. Most often it will be
in question. This means that the design and the these actors who are involved in innovation and
further development of the technological object technological development processes (Bijker et
is a result of gradual and mutual debates between al., 1989; Bruun Jensen et al., 2007).
the different social groups, and that, besides rather According to the SCOT theory, studies of a
narrow purely professional considerations, social technological development process must start
factors (interests, power) determine which options out by identifying the relevant social groups.
are chosen. This results in a “multidirectional” It is obvious that end-users represent a relevant
development model in contrast to the linear model social group, but also less obvious groups must
often described in the history of technology (Bijker be identified. Once the relevant social groups are
et al., 2007). identified, the next step is to identify the different
During the technological development process problems that each social group attaches to the
different problems will appear as a result of the dif- technological object. For each problem it is then
ferent meanings each social group attaches to the possible to identify a range of solutions. This way
technological object. These problems are resolved of describing the development process clearly
through negotiations between the groups - the final demonstrates the interpretive flexibility of the
solution being highly dependent on the interests technological object. It also clearly demonstrates
that during a technological development process,
323
a number of conflicts will arise as different social in achieving the key objectives in healthcare -
groups express various technical needs, numer- patient safety and quality of treatment and care.
ous solutions to a problem, moral conflicts, etc.
(Bijker et al., 1989). When to Use the Method
In the EUPHIT method, the SCOT theory pro-
vides a framework for understanding the underly- HIT development can be divided into different
ing reasons for the different social groups’ vested stages (e.g., research, planning, implementation,
interests in the technological object in question operation). Depending on whether the same so-
and the power they have, or do not have, to bring cial groups (e.g., clinicians, IT-professionals and
forward these interests. The historical perspective administrators) are involved in decision-making
is, therefore, included in an attempt to uncover throughout all stages or different groups are in-
the meaning that has been attributed by the dif- volved in the different stages, the EUPHIT method
ferent relevant social groups to the technological should be completed one or several times. Thus, the
object from its outset until today; and in order to EUPHIT method should be completed each time
gain an understanding of the historical and cur- a new social group is involved in HIT develop-
rent interests and the balance of power between ment and if all actors within one social group are
the social groups. replaced (e.g., all clinicians in the clinicians-group
are replaced by other clinicians), because the in-
teractions between the different social groups are
OBJECTIVE OF THE STUDY highly dependent on which groups are involved
as well as the actors within each group.
The STST and the SCOT theories are both well
known. What is new is the act of combining them A Step-by-Step Presentation
and employing them within the healthcare sector, of the EUPHIT Method
which has not been done previously. HIT research
at Aalborg University is known for its focus on 1. Identify and define the technological object
how to apply new theories in practice. Thus, the in the local, present perspective
key objective for developing the new STST-SCOT 2. Conduct the technology-carrier analysis:
framework was to create a theoretical framework a. Identify the different relevant social
for a new methodological approach to collect more carrier groups of technology (the local,
detailed information on social groups participating present perspective)
in HIT development and, in doing so, allow HIT b. Conduct an analysis of:
management to explore new avenues during HIT ▪▪ the vested interests of the identi-
development in order to support and facilitate real fied social groups (interests)
end-user participation. ▪▪ their options to carry through
these interests (power);
Target Groups ▪▪ the degree of support from
their professional associations
The EUPHIT method is intended for HIT project (organization);
management at all levels as well as others working ▪▪ their opportunities to see and
in the field of information systems success. It is test the technological object in
important to stress that a precondition for using the practice (access);
method is that management at all levels recognize ▪▪ the amount of knowledge and
end-user participation as a crucial success factor information they have acquired
324
about the technological object explain how the actors’ actions are
both in general and particular- directed by the content in the techno-
ly regarding the different op- logical frame
tions available (knowledge and e. Synthesize the results in order to gain
information) an understanding of the results of the
c. Synthesize the results in order to technology carrier analysis
gain an insight into the social groups
participating in the HIT development
process If HIT project management (or others employing
3. Conduct the SCOT analysis: the method) only need more detailed information
a. Identify and define the technologi- on the social groups participating in the HIT de-
cal object in the national, historical velopment process and their interaction during the
perspective (often, a broader concept development – and not an understanding of the
of the technological object is identi- underlying reasons for these - the method could
fied compared to the local, present be reduced to steps 1 and 2.
perspective) The EUPHIT method was developed and
b. Identify the different relevant social used for the first time in a research study of an
groups (the national, historic perspec- EHR planning process in a Danish region. In the
tive). Study the problems that each following section a thorough description of this
group has attached to the technological study will provide a comprehensive presentation
object from its outset until today, as of how to use the method in practice.
well as the solutions to the problems
that each group has presented
c. Identify the meaning (objective, goal, A COMPREHENSIVE
interests) that each group attaches to PRESENTATION OF THE
the technological object (e.g., how EUPHIT METHOD: A RESEARCH
does the different groups understand STUDY OF THE EHR PLANNING
the technology? When, how and by PROCESS IN A DANISH REGION
whom was the technological develop-
ment process started? Which group’s Background for the Research Study
were/are the most dominant in the de-
velopment process? Which arguments In Denmark, where the healthcare system is public
have the different groups been using and financed by taxation and the five regions1
in the course of time to support their govern the hospitals, there has been extensive
own interests? Have they reached any discussion about the EHR in Danish hospitals.
agreements? How?). This step should Since 1999, the Danish national strategy for IT in
be repeated iteratively until all different the healthcare sector has required that all Danish
meanings attached to the technological counties implement an EHR (Sundhedsministeriet
object are identified (The Ministry of Health), 1999). In Denmark, the
d. Study the technological frame attached concept of “EHR” is defined as a platform with
to each social group (e.g., what is the different modules delivered by different vendors.
content? To which extent are the actors This means that the EHR in the five Danish re-
in each group included in relation to gions each have their own development strategy
the respective frame?). This step may and different platforms. The national strategy for
325
IT resulted in the County of North Jutland (CNJ) status from potential to actual social carriers2
developing an overall IT-strategy for the EHR of the EHR-technology?
development process (Nordjyllands Amt (The 2. Can the answer to the first research ques-
County of North Jutland), 2000). In October 2003, tion be understood by studying the different
the EHR planning process was initiated in the meanings that each of the relevant social
CNJ and, in March 2004, a local EHR working groups, (physicians, IT-professionals and
group was established with a view to producing administrators), attached/attaches to the
the requirement specifications for the EHR and EHR?
choosing between four possible systems.
The EHR working group was set up with clini- Methods for the Research Study
cians and IT-professionals; the clinicians compris-
ing nurses, medical secretaries and physicians. In accordance with the two research questions,
When it comes to achieving a successful imple- the research project was divided into two parts:
mentation of the EHR, all groups of clinicians a case study and a literature study.
are important end-user participation groups in
the development process. However, more studies Case Study
show that physicians are a very important group
because their acceptance is crucial as to whether The three Social-Carrier-of-Technology groups
or not the EHR is implemented in the intended (the STCT theory) were studied in the local, pres-
way (Mediq et al., 2005; Nøhr et al., 2004; Scott ent perspective with a focus on process-oriented
et al., 2005). In this perspective, a research study technology analysis. They consisted of physicians
of the EHR planning process in the CNJ was and IT-professionals in the EHR working group
conducted from 2003 to 2006, focusing specifi- and the ICT-board of the CNJ. The context was
cally on physicians as a relevant social group in the EHR planning process in the CNJ in Denmark,
EHR development. Besides the physicians, IT- and the research period lasted from October 2003
professionals and administrators were identified until April 2006.
as significant groups based on prior research and
own experience (Ash et al., 2003; Gregory, 2000; Literature Study
Mediq et al., 2005; Nøhr et al., 2004).
In the literature study the three relevant social
Objective of the Research Study groups (the SCOT theory) were studied in the
national, historical perspective. They consisted
The objective of the research was to develop a new of physicians and IT-professionals associated with
method for identifying, studying and understand- the Danish hospital sector and hospital managers
ing end-user participation in HIT development in and EHR decision-makers at county/regional level.
order to be able to support and facilitate it. The
objective was achieved by answering the follow- Data Collection and Analysis
ing two research questions:
Data in the case study (the local, present perspec-
1. Did the physicians have the necessary tive) were collected as presented in Table 1.
resources (interest, power, organization, Data were analysed using a Socio-Technical
information, access and knowledge) during “technology-carrier analysis” developed by re-
the EHR planning process to change their searchers at Aalborg University (Müller, 1990;
Müller et al.,., 1988; Müller et al., 1989). The
326
Table 1. Data collection in the case study
Data collection techniques Techniques used for this study

Interviews Semi structured interviews with 11 physicians (including all physicians in the EHR
working group), 2 IT-professionals, 3 Administrators. The interviews, lasting from 60
to 90 min., were recorded and transcribed.
Observation Participant observation at EHR working group meetings.
Insight into documents Minutes from meetings, Project plans, tender material etc.
Validation of data Data-triangulation, transcripts sent to interview-persons, written report sent to
participants
software program ATLAS (Muhr, 2004) was used • the history and the development of the
to analyse interviews. The focus was on the six Danish health record – paper-based as well
conditions required to achieve the status of an as electronic-based
actual social carrier of EHR (interest, power, • the history of Danish physicians, their in-
organization, information, access and knowledge) terest in the health record and the way in
and an “open mind” towards other themes. which they were/are organized
A new way of visualizing the results of the • the history of Danish IT-professionals and
technology-carrier analysis was developed as Danish administrators, their respective in-
shown in Table 2. A number of marks are given terests in the health record and the way in
to indicate whether the conditions to become an which they were/are organized
actual social carrier of the EHR are met for the
different social groups involved. Three marks Relevant textbooks were also used within
() indicate that the group achieved the status the areas mentioned above (Bruun Jensen, 2006;
of an accrual social carrier of EHR. Two marks Jacobsen & Larsen, 2007; Jespersen, 2004). Data
() indicate that the group was a very potential were analysed using SCOT analysis (Bijker et
social carrier – and one () indicates that the group al., 1989; Bruun Jensen.et al., 2007. The analysis
was a less potential carrier. It is important to stress included:
that the table only serves as a visualization of the
results; the number of marks given are based on • identifying the technological object and
the researchers thorough qualitative analysis of the relevant social groups and
each of the six conditions involved. • studying the different meaning that these
In the literature study (the national, historical groups attach to the technological object
perspective), data were collected through search- and their respective technological frames.
es in PubMed and Google Scholar. The research
strategy was as follows:
Table 2. Framework for visualization the technology-carrier analysis (The results of the analysis will
be added in the table body)
Social group Interest Power Organization Information Access Knowledge

Physicians
IT-professionals
IT-Board
327
Results of the Technology- hospitals outside Aalborg were not represented

Carrier Analysis in the EHR working group.
During the planning process, the EHR-project
Identification of the Technological management made no attempts to learn from the
Object and the Different Relevant experiences of management of EHR planning-
Social Carrier Groups of Technology processes in the other Danish counties – including
experiences about the need for workload reduc-
The technological object in the local present tions in one form or another. The EHR-project
perspective was one out of four possible EHR management’s main argument for not allowing
systems. The relevant social carrier groups com- workload reduction (paid by the county) was a
prised the physicians and IT-professionals in the general principle about leaving decisions about
EHR working group and the ICT-board of the workload reduction to the individual hospitals in
CNJ – focusing specifically on the physicians. the county. From the ICT-board’s point of view,
the local hospital’s incentive to pay for workload
The Planning Strategy: Factors reduction was the fact that the individual hospitals
Affecting All Six Conditions were allowed to keep any rationalization gains
from the EHR implementation process. This
Some factors related to the planning strategy af- principle meant that the responsibility for priori-
fected all six conditions. They are reported below. tizing costs for workload reduction, e.g. within
An EHR working group was established by treatment and care, was moved from county-level
the ICT-board to draw up requirement specifica- to hospital-level. Based on experiences regarding
tions and choose between four EHR systems. The the prerequisites for a successful implementation
members of the working group comprised physi- of EHR, a decision like this should be made at
cians, nurses, secretaries and IT-professionals. county-level to ensure the best possible implemen-
Two out of eight physicians in the EHR working tation across the county. The fact that no sharing
group were members from the start. The other six of past experiences at any level (strategic, tactical
joined the group 18 months later. and operational) took place before or during the
The – predominantly informal – planning EHR planning process could indicate that the
strategy for the EHR planning process meant that EHR-project management had underestimated the
there was no clinical workload reduction at all workload associated with the planning process.
during the process. Physicians were expected to
handle their full-time clinical duties while, at the Interest
same time, participating in the planning process.
They were expected to read a vast number of ICT- The three technology-carrier groups had different
technical papers and reports. Consequently, seven interests in the implementation of the EHR. The
out of eight physicians in the working group were physicians’ main interest was to ensure positive
senior physicians from Aalborg Hospital (the main clinical benefits. To achieve this, physicians found
hospital in the CNJ) who to some extent mastered it important that it was physicians who formu-
their clinical tasks. Only one junior physician – lated the medical demands in the requirement
from a hospital outside Aalborg – joined the group. specification. The IT-professionals’ main interests
He participated in two meetings whereupon he were concerned with optimizing administrative
had to leave the group due to a lack of time. It functions in the EHR, while the ICT-boards had
meant that junior physicians (more than half of a major interest in complying with the national
the physicians in the CNJ) and physicians from
328
requirements about implementing EHR in all of the EHR system. Now only technical differ-
Danish counties. ences were used to choose between the systems.
A new way of implementing the planning Thus, the physicians made their choice accord-
process was used in the CNJ: a “dialogue based ing to feelings, sensations – and advice from the
planning process”. It facilitated dialogue between IT-professionals. Thus, the final choice of EHR
the members of the working group and the four system was primarily made on the basis of the
vendors, and it made it possible for each of the interests of the IT-professionals and the ICT-board,
vendors to change system functions – e.g. the i.e. according to technical and economic criteria.
configuration of the user interface – during the
process. The physicians felt that this enabled them Power and Organization
to gain some insight into more aspects of the sys-
tems and to argue for their clinical demands and The Danish Medical Association was not involved
interests in the EHR. However, the physicians did in the planning process in any way. The orga-
not have sufficient time (no workload reduction) nizational support and encouragement that the
to go through all documents related to the process. physicians in the working group achieved during
Neither did they have the time to participate in the planning process came from colleagues at
all meetings in the EHR working group or meet- Aalborg Hospital. Compared to the physicians,
ings and workshops in the sub-groups established the IT-professionals had much more organizational
in relation to the process. As a consequence, support – and power. The fact that their organi-
IT-professionals with past clinical background zation was in charge of the project management
(non-physicians) developed most of the medical and – compared to the physicians – they were well
demands in the requirement specifications. This represented in both the ICT-board and all other
implies, according to Kensing et al., Brandt and groups related to the process; and that most of the
Simonsen et al., a great risk; namely that the documents prepared during the EHR process were
medical requirements do not reflect clinical prac- prepared by members of their organization, gave
tice because experience shows that professional them a significant amount of power. The director
knowledge has to be presented by professionals of IT-Health was also the EHR-project manager.
(Kensing et al., 1998; Brandt, 2006; Simonsen At the same time, the director and two heads of
& Hertzum, 2008). At the same time, it implies department at “IT-Health” were members of both
a great risk in that the physicians’ interests were the ICT-board and the EHR Steering Group. Thus,
not, or only partially, met during the process. the IT-professionals were part of the decision-
All four EHR systems were at a very early making authority as illustrated in Figure 2.
developmental stage at the time of the planning The physicians and the IT-professionals also
process. None of them were in operation in any used the power associated with professional
hospital ward - they only existed as the vendor’s knowledge to accommodate their own respective
schematic diagrams and early prototypes. This interests. However, the physicians’ clinical knowl-
fact taken together with the fact that all the edge turned out to be inadequate as a basis for
vendors changed the system functions related choosing between systems. Technical knowledge
to clinical work practices (e.g. configuration of was essential. The ICT-board had the power to
the user interface, the number of mouse clicks make final decisions – which they did when they
and integration with other systems) during the ordered all physicians to choose one of the systems
development process, resulted in the four systems despite the physicians’ request to wait until more
becoming almost alike. Therefore clinical related useful information, access and knowledge were
functions were abandoned as criteria for selection available. In fact, the physicians were powerless
329
Figure 2. Visualization of interlocking of persons in the EHR Working Group, the Steering Group and
the ICT-board respectively. Only members belonging to the three technology carrier groups are shown
when it came to the possibility of being able to between the systems. The fact that the four EHR
exert real influence on which system to choose. systems could not be tested in a real live setting
meant that clinical knowledge was not enough to
Access chose between the systems. Technical knowledge
became essential (see the knowledge aspect).
The four systems selected were not in operation in
any hospital ward and, therefore, it was not pos- Information
sible to test them in a “real live setting”. This was
critical, especially to the physicians as they were The members of the working group, who were
asked to specifically concentrate on the system involved from the start, were invited to different
functions related to clinical work practices, (e.g. arrangements as preparation for the different tasks
configuration of the user interface, the number they were asked to perform. The six physicians,
of mouse clicks and integration with other sys- who joined the working group 18 months after
tems). Recognizing this problem, the physicians it was established, did not receive any kind of
were offered to test four prototypes. None of the introduction or update on the work conducted
physicians were able to participate in all four tests so far - or any other kind of preparation for the
because of clinical duties (no workload reduction), task. This fact, together with the fact that clinical
which meant that they were unable to compare workload reduction was not possible – meaning
the four systems. Thus, the tests did not provide that the physicians had their full-time clinical work
the physician with a better basis for choosing beside the work in the EHR-working group, made
330
Table 3. Presentation of the results of the technology-carrier analysis
Social group Interest Power Organization Information Access Knowledge

Physicians     
IT-professionals     
IT-Board     
it impossible for the six last physician members of Results of the SCOT Analysis
the group to obtain the level of information neces-
sary to exert real influence in decision-making. Identification of the Technological Object and
the Social Groups
Knowledge In the historical, national perspective, the techno-
logical object was the health record – paper-based
Because the four systems became almost alike or electronic. The social groups comprised phy-
with respect to system functions related to clinical sicians and IT-professionals associated with the
work practices, these were given up as criteria for Danish hospital sector and hospital managers and
selection between systems. This meant that clini- EHR decision-makers at county/regional level.
cal knowledge became inadequate as a basis for
choosing between systems – technical knowledge Meanings and Technological Frames
was essential. Only one of the eight physicians
had the necessary technical knowledge for true Danish physicians form an inhomogeneous group
participation in the often very technical debates in with respect to their opinion on both problems and
the working group. The old saying: “knowledge solutions associated with the paper-based health
is power” proved to be very true. The reality record. Therefore they were divided into two
was: the more technical knowledge – the more groups by the researcher: “Clinical physicians”
power. The lack of technical knowledge made the and “Early adopters”. Throughout the history of
physicians incapable of exerting real influence the health record the two groups of physicians
on several important decisions made during the have had an internal struggle for the power and
process – e.g. the final choice of EHR system. the right to define the purpose of the health record.
Throughout history, the “Early adopters” have
Synthesis of the Technology advocated the introduction of new versions based
Carrier Analysis on visions of future clinical benefits – primary
(clinical work) as well as secondary (teaching
The results of the technology carrier analysis are and research). The “Clinical physicians” have
synthesized and visualized in Table 3. It shows tried to “arrest” this development, partly because
that none of the conditions necessary to achieve they have felt no need for new versions, partly
the status of an actual carrier of the EHR were in order to “keep up” with daily clinical prac-
met by the physicians during the EHR planning tice. However, in the long-term perspective the
process in the CNJ. Thus their status as potential “Clinical physicians” have always had to accept
carries of EHR remained unchanged. new – increasingly standardized – versions of the
health record.
In the short-term perspective, however, several
examples show that the “Clinical physicians”
331
have succeeded in curbing the development for Synthesis of the SCOT Analysis
the a time (Bossen, 2007; Bossen et al., 2005;
Scott et al., 2005). The two groups of physicians Historically, physicians and administrators have
have, however, recently taken a common external had, and still have, different understandings of the
position when it comes to the primary purpose of purpose of the patient record. Throughout history
patient data due to a growing interest in the use of the two groups have fought for their own under-
health data for secondary non-clinical purposes standing whenever new versions of the record have
(management and governance) among administra- been introduced. The planning process in the CNJ
tors and IT-professionals. This primary purpose is in Denmark was an example proving this. In this
clinical use in daily practice. Secondary clinical case, physicians fought for their clinical interest
use – and other secondary purposes – must not against administrators and IT-professionals – the
compromise the primary use. interests of the two latter groups to a large degree
Besides the internal power struggle about the being the same. All three groups fought for the
right to define the purpose of the health record, purpose of patient records to meet and support
Danish physicians have also faced an external their own interests and work areas.
struggle against administrators, which has taken
place ever since the outset of the Danish health Conclusion on the Research Study
record approximately 150 years ago. Historically,
administrators have shown a growing interest in The results showed absence of workload reduction
health records because better possibilities for as the main barrier to the physicians to achieve
extracting data for primary and secondary clini- true involvement in the process, and it affected
cal purposes at the same time also meant better all six conditions – interest, power, organization,
possibilities for extracting data for secondary non- information, access and knowledge – required
clinical purposes. The development of the EHR to obtain the status of actual social carriers of
was originally initiated by the “Early adopters” the EHR. The clinician’s role in the process was
with the internal control of treatment quality as reduced to clinical consultants informing about
its objective. This has, over the years, been over- physicians’ needs in the requirement specifications
taken by administrators with external control of and other documents. However, they were not
quality, efficiency and financing as its objective. able to fill this role completely due to a lack of
At the same time, the argument about patient time. Therefore, this work was, to a large extent,
safety, which for many years was used solely by handled by IT-professionals from “IT-Health”.
physicians as an argument for using patient data The answer to the first research question is that
for clinical purposes, is now also used by admin- none of the conditions required in order to obtain
istrators to legitimize the use of patient data for the status of actual social carriers of the EHR –
non-clinical purposes. In recent years, the “Early interest, power, organization, information, access
adopters” – and thus the medical profession – have and knowledge – were met by the physicians in
lost most of their influence on the development the EHR working group. Their initial status as
of the health record to the administrators, and the potential carriers of the EHR remained unchanged.
physicians’ power and right to define the purpose The answer to the second research question
of health records appears more diminished today is that the different meanings the three social
than ever before. groups attached to the EHR are rooted in an
inherited balance of power between physicians
and administrators specifically. Ever since the
outset of the Danish patient record, clinicians and
332
administrators have fought for the power and the computer systems on the employment satisfaction
right to define its purpose. So far, administrators of office clerks (Mumford & Ward, 1968), and
have won this battle and seem to have a stronger so far it has not been used within the healthcare
position today than ever. This inherited battle sector. The method has been criticized for being
of power was the major reason for the approach unrealistic and impossible to employ as it implies
chosen for the planning process in the CNJ. This that end-users without the necessary technical
battle is considered to be the reason for the condi- skills have a say in decision-making about very
tions made available to the EHR working group technical questions (Avison & Fitzgerald, 1995b).
by the ICT-board during the planning process; It has also been deemed naive due to the fact that
conditions that consequently reduced the role it presupposes that managers will accept that their
of physicians in the planning process in North right to manage is reduced (Avison & Fitzgerald,
Jutland to one of clinical consultants, rather than 1995b). The EUPHIT method may be subject to
real participants. similar criticism, as a precondition for using the
method is upper management recognizing a need
Solutions and Recommendations for involving end-users in HIT development in
order to achieve a successful result – and that
Relating the EUPHIT Method they accept the different implications of end-user
to Other Approaches participation in terms of time, costs and delegation
of decision-making to the end-users.
The EUPHIT is a step-by-step methodological In a newly published article, the ETHICS
approach, which has been developed to cover method is recognized as a valuable contribution
the entire IT development process and for the to support end-user participation in IT develop-
healthcare sector specifically. Only very few ment. The authors argue that the limitations are
other step-by-step approaches for supporting and outweighed by the strengths, and that Mumford’s
improving end-user participation in IT develop- research focuses on the fact that the discourse be-
ment can be found, and they are all developed for tween technology, organizations and staff within
the design stage and for other organizations than organizations is as important now as it was in the
the healthcare sector. late 1960s (Leitch & Warren, 2010). This is also
The “MUST method” is a method for participa- true for the EUPHIT method.
tory design developed by Danish researchers in
the 1990s (MUST is an acronym for theories of Strengths and Limitations
and methods for initial analysis and design activi- of the EUPHIT Method
ties). This method has been developed throughout
projects within a range of different organizations, The research study, throughout which the EUPHIT
e.g. film companies, R/D-lab, radio-station – ex- method was developed, was a qualitative study
cept for the healthcare sector. It is intended for using triangulation of different data collecting
supporting IT professionals responsible for design methods to validate the data. The analytical frame-
in IT development (Bødker et al., 2004; Kensing work supported and strengthened data analysis
et al., 1996) in contract to the EUPHIT method, as it was based on an integration of well-known
which is intended for HIT management at all levels. theories. In qualitative studies there is a risk that
Enid Mumford’s socio-technical “ETHICS the researcher has a predetermined opinion on
Method” (effective technical and human imple- the subject in question. There is also the risk that
mentation of computer based systems) was origi- the researcher is “seduced” by the position taken
nally developed to determine the impact of office by one group. To account for this, all activities
333
throughout the process were thoroughly described used at any stage as the importance of acquiring
(transparency). The fact that the researcher’s have knowledge about and understanding the social
participated throughout the planning process in interactions occurring between the social groups
all meetings and have had access to most docu- involved is equally important during all stages of
ments – electronic as well as paper-based – has HIT development.
made it possible to assess the truthfulness of, The most important limitation of/precondi-
e.g., time pressure and the amount of documents tion for using the EUPHIT method is that upper
related to the process. management, as well as management at all other
On the basis of the findings in the research levels in the healthcare organization, recognize
study it is concluded that the six conditions nec- end-user participation as a crucial success factor
essary to achieve the status of an actual carrier in HIT development – and thus for achieving the
of EHR (interest, power, organization, informa- key objectives in healthcare.
tion, access and knowledge) are good analytical Another important limitation/precondition is
markers for whether the preconditions for real that upper management are actively supporting
participation (early involvement, representation the method throughout the process. This means
and workload reduction) are met or not in HIT that they must provide the resources necessary
development. Through the thorough examination in terms of time and personnel for the method to
of the these conditions, the interactions between be employed, as the EUPHIT method could be
the social groups involved in HIT planning were rather time consuming, depending on which and
disclosed, while the SCOT analysis shed light on how many empirically data collection-methods
the underlying reasons for these. However, the are used.
condition access proved to be less relevant than the Using the method also implies that resources
other five in the early stages of HIT development. are available for the clinicians to achieve workload
In the early stages, different health information reduction and provide them with the technical
technologies most often only exist as schematic skills required in order to make informed decisions
diagrams or early prototypes, which makes access in the early stages of the development process. It
to testing the technology in real clinical settings also means that management at different levels
impossible. must accept that their right to manage is reduced
The EUPHIT method was developed through- because using the method implies that end-users
out a research study in a major Danish healthcare have a say in decision-making.
organization. However, research shows that the
interactions and the battle of power between dif-
ferent social groups are not specifically Danish CONCLUSION
phenomena, they can be found in HIT development
in general (Berg M. & Winthereik B.R., 2004). Real end-user participation is essential for the
Thus, a thorough study of the six conditions successful outcome of HIT development - and
needed for the end-users to achieve the status of thus for fulfilling the key objectives for healthcare.
an active carrier of a new technology, and thus for However, the interests of and the balance of power
them to be real participants, will also in a broader between the different social groups involved in
context provide HIT management with valuable HIT development are decisive for real end-user
information on the social groups participating participation.
in HIT development and the reasons for the The EUPHIT method presented in this chapter
variations. The method was developed and used is a new methodological approach, which has
during the planning stage. However, it can be proven effective for collecting more detailed
334
information on end-users participation in HIT- Berg, M. (2001). Implementing information

development by disclosing the social interactions systems in health care organizations: Myths and
that occur between the social groups involved. challenges. International Journal of Medical
Providing an understanding of the underlying Informatics, 64, 143–156. doi:10.1016/S1386-
reasons for the variations in the social groups 5056(01)00200-3
participating by uncovering the different meanings
Berg, M., & Winthereik, B. R. (2004). Waiting
attached to the HIT in question by the different
for Godot, episodes from the history of patient
relevant social groups has also proven effective.
records. In Berg, M. (Ed.), Health information
This new method for disclosing and understanding
management: integrating Information Technol-
the social interactions between social groups in
ogy in health care work (pp. 11–44). London,
HIT development provides an important tool for
UK: Routledge.
HIT project management at any level. It allows
new avenues to be explored during the process Bijker, W. E., Huges, T. P., & Pinch, T. (1989).
in order to support, facilitate and improve real The social construction of technological systems.
end-user participation. Massachusetts: MIT Press.
Black, A. D., Car, J., Pagliari, C., Anandan, C., &
Cresswell, K. (2011). The impact of eHealth on
REFERENCES
the quality and safety of health care: A system-
Ash, J., Stavre, Z., & Kuperman, G. J. (2003). atic overview. Public Library of Science (PLoS).
A consensus statement on considerations for a Medicine, 18(8), e1000387. doi:doi:10.1371/
successful CPOE implementation. Journal of the journal.pmed.1000387
American Medical Informatics Association, 10, Bødker, K., Kensing, F., & Simonsen, J. (2004).
229–234. doi:10.1197/jamia.M1204 Participatory IT design. Designing for business
Avison, D. E., & Fitzgerald, G. (1995a). Partici- and workplace realities. MIT Press.
pation. In Avison, D. E., & Fitzgerald, G. (Eds.), Bossen, C. (2007). Evaluation of a computerized
Information Systems development: Methodolo- problem-oriented medical record in a hospital de-
gies, techniques and tools (pp. 87–93). Berkshire, partment: Does it support daily clinical practice?
UK: The McGraw-Hills Companies. International Journal of Medical Informatics, 76,
Avison, D. E., & Fitzgerald, G. (1995b). Effective 592–600. doi:10.1016/j.ijmedinf.2006.04.007
technical and human implementation of computer- Bossen, C., Krüger, E., & Weeke, J. (2005).
based systems (ETHICS). In Avison, D. E., & Rapport om afprovning af Aarhus amts G-EPJ
Fitzgerald, G. (Eds.), Information Systems de- prototype (report on the testing of the G-EHR in
velopment: Methodologies, techniques and tools. the county of Aarhus). Aarhus, Denmark: Informa-
Berkshire, UK: The McGraw-Hills Companies. tions- og Medievidenskab, Aarhus Universitet.
Berg, M. (1999). Patient care Information Sys- Brandt, E. (2006). Designing exploratory design
tems and health care work: A sociotechnical games: A framework for participation in partici-
approach. International Journal of Medical patory design? Proceedings Participatory Design
Informatics, 55(2), 87–101. doi:10.1016/S1386- Conference.
5056(99)00011-8
335
Bruun Jensen, C. (2006). En performativ historie: Høstgaard, A. M., & Nøhr, C. (2004). Metode-
EJP som en fremtids-genererende anordning (A haandbog i forandringsparathed i forbindelse med
performative history: EHR as a future-generating udvikling og implementering af nye it-systemer in-
device). The Founding Meeting of the Danish denfor sundhedsvaesnet (handbook in methods for
Association for Science and Technology Studies change readiness in development and implemen-
(DASTS), DTU. tation of new IT-systems in healthcare). Aalborg,
Denmark: Virtual Center for Health Informatics.
Bruun Jensen, C., Lauritsen, P., & Olesen, F.
(2007). Introduktion til STS (Introduction to STS). Jacobsen, K., & Larsen, K. (2007). Ve og velfaerd;
København, Denmark: Hans Reitzels Forlag. laeger, sundhed og samfund gennem 200 år (woe
and welfare: Physicians, health and society dur-
Edmondson, A. C. (2003). Framing for learning:
ing 200 years). København, Denmark: Linhart
Lessons in successful technology implementation.
og Ringhof.
California Management Review, 45(2), 34–51.
Jespersen, H. (2004). Sygepleje og EPJ (Nursing
Edquist, C., & Edquist, O. (1979). Social car-
and EHR). Aarhus, Denmark: Institut for Sygeple-
riers of techniques for development. Jour-
jevidenskab. Denmark: Aarhus University.
nal of Peace Research, 16(4), 313–331.
doi:10.1177/002234337901600403 Johnson, C. W. (2006). Why did that happen? Ex-
ploring the proliferation of barely usable software
Gregory, J. (2000). Sorcerer’s apprentice: Cre-
in healthcare systems. Quality & Safety in Health
ating the electronic health record, re-inventing
Care, 15(supplement 1), i76–i81. doi:10.1136/
medical records and patient care. California:
qshc.2005.016105
University of San Diego.
Kaplan, B. (2000). Culture counts: How institu-
Heeks, R. D., Mundy, D., & Salazar, A. (1999).
tions values affect computer use. M.D. Computing,
Why health care Information Systems succeed
17, 23–26.
or fail: Information Systems for public sector
management working paper no. 9. Manchester, Kaplan, B., & Harris-Salamone, K. (2009). Health
UK: Institute for Development Policy and Man- IT success and failure: Recommendations from
agement, University of Manchester. literature and an AMIA workshop. Journal of the
American Medical Informatics Association, 16(3),
Høstgaard, A. M. (2009). Fryder forandring?
291–299. doi:10.1197/jamia.M2997
casestudie af EPJ udbudsprocessen i region nord
- belyst gennem en procesevaluering med fokus Kaye, R., Kokia, E., & Shalev, V., Ida,r D., &
på laegerne som sociale baerere af den elektroni- Chinitz, D. (2010). Barriers and success factors
ske patientjournal, EPJ (Change is the spice of in health Information Technology: A practitioner’s
life? process analysis case study of a region in perspective. Journal of Management & Marketing
Denmark) (Unpublished doctoral dissertation). in Healthcare, 3(2), 163–175. doi:10.1179/1753
Aalborg University, Denmark. 30310X12736577732764
Kensing, F., & Blomberg, J. (1998). Participa-
tory design: Issues and concerns. Computer
Supported Cooperative Work, 7, 167–185.
doi:10.1023/A:1008689307411
336
Kensing, F., Simonsen, J., & Bødker, K. (1996). Money, A. G., Barnett, J., Kuljis, J., Craven,
MUST - A method for participatory design. The M. P., Martin, J. L., & Young, T. (2011). The
Fourth Biennial Conference on Participatory role of the user within the medical device de-
Design, (pp. 129-140). sign and development process: Medical device
manufacturers’ perspectives. BioMedCentral:
Kuanda, J. (2003). Culture and technological
Medical Information and Decision Making, 11,
transformation in the south (Müller, J., Ed.).
15. doi:10.1186/1472-6947-11-15
Denmark: Samfundslitteratur Press.
Muhr, T. (2004). ATLAS the knowledge work-
Kushniruk, A., & Turner, P. (2011). Who’s us-
bench. V5.0 user’s guide and reference. Berlin,
ers? Participation and empowerment in socio-
Germany: ATLAS.ti.Scientific Software Devel-
technical approaches to health IT development.
opment GmbH.
International Perspectives in Health Informatics,
160, 280–285. Müller, J. (1990). Teknologivurdering og trovaer-
dighed (technology assessment and credibility). In
Leitch, S., & Warren, J. (2010). ETHICS: The
Cronberg T., & Friis D. (Eds.), Metoder i teknolo-
past, present and future of socio-technical systems
givurdering, erfaring og fornyelse (Methods in
design. In Tatnall, A. (Ed.), History of computing.
technology assessment - knowledge and experi-
Learning from the past (pp. 189–197). Victoria,
ences) (pp. 267-279). København, Denmark:
Australia: International Federation for Information
Blytmanns forlag.
Processing. doi:10.1007/978-3-642-15199-6_19
Müller, J. (1991). Hvad er teknologi? (What is
Lorentzen, A. (1994). Teknologi og udvikling i
technology?). In Rostgaard, M., Remmen, A., &
den nordjyske maskinindustri (technology and
Christensen, J. (Eds.), Samfundet i teknologien
development in engineering in the northern part
(pp. 27–47). Aalborg, Denmark: Aalborg Uni-
of Denmark). Aalborg, DK: Aalborg Univer-
versitetsforlag.
sitetsforlag.
Müller, J., Kjær-Rasmussen, J., & Nøhr, C.
Lorenzi, N., & Riley, R. T. (1995). Organizational
(1988). Perspektiver for EDB-teknologi i sygehu-
aspects of health informatics. Springer Verlag.
svaesenet (Perspectives for IT-technology in the
Lorenzi, N., Riley, R. T., Blyth, A. J. C., Southon, hospital sector). Aalborg, Denmark: Institut for
G., & Dixon, B. J. (1997). Antecedents of the Samfundsudvikling og Planlaegning, Aalborg
people and organizational aspects of medical Universitetscenter, Aalborg, DK.
informatics: Review of the literature. Journal of
Müller, J., Kjær-Rasmussen, J., & Nøhr, C. (1989).
the American Medical Association, 4(2), 79–93.
Proactive technology assessment as a tool in HIS
doi:10.1136/jamia.1997.0040079
development. In Barber, B. (Ed.), MEDINFO
Mediq, N. C., & Høstgaard, A.M. (2005). Bruger- 1989. Singapore.
erfaringer fra GEPKA projektet paa afdeling M,
Mumford, E. (1983). Designing human systems
Aarhus sygehus (User experience from the GEPKA
for new technology - The ETHICS method. Man-
project in Aarhus hospital). Aalborg University:
chester, UK: Manchester Business School.
Virtual Center for Health Informatics.
Mumford, E., & Ward, T. B. (1968). Computers:
Planning for people. UK: Batsford Limited.
337
Mumford, E., & Weir, M. (1979). Computer Robertson, A., Cresswell, K., Takian., A., et al.
systems in work design: The ETHICS method. (2010). Implementation and adaption of nation-
London, UK: Associated Business Press. wide electronic health records in secondary care
in England: Qualitative analysis of interim results
Nies, J., & Pelayo, S. (2010). From users involve-
from a prospective national evaluation. BMJ On-
ment to users’ needs understanding: A case study.
line Journal, 314.
International Journal of Medical Informatics,
79(4), 76–82. doi:10.1016/j.ijmedinf.2009.06.007 Scott, J. T., Rundall, T. G., Vogt, T. M., & Hsu, J.
(2005). Kaiser Permanente’s experience of imple-
Nøhr, C., & Bernstein, K. (1992). Implementation
menting an electronic medical record: A qualita-
and diffusion of medical informatics: The case of
tive study. BMJ Online Journal, 331(7528), 1–5.
electronic prescription communication. between
practitioners and pharmacies. Proceedings of Pri- Shah, S. G., Farrow, A., & Robinson, I. (2009).
mary Health Care Specialist Group of the British The representation of healthcare end users’ per-
Computer Society, (pp. 14-18). spectives by surrogates in healthcare decisions: A
literature review. Scandinavian Journal of Caring
Nøhr, C., Høstgaard, A. M., Botin, L., & Kjær-
Sciences, 23(4), 809–819. doi:10.1111/j.1471-
Andersen, S. (2004). Evaluering af GEPKA-
6712.2008.00674.x
projektet, delrapport 2, klinisk afprovning (evalu-
ation of the GEPKA-project, report 2, clinical Shah, S. G., & Robinson, I. (2006). User
testing). Aalborg University: Virtual Center of involvement in healthcare technology de-
Health Informatics. velopment and assessment: Structured lit-
erature review. International Journal of Health
Nordjyllands Amt (The County of North Jutland).
Care Quality Assurance, 19(6-7), 500–515.
(2000). IT-handlingsplan for sundhedsomraadet
doi:10.1108/09526860610687619
i Nordjyllands amt 2002-2004 (IT-plan for the
healthcare area in the county of north Jutland Shah, S. G., & Robinson, I. (2009). Developing
2002-2004). medical device technologies from users’ perspec-
tives: A theoretical framework for involving users
Pare, G., Sicotte, C., & Jacques, H. (2006). The
in the development process. International Journal
effects of creating psychological ownership on
of Technology Assessment in Health Care, 25(4),
physicians’ acceptance of clinical Information
514–521. doi:10.1017/S0266462309990328
Systems. Journal of the American Medical Infor-
matics Association, 13(2), 197–205. doi:10.1197/ Simonsen, J., & Hertzum, J. (2006). A regional
jamia.M1930 PD strategy for EPR systems: Evidence-based
IT development. Ninth Biennial Conference on
Protti, D. (2002). What can the American elec-
Participatory Design, 2, (pp. 125-128).
tronic health record (EHR) pioneers tell us about
what it takes to be successful? Health Manage- Simonsen, J., & Hertzum, J. (2008). Participative
ment Forum, 15(2), 33–35. doi:10.1016/S0840- design and the challenges of large-scale systems:
4704(10)60578-9 Extending the iterative PD approach. Proceedings
Participatory Design Conference.
338
Southon, G., Sauer, c., & Dampney, K. (1999). Barki, H., & Hartwick, J. (1994). Measuring user
Lessons form a failed information system initia- participation, user involvement, and user attitude.
tive: Issues for complex organizations. Interna- Management Information Systems Quarterly,
tional Journal of Medical Informatics, 55, 33–46. 18(1), 59–82. doi:10.2307/249610
doi:10.1016/S1386-5056(99)00018-0
Berg, M., & Aarts, J., & Van der Lei. (2003). ICT
Sundhedsministeriet (The Ministry of Health). in health care: Sociotechnical approaches. Meth-
(1999). National strategi for IT i sygehusvæsenet ods of Information in Medicine, 42(4), 297–301.
2000-2002 (national strategy for IT in the hos-
Bødker, K., & Pedersen, J. S. (1991). Workplace
pital sector 2000-2002). København, Denmark:
cultures: Looking at artifacts, symbols, and
Sundhedsstyrelsen.
practices. In Greenbaum, J., & Kyng, M. (Eds.),
Van der Meijden, M. J., Tange, H., Troost, J., & Design at work: Cooperative design of computer
Hasman, A. (2001). Development and implemen- systems (pp. 121–136). Chichester, UK: Lawrence
tation of an EPR: How to encourage the user. Erlbaum associates.
International Journal of Medical Informatics, 64,
Bødker, S. (1994). Creating conditions for par-
173–185. doi:10.1016/S1386-5056(01)00208-8
ticipation: Conflicts and resources in systems
Van der Meijden, M. J., Tange, H., Troost, J., & design. Proceedings of Computer Professionals
Hasman, A. (2003). Determinants of success of for Social Responsibility, (pp. 13-20).
inpatient clinical Information Systems: A literature
Coombs, C. R., Doherty, N. F., & Loan-Clarke,
review. Journal of the American Medical Infor-
J. (2002). The role of user ownership and posi-
matics Association, 10, 235–243. doi:10.1197/
tive user attitudes in the successful adoption of
jamia.M1094
information systems within NHS community trusts.
Vingtoft, S., Lippert, S., Bernstein, K., Bruun- Advanced topics in end user computing. Hershey,
Rasmussen, M., Kristensen, M., & Nøhr, C. PA: IGI Publishing.
(2000). EPJ-observatoriets statusrapport 2000
Dewan, N. A., & Lorenzi, N. (2000). Evaluating
(EHR status report 2000)
readiness and user acceptance. M.D. Computing,
Wears, R. L., & Berg, M. (2005). Computer 17(3), 50–52.
technology and clinical work. Still waiting for
Glasemann, M., Kanstrup, A. M., & Ryberg, T.
Godot. Journal of the American Medical As-
(2010). Design and exploration of a mobile game
sociation, 293(10), 1261–1263. doi:10.1001/
scenario in a diabetic youth camp. International
jama.293.10.1261
Conference Mobile Learning 2010, (pp. 132-140).
Granlien, M. F. (2009). Effect map - A participa-
tory tool for specifying formative effects to sup-
ADDITIONAL READING
port mutual adaptation. 32st Information Systems
Ash, J., Anderson, N. R., & Tarzy-Hornoch, P. Research Seminar in Scandinavia, (pp. 1-22).
(2008). People and organizational issues in re- Høstgaard, A. M., Bertelsen, P., & Nøhr, C. (2010).
search systems implementation. Journal of the How are clinicians involved in EHR planning? A
American Medical Informatics Association, 15(3), process analysis case study of a region in Denmark.
283–289. doi:10.1197/jamia.M2582 Medical Informatics Conference 2010 (MEDINFO
2010), (pp. 121-125).
339
Jensen, T. B., & Aanestad, M. (2007). Hospitality End-User Participation: In this chapter the
and hostility in hospitals: A case study of an EPR concept is used in the sense of enabling end-users to
adoption among surgeons. European Journal of exert real influence in decision-making throughout
Information Systems, 16, 672–680. doi:10.1057/ the technological development process.
palgrave.ejis.3000713 HIT (Health Information Technology): The
infrastructure through which clinical and admin-
Kyng, M. (1995). Making representation work.
istrative information is passed.
Communications of the ACM, 38(9), 46–55.
HIT Development: The process of developing
doi:10.1145/223248.223261
new health information technology from the first
Olesen, F., & Markussen, R. (2003). Reconfigured stage through to the last.
medication: Writing medicine in a sociotechnical Organizational Decision-Making: The pro-
practice (No. 11). The Johns Hopkins University cess of making decisions within an organization.
Press and the Society for Literature and Science. Social Carriers of Technology: A group of
actors, or a social entity who choose the new
Petersen, L. S., Bjørnes, C. D., & Bertelsen, P.
technology and carry it forward towards the next
(2010). Communication challenges in system
stage during the process.
development: Involvement of system developers
Workload Reduction: Colleagues taking
in small-scale IT-projects. In Nøhr, C., & Aarts,
over part/all of the clinical duties of the clinician
J. (Eds.), Information Technology in health care:
involved – it does not include extra payment.
Socio-technical approaches 2010 - From safe
systems to patient safety (pp. 72–78). IOS Press.
ENDNOTES

1
On 1 January 2007, 13 Danish counties
merged into five regions.
EHR: In Denmark it is defined as a platform 2
A group of actors, or a social entity, who
with different modules supplied by different choose the new technology and carry it for-
vendors. ward towards the next stage in the process.
End-User Involvement: In this chapter the
concept is used synonymous with “End-user
participation.”
340
341
Chapter 17
The Development of a
Model for Information
Systems Security Success
Kimberley D. Dunkerley
Nova Southeastern University, USA
Gurvirender Tejay
Nova Southeastern University, USA
ABSTRACT
Information security has received a great deal of attention from a number of researchers. However, there
has been little research aimed at understanding the dimensions critical for the success of organizational
information security programs. This chapter considers a large body of information security literature and
organizes the research based on their findings. This taxonomy is used to develop a parsimonious model
for information security success within organizations. Also, the utility of the proposed model within the
contexts of government and healthcare is considered.
INTRODUCTION and water purification and distribution. While

the IS assets are themselves valuable, the value
Modern society has become reliant on informa- of the information assets they process and store
tion systems (IS) and the information assets they must also be considered. Research has shown the
process and store. This reliance is quite impressive greatest damage often offered by a breach of IS
when presented with the facts: IS power much of security to be the loss of information resources
the modern infrastructure, including such vital and their resultant strategic advantages (CSIS,
assets as telecommunications, power produc- 2009; Pisello, 2004).
tion and distribution, oil and gas distribution, Even while organizations are beginning to
fully realize the value of their IS and information
DOI: 10.4018/978-1-4666-0170-3.ch017 assets, IS security incidents do occur, and with
The Development of a Model for Information Systems Security Success
potentially significant losses. These losses are of In order to effectively contribute towards this
both a monetary nature, as well as compromises greater understanding, this chapter will detail
to information assets. While it can be difficult the seminal works within IS success and discuss
to determine the full extent of losses suffered how they can be extended towards the IS security
through IS security exploits (Cavusoglu, Mishra, domain to create a model for IS security success
& Raghunathan, 2004; CSIS, 2008; Gaudin, within organizations. For this purpose, the key
2007), threats certainly have been realized at the research themes within the IS security domain will
corporate, state, and federal levels. The 2008 CSI be presented within the context of those theoreti-
Computer Crime and Security Survey found an cal foundations. Because we have chosen to adopt
average annual loss to corporate entities reported the model development approach used to develop
at $300,000 USD, with the potential to rise sharply seminal IS success models, our strategy differs
depending on the type and scope of the incident from the approach used to develop alternative
(CSI, 2008). The sheer losses borne by organiza- models of IS security success such as Kankan-
tions fundamentally underline the problems that halli, Teo, Tan, and Wei (2003). The review of
face corporate entities and nation-states as their IS security literature was conducted as outlined
infrastructures become increasingly technological by Webster and Watson (2002). Selected papers
and enemies become increasingly sophisticated (no time period constraint) from the IS field were
in their attack techniques. discovered using keywords. Those reference lists
To combat threats to their IS and information were then also examined to expand the reviewed
assets, a number of prescriptive IS security pro- research.
grams with varying content (e.g., U.S. department
of defense information assurance certification and
accreditation process (DIACAP), international IS SUCCESS MODELS
organization for standardization (ISO)/interna-
tional electrotechnical commission (IEC) 27000 The following section introduces three models
series, national institute of standards and technol- influential to the development of the IS security
ogy (NIST) SP 800-53) have been developed. success model: Shannon and Weaver (1949),
These programs differ widely in their content and Mason (1978), and DeLone and McLean (1992).
context, but they have one common aim: secur- DeLone and McLean (1992) is a seminal work
ing IS and their concomitant information assets. in IS success (Elleithy, 2008), and itself builds
However, even after the implementation of one upon the works of Shannon and Weaver (1949)
or more IS security programs, there is no assur- and Mason (1978). Understanding the evolution
ance that IS and information assets are secure. of these models and the lessons learned along
Arguably, in order to properly address the known the way is important for the development of an
and unknown threats towards an organization, a effective and efficient IS security success model.
thorough understanding of the factors constitut-
ing a successful IS security program is essential. Shannon and Weaver (1949)
Without this understanding, it could prove difficult
for organizations to accurately state the benefits Shannon and Weaver (1949) developed a model
the IS security program provides (DeLone & that identified three constructs involved in ef-
McLean, 2003; Gable, Sedera, & Chan, 2008) fective communications. The Shannon and
as well as ensure optimal resource utilization for Weaver model is considered a seminal model
future efforts (Zviran & Haga, 1999). for understanding communication and a seed for
communication studies (Fiske, 1982) as well as
342
information theory (Chandler, 2008). Within the In this sense, information flows through a series
Shannon and Weaver (1949) model, the technical of stages from its production through its use or
level of communications involves the accuracy consumption to its influence on individual and/or
and efficiency of the communication system that organizational performance” (DeLone & McLean,
produces information, or “how accurately can the 1992, p. 61).
message be transmitted?” Second, the semantic DeLone and McLean further suggested that
level relates to the success of the information in Mason’s work implied a need for separate success
conveying the intended meaning from sender to measures for each level of information. To this end,
receiver or “how precisely is the meaning ‘con- DeLone and McLean (1992) reviewed the seminal
veyed’?” Finally, the effectiveness level is the IS literature and collected empirical measures for
result the information actually has on the user’s each of the six dimensions of their model. DeLone
behavior or “how effectively does the received and McLean (1992) provided the foundational
meaning affect behaviour?” framework of IS success and extended the idea
The Shannon and Weaver (1949) model is of IS as distinct levels of communication into a
notable for its simplicity, its generality, and its measure of IS effectiveness, positing the concepts
quanitifiability, making it particularly attractive of system quality, information quality, use, user
to adoption within various academic disciplines. satisfaction, individual impact, and organizational
However this simplicity has led to studies accusing impact as composing success. Further research
the model of being misleadingly oversimplified extended the DeLone and McLean work into new
(Chandler, 2008). directions, incorporating new or refined concepts
of usefulness (Seddon & Kiew, 1996), use (Seddon,
Mason (1978) 1997), and service quality (Pitt et al., 2005) into
the mix. However, these works tend to venture into
Mason (1978) adapted the work by Shannon a more means-based than end-state approach, and
and Weaver in order to relate it specifically to do not provide parsimonious definitions for the
information systems. Mason’s model relabeled concept of IS success. Further, Petter, DeLone,
the variable of effectiveness as influence and and McLean (2008) described organizations
presented the influence level as a hierarchy of looking to define success beyond the traditional
events taking place at the receiving end of an financial measures, such as Return on Investment,
information system; these events are receipt of incorporating more intangible benefits provided
the information; influence of the information on by an IS. This concept of benefits, both tangible
the receiver; and the influence of the information and intangible, is important when discussing IS
on the performance of the system (Mason, 1978; security – an IS security program should ideally
Myers, Kappelman, & Prybutok, 1997). provide readily tangible benefits (e.g., limiting
IS breaches, limiting sensitive information spill-
DeLone and McLean (1992) age), but the intangible benefits (e.g., IS security
usability, enablement of every day operations)
DeLone and McLean (1992) built upon both the cannot be underestimated.
Shannon and Weaver and Mason works in an at-
tempt to reflect the interdependent, process nature
of IS success (Myers, Kappelman, & Prybutok,
1997) “The concept of levels of output from com-
munication theory demonstrates the serial nature
of information (i.e., a form of communication) …
343
TECHNICAL ASPECTS CIA has become a cornerstone of IS security; even

OF IS SECURITY while a host of other factors have been proposed,
such as responsibility, privacy (Krauss & Tipton,
Technical research has focused on providing 2002) trust, non-repudiation and authenticity
practical solutions for securing various levels (Ma, Johnston, & Pearson, 2008) are mentioned
of organizational assets. Technical means have alongside, the CIA triad is a mainstay. Most secure
historically been the cornerstone of IS security system development activities and organizational
research and have dominated the field to date security policies have been exclusively based the
(Chang & Ho, 2006). Numerous studies have pursuit of these core principles, and many studies
been conducted to find solutions for IS security are based on the assumption that achieving the
vulnerabilities through the application of techni- CIA of an organization’s assets are the essential
cal controls. objectives of an IS security program (Hayam &
Oz, 1993; Leiwo, Gamage, & Zheng, 1999; By-
Technical Security Controls rnes & Proctor, 2002; Rosenthal, 2002; Krauss
and Tipton, 2002; Moulton & Coles, 2003; Von
One stream of research has dealt with controls for Solms, 2005; Ma, Johnston, & Pearson, 2008).
securing data itself, often using different types of However, Anderson (2003) argues that IS se-
encryption (Kaliski, 1993; Blythe, 2008). Other curity cannot ultimately only mean CIA; in order
research has focused on the use of digital signa- to have a measure of security effectiveness there
tures to facilitate trusted transactions between must be generally accepted measurements for
parties (Rivest, 1978; Tompkins & Handley, CIA. To date, there are none. Further, Anderson
2003; Blythe, 2008). Software controls have urges for the establishment of metrics, not only
been considered, looking to secure infrastructures for CIA, but also for quantifying the value of
from the inside by strengthening the applications the IS security program and how the program
that are present on IS (Walsh, 1981; Shimeall provides a “well assured sense of assurance” to
& McDermott, 1999; August & Tunca, 2006). an organization (p. 313).
Finally, many studies have focused on numerous
hardware solutions, including intrusion detection IS Security Checklists and Standards
and firewalls (Denning, 1987; Daniels & Spafford,
1999; Vigna & Kemmerer, 1999; Axelsson, 2000; IS security evolved with a reliance on checklists
Frincke, 2000). Other researchers have developed and other “one-size-fits-all” measures aimed at
models – Straub’s computer security model, the finding the specific minimum control set that
Bell-LaPadula model, the Biba integrity model, will best protect information systems in general
the Clark-Wilson model, the Goguen-Mesequer (Baskerville, 1993). These measures have evolved
model, the Southerland model, and the Brewer- primarily from the government sector, which has
Nash model, to name a few – focusing on technical attempted to achieve IS security success through
methodologies (Foltz, Schwager, & Anderson, the use of regulated certification and accreditation
2008). requirements. The U.S. government, for example,
While technical research has occasionally has developed a series of control frameworks (e.g.,
interspersed external theories developed within department of defense information technology
the social, criminological, or behavioral domains, security certification and accreditation program
technical studies have largely focused on protect- (DITSCAP), department of defense information
ing infrastructure by facilitating the classic CIA assurance certification and accreditation program
(confidentiality, integrity, and availability) triad. (DIACAP)) that mandate sets of controls across
344
the board based on the integrity, availability, and security, developed through the University of
sensitivity requirements of the IS. These required Southern California (ISACA, 2009) and licensed
controls often involve lengthy risk assessments through the Information Systems Audit and Con-
and documentation creation along with stringent trol Association.
technical controls, attempting to secure the people, IS security maturity criteria have been a
processes, and technology that power the IS. burgeoning topic of research. Maturity criteria
Internal or third-party certification exercises are aim to offer an objective scale for classifying an
often required to validate the implementation. organization’s IS security posture, from low to
After successful accreditation is received, regu- high. These criteria not only offer a “goal” for
lar reporting requirements are the norm. Finally, improvement, but can be viewed as differentiating
the process is often required on a recurring basis an organization from its competitors based on a
dependent on the sensitivity of the IS. quantified assessment of successful IS security
Closely related to certification and accredita- control implementation. The system security
tion frameworks are IS governance and manage- engineering capability maturity model, a product
ment frameworks. While the context (Moulton & of research done at Carnegie Mellon University
Coles, 2003; Poore, 2006; Da Veiga & Eloff, 2007) has received the most attention (Siponen, 2005),
differs from governmental control structures, they but alternate models (Murine & Carpenter, 1984;
are very similar in their stated goals: IS security Stacey, 1996) do exist.
frameworks attempt to ensure the CIA of business Another segment of IS security research has
information coming into contact with the people, focused on the development of standards-based
processes, and technology that comprise everyday security, such as the generally accepted systems
business operations (Posthumus & von Solms, security principles (1999) and the ISO/IEC 27000
2004) through the use of mandated controls. IS series. These frameworks purport to best secure
security governance and management frameworks anything from an individual asset to an entire
have evolved from IT governance and manage- organization through implementation of a set of
ment frameworks, such as the Control Objective controls, usually covering people, processes, and
for information and related technology (COBIT) technology. While not as thoroughly studied as
and the information technology infrastructure purely technical controls (Straub & Welke, 1998),
library (ITIL). These frameworks have a very it has been argued that one of the most important
limited focus on IS security, with a small number IS security controls that can be introduced into
of controls considered alongside other areas like an organization is the IS security policy (Parker,
service desks. Purely IS security frameworks, 1998; Perry, 1985; Schweitzer, 1982; Warman,
such as the ISO/IEC 27001 (formerly the BS 1992; Hone & Eloff, 2002).
7799/ISO 17799), have included the Plan/Do/ Studies have suggested that most IS security
Check/Act cycle that evolved from IT governance decisions within small to medium-sized organi-
frameworks, implementing cycles to establish zations are directly guided by IS security policy
controls, implement controls, assess controls, (Briney & Prince, 2002) while large organizations
and refine based on the results of assessment. institutionalize IS security in their culture through
These standards have generally developed within the use of IS security policy (Baskerville & Si-
industry, but academia has begun development of ponen, 2002). The term “policy” itself has been
frameworks that attempt to apply cutting-edge argued, with Baskerville and Siponen (2002) divid-
theories for industry practice. An example is the ing research into two schools of thought: technical/
von Solms and von Solms (2006) direct-control computer security and non-technical/management
model, and the business model for information security. Technical security policy generally refers
345
to the automated implementation of management between IS security and usability (Baskerville,

policies (Sterne, 1991; Tanenbaum, 1992). This is 1992). Research has shown that policies must be as
confused by the term “policy” being used within flexible to the changing needs of the organization
technical contexts, such as group policies in a as the changes are fluid, facilitating rather than
directory environment, or access control policies inhibiting organizational emergence (Baskerville
on a firewall. Management policy, as defined & Siponen, 2002).
within Baskerville and Siponen (2002), is a high-
level plan embracing the organization’s general Risk Analysis
security goals and acceptable procedures. Within
this perspective, there has been significant study When attempting to create a balanced IS security
conducted as to the role of IS security policy program, research has shown that the security risks
within the organization. of the organization must be considered alongside
One area of IS security policy research has the organizational strategies, maximizing gain
worked to inform the development of effective IS while minimizing loss (Kotulic & Clark, 2004).
security policies, to include the determination of However, the evidence suggests that organiza-
proper scope and breadth (Hone & Eloff, 2002) as tions do not understand what the risks are to their
well as key internal and external influences during organization; Straub and Welke (1998) found that
development (Knapp, Morris, Marshall, & Byrd, organizations would be glad to accept theory-based
2009). Baskerville and Siponen (2002) suggested risk management programs if they understood the
a “meta-policy” or policy for the development of risks to their organization and understood how to
policy, as the best method for developing effective implement a risk management program.
IS security policies tailored to an organizational A key point within risk management research
perspective. is the assumption that a clear analysis and under-
Another area of IS security policy research has standing of risks is critical to achieving effective
focused on the human interaction with IS security security within an organization. The goal of risk
policy, from the senior management (Perry, 1985; analysis is to help management make informed
Gondek, 1989; Kwok & Longley, 1997; Dutta & decisions about investments and to develop risk
McCrohan, 2002; von Solms, 2005) to the end management and IS security policies (McFadz-
user (Belden, 1989; Warman, 1992; Herath & ean et al., 2006) while fully considering the
Rao, 2009). D’Arcy and Hovav (2007) suggested constraints in place inherent to the organization
that the human interaction has the potential to (Sun, Srivastava, & Mock, 2006). Risk analysis
completely invalidate the effectiveness of security methodologies generally measure risk either as
policies, but also that proper implementation of the probability of a negative outcome or a product
policies within an organization has the potential of the probability of a negative outcome due to a
to reduce misuse. threat and the probability that the corresponding
It has been argued that, in order for the IS control will fail to eliminate the threat (Perschke,
security program to be successful, IS security Karabin, & Brock, 1986; Guarro, 1987; Pickard,
policy must be aligned closely with the needs 1989). Within that context, many IS risk analysis
of the organization. Researchers have found that methodologies have been developed within both
organizations have unique needs that must be academia and industry. These include quantita-
considered (Schweitzer, 1982; Wood, 1999) and tive methods (e.g., expected value (EV) analysis
that a one-size-fits-all perspective is not ideal; (Perschke et al., 1986; Pickard, 1989; Rainer,
further, inflexibility in IS security policy can en- Snyder, & Carr, 1991), stochastic dominance
courage “developmental duality” or an imbalance approach (Post & Diltz, 1986), Livermore Risk
346
Analysis Methodology (LRAM) (Guarro, 1987)), only technical means are considered without un-
qualitative methods (e.g., scenario analysis, ques- derstanding the social, psychological, and cultural
tionnaire, and fuzzy metrics), and tool kits (e.g., variables, the results will lack usefulness (Webler,
information risk analysis methodologies (IRAM), Rakel, & Ross, 1992). All things considered, risk
the CCTA risk analysis and management method analysis as a process is still considered by many
(CRAMM) (Sun et al., 2006), national institutes researchers – even those critical to the current
of standards and technology (NIST) special pub- methods – to have merit for providing order and
lication (SP) 800-37, and the CERT operationally process, and helping to gain management support
critical threat, asset, and vulnerability evaluation for the IS security program (Baskerville, 1991).
(OCTAVE) method (Woody, 2006). In turn, risk Risk analysis is just one part of the risk man-
analysis methodologies have evolved from more agement process that has been considered; after
checklist-based approaches (McFadzean et al., threats have been assessed and risks determined,
2006) to include more sophisticated theories the management of those risks is key – with the
such as theory of belief function (e.g. Sun et al., ultimate goal maximizing gain for the organization
(2006) and finally, strategic conceptual modeling while minimizing loss (Kotulic & Clark, 2004).
approaches (Misra, Kumar, & Kumar, 2007). This is a long-term process with outputs that feed
Analysis of risks requires an understanding directly into a healthy governance model, with
of what threats are actually present. A number the expectation that senior management must
of works have attempted to classify threats into fully understand organizational risk in order to
various taxonomies, including categorical (Ches- incorporate it into the strategic outlook. To this
wick & Bellovin, 1994), results-based (Russell end, risk management is not a tool for reflection;
& Gangemi, 1991; Cohen, 1995), empirical risk management, when executed properly, di-
data-based (Neumann & Parker, 1989; Amoroso, rectly contributes to organizational effectiveness
1994), matrix-based (Perry and Wallich, 1984; (Jarvenpaa & Ives, 1991), should be proactive
Landwehr, Bull, McDermott, & Choi, 1994), and in nature (Kotulic & Clark, 2004), and should
process-based (Stallings, 1995). be integrated into business processes (Coles &
Risk analysis methodologies have been Moulton, 2003).
criticized for a variety of perceived weaknesses Management of risk involves a carefully cal-
(Dhillon & Backhouse, 2001), including simplicity culated application of selected controls. Straub
(Kokolakis, Demopoulos, & Byrd, 2000), lack of and Welke (1998) posited that, based on the extant
scientific approach (Baskerville, 1991), lack of research, controls would fall into one of four dis-
clarity (Alter & Sherer, 2004) and the random- tinct categories: deterrence, prevention, detection,
ness of actual attacks (Clements, 1977). Further, and recovery. Studies suggesting controls often
criticisms have been leveled at functionalist ap- use General Deterrence Theory to provide expla-
proaches to risk analysis, claiming that organiza- nations their proposed method will be effective
tions over-rely on risk analysis as a predictive at controlling risk. A number of methodologies
model without fully considering other factors, have been developed to facilitate risk manage-
such as user behavior (Baskerville, 1991; Beck, ment implementation including the business
1992). Again, human factors become key: research process information risk management (BPIRM)
has shown that humans take risks only through approach (Coles & Moulton, 2003; Moulton &
the creation of IS security incidents (Hitchings, Coles, 2003), the fundamental information risk
1995), but also through poor decision-making management (FIRM) methodology (Rycroft &
when an incident occurs (McGaughey, Snyder, Tully, 2007), and the perceived composite risk
& Carr, 1994). Again, research has shown that if (PCR) metric (Bodin, Gordon, & Loeb, 2008).
347
Table 1. Themes in technical security research
Research Theme Definition Seminal Literature

Technical security controls Technical solutions within the various levels of assets. Rivest, 1978; Denning, 1987; Kaliski, 1993
IS security checklists, stan- The automated implementation of management poli- Sterne, 1991; Tanenbaum, 1992; Baskerville
dards, and policy cies; or the organization’s general security goals and and Siponen, 2002
acceptable procedures.
Risk analysis The carefully calculated application of selected con- Straub and Welke, 1998; Kotulic & Clark, 2004
trols, maximizing gain while minimizing loss.
However, in spite of the research conducted, Internal Threats

the methodology followed, and the controls
implemented, researchers have argued that there One line of research deals with counterproductive
will always be a residual amount of risk to an IS, computer usage and malicious extremes, includ-
regardless of the actions taken or decisions made ing insider threats (Dhillon, 1999; Dhillon, 2001;
(Straub & Welke, 1998; Kotulic & Clark, 2004; Siponen, 2001; Trompeters & Eloff, 2001; Schultz,
Sun et al., 2006; Bodin et al., 2008). Risk manage- 2002; Whitman, 2003; Stanton et al., 2005; D’Arcy
ment, while unable to completely solve the issue & Hovav, 2009; Warkentin & Willison, 2009).
of risk, can provide a measure of mitigation. The While firms are shown to spend more resources
major themes within technical security research countering perceived threats originating from
are presented within Table 1. external forces (Stanton et al., 2005), it has been
argued that the insider threat is perhaps the most
significant threat an organization should consider
SEMANTIC ASPECTS OF IS (Dhillon, 1999; Whitman, 2003) and that the actual
SECURITY number of internally-led breaches suffered cannot
be known due to the vast amount of unreported
Research has suggested that IS security has an and unknown breaches (D’Arcy & Hovav, 2009).
almost “self canceling” phenomenon to consider: Much research centers around general deterrence
the user (Dodge et al., 2007). Lack of user com- theory-based approaches to solving insider threat
pliance has been directly tied to a decrease in IS (Tittle, 1980; Straub, 1990), theorizing that mis-
security effectiveness (Adams & Sasse, 1999). use will decrease as the disincentives increase.
Since the effectiveness of controls that are put into Further, studies have shown that increasing in-
place to protect information assets are constrained ternal knowledge of IS security policy and other
by behaviors of human agents who access, use, countermeasures, while not consistent, has the
administer, and maintain them (Vroom & von effect of decreasing misuse from certain internal
Solms, 2004; Stanton, Stam, Mastrangelo, & groups (D’Arcy & Hovav, 2009). However, policy
Jolton, 2005; Ma, Johnston, & Pearson, 2008), it alone cannot be relied upon as a deterrent; Sipo-
is clear that the user and their effect on IS secu- nen, Pahnila, and Mahmood (2010) found social
rity must be considered. Anderson (2001) even pressures, employee assessments of vulnerability,
argued that information insecurity is as much due and the immediacy of threats all play a part in
to “perverse incentives” as it is to weaknesses in determining employee intention to comply with
the technical infrastructure. IS security policy. To shed new light on internal
threats using fresh perspectives, criminological
348
theories have been introduced to the IS domain not enough; they must be convinced of it (Pfleeger
(Willison & Backhouse, 2006). & Pfleeger, 2003).
External Threats User Intention
Another group of research focuses on external Another research stream attempts to better under-
threats. These are the threats perhaps most closely stand the user’s intentions and their effect on IS
identified as hacking (Hausken, 2006) or competi- security. These studies often incorporate theories
tion (Gordon & Loeb, 2001). Stanton et al. (2005) such as the Theory of Planned Behavior or Theory
found that firms are more concerned with threats of Reasoned Action to explain user intention and
originating from external sources; this is perhaps its affect on subsequent behavior. Research sug-
due to the dominance of externally exploited gests that user intention is affected by a number
breaches reported in the press (Campbell et al., of external moderators, including organizational
2003). Studies have shown that the perception of commitment (Herath & Rao, 2009), codes of
external threats – hackers, viruses, and spyware – ethics (Harrington, 1996), cultural factors (Von
so dominate IS security programs that even secu- Solms, 2000; Dinev et al., 2008), and social pres-
rity policy development first considers protection sures (Siponen et al., 2010). Further studies have
against the external, rather than internal, threat discussed the link between the user’s awareness
(Smith, 1994). Research has typically considered and their intentions towards IS (Stanton et al.,
the external threat to be fixed and immutable 2005; Choi, Kim, Goo, & Whitmore, 2008) and
(Hausken, 2007), but it has been suggested that suggest that user awareness has a direct link to
external threats do consider the costs and benefits their intentions, which in turn affects behavior.
of attack based on information identified through These findings suggest that user intention—rang-
competitor analysis (Gordon & Loeb, 2001). ing from the malicious to the beneficial – might
be a key to understanding why users behave in
User Awareness the manner that they do, and the measures that
must be taken to prevent or protect against mali-
A second subset of user research focuses on the cious behavior. The major themes within semantic
awareness of users towards the systems—both security research are presented within Table 2.
the information system and its protective tech-
nologies—with which they interact (Thomson &
von Solms, 1998; Siponen, 2000). Research has EFFECTIVENESS ASPECTS OF IS
shown that awareness of technology is central SECURITY
to the formation of user attitudes, and in turn,
the user’s concern for IS security (Goodhue & The level of effectiveness relates to the benefits –
Straub, 1991; Dinev & Hu, 2007) but is difficult both tangible and intangible – that an organization
to characterize due to the individual nature of the enjoys. A number of researchers have discussed
variable itself (Dodge et al., 2007). For instance, various organizational benefits, such as financial
awareness towards the negative consequences value (e.g., Anderson, 2001; Gordon & Loeb,
of spyware has been found to motivate users to 2002, 2006) and balance (e.g., Anderson, 2003),
develop positive attitudes towards protective tech- and how the protection of IS and information as-
nologies and their intention to use them (Dinev sets supports and furthers the business as a whole.
et al., 2008). However, research suggests that
simply telling users to follow secure practices is
349
Table 2. Themes within semantic security research

Internal threats Threats originating from internal sources. Dhillon, 2001; Siponen, 2001
External threats Threats originating from external sources. Gordon & Loeb, 2001; Hausken, 2006
User awareness The awareness of users towards the systems—both the Thomson & von Solms, 1998; Siponen,
information system and its protective technologies—with 2000
which they interact.
User intention User intention towards IS and their protective technologies, Straub and Welke, 1998; Kotulic & Clark,
ranging from the malicious to the beneficial. 2004; Stanton et al., 2005
Financial Value A further factor that has been considered is

the true cost of IS insecurity; it has been found
Since most measures – technical, personnel, that there is a highly significant negative market
procedural – involve some level of resource al- reaction to IS security breaches, especially when
location, spending on IS security has become an involving unauthorized access to confidential data
important priority within organizations (Gordon (Campbell, Gordon, Loeb, and Zhou, 2003). This
& Loeb, 2002). Understanding how to create fact is further compounded for certain market seg-
financial value – investing the optimal amount ments, such as Internet-specific firms and software
in protecting assets and creating balance – is key. vendors, who are subjected to even greater risk of
A good deal of research has focused on deriving losses due to security breaches (Hovav & D’Arcy,
the optimal amount for an organization to invest 2003; Telang & Wattal, 2007). Further, research
in securing its IS and related assets (e.g., Millen, has shown that even unpublished breaches can
1992; Luotonen, 1993; McKnight, Solomon, have a devastating economic effect on a firm
Reagle, Carver, Johnson, Gerovac, et al., 1997; (Arora, Nandkumar, & Telang, 2006); organiza-
Finne, 1998; Jones, 1997; Buzzard, 1999; Hoo, tions cannot hide from their vulnerabilities and
2000; Anderson, 2001; Meadows, 2001; Gordon expect to come out unscathed. Incentives are not
& Loeb, 2002; Gordon & Loeb, 2006). This re- only monetary, however; multiple studies have
search stream has culminated in the development discussed the incentives created by regulations
of models for predicting this optimal amount of IS like Sarbanes Oxley (Gordon, Loeb, Lucyshyn,
security investment (e.g., Gordon & Loeb, 2002; & Sohail, 2006; Hausken, 2006). Within these
Hausken, 2006; Huang, Hu, & Behara, 2008). guidelines, there are often economic penalties
Finally, as large amounts of money are allotted for non-compliance. This is another economic
for IS security measures, stakeholders have begun factor that must be considered when quantifying
to demand results that they can see, in order to the cost of IS security.
justify these expenditures. Traditional economic It is important for stakeholders to stress the
ideas, such as return on investment (ROI), have value that IS security can create within an orga-
been discussed; with researchers attempting to nization; however, when attempting to explain
determine if tools such as return on security in- how an IS security program creates value for an
vestment (RoSI) (Gordon & Loeb, 2002) and the organization, one cannot focus solely on financial
analytic hierarchy process (AHP) (Bodin, Gordon, aspects. Research has discussed at length the socio-
& Loeb, 2005) would be useful for explaining IS organizational considerations involved with IS
security investments. security, such as effects on organizational culture,
and their value to the organization (Backhouse,
350
Table 3. Themes within effectiveness security research

Financial value Investing the optimal amount in protecting assets. Gordon & Loeb, 2002, 2006; Hausken,
2006
Security balance Organizational risks and controls in balance. Anderson, 2001, 2003; Gordon & Loeb,
2002; Huang, Hu, & Behara, 2008
Hsu, & Silva, 2006; Dhillon & Torkzadeh, 2006; upon the body of work by Shannon and Weaver
Drevin, Kruger, & Stegn, 2007; Dinev, Goo, Hu, (1949) and Mason (1978). This model, in turn,
& Nam, 2008). adopts the Shannon and Weaver (1949) taxonomy
of communication levels and applied it within the
IS security domain.
UNDERSTANDING IS SECURITY The streams of research within IS security dif-
SUCCESS fer in their nature, but there are definite themes
recurring throughout the domain. Early works
Anderson (2003) argued that many definitions of within IS security research were significantly
information systems security actually described the technical (Chang & Ho, 2006), and highly pre-
processes or concepts adopted towards IS security scriptive, with a heavy dependence on checklists
without defining the end state – again considering and methodological-based approaches aimed at
the means without the end. Many definitions of IS producing a “one-size-fits-all” method of protec-
security focus on the concepts of confidentiality, tion. This mindset, while long deemed inadequate
integrity, and availability, the so-called CIA Triad. by researchers (Baskerville & Siponen, 2002),
However, this research is based on the fundamental does continue to persist through a number of gov-
perspective presented by Anderson (2003) that CIA ernance and standards-based measures currently
(and other concepts such as non-repudiation and in use. However, the field as a whole is evolving
authenticity) are worthy goals to be achieved, but with the times; researchers have begun to expand
are not the fulfillment of an IS security program. into organizational optimization (e.g., Gordon
Further, Anderson (2003) suggested that a proper & Loeb, 2002, 2006), considering the concepts
definition of IS security must be attainable, while of balance (Anderson, 2003) and emergence
flexible to meet the organizational context in which (Baskerville & Siponen, 2002). These concepts
it is implemented. To that end, this research adopts weave through a considerable number of studies
the Anderson (2003) definition of IS security as a across the IS security domain. An example is the
well-informed sense of assurance that information economic research of Gordon and Loeb (2002,
risks and information security controls are in bal- 2006), promoting the idea of a balanced IS secu-
ance. This definition seeks to promote the crucial rity program as value maximization by optimal
concept of balance – determining the extent to investment into the protection of assets, a highly
which IS security can be implemented without context dependent concept. These concepts align
trampling the business objectives. This definition with Anderson’s (2003) definition of IS security
may differ widely between organizations, based as risks and controls being in balance.
on the sensitivity of the information assets and the Another major theme emerging within the IS
emergent nature of the organization itself. security domain is the importance of considering
The development of a model for predicting IS the human factor present within the IS. While the
security success within organizations will be built IS is not solely technical in nature, early research
351
streams within the IS security domain focused is transformed into knowledge as it is combined
primarily on achieving CIA and its fellow tenets with context and personalized into organizational
through technical methods. A paradigm shift in the “know-how” (Alavi & Leidner, 2001). Kane
domain occurred when the human aspect began to (2006) suggested that data, information, and
be considered. Da Veiga and Eloff (2007) described subsequent knowledge actually reside along a
IS security as having distinct phases of evolution: single continuum. This is an important concept
the first phase, purely technical in nature, heavily for the development of this IS security success
depended on the technological means of securing model, as the end benefits provided by knowl-
the IS. The second phase began when the realiza- edge synthesis and exploitation are impossible if
tion was made that the human element urgently the information itself proves to be irretrievable,
needed to be addressed. This realization has been unusable, or without value. For the purposes of
reflected within the body of research; the IS se- this chapter, the definition proposed by Liebenau
curity domain has moved from purely technical and Backhouse (1990) as information being data
considerations to the inclusion of a great number arranged in a meaningful way for some perceived
of studies focusing on socio-organizational areas purpose has been adopted. This definition honors
such as culture (Von Solms, 2000; Dinev et al., information as having more meaning than the raw
2008), user awareness (Thomson & von Solms, data itself, waiting for some unique application
1998; Siponen, 2000), and user behavior (Stanton into the organizational context.
et al., 2005; Choi et al., 2008; Siponen et al., 2010). Likewise, the information system has been
Clearly, as research has suggested a powerful studied at length. While using the term IS often
mitigating effect presented by the user (Adams & leads to assumptions of the technical specifica-
Sasse, 1999; Dodge, Carver, & Ferguson, 2007), tions itself (Lovata, 1987), the true domain actu-
it can be expected that the human factor will ally encompasses far more than this simple view.
continue to be an important consideration across Understanding what IS includes as a domain
the IS security domain. is fundamental to translating its role into the
organizational context. O’Donovan and Roode
(2002) suggested that IS cannot only be concerned
IS SECURITY SUCCESS MODEL with the exploitation of technology, but must
also consider the effects of technology and the
Key Concepts changes – both challenged and opportunities – it
can bring. This concept of “inherent contradic-
There have been many perspectives presented tions” – a technological emphasis imposed by
arguing what information is; Liebenau and computer specialists and a social, organizational,
Backhouse (1990) suggested that the regrettable and business emphasis imposed by specialists from
consequence of this trend was a bulk of definitions a number of domains – truly describes what the IS
only serving the narrow interests of those defin- means to an organization and the challenges that
ing them. Recent studies have placed information attempting to secure that environment can mean
into a framework alongside the concepts of data to an organization.
and knowledge, and often further including wis- Many researchers have attempted to define IS
dom. The data-information-knowledge hierarchy on the basis of levels representing these inher-
describes data as a set of signs formulated in a ent contradictions. Shannon and Weaver (1949)
structure and governed by formal rules (Tejay, described an information system as having three
Dhillon, & Chin, 2005), being processed and levels: technical, encompassing the production
interpreted to form information. This information of the information; semantic, or the success in
352
conveying the intended message to the receiver; seeks to promote the crucial concept of balance
and finally, effectiveness, or the level of effect – determining the extent to which IS security can
the information actually has on the receiver. be implemented without trampling the business
This reinforces the belief that technical consid- objectives. This definition may differ widely be-
erations of an IS must be considered alongside tween organizations, based on the sensitivity of
the socio-organizational considerations. We have the information assets and the emergent nature of
adopted the definition presented by Liebenau and the organization itself.
Backhouse (1990), which defines an information The concept of success has been studied count-
system as an aggregate of information handling less times within the IS domain, to many outcomes.
activities at the technical, formal and informal DeLone and McLean (1992) provided the founda-
levels of an organization. This definition provides tional framework of IS success and extended the
an effective representation of the various aspects idea of IS as distinct levels of communication into
of consideration within an IS: the technical level a measure of IS effectiveness, positing the concepts
includes the information technology present within of system quality, information quality, use, user
the organization, the technology often mistaken satisfaction, individual impact, and organizational
as the IS itself. The formal level includes the impact as composing success. Further research
bureaucracy, rules, and forms concerned with the extended the DeLone and McLean work into new
inter-organizational and the intra-organizational directions, incorporating new or refined concepts
use of information. Finally, the informal level of usefulness (Seddon & Kiew, 1996), use (Seddon,
includes the organizational sub-cultures where 1997), and service quality (Pitt et al., 2005) into
meanings are established, intentions understood, the mix. However, these works tend to venture into
beliefs, commitments, and responsibilities are a more means-based than end-state approach, and
made, altered, and discharged (Dhillon & Back- do not provide parsimonious definitions for the
house, 1994). concept of IS success. Further, Petter, DeLone,
Anderson (2003) argued that many definitions and McLean (2008) described organizations
of information systems security actually described looking to define success beyond the traditional
the processes or concepts adopted towards IS financial measures, such as Return on Investment,
security without defining the end state – again incorporating more intangible benefits provided
considering the means without the end. Many by an IS. This concept of benefits, both tangible
definitions of IS security focus on the concepts and intangible, is important when discussing IS
of confidentiality, integrity, and availability, the security – an IS security program should ideally
so-called CIA Triad. However, this work is based provide readily tangible benefits (e.g., limiting
on the fundamental perspective presented by An- IS breaches, limiting sensitive information spill-
derson (2003) that CIA (and other concepts such age), but the intangible benefits (e.g., IS security
as non-repudiation and authenticity) are worthy usability, enablement of every day operations)
goals to be achieved, but are not the fulfillment of cannot be underestimated.
an IS security program. Further, Anderson (2003) Based on this understanding, this research
suggested that a proper definition of IS security adopted the definition presented within Petter,
must be attainable, while flexible to meet the or- DeLone, and McLean (2008) as success being
ganizational context in which it is implemented. the tangible and intangible benefits a program
To that end, we have adopted the Anderson (2003) provides to an organization. Further, this line
definition of IS security as a well-informed sense of thought will be extended to synthesize the
of assurance that information risks and information concepts of IS security and success together into
security controls are in balance. This definition a definition of information systems security suc-
353
cess as the cumulative effect of the relationship ability and flexibility to deal with emergent situ-
between information systems experience and ations or exceptions. Quite simply, the IS security
user experience, manifested as the tangible and program should help the organization get the job
intangible benefits provided by an organization’s done. These concepts – securing the information
IS security program. and IS assets, and enabling operations – form
the basis of the Technical Level as described by
Model Development Shannon and Weaver.
As the extant literature has suggested, the user
There are notable findings from IS security is a key factor when considering how IS security
research that can readily be presented within actually fits into the organizational dynamic.
the framework of Shannon and Weaver (1949). Stanton (2005) found that the user dimension
First, the technical streams of IS security have was composed of two elements, the user’s inten-
demonstrated the importance of securing IS assets tions towards the IS and their knowledge. In the
through technical means, or ensuring that infor- context of organizational IS security, it becomes
mation “outputs fully and fairly reflect [inputs] extremely vital to understand the users intentions
and [processes] are complete, timely, authorized, towards IS and its protective technologies, on a
and accurate” (Boritz, 2004). This is crucial, as spectrum from positive to negative. Further, the
information is arguably the most important asset user’s knowledge describes user’s cognition of
a firm controls. However, the IS itself cannot be the organizational environment, both technical
forgotten; as suggested in Flowerday and Von and non-technical. Technical knowledge refers
Solms (2005), the information cannot be relied to the user’s savvy with technical assets. On the
upon if the “accuracy, completeness, timeliness, other hand, non-technical knowledge involves
validity, and processing methods” of the IS itself user’s awareness about security policy and vari-
has not been secured properly. Therefore, the ous controls to protect information assets and the
concepts of information security and IS security organizational mission. The user factors form the
go hand-in-hand towards a common end. basis of the semantic level, as discussed by Shan-
Further, the IS security program should not non and Weaver.
only create value to the organization through the Based on the previously discussed concepts
security provided to the information and IS assets, of balance, flexibility, and emergence, IS security
but also through operations enablement. Security success can – and often will – mean different
controls applied in a “one-size-fits-all” manner things within different organizational contexts.
have the ability to stifle not only the everyday For example, an organization that relies heavily
mission of an organization through laborious on highly classified information assets might
mechanisms or heavy-handed policies, but can be a find success to mean zero information spillages
real threat to the organizational objectives. Again, occurring within a given time period. Another
considering the Anderson (2003) definition of IS organization might determine success to mean
security, an organization must be well-informed compliance with regulatory guidance. Being well-
in regards to its strategic goals and how the IS informed provided the organizational leadership
security program must balance those goals with with the tools to determine what success means
the quantified threats to the organization. Consid- within the particular organizational context. Fur-
ering this, the IS security program operating in a ther, IS security success is not predetermined to
balanced manner should not hinder the strategic be positive in nature; it is a realistic possibility
goals of the organization; instead, the IS security that, based on the inputs provided, the level of
program should incorporate the concepts of us- success is found to be negative in nature. The
354
Table 4. Model dimensions for Shannon and Weaver (1949) communication levels
Communication Definition Model Dimensions Seminal Literature

Levels
Technical The accuracy and efficien- Information Integrity, Anderson (1972), Wiseman (1986), Denning
cy of the system producing IS Assurance, (1987), Muralidhar et al. (1995), Sandhu et al.
information. Operations Enablement (1996), Daniels & Spafford (1999)
Semantic The success the informa- User Intention towards Dhillon (2001), Siponen (2001), Trompeters &
tion has in conveying the Protective Technologies, Eloff (2001), Schultz (2002), Vroom & von Solms
intended meaning from User Security Knowl- (2004), Stanton et al. (2005), Dinev et al. (2008)
sender to receiver. edge
Effectiveness Effect of information on IS Security Success Anderson (2001), Gordon and Loeb (2002),
the user’s behavior. Campbell et al. (2003), Hovav and D’Arcy
(2003), Tanaka et al. (2005), Arora et al. (2006)
Figure 1. Model for IS security success
success of the IS security program itself forms semantic level factors (user intention towards pro-
the basis for the effectiveness level as detailed tective technologies and user security knowledge)
by Shannon and Weaver. lead to the effectiveness level proffered upon the
Understanding these factors presented within organization, IS Security Success. These factors
the structure provided by Shannon and Weaver and their respective Shannon and Weaver (1949)
(1949), the benefits provided through the dynamic communication levels are presented in Table 4.
relationship between the Technical Level fac- The resulting model is presented within Figure
tors (information integrity, information systems 1.
assurance, and operations enablement) and the
355
UTILITY OF IS SECURITY SUCCESS responsibilities. “[An] industrial age organization

MODEL makes a cyber-dependent government vulnerable
and inefficient” (CSIS Commission on Cyberse-
The following section will discuss the utility of curity, 2009, p. 51). Understanding the necessity
the IS security success model within two contexts: and efforts towards remediating this, the proposed
government and healthcare. model could help government organizations focus
their efforts on the dimensions of IS security that
Government Context are truly necessary to protect information assets.
IS and information assets are more important than Healthcare Context

ever within organizations, while at the same time,
the advent of online tools and services is having a Much like with e-government entities, there are
dramatic impact on business operations (DeLone challenges facing healthcare organizations as
& McLean, 2003). This impact is affecting change technology evolves. One of these is the need to
within all organizations, but perhaps most so balance IS security with the growing requirement
within the government sector. The advent of e- to provide e-enabled services to patients. The trend
Government, the intersection between traditional towards digital patient records and the increasing
government and the new age of delivering services exchange of information between patients, pro-
to citizens via the Internet, has necessitated a shift viders, and payers creates a difficult conundrum
in the paradigm of protecting our national interests for organizations more accustomed to dealing
while serving the citizen in a most effective man- with paper solutions (Appari & Johnson, 2010).
ner. These national interests include everything Study has shown that the emerging technolo-
from banking to electrical systems that power our gies embraced within the healthcare industry do
homes and businesses. not have values fundamentally conducive to IS
Government organizations have historically security concerns (Subramoniam & Sadi, 2010).
relied heavily on checklists and minimum-security Healthcare organizations are also lagging behind
requirements as their baseline to information in the effective implementation of IS security
systems security success (Baskerville, 1993). solutions; a 2009 survey found that nearly all
Furthermore, government entities are required healthcare organizations shared digital patient
to comply with a variety of regulatory require- records with governmental agencies, and one-third
ments, such as the federal information security of the organizations had experienced at least one
management act (FISMA) and the department identity theft (HIMSS, 2009). This indicates a need
of defense information assurance certification for more effective IS security controls.
and accreditation process (DIACAP). These Further, the same 2009 survey of healthcare
constraints add a layer of complexity not present organizations found that 85% of the respondents
within the standard commercial environment. had implemented some form of IS security con-
Regulatory mandates coupled with the changes trol, and only two-thirds of those who had were
presented by advent of the Internet age shape the measuring the success of the IS security controls
important, but often begrudged, role IS security (HIMSS, 2009). Because of the lack of a model for
plays within government sector. While industry IS security success within healthcare context, it is
has plunged forward with sophisticated techniques unclear that organizations understand what success
such as biometric access controls and intrusion means for their context. Also, without understand-
prevention devices, government organizations ing the factors that compose success, effective and
are often merely struggling to define roles and efficient measures cannot be developed.
356
The proposed IS security model could help REFERENCES

address these challenges by better focusing orga-
nizational efforts on the true path to IS security: Adams, A., & Sasse, M. A. (1999). Users are not
defining what success means to the organiza- the enemy. Communications of the ACM, 42(12),
tion, operationalizing the constructs that lead to 40–46. doi:10.1145/322796.322806
success for that organization, and understanding Alavi, M., & Leidner, D. E. (2001). Knowledge
the intervening power of users interacting with management systems: Conceptual foundations and
the information system. By doing these things, research issues. Management Information Systems
a healthcare organization can better assure the Quarterly, 25(1), 107–136. doi:10.2307/3250961
privacy and safety of the information assets that
will power healthcare transformation for years Alter, S., & Sherer, S. (2004). A general, but read-
to come. ily adaptable model of Information System risk.
Communications of the AIS, 14(1), 1–28.
Amoroso, E. G. (1994). Fundamentals of com-
CONCLUSION puter security technology. Upper Saddle River,
NJ: Prentice-Hall PTR.
In conclusion, significant literature has been
reviewed to better understand the fundamental Anderson, J. M. (2003). Why we need a new
elements of IS security success that have been definition of information security. Computers
proposed by researchers. From this literature, & Security, 22(4), 308. doi:10.1016/S0167-
six interrelated elements have emerged as core 4048(03)00407-3
dimensions of IS security success. From these
Anderson, R. (2001). Why information security is
findings a model for IS security success has been
hard – An economic perspective. In Proceedings
proposed. In addition, utility of the model within
of 17th Annual Computer Security Applications
the contexts of e-Government and healthcare
Conference, New Orleans, Louisiana, (pp. 10-14).
have been discussed. The model introduced in
this paper should be useful for future research Appari, A., & Johnson, M. E. (2010). Information
efforts in several ways. The security elements of security and privacy in healthcare: Current state
the model have never been studied as interrelated of research. International Journal of Internet and
dimensions of an encompassing model. Research Enterprise Management, 6(4), 279. doi:10.1504/
on these relationships would provide a better IJIEM.2010.035624
understanding of the interaction between security
Arora, A., Nandkumar, A., & Telang, R. (2006).
structure within an organization and an individual
Does information security attack frequency in-
utilizing the IS and its protective technologies.
crease with vulnerability disclosure? An empirical
In addition, implementation of the model within
analysis. Information Systems Frontiers, 8(5),
various organizational contexts would provide a
350–362. doi:10.1007/s10796-006-9012-5
better understanding of how IS security varies
within these contexts. Assistant Secretary of Defense for Networks and
Information Integration/Department of Defense
Chief Information Officer. (2010). DoD 8570.01-
M, information assurance workforce improvement
program, incorporating change 2, April 20, 2010.
Washington, DC: Department of Defense.
357
August, T., & Tunca, T. I. (2006). Network Bodin, L., Gordon, L. A., & Loeb, M. P.
software security and user incentives. Manage- (2005). Evaluating information security invest-
ment Science, 52(11), 1703–1721. doi:10.1287/ ments using the analytic hierarchy process.
mnsc.1060.0568 Communications of the ACM, 48(2), 78–83.
doi:10.1145/1042091.1042094
Axelsson, S. (2000). The base-rate fallacy and the
difficulty of intrusion detection. ACM Transac- Bodin, L. D., Gordon, L. A., & Loeb, M. P.
tions on Information and System Security, 3(3), (2008). Information security and risk manage-
186–205. doi:10.1145/357830.357849 ment. Communications of the ACM, 51(4), 64–68.
doi:10.1145/1330311.1330325
Backhouse, J., Hsu, C. W., & Silva, L. (2006).
Circuits of power in creating de jure standards: Boritz, J. E. (2004). Information integrity for the
Shaping an international Information Systems information era. Rolling Meadows, IL: ISACA.
security standard. Management Information
Briney, A., & Prince, F. (2002). Does size matter?
Systems Quarterly, 30, 413–438.
Retrieved from www.infosecuritymag.com/2002/
Baskerville, R. (1991). Risk analysis: An interpre- sep/2002survey.pdf
tive feasibility tool in justifying Information Sys-
Buzzard, K. (1999). Computer security - What
tems security. European Journal of Information
should you spend your money on. Computers &
Systems, 1(2), 121–130. doi:10.1057/ejis.1991.20
Security, 18(4), 322–334. doi:10.1016/S0167-
Baskerville, R. (1992). The developmental dual- 4048(99)80078-9
ity of Information Systems security. Journal of
Byrnes, F., & Proctor, P. (2002). The secured
Management Systems, 4(1), 1–12.
enterprise: Protecting your information assets.
Baskerville, R. (1993). Information Systems secu- Upper Saddle River, NJ: Prentice Hall.
rity design methods: Implications for Information
Campbell, K., Gordon, L.A., Loeb, M.P., &
Systems development. ACM Computing Surveys,
Zhou, L. (2003). The economic cost of publicly
25(4), 375–413. doi:10.1145/162124.162127
announced information security breaches: Empiri-
Baskerville, R., & Siponen, M. (2002). An informa- cal evidence from the stock market. Journal of
tion security meta-policy for emergent organiza- Computer Security, 431-448.
tions. Logistics Information Management, 15(5/6),
Cavusoglu, H., Mishra, B., & Raghunathan, S.
337–346. doi:10.1108/09576050210447019
(2004). The effect of internet security breach
Beck, U. (1992). Risk society. London, UK: Sage. announcements on market value: Capital market
reactions for breached firms and internet security
Belden, M. (1989). The employee’s role in
developers. International Journal of Electronic
protecting information assets. Computers &
Commerce, 9(1), 69–104.
Security, 8(6), 487–493. doi:10.1016/0167-
4048(89)90077-1 Center for Strategic and International Studies
(CSIS). (2008). Securing cyberspace for the 44th
Blythe, S. E. (2008). Croatia’s computer laws:
presidency. Retrieved 22 December, 2009, from
Promotion of growth in e-commerce via greater
http://www.csis.org/media/csis/pubs/081208_se-
cyber-security. European Journal of Law and
curingcyberspace_44.pdf
Economics, 26, 75–103. doi:10.1007/s10657-
008-9053-y
358
Center for Strategic and International Studies D’Arcy, J., & Hovav, A. (2007). Towards a best fit
(CSIS). (2009). Significant cyber events since between organizational security countermeasures
2006. Retrieved 22 December, 2009, from http:// and Information Systems misuse behaviors. Jour-
csis.org/files/publication/091109_cyber_events_ nal of Information Systems Security, 3(2), 3–30.
since_2006.pdf
D’Arcy, J., & Hovav, A. (2009). Does one size fit
Chandler, D. (2008). The transmission model of all? Examining the differential effects of IS secu-
communication. Retrieved 15 May, 2011, from rity countermeasures. Journal of Business Ethics,
http://www.aber.ac.uk/media/Documents/short/ 89, 59–71. doi:10.1007/s10551-008-9909-7
trans.html
Da Veiga, A., & Eloff, J. H. P. (2007). An infor-
Chang, S. E., & Ho, C. B. (2006). Organiza- mation security governance framework. Infor-
tional factors to the effectiveness of implement- mation Systems Management, 24(4), 361–372.
ing information security management. Industrial doi:10.1080/10580530701586136
Management & Data Systems, 106(3), 345–361.
Daniels, T. E., & Spafford, E. H. (1999). Identifica-
doi:10.1108/02635570610653498
tion of host audit data to detect attacks on low-level
Cheswick, W. R., & Bellovin, S. M. (1994). IP. Journal of Computing Security, 7(1), 3–35.
Firewalls and Internet security: Repelling the
wily hacker. Reading, MA: Addison-Wesley
tion System success: The quest for the dependent
Publishing Company.
variable. Information Systems Research, 3(1),
Choi, N., Kim, D., Goo, J., & Whitmore, A. (2008). 60–95. doi:10.1287/isre.3.1.60
Knowing is doing: An empirical validation of
Denning, D. (1987). An intrusion-detection model.
the relationship between managerial information
IEEE Transactions on Software Engineering,
security awareness and action. Information Man-
13(2), 222–226. doi:10.1109/TSE.1987.232894
agement & Computer Security, 16(5), 484–501.
doi:10.1108/09685220810920558 Dhillon, G. (1999). Managing and control-
ling computer misuse. Information Manage-
Clements, D. P. (1977). Fuzzy ratings for computer
ment & Computer Security, 7(4), 171–175.
security evaluation. PhD Dissertation, University
doi:10.1108/09685229910292664
of California, Berkeley.
Dhillon, G. (2001). Violation of safeguards by
Cohen, F. B. (1995). Protection and security on
trusted personnel and understanding related
the information superhighway. New York, NY:
information security concerns. Computers &
John Wiley.
Security, 20, 165–172. doi:10.1016/S0167-
Coles, R. S., & Moulton, R. (2003). Opera- 4048(01)00209-7
tionalizing IT risk management. Computers &
Dhillon, G., & Backhouse, J. (1994). Responsibil-
Security, 22(6), 487–493. doi:10.1016/S0167-
ity analysis: A basis for understanding complex
4048(03)00606-0
managerial situations. In Proceedings of the
Computer Security Institute (CSI). (2008). CSI International System Dynamics Conference, (pp.
computer crime & security survey. Retrieved 70-79).
22 December, 2009, from http://i.cmpnet.com/
v2.gocsi.com/pdf/CSIsurvey2008.pdf
359
Dhillon, G., & Backhouse, J. (2001). Current Fiske, J. (1982). Introduction to commu-
directions in IS security research: Towards socio- nication studies. London, UK: Routledge.
organizational perspectives. Information Systems doi:10.4324/9780203323212
Journal, 11(2), 127–153. doi:10.1046/j.1365-
Flowerday, S., & von Solms, R. (2005). Real-
2575.2001.00099.x
time information integrity = system integrity +
Dhillon, G., & Torkzadeh, G. (2006). Value-fo- data integrity + continuous assurances. Comput-
cused assessment of Information System security ers & Security, 24(8), 604–613. doi:10.1016/j.
in organizations. Information Systems Journal, 16, cose.2005.08.004
293–314. doi:10.1111/j.1365-2575.2006.00219.x
Foltz, C. B., Schwager, P. H., & Anderson,
Dinev, T., Goo, J., Hu, Q., & Nam, K. (2008). J. E. (2008). Why users (fail to) read com-
User behaviour towards protective Information puter usage policies. Industrial Manage-
Technologies: The role of national cultural dif- ment & Data Systems, 108(6), 701–712.
ferences. Information Systems Journal, 19(4), doi:10.1108/02635570810883969
391–412. doi:10.1111/j.1365-2575.2007.00289.x
Fricke, M. (2009). The knowledge pyramid:
Dinev, T., & Hu, Q. (2007). The centrality of A critique of the DIKW hierarchy. Jour-
awareness in the formation of user behavioral nal of Information Science, 35(2), 131–142.
intention toward protective information technolo- doi:10.1177/0165551508094050
gies. Journal of the Association for Information
Systems, 8(7), 386–408.
conceptualizing Information Systems success:
Dodge, R. C., Carver, C., & Ferguson, A. J. The IS-impact measurement model. Journal of
(2007). Phishing for user security awareness. the Association for Information Systems, 9(7),
Computers & Security, 26, 73–80. doi:10.1016/j. 377–408.
cose.2006.10.009
Gaudin, S. (2007, April 11). Security breaches cost
Drevin, L., Kruger, H. A., & Stegn, T. (2007). Val- $90 to $305 per lost record. Information Week.
ue-focused assessment of ICT security awareness Retrieved October 18, 2010, from http://www.in-
in an academic environment. Computers & Secu- formationweek.com/news/security/showArticle.
rity, 26, 36–43. doi:10.1016/j.cose.2006.10.006 jhtml?articleID=199000222
Dutta, A., & McCrohan, K. (2002). Management’s Gondek, C. (1989). Establishing information
role in information security in a cyber economy. security. Management Accounting, 70(10), 34–37.
California Management Review, 45(1), 67–87.
Goodhue, D. L., & Straub, D. W. (1991). Security
Elleithy, K. (2008). Innovations and advanced concerns of system users: A study of perception
techniques in systems, computing sciences, and of the adequacy of security measures. Information
software engineering. Dordrecht, The Nether- & Management, 20(1), 13–27. doi:10.1016/0378-
lands: Springer. doi:10.1007/978-1-4020-8735-6 7206(91)90024-V
Finne, T. (1998). A conceptual framework for Gordon, L. A., & Loeb, M. P. (2001). Using
information security management. Computers information security as a response to competitor
& Security, 17(4), 303–307. doi:10.1016/S0167- analysis systems. Communications of the ACM,
4048(98)80010-2 44(9), 70–75. doi:10.1145/383694.383709
360
Gordon, L. A., & Loeb, M. P. (2002). The eco- Herath, T., & Rao, H. R. (2009). Protection moti-
nomics of information security investment. ACM vation and deterrence: A framework for security
Transactions on Information and System Security, policy compliance in organisations. European
5(4), 438–457. doi:10.1145/581271.581274 Journal of Information Systems, 18(2), 106–126.
doi:10.1057/ejis.2009.6
Gordon, L. A., & Loeb, M. P. (2006). Budget-
ing process for information security expendi- HIMSS Security Survey. (2009). Healthcare
tures. Association for Computing Machinery. information and management systems society.
Communications of the ACM, 49(1), 121–125. Retrieved 13 May 2011 from www.himss.org/con-
doi:10.1145/1107458.1107465 tent/files/HIMSS2009SecuritySurveyReport.pdf
Gordon, L. A., Loeb, M. P., Lucyshyn, W., & So- Hitchings, J. (1996). A practical solution to the
hail, T. (2006). The impact of the Sarbanes-Oxley complex human issues of information security
Act on the corporate disclosures of information design. In Katsikas, S. (Ed.), Information Systems
security activities. Journal of Accounting and security: Facing the information society of the 21st
Public Policy, 25(5), 503–530. doi:10.1016/j. century. London, UK: Chapman & Hall.
jaccpubpol.2006.07.005
Hone, K., & Eloff, J. H. P. (2002). Information
Guarro, S. B. (1987). Principles and procedures security policy - What do international infor-
of the LRAM approach to information systems mation security standards say? Computers &
risk analysis and management. Computers & Security, 21(5), 402–409. doi:10.1016/S0167-
Security, 6(6), 493–504. doi:10.1016/0167- 4048(02)00504-7
4048(87)90030-7
Hoo, K. (2000). How much is enough? A risk-
Harrington, S. J. (1996). The effect of codes of management approach to computer security. A
ethics and personal denial of responsibility on Consortium for Research on Information Security
computer abuse judgments and intentions. Man- Policy (CRISP) Working Paper. Stanford, CA:
agement Information Systems Quarterly, 20(3), Stanford University.
257–278. doi:10.2307/249656
Hovav, A., & D’Arcy, J. (2003). The impact of
Hausken, K. (2006). Income, interdependence, denial-of-service attack announcements on the
and substitution effects affecting incentives for market value of firms. Risk Management & Insur-
security investment. Journal of Accounting and ance Review, 6(2), 97–121. doi:10.1046/J.1098-
Public Policy, 25, 629–665. doi:10.1016/j.jac- 1616.2003.026.x
cpubpol.2006.09.001
Huang, C. D., Hu, Q., & Behara, R. S. (2008). An
Hausken, K. (2007). Information sharing among economic analysis of the optimal information se-
firms and cyber attacks. Journal of Accounting curity investment in the case of a risk-averse firm.
and Public Policy, 26, 639–688. doi:10.1016/j. International Journal of Production Economics,
jaccpubpol.2007.10.001 114(2), 793–804. doi:10.1016/j.ijpe.2008.04.002
Hayam, A., & Oz, E. (1993). Integrating data se- Hunsinger, D. S., & Smith, M. A. (2009). IT
curity into the systems development life. Journal certification use by hiring personnel. Journal of
of Systems Management, 44(8), 16–21. Computer Information Systems, 50(2), 71–82.
361
Information Systems Audit and Control Asso- Kokolakis, S. A., Demopoulos, A. J., & Ki-
ciation (ISACA). (2009). An introduction to the ountouzis, E. A. (2000). The use of business
business model for information security. Retrieved process modelling in Information Systems
15 November, 2009, from http://www.isaca.org/ security analysis and design. Information Man-
AMTemplate.cfm?Section=Deliverables&Te agement & Computer Security, 8(3), 107–116.
mplate=/ContentManagement/ContentDisplay. doi:10.1108/09685220010339192
cfm&ContentID=48017
Kotulic, A. G., & Clark, J. G. (2004). Why there
Institute for Information Infrastructure Protection. aren’t more information security research stud-
(I3P). (2009). National cyber security research ies. Information & Management, 41, 597–607.
and development challenges: An industry, aca- doi:10.1016/j.im.2003.08.001
demic, and government perspective. Retrieved
Krauss, M., & Tipton, H. (2002). Handbook of
22 December, 2009, from http://www.thei3p.org/
information security management. Boca Raton,
docs/publications/i3pnationalcybersecurity.pdf
FL: CRC Press.
Jarvenpaa, S. L., & Ives, B. (1991). Executive
Kwok, L., & Longley, D. (1997). Code of practice:
involvement and participation in the manage-
A standard for information security management.
ment of Information Technology. Management
In Proceedings of the IFIP TC11 13th International
Conference on Information Security.
doi:10.2307/249382
Landwehr, C. E., Bull, A. R., McDermott, J. P.,
Jones, A. (1997). Penetration testing and sys-
& Choi, W. S. (1994). A taxonomy of computer
tem audit. Computers & Security, 16, 595–602.
security flaws. ACM Computing Surveys, 26(3),
doi:10.1016/S0167-4048(97)80796-1
211–254. doi:10.1145/185403.185412
Kaliski, B. (1993). A survey of encryption
Leiwo, J., Gamage, C., & Zheng, Y. (1999). Orga-
standards. IEEE Microcomputers, 13, 74–81.
nizational modeling for efficient specification of
doi:10.1109/40.248057
information security requirements. In Proceedings
Kane, G. C. (2006). Casting the net: A multimodal of the Advances in Databases and Information
network perspective on knowledge sharing. Dis- Systems: 3rd East European Conference (ABDIS),
sertation Abstracts International, 67(09). (UMI (pp. 247-260).
No: 1232407861)
Liebenau, J., & Backhouse, J. (1990). Under-
Kankanhalli, A., Teo, H. H., Tan, B. C. Y., & Wei, standing information: An introduction. London,
K. K. (2003). An integrative study of Information UK: Macmillan.
Systems security effectiveness. International
Lovata, L. M. (1987). Behavioral theories relating
Journal of Information Management, 23, 139–254.
to the design of information systems. Management
doi:10.1016/S0268-4012(02)00105-6
Knapp, K. J., Morris, R. F., Marshall, T. E., & doi:10.2307/249354
Byrd, T. A. (2009). Information security policy:
Luotonen, O. (1993). Risk management and in-
An organizational-level process model. Comput-
surances. Helsinki, Finland: Painatuskeskus Oy.
ers & Security, 28(7), 493–508. doi:10.1016/j.
cose.2009.07.001
362
Ma, Q., Johnston, A. C., & Pearson, J. M. (2008). O’Donovan, B., & Roode, D. (2002). A framework
Information security management objectives and for understanding the emerging discipline of infor-
practices: A parsimonious framework. Informa- mation systems. Information Technology & People,
tion Management & Computer Security, 16(3), 15(1), 26–41. doi:10.1108/09593840210423217
251–270. doi:10.1108/09685220810893207
Parker, D. B. (1998). Fighting computer crime - A
McFadzean, E., Ezingeard, J., & Birchall, D. new framework for protecting information. New
(2006). Anchoring information security gover- York, NY: John Wiley & Sons.
nance research: Sociological groundings and
Perry, T., & Wallich, P. (1984). Can computer
future directions. Journal of Information System
crime be stopped? IEEE Spectrum, 21(5).
Security, 2(3), 3–48.
Perry, W. E. (1985). Management strategies for
McGaughey, R. E., Snyder, C. A., & Carr, H. H.
computer security. Boston, MA: Butterworth-
(1994). Implementing Information Technology for
Heinemann.
competitive advantage: Risk management issues.
Information & Management, 26(5), 273–280. Perschke, G. A., Karabin, S. J., & Brock, T. L.
doi:10.1016/0378-7206(94)90066-3 (1986). Four steps to information security. Journal
of Accountancy, 161(4), 104–113.
McKnight, L., Solomon, R., Reagle, J., Carver, D.,
Johnson, C., Gerovac, B., & Gingold, D. (1997). Petter, S., DeLone, W., & McLean, E. (2008).
Information security of Internet commerce. In Measuring information systems success: Models,
McKnight, L. W., & Bailey, J. P. (Eds.), Internet dimensions, measures, and interrelationships.
economics. Cambridge, MA: MIT Press. European Journal of Information Systems, 17,
236–263. doi:10.1057/ejis.2008.15
Meadows, C. (2001). A cost-based framework for
analysis of denial of service in networks. Journal Pfleeger, C. P., & Pfleeger, S. L. (2003). Security
of Computer Security, 9(1/2), 143–164. in computing (3rd ed.). Prentice Hall.
Millen, J. (1992). A resource allocation model Pickard, R. (1989). Computer crime. Information
for denial of service. In Proceedings of the 1992 Center, 5(9), 18–27.
IEEE Symposium on Security and Privacy. Los
Pisello, T. (2004, October). Is there a business
Alamitos, CA: IEEE Computer Society Press.
case for IT security? Security Management.
Misra, S. C., Kumar, V., & Kumar, U. (2007). Retrieved October 18, 2010 from http://www.
A strategic modeling technique for information securitymanagement.com/article/there-business-
security risk assessment. Information Man- case-it-security
agement & Computer Security, 15(1), 64–77.
Poore, R. (2006). Information security gover-
doi:10.1108/09685220710738787
nance. In Handbook of information security
Moulton, R., & Coles, R. S. (2003). Applying management. Boca Raton, FL: CRC Press.
information security governance. Computers &
Post, G. V., & Diltz, J. D. (1986). A stochastic
Security, 22(7), 580–584. doi:10.1016/S0167-
dominance approach to risk analysis of computer
4048(03)00705-3
systems. Management Information Systems Quar-
Neumann, P., & Parker, D. (1989). A summary of terly, 10(4), 363–375. doi:10.2307/249191
computer misuse techniques. In Proceedings of
the 12th National Computer Security Conference.
363
Posthumus, S., & von Solms, R. (2004). A Siponen, M., Pahnila, S., & Mahmood, M. A.
framework for the governance of information (2010). Compliance with information security
security. Computers & Security, 23, 638–646. policies: An empirical investigation. Computer,
doi:10.1016/j.cose.2004.10.006 43(2), 64–71. doi:10.1109/MC.2010.35
Rainer, R. K., Snyder, C. A., & Carr, H. H. (1991). Siponen, M. T. (2001). On the role of human moral-
Risk analysis for Information Technology. Jour- ity in Information Systems security. Information
nal of Management Information Systems, 8(1), Resources Management Journal, 14(4), 15–23.
129–147. doi:10.4018/irmj.2001100102
Rivest, R. L. (1978). A method for obtaining Siponen, M. T. (2005). Analysis of modern IS
digital signatures and public key cryptosystems. security development approaches: Toward the next
Communications of the ACM, 21(2), 120–126. generation of social and adaptable ISS methods.
doi:10.1145/359340.359342 Information and Organization, 15, 339–375.
doi:10.1016/j.infoandorg.2004.11.001
Rosenthal, D. (2002). Intrusion detection tech-
nology: Leveraging the organization’s security Smith, H. J. (1994). Managing privacy: Informa-
posture. Information Systems Management, 19(1), tion Technology and corporate America. Chapel
35–44. doi:10.1201/1078/43199.19.1.20020101/ Hill, NC: University of North Carolina Press.
31474.5
Stallings, W. (1995). Network and internetwork
Russell, D., & Gangemi, G. T. (1991). Computer security principles and practice. Englewood
security basics. Sebastopol, CA: O’Reilly. Cliffs, NJ: Prentice Hall.
Rycroft, S., & Tully, M. (2007). Building an Stanton, J. M., Stam, K. R., Mastrangelo, P., &
information security meta standard. BT Technol- Jolton, J. (2005). Analysis of end user security
ogy Journal, 25(1), 37–40. doi:10.1007/s10550- behaviors. Computers & Security, 24, 124–133.
007-0006-8 doi:10.1016/j.cose.2004.07.001
Schultz, E. E. (2002). A framework for understand- Sterne, D. F. (1991). On the buzzword ‘security
ing and predicting insider attacks. Computers & policy’. In Proceedings of the IEEE Computer
Security, 21(6), 526–531. doi:10.1016/S0167- Society Symposium on Research in Security and
4048(02)01009-X Privacy, 219-230.
Schweitzer, J. A. (1982). Managing information Straub, D. W. (1990). Effective IS security: An
security: A program for the electronic informa- empirical study. Information Systems Research,
tion age. Boston, MA: Butterworth-Heinemann. 1(3), 255–276. doi:10.1287/isre.1.3.255
Shannon, C. E., & Weaver, W. (1949). The math- Straub, D. W., & Welke, R. J. (1998). Coping
ematical theory of communication. Urbana, IL: with systems risk: Security planning models
University of Illinois Press. for management decision making. Management
Shimeall, T. J., & McDermott, J. J. (1999).
doi:10.2307/249551
Software security in an internet world: An ex-
ecutive summary. IEEE Software, 16(4), 58–62. Subramoniam, S., & Sadi, A. H. M. S. (2010).
doi:10.1109/52.776950 Information Technology Professional, Nov/Dec,
46-51.
364
Sun, L., Srivastava, R. P., & Mock, T. J. (2006). von Solms, B. (2000). Information security –
An information systems security risk assessment The third wave? Computers & Security, 19(7),
model under the Dempster-Shafer theory of belief 615–620. doi:10.1016/S0167-4048(00)07021-8
functions. Journal of Management Information
von Solms, R., & von Solms, B. (2004). From
Systems, 22(4), 109–142. doi:10.2753/MIS0742-
policies to culture. Computers & Security, 23(4),
1222220405
275–279. doi:10.1016/j.cose.2004.01.013
Tanaka, H., Matsuura, K., & Sudoh, O. (2005).
von Solms, R., & von Solms, S. H. (2006). Infor-
Vulnerability and information security investment:
mation security governance: A model-based on the
An empirical analysis of e-local government in
direct-control cycle. Computers & Security, 25,
Japan. Journal of Accounting and Public Policy,
408–412. doi:10.1016/j.cose.2006.07.005
24, 37–59. doi:10.1016/j.jaccpubpol.2004.12.003
von Solms, S. H. (2005). Information security
Tanenbaum, A. (1992). Modern operating systems.
governance - Compliance management vs opera-
Englewood Cliffs, NJ: Prentice-Hall.
tional management. Computers & Security, 24,
Tejay, G., Dhillon, G., & Chin, A. G. (2005). 443–447. doi:10.1016/j.cose.2005.07.003
Data quality dimensions for Information Systems
Vroom, C., & von Solms, R. (2004). Towards in-
security: A theoretical exposition. In Dowland,
formation security behavioural compliance. Com-
P., Furnell, S., Thuraisingham, B., & Wang, X.
puters & Security, 23(3), 191–198. doi:10.1016/j.
S. (Eds.), Security management, integrity, and
cose.2004.01.012
internal control in Information Systems (pp.
21–39). New York, NY: Springer. doi:10.1007/0- Walsh, M. E. (1981). Software security. Journal
387-31167-X_2 of Systems Management, 32(10), 6–14.
Telang, R., & Wattal, S. (2007). An empirical Warkentin, M., & Willison, R. (2009). Behavioral
analysis of the impact of software vulnerability and policy issues in Information Systems security:
announcements on firm stock price. IEEE Trans- The insider threat. European Journal of Infor-
actions on Software Engineering, 33(8), 544. mation Systems, 18(2), 101–105. doi:10.1057/
doi:10.1109/TSE.2007.70712 ejis.2009.12
Tittle, C. R. (1980). Sanctions and social devi- Warman, A. R. (1992). Organizational computer
ance: The question of deterrence. New York, security policy: The reality. European Journal of
NY: Praeger. Information Systems, 1(5), 305–310. doi:10.1057/
ejis.1992.2
Tompkins, T., & Handley, D. (2003). Giving e-
mail back to the users: Using digital signatures Webler, T., Rakel, H., & Ross, R. J. S. (1992). A
to solve the spam problem. First Monday, 8(9). critical theoretical look at technical risk analysis.
Industrial Crisis Quarterly, 6, 23–38.
Trompeters, C. M., & Eloff, J. H. P. (2001). A
framework for the implementation of socio-ethical Webster, J., & Watson, R. T. (2002). Analyzing
controls in information security. Computers the past to prepare for the future: Writing a lit-
& Security, 20, 384–391. doi:10.1016/S0167- erature review. Management Information Systems
4048(01)00507-7 Quarterly, 26(2), xiii–xxiii.
Vigna, G., & Kemmeerer, R. A. (1999). NetSTAT: Whitman, M. E. (2003). Enemy at the gate: Threats
A network-based intrusion detection system. Jour- to information security. Communications of the
nal of Computing Security, 7(1), 37–71. ACM, 46(8), 91–95. doi:10.1145/859670.859675
365
Whitman, M. E. (2004). In defense of the realm: KEY TERMS AND DEFINITIONS

Understanding the threats to information se-
curity. International Journal of Information Information: Data arranged in a meaningful
Management, 24(1), 43–57. doi:10.1016/j.ijin- way for some perceived purpose (Liebenau &
fomgt.2003.12.003 Backhouse, 1990)
Information System: An aggregate of infor-
Willison, R., & Backhouse, J. (2006). Opportu- mation handling activities at the technical, formal
nities for computer crime: Considering systems and informal levels of an organization (Liebenau
risk from a criminological perspective. European & Backhouse, 1990)
Journal of Information Systems, 15, 403–414. Information Systems Security: A well-
doi:10.1057/palgrave.ejis.3000592 informed sense of assurance that information risks
Wood, C. C. (1999). Information security policies and information security controls are in balance
made easy. San Rafael, CA: Baseline Software. (Anderson, 2003)
Information Systems Security Success: The
Woody, C. (2006). Applying OCTAVE: Practitio- cumulative effect of the relationship between in-
ners report. Carnegie Mellon University. formation systems experience and user experience,
Zviran, M., & Haga, W. J. (1999). Password secu- manifested as the tangible and intangible benefits
rity: An empirical study. Journal of Management provided by an organization’s IS security program.
Information Systems, 15(4), 161–184. Success: The tangible and intangible benefits
a program provides to an organization (Petter,
DeLone, & McLean, 2008).
366
367
Chapter 18
Evolutional Patterns of
Intranet Applications:
Organizational Issues and
Pietro Previtali
University of Pavia, Italy
ABSTRACT
The objective of this chapter is to propose a theoretical examination, strengthened by an empiric survey
of intranet evolutional patterns and the neologism that designates a communication system, for access
to and searching of business information based on Internet technologies. This chapter analyses intranet
applications and functionalities in order to classify them according to a taxonomy that allows us to dis-
tinguish between an institutional intranet, a knowledge management intranet, and an operating intranet.
The main research hypothesis is the existence of an intranet life cycle, as an evolutional model starting
from an institutional intranet that moves to a knowledge management intranet and then to an operating
one. This last one is considered as a proxy for successful IS implementation. To substantiate the above-
mentioned hypothesis an empirical study was conducted among 110 large Italian corporations, with a
response rate of 66% (73 corporations). The method used was a survey conducted during the months of
March, April, and May 2010, applying a mix of random sampling (randomly selected interviewees from
the directory) and “snowball” sampling (contacting interviewees through leads). The results show how,
basically, companies approach intranet implementation processes in an incremental way, which begins
with the integration of the basic functionalities as “communication channel,” “service platform,” or
“document management.”
DOI: 10.4018/978-1-4666-0170-3.ch018
Evolutional Patterns of Intranet Applications
BACKGROUND the efforts to find, organize and make available

systematically a company’s intellectual capital
The external environment has an overwhelming and to foster a culture of continuous learning and
impact on management uncertainty and organiza- knowledge sharing (Averweg, 2010) so that the
tional functioning. The changes and complexity organizational activities build on what is already
in environmental domains have major implica- known. An intranet can also be considered as an
tions for organizational design and action. In this information technology application to a defined
context, organizational changes are assuming a community of users within an organisation (Das-
character of discontinuity and an acceleration gupta, 2001). Typically, intranets use the same
without precedent in today’s business world. So, technologies and software as the Internet. As such,
it is more and more important for each corporation intranets are typically accessed using the same
to consider variables such as dynamic efficiency, web browser applications used for accessing the
flexibility, autonomy and cooperation in their Internet (Baker, 2000).
organizational strategies. The main challenge of intranets is the ability
In order to provide an effective solution to the to meet business goals through improved pro-
new organizational and strategic requirements, ductivity and efficiency of employees (Lederer et
first of all, organizations should implement an al., 2000). Intranets can offer significant benefits
adequate information and communication system. through internal communication, collaborative/
The related logic and technological architecture cooperative work, knowledge management and
must be simple, integrated and without barriers process redesign (Baker, 2000). More in details,
in terms of data exchange or the geographic lo- Lloyd (1998) identifies many particular benefits
calization of the information. Moreover, it should experienced by organisations from the develop-
allow the connection of the different peripheral ment of intranets, including improved competi-
organizational units, carrying out effective data tiveness gained through operational efficiency
and knowledge sharing, without any spatial or and improved productivity through increased
temporal boundaries (Buniyamin, 2004; Geczy, access to timely and up-to-date information;
2007; Mateo, 2011). One primary goal for today’s cost savings related to reduction of papers use,
information technology systems is to support warehousing documents and distribution, time
efforts to manage and leverage organizational savings on searching for and locating information,
knowledge (O’Boyle, 2009). Having greater ac- improved productivity due to faster communica-
cess to information is useless unless that information of information, and easier understanding of
tion is put to use to further the goals and success information due to instinctive intranet interfaces.
of the company (Denton, 2010). An intranet implementation brings the fol-
An organization technology that seems to lowing most common advantages (Hills, 1997a,
satisfy these informative requirements is the Hills, 1997b):
intranet (Goles & Hirschheim, 1997; Crowston,
2002; Norzaidi, 2009 A). This neologism identi- a. technological advantages:
fies a system for communication, access to and ◦◦ web browser interface, standard and
searching of business information, which is an user- friendly;
Internet-based technology (TCP-IP protocol and ◦◦ flexible and adaptable to business
so on) that led to a new way of thinking about processes and end user preferences;
organizing and sharing an organization’s intel- ◦◦ based on open standards;
lectual and creative resources (Denton, 2003; ◦◦ connectivity with other systems;
Horton et al., 2001; Lloyd, 1998). It refers to
368
◦◦ scalable; hypertext links and continued and effortlessly access all

multimedia through the company using web tech-
◦◦ single interface to information re- nologies and hypermedia. The famil-
sources and services iar examples of web publishing con-
b. organizational advantages: sist of training, news feed, company
◦◦ access to internal and external infor- polices, documents, and employee
mation, it improves communication, manual. Each unit can bring up to
increases collaboration and coordina- date the online copy of a document
tion, supports links with customers and intranet always provides the most
and partners, can capture and share recent version to employees;
knowledge; ◦◦ intranet offers business operations
◦◦ through intranet common corporate and administration solutions because
culture every user can view the simi- it also being used as a platform of
lar information. mounting and organizing applica-
◦◦ intranet improves teamwork and tions across the internet world;
all certified users can get access to ◦◦ intranet can help users to locate and
information. view information faster and use ap-
c. economic advantages: plications relevant to their roles and
◦◦ intranet is time saving because there responsibilities. With the help of a
is no need to maintain physical docu- web browser interface, users can ac-
ments such as procedure manual, cess data held in any database the or-
requisition forms, and internet phone ganization wants to make available,
list; anytime and - subject to security pro-
◦◦ intranet facilitates their user to view visions - from anywhere within the
and gets information and data via web company workstations, increasing
browser. Intranet also save the money employees’ ability to perform their
of any organization on printing, pub- jobs faster, more accurately, and with
lishing and overall maintenance; confidence that they have the right
◦◦ intranet is scalable and flexible. A information. It also helps to improve
small company can be up and run- the services provided to the users;
ning at practically no extra cost. As ◦◦ intranet is also being used as a plat-
the company grows, it can invest in form for developing and deploying
a more robust web server, hire a web applications to support business op-
developer to design more dynamic in- erations and decisions across the in-
tranet web pages and make more and ternetworked enterprise;
more information and software avail- e. integration advantages:
able online. ◦◦ intranet improves workforce produc-
d. operational advantages: tivity which can help user to find and
◦◦ intranets are helpful to converse observe information very fast. User
planned initiative that has an in- may also use applications accord-
ternational reach all through the ing to their roles and tasks. Through
organization; web browser a user can get access to
◦◦ web publishing which permits bur- entire contents of any website from
densome corporate knowledge to be anywhere or any time. Intranet also
369
increase the ability of employee’s by like film production and less like a traditional
performing their job confidently very engineering activity. Traditional distinctions - for
fast, and accurately; example between designers and users - will be-
◦◦ intranet permits business companies come blurred and multiskilled teams that combine
to share out information to employ- high levels of both technical and artistic skills
ees according to their need or require- will develop IS services. Whether an Intranet has
ments. Employees may also link to been implemented as an operational tool or as a
appropriate data at their expediency strategic vehicle, it must be designed to be usable
◦◦ intranet offers to their user to write by employees. It’ important to highlight that a
applications on their browser without number of attributes are central to the usability
cross-browser compatibility issues. of an Intranet such as the application should be
useful, easy to learn, easy to use and consistent
According to Lyytinen et al. (1998) and Bansler (Gould and Lewis 1985; Palmer 2002).
et al. (2000), intranets led to four major changes As regards the make or buy strategy, an In-
in IS services: tranet technology is relatively simple, and most
organizations will have sufficient in-house com-
• ubiquity of services: services will be avail- petences to address the technical challenges in
able at any time and at any place; the implementation of an intranet. Exclusive use
• speed of change: new technologies are be- of internal resources, however, will not readily
ing invented and adopted an order of mag- allow the organization to benefit from the experi-
nitude faster than were earlier platforms. ence, expertise, and economies of scale inherent
As a result many technologies and related to existing intranet products on the market or
skills will become obsolete overnight; vendors specializing in intranet development.
• component-based development: the new Karlsbjerg and Damsgaard (2001) identify four
platform is founded on the use of compo- archetypes of intranet implementation strategies.
nent architectures that will lead to the cre- The first type is characterized by a tailor- made
ation of software component markets and architecture and in-house implementation. In this
the delivery of new software components organizations the employees must have certain
through the network; technical skills and expertise on subjects such as
• media design: software development will web-server technology, web development tools
coalesce with media design. IS services and programming languages. This type leads some
will become media oriented in contrast to advantages like inexpensive in the beginning of
computation orientations of the past. the implementation process and potential high
degree of tailoring to the organization, but also
The authors predict that these changes have some disadvantages as high hidden costs of use
deep and pervasive implications for the way of internal competences, experience from similar
organizations will use, manage, and organize intranets unlikely, the functionality of the intranet
their IS resources in the future. First, new skills has not been widely tested, poor technical design.
become critical in developing IS services. These The second type is characterized by a tailor-
include telecommunications skills, artistic and made architecture and outsourced implementation.
content skills, as well as broad organizational This strategy allows the organization to benefit
design and change management skills. Secondly, from the consultants’ experiences from similar
the organization of systems development will projects aiding in the requirement specification
change. Systems development will become more and development of an intranet tailored to the
370
organization’s specific needs. Organizations that implementation into the organization. The in-
do not have time or in-house skills to develop tranet product will be based on thoroughly tested
an in-house solution may turn to the consultant. advanced applications, and due to the choice of
Another reason for contacting expertise from implementation sourcing, the implementation
outside contractors may be low availability or effort will benefit from the experience of the con-
high costs of a knowledgeable workforce on the sultants. By using this implementation strategy,
market. This type leads some advantages like the organization can adjust its intranet to more
less internal expertise is required, benefits from demanding requirements without bearing the bur-
the consultants’ experience and expertise, likely den of the required learning. This type leads both
to be technically well designed, an opportunity advantages like advanced standard functionality
to improve the internal competence through edu- can be achieved fast, thoroughly tested functional-
cation but also some disadvantages as a risk of ity, a chance to increase the internal competence
becoming too dependent on one supplier and often through education, Benefits from the consultants’
an expensive solution. experience with similar projects, likely to be
The third type is characterized by a ready- made technically well designed and disadvantages as
architecture and in-house implementation. In such a risk of becoming locked-in to the product and
case, the organization’s requirements may be met to the supplier, it can be an expensive solution
by the standard functionality or by simple param- due to initial purchase costs, product licenses,
eter adjustment of an intranet product, requiring and consultant fees.
little or no application development knowledge of As concern the implementation process,
the organization. The organization must estimate Bansler et al. found that the influence and action
the extent of adjustments and alterations necessary of mediators can play a critical role in the suc-
before the organization’s requirements are met, cessful implementation of intranets. They may
and the result must be weighed against the avail- actively promote the use of the intranet, adapt the
ability of internal resources. This implementation technology to its local context, and shape the way
strategy lends itself both to organizations with other users adopt and use the technology. Organiza-
organizational processes similar to those supported tions should, therefore, carefully consider how to
in the intranet product, as well as to organizations define the mediator role and how many resources
with a high level of IT skills but too few of these to spend on mediation activities. In addition, the
skilled personnel available to implement an in- authors suggest that issues of power and control are
tranet in-house. This type leads both advantages crucial to intranet implementation and evolution
like standard thoroughly tested functionality, a over time. The distribution of authority among
chance of sedimentation of knowledge about the organizational actors and the degree of manage-
product and its implementation, and disadvantages ment control have decisive impact on the intranet’s
as not tailored to specific requirements, a risk of development. Consequently, senior management
becoming too dependent on the suppliers, it can should carefully consider the design of the support
be a relative expensive solution for large organi- organization and the delegation of power.
zations due to licenses, some internal expertise is
required for the implementation process.
The fourth type is characterized by a ready- RESEARCH METHODOLOGY
made architecture and outsourced implementation.
In this implementation strategy the organization The central aim of this chapter is to propose a
purchases both a standard intranet product and theoretical discussion, strengthened by an em-
outside expertise to perform the installation and pirical survey, of the existence of some typical
371
Figure 1. Sample structure according to the rev-

evolutional processes in the implementation of
enue of the respondents (mln €)
an intranet. If that is true, these typical increas-
ing paths could be used as a dynamic variable
for organizational design and can effect strong
managerial implications. Carrying out empirical
research on ICT topics presents various factors
of complexity. The obstacles concern first of all
the difficult analysis of trends and prescriptions
valid for a long period (Checkland & Holwell,
1998); second, the large number and the diversity
of perspectives and paradigms used by the wide
literature about the organizational impact of ICT
(Stowell, 1995); then, the ambiguities in terms of
lasted about one hour. Each interview was tape-
semantics and contents that concern these topics
recorded and summaries were written and ap-
(Scott, 1994); and finally a concrete obstacle is
proved by each interviewee. In addition, we had
analysing technological variables without con-
access to the 5 intranets and were thus able to get
sidering the organizational ones (Gerwin, 1979),
experience of their structure, design, and content.
according to a systematic approach. Being aware of
The first stage involved interviews with 4
these limits, our aim is not to insist on fine-tuning
champion IT managers within organizations who
of comparative analyses, in order to measure and
have implemented intranet as a strategic technol-
correlate static variables (instant observation of
ogy, developing a large number of applications.
data); the chapter is the result of a longitudinal
The central purpose of this stage was to allow the
analysis of the dynamic phenomenon that is the
researcher to gain an understanding of the rationale
evolutional path of the intranet, in terms of func-
behind the implementation of the intranet and its
tionalities that are progressively implemented.
main characteristics and functionalities. After this
In order to achieve these objectives, our re-
first stage which allowed an important fine tuning
search was conducted among 110 large Italian
of the check list, the second stage was the exten-
corporations, with a response rate of 66% (73
sion of the survey to the remaining 68 companies.
corporations). The method used was a survey
The check list for the interview was structured
that was conducted during the months of March,
in the following sections:
April and May 2010, applying a mix of random
sampling (randomly selected interviewees from
• section1, which explores the intranet func-
our directory) and “snowball” sampling (contact-
tionalities trough the taxonomy showed in
ing interviewees through leads).
the next paragraph which identifies 6 type
In particular, data were collected through
of applications: communication channel,
unstructured and semi-structured interviews, re-
service platform, document management,
view of documents, and examination of intranet
community, collaborative work, process
implementations. Figures 1 and 2 show the main
support;
characteristics of the sample, in terms of company
• section 2, which analyzes the orga-
size and industry.
nizational positioning of the intranet
Participants to interviews spanned vertical
manager and/or the staff dedicated to
levels and functional groupings and included
intranet management, distinguishing be-
senior vice presidents, corporate IT managers, IT
tween IS Department, Human Resources
project leaders and key users. The interviews
372
Figure 2. Sample structure according to the industry of the respondents
Department, Communication Department, OBSERVATION PERSPECTIVE

line departments and other; AND BACKGROUND: THE
• section 3, which analyzes the existence of MAIN FUNCTIONALITIES
an incremental evolutional patter, divid-
ing into phases the evolutional path. The According to a survey conducted by Information-
longitudinal aspect allowed the impact of Week in 1998, with nearly a thousand responding
intranet on the organization intranet to be managers, the information that is most frequently
assessed over a medium – long period. included in intranets is in the form of product
When someone formally decides to imple- catalogues (49 percent of all companies), corporate
ment an intranet, an evolving process has policies and procedures (35 percent), purchase
often already started at the grass root level ordering (42 percent), document sharing (39 per-
(Lamb and Davidson, 2000). Therefore the cent), corporate phone directories (40 percent), and
implementation does not start at a definite human resource forms (35 percent) (McGee 1998).
point in time, but emerges through a series Also included, in lower percentages, were training
of implementation initiatives that com- programs, customer databases, data warehouse
bine existing initiatives with novel ones. and decision support access, image archives, and
Likewise the implementation never seems travel reservation services. Nowadays, an intranet
to end, because new functionality and pos- can be used as a platform for the development of
sibilities evolve over time. different applications (Denton, 2006; Norzaidi,
2009b; Fong, 2010). Our taxonomy of intranet
The use of structured interviews for data col- functionalities distinguishes between six groups
lection helped address the objectives outlined of applicative solutions:
above and to ensure valid and reliable results are
gathered from this study, all participants chosen a. communication channel: functionalities that
for the interview stages had adequate knowledge support information transfer and diffusion,
of the topics under study. which are in general independent in respect
of the user profile, and with a low degree of
interactivity (e.g. bulletins, news, corporate
373
Figure 3. Intranet functionalities
guidelines, market reports, product specifi- ANALYSIS AND REFLECTIONS

cations, internal documents, organizational
charts); The first results provided by the univariate statis-
b. service platform: services for employees, tical analysis lead to the confirmation of a very
for example facilities reservation, training, diffuse proposition in the actual ICT literature:
library service, help desk, e-learning, online in general, intranets have been developed with
support; functionalities characterized by a low degree of
c. document management: support for docu- interactivity and without considering the business
ment publication and management, includ- users’ profile (Scheepers, 2003; Welch, 2007;
ing research, recording and versioning Masrek, 2008). Services for employees and docu-
functionalities; ment management solutions are less diffused but
d. community: support for communication nevertheless important, as well as virtual commu-
and interaction between groups of users, nication and interaction among users. However,
for example solutions such as forums, chat, above all, figure 3 shows that in the Italian context
mailing lists, instant messaging; there is a low adoption of functionalities for the
e. collaborative work: support for team work- creation of virtual teams and for the support of
ing carried out by the creation of a virtual operative processes.
space (groupware, video conferencing, and In addition, only 23% of the sample has ad-
so on); opted all 6 functionalities previously described,
f. process support: a vertical workflow system while the configurations that combine a different
to initiate and execute workflow-related mix of functionalities are much more diffused.
organization transactions and standardized Instead, figure 4 shows the organizational posi-
administrative processes (e.g. field-force tioning of the intranet manager and/or the staff
automation, sales-force automation, budget- dedicated to intranet management. In particular,
ing processing, claim processing, remote the main evidence is that this role is played by
control of production). the IS Department, according to 43% of the
374
sample; secondly by the Human Resources De- intranet only in the last stages of the evolutional
partment with 16% of the sample; the Commu- path.
nication Department manages the local intranet In terms of implementation modalities, 95%
in 13% of the cases; while line departments do of the sample adopted a strategy that could be
not assume any relevant importance. Finally, for called “progressive integration,”1 while only 5%
a large part of the sample, about 22%, there is no of the sample preferred a “big bang approach”;
prevailing function that controls and manages the the implementation process needs a learning path
intranet; in these cases, such responsibility is for users and incremental fine-tuning of these
spread among several organizational units. solutions (Tyre & Orlikowski, 1993; Leonard-
With reference to the central aim of this chap- Barton, 1998). For this reason, Weick (1990)
ter – the dynamic analysis of the intranet phenom- defined ICT as an “equivocal” technology, open
enon – the effective existence of an incremental to many possible interpretations: as a “stochastic”
evolutional pattern finds first confirmation from technology, because of the objective difficulty of
the data shown in figure 5. We asked our sample
to analyse their “intranet history,” dividing into
phases the evolutional path. A phase/stage is a Figure 4. Intranet managers’ organizational
topical momentum in which a company creates/ positioning
adds a new important functionality to the intranet.
In particular, it emerges with clarity that the most
simple functionalities – such as information shar-
ing with a low degree of interactivity – are intro-
duced at the very beginning of the evolutional
path. Consequently, it seems that the subsequent
implementations concern services for employees
and solutions for document management. More-
over, figure 5 suggests that the functionality for
the creation of a virtual team and for the support
of operative processes are integrated into the
Figure 5. The evolutional path of intranet functionalities
375
Table 1. Percentage frequencies of implementation of different functionalities in the different phases
Phase A B C E F G
1 72.10% 24.10% 29.20% 12.80% 9.10% 7.70%
2 14.80% 37.00% 41.70% 35.90% 9.10% 23.10%
3 13.10% 29.60% 14.55% 15.40% 22.70% 26.90%
4 0 9.30% 14.55% 23.10% 13.65% 19.20%
5 0 0 0 7.70% 31.80% 11.55%
6 0 0 0 5.10% 13.65% 11.55%
100.00% 100.00% 100.00% 100.00% 100.00% 100.00%
A) communication channel; B) service platform; C) document management; D) community; E) collaborative work; F) process support
combining determinants and effects; as a “con- sition is true, these hypothetical evolutional
tinuous” technology, for its continuous and fre- processes are not simply random events. So,
netic modernization trends; and finally as an searching for a few systematic evolutional pro-
“abstract” technology, because of the employees’ cesses, the data elaboration has been approached
work using symbolic processes that unavoidably with a statistic fit analysis, in particular with a
create ambiguity and different interpretations calculus of odds ratios for each couple of possible
(Orlikowski, 1992; Sampler, 1996). The empirical situations.2 The result is a three-dimension dicho-
analysis of the evolutional processes of the intranet tomic data set (figure 6) in which each cell E(x;y;t)
is carried out starting with the data shown in contains a value of one if the functionality “x” is
Table 1 that represent the percentage frequencies integrated into the intranet by enterprise “y” at
of implementation of different functionalities instant “t,” and zero otherwise.
during the six theoretical phases. The second step of the data elaboration is to
Therefore, coherently with the aims of the assume six start situations, instant [t1], correspon-
chapter, the main research question is whether dent to the six functionalities, and therefore to
there are some evolutional paths in the intranet proceed with the analysis by searching for the
life cycle that are meaningfully more frequent typical evolution, in terms of the progressive
with respect to other combinations. If this propo- integration of functionalities, for each starting
Figure 6. Logic structure of the data set
376
Figure 7. Start from “A” functionality – path graph
point. As a first outcome of this elaboration, there The second typical path that begins with “A,”
is no typical evolutional path characterized by on the right of figure 7, implements “document
statistical significance that begins with the func- management” functionalities in the second phase
tionalities “D” or “F.” Conversely, table 3 show and subsequently continues with the integration
how the intranets that begin their progressive of “service platform” solutions. Sometimes this
implementation at functionality “A” follow two path proceeds to the “process support” function-
typical evolutional paths. Figure 7 presents the ality. In this case, there is a strong organizational
details of these two evolutional patterns3 that start impact; therefore, an incremental learning process
from the functionality called “communication is presumed (Cyert & March, 1963; Duncan &
channel.” In the first case, on the left of the graph, Weiss, 1978; Nelson & Winter, 1982; March,
the intranet evolves with “service platform” func- 1991). It is probable that the company – before
tionality and then sometimes with “document the implementation of pervasive functionalities
management” solutions. Therefore, in this case it – wants to experiment and to create some frames
seems that the final result is partial development and some scripts in order to manage the process
of the intranet, without functionalities character- of diffusion and adoption of new technologies.
ized by a high degree of organizational impact Similarly to the previous table, table 4 shows
(“D” or “F” functionalities); this configuration the evolutional process that begins with function-
leads to an institutional intranet. ality B. In this case, the data elaborations have
found only one meaningful typical path.
377
Figure 8. Start from “B” functionality – path graph
Figure 8 shows the evolutional path that starts “communication channel,” “service platform” or
with “service platform” functionality, but also “document management.” No typical evolutional
continues with an uncertain trajectory. In fact, paths that start with other functionalities have been
there is a clear incremental process after instant found. This empirical issue could be interpreted
[t1] but it seems very difficult to characterize a with the necessity of some cautious learning
typical path, except for a possible evolution to processes about the modalities of diffusion and
the “process support” functionality. adoption of intranet solutions, especially with
Finally, figure 9 represents two evolutional reference to such functionalities as “collaborative
processes that start from the implementation of work,” “community” and/or “process support,”
the “document management” functionality. In the which involve a greater organizational impact.
first path shown, on the left of the graph, the in- In our understanding, the development of
tranet evolves through the integration of “com- these last functionalities requires, first of all,
munication channel” applications and subse- the adoption of a new organizational culture and
quently it continues without a distinct trend. new managerial paradigms. The intranet is not a
Conversely, the second one is more complex but revolutionary technology; what is revolutionary
also more clear: the intranet evolves – with a high is that this technology can stimulate a better or-
probability – to the “community” functionalities, ganization redesign process and a more efficient
after it follows different trajectories during the information system.
third phase and finally arrives at “collaborative The company’s intellectual capital is the sum
work” in the fourth step. The result is a configu- of its information, experience, understanding,
ration with a high degree of organizational impact. relationships, processes, innovations and dis-
coveries. Although IT plays an important role by
enabling the storage and dissemination of data and
CONCLUSION information across the organization technology,
this is only one part of a larger puzzle. A complete
In synthesis, the results show how, basically, knowledge management system includes not only
companies approach intranet implementation pro- processes for capturing and storing knowledge
cesses in an incremental way, which begins with and organizing it for easy access, but also ways
the integration of the following functionalities: to generate new knowledge through learning and
378
Figure 9. Start from “C” functionality – path graph
to share knowledge throughout the organization. Making this work requires not only a new in-
IT alone is not enough to handle this complex formation infrastructure, but a shift in attitude and
problem. The main key to leveraging knowledge culture, from a push mentality to a pull mentality.
is changing the organizational culture to encour- As in all democratic models, for digital democracy
age people to share rather than hoard knowledge. to occur, the citizens must take the initiative and
In fact, our traditional organizational structures responsibility to create and maintain their own
have focused on a central command and control digital landscape.
model. The organization was designed to bring This evolutional process should probably be
information to the central command site and approached with a progressive and incremental
distribute and enforce the decisions back to the path, and not with a “big bang” implementation,
edges of the enterprise. For a system to function, according to the main results of our empirical
it must have ways of self-organizing and even research. Further useful research would be fo-
directing behaviour. If command and control are cused on which factors are associated with more
distributed to the subsystems, then we must look effective intranet applications, with reference to
elsewhere for the self-organizing capabilities of organizational (pre-) conditions that can promote
the complex system. The paradigm shift, then, the evolutional path towards intranet solutions
is one of moving from a central command and with a high degree of interactive and collabora-
control model to a distributed and decentralized tive functionalities, without an in-depth and posi-
adaptive innovation model. tive impact on core processes, as well as trying
379
to identify which are the best practices in their Denton, D. (2006). Using intranets to
implementation process. make virtual teams effective. Team Per-
formance Management, 12(7-8), 253–257.
doi:10.1108/13527590610711804
REFERENCES
Denton, D. (2010). Performance measure-
Averweg, U. (2010). Enabling role of an ment and intranets: A natural partnership.
intranet to augment e-coaching. Industrial International Journal of Productivity and
and Commercial Training, 42(1), 47–52. Performance Management, 59(7), 701–706.
doi:10.1108/00197851011013715 doi:10.1108/17410401011075675
Baker, S. (2000). Getting the most from your Duncan, R., & Weiss, A. (1978). Organization
Intranet and Extranet strategies. The Journal learning: Implications for organization design.
of Business Strategy, 21, 40–43. doi:10.1108/ In Staw, B. (Ed.), Research in organizational
eb040104 behavior. Greenwich, CT: Jai Press.
Bansler, J. P., Damsgaard, J., Scheepers, R., Havn, Fong, S., & Quaddus, M. (2010). Intranet use in
E., & Thommesen, J. (2000). Corporate Intranet Hong Kong public hospitals. International Jour-
implementation: Managing emergent technolo- nal of Accounting and Information Management,
gies and organizational practices. Journal of the 18(2), 156–181. doi:10.1108/18347641011048138
Association for Information Systems, 1(10). Geczy, P. (2007). Knowledge worker intranet
Buniyamin, N., & Barber, K. (2004). The intranet: behaviour and usability. International Journal
A platform for knowledge management system of Business Intelligence and Data Mining, 2(4),
based on knowledge mapping. International 447–470. doi:10.1504/IJBIDM.2007.016383
Journal of Technology Management, 28(7-8), Gerwin, D. (1979). The comparative analysis of
729–746. doi:10.1504/IJTM.2004.005780 structure and technology: A critical appraisal.
Checkland, P., & Holwell, S. (1998). Information, Academy of Management Review, 4(1), 41–51.
systems and Information Systems. Chichester, Goles, T., & Hirschheim, R. (1997). Intranets: The
UK: Wiley. next IS Solution? Information Systems Research
Crowston K. (2002). Electronic communication Center, College of Business Administration,
and new organizational forms. MIT working University of Houston.
papers series. Hills, M. (1997a). Intranet as groupware. New
Cyert, R., & March, J. (1963). A behavioral theory York, NY: John Wiley and Sons.
of the firm. Englewood Cliffs, NJ: Prentice Hall. Hills, M. (1997b). Intranet business strategies.
Dasgupta, S. (2001). Managing Internet and New York, NY: John Wiley and Sons.
Intranet technologies in organisations: Chal- Horton, R. P., Buck, T., Waterson, P. E., &
lenges & opportunities. Hershey, PA: Idea Group Clegg, C. W. (2001). Explaining Intranet use
Publishing. with the technology acceptance model. Journal
De Sanctis, G., & Scott Poole, M. (1994). Cap- of Information Technology, 16(4), 237–249.
turing the complexity in advanced technology doi:10.1080/02683960110102407
use: Adaptive structuration theory. Organization
Science, 5(2), 121–147. doi:10.1287/orsc.5.2.121
380
Karlsbjerg, J., & Damsgaard, J. (2001). Make or Mateo, R., Tanco, M., & Santos, J. (2011). Improv-
buy. A taxonomy of Intranet implementation strate- ing intranet knowledge transfer through resident
gies. Global Co-Operation in the New Millennium engineers. Journal of Knowledge Management,
the 9th European Conference on Information 15(1), 40–52. doi:10.1108/13673271111108684
Systems, Bled, Slovenia, June 27-29.
McGee, M. K. (1998, November 23). Strategic
Lamb, R., & Davidson, E. (2000). The new com- applications: Companies are forging a unique
puting archipelago: Intranet islands of practice. blend of custom and commercial software to gain
Proceedings of the IFIPWG8.2 Working Confer- a competitive advantage. Information Week.
ence on Information Technology and Changes in
Nelson, R., & Winter, S. (1982). An evolutionary
Organizational Work.
theory of economic change. Cambridge, MA:
Lederer, A. L., Maupin, D. J., Sena, M. P., & Harvard University Press.
Zhuang, Y. (2000). The technology acceptance
Norzaidi, M. D., Chong, S. C., Murali, R., & Sal-
model and the World Wide Web. Decision Support
wani, M. I. (2009). Towards a holistic model in
Systems, 29(3), 269–282. doi:10.1016/S0167-
investigating the effects of intranet usage on mana-
9236(00)00076-2
gerial performance: A study on Malaysian port
Leonard-Barton, D. (1988). Implementation industry. Maritime Policy & Management, 36(3),
as mutual adaptation of technology and or- 269–289. doi:10.1080/03088830902861235
ganization. Research Policy, 17(5), 251–267.
Norzaidi, M. D., Chong, S. C., & Salwani, M.
doi:10.1016/0048-7333(88)90006-6
I. (2009). Intranet usage, managerial satisfac-
Lloyd, P., & Boyle, P. (1998). Web-weaving: tion and performance impact: an empirical
Intranets, extranets and strategic alliances. But- analysis. International Journal of Business and
terworth - Heinemann. Systems Research, 3(4), 481–496. doi:10.1504/
IJBSR.2009.027201
Lyytinen, K., Rose, G., & Welke, R. (1998). The
brave new world of development in the inter- O’Boyle, P., Acton, T., Campion, M., Conboy, K.,
network computing architecture (InterNCA): Or & Scott, M. (2009). Towards a toolset for intranet
how distributed computing platforms will change evaluation. 22nd Bled eConference eEnablement,
systems development. Information Systems June 14 - 17. Slovenia: Bled.
Journal, 8(3), 241–253. doi:10.1046/j.1365-
Orlikowski, W. (1992). The duality of technology:
2575.1998.00037.x
Rethinking the concept of technology in organi-
March, J. (1991). Exploration and exploitation zations. Organization Science, 3(3), 398–427.
in organization learning. Organization Science, doi:10.1287/orsc.3.3.398
2(1), 71–87. doi:10.1287/orsc.2.1.71
Palmer, J. W. (2002). Web site usability, design,
Masrek, M. (2008). The effect of organiza- and performance metrics. Information Research,
tional and individual characteristics on corpo- 13(2), 151–167. doi:10.1287/isre.13.2.151.88
rate intranet utilizations. Information Manage-
Sampler, J. (1996). Exploring the relationship
ment & Computer Security, 16(2), 89–112.
between Information Technology and organiza-
doi:10.1108/09685220810879591
tional structure. In Earl, M. (Ed.), Information
management: The organizational dimension.
Oxford, UK: Oxford University Press.
381
Scheepers, R. (2003). Key roles in intranet imple- Clarke, K., & Preece, D. (2005). Constructing and
mentation: The conquest and the aftermath. Jour- using a company Intranet: It’s a very cultural thing.
nal of Information Technology, 18(2), 103–119. New Technology, Work and Employment, 20(2),
doi:10.1080/0268396032000101162 150–165. doi:10.1111/j.1468-005X.2005.00150.x
Tyre, M., & Orlikowski, W. (1994). Windows of Dasgupta, S. (2001). Managing Internet and in-
opportunity: Temporal patterns of technological tranet technologies in organizations: Challenges
adaptation in organizations. Organization Science, and opportunities. Hershey, PA: Idea Group.
5(1), 98–118. doi:10.1287/orsc.5.1.98
De Sanctis, G., & Fulk, J. (1999). Shaping orga-
Weick, K. (1990). Technology as equivoque: Sen- nization form: Communication, connection and
semaking in new technologies. In Goodman, P., & community. Thousand Oaks, CA: Sage.
Sproull, L. (Eds.), Technology and organizations.
Denton, D. K. (2003). Use an Intranet for per-
San Francisco, CA: Jossey Bass.
formance measurement. Industrial Management
Welch, E., & Pandey, S. (2007). E-government (Des Plaines), 45, 28.
and bureaucracy: Toward a better understanding
Denton, K. (2006). Strategic intranets: The
of intranet implementation and its effect on red
next big thing? Corporate Communica-
tape. Journal of Public Administration: Research
tions: An International Journal, 11(1), 5–12.
and Theory, 17(3), 379–404. doi:10.1093/jopart/
doi:10.1108/13563280610643507
mul013
Gebauer, J., & Schad, H. (1998). Building an
Internet-based workflow system. Berkeley, CA:
Fisher Center for Management and IT, working
ADDITIONAL READING
paper series n. 1030.
Aidemark, J. (2005). Implementing intranet for Golden, W., & Hughes, M. (2001). Business
social and cognitive knowledge processes. Inter- process reengineeering using Intranets: A new
national Journal of Healthcare Technology and beginning? 14th Bled Electronic Commerce
Management, 6(4/5/6), 357-367. Conference Bled, Slovenia, June 25 - 26.
Alleyne, C., Kakabadse, A., & Kakabadse, N. Griffiths, M. (2007). Oxygen: Social intranets,
(2007). Using the HR intranet: An exploratory collective intelligence, and government practices.
analysis of its impact on managerial satisfaction The Electronic. Journal of E-Government, 5(2),
with the HR function. Personnel Review, 36(2), 177–190.
295–310. doi:10.1108/00483480710726154
Hill, S., et al. (2005). Intranet redesign: Manage-
Beer, M. (2002). Building organizational fitness in ment and employee perspectives on usability. 18th
the 21st century. Harvard Business School working Bled eConference, Slovenia, June 6 - 8.
paper series, February.
Klecun, E., & Cornford, T. (2004). An inter-
Brynjolfsson, E., & Hitt, L. (2000). Beyond com- pretative evaluation of a healthcare intranet.
putation: Information Technology, organizational International Journal of Healthcare Technology
transformation and business performance. The and Management, 5(6), 407–421. doi:10.1504/
Journal of Economic Perspectives, 14(4), 23–48. IJHTM.2003.004163
doi:10.1257/jep.14.4.23
382
Lamb, R. (2002). Intranet boundaries: Social ac- KEY TERMS AND DEFINITIONS
tors and systems integration. 15th Bled Electronic
Commerce Conference, Slovenia, June 17-19. Collaborative Work: Support for team work-
ing carried out by the creation of a virtual space
Martini, A., & Pellegrini, L. (2008). Which in- (groupware, video conferencing and so on).
tranet for healthcare? Models and trends from Communication Channel: Intranet func-
a case-based approach. International Journal of tionalities that support information transfer and
Healthcare Technology and Management, 9(3), diffusion, which are in general independent in
213–230. doi:10.1504/IJHTM.2008.018214 respect of the user profile, and with a low degree
Okunoye, A., & Bada, A. (2006). Institutional of interactivity (e.g. bulletins, news, corporate
drivers of intranet use in a global context: Case guidelines, market reports, product specifications,
of a distributed international research organisa- internal documents, organizational charts).
tion. World Review of Science. Technology and Community: Support for communication and
Sustainable Development, 4, 73–85. doi:10.1504/ interaction between groups of users, for example
WRSTSD.2007.012661 solutions such as forums, chat, mailing lists and
instant messaging.
Olson, G., Malone, T., & Smith, J. (2001). Co- Document Management: Support for docu-
ordination theory and collaboration technology. ment publication and management, including
Mahwah, NJ: Lawrence Erlbaum Associates. research, recording and versioning functionalities.
Pellegrino, G. (2004). Intranet-based internal Evolutional Pattern: This deals with the study
communication: Routines, knowledge transfer of processes that transform economy for firms,
and learning strategies. International Jour- institutions, industries, employment, production,
nal of Human Resources Development and trade and growth, through the actions of diverse
Management, 4(4), 390–402. doi:10.1504/ agents from experience and interactions, using
IJHRDM.2004.005046 evolutionary methodology.
Intranet: This neologism identifies a system
Rose, J., & Scheepers, R. (2001). Organizational for communication, access and searching of busi-
Intranets: Cultivating Information Technology ness information, and is an Internet-based technol-
for the people by the people. Hershey, PA: Idea ogy (TCP-IP protocol and so on) that led to a new
Group Publishing. way of thinking about organizing and sharing an
Stenmark, D. (2006). Corporate intranet failures: organization’s intellectual and creative resources.
Interpreting a case study through the lens of for- Intranet Applications: An intranet can be
mative context. International Journal of Business analysed through six groups of applicative solu-
Environment, 1(1), 112–125. tions and functionalities: communication channel,
service platform, document management, com-
Szuprowicz, B. (1998). Extranets and Intranets. munity, collaborative work and process support.
E-commerce strategies for the future. Computer Organizational Issues and Information
Technology Research Corporation. System Success: These deal with organizational
Zeleny, M. (2000). The IEBM handbook of In- (pre-) conditions that contribute to information
formation Technology in business. London, UK: systems’ effectiveness.
Business Press. Service Platform: Services for employees,
for example facilities reservation, training, library
service, help desk, e-learning and online support.
383
ENDNOTES instant [t1], followed by implementation of

functionality “C” at instant [t2]; (...) case
1
By “progressive implementation” we mean a n: implementation of functionality “X” at
sequential integration of different function- instant [tz], followed by implementation of
alities (according to the taxonomy proposed) functionality “Y” at instant [tz+1] and so on.
in different temporal instants, where the 3
Legend of path graphs: the circle with a
temporal distance between [ tx ] and [ tx+1 ] letter inside indicates the implementation
is a medium or short period. Instead, by a of that functionality; the arrow indicates a
“big bang approach” we mean the parallel meaningful evolutional trend; to each ar-
implementation of more than one function- row corresponds a rectangle that contain
ality at the same time [ t1 ], corresponding the odds value corresponding to the related
to the launch of the intranet, but also to the evolutional passage and then the confidence
completion of its evolutional path (in the limits (alpha=0.10); at the opposite side, the
short and medium period). cut line represents resistance in respect of
2
Case 1: implementation of functionality “A” the related passage (negative association); an
at instant [t1], followed by implementation outlined line/circle represents a meaningful
of functionality “B” at instant [t2]; case but weak association.
2: implementation of functionality “A” at
384
385
Chapter 19
Designing an Information
Systems Performance
Management System:
The Case of an Insurance Corporation
Angela Perego
SDA Bocconi School of Management, Italy
ABSTRACT
Contribution of Information Systems (IS) to business has been widely debated among both business
scholars and practitioners. Even though a consistent body of literature has examined the problem over
a time frame of more than 20 years, and a plethora of theoretical contributions has been produced, the
issue of evaluating IS effectiveness remains unresolved. Starting from the assumption that real-world
experiences differ from theoretical explications, and with the intent to contribute to IS Performance
Management field bringing evidences from the reality, this chapter describes and discusses the design
of an IS performance management system implemented by an insurance corporation.
INTRODUCTION its contributions to the firm (Otley, 1999, Bourne

et al. 2003, Ferreira & Otley, 2009, Kaplan, 2009).
Performance evaluation is critical for all business In particular, the control and governance of inter-
functional departments (accounting, marketing nal services such as information systems (IS) has
and operations etc.); each department is involved in become quite critical in organizations due to the
performance measurement and must demonstrate large degree of expenditure and investment. IS
managers face mounting pressure to measure IS
DOI: 10.4018/978-1-4666-0170-3.ch019
Designing an Information Systems Performance Management System
department performance, justify the appreciable sumer surplus, or advantages in the supply
investment they require to operate, and evaluate chain;
IS in terms of tangible business value and return • IT-based value could be latent: there exists
on investment (ROI). a difference between creating value and
In the light of the depicted scenario, IS per- creating differential value;
formance management systems (PMS) should • There are numerous factors mediating IT
help IS departments to evaluate the outcomes of and value: there might be latencies in the
activities, practices and processes at all levels of manifestation of IT value;
the IS organization. They should also help IS de- • Causality for IT value is elusive: it is dif-
partments to face a serious credibility problem due ficult to fully capture and properly attribute
to lack of management practices that can provide the value generated by IT investments.
real benefits in business operations. Therefore, IS
PMS seem to be the right solution for the CIO Therefore, research in the IS business value
and IS department’s problems, but they are not field is far to a conclusion. There is the need of
so widespread in companies and implementations studies that propose practically applicable frame-
often fail to achieve full benefits (Miranda, 2004, work and methodologies, something that has also
Ariyachandra & Frolick, 2008, Aho, 2009). Pres- been identified as a limitation of several previous
ently, a robust and complete model with which studies (Leem et al., 2004, Petter, 2008). This lack
to evaluate IS Performance that practitioners can of knowledge increases the difficulties companies
apply in their companies does not exist (Gable et face in the evaluation of IS performance. In addi-
al., 2008). As scholars’ research has been able to tion, according to research in both PMS (Ferreira
define quantitative and perceptual measures to & Otley, 2009) and management control systems
assess the efficiency of IS, the issue of evaluating (Otley, 1999), the difficulty in implementing this
IS effectiveness remains unresolved. type of system has been determined by internal
A good summary of the current state of the art factors such as the culture and organizational
of research on IS business value is provided by tensions.
Kohli and Grover (2008) who, on the basis of an Notwithstanding these challenges, the criti-
extensive literature review, affirm that: cality of the issue leads companies to launch IS
PMS implementations even though they cannot
• IT does create value: a consistent mass of appropriately evaluate the results in economic
studies demonstrate that there is a relation- terms. A survey by Gartner (2009) reveals that
ship between IT and some aspect of firm while performance management is a high priority
value; for CIOs, at least one half of the companies that
• IT creates value under certain conditions: implement PMS in the next two years will fail to
to produce value, IT must be part of a realize the full benefits. Firms are therefore very
business value creation process where it interested in improving their understanding of
interacts with other IT and organizational IS PMS design, implementation and evaluation
resources; processes. They also seek guidance to help them
• IT-based value manifests itself in many during the critical challenges they will face and
ways: studies have shown so far that IT to exploit experiences and knowledge of other
value can reveal itself in several ways, like firms to improve their chances of success and their
productivity improvements, business pro- ability to evaluate IS performance. This chapter
cesses improvements, profitability, con- describes and discusses the design process of an
IS performance management system implemented
386
by an iInsurance corporation, in order to provide technology-related variables and performance. At

companies with recommendations to support them the industry level, on the other hand, the results are
in the design process of an IS PMS. mixed, with some studies documenting a positive
impact of IS (Kelley, 1994; Siegel & Griliches,
1992) while other studies detect no significant
BACKGROUND advantage to IS investments (Berndt & Morrison,
1995; Koski, 1999). Finally, at the more-detailed
The assessment of the real contribution of informa- firm level, a lot of studies present results that in-
tion systems (IS) to the firm, in terms of business dicate a positive relationship between technology
value, is the core of a wide and intense debate that and performance (Diewert & Smith, 1994; Hitt
has engaged, and still engages, both academics & Brynjolfsson, 1995; Dewan & Min, 1997). For
and practitioners for years. Interest in the debate example, a study by Lee et al. (2003) describes
has increased even though the conclusions of how knowledge management systems produce an
several studies in this area can be summed up organizational performance payoff that stimulates
using Robert Solow’s famous remark: “we see positive feedback to initial enablers, processes
computers everywhere except in the productiv- and intermediate outcomes. Furthermore, Aral
ity statistics” (Solow, 1987) and Nicholas Carr’s et al. (2006) found that enterprise resource plan-
affirmation: “IT doesn’t matter” (Carr, 2003). ning usage improved organizational performance,
Brynjolfsson called this phenomenon the “IT which enabled the subsequent adoption and use
productivity paradox” (Brynjolfsson, 1993) and of supply chain management and customer re-
suggested that traditional productivity measures lationship management systems, which further
may not be appropriate to estimate the contribu- improved performance.
tion of IS to business outcomes. Starting with Other researchers have moved the debate “from
Brynjolfsson’s studies, several other researchers the question of whether IT creates value to how,
have tried to examine the relationship between when and why benefits occur or fail to do so” (Soh
IS investments and organizational performance. & Markus, 1995 p.29) and focused their attention
A plethora of different research methodologies on the IS business value creation process. One
has been adopted in this research stream that of the first to move towards this new direction
also shows contributions coming from several was Weill (1992) who introduced the concept of
disciplines like economics, strategy, accounting, “conversion effectiveness” that represents the
operational research and, of course, information aspects of the firm’s climate which influence IS.
systems (Brynjolfsson & Hitt, 1996; Lee & Barua, In 1995, Markus and Soh gave an extremely
1999; Brynjolfsson & Hitt, 2003; Melville et al., relevant contribution to the debate proposing a
2004). Nevertheless, the connection between IS theoretical model of IS value creation. Their model
and productivity is still elusive (Devaraj & Kohli, synthesized prior contributions in a chain of three
2003). One reason could be the level of analysis at different process models which would specify a
the organizational level, which makes it difficult sequence of necessary (but not sufficient) condi-
to isolate the impact of any individual technology. tions that explains how the IS outcomes occur or
Devaraj & Kohli (2003) suggested that “the more not. “[…] organizations spend on IT and, subject
detailed the level of analysis, the better the chance to the varying degrees of effectiveness during
to detect the impact, if any, of a given technology” the IT management process, obtain IT assets.
(p.275). In confirmation of that, many economy- Quality IT assets, if combined with the process
level studies (Roach, 1987; Morrison & Berndt, of appropriate IT use, then yield favorable IT
1991) observed a negative relationship between impacts. Favorable IT impacts, if not adversely
387
affected during the competitive process, lead to business performance (Anthony, 1965; Simons,
improved organizational performance” (Soh & 1995; Neely, 1999; Otley, 1999; Ferreira & Otley,
Markus, 1995 p. 39). The main result of their 2009) and lesson learned coming from them. As a
study was to highlight the distance between IS matter of fact, they give relevant contributions in
investment and organizational performance. Since the definition of key performance measures and
then, a lot of researchers have undertaken studies in the design process of performance manage-
on the factors which lead to IS business value. A ment systems.
synthesis of the major highlights can be found To improve the understanding on the con-
in the “integrative model of IT business value” venient performance measure set, Otley (1999)
proposed by Melville, Kraemer & Gurbaxani pointed out that his field “has tended to restrict
(2004). They identified the organization as the itself to considering only financial performance,
locus of IS business value creation and pointed out and to use frameworks and theories drawn pri-
IS business value is generated by the employment marily from the discipline of economics” and
of IS resources and complementary organizational also concluded that management control systems
resources. They also emphasized the role of ex- “need also be studied as part of a wider organiza-
ternal factors (industry characteristics, trading tional control system.” Otley goes on to state that
partners and political, regulatory, educational, performance management “goes well beyond the
social and cultural context) in generating of IS traditional boundaries of management accounting.
business value. […] there is a need to understand the operational
Other research has also focused on the assess- activities of the organizations. […] there is a need
ment of IS contribution to firm performance and to connect control systems design with issue of
the definition of IS success measures. The first strategy […] there is a need to focus on the ex-
study that sought to impose some form of order on ternal context within which the organization is
IS researchers’ choices of success measures was set […] central is the value that an organization
DeLone & McLean (1992). They proposed an IS is delivering to its customers. […] performance
success model based on six distinct constructs of measurement practices need to be evaluated not
information system: system quality, information just from an economic perspective, but from a
quality, use, user satisfaction, individual impact, social, behavioural and managerial perspective,
and organizational impact. Pitt, Watson & Kavan within an overall organizational context” (Otley,
(1995) gave a relevant contribution to the develop- 1999 pp 380-381). This statement has been sup-
ment of the IS success model. They pointed out porting by several other researchers in accounting
that IS department has expanded its role from field (Simons, 1995; Kaplan & Norton, 1996; Fer-
product developer and operations manager to reira & Otley, 2009). Thus, researchers propose
service provider. Therefore the quality of the IS the adoption of the balanced scorecard (BSC)
department’s services, as perceived by its users, concept (Kaplan & Norton, 1996) to measure
is a key indicator of IS success which affects both IS business value and evaluate IS performance.
use and user satisfaction. As a matter of fact, the BSC has been applied in
According to DeLone & McLean “research- many industries and to various business activi-
ers should systematically combine individual ties (Park et al., 2005). Martinsons et al. (1999)
measures from the IS success categories to cre- developed a BSC for information systems that
ate a comprehensive measurement instrument” “allows managers to see the positive and negative
(DeLone & McLean, 1992 p. 88). The aim of impacts of IT applications and IS activities on the
developing a system to evaluate IS contribution factors that are important to the organization as
to business has to take into account studies on a whole” (p.85). They pointed out that measure-
388
ment is a prerequisite to management and, as a oriented and therefore their IT skills are not so
consequence, they proposed IS BSC as a strategic high, technology represents an opportunity, or
IS management tool that can be used to assess IS may be the only chance for insurance companies
contribution to business and guide performance to innovate their product and seek a competitive
improvement efforts. Therefore it suggests a new advantage, even if temporary.
point of view of IS evaluation which should support The corporation has one IS department for all
an IS Governance approach (Martinsons, 1999; the companies that come under the umbrella of
Saull, 2000; Van Grembergen, 2000). the parent company, and it reports directly to the
managing director. Furthermore, each company
has a small group which is only responsible for
CASE DESCRIPTION providing technical support on site. This IS de-
partment consists of three staff units: quality and
IG1 is a big Insurance Group with 770 agencies security policies unit, IT infrastructure standards
in Europe, 2812 employees and 3 billion euro of and policies, and administration and control. The
premiums collected in 2009. The Group, which IS department also consists of six other units cover-
consists of 8 companies, operates through a va- ing the various phases of a hypothetical IS service
riety of distribution channels: traditional agent, value chain namely: demand management, system
financial advisor, broker and bank channels and integration, software factory, IT infrastructure,
the new telephone and Internet channels. The application delivery and operations, and service
Group provides traditional insurance services desk. In this IS department, there are 181 internal
(automobile, life and other damages) but in re- IS professionals and 340 members of external staff,
cent years it has diversified the business towards with a body rental agreement. The choice to use
banking and real estate services. The IG Group body rental agreements instead of outsourcing
is not only present in Italy, but also in Spain and contracts is consistent with the group culture of
France, and its international presence is going to maintaining activities in house, and consequently,
increase in the years to come. Business strategy the necessary competences and skills: through
is focused on the following points: body rental, a portion of the human resource
costs becomes variable and flexible, without los-
• Business growth through acquisitions and ing control of IS activities and more importantly
diversification (i.e. banking and call cen- the skills and the autonomy to perform them. As
tres for health insurance services). a matter of fact, only the service desk activities
• Multichannel strategy, by strengthening have been outsourced, but the manager respon-
and exploiting agency relationships. sible for the service desk unit is an IG employee
• Profitability consolidation through Group who constantly monitors and controls the jobs
re-organization and business process performed by operators. Finally, there are some
re-engineering. external software providers that develop software
applications for IG, but this is justifiable because
IG’s organizational structure is a typical mix IG’s trend is to buy software applications avail-
of divisions and functions. The divisions refer to able in the marketplace and eventually acquire
the business line (automobile, life, non-life, bank, customizations from the software providers. This
etc.), whereas the functions are shared services IG’s business context complexity makes informa-
such as information systems, human resources, tion systems even more strategic and relevant as
accounts and a purchase department. Although far as maintaining its market share is concerned.
IG employees are not particularly technology- The IS department’s task therefore becomes more
389
difficult because an increase in IS strategic value ment system in order to learn how to deal with
leads to an increase in the complexity of IS re- these new challenges, management practices and
sources management and the need to consider IS work methods, and to be ready to answer future
as a service, and consequently the need to apply questions about IS performance and possible
service management rules and practices. service quality improvement.
The project started in March 2007 with the
Objectives and Boundaries goal of measuring the efficiency of the IS depart-
of IS PMS Project ment and the effectiveness of IS services in order
to have an initial assessment of IS performance.
In 2005, the IS department started to shift from According to the project origin, the evaluation
product driven organization to process driven perspective adopted in IG was that of the CIO,
organization. Therefore, IS service managers were even though the final scope is to share IS perfor-
introduced, being focused on business processes, mance results with managing directors of group
so as to dispose all-round business process knowl- companies in order to achieve agreement on the
edge, to go beyond a “silos” approach and be able convenient level of performance considering the
to provide services covering all users needs. This cost and benefit analysis, the measures therefore
approach should have simplified the relationship should be understandable to IS users.
between users and IS professionals because each The evaluation would be conducted from both
user essentially would have been just one service the department and individual perspective in order
manager to refer to, and should have provided the to build a complete picture of the effectiveness of
IS department with a better understanding of the IS services, and would use objective and perceptual
business complexity and thus they will be able to measures, even though the internal IS perspective
provide IS services, more aligned with user needs provides more relevant objective measures, which
in terms of supported activities and performance. are the subject of eventual negotiations with us-
Nevertheless, the IS department and IG organiza- ers to define the quality service level threshold.
tion were not ready to implement this change and The project was divided in three phases:
manage IS as a service or handle IS performance
indicators. IS people were not used to tracking 1. Design. The aim of this phase is to define
their activities or guaranteeing quality levels. The the IS performance measure set according
IS department could not measure and, as a result, to the company’s characteristics (strategy,
could not know much about its performances. So organizational structure, size, procedures,
it would have been difficult to discuss improve- etc.) and the algorithms to calculate the
ments in performance or in economic. On the other measures. To reach this goal, the team had
hand, top management and user departments were to:
not particularly interested in analysing how the a. Collect information on the company
IS department performed and consequently they context (internal processes, organiza-
did not waste time attempting to understand the tional procedures, etc.) and the external
peculiarity of the IS environment, even though environment (competitors, industry
often they complained about IS services. As a characteristics, etc.).
matter of fact, the company was doing well and b. Define the appropriate IS Performance
so the pressure to reach cost effectiveness and measure set to include in the IS PMS.
to improve level of quality was not so high, but c. Identify data necessary to calculate the
the situation was changing. As a result, the CIO measures.
decided to implement an IS performance manage-
390
2. Refining. The aim of this phase is to verify the definition of technological standards and the
the existence of data source and evaluate the monitoring of the technology market, allowing
convenience of creating new procedures to new technological opportunities to be identified.
gather data which is not available. Therefore Within this organizational model, IS units were
this phase can lead to the modification of free to adopt internal procedures and tools to
the IS performance measure set according handle daily tasks and plan their activities, if not
to data source that is or will be available. covered by official procedures. Nevertheless, the
3. Implementation. The aim of this phase is to interviews showed some contradictions between
create manual or automatic procedures to the descriptions made by managers in charge of
feed the IS PMS. the IS units and the official documents provided
by the quality and security policies unit.
This chapter focuses only on the design and Finally, the interviews showed a good level
refining phases. of cooperation among IS units and also with user
departments. In particular, the relationship with
Design Phase user departments and other companies of the
Group was handled by the demand management
The first step of the project was to gather infor- unit which consisted of service managers who
mation on the competition environment, business were in charge of collecting business require-
strategy and the organizational context. Subse- ments, supporting the system integration unit in
quently, the project team analysed the IS depart- the design of a solution, planning the activities
ment’s organization by studying the available with the software factory and updating users
documents and interviewing the IS professionals through work in progress situation reports. They
responsible for the various IS units in order to also asked the directors of user departments once
make an in-depth analysis of internal processes, a year to fill in a questionnaire so as to evaluate
practices and procedures in use and IS manage- their satisfaction.
ment tools. The IS department was an extremely After the context analysis, the team project
large organization in which task specialization was able to start the design process of the IS PMS
among IS units was well defined. Coordination adopting the balanced scorecard model. The four
among the several IS units was managed through key measurement areas were designed as follows:
formal procedures which should have reduced the
tension among them. These formal procedures • Business contribution and value, whose
also guaranteed quality and security, and they aim is to demonstrate to stakeholders how
were defined by the quality and security policies IS services support business strategic ob-
unit. As a result, the internal process were rather jectives, in terms of impacts on key busi-
well-structured, and corresponding management ness processes, IS cost dynamics and IS
tools were available (e.g. IT portfolio manage- portfolio prioritization.
ment, project management, software development, • Customer orientation, which measures the
application management and IS cost accounting). satisfaction of internal customers, SLA
On the contrary, the maturity level of IS person- compliance, the collaboration level be-
nel management tools was low. For example, tween IS department and user departments,
there were no up-dated job descriptions, skill how demand process is managed and ser-
maps and skill development plans. In order to vice support is delivered.
manage the complexity of the IS context, another • IT processes, which evaluates if IS pro-
essential part of the IS governance approach was cesses (IT sourcing, IT development and
391
implementation, IT security, IT operation The data necessary to calculate these measures

and service delivery, etc.) are efficient, should have been included in help desk manage-
complete and compliant to the best inter- ment, application management, project manage-
national practices. ment, human resources management and system
• Change and innovation, which inquires if and network management tools. It also required
IS department has got the necessary re- the IS staff timesheet, which was not available
sources (managerial and technical skills, at the time, in order to be able to divide IS staff
IT intelligence capabilities, etc.) or condi- time over the various activities.
tions (organizational climate or IT infra- The second measurement area was “customer
structure obsolescence) to deliver organi- orientation”. The project team started to design
zational and technical innovation. customer satisfaction measures and defined the
following relevant measures:
The project team initially focused on the “IS
processes” measurement area, in particular on • Index of user satisfaction referred to IS de-
system quality, speed in execution, project man- partment activities.
agement capability, IT sourcing efficiency and • Index of respect for the service level agree-
IS staff workload. ment settled with user departments.
The project team identified the following
measures as relevant: The calculus of these measures would have
required first of all the definition of a method with
• Server unavailability percentage. which to evaluate user satisfaction, and secondly
• Network unavailability percentage. the establishment of a service level agreement
• Number and severity of incidents and which in turn required the definition of the IS
malfunctions. service catalogue.
• Number of application bugs. The third analysed measurement Area was
• Index of IS reliability. “change and innovation”. The project team worked
• Percentage of incidents solved by first line on the following measurement categories: the
Support Help Desk. organizational climate, the permanent education
• Average time to answer user calls. of IS staff, the expertise and skill of IS staff and
• Average resolution time for type of age of the application portfolio. In particular, it
incidents. decided to use the following measures:
• Percentage of projects “on time”.
• Percentage of projects “on budget”. • IS department turnover.
• Percentage of external FTE. • IS department Absenteeism rate.
• Concentration index of IS suppliers. • Average age of IS staff.
• Percentage of contracts with IS suppliers • Number of educational days per person.
including SLA. • Percentage of IS budget allocated to
• Percentage of IS hours per type of applica- education.
tion (operational and managerial). • Number of years of IS experience per staff
• Percentage of maintenance and evolution- member.
ary maintenance activities. • Percentage of necessary skills covered by
• Backlog. IS staff.
• Saturation index of IS capacity. • Number of applications for age category.
• Average age of software applications.
392
• Average age of hardware. Refining Phase
Necessary data would have been included in In the refining phase the project team performed
the IS staff skills and development management two types of actions required before starting the
tool, the Application Management tool and IS implementation of the IS performance manage-
cost Accounting. ment system: analysis of data source identified in
The last area was “business contribution and the design phase and the construction of new tools
value”. The project team identified the appropriate and procedures (i.e. the IS service catalogue and
dimensions for this area as being contribution to IS staff timesheet) to gather required data which
achieving goals, business value of IS projects and was not already available. Some data was not
IS cost control. Some problems arose in the design available in the management tools, for example
of measures for the evaluation of IS contributions the project management system did not include
to the achievement of business goals due to the a classification of projects on the grounds of the
difficulty in directly linking IS investments to type of business process and success key factors,
business outcomes. The solution reached consisted and information on skills was not available in the
of seeking to use measures that showed how much human resource system. Instead, other data, like IS
businesses use IS in order to be competitive. Thus costs data, was scattered over several spreadsheets
the project team defined as measures: and required manual elaborations. In addition, IS
staff members proposed changes to highlight dif-
• Number of new products based on IS ferent aspects of the same phenomenon in order
services. to reduce possible criticism regarding their jobs.
• Number of critical applications. This behaviour showed that the IS units were used
• Percentage of projects on innovative chan- to working as a black box and not giving evidence
nel (e.g. bank and web). of what went on within the units, and thus they
were worried about performance measures which
Whereas, the following measures were defined could have highlighted inefficiency in their units.
to cover the other dimensions: The collection of customer satisfaction data,
for example, proved to be a very critical point.
• Percentage of projects focused on critical The IS department did not wish to involve users
processes. in this project or carry out a survey which would
• Percentage of projects on strategic goals. have exposed the IS department to requests
• IS costs (investments and operating ex- for explanations and, as a result, the sharing of
penses) to support success key factors. measures, defined IS services and correspond-
• IS costs per staff member. ing service levels with the inevitable request of
• IS costs distributed across innovation, change in order to consider the IS performance
growth and run activities. management system as a company tool through
• Percentage of IS expenses above or within which to manage IS services and user needs. This
budget. was not due to a bad relationship between user
departments and the IS department, but rather
The data necessary to calculate the measures the IS department’s desire to avoid sharing infor-
presented above should have come from IS costs mation with users that could have modified the
accounting, application management and project power balance in favour of user departments. As
management systems. a matter of fact, the IS department was not really
under pressure from user departments to become
393
Table 1. IG IS performance management system
“Business Contribution and Value” “Customer Orientation”

Contribution to achieving business goals: User Satisfaction:
• Number of new products based on IS services. • Index of user satisfaction referred to IS department activities.
• Number of critical applications. • Index of respect for the Service Level Agreement settled with
• Percentage of projects on innovative channels (bank, web, etc.). user departments.
Business Value of IS projects: • Percentage of projects “on time” and “on budget” per type of
• Percentage of projects focused on critical processes. project.
IS Cost control: Partnership with user:
• IS costs to support Success Key Factors • Percentage of IS hours charge for meeting with users.
• IS costs distributed across innovation, growth and run activities.
• IS costs per staff member.
• IS costs per User department and Company.
“Change and Innovation” “IS processes”
Organizational Climate: System Quality:
• IS department Absenteeism rate. • Server unavailability percentage.
• IS department turnover. • Network unavailability percentage.
• Average age of IS staff. • Number and severity of incidents and malfunctions.
Permanent Education of IS Staff: • Number of application bugs.
• Number of educational days per person. • Index of IS reliability.
• Percentage of IS budget allocated to education. Project Management Capability:
Expertise and skill of IS Staff: • Percentage of projects “on time”.
• Number of years of IS experience per staff member. • Percentage of projects “on budget”.
Age of the application portfolio: Speed in execution:
• Number of applications for age category. • Percentage of incidents solved by first line Support Help Desk.
• Average age of software applications. • Average time to answer user calls.
• Average age of hardware. • Average resolution time for type of incidents.
IS sourcing efficiency:
• Concentration index of IS suppliers.
• Percentage of contracts with IS suppliers including the SLA.
IS Staff workload:
• Percentage of external FTE.
• Percentage of IS hours allocated to maintenance and evolution-
ary maintenance activities
• Backlog.
• Saturation index of IS capacity
faster or save money, and thus the IS department management, and system and network systems)
was not focused on pursuing efficiency. Sharing and discussed the opportunity to improve them
performance measures would have meant reducing in order to collect more data and to automatically
the information asymmetry on IS subjects and al- feed the IS performance management system. The
lowing users to ask for more. Finally, data derived CIO decided not to make specific investments in it,
from the yearly questionnaires, which were filled but to include the necessary improvement in any
in on demand by managers and the directors of future projects regarding IS management tools.
user departments, was used to calculate the user Table 1 shows the final set of IS performance
satisfaction index. Therefore, the project team measures that was implemented in IG.
provided an in-depth analysis of problems that
arose and evaluated new opportunities to enrich
the IS measure set. The project team also made Discussion and Recommendations
an in-depth analysis of the existing IS manage-
ment tools (e.g. IS cost accounting, portfolio The IG case describes the design of an IS PMS
management, project management, application and the factors that affect it. In particular, the
394
analysis of the case shows the final IS measure management system for two reasons. The first
set is affected by several variables related to IS reason is that it affects the maturity level of the IS
department organization (role, size, maturity department in terms of the standards and policies
etc.) and organizational tensions (e.g. between definition, procedures formalization and the use of
IS department and User department). management tools. The second reason is related
The role of IS appears to be extremely rel- to relational complexity that increases with the
evant in the design of IS performance measures. size of the IS department. As a result, the size of
IS played a strong strategic role, and as a result, the IS department also affects the climate inside
IS departments were able to translate business the IS department.
strategy into IS strategy and link IS activities and The maturity of the IS department is also found
projects to success key factors. The perception of to affect the shape of the IS performance manage-
relevance and the motivation to design specific ment system in an extremely strong way because
measures connected to business strategy were the availability of input data necessary to calculate
extremely high. the measures depends on it. Evidence of a high
Another variable which has been shown to af- maturity level is firstly seen in having formal and
fect the IS measure set according to the analysis structured rules and procedures, and management
of the case is IS sourcing strategy. As a matter of tools which provide all the data necessary to moni-
fact, the outsourcing of a relevant part of the IS tor and manage the IS activities. In this situation,
activities, e.g. the help desk and new software the effect of the IS department’s maturity would
development, led to these activities not being be very low because all necessary data would be
considered for measurement in the “IS processes” immediately available or after small changes to
area. In addition it determined the introduction of the respective systems. The sophistication of IS
a specific dimension to the “IS processes” area, performance measures depends on the available
i.e. IS sourcing efficiency with its corresponding data and, as a consequence, on the existence of
measures (concentration index of IS suppliers management tools and formal procedures that
and percentage of contracts with the IS supplier require the production of reports and documents
including the SLA). Finally, the body rental use in order to collect all the data and information
led to the percentage of external FTE being added needed to manage IS activities. In IG case, for
as a new measure. example, it was not possible to allocate IS staff
The size and structure of the IS department time to the IS activities categories so as they used
directly affects measures that relate to the organi- estimate, but the measures reliability was lower.
zational climate, permanent education, expertise IG had also problems in evaluating the level of
and skill of IS staff. In fact, in small departments correspondence between business requirements
measures such as the index of turnover and absen- and developed software applications so as they
teeism are not meaningful as they generally equal had to improve their project management tools
zero. Secondly, measures relating to permanent in order to maintain this measure.
education could also be less relevant because the Nevertheless, IG case shows that the design of
IS budget allocated for these activities is usually the IS performance measures was an opportunity
low and thus the opportunity to attend training or to verify and improve the supervision and manage-
management courses is not so frequent. Finally, ment of IS processes. For example, the definition
in such a small department, it is not so common of sophisticated performance measures for project
to perform an analysis of IS staff skills and plan management triggers the introduction of more
skill development courses. Size also has an indi- precise checks on IS projects and processes. At
rect influence on the shape of the IS performance the same time, the automatic report on the help
395
desk activities was enhanced by the inclusion of and “customer orientation”. This good climate
customer satisfaction indexes or the recognition avoids the likelihood of the IS staff considering
of the causes of IS malfunctions. Finally, the pos- the IS evaluation project as an exam and the IS
sibility to intervene to improve the IS accounting performance management system as a control
tool seems to be helpful (e.g. the introduction of tool seeking to manipulate the design of the
new items to attribute IS costs directly to an IS measure, especially referred to the efficiency of
process or to a project, to distinguish between run IS processes, so as to highlight only some aspects
activities, maintenance activities and innovative and not others, as happened in IG case. Finally, a
activities). good climate favours the possibility of introduc-
In confirmation of the above, CIO has thought ing a specific section on the IS department to the
of the opportunity to enhance their management user evaluation due to the fact that IS employees
tools. However, in IG case, the improvements are ready to work together on the results without
were limited in confirmation of the fact that the allocating the responsibility of possible negative
IS performance management system was designed feedback to each other.
for future needs and it is not the priority at the time. The IG case also shows that climate and
According to the analysis of the case, one of power balances between the IS department and
the main results of the design and development user departments has an impact on the decision
of the IS performance management system is not to share the results of the IS evaluation with us-
only the definition of IS measures but the devel- ers. In particular, an apparent good climate is not
opment of a solid system of IS governance and enough because power balances are also relevant
systems that produce input data. The CIO did not in order to decide whether or not to involve users
consider it to be a by-product of the project but in the project. IG’s IS department decided not to
rather one of its main outputs. In other word, the involve users because doing so would have given
design and development of the IS performance them useful information which could have changed
management system lead to a real diagnostic ef- the power balance. At the time, IS users were
fort on IS management systems and policies, and not really interested in understanding IS activi-
encouraged an effort to re-engineer IS processes ties and they trusted the IS department. Sharing
and IS related policies. information about IS performance would have
This IS performance management system be- increased the users’ knowledge of the IS depart-
came the decision point for the design of further IS ment, and thus made the users more powerful in
development activities, and became an IS manage- a possible negotiation on project timescales and
ment cockpit for the CIO. This helped to establish costs, evolutionary maintenance activities and
an opportunity for continuous improvement that the like. IS department of IG was more powerful
links performance measurement to analysis, and than user department due to the lack of interest
to decisions of possible actions. Improvement of users, which led users to not negotiate better
activities are normally tactical and not strategic, IS service levels with IS department. Therefore
as they deal with the modification of resource IS department wanted to maintain this advantage.
allocation, for instance, an increase in help-desk In the same way, power can also affect the
resources or the redefinition of measures to evalu- choice of IS measures in order to maintain, for as
ate IS programs and projects after the evaluation long as possible, the existing information asym-
of the “on time” and “on budget” measures. metry between the IS department and user depart-
The effects of a good climate on IS performance ments. Therefore, the IS department does not seek
measures are shown in different measurement to highlight all the possible process measures so
areas: “change and innovation”, “IS processes”
396
that it has not controlled part of the process that In addition, the IG case has the limit to be
can guarantee a buffer for possible inefficiencies. conducted involving only IS staff in decision
The analysis of the case also highlight that making process. The users’ involvement would
IS department was aware of the necessity of have changed the final IS Performance measure
employing business measures, which are easily set because the user perspective would have been
understandable to users audience, rather than more relevant and the role of power balance would
technical ones in order to reach this goal. This have been different. Therefore further research
aspect is quite important because IS department could involve both IS staff and users in order to
had some problems to build understandable investigate how the final IS measure set is affected
measures. They had to adopt user language and by users. In order to improve the understanding
also to consider the users’ technical skills and of IS performance management systems, future
background in order to provide meaningful and research should also investigate the impact of
understandable IS performance measures to users. these systems on IS management activities over
That had a relevant impact on the IS measures time. In particular it is possible to approach the
design because the focus was more on IS service research from two perspectives: on the one hand
and less on IS processes. As a matter of fact, in it would be relevant to study how IS performance
IG case, the first planned action was to define an management systems change and improve in
IS Service Catalogue, which was the basis of IS order to become more effective and aligned with
performance measures understandable to users. company’s needs, on the other hand it would be
Finally, the analysis of the case shows that the useful to evaluate their real contribution and sup-
design of the IS performance management system port to IS improvement decisions.
triggers the opportunity to start the communication
and internal marketing of IS to internal customers
and can help the IS department to establish better CONCLUSION
its relationship, reputation and trustworthiness.
This chapter describes a case of the design of IS
PMS. It presents the experience of IG company
FUTURE RESEARCH DIRECTIONS and the difficulties that arose in terms of necessary
organizational preconditions and required input
This chapter has attempted to contribute to the IS data, and thus it can give some recommendations
performance management system field by bringing to companies which intend to launch this type of
evidences from real-world experience. In particu- projects. Reviews of the literature and analyses
lar, it highlights the factors that affect the design of the studies reveal that all IS assessment models
of an IS PMS, using the results emerging from the aim to provide the most relevant, reliable, and
analysis of IG Case. Therefore, the generalization representative set of IS performance dimensions
of research findings is not possible. However, the and measures that refer to the specific internal
transferability of research results can only occur and external environment of the organization
in similar “structural configurations” according considered. This objective implies clarity and a
to the model proposed by Modol & Muniategui willingness to pursue organizational goals rather
(2008). To enforce the research results and apply than private aims, and that only one appropriate set
them to other organizational contexts, further of dimensions and measures exists for a company.
research could widen the research to other cases Whereas the IG case shows that individual, unit
following the multiple cases approaches proposed or department interests affect the choice of the IS
by Yin (2009). performance measure set. In addition, variables
397
related to IS context (role, IS department size, Ariyachandra, T. R., & Frolick, M. N. (2008).
maturity etc.) have a really relevant impact on the Critical success factors in business performance
choice of IS performance measures. Finally, also management—Striving for success. Informa-
relational variables such as the climate between tion Systems Management, 25(2), 113–120.
the IS department and user departments and the doi:10.1080/10580530801941504
existence of organizational tension, affect the
Berndt, E., & Morrison, C. (1995). High-tech
quality of IS performance systems because they
capital formation and economic performance in
can lead people involved in the design process
U.S. manufacturing industries: An exploratory
to take the defensive rather than promote a col-
analysis. Journal of Econometrics, 65, 9–43.
laborative context.
doi:10.1016/0304-4076(94)01596-R
Therefore, the IG case shows that the design
of an IS performance management system is an Bourne, M., Franco, M., & Wilkes, J. (2003).
iterative process. Starting from an appropriate Corporate performance management. Mea-
set of IS performance dimensions and measures, suring Business Excellence, 3(3), 15–21.
it is possible to define, in an iterative cycle, doi:10.1108/13683040310496462
what actions are necessary to implement it, how
Brynjolfsson, E. (1993). The productivity paradox
variables related to organizational and relational
of IT. Communications of the ACM, 36(12), 66–77.
environment offer resistance to the construction
doi:10.1145/163298.163309
of missing input and attempt to modify the ap-
propriate set, and in what way actions could be Brynjolfsson, E., & Hitt, L. (1996). Paradox lost?
taken to find a solution to the problems that have Firm-level evidence on the returns to Information
arisen and avoid the modification of the set of Systems spending. Management Science, 42(4),
dimensions and measures. 541–558. doi:10.1287/mnsc.42.4.541
Brynjolfsson, E., & Hitt, L. (2003). Computing
productivity: Firm-level evidence. The Review
REFERENCES
of Economics and Statistics, 85(4), 793–808.
Aho, M. (2009). A capability maturity model for doi:10.1162/003465303772815736
corporate performance management, an empirical Carr, N. (2003). IT doesn’t matter. Harvard Busi-
study in large Finnish manufacturing companies. ness Review, 81(5), 41–49.
Proceedings from the eBRF 2009- A Research
Forum to Understand Business in Knowledge DeLone, W. H., & McLean, E. R. (1992). Informa-
Society, Jyväskylä, Finland, September 23 - Sep- tion Systems success: The quest for the dependent
tember 25, 2009. variable. Information Systems Research, 3(1),
60–95. doi:10.1287/isre.3.1.60
Anthony, R. (1965). Planning and control systems:
A framework for analysis. Boston, MA: Harvard Deveray, S., & Kohli, R. (2003). Performance
Business Press. impact of Information Technology: Is actual usage
the missing link? Management Science, 49(3),
Aral, S., Brynjolfsson, E., & Wu, D. J. (2006). 273–289. doi:10.1287/mnsc.49.3.273.12736
Which came first, IT or productivity? The virtuous
cycle of investment and use in enterprise systems.
Twenty-Seventh International Conference on
Information Systems, (pp. 1819-1840).
398
Dewan, S., & Min, C. (1997). The substitution Kohli, R., & Grover, V. (2008). Business value of
of Information Technology for other factors IT: An essay on expanding research directions to
of production: A firm level analysis. Manage- keep up with the times. Journal of the Association
ment Science, 43(12), 1660–1675. doi:10.1287/ for Information Systems, 9(1), 23–39.
mnsc.43.12.1660
Koski, H. (1999). The implications of network
Diewert, E. W., & Smith, A. M. (1994). Productiv- use, production network externalities and public
ity measurement for a distribution firm. National networking programmes for firm’s productivity.
Bureau of Economic Research working paper no. Research Policy, 28(4), 423–439. doi:10.1016/
4812, Washington, D.C. S0048-7333(98)00127-9
Ferreira, A., & Otley, D. (2009). The design and Lee, B., & Barua, A. (1999). An integrated
use of performance management systems: An ex- assessment of productivity and efficiency
tended framework for analysis. Management Ac- impacts of Information Technology invest-
counting Research, 20(4), 263–282. doi:10.1016/j. ments: Old data, new analysis and evidence.
mar.2009.07.003 Journal of Productivity Analysis, 12, 21–43.
doi:10.1023/A:1007898906629
conceptualizing system success: The IS-impact Lee, H., & Choi, B. (2003). Knowledge man-
measurement model. Journal of the Association agement enablers, processes, and organizational
for Information Systems, 9(7), 377–408. performance: An integrative view and empirical
examination. Journal of Management Information
Gartner, (2009). Gartner EXP worldwide survey
Systems, 20(1), 179–228.
of more than 1,500 CIOs shows IT spending to
be flat in 2009. Retrieved April 22, 2010, from Leem, C. S., Yoon, C. Y., & Park, S. K. (2004). A
http://www.gartner.com/it/page.jsp?id=855612 process-centered IT ROI analysis with a case study.
Information Systems Frontiers, 6(4), 369–383.
Hitt, L., & Brynjolfsson, E. (1995). Productiv-
doi:10.1023/B:ISFI.0000046378.42438.5f
ity, business profitability, and consumer surplus:
Three different measures of information technol- Martinsons, M., Davison, R., & Tse, D. (1999). The
ogy value. Management Information Systems balanced scorecard: A foundation for the strategic
Quarterly, 20(2), 121–142. doi:10.2307/249475 management of Information Systems. Decision
Support Systems, 25, 71–88. doi:10.1016/S0167-
Kaplan, R. (2009). Measuring performance
9236(98)00086-4
(Pocket Mentor). Boston, MA: Harvard Busi-
ness Press. Melville, N., Kraemer, K., & Gurbaxani, V. (2004).
Information Technology and organizational per-
Kaplan, R., & Norton, D. (1996). The balanced
formance: An integrative model of IT business
scorecard: Translating strategy into action. Bos-
value. MIS Quarterly Review, 28(2), 283–322.
ton, MA: Harvard Business School Press.
Miranda, S. (2004). Beyond BI: Benefiting from
Kelley, M. (1994). Productivity and Information
corporate performance management solutions.
Technology: The elusive connection. Manage-
Financial Executive, 20(2), 58–61.
ment Science, 40(11), 1406–1425. doi:10.1287/
mnsc.40.11.1406 Modol, J. R., & Muniategui, F. S. (2008). Tackling
the problem of transferability in IS qualitative
research. Paris: ICIS Proceedings.
399
Morrison, C., & Berndt, E. (1991). Assessing the Simons, R. (1995). Levers of control: How man-
productivity of information technology equipment agers use innovative control systems to drive
in U.S. manufacturing industries. National Bureau strategic renewal. Boston, MA: Harvard Business
of Economic Research working paper no. 3582, School Press.
Washington, D.C.
Soh, C., & Markus, M. L. (1995.) How IT creates
Neely, A. (1999). The performance measurement business value: A process theory synthesis. Pro-
revolution: Why now and what next? International ceedings of the Sixteenth International Conference
Journal of Operations & Production Management, on Information Systems.
19(2), 205–228. doi:10.1108/01443579910247437
Solow, R. S. (1987). We’d better watch out. New
Otley, D. (1999). Performance management: York Times Book Review.
A framework for management control systems
Van Grembergen, W. (2000). The balanced score-
research. Management Accounting Research, 10,
card and IT governance. Information Systems
363–382. doi:10.1006/mare.1999.0115
Control Journal, 2, 40–43.
Park, J. H., Lee, J. K., & Yoo, J. S. (2005). A
Weill, P. (1992). The relationship between in-
framework for designing the balanced supply
vestment in Information Technology and firm
chain scorecard European. Journal of Information
performance: A study of the value manufactur-
Systems, 14(4), 335–346. doi:10.1057/palgrave.
ing sector. Information Systems Research, 3(4),
ejis.3000544
307–333. doi:10.1287/isre.3.4.307
Petter, S., DeLone, W., & McLean, E. (2008).
Yin, R. K. (2009). Case study research: Design
Measuring Information Systems success: Models,
and methods (4th ed.). SAGE Publications.
dimensions, measures, and interrelationships.
236–263. doi:10.1057/ejis.2008.15
ADDITIONAL READING
Pitt, L. F., Watson, R. T., & Kavan, C. B. (1995).
Service quality: A measure of Information Systems Cameron, K. S., & Whetten, D. A. (1983). Some
effectiveness. Management Information Systems conclusions about organizational effectiveness.
Quarterly, 19(2), 173–188. doi:10.2307/249687 In Cameron, K. S., & Whetten, D. A. (Eds.),
Roach, S. (1987). America’s technology dilemma: Organizational effectiveness: A comparison of
A profile of the information economy. Special multiple models. New York, NY: Academic Press.
economic study. New York, NY: Morgan Stanley. Chan, Y. E. (2000). IT value: The great divide be-
Saull, R. (2000). The IT balanced scorecard – A tween qualitative and quantitative and individual
roadmap to effective governance of a shared ser- and organizational measures. Journal of Manage-
vices IT organization. Information Systems Control ment Information Systems, 16(4), 225–261.
Journal, 2, 31–38. doi:10.1023/A:1010041819361 Davern, M. J., & Wilkin, C. L. (2010). Towards an
Siegel, D., & Griliches, Z. (1992). Purchased integrated view of IT value measurement. Interna-
services, outsourcing, computers, and productivity tional Journal of Accounting Information Systems,
in manufacturing. In Griliches, Z. (Ed.), Output 11, 42–60. doi:10.1016/j.accinf.2009.12.005
measurement in the service sectors (pp. 429–458).
Chicago, IL: University of Chicago Press.
400
Grover, G., Jeong, S. R., & Segars, A. H. (1996). Saunders, C. S., & Jones, J. W. (1992). Measuring
Information Systems effectiveness: The construct performance of the information systems function.
space and patterns of application. Information & Journal of Management Information Systems,
Management, 31, 177–191. doi:10.1016/S0378- 8(4), 63–82.
7206(96)01079-8
Scheepers, H., & Scheepers, R. (2008). A pro-
Myers, B. L., Kappelman, L. A., & Prybuto, V. cess-focused decision framework for analyzing
R. (1997). Comprehensive model for assessing the business value potential of IT investments.
the quality and productivity of the Information Information Systems Frontiers, 10(3), 321–330.
System function. In Garrity, E., & Sanders, L. doi:10.1007/s10796-008-9076-5
(Eds.), Toward a theory for Information Systems
Seddon, P. B., Staples, S., Patnayakuni, R., &
assessment: Information Systems success mea-
Bothwell, M. (1999). Dimension of Information
surement. Hershey, PA: IDEA Group Publishing.
System success. Communication of AIS, 20(2),
Neely, A. (1995). Performance measurement 2–39.
system design: Theory and practice. International
Simons, R. (2000). Performance measurement &
Journal of Operations & Production Management,
control systems for implementing strategy. Upper
15, 80–116. doi:10.1108/01443579510083622
Saddle River, NJ: Prentice Hall.
Oh, W., & Pinsonneault, A. (2007). On the assess-
Strassmann, P. (1990). The business value of
ment of the strategic value of Information Tech-
computers. The Information Economics Press.
nologies: Conceptual and analytical approaches.
Management Information Systems Quarterly, Tallon, P. P. (2007). Does IT pay to focus? An
31(2), 239–265. analysis of IT business value under single and
multi-focused business strategies. The Journal of
Parasuraman, A., Zeithaml, V. A., & Berry, L. L.
Strategic Information Systems, 16(3), 278–300.
(1988). SERVQUAL: A multiple-item scale for
doi:10.1016/j.jsis.2007.04.001
measuring consumer perceptions of service qual-
ity. Journal of Retailing, 64(1), 12–40. Tanriverdi, H. (2006). Performance effects of
Information Technology synergies in multibusi-
ness firms. Management Information Systems
Quarterly, 30(1), 57–77.
tion Systems Research, 13(1), 50–69. doi:10.1287/ Tippins, M. J., & Sohi, R. S. (2003). IT competency
isre.13.1.50.96 and firm performance: Is organizational learning
a missing link? Strategic Management Journal,
Ranganathan, C., & Jha, S. (2008). Do CIOs Mat-
24(8), 745–761. doi:10.1002/smj.337
ter? Assessing the value of CIO presence in top
management teams. ICIS 2008 Proceedings, Paris. Van Grembergen, W., & De Haes, S. (2009).
Enterprise governance of Information Technol-
Sabherwal, R., Jeyaraj, A., & Chowa, C. (2006).
ogy: Achieving strategic alignment and value.
Information System success: Individual and or-
New York, NY: Springer. doi:10.1007/978-0-
ganizational determinants. Management Science,
387-84882-2
52(12), 1849–1864. doi:10.1287/mnsc.1060.0583
Weill, P., & Olson, M. H. (1989). An assessment
of the contingency theory of management Infor-
mation Systems. Journal of MIS, 6(1), 59–85.
401
KEY TERMS AND DEFINITIONS Objective Measure: The measurement of

objective phenomena (e.g. time response, data
IS Business Value: The organizational per- quality, etc.)
formance impacts of IS, including productivity Perceptual Measure: The measurement of end
enhancement, profitability improvement, cost user perceptions of IS efficiency and effectiveness.
reduction, competitive advantage, inventory Performance: The result of an action or to
reduction and other measures of performance. the way in which someone or something func-
IS Performance: The efficiency and effective- tions. Usually the performance is related to the
ness of IS actions. accomplishment of a target.
IS Performance Management System: the set
of measures used to quantify both the efficiency
and effectiveness of IS actions. ENDNOTE
1
IG is a fictitious name.
402
403
Compilation of References
Abbad, M., Morris, D., & de Nahlik, C. (2009). Looking Agrawal, E., Jain, P., & Jain, V. (2010). A comparative
under the bonnet: Factors affecting student adoption of study of ISO 9001, CMMI and Six Sigma with reference
e-learning systems in Jordan. International Review of to software process quality. Global Journal of Enterprise
Research in Open and Distance Learning, 10(2), 1–25. Information System, 1(1).
Abrahamsson, P., Salo, O., Ronkainen, J., & Warsta, J. Ahmadjian, V., & Paracer, S. (2000). Symbiosis: An
(2002). Agile software development methods: Review and introduction to biological associations. Oxford, UK:
analysis (No. 478). Espoo, Finland: Technical Research Oxford University Press.
Centre of Finland (VTT Publications.
Ahmed, H., Daim, T., & Basoglu, N. (2007). Infor-
Abramo, G., D’Angelo, C. A., & Caparasecca, A. (2009). mation Technology diffusion in higher education.
Allocative efficiency in public research funding: Can Technology in Society, 29(4), 469–482. doi:10.1016/j.
bibliometrics help? Research Policy, 38(1), 206–215. techsoc.2007.08.011
doi:10.1016/j.respol.2008.11.001
Aho, M. (2009). A capability maturity model for corporate
Abran, A., & Buglione, L. (2003). A multidimensional performance management, an empirical study in large
performance model for consolidating balanced scorecards. Finnish manufacturing companies. Proceedings from the
Advances in Engineering Software, 34(6), 339–349. eBRF 2009- A Research Forum to Understand Business
doi:10.1016/S0965-9978(03)00033-4 in Knowledge Society, Jyväskylä, Finland, September
23 - September 25, 2009.
Adam, F., & Sammon, D. (2004). The enterprise resource
planning decade: Lesson learned and issues for the future. Ainsworth, H. (1999). Why do projects break down?
Hershey, PA: Idea Group Publishing. Charter, 70(February), 46.
Adams, A., & Sasse, M. A. (1999). Users are not the Ajzen, I. (1991). The theory of planned behavior. Organi-
enemy. Communications of the ACM, 42(12), 40–46. zational Behavior and Human Decision Processes, 50(2),
doi:10.1145/322796.322806 179–211. doi:10.1016/0749-5978(91)90020-T
Adeyinka, T., & Mutula, S. (2010). A proposed model for Ajzen, I., & Fishbein, M. (1969). The prediction of
evaluating the success of WebCT course content man- behavioral intentions in a choice situation. Journal
agement system. Computers in Human Behavior, 26(6), of Experimental Social Psychology, 5(4), 400–416.
1795–1805. doi:10.1016/j.chb.2010.07.007 doi:10.1016/0022-1031(69)90033-X
Agarwal, R., & Prasad, J. (1999). Are individual differ- Akkermans, H., & Helder, K. (2002). Vicious and virtuous
ences germane to the acceptance of new Information cycles in ERP implementation: A case study of interre-
Technologies? Decision Sciences, 30(2), 361–391. lations between CSF. European Journal of Information
doi:10.1111/j.1540-5915.1999.tb01614.x Systems, 11, 35–46. doi:10.1057/palgrave/ejis/3000418
Aksnes, D. W., & Taxt, R. E. (2004). Peers reviews and Al-Qirim, N. (2003). Electronic commerce in small to
bibliometric indicators: A comparative study at Norwe- medium-sized enterprises: Frameworks, Issues and
gian University. Research Evaluation, 13(1), 33–41. implications. Hershey, PA: Idea Group Publishing.
doi:10.3152/147154404781776563 doi:10.4018/978-1-59140-146-9
Aladwani, A., & Palvia, P. (2002). Developing and vali- Al-Qirim, N. (2005). An empirical investigation of an e-
dating an instrument for measuring user-perceived Web commerce adoption-capability model in small businesses
quality. Information & Management, 39(6), 467–476. in New Zealand. Electronic Markets, 15(4), 418–437.
doi:10.1016/S0378-7206(01)00113-6 doi:10.1080/10196780500303136
Alavi, M., & Joachimsthaler, E. A. (1992). Revisiting DSS Alter, S., & Sherer, S. (2004). A general, but readily adapt-
implementation research: A meta-analysis of the literature able model of Information System risk. Communications
and suggestions for researchers. Management Information of the AIS, 14(1), 1–28.
Systems Quarterly, 16(1), 95–113. doi:10.2307/249703
Amaratunga, D., Haigh, R., Sarshar, M., & Baldry, D.
Alavi, M., & Leidner, D. E. (2001). Knowledge man- (2002). Application of the balanced scorecard concept
agement systems: Conceptual foundations and research to develop a conceptual framework to measure facilities
issues. Management Information Systems Quarterly, management performance within NHS facilities. Interna-
25(1), 107–136. doi:10.2307/3250961 tional Journal of Health Care Quality Assurance, 15(4),
141–151. doi:10.1108/09526860210431805
Aldag, R., & Power, D. (1986). An empirical assessment
of computer-assisted decision analysis. Decision Sciences, Ambler, S. W., & Constantine, L. L. (2000). The unified
17(4), 572–588. doi:10.1111/j.1540-5915.1986.tb00243.x process inception phase. Lawrence, KS: CMP Books.
Aldridge, A., Forcht, K., & Pierson, J. (1997). Get linked Amit, R., & Zott, C. (2001). Value creation in e-business.
or get lost: Marketing strategy for the Internet. Internet Strategic Management Journal, 22(6 7), 493-520.
Research: Electronic Networking Applications and Policy,
Amit, R., & Zott, C. (2002). Value drivers of e-commerce
7(3), 161–169. doi:10.1108/10662249710171805
business models. In Hitt, M. A., Amit, R., Lucier, C., &
Ally, M., Cater-Steel, A., & Toleman, M. (2007, 5-7 Dec Nexon, R. (Eds.), Creating value: Winners in the new
2007). A website sophistication model based on value- business environment (pp. 15–47). Oxford, UK: Blackwell.
added technology solutions and services. Paper presented
Ammenwerth, E. (2003). Die Bewertung von Informa-
at the 18th Australasian Conference on Information
tionssystemen des Gesundheitswesens – Beiträge für ein
Systems (ACIS), Toowoomba, Australia.
umfassendes Informationsmanagement. Habilitations-
Al-Mashari, M., Al-Mudimigh, A., & Zairi, M. (2003). schrift zur Erlangung der Venia docendi für das Fach
Enterprise resource planning: Taxonomy of critical fac- Medizinische Informatik.
tors. European Journal of Operational Research, 146(2),
Ammenwerth, E., Iller, C., & Mahler, C. (2006). IT-
352–364. doi:10.1016/S0377-2217(02)00554-4
adoption and the interaction of task, technology and indi-
Aloini, D., Dulmin, R., & Mininno, V. (2007). Risk viduals: A fit framework and a case study. BMC Journal
management in ERP project introduction: Review of the of Medical Informatics and Decision Making, 6(3).
literature. Information & Management, 44(6), 547–567.
Ammenwerth, E., Graber, S., Herrmann, G., Burkle, T.,
doi:10.1016/j.im.2007.05.004
& Konig, J. (2003). Evaluation of health Information
Alpar, P. (2001). Measuring the efficiency of website Systems - Problems and challenges. International Journal
traffic generation. International Journal of Electronic of Medical Informatics, 71(2/3), 125–135. doi:10.1016/
Commerce, 6(1), 53–74. S1386-5056(03)00131-X
Alpar, P., & Reeves, S. (1990). Predictors of MS or ap- Ammenwerth, E., & Shaw, N. T. (2005). Bad health
plication in small businesses. Interfaces, 20(2), 2–11. informatics can kill – Is evaluation the answer? Methods
doi:10.1287/inte.20.2.2 of Information in Medicine, 44(1), 1–3.
404
Amoako-Gyampah, K., & Salam, A. F. (2004). An ex- Arbaugh, J. (2000). How classroom environment and
tension of the technology acceptance model in an ERP student engagement affect learning in Internet-based MBA
implementation environment. Information & Manage- courses. Business Communication Quarterly, 63(4), 9–26.
ment, 41(6), 731–745. doi:10.1016/j.im.2003.08.010 doi:10.1177/108056990006300402
Amoroso, E. G. (1994). Fundamentals of computer Ardito, C., Costabile, M., Marsico, M., Lanzilotti, R.,
security technology. Upper Saddle River, NJ: Prentice- Levialdi, S., & Roselli, T. (2006). An approach to usability
Hall PTR. evaluation of e-learning applications. Universal Access
in the Information Society, 4(3), 270–283. doi:10.1007/
Anderlini, L., & Felli, L. (2004). Bounded rationality and
s10209-005-0008-6
incomplete contracts. Research in Economics, 58, 3–30.
doi:10.1016/j.rie.2003.12.001 Ariyachandra, T. R., & Frolick, M. N. (2008). Critical
success factors in business performance management—
Anderson, R. (2001). Why information security is hard
Striving for success. Information Systems Management,
– An economic perspective. In Proceedings of 17th An-
25(2), 113–120. doi:10.1080/10580530801941504
nual Computer Security Applications Conference, New
Orleans, Louisiana, (pp. 10-14). Armstrong, B., Fogarty, G., Dingsdag, D., & Dimbleby,
J. (2005). Validation of a computer user satisfaction
Anderson, S., & Felici, M. (2002). Quantitative aspects
questionnaire to measure IS success in small business.
of requirements evolution. Paper presented at the 26th
Journal of Research and Practice in Information Technol-
Annual International Computer Software and Applica-
ogy, 37(1), 27–42.
tions Conference (COMPSAC).
Arnott, R., & Small, K. (1994). The economics of traffic
Anderson, J. M. (2003). Why we need a new definition
congestion. American Scientist, 82, 446–446.
of information security. Computers & Security, 22(4),
308. doi:10.1016/S0167-4048(03)00407-3 Arora, A., Nandkumar, A., & Telang, R. (2006). Does
information security attack frequency increase with
Anderson, K., & McAdam, R. (2004). A critique of
vulnerability disclosure? An empirical analysis. Infor-
benchmarking and performance: Lead or Lag. Bench-
mation Systems Frontiers, 8(5), 350–362. doi:10.1007/
marking: An International Journal, 11(5), 465–483.
s10796-006-9012-5
doi:10.1108/14635770410557708
Arunachalam, S., & Jinandra, D. M. (2000). Mapping
Anthony, R. (1965). Planning and control systems: A
international collaboration in science in Asia through
framework for analysis. Boston, MA: Harvard Business
coauthorship analysis. Current Science, 79(5), 621–628.
Press.
Ash, J. S., Berg, M., & Coiera, E. (2004). Some unintended
Antlova, K. (2009). Motivation and barriers of ict adoption
consequences of Information Technology in health care:
in small and medium-sized enterprises. E & M Ekonomie
The nature of patient care Information System-related
a Management, 12(2), 140-155.
errors. Journal of the American Medical Informatics As-
Appari, A., & Johnson, M. E. (2010). Information security sociation, 11(2), 104. doi:10.1197/jamia.M1471
and privacy in healthcare: Current state of research. Inter-
Ash, J., Stavre, Z., & Kuperman, G. J. (2003). A consensus
national Journal of Internet and Enterprise Management,
statement on considerations for a successful CPOE imple-
6(4), 279. doi:10.1504/IJIEM.2010.035624
mentation. Journal of the American Medical Informatics
Aral, S., Brynjolfsson, E., & Wu, D. J. (2006). Which Association, 10, 229–234. doi:10.1197/jamia.M1204
came first, IT or productivity? The virtuous cycle of in-
Assistant Secretary of Defense for Networks and Informa-
vestment and use in enterprise systems. Twenty-Seventh
tion Integration/Department of Defense Chief Information
International Conference on Information Systems, (pp.
Officer. (2010). DoD 8570.01-M, information assurance
1819-1840).
workforce improvement program, incorporating change 2,
April 20, 2010. Washington, DC: Department of Defense.
405
Atoji, Y., Koiso, T., & Nishida, S. (2002). Information Axelrod, R. (1976). Structure of decision: The cogni-
filtering for emergency management. tive maps of political elites. Princeton, NJ: Princeton
University Press.
Aubert, B. A., Patry, M., & Rivard, S. (2005). A framework
for Information Technology outsourcing risk management. Axelsson, S. (2000). The base-rate fallacy and the
The Data Base for Advances in Information Systems, difficulty of intrusion detection. ACM Transactions
36(4), 9–28. on Information and System Security, 3(3), 186–205.
doi:10.1145/357830.357849
Aubert, B. A., Rivard, S., & Patry, M. (2004). A transaction
cost model of IT outsourcing. Information & Management, Azzone, G., Masella, C., & Bertelè, U. (1991). Design of
41(7), 921–932. doi:10.1016/j.im.2003.09.001 performance measures for time-based companies. Interna-
tional Journal of Operations & Production Management,
Audretsch, D. B., Prince, Y. M., & Thurik, R. (1999). Do
11(3), 77–85. doi:10.1108/01443579110143412
small firms compete with large firms. Atlantic Economic
Journal, 27(2), 201–209. doi:10.1007/BF02300239 Backhouse, J., Hsu, C. W., & Silva, L. (2006). Circuits of
power in creating de jure standards: Shaping an interna-
Auer, C. (2004). Performance Mesurement für das Cus-
tional Information Systems security standard. Manage-
tomer Relationship Management – Controlling des IKT-
ment Information Systems Quarterly, 30, 413–438.
basierten Kundenbeziehungsmanagements. Deutscher
Universitäts-Verlag, 2004. Baets, W. (1992). Aligning information systems with busi-
ness strategy. The Journal of Strategic Information Sys-
Auger, P. (2005). The impact of interactivity and de-
tems, 1(4), 205–213. doi:10.1016/0963-8687(92)90036-V
sign sophistication on the performance of commercial
websites for small businesses. Journal of Small Busi- Bagozzi, R., & Pieters, R. (1998). Goal-directed
ness Management, 43(2), 119–137. doi:10.1111/j.1540- emotions. Cognition and Emotion, 12(1), 1–26.
627x.2005.00129.x doi:10.1080/026999398379754
August, T., & Tunca, T. I. (2006). Network software se- Bagranoff, N. A., & Brewer, P. C. (2003). PMB invest-
curity and user incentives. Management Science, 52(11), ments: An enterprise system implementation. Journal
1703–1721. doi:10.1287/mnsc.1060.0568 of Information Systems, 17(1), 85–106. doi:10.2308/
jis.2003.17.1.85
Australian Flexible Learning Framework. (2009). 2009 E-
learning benchmarking survey. Brisban, Australia: Author. Bahli, B., & Rivard, S. (2003). The Information Tech-
nology outsourcing risk: A transaction cost and agency
Averweg, U. (2010). Enabling role of an intranet to aug-
theory-based perspective. Journal of Information Technol-
ment e-coaching. Industrial and Commercial Training,
ogy, 18(3), 211–221. doi:10.1080/0268396032000130214
42(1), 47–52. doi:10.1108/00197851011013715
Bailey, J. E., & Pearson, S. W. (1983). Development of a
Avison, D. E., & Fitzgerald, G. (1995a). Participation. In
tool for measuring and analyzing computer user satisfac-
Avison, D. E., & Fitzgerald, G. (Eds.), Information Systems
tion. Management Science, 29(5), 530–545. doi:10.1287/
development: Methodologies, techniques and tools (pp.
mnsc.29.5.530
87–93). Berkshire, UK: The McGraw-Hills Companies.
Bailey, J., & Pearson, S. (1983). Development of a tool
Avison, D. E., & Fitzgerald, G. (1995b). Effective technical
for measuring and analyzing computer user satisfaction.
and human implementation of computer-based systems
Management Science, 29(5), 530–545. doi:10.1287/
(ETHICS). In Avison, D. E., & Fitzgerald, G. (Eds.), Infor-
mnsc.29.5.530
mation Systems development: Methodologies, techniques
and tools. Berkshire, UK: The McGraw-Hills Companies. Baker, S. (2000). Getting the most from your Intranet and
Extranet strategies. The Journal of Business Strategy, 21,
Avison, D., Gregor, S., & Wilson, D. (2006). Managerial
40–43. doi:10.1108/eb040104
IT unconsciousness. Communications of the ACM, 49(7),
89–93. doi:10.1145/1139922.1139923
406
Bakker, K., Boonstra, A., & Wortmann, H. (2010). Does Barki, H., & Huff, S. J. (1990). Implementing decision
risk management contribute to IT project success? A support systems: Correlates of user satisfaction and system
meta-analysis of empirical evidence. International Journal usage. INFOR (Canada), 28(2), 89–101.
of Project Management, 28(5), 493–503. doi:10.1016/j.
Barnes, S., & Vidgen, R. (2003). Interactive e-government:
ijproman.2009.07.002
Evaluating the website of the UK inland revenue. Journal
Ballantine, J., Bonner, M., Levy, M., & Martin, A. (1996). of Electronic Commerce in Organizations, 2(1), 22.
The 3-D model of Information Systems success: The
Barnes, S., & Vidgen, R. (2003). Measuring website
search for the dependent variable continues. Information
quality improvements: A case study of the forum on
Resources Management Journal, 9(4), 5–14.
strategic management knowledge exchange. Indus-
Baloglu, S., & Pekcan, Y. (2006). The website design and trial Management & Data Systems, 103(5), 297–309.
internet site marketing practices of upscale and luxury doi:10.1108/02635570310477352
hotels in Turkey. Tourism Management, 27(1), 171–176.
Barney, J. B. (1991). Firm resources and sustained
doi:10.1016/j.tourman.2004.07.003
competitive advantage. Journal of Management, 17(1),
Bancroft, N., Seip, H., & Sprengel, A. (2001). Imple- 99–120. doi:10.1177/014920639101700108
menting SAP R/3 (2nd ed.). Greenwich, CT: Manning
Baroudi, J., Olson, M., & Ives, B. (1986). An empirical
Publications Co.
study of the impact of user involvement on system usage
Banker, R. D., & Slaughter, S. A. (1997). A field study and information satisfaction. Communications of the ACM,
of scale economies in software maintenance. Man- 29(3), 232–238. doi:10.1145/5666.5669
Baroudi, J., & Orlikowski, W. (1988). A short-form
mnsc.43.12.1709
measure of user information satisfaction: A psychometric
Bannerman, P. L. (2008). Risk and risk management evaluation and notes on use. Journal of Management
in software projects: A reassessment. Journal of Sys- Information Systems, 4(4), 44–59.
tems and Software, 81(12), 2118–2133. doi:10.1016/j.
Bartis, E., & Mitev, N. (2008). A multiple narrative ap-
jss.2008.03.059
proach to information systems failure: A successful system
Bansler, J. P., Damsgaard, J., Scheepers, R., Havn, E., & that failed. European Journal of Information Systems,
Thommesen, J. (2000). Corporate Intranet implementa- 17(2), 112–124. doi:10.1057/ejis.2008.3
tion: Managing emergent technologies and organizational
Barua, Konana, P., Whinston, A., & Fang, Y. (2001).
practices. Journal of the Association for Information
Driving e-business excellence. MIT Sloan Management
Systems, 1(10).
Review, 43(1), 36-44.
Bao, Y., Sun, X., & Trivedi, K. S. (2005). A workload-
Barua, A., & Lee, B. (1997). The Information Technology
based analysis of software aging, and rejuvenation. IEEE
productivity paradox revisited: A theoretical and empirical
Transactions on Reliability, 54(3), 541–548. doi:10.1109/
investigation in the manufacturing sector. International
TR.2005.853442
Journal of Flexible Manufacturing Systems, 9, 145–166.
Barafort, B., Di Renzo, B., & Merlan, O. (2002). Benefits doi:10.1023/A:1007967718214
resulting from the combined use of ISO/IEC 15504 with
Barua, Kriebel, C., & Mukhopadhyay, T. (1995). Infor-
the Information Technology infrastructure library (ITIL).
mation Technologies and business value: An analytic and
4th International Conference on Product Focussed Soft-
empirical investigation. Information Systems Research,
ware Development and Process Improvement (PROFES).
6(1), 3–23. doi:10.1287/isre.6.1.3
Barafort, B., Di Renzo, B., Lejeune, V., Prime, S., &
Bärwolff, H., Victor, F., & Hüsken, V. (2006). IT-Systeme
Simon, J.-M. (2005). ITIL based service management
in der Medizin. Vieweg Verlag.
measurement and ISO/IEC 15504 process assessment:
A win-win opportunity. SPICE 2005 Conference. Austria.
407
Ba, S., Stallaert, J., & Whinston, A. B. (2001). Research Beck, U. (1992). Risk society. London, UK: Sage.
commentary: Introducing a third dimension in information
Behrens, S. (2009). Shadow systems: The good, the bad
systems design - The case for incentive alignment. Infor-
and the ugly. Communications of the ACM, 52(2), 124–129.
mation Systems Research, 12(3), 225–239. doi:10.1287/
doi:10.1145/1461928.1461960
isre.12.3.225.9712
Bekkers, V. (2009). Flexible information infrastructures in
Basili, V., Caldiera, G., Rombach, D., & Marciniak, J.
Dutch e-government collaboration arrangements: Experi-
(1994). The goal question metric approach. In Marciniak,
ences and policy implications. Government Information
J. J. (Ed.), Encyclopedia of software engineering. Wiley.
Quarterly, 26(1), 60–68. doi:10.1016/j.giq.2007.09.010
Baskerville, R. (1991). Risk analysis: An interpretive
Belanger, F., Fan, W., Schaupp, L. C., Krishen, A., Ever-
feasibility tool in justifying Information Systems security.
hart, J., & Poteet, D. (2006). Website success metrics:
European Journal of Information Systems, 1(2), 121–130.
Addressing the duality of goals. Communications of the
doi:10.1057/ejis.1991.20
ACM, 49(12), 114–116. doi:10.1145/1183236.1183256
Baskerville, R. (1992). The developmental duality of
Belanger, F., Hiller, J. S., & Smith, W. J. (2002). Trust-
Information Systems security. Journal of Management
worthiness in electronic commerce: The role of privacy,
Systems, 4(1), 1–12.
security, and site attributes. The Journal of Strategic
Baskerville, R. (1993). Information Systems security Information Systems, 11(3-4), 245–270. doi:10.1016/
design methods: Implications for Information Systems S0963-8687(02)00018-5
development. ACM Computing Surveys, 25(4), 375–413.
Belden, M. (1989). The employee’s role in protecting
doi:10.1145/162124.162127
information assets. Computers & Security, 8(6), 487–493.
Baskerville, R., & Siponen, M. (2002). An informa- doi:10.1016/0167-4048(89)90077-1
tion security meta-policy for emergent organizations.
Bellman, R. E., & Zadeh, L. A. (1970). Decision-making
Logistics Information Management, 15(5/6), 337–346.
in a fuzzy environment. Management Science, 17(4),
doi:10.1108/09576050210447019
141–164. doi:10.1287/mnsc.17.4.B141
Bauer, C., & Scharl, A. (2000). Quantitive evaluation of
Benbasat, I., & Barki, H. (2007). Quo vadis, TAM? Journal
Web site content and structure. Internet Research, 10(1),
of the Association for Information Systems, 8(4), 211–218.
31–44. doi:10.1108/10662240010312138
Benbasat, I., & Dexter, A. (1986). An investigation of the
Bauer, H. H., Falk, T., & Hammerschmidt, M. (2006).
effectiveness of color and graphical information presenta-
E-transQual: A transaction process-based approach for
tion under varying time constraints. Management Informa-
capturing service quality in online shopping. Journal
tion Systems Quarterly, 10(1), 59–83. doi:10.2307/248881
of Business Research, 59, 866–875. doi:10.1016/j.
jbusres.2006.01.021 Benbasat, I., & Zmud, R. W. (2003). The identity crisis
within the is discipline: Defining and communicating the
Bayer, A. E., & Folger, J. (1966). Some correlates of a
discipline’s core properties. Management Information
citation measure of productivity in science. Sociology of
Systems Quarterly, 27(2), 183–194.
Education, 39(4), 381–390. doi:10.2307/2111920
Benckendorff, P., & Black, N. (2000). Destination mar-
Beaudry, A., & Pinsonneault, A. (2005). Understanding
keting on the Internet. The Journal of Tourism Studies,
user responses to Information Technology: A coping model
11(1), 11–21.
of user adaptation. Management Information Systems
Quarterly, 29(3), 493–524. Bendoly, E., & Jacobs, F. R. (2004). ERP architecture/op-
erational alignment for order-processing performance. In-
Beck, K., & Cleal, D. (1999). Optional scope contracts.
ternational Journal of Operations & Production Manage-
Retrieved 11th July, 2011, from http://www.xprogram-
ment, 24(1), 99–117. doi:10.1108/01443570410511013
ming.com/ftp/Optional+scope+contracts.pdf
408
Bennett, J. (1977). User-oriented graphics systems for Bernroider, E. W. N., & Koch, S. (2002). A framework
decision support in unstructured tasks. In Treu, S. (Ed.), for the selection of ERP packages for small to medium
User oriented design of interactive graphics systems. and large organisations. In Hossain, L., Patrick, J. D.,
New York, NY: Association for Computing Machinery. & Rashid, M. A. (Eds.), Enterprise resource planning:
doi:10.1145/1024273.1024275 Global opportunities & challenges (pp. 182–196). Her-
shey, PA: Idea Group Publishing.
Berander, P., & Jönsson, P. (2006). A goal question metric
based approach for efficient measurement framework Berry, L., & Parasuraman, A. (1997). Listening to the
definition. Paper presented at the ISESE ‘06: 2006 ACM/ customer-the concept of a service-quality Information
IEEE İnternational Symposium on Empirical Software System. Sloan Management Review, 38(3), 65–76.
Engineering, Rio de Janeiro, Brazil.
Beynon-Davies, P. (1999). Human error and Informa-
Berander, P., & Svahnberg, M. (2009). Evaluating two tion Systems failure: The case of the London ambulance
ways of calculating priorities in requirements hierarchies service computer-aided despatch system project. Inter-
– An experiment on hierarchical cumulative voting. Jour- acting with Computers, 11(6), 699–720. doi:10.1016/
nal of Systems and Software, 82, 836–850. doi:10.1016/j. S0953-5438(98)00050-2
jss.2008.11.841
Bhagwat, R., & Sharma, M. K. (2007). Performance
Berghout, E., & Remenyi, D. (2005). The eleven years of measurement of supply chain management: A balanced
the European conference on it evaluation: Retrospectives scorecard approach. Computers & Industrial Engineering,
and perspectives for possible future research. Electronic 53(1), 43–62. doi:10.1016/j.cie.2007.04.001
Journal of Information Systems Evaluation, 8(2), 81-98.
Bharadwaj, A. S. (2000). A resource-based perspective on
Berg, M. (1999). Patient care Information Systems and Information Technology capability and firm performance:
health care work: A sociotechnical approach. Interna- An empirical investigation. Management Information
tional Journal of Medical Informatics, 55(2), 87–101. Systems Quarterly, 24(1), 169–196. doi:10.2307/3250983
doi:10.1016/S1386-5056(99)00011-8
Bharadwaj, A. S., Bharadwaj, S. G., & Konsynski, B.
Berg, M. (2001). Implementing information systems in R. (1999). Information Technology capability and firm
health care organizations: Myths and challenges. Inter- performance: An empirical investigation. Management
national Journal of Medical Informatics, 64, 143–156. Information Systems Quarterly, 45(7), 1008–1024.
doi:10.1016/S1386-5056(01)00200-3
Bharadwaj, A., Bharadwaj, S., & Konsynski, B. (1999).
Berg, M., & Winthereik, B. R. (2004). Waiting for Godot, Information Technology effects on firm performance
episodes from the history of patient records. In Berg, as measured by Tobin’s q. Management Science, 45(6),
M. (Ed.), Health information management: integrating 1008–1024. doi:10.1287/mnsc.45.7.1008
Information Technology in health care work (pp. 11–44).
Bharadwaj, A., Keil, M., & Mahring, M. (2009). Effects
London, UK: Routledge.
of Information Technology failures on the market value
Berndt, E., & Morrison, C. (1995). High-tech capital for- of firms. The Journal of Strategic Information Systems,
mation and economic performance in U.S. manufacturing 18(2), 66–79. doi:10.1016/j.jsis.2009.04.001
industries: An exploratory analysis. Journal of Econo-
Bharati, P., & Berg, D. (2005). Service quality from the
metrics, 65, 9–43. doi:10.1016/0304-4076(94)01596-R
other side: Information Systems management at Duquesne
Bernroider, E. W. N. (2008). IT governance for enterprise Light. International Journal of Information Management,
resource planning supported by the DeLone-McLean mod- 25(4), 367–380. doi:10.1016/j.ijinfomgt.2005.04.008
el of Information Systems success. Information & Man-
Bharati, P., & Chaudhury, A. (2004). An empirical inves-
agement, 45(5), 257–269. doi:10.1016/j.im.2007.11.004
tigation of decision-making satisfaction in Web-based
decision support systems. Decision Support Systems,
37(2), 187–197. doi:10.1016/S0167-9236(03)00006-X
409
Bharati, P., & Chaudhury, A. (2006). Studying the cur- Blythe, S. E. (2008). Croatia’s computer laws: Promo-
rent status of technology adoption. Communications of tion of growth in e-commerce via greater cyber-security.
the ACM, 49(10), 88–93. doi:10.1145/1164394.1164400 European Journal of Law and Economics, 26, 75–103.
doi:10.1007/s10657-008-9053-y
Bhattacharya, S., Kretschmer, H., & Meyer, M. (2003).
Characterizing intellectual spaces between science Bobek, S., & Sternad, S. (2010). Management of ERP
and technology. Scientometrics, 58(2), 369–390. solutions. In Vaish, A. (Ed.), Recent advances in manage-
doi:10.1023/A:1026244828759 ment and information security (pp. 31–47). New Delhi,
India: Shree Publishers & Distributors.
Bhatt, G., Emdad, A., Roberts, N., & Grover, V. (2010).
Building and leveraging information in dynamic environ- Bodin, L. D., Gordon, L. A., & Loeb, M. P. (2008). Infor-
ments: The role of IT infrastructure flexibility as enabler mation security and risk management. Communications
of organizational responsiveness and competitive advan- of the ACM, 51(4), 64–68. doi:10.1145/1330311.1330325
tage. Information & Management, 47(7-8), 314–349.
Bodin, L., Gordon, L. A., & Loeb, M. P. (2005). Evaluat-
doi:10.1016/j.im.2010.08.001
ing information security investments using the analytic
Bhatt, P., Shroff, G., & Misra, A. K. (2004). Dynamics hierarchy process. Communications of the ACM, 48(2),
of software maintenance. ACM SIGSOFT Software Engi- 78–83. doi:10.1145/1042091.1042094
neering Notes, 29(5), 1–5. doi:10.1145/1022494.1022513
Bødker, K., Kensing, F., & Simonsen, J. (2004). Partici-
Bhatt, P., Shroff, G., & Misra, A. K. (2006). Influenc- patory IT design. Designing for business and workplace
ing factors in outsourced software maintenance. ACM realities. MIT Press.
SIGSOFT Software Engineering Notes, 31(3), 1–6.
Boehm, B. W. (1991). Software risk management:
doi:10.1145/1127878.1127883
Principles and practices. IEEE Software, 8(1), 32–41.
Bijker, W. E., Huges, T. P., & Pinch, T. (1989). The social doi:10.1109/52.62930
construction of technological systems. Massachusetts:
Boehm, B., & Turner, R. (2005). Management challenges
MIT Press.
to implementing agile processes in traditional development
Binbasioglu, M., & Winston, E. (2004). Systems thinking organizations. IEEE Software, 22(5), 30–39. doi:10.1109/
for identifying unintended consequences of IT: Packaged MS.2005.129
software implementation in small businesses. Journal of
Boertien, N., Steen, M. W. A., & Jonkers, H. (2005).
Computer Information Systems, 45(1), 86–93.
Evaluation of component-based development methods.
Black, A. D., Car, J., Pagliari, C., Anandan, C., & Cress- In Krogstie, J., Halpin, T., & Siau, K. (Eds.), Information
well, K. (2011). The impact of eHealth on the quality modeling methods and methodologies. Hershey, PA: Idea
and safety of health care: A systematic overview. Public Group Publishing.
Library of Science (PLoS). Medicine, 18(8), e1000387.
Bolton, J. E. (1971). Small firms: Report of the committee
doi:doi:10.1371/journal.pmed.1000387
of inquiry on small firms. London, UK: Department of
Black, J., Draper, C., Lococo, T., Matar, F., & Ward, C. Trade and Industry.
(2007). An integration model for organizing IT service
Boonstra, A., & Govers, M. J. G. (2009). Understanding
management. IBM Systems Journal, 46(3), 405–422.
ERP system implementation in a hospital by analysing
doi:10.1147/sj.463.0405
stakeholders. New Technology, Work and Employment,
Bloch, M., Pigneur, Y., & Segev, A. (1996). On the road 24(2), 177–193. doi:10.1111/j.1468-005X.2009.00227.x
of electronic commerce-A business value framework,
Boritz, J. E. (2004). Information integrity for the informa-
gaining competitive advantage and some research issues.
tion era. Rolling Meadows, IL: ISACA.
410
Borner, K., Hardy, E. F., Herr, B. W., Hooloway, T., & Brady, T., & Davies, A. (2010). 9 learning to deliver a
Paley, W. B. (2007). Taxonomy visualization in support mega-project. Procuring Complex Performance: Stud-
of the semi-automatic validation and optimization of ies of Innovation in Product-Service Management, 174.
organizational schemas. Journal of Informatrics, 1(3),
Brandt, E. (2006). Designing exploratory design games:
214–225. doi:10.1016/j.joi.2007.03.002
A framework for participation in participatory design?
Bossen, C. (2007). Evaluation of a computerized problem- Proceedings Participatory Design Conference.
oriented medical record in a hospital department: Does
Braun, T., Schubert, A. P., & Kostoff, R. N. (2000).
it support daily clinical practice? International Journal
Growth and trends of fullerene research as reflected in
of Medical Informatics, 76, 592–600. doi:10.1016/j.
its journal literature. Chemical Reviews, 100(1), 23–37.
ijmedinf.2006.04.007
doi:10.1021/cr990096j
Bossen, C., Krüger, E., & Weeke, J. (2005). Rapport
Brigham, D. (2001). Converting student support services
om afprovning af Aarhus amts G-EPJ prototype (report
to online delivery. International Review of Research in
on the testing of the G-EHR in the county of Aarhus).
Open and Distance Learning, 1(2), 1–16.
Aarhus, Denmark: Informations- og Medievidenskab,
Aarhus Universitet. Brignall, S., & Ballantine, J. (1996). Performance mea-
surement in service businesses revisited. International
Boudon, R. (1982). The unintended consequences of
Journal of Service Industry Management, 7(1), 6–31.
social action. Macmillan.
doi:10.1108/09564239610109393
Boudreau, M. C. (2002). Learning to use ERP technol-
Briney, A., & Prince, F. (2002). Does size matter?
ogy: A causal model. The 36th Hawaii International
Retrieved from www.infosecuritymag.com/2002/
Conference on System Sciences. Retrieved July 7, 2008,
sep/2002survey.pdf
from http://csdl2.computer.org/comp/ proceedings/
hicss/2003/1874/08/187480235b.pdf Broadbent, M., & Weill, P. (1997). Management by
maxim: How business and IT managers can create IT
Bourne, M., Franco, M., & Wilkes, J. (2003). Corporate
infrastructures. Sloan Management Review, 38, 77–92.
performance management. Measuring Business Excel-
lence, 3(3), 15–21. doi:10.1108/13683040310496462 Brooks, P. (2006). Metrics for IT Service Management
(1st ed.). Zaltbommel, The Netherlands: Van Haren Pub.
Bourque, P., Wolff, S., Dupuis, R., Sellami, A., & Abran,
A. (2004). Lack of consensus on measurement in software Brown, M. G. (1996). Keeping score using the right metrics
engineering: Investigation of related ıssues. Paper pre- to drive world-class performance (p. ix).
sented at the 14th International Workshop on Software
Measurement IWSM/MetriKon, Magdeburg, Germany. Bruun Jensen, C. (2006). En performativ historie: EJP
som en fremtids-genererende anordning (A performative
Bradford, M. (2008). Modern ERP – Select, implement & history: EHR as a future-generating device). The Found-
use today’s advanced business systems. North Carolina ing Meeting of the Danish Association for Science and
State University: College of Management. Technology Studies (DASTS), DTU.
Bradford, M., & Florin, J. (2003). Examining the role Bruun Jensen, C., Lauritsen, P., & Olesen, F. (2007).
of innovation diffusion factors on the implementation Introduktion til STS (Introduction to STS). København,
success of enterprise resource planning systems. Interna- Denmark: Hans Reitzels Forlag.
tional Journal of Accounting Information Systems, 4(3),
205–225. doi:10.1016/S1467-0895(03)00026-5 Bryant, L., Jones, D. A., & Widener, S. K. (2004). Man-
aging value creation within the firm: An examination of
Bradley, J., & Lee, C. C. (2007). ERP training and user multiple performance measures. Journal of Management
satisfaction: a case study. International Journal of En- Accounting Research, 16(1), 107–131. doi:10.2308/
terprise Information Systems, 3(4), 33–55. doi:10.4018/ jmar.2004.16.1.107
jeis.2007100103
411
Brynjolfsson, E. (1993). The productivity paradox Busse, R., Schreyögg, J. A., & Gericke, C. (2006). Manage-
of IT. Communications of the ACM, 36(12), 66–77. ment im Gesundheitswesen (2nd ed.). Berlin, Germany:
doi:10.1145/163298.163309 Springer. doi:10.1007/978-3-540-29465-8
Brynjolfsson, E., & Hitt, L. (1996). Paradox lost? Buzzard, K. (1999). Computer security - What should
Firm-level evidence on the returns to Information Sys- you spend your money on. Computers & Security, 18(4),
tems spending. Management Science, 42(4), 541–558. 322–334. doi:10.1016/S0167-4048(99)80078-9
doi:10.1287/mnsc.42.4.541
Byrd, T., Thrasher, E., Lang, T., & Davidson, N. (2006).
Brynjolfsson, E., & Hitt, L. (2003). Computing productivity: A process-oriented perspective of IS success: Examin-
Firm-level evidence. The Review of Economics and Statis- ing the impact of IS on operational cost. Omega, 34(5),
tics, 85(4), 793–808. doi:10.1162/003465303772815736 448–460. doi:10.1016/j.omega.2005.01.012
Brynjolfsson, E., & Hitt, L. M. (1996). Paradox lost? Byrne, M. M. (1998). Hermeneutics 101. Paper presented
Firm-level evidence of the returns to Information Sys- at the Qualitative Interest Group.
tems spending. Management Science, 42(4), 541–558.
Byrnes, F., & Proctor, P. (2002). The secured enterprise:
doi:10.1287/mnsc.42.4.541
Protecting your information assets. Upper Saddle River,
Brynjolfsson, E., & Hitt, L. M. (1998). Beyond the pro- NJ: Prentice Hall.
ductivity paradox. Communications of the ACM, 41(8),
Cahlik, T. (2000). Search for fundamental articles
49–55. doi:10.1145/280324.280332
in economics. Scientometrics, 49(3), 389–402.
Bucchiarone, A., Polini, A., Pelliccione, P., & Tivoli, M. doi:10.1023/A:1010533506061
(2006). Towards an architectural approach for the dynamic
Caleromedina, C., & Noyons, E. (2008). Combining
and automatic composition of software components. Paper
mapping and citation network analysis for a better under-
presented at the ROSATEA ‘06: ISSTA 2006 Workshop on
standing of the scientific development: The case of the
Role of Software Architecture for Testing and Analysis.
absorptive capacity field. Journal of Informatrics, 2(4),
Bueno, S., & Salmeron, J. L. (2008). TAM-based success 272–279. doi:10.1016/j.joi.2008.09.005
modelling in ERP. Interacting with Computers, 20(6),
Calisir, F., Gumussoy, C. A., & Bayram, A. (2009).
515–523. doi:10.1016/j.intcom.2008.08.003
Predicting the behavioural intention to use enterprise
Bueno, S., & Salmeron, J. L. (2009). Benchmarking main resource planning systems – An exploratory extension of
activation functions in fuzzy cognitive maps. Expert Sys- the technology acceptance model. Management Research
tems with Applications, 36(3), 5221–5229. doi:10.1016/j. News, 32(7), 597–613. doi:10.1108/01409170910965215
eswa.2008.06.072
Callon, M., Courtial, J. P., & Laville, F. (1991). Co-word
Bullen, C. V., & Rockart, J. F. (1981). A primer on critical analysis as a tool for describing the network of interac-
success factors. Working paper 1220-81. Report. Mas- tions between basic and technological research: The case
sachusetts Institute of Technology (MIT), Sloan Scool of of polymer chemistry. Scientometrics, 22(1), 155–205.
Management, Center for Information System Research. doi:10.1007/BF02019280
Buniyamin, N., & Barber, K. (2004). The intranet: A Cambrosio, A., Limoges, C., Courtial, J. P., & Laville,
platform for knowledge management system based on F. (1993). Historical scientometrics? Mapping over 70
knowledge mapping. International Journal of Tech- years of biological safety research with co-word analysis.
nology Management, 28(7-8), 729–746. doi:10.1504/ Scientometrics, 27(2), 119–143. doi:10.1007/BF02016546
IJTM.2004.005780
Cameron, K. (1978). Organisational effectiveness: Its mea-
Burnes, B. (2010). Call for papers: Journal of Change surement and prediction in higher education. Administra-
Management. Journal of Change Management, 10(2), tive Science Quarterly, 32, 604–632. doi:10.2307/2392582
241–242. doi:10.1080/14697011003798085
412
Cameron, K. (1981). Domains of organizational effective- Cater-Steel, A., & Tan, W.-G. (2005). Implementation
ness in colleges and universities. Academy of Management of IT infrastructure library (ITIL) in Australia: Progress
Journal, 24(1), 25–47. doi:10.2307/255822 and success factors. 2005 IT Governance International
Conference, Auckland, New Zealand.
Cameron, K. (1986a). A study of organisational effec-
tiveness and its predictors. Management Science, 32(1), Cater-Steel, A., Tan, W.-G., & Toleman, M. (2007). itSMF
87–112. doi:10.1287/mnsc.32.1.87 Australia 2006 Conference: Summary of ITIL adoption
survey responses. Toowoomba, Australia: University of
Cameron, K. (1986b). Effectiveness as paradox: Con-
Southern Queensland.
sensus and conflict in conceptions of organisational
effectiveness. Management Science, 32(5), 539–553. Cater-Steel, A., Tan, W.-G., & Toleman, M. (2008). itSMF
doi:10.1287/mnsc.32.5.539 Australia 2007 Conference: Summary of ITSM standards
and frameworks survey responses. Toowoomba, Australia:
Cameron, K. S., & Whetten, D. A. (1983). Organizational
University of Southern Queensland.
effectiveness: A comparison of multiple models. New
York, NY: Academic Press. Cater-Steel, A., Tan, W.-G., & Toleman, M. (2009). itSMF
Australia 2008 Conference: Summary of ITSM standards
Campbell, K., Gordon, L.A., Loeb, M.P., & Zhou, L.
and frameworks survey responses. Toowoomba, Australia:
(2003). The economic cost of publicly announced infor-
University of Southern Queensland.
mation security breaches: Empirical evidence from the
stock market. Journal of Computer Security, 431-448. Cater-Steel, A., Toleman, M., & Tan, W.-G. (2006).
Transforming IT service management the ITIL impact.
Campbell, K. (2001). Seven steps to success: Creating
ACIS 2006 17th Australasian Conference on Information
the perfect e-commerce system. Journal of Convergence,
Systems. Adelaide, Australia: Australasian Association
1(2), 11–18.
for Information Systems.
Caniëls, M. C. J., & Bakens, R. J. J. M. (2011). The effects
Cater-Steel, A. (2009). IT service departments struggle to
of project management Information Systems on decision
adopt a service-oriented philosophy. International Journal
making in a multi project environment. International
of Information Systems in the Service Sector, 1(2), 69–77.
Journal of Project Management, 30(2).
doi:10.4018/jisss.2009040105
Cannon, J. A. (1994). Why IT applications succeed or
Cavusoglu, H., Mishra, B., & Raghunathan, S. (2004).
fail: The interaction of technical and organizational fac-
The effect of internet security breach announcements on
tors. Industrial and Commercial Training, 26(1), 10–15.
market value: Capital market reactions for breached firms
doi:10.1108/00197859410051226
and internet security developers. International Journal
Carmel, E., & Nicholson, B. (2005). Small firms and of Electronic Commerce, 9(1), 69–104.
offshore software outsourcing: High transaction costs and
Cayir, S., & Basoglu, A. N. (2008, 27-31.July.2008).
their mitigation. Journal of Global Information Manage-
Information Technology interoperability awareness:
ment, 13(3), 33–54. doi:10.4018/jgim.2005070103
A taxonomy model based on information requirements
Carr, N. (2003). IT doesn’t matter. Harvard Business and business needs. Paper presented at the Portland In-
Review, 81(5), 41–49. ternational Conference on Management of Engineering
& Technology (PICMET), Cape Town, South Africa.
Cater-Steel, A. (2004). An evaluation of software develop-
ment practice and assessment-based process improvement Center for Strategic and International Studies (CSIS).
in small software development firms. Doctoral Disserta- (2008). Securing cyberspace for the 44th presidency.
tion, Griffith University. Retrieved 22 December, 2009, from http://www.csis.
org/media/csis/pubs/081208_securingcyberspace_44.pdf
413
Center for Strategic and International Studies (CSIS). Chan, Y. E., Huff, S. L., Barclay, D. W., & Copeland, D.
(2009). Significant cyber events since 2006. Retrieved G. (1997). Business strategic orientation, information
22 December, 2009, from http://csis.org/files/publica- systems strategic orientation, and strategic alignment. In-
tion/091109_cyber_events_since_2006.pdf formation Systems Research, 8(2), 125–150. doi:10.1287/
isre.8.2.125
Chakraborty, G., Lala, V., & Warren, D. (2002). An
empirical investigation of antecedents of B2B websites’ Checkland, P., & Holwell, S. (1998). Information, systems
effectiveness. Journal of Internet Marketing, 16(4), 51–72. and Information Systems. Chichester, UK: Wiley.
doi:10.1002/dir.10044
Chelladurai, P. (1987). Multidimensionality and multiple
Chakravarthy, B. S. (1986). Measuring strategic perfor- perspectives of organisational effectiveness. Journal of
mance. Strategic Management Journal, 7(September/ Sport Management, 1(1), 37–47.
October), 437–458. doi:10.1002/smj.4250070505
Chen, L., & Mohamed, S. (2008). Contribution of knowl-
Chandler, D. (2008). The transmission model of com- edge management activities to organisational business
munication. Retrieved 15 May, 2011, from http://www. performance. Journal of Engineering. Design and Tech-
aber.ac.uk/media/Documents/short/trans.html nology, 6(3), 269–285. doi:10.1108/17260530810918289
Chang, H. H., Wang, Y.-H., & Yang, W.-Y. (2009). The Cheswick, W. R., & Bellovin, S. M. (1994). Firewalls
impact of e-service quality, customer satisfaction and and Internet security: Repelling the wily hacker. Reading,
loyalty on e-mrketing: Moderating effect of perceived MA: Addison-Wesley Publishing Company.
value. Total Quality Management, 20(4), 423–443.
Chia, A., Goh, M., & Hum, S. (2009). Performance mea-
doi:10.1080/14783360902781923
surement in supply chain entities: Balanced scorecard
Chang, J., & King, W. (2005). Measuring the performance perspective. Benchmarking: An International Journal,
of Information Systems: A functional scorecard. Journal 16(5), 605–620. doi:10.1108/14635770910987832
Chiapello, E., & Lebas, M. (1996). The Tableau de Bord,
Chang, S. E., & Ho, C. B. (2006). Organizational factors A French approach to management information. Paper
to the effectiveness of implementing information security presented at the 19th Annual Meeting of the European
management. Industrial Management & Data Systems, Accounting Association.
106(3), 345–361. doi:10.1108/02635570610653498
Chin, W. W. (1998). Issues and opinion on structural
Chang, S. I., Hung, S. Y., Yen, D. C., & Lee, P. J. (2010). equation modelling. Management Information Systems
Critical factors of ERP adoption for small- and medium- Quarterly, 22(1), 7–16.
sized enterprises: An empirical study. Journal of Global
Chin, W. W., & Gopal, A. (1995). Adoption intention
Information Management, 18(3), 82–106. doi:10.4018/
in GSS: Relative importance of beliefs. DATA BASE
jgim.2010070104
Advances, 26(2/3), 42–64. doi:10.1145/217278.217285
Chan, Y. E. (2002). Why haven’t we mastered alignment?
Chiu, C., Hsu, M., Sun, S., Lin, T., & Sun, P. (2005). Us-
The importance of the informal organizational structure.
ability, quality, value and e-learning continuance decisions.
MIS Quarterly Executive, 2(2), 97–112.
Computers & Education, 45(4), 399–416. doi:10.1016/j.
Chan, Y. E., & Huff, S. L. (1992). Strategy: An Information compedu.2004.06.001
Systems research perspective. The Journal of Strategic
Chiu, W. T., & Ho, Y. S. (2005). Bibliometric analysis
Information Systems, 1(14), 191–204. doi:10.1016/0963-
of homeopathy research during the period of 1991 to
8687(92)90035-U
2003. Scientometrics, 63(1), 3–23. doi:10.1007/s11192-
005-0201-7
414
Chiu, W. T., & Ho, Y. S. (2007). Bibliometric analysis of Cohen, E., & Nycz, M. (2006). Learning objects and
tsunami research. Scientometrics, 73, 3–17. doi:10.1007/ e-learning: An informing science perspective. Interdis-
s11192-005-1523-1 ciplinary Journal of Knowledge and Learning Objects,
2, 23–34.
Choi, N., Kim, D., Goo, J., & Whitmore, A. (2008). Know-
ing is doing: An empirical validation of the relationship Cohen, F. B. (1995). Protection and security on the
between managerial information security awareness and information superhighway. New York, NY: John Wiley.
action. Information Management & Computer Security,
Cohen, M. (1992). Epidemiology of drug resistance: Im-
16(5), 484–501. doi:10.1108/09685220810920558
plications for a post-antimicrobial era. Science, 257(5073),
Cho, P.-A. J. H. V. (2011). Comparing e-learning tools’ 1050. doi:10.1126/science.257.5073.1050
success: The case of instructor–student interactive vs. self-
Cohen, S., Thiraios, D., & Kandilorou, M. (2008).
paced tools. Computers & Education, 57(3), 2025–2038.
Performance parameters interrelations from a balanced
doi:10.1016/j.compedu.2011.05.002
scorecard perspective: An analysis of Greek compa-
Chua, C., Cao, L., Cousins, K., & Straub, D. (2002). nies. Managerial Auditing Journal, 23(5), 485–503.
Measuring researcher-production in Information Systems. doi:10.1108/02686900810875307
Journal of the Association for Information Systems, 3,
Coles, R. S., & Moulton, R. (2003). Operationalizing IT
145–215.
risk management. Computers & Security, 22(6), 487–493.
Chuan-Chuan, J. L., & Hsipeng, L. (2000). Towards an doi:10.1016/S0167-4048(03)00606-0
understanding of the behavioural intention to use a web
Colosimo, M., Lucia, A. D., Scanniello, G., & Tortora, G.
site. International Journal of Information Management,
(2009). Evaluating legacy system migration technologies
20(3), 197–208. doi:10.1016/S0268-4012(00)00005-0
through empirical studies. Information and Software Tech-
Chuang, K. Y., Huang, Y. L., & Ho, Y. S. (2007). A bib- nology, 51, 433–447. doi:10.1016/j.infsof.2008.05.012
liometric and citation analysis of stroke-related research
Commission, E. (2003). Commission recommendation
in Taiwan. Scientometrics, 72(2), 201–212. doi:10.1007/
2003/361/EC. Official Journal of the European Union,
s11192-007-1721-0
46(L 124), 6.
Churchill, G. A. (1979). A paradigm for developing bet-
Commission, E. (2004). Go digital awareness campaign
ter measures of marketing constructs. JMR, Journal of
2001-2003: The main lessons to be learnt.
Marketing Research, 16(1), 64–73. doi:10.2307/3150876
Computer Security Institute (CSI). (2008). CSI computer
Ciborra, C. U. (1993). Teams, markets and systems: Busi-
crime & security survey. Retrieved 22 December, 2009,
ness innovation and Information Technology. Cambridge
from http://i.cmpnet.com/v2.gocsi.com/pdf/CSIsur-
University Press.
vey2008.pdf
Claeys, T. (1996). Medical filling (2nd ed.). Cengage
Conboy, K. (2010). Project failure en masse: A study of
Learning.
loose budgetary control in ISD projects. European Journal
Clements, D. P. (1977). Fuzzy ratings for computer of Information Systems, 19(3), 273–287. doi:10.1057/
security evaluation. PhD Dissertation, University of ejis.2010.7
California, Berkeley.
Conger, S., Winniford, M., & Erickson-Harris, L. (2008,
Clifton, B. (2010). Advanced Web metrics with Google 14-17 August). Service management in operations. Paper
analytics (2nd ed.). Indianapolis, IN: Sybex. presented at the Fourteenth Americas Conference on
Information Systems, Toronto.
Cockburn, C., & Wilson, T. (1996). Business use of the
World Wide Web. International Journal of Informa-
tion Management, 16(2), 83–102. doi:10.1016/0268-
4012(95)00071-2
415
Connell, N. A. D., & Young, T. P. (2007). Evaluating Cyert, R., & March, J. (1963). A behavioral theory of the
healthcare information systems through an “enterprise” firm. Englewood Cliffs, NJ: Prentice Hall.
perspective. Information & Management, 44(4), 433–440.
D’Arcy, J., & Hovav, A. (2007). Towards a best fit between
doi:10.1016/j.im.2007.04.002
organizational security countermeasures and Information
Constantinides, E. (2002). The 4S Web-marketing mix Systems misuse behaviors. Journal of Information Systems
model. Electronic Commerce Research and Applications, Security, 3(2), 3–30.
1(1), 57–76. doi:10.1016/S1567-4223(02)00006-6
Da Veiga, A., & Eloff, J. H. P. (2007). An information secu-
Cooper, R. B., & Zmud, R. W. (1990). Information rity governance framework. Information Systems Manage-
Technology implementation research: A technological ment, 24(4), 361–372. doi:10.1080/10580530701586136
diffusion approach. Management Science, 36(2), 123–139.
Dalakas, V. (2005). The effect of cognitive appraisals
doi:10.1287/mnsc.36.2.123
on emotional responses during service encounters. Ser-
Cormican, K., & O’Sullivan, D. (2004). Auditing best vices Marketing Quarterly, 27(1), 23–41. doi:10.1300/
practice for effective product innovation management. J396v27n01_02
Technovation, 24(10), 819–829. doi:10.1016/S0166-
Daneshgar, F., & Toorn, V. (2009). E-learning in work-
4972(03)00013-0
place versus e-learning in higher education. Australian
Cortex, I. T. (2002). Success assessment: Where is the Education Computing, 1(24), 16–22.
limit between success and failure? Retrieved 18 October,
Daniels, T. E., & Spafford, E. H. (1999). Identification of
2002, from http://www.it-cortex.com/Failure.htm
host audit data to detect attacks on low-level IP. Journal
Cox, J., & Dale, B. G. (2001). Service quality and e- of Computing Security, 7(1), 3–35.
commerce: An exploratory analysis. Managing Service
Dannenberg, A. L. (1985). Use of epidemiology in medical
Quality, 11, 121–131. doi:10.1108/09604520110387257
specialties: An examination by citation analysis. American
Cragg, P. (2008). Identifying key Information Sys- Journal of Epidemiology, 121(1), 140–151.
tems competencies in small firms. Total Quality
D’Arcy, J., & Hovav, A. (2009). Does one size fit all?
Management & Business Excellence, 19(1-2), 29–35.
Examining the differential effects of IS security coun-
doi:10.1080/14783360701601926
termeasures. Journal of Business Ethics, 89, 59–71.
Cristobal, E., Flavian, C., & Guinaliu, M. (2007). Per- doi:10.1007/s10551-008-9909-7
ceived e-service quality (PeSQ) measurement valida-
Dasgupta, S. (2001). Managing Internet and Intranet
tion and effects on consumer satisfaction and website
technologies in organisations: Challenges & opportuni-
loyalty. Managing Service Quality, 17(3), 317–340.
ties. Hershey, PA: Idea Group Publishing.
doi:10.1108/09604520710744326
Dasgupta, S., Sarkis, J., & Talluri, S. (1999). Influence of
Cronbach, L. J., & Meehl, P. E. (1955). Construct vaildity
information technology investment on firm productivity: A
in psychological tests. Psychological Bulletin, 52.
cross-sectional study. Logistics Information Management,
Cronk, M. C., & Fitzgerald, E. P. (2002). Constructing a 12(1/2), 120–129. doi:10.1108/09576059910256493
theory of IS business value from the literature. Electronic
Davern, M., & Wilkin, C. (2010). Towards an integrated
Journal of Business Research Methods, 1(1), 11–17.
view of IT value measurement. International Journal
Crowston K. (2002). Electronic communication and new of Accounting Information Systems, 11(1), 42–60.
organizational forms. MIT working papers series. doi:10.1016/j.accinf.2009.12.005
Culnan, M. J. (1987). Mapping the intellectual structure Davis, F. D. (1986). A technology acceptance model for
of MIS, 1980-1985: A co-citation analysis. Manage- empirically testing new end-user Information Systems:
ment Information Systems Quarterly, 11(3), 341–353. Theory and results. Massachusetts Institute of Technol-
doi:10.2307/248680 ogy: Doctoral dissertation, Sloan School of Management.
416
Davis, F. D. (1989). Perceived usefulness, perceived De Sanctis, G., & Scott Poole, M. (1994). Capturing
ease-of-use, and user acceptance of Information Technol- the complexity in advanced technology use: Adaptive
ogy. Management Information Systems Quarterly, 13(3), structuration theory. Organization Science, 5(2), 121–147.
319–340. doi:10.2307/249008 doi:10.1287/orsc.5.2.121
Davis, F. D. (1993). User acceptance of Information De Toni, A., & Tonchia, S. (2001). Performance measure-
Technology: System characteristics, user perceptions and ment systems - Models, characteristics and measures. In-
behavioural impacts. International Journal of Man-Ma- ternational Journal of Operations & Production Manage-
chine Studies, 38, 475–487. doi:10.1006/imms.1993.1022 ment, 21(1/2), 46–71. doi:10.1108/01443570110358459
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). Deans, K. R., & Adam, S. (2000). Internet survey data
User acceptance of computer technology: A comparison collection: The case of WebQUAL. Paper presented at
of two theoretical models. Management Science, 35(8), the ANZMAC Conference, University of New South
982–1003. doi:10.1287/mnsc.35.8.982 Wales, Sydney.
Davis, F. D., Bagozzi, R., & Warshaw, P. R. (1989). Dedrick, J., Gurbaxani, V., & Kraemer, K. L. (2003).
User acceptance of computer technology: A comparison Information Technology and economic performance: A
of two theoretical models. Management Science, 35(8), critical review of the empirical evidence. ACM Comput-
982–1003. doi:10.1287/mnsc.35.8.982 ing Surveys, 35(1), 1–28. doi:10.1145/641865.641866
Davis, F., Bagozzi, R., & Warshaw, P. (1989). User Dehning, B., & Richardson, V. J. (2002). Returns on
acceptance of computer technology: A comparison of investments in Information Technology: A research
two theoretical models. Management Science, 35(8), synthesis. Journal of Information Systems, 16(1), 7–30.
982–1003. doi:10.1287/mnsc.35.8.982 doi:10.2308/jis.2002.16.1.7
Davis, J. C., & Gonzalez, J. G. (2003). Scholarly jour- Dekleva, S. M. (1992). The ınfluence of the Information
nal articles about the Asian tiger economies: Authors, Systems development approach and maintenance. Man-
journals and research fields, 1986–2001. Asian-Pacific agement Information Systems Quarterly, 16(3), 355–372.
Economic Literature, 17(2), 51–66. doi:10.1046/j.1467- doi:10.2307/249533
8411.2003.00131.x
DeLone, & McLean, E. (2003). The DeLone and McLean
Davis, J. D., & Kottemann, J. E. (1994). User percep- Model of Information Systems success: A ten-year update.
tion of decision support effectiveness: Two production Journal of Management Information Systems, 19(4), 9–30.
planning experiments. Decision Sciences, 25(1), 57–78.
DeLone, & McLean, E. (2004). Measuring e-commerce
doi:10.1111/j.1540-5915.1994.tb00516.x
success: Applying the DeLone & McLean Information
Davis, L., Dehning, B., & Stratopoulos, Th. (2003). Does Systems success model. International Journal of Elec-
the market recognize IT-enabled competitive advantage? tronic Commerce, 9(1), 31–47.
Information & Management, 40(7), 705–716. doi:10.1016/
DeLone, & McLean, E. R. (2003). The DeLone and
S0378-7206(02)00092-7
McLean model of Information Systems success: A ten-
Day, A. (1997). A model for monitoring website year update. Journal of Management Information Systems,
effectiveness. Internet Research: Electronic Net- 19(4), 9–30.
DeLone, W. H., & McLean, E. R. (2002). Information
doi:10.1108/10662249710165244
Systems success revisited. Paper presented at the The 35th
De Jong, C. M., Huizingh, K. R. E., Oude Ophuis, P. A. Hawaii International Conference on System Sciences-
M., & Wierenga, B. (1994). Critical success factors for HICSS-35, Hilton Waikoloa Village/Island of Hawaii.
marketing decision support systems. Delft, The Nether-
lands: Eburon.
417
DeLone, W. H., & McLean, E. R. (1992). Information Devos, J., Van Landeghem, H., & Deschoolmeester,
Systems success: The quest for the independent variable. D. (2008). Outsourced Information Systems Failures
Information Systems Research, 3(1), 60–95. doi:10.1287/ in SMEs: A multiple case study. Electronic Journal of
isre.3.1.60 Information Systems Evaluation, 11(2), 73–84.
DeLone, W. H., & McLean, E. R. (2003). The DeLone Devos, J., Van Landeghem, H., & Deschoolmeester, D.
and McLean Model of Information Systems success: A (2010). IT/IS and small & medium-sized enterprises:
ten-year update. Journal of Management Information Literature overview. Ghent University Association.
Systems, 19(4), 9–30.
Devos, J., Van Landeghem, H., & Deschoolmeester, D.
DeLone, W. H., & McLean, E. R. (2004). Measuring (2010). The theory of the lemon markets in IS research.
e-commerce success: Applying the DeLone & McLean In Dwivedi, Y. K., Wade, M., & Schneberger, S. L. (Eds.),
Information Systems success model. International Journal Information Systems theory: Explaining and predicting
of Electronic Commerce, 9(1), 31–47. our digital society. Springer. doi:10.1007/978-1-4419-
6108-2_11
DeLone, W., & McLean, E. (1992). Information Systems
success: The quest for the dependent variable. Information Devos, J., Van Landeghem, H., & Deschoolmeester, D.
Systems Research, 3(1), 60–95. doi:10.1287/isre.3.1.60 (2011). Does size and sector matter? A discussion of
the idiographic character of the relation SMEs and IT/
Denning, D. (1987). An intrusion-detection model. IEEE
IS. Howest.
Transactions on Software Engineering, 13(2), 222–226.
doi:10.1109/TSE.1987.232894 Dewan, S., & Min, C. (1997). The substitution of Informa-
tion Technology for other factors of production: A firm
Denton, D. (2006). Using intranets to make virtual teams
level analysis. Management Science, 43(12), 1660–1675.
effective. Team Performance Management, 12(7-8),
doi:10.1287/mnsc.43.12.1660
253–257. doi:10.1108/13527590610711804
Dhillon, G., & Backhouse, J. (1994). Responsibility
Denton, D. (2010). Performance measurement and
analysis: A basis for understanding complex managerial
intranets: A natural partnership. International Journal
situations. In Proceedings of the International System
of Productivity and Performance Management, 59(7),
Dynamics Conference, (pp. 70-79).
701–706. doi:10.1108/17410401011075675
Dhillon, G. (1999). Managing and controlling computer
Dess, G., & Robinson, R. (1984). Measuring organiza-
misuse. Information Management & Computer Security,
tional performance in the absence of objective measures:
7(4), 171–175. doi:10.1108/09685229910292664
The case of the privately-held firm and conglomerate busi-
ness unit. Strategic Management Journal, 5(3), 265–273. Dhillon, G. (2001). Violation of safeguards by trusted
doi:10.1002/smj.4250050306 personnel and understanding related information se-
curity concerns. Computers & Security, 20, 165–172.
Deveray, S., & Kohli, R. (2003). Performance impact
doi:10.1016/S0167-4048(01)00209-7
of Information Technology: Is actual usage the missing
link? Management Science, 49(3), 273–289. doi:10.1287/ Dhillon, G., & Backhouse, J. (2001). Current directions
mnsc.49.3.273.12736 in IS security research: Towards socio-organizational per-
spectives. Information Systems Journal, 11(2), 127–153.
Devos, J., Van Landeghem, H., & Deschoolmeester, D.
doi:10.1046/j.1365-2575.2001.00099.x
(2009). IT governance in SMEs: A Theoretical framework
based on the outsourced Information System failure. The Dhillon, G., & Torkzadeh, G. (2006). Value-focused
3rd European Conference on Information Management assessment of Information System security in organi-
and Evaluation. zations. Information Systems Journal, 16, 293–314.
doi:10.1111/j.1365-2575.2006.00219.x
418
Dholakia, U., & Rego, L. (1998). What makes commercial Ding, Y., Chowdhury, G. G., & Foo, S. (2001). Bibliometric
Web pages popular? European Journal of Marketing, cartography of information retrieval research by using co-
32(7/8), 724–736. doi:10.1108/03090569810224119 word analysis. Information Processing & Management,
37(6), 817–842. doi:10.1016/S0306-4573(00)00051-0
Diamantopoulos, A., & Siguaw, J. A. (2006). Formative
versus reflective indicators in organisational measure Dodge, R. C., Carver, C., & Ferguson, A. J. (2007). Phish-
development: A comparison and empirical illustra- ing for user security awareness. Computers & Security,
tion. British Journal of Management, 17(4), 263–282. 26, 73–80. doi:10.1016/j.cose.2006.10.009
doi:10.1111/j.1467-8551.2006.00500.x
Doig, G. (2002). A study of the factors that influence the
Dibbern, J., & Heinzl, A. (2009). Outsourcing of Infor- perception of strategic information value in major UK
mation Systems functions in small and medium sized based organisations. Doctoral Thesis Loughborough
enterprises: A test of a multi-theoretical model. Business University.
& Information Systems Engineering, 1(1), 101–110.
Doll, W. J., & Torkzadeh, G. (1988). The measurement of
doi:10.1007/s12599-008-0008-1
end-user computing satisfaction. Management Information
Dibrell, C., Davis, P. S., & Craig, J. (2008). Fueling Systems Quarterly, 12(2), 259–276. doi:10.2307/248851
innovation through Information Technology in SMEs.
Doll, W. J., & Torkzadeh, G. (1998). Developing a multi-
Journal of Small Business Management, 46(2), 203–218.
dimensional measure of system-use in an organizational
doi:10.1111/j.1540-627X.2008.00240.x
context. Information & Management, 33(4), 171–185.
Diewert, E. W., & Smith, A. M. (1994). Productivity doi:10.1016/S0378-7206(98)00028-7
measurement for a distribution firm. National Bureau
Doll, W., & Torkzadeh, G. (1988). The measurement of
of Economic Research working paper no. 4812, Wash-
end-user computing satisfaction. Management Information
ington, D.C.
Dikerson, J., & Kosko, B. (1994). Virtual worlds as fuzzy
Donaldson, T., & Preston, L. E. (1995). The stakeholder
cognitive maps. Proceedings of IEEE Virtual Reality
theory of the corporation: Concepts, evidence, and impli-
Symposium, 3(2), 173-189.
cations. Academy of Management Review, 20(1), 65–91.
Dinev, T., Goo, J., Hu, Q., & Nam, K. (2008). User be-
Donko, D., & Traljic, I. (2009). Measurement of service
haviour towards protective Information Technologies:
effectiveness and establishment of baselines. WSEAS
The role of national cultural differences. Information
Transactions on Information Science and Applications,
Systems Journal, 19(4), 391–412. doi:10.1111/j.1365-
6(8), 1310–1319.
2575.2007.00289.x
Downing, C. E. (1999). System usage behavior as a
Dinev, T., & Hu, Q. (2007). The centrality of awareness in
proxy for user satisfaction: An empirical investigation.
the formation of user behavioral intention toward protec-
tive information technologies. Journal of the Association
S0378-7206(98)00090-1
for Information Systems, 8(7), 386–408.
Drevin, L., Kruger, H. A., & Stegn, T. (2007). Value-
Ding, Y., & Straub, D. (2007). Using marketing exchange
focused assessment of ICT security awareness in an
theory to conceptualize IS quality and re-conceptualize the
academic environment. Computers & Security, 26, 36–43.
IS success model. Twenty Eighth International Conference
doi:10.1016/j.cose.2006.10.006
on Information Systems, Montréal.
Drèze, X., & Zufryden, F. (1997). Testing website design
Ding, D. X., Hu, P. J.-H., & Sheng, O. R. L. (2010). E-
and promotional content. Journal of Advertising Research,
SELFQUAL: A scale for measuring online self-service
37(2), 77–91.
quality. Journal of Business Research, 64(5).
419
Drèze, X., & Zufryden, F. (2004). Measurement of online Edwards, J. R. (2001). Multidimensional constructs in
visibility and its impact on Internet traffic. Journal of In- organisational behavior research: An integrative analyti-
teractive Marketing, 18(1), 20–37. doi:10.1002/dir.10072 cal framework. Organizational Research Methods, 4(2),
144–192. doi:10.1177/109442810142004
Du, J., Yang, X., & Wang, Z. (2008). Effective runtime scal-
ability metric to measure productivity in high performance Eick, S. G., Graves, T. L., Karr, A. F., Marron, J. S., &
computing systems. Paper presented at the Conference on Mockus, A. (2001). Does code decay? Assessing the evi-
Computing Frontiers, Ischia, Italy. dence from change management data. IEEE Transactions
on Software Engineering, 27(1). doi:10.1109/32.895984
Duncan, R., & Weiss, A. (1978). Organization learning:
Implications for organization design. In Staw, B. (Ed.), Eisenhardt, K. M. (1989). Building theories from case
Research in organizational behavior. Greenwich, CT: study research. Academy of Management Review, 14(4),
Jai Press. 532–550.
Dutta, A., & McCrohan, K. (2002). Management’s role Elleithy, K. (2008). Innovations and advanced techniques
in information security in a cyber economy. California in systems, computing sciences, and software engineering.
Management Review, 45(1), 67–87. Dordrecht, The Netherlands: Springer. doi:10.1007/978-
1-4020-8735-6
Dybå, T., & Dingsøyr, T. (2008). Strength of evidence in
systematic reviews in software engineering. Paper pre- Elliot, S. R. Moّrup-Petersen, A. S., & Bjoّrn-Andersen,
sented at the 2nd ACM-IEEE International Symposium N. (2000, 19-21, 2000). Towards a framework for evalu-
on Empirical Software Engineering and Measurement. ation of commercial websites. Paper presented at the 13th
International Bled Electronic Commerce Conference,
Eastin, M. S. (2002). Diffusion of e-commerce: An
Bled, Slovania.
analysis of the adoption of four e-commerce activities.
Telematics and Informatics, 19(3), 251–267. doi:10.1016/ Emery, J. C. (1971). Cost/benefit analysis of Information
S0736-5853(01)00005-3 Systems. SMIS Workshop report number 1, The Society
for Management Information Systems, Chicago, IL.
Eck, N. J. P., & Waltman, L. R. (2007). VOS: A new method
for visualizing similarities between objects. In Decker, Encyclopaedia Britannica. (2011a). Amensalism. Re-
R., & Lenz, H.-J. (Eds.), Advances in data analysis (pp. trieved 5th May, 2011, 2011, from http://www.britannica.
299–306). Berlin, Germany: Springer. doi:10.1007/978- com/EBchecked/topic/19211/amensalism
3-540-70981-7_34
Encyclopaedia Britannica. (2011b). Symbiosis. Retrieved
Eden, C., & Ackermann, F. (2004). Analyzing cognitive 5th May, 2011, 2011, from http://www.britannica.com/
maps to help structure issues or problems. European EBchecked/topic/577677/symbiosis
Journal of Operational Research, 159(3), 673–686.
Enderle, G., & Tavis, L. A. (1998). A balanced concept of
doi:10.1016/S0377-2217(03)00431-4
the firm and the measurement of its long-term planning
Edmondson, A. C. (2003). Framing for learning: Lessons and performance. Journal of Business Ethics, 17(11),
in successful technology implementation. California 1129–1144. doi:10.1023/A:1005746212024
Management Review, 45(2), 34–51.
Epstein, M. J. (2004). Implementing e-commerce strate-
Edquist, C., & Edquist, O. (1979). Social carriers of gies: A guide to corporate success after the dot.com bust
techniques for development. Journal of Peace Research, (1st ed.). Westport, CT: Praeger Publishers.
16(4), 313–331. doi:10.1177/002234337901600403
Epstein, M. J. (2005). Implementing successful e-
Edvinsson, L. (1997). Developing intellectual capital commerce initiatives. Strategic Finance, 86(9), 22–29.
at Skandia. Long Range Planning, 30(3), 366–373.
doi:10.1016/S0024-6301(97)90248-X
420
Erickson, J., Lyytinen, K., & Siau, K. (2005). Agile model- Feindt, S., Jeffcoate, J., & Chappell, C. (2002). Iden-
ing, agile software development, and extreme program- tifying success factors for rapid growth in SME e-
ming: The state of research. Journal of Database Man- commerce. Small Business Economics, 19(1), 51–62.
agement, 16(4), 88–100. doi:10.4018/jdm.2005100105 doi:10.1023/A:1016165825476
Estabrooks, C. A., Winther, C., & Derksen, L. (2004). Ferreira, A., & Otley, D. (2009). The design and use of
Mapping the field: A bibliometric analysis of the research performance management systems: An extended frame-
utilization literature in nursing. Nursing Research, 53(5), work for analysis. Management Accounting Research,
293–303. doi:10.1097/00006199-200409000-00003 20(4), 263–282. doi:10.1016/j.mar.2009.07.003
Esteves, J., Pastor, J., & Casanovas, J. (2002). Monitor- Fetterman, D. M. (1998). Ethnography: Step-by-step
ing business process redesign in ERP implementation (applied social research methods) (2nd ed., Vol. 17).
projects. Americas Conference on Information Systems Sage Publications.
2002, Proceedings Paper 125.
Feyzioglu, O., Buyukozkan, G., & Ersoy, M. S. (2007).
Etezadi-Amoli, J., & Farhoomand, A. F. (1996). A struc- Supply chain risk analysis with fuzzy cognitive maps.
tural model of end user computing satisfaction and user IEEE International Conference on Industrial Engineering
performance. Information & Management, 30(2), 65–73. and Engineering Management, (pp. 1447 –1451).
doi:10.1016/0378-7206(95)00052-6
Fink, L., & Neumann, S. (2007). Gaining agility through
Evans, J., & Schaefer, C. (2001). Ten tasks of change: IT personnel capabilities: The mediating role of IT in-
Demystifying changing organizations. San Francisco, frastructure capabilities. Journal of the Association for
CA: Jossey-Bass/Pfeiffer. Information Systems, 8(8), 440–462.
Ewusi-Mensah, K. (1997). Critical issues in abandoned in- Fink, L., & Neumann, S. (2009). Exploring the perceived
formation systems development projects. Communications business value of the flexibility enabled by Information
of the ACM, 40(9), 74–80. doi:10.1145/260750.260775 Technology infrastructure. Information & Management,
46(2), 90–99. doi:10.1016/j.im.2008.11.007
Ezhkova, I. (2005). Self-organizing representa-
tions. Cybernetics and Systems, 36(8), 861–875. Finne, T. (1998). A conceptual framework for informa-
doi:10.1080/01969720500306345 tion security management. Computers & Security, 17(4),
303–307. doi:10.1016/S0167-4048(98)80010-2
Fan, M., Stallaert, J., & Whinston, A. B. (2003). Decen-
tralized mechanism design for supply chain organisations Fischer, W. (2000). Grundzüge von DRG-Systemen. In
using an auction market. Information Systems Research, KH-Report (pp. 13–31).
14(1), 1–22. doi:10.1287/isre.14.1.1.14763
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention,
Farbey, B., Land, F., & Targett, D. (1999). Moving IS and behavior: An introduction to theory and research.
evaluation forward: Learning themes and research is- MA: Addison-Wesley Reading.
sues. The Journal of Strategic Information Systems, 8(2),
Fisher, J., Annemieke, C., & Bentley, J. (2007).
189–207. doi:10.1016/S0963-8687(99)00021-9
Moving from a Web presence to e-commerce: The
Fassancht, M., & Koese, I. (2006). Quality of elec- importance of a business—Web strategy for small-
tronic service: Conceptualizing and testing a hierarchi- business owners. Electronic Markets, 17(4), 253–262.
cal model. Journal of Service Research, 9(1), 19–37. doi:10.1080/10196780701635864
doi:10.1177/1094670506289531
Fiske, J. (1982). Introduction to communication studies.
Fearon, C., & Philip, G. (1999). An empirical study of the London, UK: Routledge. doi:10.4324/9780203323212
use of EDI in supermarket chains using a new conceptual
Fitzgerald, L., Johnston, R., Brignall, S., Silvestro, R.,
framework. Journal of Information Technology, 14(1),
& Voss, C. (1994). Performance measurement in service
3–21. doi:10.1080/026839699344719
businesses. Cambridge, UK: CIMA.
421
Fitzpatrick, R. (2000). Additional quality factors for the Früh, K. F., Maier, U., & Schaudel, D. (2008). Handbuch
World Wide Web. Paper presented at the The Second der Prozessautomatisierung: Prozessleittechnik für ver-
World Congress for Software Quality Yokohama, Japan. fahrenstechnische Anlagen. Oldenbourg Industrieverlag.
Fletcher, C. (2001). Performance appraisal and man- Frydenberg, J. (2002). Quality standards in e-learning: A
agement: The developing research agenda. Journal of matrix of analysis. International Review of Research in
Occupational and Organizational Psychology, 74(4), Open and Distance Learning, 3(2), 1–15.
473–487. doi:10.1348/096317901167488
Fugate, M., Kinicki, A. J., & Prussia, G. E. (2008). Em-
Floropoulos, J., Spathis, C., Halvatzis, D., & Tsipou- ployee coping with organisational change: An examina-
ridou, M. (2010). Measuring the success of the Greek tion of alternative theoretical perspectives and models.
taxation Information System. International Journal of Personnel Psychology, 61(1), 1–36. doi:10.1111/j.1744-
Information Management, 30(1), 47–56. doi:10.1016/j. 6570.2008.00104.x
ijinfomgt.2009.03.013
Gable, G., Sedera, D., & Chan, T. (2003). Enterprise
Flowerday, S., & von Solms, R. (2005). Real-time in- systems success: A measurement model. Twenty Fourth
formation integrity = system integrity + data integrity International Conference on Information Systems, Seattle.
+ continuous assurances. Computers & Security, 24(8),
Gable, G. (1996). A multidimensional model of client
604–613. doi:10.1016/j.cose.2005.08.004
success when engaging external consultants. Management
Foltz, C. B., Schwager, P. H., & Anderson, J. E. (2008). Science, 42(8), 1175–1198. doi:10.1287/mnsc.42.8.1175
Why users (fail to) read computer usage policies. Indus-
Gable, G. G., Sedera, D., & Chan, T. (2008). Re-concep-
trial Management & Data Systems, 108(6), 701–712.
tualizing Information Systems success: The IS-impact
doi:10.1108/02635570810883969
measurement model. Journal of the Association for
Fong, S., & Quaddus, M. (2010). Intranet use in Hong Information Systems, 9(7), 377–408.
Kong public hospitals. International Journal of Ac-
Gable, G., Sedera, D., & Chan, T. (2008). Re-concep-
counting and Information Management, 18(2), 156–181.
tualizing Information System success: The IS-impact
doi:10.1108/18347641011048138
measurement model. Journal of the Association for
Fornell, C., & Larcker, D. F. (1981). Evaluating structural Information Systems, 9(7), 377–408.
equation models with unobservable variables and mea-
Gacenga, F., Cater-Steel, A., & Toleman, M. (2010). An
surements errors. JMR, Journal of Marketing Research,
international analysis of IT service management benefits
18(1), 39–50. doi:10.2307/3151312
and performance measurement. Journal of Global Infor-
Foster, J., & Lin, A. (2004). Collaborative e-business mation Technology Management, 13(4), 28–63.
planning: Developing an enterprise learning tool for
Gaertner, N., & Smith, M. (2001). E-commerce in a Web-
information management and Information Systems cur-
based environment: Auditing relative advantages in the
ricula. Journal of Electronic Commerce in Organizations,
Australian health sector. Managerial Auditing Journal,
2(2), 28–43. doi:10.4018/jeco.2004040103
16(6), 347–365. doi:10.1108/02686900110395497
Fowler, M., & Highsmith, J. (2001). The agile manifesto.
Gallagher, C. A. (1974). Perceptions of the value of a
Software Development, 9(8), 28–32.
management Information System. Academy of Manage-
Fricke, M. (2009). The knowledge pyramid: A critique ment Journal, 17(1), 46–55. doi:10.2307/254770
of the DIKW hierarchy. Journal of Information Science,
Galletta, D. F., & Lederer, A. L. (1989). Some cautions
35(2), 131–142. doi:10.1177/0165551508094050
on the measurement of user information satisfaction.
Frodl, A. (2010). Gesundheitsbetriebslehre: Betrieb- Decisions Science Journal, 20(2), 168–187.
swirtschaftslehre des Gesundheitswesens. Gabler Verlag.
422
Galup, S., Dattero, R., Quan, J., & Conger, S. (2009). An Gelderman, M. (1997). Success of management support
overview of IT service management. Communications of systems: A literature review and an empirical investiga-
the ACM, 52(5), 124–127. doi:10.1145/1506409.1506439 tion. Amsterdam, The Netherlands: Thesis Publishers
Amsterdam.
Garfield, E. (1972). Citation analysis as a tool in journal
evaluation. Science, 178(60), 471–479. doi:10.1126/sci- Gelderman, M. (1998). The relation between user satis-
ence.178.4060.471 faction, usage of Information Systems and performance.
Garfield, E. (1990). Keywords plus-ISIS breakthrough
S0378-7206(98)00044-5
retrieval method. 1. Expanding your searching power on
current-contents on diskette. Current Contents, 32, 5–9. Gelderman, M. (2002). Task difficulty, task variability
and satisfaction with management support systems. In-
Garg, S., Moorsel, A. v., Vaidyanathan, K., & Trivedi, K.
formation & Management, 39(7), 593–604. doi:10.1016/
S. (1998, 4-7 November). A methodology for detection and
S0378-7206(01)00124-0
estimation of software aging. Paper presented at the The
Ninth International Symposium on Software Reliability Georgina, D., & Olson, M. (2008). Integration of tech-
Engineering, Paderborn, Germany. nology in higher education: A review of faculty self-
perceptions. The Internet and Higher Education, 11(1),
Garrity, E. J., Glassberg, B., & Kim, Y, J., Sanders, G. L.,
1–8. doi:10.1016/j.iheduc.2007.11.002
& Shin, S, K. (2005). An experimental investigation of
Web-based Information Systems success in the context of Georgopoulos, B., & Tannenbaum, A. (1957). A study
electronic commerce. Decision Support Systems, 39(3), of organisational effectiveness. American Sociological
485–503. doi:10.1016/j.dss.2004.06.015 Review, 22(5), 534–540. doi:10.2307/2089477
Gartner Inc. (2011). Quarterly IT spending forecast. Re- Gerwin, D. (1979). The comparative analysis of structure
trieved 31 January, 2011, from http://www.gartner.com/ and technology: A critical appraisal. Academy of Manage-
technology/research/quarterly-it-forecast/ ment Review, 4(1), 41–51.
Gartner, (2009). Gartner EXP worldwide survey of more Ghandour, A., Benwell, G., & Deans, K. (2011). Measur-
than 1,500 CIOs shows IT spending to be flat in 2009. ing the performance of ecommerce websites–An owner’s
Retrieved April 22, 2010, from http://www.gartner.com/ perspective. Pacific Asia Journal of the Association for
it/page.jsp?id=855612 Information Systems, 3(1), Article 2.
Gattiker, T. F., & CFPIM. (2002). Anatomy of an ERP Ghandour, A., Benwell, G., & Deans, K. R. (2007). The
implementation gone awry. Production and Inventory contribution of goal setting to the success of ecommerce
Management Journal, 43(3/4), 96–105. systems among small and medium enterprises in New
Zealand. Paper presented at the Pan-Pacific Conference
Gattiker, T. F., & Goodhue, D. L. (2005). What happens
XXIV of the Pan-Pacific Business Association.
after ERP implementation: Understanding the impact
of interdependence and differentiation on plant-level Giaglis, G., Paul, R., & Doukidis, G. (1999). Dynamic
outcomes. Management Information Systems Quarterly, modeling to assess the business value of electronic com-
29(3), 559–585. merce. International Journal of Electronic Commerce,
3(3), 35–52.
Gaudin, S. (2007, April 11). Security breaches cost $90
to $305 per lost record. Information Week. Retrieved Oc- Gill, N. S., & Balkishan. (2008). Dependency and
tober 18, 2010, from http://www.informationweek.com/ ınteraction oriented complexity metrics of component-
news/security/showArticle.jhtml?articleID=199000222 based systems. ACM SIGSOFT Software Engineering
Notes, 33(2).
Geczy, P. (2007). Knowledge worker intranet behaviour
and usability. International Journal of Business Intel- Gill, N. S., & Tomar, P. (2010). Modified development
ligence and Data Mining, 2(4), 447–470. doi:10.1504/ process of component-based software engineering. ACM
IJBIDM.2007.016383 SIGSOFT Software Engineering Notes, 35(2).
423
Glass, R. L. (1998). Editor’s corner: How long do infor- Gordon, L. A., & Loeb, M. P. (2001). Using informa-
mation systems last? Journal of Systems and Software, tion security as a response to competitor analysis
40(2), 91. doi:10.1016/S0164-1212(97)00002-2 systems. Communications of the ACM, 44(9), 70–75.
doi:10.1145/383694.383709
Goi, C. L. (2007). A review of existing website models
for e-commerce. Journal of Internet Banking and Com- Gordon, L. A., & Loeb, M. P. (2002). The economics
merce, 12(1), 1–17. of information security investment. ACM Transactions
on Information and System Security, 5(4), 438–457.
Goldratt, E. M. (1990). What is this thing called theory
doi:10.1145/581271.581274
of constraints and how should it be implemented?Great
Barrington, MA: North River Press. Gordon, L. A., & Loeb, M. P. (2006). Budgeting process
for information security expenditures. Association for
Goles, T., & Hirschheim, R. (1997). Intranets: The next
Computing Machinery. Communications of the ACM,
IS Solution? Information Systems Research Center, Col-
49(1), 121–125. doi:10.1145/1107458.1107465
lege of Business Administration, University of Houston.
Gordon, L. A., Loeb, M. P., Lucyshyn, W., & Sohail, T.
Gondek, C. (1989). Establishing information security.
(2006). The impact of the Sarbanes-Oxley Act on the
Management Accounting, 70(10), 34–37.
corporate disclosures of information security activities.
Gong, H., Tate, M., & Alborz, S. (2007). Managing the Journal of Accounting and Public Policy, 25(5), 503–530.
outsourcing marriage to achieve success. System Sciences, doi:10.1016/j.jaccpubpol.2006.07.005
2007. HICSS 2007. 40th Annual Hawaii International
Gorla, N., & Lin, S. (2010). Determinants of software qual-
Conference on, 239c-239c.
ity: A survey of Information Systems project managers.
Gonza’lez, M., & Palacios, B. (2004). Quantitative Information and Software Technology, 52(6), 602–610.
evaluation of commercial websites: An empirical study doi:10.1016/j.infsof.2009.11.012
of Spanish firms. International Journal of Informa-
Gorla, N., Somers, T., & Wong, B. (2010). Organizational
tion Management, 24(4), 313–328. doi:10.1016/j.ijin-
impact of system quality, information quality, and service
fomgt.2004.04.009
quality. The Journal of Strategic Information Systems,
Goodhue, D. L., Klein, B. D., & March, S. T. (2000). User 19(3), 207–228. doi:10.1016/j.jsis.2010.05.001
evaluation of IS as surrogates for objective performance.
Grandon, E. E., & Pearson, J. M. (2004). Electronic
commerce adoption: An empirical study of small and
S0378-7206(00)00057-4
medium US businesses. Information & Management,
Goodhue, D. L., & Straub, D. W. (1991). Security concerns 42(1), 197–216. doi:10.1016/j.im.2003.12.010
of system users: A study of perception of the adequacy
Grandon, E., & Pearson, J. M. (2004). Electronic com-
of security measures. Information & Management, 20(1),
merce adoption: An empirical study of small and medium
13–27. doi:10.1016/0378-7206(91)90024-V
US businesses. Information & Management, 42(1),
Goodhue, D. L., & Thompson, R. L. (1995). Task- 197–216. doi:10.1016/j.im.2003.12.010
technology fit and individual performance. Manage-
Grant, D., & Osei-Bryson, K. (2005). Using fuzzy cogni-
ment Information Systems Quarterly, 19(2), 213–235.
tive maps to assess MIS organizational change impact.,
doi:10.2307/249689
Proceedings of the 38th Annual Hawaii International
Gopal, A., Sivaramakrishnan, K., Krishnan, M. S., & Muk- Conference on System Sciences, (pp. 1-11).
hopadhyay, T. (2003). Contracts in offshore software de-
Gregor, S. (2006). The nature of theory in information
velopment: An empirical analysis. Management Science,
systems. Management Information Systems Quarterly,
49(12), 1671–1683. doi:10.1287/mnsc.49.12.1671.25120
30(3), 611–642.
424
Gregory, J. (2000). Sorcerer’s apprentice: Creating the Gupta, Y. P., & Raghunathan, T. S. (1988). A preliminary
electronic health record, re-inventing medical records model for İnformation System replacement. Omega, 16(4),
and patient care. California: University of San Diego. 289–296. doi:10.1016/0305-0483(88)90065-5
Gronau, N. (2001). Industrielle Standardsoftware – Aus- Haas, P. (2004). Medizinische Informationssysteme und
wahl und Einführung: keine Angabe. München, Germany: Elektronische Krankenakten. Springer.
Oldenbourg.
Hahn, J., & Kauffman, R. (2001). Evaluating website
Gronroos, C. (1994). From scientific manage- performance in Internet-based selling from a business
ment to service management. International Jour- value perspective. Paper presented at the International
nal of Service Industry Management, 5(1), 5–20. Conference on Electronic Commerce, Vienna, Austria.
doi:10.1108/09564239410051885
Haines, M. N. (2003). Customization, configuration,
Group, S. (2004). Third quarter research report. The or modification? A taxonomy for Information System
Standish Group International. specialization. In Khosrow-Pour, M. (Ed.), Information
Technology and organizations: Trends, ıssues, challenges
Grover, V., Jeong, S. R., & Segars, A. H. (1996). Infor-
and solutions (pp. 899–900). Hershey, PA: IGI Publishing.
mation systems effectiveness: The construct space and
patterns of application. Information & Management, Häkkinen, L., & Himola, O.-P. (2008). ERP evaluation
31(4), 177–191. doi:10.1016/S0378-7206(96)01079-8 during the shakedown phase: Lessons from an after-sales
division. Information Systems Journal, 18, 73–100.
Grubb, P., & Takang, A. A. (2003). Software maintenance:
doi:10.1111/j.1365-2575.2007.00261.x
Concepts and practice. World Scientific Publishing
Company. doi:10.1142/9789812564429 Hall, R. H. (1980). Effectiveness theory and organisational
effectiveness. The Journal of Applied Behavioral Science,
Grudin, J. (2004, 6-10.November.2004). Return on
16(4), 536–545. doi:10.1177/002188638001600408
ınvestment and organizational adoption. Paper presented
at the ACM conference on Computer supported coopera- Hamilton, S., & Chervany, N. L. (1981). Evaluating
tive work, Chicago, Illinois. Information System effectiveness. Part I. Comparing
evaluation approaches. Management Information Systems
Guarro, S. B. (1987). Principles and procedures of the
Quarterly, 5(3), 55–69. doi:10.2307/249291
LRAM approach to information systems risk analysis
and management. Computers & Security, 6(6), 493–504. Han, K., & Noh, M. (1999). Critical failure factors that
doi:10.1016/0167-4048(87)90030-7 discourage the growth of electronic commerce. Inter-
national Journal of Electronic Commerce, 4(2), 25–43.
Guenzi, P., Georges, L., & Pardo, C. (2009). The impact
of strategic account managers’ behaviors on relational Hansen, K. M., Jonasson, K., & Neukirchen, H. (2011).
outcomes: An empirical study. Industrial Marketing An empirical study of software architectures’ effect on
Management, 38(3), 300–311. doi:10.1016/j.indmar- product quality. Journal of Systems and Software, 84,
man.2007.09.011 1233–1243. doi:10.1016/j.jss.2011.02.037
Gulbins, J., Seyfried, M., & Strack-Zimmermann, H. Hanssens, D. M., Parsons, L. J., & Schultz, R. L. (1990).
(2002). Dokumenten Management (3rd ed.). Berlin, Market response models: Econometric and time series
Germany: Springer. analysis. Boston, MA: Kluwer Academic Publishers.
Gunasekaran, A., Ngai, E. W. T., & McGaughey, R. E. Hao, X., Duo-Lin, L., & Zhi-Jie, L. (2010). The research
(2006). Information Technology and systems justification: on e-commerce website success mode.
A review for research and applications. European Journal
Hardaker, G., & Smith, D. (2002). E-learning
of Operational Research, 173(3), 957–983. doi:10.1016/j.
communities, virtual markets and knowledge cre-
ejor.2005.06.002
ation. European Business Review, 14(5), 342–350.
doi:10.1108/09555340210444194
425
Harland, C. M., Caldwell, N. D., Powell, P., & Zheng, J. Heeks, R. D., Mundy, D., & Salazar, A. (1999). Why health
(2007). Barriers to supply chain information integration: care Information Systems succeed or fail: Information
SMEs adrift of eLands. Journal of Operations Manage- Systems for public sector management working paper
ment, 25(6), 1234–1254. doi:10.1016/j.jom.2007.01.004 no. 9. Manchester, UK: Institute for Development Policy
and Management, University of Manchester.
Harrington, S. J. (1996). The effect of codes of ethics
and personal denial of responsibility on computer abuse Heeks, R., Mundy, D., & Salazar, A. (1999). Why health-
judgments and intentions. Management Information care Information Systems succeed or fail. Information
Systems Quarterly, 20(3), 257–278. doi:10.2307/249656 Systems for public sector management. Working Paper
Series, Paper No. 9, 1999.
Harris, M., & Weistroffer, H. (2009). A new look at the
relationship between user involvement in systems de- Heeks, R. (2002). Information systems and develop-
velopment and system success. Communications of the ing countries: Failure, success, and local improvi-
Association for Information Systems, 24(1), 42. sations. The Information Society, 18(2), 101–112.
doi:10.1080/01972240290075039
Harrison, D. A., Mykytyn, P. P., & Riemenschneider,
C. K. (1997). Executive decisions about adoption of Heeks, R. (2006). Health Information Systems: Failure,
Information Technology in small business: Theory and success and improvisation. International Journal of
empirical tests. Information Systems Research, 8(2), Medical Informatics, 75(2), 125–137. doi:10.1016/j.
171–195. doi:10.1287/isre.8.2.171 ijmedinf.2005.07.024
Harvey, L. J., & Myers, M. D. (1995). Scholarship and Henri, J.-F. (2004). Performance measurement and organi-
practice: The contribution of ethnographic research sational effectiveness: Bridging the gap. Managerial Fi-
methods to bridging the gap. Information Technology & nance, 30(6), 93–123. doi:10.1108/03074350410769137
People, 8(3), 13–27. doi:10.1108/09593849510098244
Henseler, J., Ringle, C. M., & Sinkovics, R. (2009). The
Hausken, K. (2006). Income, interdependence, and substi- use of partial least squares path modelling in interna-
tution effects affecting incentives for security investment. tional marketing. Advances in International Marketing,
Journal of Accounting and Public Policy, 25, 629–665. 20, 277–319.
doi:10.1016/j.jaccpubpol.2006.09.001
Heo, J., & Han, I. (2003). Performance measure of Infor-
Hausken, K. (2007). Information sharing among firms and mation Systems (IS) in evolving computing environments:
cyber attacks. Journal of Accounting and Public Policy, an empirical investigation. Information & Management,
26, 639–688. doi:10.1016/j.jaccpubpol.2007.10.001 40(4), 243–256. doi:10.1016/S0378-7206(02)00007-1
Havenstein, H. (2005). Healthcare provider nears end of Herath, T., & Rao, H. R. (2009). Protection motivation
10-Year ERP journey. Computerworld, 39(51), 52–53. and deterrence: A framework for security policy compli-
ance in organisations. European Journal of Information
Hawkins, D. T. (1977). Unconventional uses of on-line
Systems, 18(2), 106–126. doi:10.1057/ejis.2009.6
information retrieval systems: On-line bibliometric
studies. Journal of the American Society for Information Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J.
Science American Society for Information Science, 28, T. (2004). Evaluating collaborative filtering recommender
13–18. doi:10.1002/asi.4630280103 systems. ACM Transactions on Information Systems,
22(1), 5–53. doi:10.1145/963770.963772
Hayam, A., & Oz, E. (1993). Integrating data security
into the systems development life. Journal of Systems Hickey, A. M., Davis, A. M., & Kaiser, D. (2003). Re-
Management, 44(8), 16–21. quirements elicitation techniques. Comparative Technol-
ogy Transfer and Society, 1(3), 279–304. doi:10.1353/
Heales, J. (2002). A model of factors affecting an infor-
ctt.2003.0026
mation system’s change in state. Journal of Software
Maintenance and Evolution: Research and Practice, 14,
409–427. doi:10.1002/smr.271
426
Hicks, B., Culley, S., McMahon, C., & Powell, P. (2010). Hochstein, A., Zarnekow, R., & Brenner, W. (2005a).
Understanding Information Systems infrastructure in Evaluation of service-oriented IT management in prac-
engineering SMEs: A case study. Journal of Engineering tice. Proceedings of the International Conference on
and Technology Management, 27(1 2),57-73. Services Systems and Services Management (ICSSSM)
(Vol. 1, pp. 80-84).
Highsmith, J., & Cockburn, A. (2001). Agile software
development: the business of innovation. Computer, Hochstein, A., Zarnekow, R., & Brenner, W. (2005b).
34(9), 120–127. doi:10.1109/2.947100 ITIL as common practice reference model for IT service
management: formal assessment and implications for
Hills, M. (1997a). Intranet as groupware. New York, NY:
practice. Proceedings of the 2005 IEEE International
John Wiley and Sons.
Conference on e-Technology, e-Commerce and e-Service
Hills, M. (1997b). Intranet business strategies. New York, (pp. 704-710). Hong Kong.
NY: John Wiley and Sons.
Hodges, M. (1997). Preventing culture clash on the World
HIMSS Security Survey. (2009). Healthcare information Wide Web. Technology Review, 100(8), 18–19.
and management systems society. Retrieved 13 May 2011
Hodges, M. S. (2007). Computers: Systems, terms and
from www.himss.org/content/files/HIMSS2009Security-
acronyms (17th ed.). SemCo.
SurveyReport.pdf
Hofer, C., & Sandberg, W. (1987). Improving new venture
Hinton, P. R., Brownlow, C., Mcmurray, I., & Cozens, B.
performance: Some guidelines for success. American
(2004). SPSS explained. London, UK: Routledge Taylor
Journal of Small Business, 12(Summer), 11–25.
& Francis Group.
Hoffman, D., Novak, T., & Chatterjee, P. (1995). Com-
Hitchings, J. (1996). A practical solution to the complex
mercial scenarios for the Web: Opportunities and chal-
human issues of information security design. In Katsi-
lenges. Journal of Computer-Mediated Communication,
kas, S. (Ed.), Information Systems security: Facing the
1(3), 1–20.
information society of the 21st century. London, UK:
Chapman & Hall. Ho, J. (1997). Evaluating the World Wide Web: A global
study of commercial sites. Journal of Computer-Mediated
Hitt, L., & Brynjolfsson, E. (1996). Productivity, busi-
Communication, 3(1). Retrieved from http://www.ascusc.
ness profitability, and consumer surplus: Three different
org/jcmc/vol3/issue1/ho.html
measures of Information Technology value. Manage-
ment Information Systems Quarterly, 20(2), 121–142. Holsapple, C. W. (2008). The pulse of multipar-
doi:10.2307/249475 ticipant systems. Journal of Organizational Com-
puting and Electronic Commerce, 18(4), 333–343.
Hitt, M. (1988). The measuring of organisational effective-
doi:10.1080/10919390802453617
ness: Multiple domains and constituencies. Management
International Review, 28(2), 28–40. Holsapple, C., & Lee Post, A. (2006). Defining, assess-
ing, and promoting e learning success: An Information
Hochstein, A. (2004). Managing IT service quality as
Systems perspective. Decision Sciences Journal of
perceived by the customer: The service oriented IT
Innovative Education, 4(1), 67–85. doi:10.1111/j.1540-
SERVQUAL. Paper presented at the International Tele-
4609.2006.00102.x
communications Society (ITS) 15th Biennial Conference.
Hone, K., & Eloff, J. H. P. (2002). Information secu-
Hochstein, A., Tamm, G., & Brenner, W. (2005, May
rity policy - What do international information security
26 - 28). Service-oriented IT management: Benefit, cost
standards say? Computers & Security, 21(5), 402–409.
and success factors. Paper presented at the Thirteenth
doi:10.1016/S0167-4048(02)00504-7
European Conference on Information Systems, Regens-
burg, Germany.
427
Hong, & Kim. (2004). Architectural criteria for web- Høstgaard, A. M., & Nøhr, C. (2004). Metodehaandbog
site evaluation–Conceptual framework and empirical i forandringsparathed i forbindelse med udvikling og
validation. Behaviour & Information Technology, 23(5), implementering af nye it-systemer indenfor sundheds-
337-357. vaesnet (handbook in methods for change readiness in
development and implementation of new IT-systems in
Hong, I. (2007). A survey of website success metrics used
healthcare). Aalborg, Denmark: Virtual Center for Health
by Internet-dependent organizations in Korea. Internet Re-
Informatics.
search, 17(3), 272–290. doi:10.1108/10662240710758920
Hovav, A., & D’Arcy, J. (2003). The impact of denial-
Hong, S.-J., & Lerch, F. J. (2002). A laboratory study of
of-service attack announcements on the market value
consumers’ preferences and purchasing behavior with
of firms. Risk Management & Insurance Review, 6(2),
regards to software components. ACM SIGMIS Database,
97–121. doi:10.1046/J.1098-1616.2003.026.x
33(3), 23–37. doi:10.1145/569905.569909
Hsieh, J. J. P. A., & Wang, W. (2007). Explaining em-
Hong, W., & Zhu, K. (2006). Migrating to Internet-based
ployees’ extended use of complex Information Systems.
e-commerce: Factors affecting e-commerce adoption and
European Journal of Information Systems, 16(3), 216–227.
migration at the firm level. Information & Management,
43(2), 204–221. doi:10.1016/j.im.2005.06.003
Hsiu-Fen, L., & Gwo-Guang, L. (2006). Determinants
Hoo, K. (2000). How much is enough? A risk-management
of success for online communities: An empirical study.
approach to computer security. A Consortium for Research
Behaviour & Information Technology, 25(6), 479–488.
on Information Security Policy (CRISP) Working Paper.
doi:10.1080/01449290500330422
Stanford, CA: Stanford University.
Huang, C. D., Hu, Q., & Behara, R. S. (2008). An economic
Hopkins, C. D., Roster, C. A., & Wood, C. M. (2006).
analysis of the optimal information security investment
Making the transition to retirement: Appraisals, post-
in the case of a risk-averse firm. International Journal of
transition lifestyle, and changes in consumption pat-
Production Economics, 114(2), 793–804. doi:10.1016/j.
terns. Journal of Consumer Marketing, 23(2), 89–101.
ijpe.2008.04.002
doi:10.1108/07363760610655023
Huang, H. (2009). Designing a knowledge-based system
Horch, J. W. (2003). Practical guide to software quality
for strategic planning: A balanced scorecard perspec-
management (2nd ed.). Artech House.
tive. Expert Systems with Applications, 36(1), 209–218.
Horton, R. P., Buck, T., Waterson, P. E., & Clegg, C. doi:10.1016/j.eswa.2007.09.046
W. (2001). Explaining Intranet use with the technology
Huang, H. H., & Hsu, J. S. C. (2005). An evaluation of
acceptance model. Journal of Information Technology,
publication productivity in Information Systems: 1999 to
16(4), 237–249. doi:10.1080/02683960110102407
2003. Communications of the Association for Information
Høstgaard, A. M. (2009). Fryder forandring? casestudie Systems, 15(31), 555–564.
af EPJ udbudsprocessen i region nord - belyst gennem
Huang, L. (2008). Strategic orientation and perfor-
en procesevaluering med fokus på laegerne som sociale
mance measurement model in Taiwan’s travel agen-
baerere af den elektroniske patientjournal, EPJ (Change
cies. Service Industries Journal, 28(10), 1357–1383.
is the spice of life? process analysis case study of a region
doi:10.1080/02642060802250179
in Denmark) (Unpublished doctoral dissertation). Aalborg
University, Denmark. Huang, R., Zmud, R. W., & Price, R. L. (2010). Influenc-
ing the effectiveness of IT governance practices through
steering committees and communication policies. Euro-
pean Journal of Information Systems, 19(3), 288–302.
doi:10.1057/ejis.2010.16
428
Huang, S., Lee, C., & Kao, A. (2006). Balancing Huq, Z., & Martin, T. N. (2006). The recovery of BPR
performance measures for information security man- implementation through an ERP approach – A hospital
agement: A balanced scorecard framework. Indus- case study. Business Process Management Journal, 12(5),
trial Management & Data Systems, 106(2), 242–255. 576–587. doi:10.1108/14637150610691000
doi:10.1108/02635570610649880
Hwang, H., Yeh, R., Chen, H., Jiang, J., & Klein, G. (2002).
Hubbard, G. (2009). Measuring organisational perfor- IT investment strategy and IT infrastructure services. The
mance: Beyond the triple bottom line. Business Strategy Review of Business Information Systems, 6(2), 55–63.
and the Environment, 18(3), 177–191. doi:10.1002/
Hwang, Y., & Kim, D. J. (2007). Customer self-service
bse.564
system: The effect of perceived Web quality with service
Hübner-Bloder, G., & Ammenwerth, E. (2009). Bench- contents on enjoyment, anxiety, and e-trust. Decision Sup-
mark hospital Information Systems – A Delphi study. port Systems, 43, 446–760. doi:10.1016/j.dss.2006.12.008
Methods of Information in Medicine, 48, 508–518.
Iacovou, C. L., & Nakatsu, R. (2008). A risk profile of off-
doi:10.3414/ME09-01-0044
shore-outsourced development projects. Communications
Huchendorf, S. C. (2005). Information Technology infra- of the ACM, 51(6), 89–94. doi:10.1145/1349026.1349044
structure library (ITIL) performance metrics. University
IEEE. (1990). IEEE standard computer dictionary: A
of Minnesota, College of Continuing Education 2005.
compilation of IEEE standard computer glossaries -
Hughes, M., & Golden, W. (2004). Achieving business 610. IEEE.
process change in the public sector: Revolution or evolu-
IEEE. (1993). IEEE standard for software productivity
tion? Electronic Government, an International Journal,
metrics. (IEEE Std 1045-1992).
1(2), 152-165.
IEEE. (2006). IEEE Std 982.1 - 2005 IEEE standard
Huizingh, K. A., Bijmolt, T., & Hoekstra, J. (2007). How
dictionary of measures of the software aspects of depend-
important are transactional or informational functions
ability. (Revision of IEEE Std 982.1-1988: IEEE).
for website success? Paper presented at the ANZMAC
2007, Dunedin, New Zealand. Ifinedo, P. (2005). Do organisational technological con-
tingency factors influence the perception of ERP systems
Huizingh, K. R. E. (1995), Marketing and sales Informa-
success? An exploratory study in the Baltic-Nordic Region
tion Systems. Schoonhoven, The Netherlands: Academic
of Europe. Conference Proceedings, The 4th International
Service (Dutch text).
Business Information Management Association (IBIMA)
Huizingh, E. (2002). The antecedents of website per- Conference, Lisbon, pp. 427 – 437.
formance. European Journal of Marketing, 36(11/12),
Ifinedo, P. (2006). Extending the Gable et al. enterprise
1225–1247. doi:10.1108/03090560210445155
systems success measurement model: A preliminary study.
Hung, S. (2003). Expert versus novice use of the execu- Journal of Information Technology Management,17(1),
tive support systems: An empirical study. Information 14-33.
& Management, 40(3), 177–189. doi:10.1016/S0378-
Ifinedo, P., & Nahar, N. (2006). Quality, impact and suc-
7206(02)00003-4
cess of ERP-systems: A study involving some firms in the
Hung, W., & McQueen, R. (2004). Developing an Nordic-Baltic region. Journal of Information Technology
evaluation instrument for e-commerce websites from the Impact, 6(1), 1–28.
first-time buyer’s viewpoint. The Electronic Journal of
Ifinedo, P., & Nahar, N. (2009). Interactions between
Information Systems Evaluation, 7(1), 31–42.
contingency, organizational IT factors, and ERP suc-
Hunsinger, D. S., & Smith, M. A. (2009). IT certification cess. Industrial Management & Data Systems, 109(1-2),
use by hiring personnel. Journal of Computer Information 118–137. doi:10.1108/02635570910926627
Systems, 50(2), 71–82.
429
Igbaria, M., Zinatelli, N., Cragg, P., & Cavaye, A. L. IT Governance Institute. (2007a). Cobit quickstart (2nd
M. (1997). Personal computing acceptance factors in ed.). IT Governance Institute.
small firms: A structural equation model. Manage-
IT Governance Institute. (2007b). Cobit v.4.1. IT Gov-
ernance Institute.
doi:10.2307/249498
itSMF International. (2008). Self assessment (ITIL V2).
Iivari, J. (2005). An empirical test of the DeLOne-McLean
Retrieved 07 February, 2011, from http://www.itsmfi.org/
model of Information Systems success. The Data Base for
content/self-assessment-itil-v2
Advances in Information Systems, 36(2), 8–27.
Ittner, C. D., & Lacker, D. F. (2003). Coming up short
Information Systems Audit and Control Association
on nonfinancial performance measurement. Harvard
(ISACA). (2009). An introduction to the business model
Business Review, (November): 1–9.
for information security. Retrieved 15 November, 2009,
from http://www.isaca.org/AMTemplate.cfm?Section= Iversen, J. H., Mathiassen, L., & Nielsen, P. A. (2004).
Deliverables&Template=/ContentManagement/Content- Managing risk in software process improvement: An ac-
Display.cfm&ContentID=48017 tion research approach. Management Information Systems
Quarterly, 28(3), 395–433.
Insiti, M. (2007). ERP end-users business productivity:
A field study of SAP & Microsoft. Keystone Strategy Ives, B., Hamilton, S., & Davis, G. (1980). A framework
Research 2007. Retrieved June 12, 2010, from http:// for research in computer-based management Informa-
download.microsoft.com tion Systems. Management Science, 26(9), 910–934.
doi:10.1287/mnsc.26.9.910
Institute for Information Infrastructure Protection. (I3P).
(2009). National cyber security research and develop- Ives, B., & Olson, M. H. (1984). User involvement and
ment challenges: An industry, academic, and government MIS success: A review of research. Management Science,
perspective. Retrieved 22 December, 2009, from http:// 30(5), 586–603. doi:10.1287/mnsc.30.5.586
www.thei3p.org/docs/publications/i3pnationalcyberse-
curity.pdf Ives, B., Olson, M. H., & Baroudi, J. J. (1983). The measure-
ment of user information satisfaction. Communications
Ip, C., Law, R., & Lee, H. A. (2010). A review of website of the ACM, 26, 785–793. doi:10.1145/358413.358430
evaluation studies in the tourism and hospitality fields from
1996 to 2009. International Journal of Tourism Research, Ives, B., Olson, M., & Baroudi, J. (1983). The measure-
Early View (Online Version of Record published before ment of user information satisfaction. Communications of
inclusion in an issue). the ACM, 26(10), 785–793. doi:10.1145/358413.358430
Ishman, M. D., Pegels, C. C., & Sanders, G. L. (2001). Jacobs, B. (2004). Using soft systems methodology for
Managerial information: System success factors within performance improvement and organisational change
the cultural context of North America and a former Soviet in the English national health service. Journal of Con-
Republic. The Journal of Strategic Information Systems, tingencies and Crisis Management, 12(4), 138–149.
10(4), 291–312. doi:10.1016/S0963-8687(01)00055-5 doi:10.1111/j.0966-0879.2004.00444.x
ISO/IEC. (2005). ISO/IEC 20000-1:2005 Information Jacobsen, K., & Larsen, K. (2007). Ve og velfaerd; laeger,
technology service management, part 1: Specification (pp. sundhed og samfund gennem 200 år (woe and welfare:
16). Sydney, Australia: Standards Australia. itSMF Ger- Physicians, health and society during 200 years). Køben-
many. (2008). Total value of IT (TVITEM). Retrieved 07 havn, Denmark: Linhart og Ringhof.
February, 2011, from http://www.itsmf.de/ak_tv_it.html James, L., Mulaik, S., & Brett, J. (1982). Causal analysis:
IT Governance Institute. (2003). Board briefing on IT gov- Assumptions, models, and data. Beverly Hills, CA: Sage.
ernance. Rolling Meadows, IL: IT Governance Institute.
430
Jamieson, D., Vinsen, K., & Callender, G. (2005). Agile Jespersen, H. (2004). Sygepleje og EPJ (Nursing and
procurement: New acquisition approach to agile soft- EHR). Aarhus, Denmark: Institut for Sygeplejevidenskab.
ware development. 31st EUROMICRO Conference on Denmark: Aarhus University.
Software Engineering and Advanced Applications, 2005,
Jiang, J. J., Klein, Roan, G., J., & Lin, J. T. M. (2001).
(pp. 266-273).
IS service performance: Self-perceptions and user per-
Jamieson, K. (2007). Information systems decision mak- ceptions. Information & Management, 38(8), 499–506.
ing: Factors affecting decision makers and outcomes. doi:10.1016/S0378-7206(01)00072-6
Central Queensland University, Rockhampton.
Jing, Y., Jian, X., Xue-Long, Z., & Feng-Yu, L. (2005,
Janssen, M., & Klievink, B. (2010). ICT-project failure in 10-12.October.2005). Modeling and availability analysis
public administration: the need to include risk manage- of nested software rejuvenation policy. Paper presented
ment in enterprise architectures. Paper presented at the at the IEEE International Conference on Systems, Man
11th Annual International Digital Government Research and Cybernetics.
Conference on Public Administration Online: Challenges
Johnson, C. W. (2006). Why did that happen? Exploring
and Opportunities, Puebla, Mexico.
the proliferation of barely usable software in healthcare
Jarvenpaa, S. L., & Ives, B. (1991). Executive involve- systems. Quality & Safety in Health Care, 15(supplement
ment and participation in the management of Information 1), i76–i81. doi:10.1136/qshc.2005.016105
Technology. Management Information Systems Quarterly,
Jones, A. (1997). Penetration testing and system audit.
15(2), 205–227. doi:10.2307/249382
Computers & Security, 16, 595–602. doi:10.1016/S0167-
Jarvis, C. B., MacKenzie, S. B., & Podsakoff, P. M. 4048(97)80796-1
(2003). A critical review of construct indicators and
Jones, C. (2003). Variations in software development
measurement model misspecification in marketing and
practices. Software, IEEE, 20(6), 22–27. doi:10.1109/
consumer research. The Journal of Consumer Research,
MS.2003.1241362
30(2), 199–218. doi:10.1086/376806
Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online uni-
Jazayeri, M. (2005, 5-6.September. 2005). Species evolve,
versity students’ satisfaction and persistence: Examining
individuals age. Paper presented at the Eighth Interna-
perceived level of presence, usefulness and ease of use as
tional Workshop on Principles of Software Evolution
predictors in a structural model. Computers & Education,
(IWPSE’05), Lisbon, Portugal.
57, 1654–1664. doi:10.1016/j.compedu.2011.02.008
Jehangir, M., Dominic, P., Naseebullah, N., & Khan, A.
Jørgensen, M., & Grimstad, S. (2005). Over-optimism
(2011). Towards digital economy: The development of ICT
in software development projects: “The winner’s curse”.
and e-commerce in Malaysia. Modern Applied Science,
Proceedings of IEEE ConieleComp, (pp. 280–285).
5(2), 171–178. doi:10.5539/mas.v5n2p171
Jusoh, R., Ibrahim, D. N., & Zainuddin, Y. (2008).
Jensen, J. (2003). Issues facing SMEs in their adoption
The performance consequences of multiple perfor-
of electronic commerce. Retrieved August 22, 2006, from
mance measures usage: Evidence from the Malaysian
http://www.ica.ogit.gov.au/papers03/jensenpaper12final.
manufacturers. International Journal of Productiv-
pdf
ity and Performance Management, 57(2), 119–136.
Jensen, M. C. (2002). Value maximization, stakeholder doi:10.1108/17410400810847393
theory, and the corporate objective function. Business
Kahn, B., Strong, D., & Wang, R. (2002). Informa-
Ethics Quarterly, 12(2), 235–256. doi:10.2307/3857812
tion quality benchmarks: Product and service perfor-
Jen, Y. W., & Chao, C. C. (2008). Measuring mobile patient mance. Communications of the ACM, 45(4), 184–192.
safety Information System success: An empirical study. doi:10.1145/505248.506007
International Journal of Medical Informatics, 77(10),
Kaliski, B. (1993). A survey of encryption standards.
689–697. doi:10.1016/j.ijmedinf.2008.03.003
IEEE Microcomputers, 13, 74–81. doi:10.1109/40.248057
431
Kalra, D., & Ingram, D. (2006). Information Technology Kaplan, R., & Norton, D. (2001). Transforming the
solutions for healthcare. Elsevier. balanced scorecard from performance measurement to
strategic management: Part I. Accounting Horizons, 15(1),
Kane, G. C. (2006). Casting the net: A multimodal network
87–104. doi:10.2308/acch.2001.15.1.87
perspective on knowledge sharing. Dissertation Abstracts
International, 67(09). (UMI No: 1232407861) Kappas, A. (2006). Appraisals are direct, immedi-
ate, intuitive, and unwitting … and some are reflec-
Kang, I., Lee, S., & Choi, J. (2004). Using fuzzy cogni-
tive…. Cognition and Emotion, 20(7), 952–975.
tive map for the relationship management in airline ser-
doi:10.1080/02699930600616080
vice. Expert Systems with Applications, 26(4), 545–555.
doi:10.1016/j.eswa.2003.10.012 Kardaras, D., & Karakostas, B. (1999). The use of fuzzy
cognitive maps to stimulate the information systems
Kankanhalli, A., Teo, H. H., Tan, B. C. Y., & Wei, K.
strategic planning process. Information and Software
K. (2003). An integrative study of Information Systems
Technology, 41(4), 197–210. doi:10.1016/S0950-
security effectiveness. International Journal of Infor-
5849(98)00125-6
mation Management, 23, 139–254. doi:10.1016/S0268-
4012(02)00105-6 Karlsbjerg, J., & Damsgaard, J. (2001). Make or buy. A
taxonomy of Intranet implementation strategies. Global
Kaplan, B. (2000). Culture counts: How institutions
Co-Operation in the New Millennium the 9th European
values affect computer use. M.D. Computing, 17, 23–26.
Conference on Information Systems, Bled, Slovenia,
Kaplan, B., & Harris-Salamone, K. (2009). Health IT June 27-29.
success and failure: Recommendations from literature
Katerattanakul, P., & Siau, K. (1999). Measuring informa-
and an AMIA workshop. Journal of the American Medical
tion quality of websites: Development of an instrument.
Informatics Association, 16(3), 291–299. doi:10.1197/
Paper presented at the The Twentieth International Con-
jamia.M2997
ference on Information Systems.
Kaplan, R. (1983). Measuring manufacturing perfor-
Katzan, H. (2008). Service science concepts, technology,
mance: A new challenge for managerial accounting
management. Bloomington, IN: iUniverse, Inc.
research. Accounting Review, 58(October), 686–705.
Kauffman, R., & Walden, E. (2001). Economics and elec-
Kaplan, R. (2009). Measuring performance (Pocket Men-
tronic commerce: Survey and directions for research. In-
tor). Boston, MA: Harvard Business Press.
ternational Journal of Electronic Commerce, 5(4), 5–116.
Kaplan, R. S. (2009). Conceptual foundations of the bal-
Kaye, R., Kokia, E., & Shalev, V., Ida,r D., & Chinitz, D.
anced scorecard. In Chapman, C. S., Hopwood, A. G.,
(2010). Barriers and success factors in health Informa-
& Shields, M. D. (Eds.), Handbook of management ac-
tion Technology: A practitioner’s perspective. Journal of
counting research (pp. 1253–1271). Oxford, UK: Elsevier.
Management & Marketing in Healthcare, 3(2), 163–175.
Kaplan, R., & Norton, D. (1993). Putting the balanced doi:10.1179/175330310X12736577732764
scorecard to work. Harvard Business Review, 71(5),
Keegan, D., Eiler, R. G., & Jones, C. R. (1989). Are your
134–142.
performance measures obsolete? Management Account-
Kaplan, R., & Norton, D. (1996a). The balanced score- ing, 70(12), 45–60.
card: Translating strategy into action. Harvard Business
Keel, A. J., Orr, M. A., Hernandez, R. R., Patrocinio, E.
School Press.
A., & Bouchard, J. (2007). From a technology-oriented
Kaplan, R., & Norton, D. (1996b). Using the balanced to a service-oriented approach to IT management. IBM
scorecard as a strategic management system. Harvard Systems Journal, 46(3), 549–564. doi:10.1147/sj.463.0549
Business Review, 74(1), 75–85.
Keen, P. G. W. (1981). Information Systems and orga-
nizational change. Communications of the ACM, 24(1),
24–33. doi:10.1145/358527.358543
432
Keider, S. P. (1984). Why systems development projects Kimiloglu, H., & Zarali, H. (2009). What signifies success
fail. Journal of Information Systems Management, 1(3), in e-CRM? Marketing Intelligence & Planning, 27(2),
33–33. doi:10.1080/07399019408963043 246–267. doi:10.1108/02634500910945011
Keiser, J., & Utzinger, J. (2005). Trends in the core litera- Kim, K. K. (1989). User satisfaction: A synthesis of
ture on tropical medicine: A bibliometric analysis from three different perspectives. The Journal of Information
1952-2002. Scientometrics, 62(3), 351–365. doi:10.1007/ Systems, 4(1), 1–12.
s11192-005-0027-3
Kim, S., Shaw, T., & Schneider, H. (2003). Website design
Kelley, H. (2001). Attributional analysis of computer benchmarking within industry groups. Internet Research,
self-efficacy. Unpublished doctoral dissertation, Richard 13(1), 17–26. doi:10.1108/10662240310458341
Ivey School of Business.
King, W., & Flor, P. (2008). The development of global
Kelley, M. (1994). Productivity and Information Technol- IT infrastructure. Omega, 36(3), 486–504. doi:10.1016/j.
ogy: The elusive connection. Management Science, 40(11), omega.2006.02.004
1406–1425. doi:10.1287/mnsc.40.11.1406
Kinnula, M., Seppanen, V., Warsta, J., & Vilminko, S.
Kelly, G. (1970). A brief introduction to personal construct (2007). The formation and management of a software
theory. London, UK: Academic Press. outsourcing partnership process. 40th Annual Hawaii
International Conference on System Sciences, HICSS
Kensing, F., Simonsen, J., & Bødker, K. (1996). MUST
2007, (p. 240b).
- A method for participatory design. The Fourth Biennial
Conference on Participatory Design, (pp. 129-140). Kippenberger, T. (2000). Management’s role in proj-
ect failure. The Antidote, 5(4), 30–33. doi:10.1108/
Kensing, F., & Blomberg, J. (1998). Participatory design:
EUM0000000006793
Issues and concerns. Computer Supported Cooperative
Work, 7, 167–185. doi:10.1023/A:1008689307411 Kirsch, L. J. (1997). Portfolios of control modes and IS
project management. Information Systems Research, 8(3),
Kent, M. L. (1998). Does your website attract or repel
215–239. doi:10.1287/isre.8.3.215
customers? Three tests of website effectiveness. Public
Relations Quarterly, 43(4), 31–34. Klecun, E., & Cornford, T. (2005). A critical approach
to evaluation. European Journal of Information Systems,
Kettinger, W., & Lee, C. (2005). Zones of tolerance:
14(3), 229–243. doi:10.1057/palgrave.ejis.3000540
Alternative scales for measuring Information Systems
service quality. Management Information Systems Quar- Knapp, K. J., Morris, R. F., Marshall, T. E., & Byrd, T. A.
terly, 29(4), 607–623. (2009). Information security policy: An organizational-
level process model. Computers & Security, 28(7),
Khalfan, A. (2003). A case analysis of business process
493–508. doi:10.1016/j.cose.2009.07.001
outsourcing project failure profile and implementation
problems in a large organisation of a developing nation. Kodama, M. (2005). New knowledge creation through
Business Process Management Journal, 9(6), 745–759. leadership-based strategic community—A case of new
doi:10.1108/14637150310506675 product development in IT and multimedia business
fields. Technovation, 25(8), 895–908. doi:10.1016/j.
Kim, & Lee. (2002). Critical design factors for successful
technovation.2004.02.016
e-commerce systems. Behaviour & Information Technol-
ogy, 21(3), 185-199. Kohavi, R., & Parekh, R. (2003). Ten supplementary
analyses to improve e-commerce websites. Paper presented
Kim, H. S., & Lee, K. C. (1998). Fuzzy implications
at the Fifth WEBKDD Workshop, Washington DC, USA.
of fuzzy cognitive map with emphasis on fuzzy causal
relationship and fuzzy partially causal relationship. Fuzzy
Sets and Systems, 97(3), 303–313. doi:10.1016/S0165-
0114(96)00349-1
433
Koh, J., & Kim, Y. (2004). Knowledge sharing in virtual Krauss, M., & Tipton, H. (2002). Handbook of informa-
communities: an e-business perspective. Expert Systems tion security management. Boca Raton, FL: CRC Press.
with Applications, 26(2), 155–166. doi:10.1016/S0957-
Krishnamurthy, S. (2003). E-commerce management:
4174(03)00116-7
Text and cases. Mason, OH: Thomson/South-Western.
Kohli, R., & Grover, V. (2008). Business value of IT: An
Krishnan, M. S., & Ramaswamy, V. (1998). An empirical
essay on expanding research directions to keep up with
analysis of customer satisfaction for Intranet market-
the times. Journal of the Association for Information
ing systems. Decision Support Systems, 24(1), 45–54.
Systems, 9(1), 23–39.
doi:10.1016/S0167-9236(98)00062-1
Kohli, R., Sherer, S., & Baron, A. (2003). Editorial—IT
Kristensen, J. K. (2001). The hidden threat to e-gov-
investment payoff in e-business environments: Research
ernment–Avoiding large government IT failures. OECD
issues. Information Systems Frontiers, 5(3), 239–247.
Public Management Policy Brief.
doi:10.1023/A:1025632525371
Kronbichler, S. A., Ostermann, H., & Staudinger, R.
Kokolakis, S. A., Demopoulos, A. J., & Kiountouzis,
(2009). A review of critical success factors for ERP proj-
E. A. (2000). The use of business process modelling
ects. The Open Information Systems Journal, 3, 14–25.
in Information Systems security analysis and design.
doi:10.2174/1874133900903010014
Information Management & Computer Security, 8(3),
107–116. doi:10.1108/09685220010339192 Kuanda, J. (2003). Culture and technological transforma-
tion in the south (Müller, J., Ed.). Denmark: Samfundslit-
Kontio, E., Korvenranta, H., Lundgren-Laine, H., &
teratur Press.
Salanterä, S. (2009). Key elements of successfull care
process of patients with heart symptoms in an emergency Kuo, Y.-F., Wu, C.-M., & Deng, W.-J. (2009). The rela-
care – Could an ERP system help? In Saranto, K. (Eds.), tionship amongservice quality, perceived value, customer
Connecting health and humans. IOS Press. satisfaction, and post-purchase intention in mobile value-
added services. Computers in Human Behavior, 25,
Kositanuri, B., Nqwenyama, O., & Osei-Bryson, K. M.
887–896. doi:10.1016/j.chb.2009.03.003
(2006). An exploration of factors that impact individual
performance in an ERP environment: An analysis us- Kuppens, P., van Mechelen, I., Smits, D. M., De Boeck, P.,
ing multiple analytical techniques. European Journal & Ceulemans, E. (2007). Individual differences in patterns
of Information Systems, 15(6), 556–568. doi:10.1057/ of appraisal and anger experience. Cognition and Emo-
palgrave.ejis.3000654 tion, 21(4), 689–713. doi:10.1080/02699930600859219
Koski, H. (1999). The implications of network use, Kushniruk, A., & Turner, P. (2011). Who’s users? Partici-
production network externalities and public networking pation and empowerment in socio-technical approaches
programmes for firm’s productivity. Research Policy, to health IT development. International Perspectives in
28(4), 423–439. doi:10.1016/S0048-7333(98)00127-9 Health Informatics, 160, 280–285.
Kosko, B. (1986). Fuzzy cognitive maps. Interna- Kusters, R. J., Trienekens, J. J. M., & Hassoldt, W. (2002).
tional Journal of Man-Machine Studies, 24(24), 65–75. On the business impact of software process improvement.
doi:10.1016/S0020-7373(86)80040-2 In D. C. Martin (Ed.), Proceedings of the IEEE 26th Inter-
national Computer Software & Applications Conference,
Kosko, B. (1996). Fuzzy engineering. Prentice Hall.
(pp. 59-67). Los Alamitos, CA: IEEE Computing Society.
Kotulic, A. G., & Clark, J. G. (2004). Why there aren’t
Kwahk, K. Y., & Lee, J. N. (2008). The role of readiness
more information security research studies. Information &
for change in ERP implementation: Theoretical bases and
Management, 41, 597–607. doi:10.1016/j.im.2003.08.001
empirical validation. Information & Management, 45(7),
Kragh, H. (1996). Cosmology and controversy: The 474–481. doi:10.1016/j.im.2008.07.002
historical development of two theories of the universe.
Princeton University Press.
434
Kwak, Y. H., & Stoddard, J. (2004). Project risk manage- Landrum, H., Prybutok, V., & Zhang, X. (2007). A
ment: lessons learned from software development envi- comparison of Magal’s service quality instrument with
ronment. Technovation, 24(11), 915–920. doi:10.1016/ SERVPERF. Information & Management, 44(1), 104–113.
S0166-4972(03)00033-6 doi:10.1016/j.im.2006.11.002
Kwok, L., & Longley, D. (1997). Code of practice: A Landwehr, C. E., Bull, A. R., McDermott, J. P., &
standard for information security management. In Pro- Choi, W. S. (1994). A taxonomy of computer secu-
ceedings of the IFIP TC11 13th International Conference rity flaws. ACM Computing Surveys, 26(3), 211–254.
on Information Security. doi:10.1145/185403.185412
Kyhlbäck, H., & Sutter, B. (2007). What does it take to Langenwalter, G. (2000). Enterprise resources planning
replace an old functioning information system with a new and beyond: Integrating your entire organization. Boca
one? A case study. International Journal of Medical In- Raton, FL: St. Lucie Press.
formatics, 76(Supplement 1), S149–S158. doi:10.1016/j.
Larsen, T., Sørebø, A., & Sørebø, Ø. (2009). The role
of task-technology fit as users’ motivation to continue
Lacity, M. (2008). Managing relationships across the information system use. Computers in Human Behavior,
life cycle of an outsourcing arrangement. In Lacity, M. 25(3), 778–784. doi:10.1016/j.chb.2009.02.006
C., & Willcocks, L. P. (Eds.), Information Systems and
Lau, A., & Tsui, E. (2009). Knowledge management
outsourcing: Studies in theory and practice (pp. 303–340).
perspective on e-learning effectiveness. Knowledge-
Palgrave MacMillan. doi:10.1057/9780230594838
Based Systems, 22(4), 324–325. doi:10.1016/j.kno-
Lagsten, J. (2011). Evaluating Information Systems ac- sys.2009.02.014
cording to stakeholders: A pragmatic perspective and
Law, C. C. H., & Ngai, E. W. T. (2005). IT business value
method. Electronic Journal Information Systems Evalu-
research: A critical review and research agenda. Interna-
ation, 14(1), 73–88.
tional Journal of Enterprise Information Systems, 1(3),
Lahtela, A., Jäntti, M., & Kaukola, J. (2010). Implement- 35–55. doi:10.4018/jeis.2005070103
ing an ITIL-based IT service management measurement
Law, K. S., Wong, C. S., & Mobley, W. H. (1998). Toward
system. Paper presented at the Fourth International Confer-
a taxonomy of multidimensional constructs. Academy of
ence on Digital Society, St. Maarten, Netherlands Antilles.
Management Review, 23(4), 741–755.
Lai, J. (2006). Assessment of employees’ perceptions of
Lazarus, R. S. (1984). Stress, appraisal, and coping. New
service quality and satisfaction with e-business. Inter-
York, NY: Springer.
national Journal of Human-Computer Studies, 64(9),
926–938. doi:10.1016/j.ijhcs.2006.04.007 Leclercq, A. (2007). The perceptual evaluation of Infor-
mation Systems using the construct of user satisfaction:
Lamb, R., & Davidson, E. (2000). The new computing
Case study of a large French group. The Data Base for
archipelago: Intranet islands of practice. Proceedings
Advances in Information Systems, 38(2), 27–60.
of the IFIPWG8.2 Working Conference on Information
Technology and Changes in Organizational Work. Lederer, A. L., Maupin, D. J., Sena, M. P., & Zhuang, Y.
(2000). The technology acceptance model and the World
Landrum, H., & Prybutok, V. (2004). A service quality
Wide Web. Decision Support Systems, 29(3), 269–282.
and success model for the information service industry.
doi:10.1016/S0167-9236(00)00076-2
European Journal of Operational Research, 156(3),
628–642. doi:10.1016/S0377-2217(03)00125-5 Lederer, A. L., & Mirani, R. (1995). Anticipating the
benefits of proposed information systems. Journal of
Landrum, H., Prybutok, V. R., & Zhang, X. (2010). The
Information Technology, 10(3), 159–169. doi:10.1057/
moderating effect of occupation on the perception of
jit.1995.20
information services quality and success. Computers &
Industrial Engineering, 58(1), 133–142.
435
Lee, Y., Kozar, K. A., & Larsen, K. R. T. (2003). The Leem, C. S., Yoon, C. Y., & Park, S. K. (2004). A
technology acceptance model: Past, present, and the future process-centered IT ROI analysis with a case study. In-
(Vol. 12, pp. 752-780). Communications of Association formation Systems Frontiers, 6(4), 369–383. doi:10.1023/
for Information Systems. B:ISFI.0000046378.42438.5f
Lee, Y., Kozar, K., & Larsen, K. (2003). The technology Lee-Post, A. (2009). E-learning success model: An In-
acceptance model: Past, present, and future. Communica- formation Systems perspective. Electronic Journal of
tions of the Association for Information Systems (Volume e-Learning, 7(1), 61-70.
12, Article 50), 12(50), 725-780.
Lee, S. M., Kim, J., Choi, Y., & Lee, S. G. (2009). Ef-
Lee, A. H., Chen, W., & Chang, C. J. (2008). A fuzzy AHP fects of IT knowledge and media selection on operational
and BSC approach for evaluating performance of IT de- performance of small firms. Small Business Economics,
partment in the manufacturing industry in Taiwan. Expert 32(3), 241–257. doi:10.1007/s11187-007-9095-5
Systems with Applications, 34(1), 96–107. doi:10.1016/j.
Lee, S., & Ahn, H. (2009). Fuzzy cognitive map based on
eswa.2006.08.022
structural equation modeling for the design of controls in
Lee, B., & Barua, A. (1999). An integrated assessment business-to-consumer e-commerce web-based systems.
of productivity and efficiency impacts of Information Expert Systems with Applications, 36(7), 10447–10460.
Technology investments: Old data, new analysis and doi:10.1016/j.eswa.2009.01.070
evidence. Journal of Productivity Analysis, 12, 21–43.
Lee, S., Kim, B. G., & Lee, K. (2004). Fuzzy cognitive
doi:10.1023/A:1007898906629
map-based approach to evaluate EDI performance: A test
Lee, B.-C., Yoon, J.-O., & Lee, I. (2009). Learners’ accep- of causal model. Expert Systems with Applications, 27(2),
tance of e-learning in South Korea: Theories and results. 287–299. doi:10.1016/j.eswa.2004.02.003
Computers & Education, 53, 1320–1329. doi:10.1016/j.
Lee, S., & Kim, K. J. (2007). Factors affecting the imple-
compedu.2009.06.014
mentation success of Internet-based information systems.
Lee, B., & Jeong, Y. (2008). Mapping Korea’s national Computers in Human Behavior, 23(4), 1853–1880.
R&D domain of robot technology by using the co-word doi:10.1016/j.chb.2005.12.001
analysis. Scientometrics, 77(1), 3–19. doi:10.1007/
Lee, Y., & Kozar, K. (2006). Investigating the effect of
s11192-007-1819-4
website quality on e-business success: An analytic hierar-
Lee, D. H., Lee, S. M., Olson, D. L., & Chung, S. H. chy process (AHP) approach. Decision Support Systems,
(2010). The effect of organizational support on ERP 42(3), 1383–1401. doi:10.1016/j.dss.2005.11.005
implementation. Industrial Management & Data Systems,
Lee, Y., Strong, D., Kahn, B., & Wang, R. (2002). AIMQ:
110(1-2), 269–283. doi:10.1108/02635571011020340
A methodology for information quality assessment. In-
Lee, H., & Choi, B. (2003). Knowledge management formation & Management, 40(2), 133–146. doi:10.1016/
enablers, processes, and organizational performance: S0378-7206(02)00043-5
An integrative view and empirical examination. Journal
Lefebvre, L. A., Mason, R., & Lefebvre, E. (1997). The
influence prism in SMEs: The power of CEOs’ percep-
Lee, J., & Lee, W. (2008). The relationship of e-learner’s tions on technology policy and its organizational impacts.
self-regulatory efficacy and perception of e-Learning Management Science, 43(6), 856–878. doi:10.1287/
environmental quality. Computers in Human Behavior, mnsc.43.6.856
24(1), 32–47. doi:10.1016/j.chb.2006.12.001
Legris, P., Ingham, J., & Collerette, P. (2003). Why do
Lee, K. C., Kim, J. S., Chung, H. N., & Kwon, S. J. (2002). people use Information Technology? A critical review
Fuzzy cognitive map approach to Web-mining inference of the technology acceptance model. Information &
amplification. Expert Systems with Applications, 22(3), Management, 40(4), 191–204. doi:10.1016/S0378-
197–211. doi:10.1016/S0957-4174(01)00054-9 7206(01)00143-4
436
Lehman, M. M. (1980, September). Programs, life Leonard, K. J. (2004). Critical success factors relating
cycles, and laws of software evolution. Paper presented to healthcare’s adoption of new technology: A guide to
at the IEEE. increasing the likelihood of successful implementation.
Electronic Healthcare, 2(4), 72–81.
Lehman, M. M., & Belady, L. A. (1985). Program evolu-
tion: Processes of software change. Levy, M., & Powell, P. (2004). Strategies for growth in
SMEs: The role of information and Information Systems.
Lehman, M. M., Ramil, J. F., Wernick, P. D., Perry, D. E.,
Oxford, UK: Elsevier.
& Turski, W. M. (1997, 5-7 November). Metrics and laws
of software evolution-The nineties view. Paper presented Levy, M., & Powell, P. (2008). Small firm transformation
at the Fourth International Software Metrics Symposium, through IS. International Journal of Technology Manage-
Albuquerque, New Mexico, USA. ment, 43(1-3), 123–141. doi:10.1504/IJTM.2008.019411
Lehman, M. (1996). Laws of software evolution revisited. Li, L., Vaidyanathan, K., & Trivedi, K. S. (2002, 3-4.
Software Process Technology, 114, 108–124. doi:10.1007/ October.2002). An approach for estimation of software
BFb0017737 aging in a Web server. Paper presented at the International
Symposium on Empirical Software Engineering.
Lehman, M. M., & Ramil, J. F. (2000). Software evolu-
tion in the age of component-based software engineering. Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., & Zhai, C.
IEE Proceedings. Software, 147, 249–255. doi:10.1049/ (2006). Have things changed now? Paper presented at the
ip-sen:20000922 1st Workshop on Architectural and System Support for
İmproving Software Dependability, San Jose, California.
Lehman, M. M., & Ramil, J. F. (2003). Software evo-
lution—Background, theory, practice. Information Liaw, S. (2008). Investigating students’ perceived
Processing Letters, 88, 33–44. doi:10.1016/S0020- satisfaction, behavioral intention, and effectiveness
0190(03)00382-X of e-learning: A case study of the Blackboard system.
Leitch, S., & Warren, J. (2010). ETHICS: The past,
compedu.2007.09.005
present and future of socio-technical systems design. In
Tatnall, A. (Ed.), History of computing. Learning from Liaw, S., Huang, H., & Chen, G. (2007). Surveying
the past (pp. 189–197). Victoria, Australia: International instructor and learner attitudes toward e-learning. Com-
Federation for Information Processing. doi:10.1007/978- puters & Education, 49(4), 1066–1080. doi:10.1016/j.
3-642-15199-6_19 compedu.2006.01.001
Leiwo, J., Gamage, C., & Zheng, Y. (1999). Organiza- Lichtenstein, Y. (2004). Puzzles in software development
tional modeling for efficient specification of information contracting. Communications of the ACM, 47(2), 61–65.
security requirements. In Proceedings of the Advances in doi:10.1145/966389.966391
Databases and Information Systems: 3rd East European
Li, E. Y. (1997). Marketing Information Systems in small
Conference (ABDIS), (pp. 247-260).
companies. Information Resources Management Journal,
Lenz, R., & Kuhn, K. A. (2003). Zur Architektur und 10(1), 27–35.
Evolution von Krankenhausinformationssystemen, In-
Liebenau, J., & Backhouse, J. (1990). Understanding
novative Informatikanwendungen. Band 2, Beiträge der
information: An introduction. London, UK: Macmillan.
33. Jahrestagung der Gesellschaft für Informatik e.V.
(GI), 29. September – 2. October. Light, J. S. (1999). When computers were women. Tech-
nology and Culture, 40(3), 455–483.
Leonard-Barton, D. (1988). Implementation as mutual ad-
aptation of technology and organization. Research Policy, Lii, Y., Hyung, J., & Tseng, D. (2004). The effects of Web
17(5), 251–267. doi:10.1016/0048-7333(88)90006-6 operational factors on marketing performance. Journal of
American Academy of Business, 5(1/2), 486–494.
437
Li, L. L., Ding, G., Feng, N., Wang, M. H., & Ho, Y. S. Liu, C., & Arnett, K. (2000). Exploring the factors as-
(2008). Global stem cell research trend: Bibliometric sociated with website success in the context of electronic
analysis as a tool for mapping of trends from 1991 to commerce. Information & Management, 38(1), 23–33.
2006. Scientometrics, 80(1), 41–60. doi:10.1016/S0378-7206(00)00049-5
Lillis, K. (2002). Despite advances, software makers and Liu, J., Chen, H., Chen, C., & Sheu, T. (2010). Relation-
hospitals still face formidable challenges. Healthcare ships among interpersonal conflict, requirements uncer-
Purchasing News, November, (pp. 16 – 20). tainty, and software project performance. International
Journal of Project Management, 29(5).
Lin, B. (1991). Health care Information System: A man-
agement perspective. Management Impacts of Information Liu, L., & Ma, Q. (2006). Perceived system performance: A
Technology, 370-387. test of an extended technology acceptance model. Journal
of Organizational and End User Computing, 18(3), 1–24.
Lin, C., Huang, Y., & Tseng, S. (2007). A study of plan-
ning and implementation stages in electronic commerce Livingstone, S., Bober, M., & Helsper, E. J. (2005).
adoption and evaluation: The case of Australian SMEs. Active participation or just more information? Infor-
Contemporary Management Research, 3(1), 83–100. mation Communication and Society, 8(3), 287–314.
doi:10.1080/13691180500259103
Lincke, R., Lundberg, J., & Löwe, W. (2008). Compar-
ing software metrics tools. Paper presented at the 2008 Lloyd, P., & Boyle, P. (1998). Web-weaving: Intranets, ex-
İnternational Symposium on Software Testing and tranets and strategic alliances. Butterworth - Heinemann.
Analysis.
Lohse, G., & Spiller, P. (1998). Electronic Shop-
Lindvall, M., Basili, V., Boehm, B., Costa, P., Dangle, ping. Communications of the ACM, 41(7), 81–88.
K., Shull, F., et al. (2002). Empirical findings in agile doi:10.1145/278476.278491
methods. Proceedings of the 2nd Annual Conference on
Loiacono., e. T., Watson., R. T., & Goodhue., D. L. (2007).
XP/Agile Universe, (pp. 197-207).
WebQual: An instrument for consumer evaluation of
Lin, H. (2007). Measuring online learning systems websites. International Journal of Electronic Commerce,
success: Applying the updated DeLone and McLean 11(3), 51-87.
model. Cyberpsychology & Behavior, 10(6), 817–820.
Loiacono, E., Watson, R., & Goodhue, D. (2002). We-
doi:10.1089/cpb.2007.9948
bQual: A measure of website quality. In Evans, K., &
Lin, H.-F. (2008). Determinants of successful virtual com- Scheer, L. (Eds.), Marketing theory and applications
munities: Contributions from system characteristics and (Vol. 13, pp. 432–438). Chicago, IL: American Market-
social factors. Information & Management, 45, 522–527. ing Association.
doi:10.1016/j.im.2008.08.002
Lorentzen, A. (1994). Teknologi og udvikling i den nor-
Little, J. D. C. (1975). Brandaid, a marketing mix model. djyske maskinindustri (technology and development in
Part 1: structure and Part 2, implementation, calibration engineering in the northern part of Denmark). Aalborg,
and case study. Operations Research, 23(4), 628–655, DK: Aalborg Universitetsforlag.
656–673. doi:10.1287/opre.23.4.628
Lorenzi, N., & Riley, R. T. (1995). Organizational aspects
Liu, Y., Trivedi, K. S., Ma, Y., Han, J. J., & Levendel, H. of health informatics. Springer Verlag.
(2002, 12-15.Nov.2002). Modeling and analysis of soft-
Lorenzi, N., Riley, R. T., Blyth, A. J. C., Southon, G., &
ware rejuvenation in cable modem termination systems.
Dixon, B. J. (1997). Antecedents of the people and orga-
Paper presented at the 13th International Symposium on
nizational aspects of medical informatics: Review of the
Software Reliability Engineering, ISSRE.
literature. Journal of the American Medical Association,
Liu, Z., & Miao, Y. (1999). Fuzzy cognitive map and 4(2), 79–93. doi:10.1136/jamia.1997.0040079
its causal inferences. IEEE International Fuzzy Systems
Conference Proceedings, 3, (pp. 1540-1545).
438
Lormans, M., van Dijk, H., van Deursen, A., Nocker, E., Lu, X. H., Huang, L. H., & Heng, S. H. (2006). Critical
& de Zeeuw, A. (2004). Managing evolving requirements success factors of inter-organizational information sys-
in an outsourcing context: An industrial experience re- tems- A case study of Cisco and Xiao Tong in China. In-
port. Proceedings of the 7th International Workshop on formation & Management, 43(3), 395–408. doi:10.1016/j.
Principles of Software Evolution, (pp. 149-158). im.2005.06.007
Loucopulos, P., Lyytinen, K., Liu, K., Gilb, T., & Ma- Lynch, R. L., & Cross, K. F. (1993). Measure up! Yard-
ciaszek, L. A. (2004). Project failures: Continuing chal- sticks for continuous improvement. Cambridge, Mass:
lenges for sustainable Information Systems. ICEIS’2004 Brackwell Business.
Sixth International Conference on Enterprise Information
Lyytinen, K., Rose, G., & Welke, R. (1998). The brave
Systems.
new world of development in the internetwork computing
Lovata, L. M. (1987). Behavioral theories relating to the architecture (InterNCA): Or how distributed computing
design of information systems. Management Information platforms will change systems development. Information
Systems Quarterly, 11(2), 147–149. doi:10.2307/249354 Systems Journal, 8(3), 241–253. doi:10.1046/j.1365-
2575.1998.00037.x
Love, P. E. D., Irani, Z., & Edwards, D. J. (2004). Industry-
centric benchmarking of Information Technology benefits, Mabert, V. A., Soni, A., & Venkataramanan, M. A. (2000).
costs and risks for small-to-medium sized enterprises in Enterprise resource planning survey of U.S. manufacturing
construction. Automation in Construction, 13(4), 507–524. firms. Production and Inventory Management Journal,
doi:10.1016/j.autcon.2004.02.002 41(2), 52–58.
Lozinsky, S. (1998). Enterprise wide software solutions: MacDonald, C., Stodel, E., Farres, L., Breithaupt, K., &
Integration practices and strategies. Reading, MA: Gabriel, M. (2001). The demand-driven learning model:
Addison-Wesley. A framework for Web-based learning. The Internet and
Higher Education, 4(1), 9–30. doi:10.1016/S1096-
Luftman, J., & Ben-Zvi, T. (2010). Key issues for IT
7516(01)00045-8
executives 2009: Difficult economy’s impact on IT. MIS
Quarterly Executive, 9(1), 49–59. MacDonald, C., & Thompson, T. (2005). Structure, con-
tent, delivery, service, and outcomes: Quality e-learning
Luftman, J., & Brier, T. (1999). Achieving and sustaining
in higher education. International Review of Research in
business-IT alignment. California Management Review,
Open and Distance Learning, 6(2), 1–25.
42(1), 109–122.
Machperson, A. (2005). Learning how to grow: Resolv-
Lu, J. (2003). A model for evaluating e-commerce based on
ing the crisis of knowing. Technovation, 10, 1129–1140.
cost/benefit and customer satisfaction. Information Systems
doi:10.1016/j.technovation.2004.04.002
Frontiers, 5(3), 265–277. doi:10.1023/A:1025645210350
Macmillan. (2003). Macmillan English dictionary for
Lu, J., Chun-Sheng, Y., Liu, C., & Yao, J. E. (2003).
advanced learners. Macmillan Publishers Limited.
Technology acceptance model for wireless Internet. In-
ternet Research: Electronic Networking Applications and Mahaney, R. C., & Lederer, A. L. (1999). Runaway In-
Policy, 13(3), 206–222. doi:10.1108/10662240310478222 formation Systems projects and escalating commitment.
Paper presented at the Special Interest Group on Computer
Lüneborg, J. L., & Nielsen, J. F. (2003). Customer-
Personnel Research Annual Conference, New Orleans,
focused technology and performance in small and large
Lousiana, USA.
banks. European Management Journal, 21(2), 258–269.
doi:10.1016/S0263-2373(03)00020-3 Mahmood, M. (1987). System development methods-A
comparative investigation. Management Information
Luotonen, O. (1993). Risk management and insurances.
Helsinki, Finland: Painatuskeskus Oy.
439
Mahmood, S., & Lai, R. (2006). A complexity measure for Marks, E. A., & Bell, M. (2006). Service-oriented archi-
UML component-based system specification. Software, tecture: A planning and ımplementation guide for business
Practice & Experience, 38(2), 117–134. doi:10.1002/ and technology. John Wiley & Sons.
spe.769
Markus, M. L., & Tanis, C. (2000). The enterprise system
Mahoney, T. A. (1967). Managerial perceptions of or- experience—From adoption to success. In Zmud, R. W.
ganisational effectiveness. Management Science, 14(2), (Ed.), Framing the domains of IT management: Project-
B76–B91. doi:10.1287/mnsc.14.2.B76 ing the future through the past. Cincinnati, OH: Pinnaflex
Educational Resources, Inc.
Mäkelä, S., & Leppänen, V. (2007). A software metric
for coherence of class roles in Java programs. Paper pre- Marthandan, G., & Tang, C. M. (2009). The multi-faceted
sented at the 5th İnternational Symposium on Principles characteristics of IT-enabled organizational effectiveness.
and Practice of Programming in Java, Lisboa, Portugal. Symposium on Progress in Information and Communi-
cation Technology (SPICT 2009), 7-8 December 2009,
Malhotra, R., & Temponi, C. (2010). Critical decisions
Kuala Lumpur, Malaysia.
for ERP integration: Small business issues. Interna-
tional Journal of Information Management, 30(1), 28–37. Marthandan, G., & Tang, C. M. (2010). Information
doi:10.1016/j.ijinfomgt.2009.03.001 Technology evaluation: Issues and challenges. Journal
of Systems and Information Technology, 12(1), 37–55.
Mansky, T. (2001). Asuwirkungen der Reform des Kran-
doi:10.1108/13287261011032643
kenhausentgeltsystems auf die stationäre Versorgung
des Krankenhausmanagement. In Ackermann, T. (Ed.), Martinez-Torres, M., Toral Marín, S., Garcia, F., Vazquez,
Schwerpunkt: Vergütungsreform mit DRGs (pp. 173–194). S., Oliva, M., & Torres, T. (2008). A technological accep-
Schattauer. tance of e-learning tools used in practical and laboratory
teaching, according to the European higher education area.
Ma, Q., Johnston, A. C., & Pearson, J. M. (2008).
Behaviour & Information Technology, 27(6), 495–505.
Information security management objectives and
doi:10.1080/01449290600958965
practices: A parsimonious framework. Information
Management & Computer Security, 16(3), 251–270. Martin, N., Gregor, S., & Hart, D. (2005). The social
doi:10.1108/09685220810893207 dimensions of business and IS/IT alignment: Case studies
of six public-sector organisations. Australian Accounting
Marcati, A., Guido, G., & Peluso, A. M. (2008). The role
Review, 15(3), 28–38.
of SME entrepreneurs’ innovativeness and personality
in the adoption of innovations. Research Policy, 37(9), Martinsons, M., Davison, R., & Tse, D. (1999). The bal-
1579–1590. doi:10.1016/j.respol.2008.06.004 anced scorecard: A foundation for the strategic manage-
ment of information systems. Journal of Decision Support
Marchand, M., & Raymond, L. (2008). Researching
Systems, 25, 71–88. doi:10.1016/S0167-9236(98)00086-4
performance measurement systems: An Information
Systems perspective. International Journal of Op- Martinsons, M., & Martinsons, V. (2002). Rethinking the
erations & Production Management, 28(7), 663–686. value of IT, again. Communications of the ACM, 45(7),
doi:10.1108/01443570810881802 25–28. doi:10.1145/514236.514256
March, J. (1991). Exploration and exploitation in orga- Maryati, M., Ray, J., & Lampros, K. (2006). Towards a
nization learning. Organization Science, 2(1), 71–87. framework for health Information Systems evaluation.
doi:10.1287/orsc.2.1.71 Paper presented at the 39th Annual Hawaii International
Conference on System Sciences (HICSS’06) Track 5.
Markovic, B., & Wood, J. (2004). User satisfaction with
CBA computer LABS. Journal of Computing Sciences Mason, R. (1978). Measuring information output: A com-
in Colleges, 19(4), 232–239. munication systems approach. Information & Manage-
ment, 1(4), 219–234. doi:10.1016/0378-7206(78)90028-9
440
Masrek, M. (2008). The effect of organizational and in- McGorry, S. (2003). Measuring quality in online pro-
dividual characteristics on corporate intranet utilizations. grams. The Internet and Higher Education, 6(2), 159–177.
Information Management & Computer Security, 16(2), doi:10.1016/S1096-7516(03)00022-8
89–112. doi:10.1108/09685220810879591
McHaney, R., Hightower, R., & Pearson, J. (2002). A vali-
Mata, F. J., Fuerst, W. L., & Barney, J. B. (1995). Informa- dation of the end-user computing satisfaction instrument
tion Technology and sustained competitive advantage: A in Taiwan. Information & Management, 39(6), 503–511.
resource based analysis. Management Information Systems doi:10.1016/S0378-7206(01)00119-7
Quarterly, 19(4), 487–505. doi:10.2307/249630
McKinney, V., & Yoon, K. (2002). The measurement of
Mata, F., Fuerst, W., & Barney, J. (1995). Information Web-customer satisfaction: An expectation and disconfir-
Technology and sustained competitive advantage: A mation approach. Information Systems Research, 13(3),
resource-based analysis. Management Information Sys- 296–315. doi:10.1287/isre.13.3.296.76
tems Quarterly, 19(4), 487–505. doi:10.2307/249630
McKinney, V., Yoon, K., & Zahedi, F. (2002). The mea-
Mateo, R., Tanco, M., & Santos, J. (2011). Improving surement of Web-customer satisfaction: An expectation
intranet knowledge transfer through resident engineers. and disconfirmation approach. Information Systems
Journal of Knowledge Management, 15(1), 40–52. Research, 13(3), 296–311. doi:10.1287/isre.13.3.296.76
doi:10.1108/13673271111108684
McKnight, L., Solomon, R., Reagle, J., Carver, D., John-
McCarthy, R., & Aronson, J. (2000). Competing in the son, C., Gerovac, B., & Gingold, D. (1997). Information
virtual world: E-commerce factors that affect the property- security of Internet commerce. In McKnight, L. W., &
casualty insurance industry. Paper presented at the Sixth Bailey, J. P. (Eds.), Internet economics. Cambridge, MA:
Americas Conference on Information Systems. MIT Press.
McFadzean, E., Ezingeard, J., & Birchall, D. (2006). McLoughlin, C., & Luca, J. (2001). Quality in online
Anchoring information security governance research: delivery: What does it mean for assessment in e-learning
Sociological groundings and future directions. Journal environments. Paper presented at the Annual Conference
of Information System Security, 2(3), 3–48. of the Australasian Society for Computer in Learning in
Tertiary Education, Melbourne, Australia
McGaughey, R. E., Snyder, C. A., & Carr, H. H. (1994).
Implementing Information Technology for competi- McManus, J. (2009). A perspective om service delivery.
tive advantage: Risk management issues. Information Management Services, Winter, 8-11.
& Management, 26(5), 273–280. doi:10.1016/0378-
McNamara, N., & Kirakowski, J. (2011). Measuring
7206(94)90066-3
user-satisfaction with electronic consumer products: The
McGee, M. K. (1998, November 23). Strategic applica- consumer products questionnaire. International Journal of
tions: Companies are forging a unique blend of custom Human-Computer Studies, 69(6), 375–386. doi:10.1016/j.
and commercial software to gain a competitive advantage. ijhcs.2011.01.005
Information Week.
McNaughton, B., Ray, P., & Lewis, L. (2010). Designing
McGill, T., & Klobas, J. (2009). A task-technology fit an evaluation framework for IT service management. In-
view of learning management system impact. Com- formation & Management, 47(4), 219–225. doi:10.1016/j.
puters & Education, 52(2), 496–508. doi:10.1016/j. im.2010.02.003
compedu.2008.10.002
Meadows, C. (2001). A cost-based framework for analysis
McGinnes, S. (2011). Barriers to client collaboration in of denial of service in networks. Journal of Computer
agile offshore Information Systems development. Paper Security, 9(1/2), 143–164.
presented at the International Conference on Information
Systems Development (ISD2011), Edinburgh.
441
Mediq, N. C., & Høstgaard, A.M. (2005). Brugerer- Meter, K. v., Cibois, P., & Saint Leger (2004). Correspon-
faringer fra GEPKA projektet paa afdeling M, Aarhus dence & co-word analysis of ten years of BMS articles
sygehus (User experience from the GEPKA project in (1993-2003). Bulletin de Méthodologie Sociologique.
Aarhus hospital). Aalborg University: Virtual Center for
Michalska, J. (2005). The usage of the balanced scorecard
Health Informatics.
for the estimation of the enterprise’s effectiveness. Journal
Medori, D., & Steeple, D. (2000). A framework for auditing of Materials Processing Technology, 162-163, 751–758.
and enhancing performance measurement systems. Inter- doi:10.1016/j.jmatprotec.2005.02.227
national Journal of Operations & Production Manage-
Mich, L., Franch, M., & Gaio, L. (2003). Evaluating
ment, 20(5), 520–533. doi:10.1108/01443570010318896
and designing the quality of websites: The 2QCV3Q
Meijden, van der, M. J., Tange, H, J., Troost, J., & Has- metamodel. IEEE MultiMedia, 10(1), 34–43. doi:10.1109/
man, A. (2003). Determinants of success of inpatient MMUL.2003.1167920
clinical Information Systems: A literature review. Journal
Middleton, C. (1995, August 25 - 27). A tale of two
of the American Medical Informatics Association, 10(3),
systems? Success and failure in a single Information
235-243.
System implementation. Paper presented at the Associa-
Meijden, M. J. d., Tange, H. J., Troost, J., & Hasman, tion for Information Systems 1995 Inaugural Conference,
A. (2003). Determinants of success of inpatient clini- Pittsburgh, PA.
cal information systems: A literature review. Journal of
Millen, J. (1992). A resource allocation model for denial
the American Medical Informatics Association, 10(3),
of service. In Proceedings of the 1992 IEEE Symposium
235–243. doi:10.1197/jamia.M1094
on Security and Privacy. Los Alamitos, CA: IEEE Com-
Mellor, S. J., Clark, A. N., & Futagami, T. (2003). Model- puter Society Press.
driven development-Guest editor’s introduction. Software,
Miller, J., & Doyle, B. (1987). Measuring the effectiveness
IEEE, 20(5), 14–18. doi:10.1109/MS.2003.1231145
of computer-based Information Systems in the financial
Mell, P., & Grance, T. (2009). Effectively and securely services sector. Management Information Systems Quar-
using the cloud computing paradigm. National Institute terly, 11(1), 107–124. doi:10.2307/248832
of Standards and Technology.
Miranda, S. (2004). Beyond BI: Benefiting from corporate
Melone, N. (1990). A theoretical assessment of the user- performance management solutions. Financial Executive,
satisfaction construct in Information Systems research. 20(2), 58–61.
Management Science, 36(1), 76–91. doi:10.1287/
Mirani, R., & Lederer, A. (1998). An instrument for as-
mnsc.36.1.76
sessing the organizational benefits of IS projects. Decision
Melville, N., Kenneth, K., & Vijay, G. (2004). Review: Sciences, 29(4), 803–838. doi:10.1111/j.1540-5915.1998.
Information Technology and organizational performance: tb00878.x
An integrative model of IT business value. Management
Misra, S. C., Kumar, V., & Kumar, U. (2007). A strategic
modeling technique for information security risk assess-
Melville, N., Kraemer, K., & Gurbaxani, V. (2004). In- ment. Information Management & Computer Security,
formation Technology and organizational performance: 15(1), 64–77. doi:10.1108/09685220710738787
An integrative model of IT business value. MIS Quarterly
Misra, S., & Kilic, H. (2007). Measurement theory and
Review, 28(2), 283–322.
validation criteria for software complexity measures.
Merono-Cerdan, A. L. (2008). Groupware uses and in- ACM SIGSOFT Software Engineering Notes, 32(2), 1–3.
fluence on performance in SMEs. Journal of Computer doi:10.1145/1234741.1234760
442
Mitchell, R. K., Agle, B. R., & Wood, D. J. (1997). Toward Morrison, A., Taylor, J., & Douglas, A. (2005). Website
a theory of stakeholder identification and salience: Defin- evaluation in tourism and hospitality. Journal of Travel
ing the principle of who and what really counts. Academy & Tourism Marketing, 17(2), 233–251. doi:10.1300/
of Management Review, 22(4), 853–886. J073v17n02_18
Modol, J. R., & Muniategui, F. S. (2008). Tackling the Motiwalla, L. F., & Thompson, J. (2009). Enterprise sys-
problem of transferability in IS qualitative research. Paris: tems for management. Upper Saddle River, NJ: Pearson
ICIS Proceedings. Prentice Hall.
Moed, H. F., De Bruin, R. E., Nederhof, A. J., & Tijssen, Moulton, R., & Coles, R. S. (2003). Applying informa-
R. J. W. (1991). International scientific cooperation tion security governance. Computers & Security, 22(7),
and awareness within the European community: Prob- 580–584. doi:10.1016/S0167-4048(03)00705-3
lems and perspectives. Scientometrics, 21(3), 291–311.
Moura, A., Sauve, J., Jornada, J., & Radziuk, E. (2006).
doi:10.1007/BF02093972
A quantitative approach to IT investment allocation to
Molla, A., & Licker, P. S. (2001). E-commerce systems improve business results. Proceedings of the Seventh
success: An attempt to extend and respecify the Delone IEEE International Workshop on Policies for Distributed
and McLean model of IS success. Journal of Electronic Systems and Networks (pp. 87-95).
Commerce Research, 2(4), 131–141.
Muhr, T. (2004). ATLAS the knowledge workbench. V5.0
Momoh, A. (2010). Challenges in enterprise resource user’s guide and reference. Berlin, Germany: ATLAS.
planning implementation: State-of-the-art. Busi- ti.Scientific Software Development GmbH.
ness Process Management Journal, 16(4), 537–565.
Mukhopadhyay, T., Kekre, S., & Kalathur, S. (1995).
doi:10.1108/14637151011065919
Business value of Information Technology: A study of
Money, A. G., Barnett, J., Kuljis, J., Craven, M. P., Martin, electronic data interchange. Management Information
J. L., & Young, T. (2011). The role of the user within the Systems Quarterly, 19(2), 137–156. doi:10.2307/249685
medical device design and development process: Medi-
Müller, J. (1990). Teknologivurdering og trovaerdighed
cal device manufacturers’ perspectives. BioMedCentral:
(technology assessment and credibility). In Cronberg T., &
Medical Information and Decision Making, 11, 15.
Friis D. (Eds.), Metoder i teknologivurdering, erfaring og
doi:10.1186/1472-6947-11-15
fornyelse (Methods in technology assessment - knowledge
Montazemi, A., Cameron, D., & Gupta, K. (1996). An and experiences) (pp. 267-279). København, Denmark:
empirical study of factors affecting software package Blytmanns forlag.
selection. Journal of Management Information Systems,
Müller, J., Kjær-Rasmussen, J., & Nøhr, C. (1988). Per-
13(1), 89–105.
spektiver for EDB-teknologi i sygehusvaesenet (Perspec-
Monteiro, E. (2003). Integrating health Information tives for IT-technology in the hospital sector). Aalborg,
Systems: A critical appraisal. Methods of Information in Denmark: Institut for Samfundsudvikling og Planlaegning,
Medicine, 42(4), 428–432. Aalborg Universitetscenter, Aalborg, DK.
Mooraj, S., Oyon, D., & Hostettler, D. (1999). The bal- Müller, J. (1991). Hvad er teknologi? (What is technol-
anced scorecard: A necessary good or an unnecessary ogy?). In Rostgaard, M., Remmen, A., & Christensen,
evil. European Management Journal, 17(5), 481–491. J. (Eds.), Samfundet i teknologien (pp. 27–47). Aalborg,
doi:10.1016/S0263-2373(99)00034-1 Denmark: Aalborg Universitetsforlag.
Morrison, C., & Berndt, E. (1991). Assessing the pro- Müller, J., Kjær-Rasmussen, J., & Nøhr, C. (1989). Proac-
ductivity of information technology equipment in U.S. tive technology assessment as a tool in HIS development.
manufacturing industries. National Bureau of Economic In Barber, B. (Ed.), MEDINFO 1989. Singapore.
Research working paper no. 3582, Washington, D.C.
443
Mumford, E. (1983). Designing human systems for new Muylle, S., Moenaert, R., & Despontin, M. (2004). The
technology - The ETHICS method. Manchester, UK: conceptualization and empirical validation of website user
Manchester Business School. satisfaction. Information & Management, 41(5), 543–560.
doi:10.1016/S0378-7206(03)00089-2
Mumford, E., & Henshall, D. (1978). Participative ap-
proach to computer systems design: A case study of the Myers, B., Kappelman, L., & Prybutok, V. (1998). A
introduction of a new computer system. New York, NY: comprehensive model for assessing the quality and pro-
Halsted Press. ductivity of the Information Systems function: Toward a
theory for Information Systems assessment. In Garritty,
Mumford, E., & Ward, T. B. (1968). Computers: Planning
E., & Sanders, G. (Eds.), Information Systems success
for people. UK: Batsford Limited.
measurement (pp. 94–121). Hershey, PA: Idea Group.
Mumford, E., & Weir, M. (1979). Computer systems in
Myers, M. D. (1994a). A disaster for everyone to see: An
work design: The ETHICS method. London, UK: Associ-
interpretive analysis of a failed IS project. Accounting.
ated Business Press.
Management and Information Technology, 4(4), 185–201.
Münch, E. (2002). Strukturwandel im Krankenhaus-Markt doi:10.1016/0959-8022(94)90022-1
– Investitionsfinanzierung im Umbruch. Handelsblatt
Myers, M. D. (1994b). Dialectical hermeneutics: A
Gesundheitskongress.
theoretical framework for the implementation of infor-
Murakami, E., Saraiva, A., Ribeiro Junior, L., Cugnasca, mation systems. Information Systems Journal, 5, 51–70.
C., Hirakawa, A., & Correa, P. (2007). An infrastructure doi:10.1111/j.1365-2575.1995.tb00089.x
for the development of distributed service-oriented Infor-
Myers, M. D. (1999). Investigating Information Systems
mation Systems for precision agriculture. Computers and
with ethnographic research. Communications of the As-
Electronics in Agriculture, 58(1), 37–48. doi:10.1016/j.
sociation for Information Systems, 2(23), 1–20.
compag.2006.12.010
Nah, F. F.-H., & Lau, J. L.-S. (2001). Critical factors
Murphy, G. B., Trailer, J. W., & Hill, R. C. (1996). Mea-
for successful implementation of enterprise systems.
suring performance in entrepreneurship research. Journal
Business Process Management Journal, 7(3), 285–296.
of Business Research, 36(1), 15–23. doi:10.1016/0148-
doi:10.1108/14637150110392782
2963(95)00159-X
Nah, F. F., Tan, X., & Teh, S. H. (2004). An empirical
Murphy, K. E., & Simon, S. J. (2002). Intangible benefits
investigation on end-users’ acceptance of enterprise
valuation in ERP projects. Information Systems Journal,
systems. Information Resources Management Journal,
12, 301–320. doi:10.1046/j.1365-2575.2002.00131.x
17(3), 32–53. doi:10.4018/irmj.2004070103
Murray, F. (2002). Innovation as co-evolution of scientific
NAO. (2004). Improving IT procurement: The impact
and technological networks: Exploring tissue engineering.
of the Office of Government Commerce’s initiatives
Research Policy, 31, 1389–1403. doi:10.1016/S0048-
on departments and suppliers in the delivery of major
7333(02)00070-7
IT-enabled projects (5 November 2004: National Audit
Musaji, Y. F. (2003). Integrated auditing of ERP systems. Office. Reich, B. H., & Benbasat, I. (2000). Factors that
New York, NY: John Wiley & Sons, inc. influence the social dimension of alignment between
business and Information Technology objectives. Man-
Mutton, P., & Golbeck, J. (2003). Visualization of seman- agement Information Systems Quarterly, 24(1), 81–113.
tic metadata and ontologies. Proceedings of the Seventh
International Conference on Information Visualization Nasco, S. A., Toledo, E. G., & Mykytyn, P. P. (2008).
(IV’03) (pp. 300-305). IEEE. Predicting electronic commerce adoption in Chilean
SMEs. Journal of Business Research, 61(6), 697–705.
doi:10.1016/j.jbusres.2007.06.047
444
Nasir, M. H. N. M., & Sahibuddin, S. (2011). Addressing Neumann, P., & Parker, D. (1989). A summary of computer
a critical success factor for software projects: A multi- misuse techniques. In Proceedings of the 12th National
round Delphi study of TSP. International Journal of the Computer Security Conference.
Physical Sciences, 6(5), 1213–1232.
Ngai, E. W. T., Law, C. C. H., & Wat, F. K. T. (2008).
Natovich, J. (2003). Vendor related risks in IT develop- Examining the critical success factors in the adoption
ment: A chronology of an outsourced project failure. of enterprise resource planning. Computers in Industry,
Technology Analysis and Strategic Management, 15(4), 59(6), 548–564. doi:10.1016/j.compind.2007.12.001
409–419. doi:10.1080/095373203000136015
Ngai, E., Poon, J., & Chan, Y. (2007). Empirical ex-
Naveh, G., Tubin, D., & Pliskin, N. (2010). Student LMS amination of the adoption of WebCT using TAM. Com-
use and satisfaction in academic institutions: The organi- puters & Education, 48(2), 250–267. doi:10.1016/j.
zational perspective. The Internet and Higher Education, compedu.2004.11.007
13(3), 127–133. doi:10.1016/j.iheduc.2010.02.004
Ngai, E., & Wat, F. (2002). A literature review and clas-
Neely, A., Adams, C., & Crowe, P. (2001). The perfor- sification of electronic commerce research. Information
mance prism in practice. Measuring Business Excellence, & Management, 39(5), 415–429. doi:10.1016/S0378-
5(2), 6-13. 7206(01)00107-0
Neely, A., Adams, C., & Kennerley, M. (2002). The Ng, H., & Wilson, T. (1998). Business use of the World
performance prism: The scorecard for measuring and Wide Web: A report on further investigations. International
managing business success. London, UK: Prentice Hall Journal of Information Management, 18(5), 291–314.
Financial Times. doi:10.1016/S0268-4012(98)00021-8
Neely, A. (1999). The performance measurement revolu- Nicolaou, A., & McKnight, D. (2006). Perceived informa-
tion: Why now and what next? International Journal of tion quality in data exchanges: Effects on risk, trust, and
Operations & Production Management, 19(2), 205–228. intention to use. Information Systems Research, 17(4),
doi:10.1108/01443579910247437 332. doi:10.1287/isre.1060.0103
Neely, A., Gregory, M., & Platts, K. (2005). Perfor- Nielsen, J. (1999). Differences between print design and
mance measurement system design: A literature review Web design. Alertbox. Retrieved from http://useit.com/
and research agenda. International Journal of Opera- alertbox/990124.html (accecced on January19, 2009).
tions & Production Management, 25(12), 1228–1263.
Nielsen, J. (1995). A home-page overhaul using other web
doi:10.1108/01443570510633639
sites. IEEE Software, 12(3), 75–78. doi:10.1109/52.382190
Neff, M. W., & Corleyb, E. A. (2009). 35 years and 160,000
Nies, J., & Pelayo, S. (2010). From users involvement
articles: A bibliometric exploration of the evolution of
to users’ needs understanding: A case study. Interna-
ecology. Scientometrics, 80(3), 657–682. doi:10.1007/
tional Journal of Medical Informatics, 79(4), 76–82.
s11192-008-2099-3
doi:10.1016/j.ijmedinf.2009.06.007
Nelson, R., & Winter, S. (1982). An evolutionary theory
Nøhr, C., & Bernstein, K. (1992). Implementation and
of economic change. Cambridge, MA: Harvard Univer-
diffusion of medical informatics: The case of electronic
sity Press.
prescription communication. between practitioners and
Nerb, J. (2007). Exploring the dynamics of the appraisal- pharmacies. Proceedings of Primary Health Care Special-
emotion relationship: A constraint satisfaction model of ist Group of the British Computer Society, (pp. 14-18).
the appraisal process. Cognition and Emotion, 21(7),
Nøhr, C., Høstgaard, A. M., Botin, L., & Kjær-Andersen,
1382–1413. doi:10.1080/02699930701193419
S. (2004). Evaluering af GEPKA-projektet, delrapport
Neufeld, D., Fang, Y., & Huff, S. (2007). The IS identity 2, klinisk afprovning (evaluation of the GEPKA-project,
crisis. Communications of the Association for Information report 2, clinical testing). Aalborg University: Virtual
Systems, 19, 447–464. Center of Health Informatics.
445
Nooteboom, B. (1993). Firm size effects on transac- Okamura, H., Miyahara, S., & Dohi, T. (2005, 4-8 April).
tion costs. Small Business Economics, 5(4), 283–295. Effect of preventive rejuvenation in communication net-
doi:10.1007/BF01516249 work system with burst arrival. Paper presented at the
Autonomous Decentralized Systems, ISADS.
Nordjyllands Amt (The County of North Jutland). (2000).
IT-handlingsplan for sundhedsomraadet i Nordjyllands Oliver, R. (2005). Quality assurance and e-learning: Blue
amt 2002-2004 (IT-plan for the healthcare area in the skies and pragmatism. Research in Learning Technology,
county of north Jutland 2002-2004). 13(3), 173–187. doi:10.1080/09687760500376389
Norzaidi, M. D., Chong, S. C., Murali, R., & Salwani, M. Olsen, K. A., & Saetre, P. (2007). IT for niche companies:
I. (2009). Towards a holistic model in investigating the ef- Is an ERP system the solution? Information Systems Jour-
fects of intranet usage on managerial performance: A study nal, 17(1), 37–58. doi:10.1111/j.1365-2575.2006.00229.x
on Malaysian port industry. Maritime Policy & Manage-
Olsina, L., Lafuente, G., & Rossi, G. (2000). E-commerce
ment, 36(3), 269–289. doi:10.1080/03088830902861235
site evaluation: A case study. Paper presented at the 1st
Norzaidi, M. D., Chong, S. C., & Salwani, M. I. (2009). International Conference on Electronic Commerce and
Intranet usage, managerial satisfaction and performance Web Technologies (EC-Web 2000).
impact: an empirical analysis. International Journal
Ong, C., & Lai, J. (2007). Measuring user satisfaction
of Business and Systems Research, 3(4), 481–496.
with knowledge management systems: Scale development,
doi:10.1504/IJBSR.2009.027201
purification, and initial test. Computers in Human Behav-
Novak, J. D. (1998). Learning, creating, and using ior, 23(3), 1329–1346. doi:10.1016/j.chb.2004.12.012
knowledge: Concept maps as facilitative tools in schools
Orlikowski, W. (1992). The duality of technology: Re-
and cooperations. Mahwah, NJ: Lawrence Erlbaum
thinking the concept of technology in organizations. Orga-
Associates.
nization Science, 3(3), 398–427. doi:10.1287/orsc.3.3.398
O’Boyle, P., Acton, T., Campion, M., Conboy, K., &
Orlikowski, W. J., & Baroudi, J. J. (1991). Studying
Scott, M. (2009). Towards a toolset for intranet evalua-
Information Technology in organizations: Research ap-
tion. 22nd Bled eConference eEnablement, June 14 - 17.
proaches and assumptions. Information Systems Research,
Slovenia: Bled.
2(1), 1–28. doi:10.1287/isre.2.1.1
O’Donovan, B., & Roode, D. (2002). A framework for
Otley, D. (1999). Performance management: A framework
understanding the emerging discipline of information
for management control systems research. Manage-
systems. Information Technology & People, 15(1), 26–41.
ment Accounting Research, 10, 363–382. doi:10.1006/
doi:10.1108/09593840210423217
mare.1999.0115
OASIG. (1996). The performance of information tech-
OUP. (2000). The Oxford English dictionary online.
nology and the role of human and organizational factors
Retrieved from http://gateway.library.qut.edu.au/
(Report). Sheffield, UK: University of Sheffieldo. Docu-
login?url=http://dictionary.oed.com/entrance.dtl
ment Number.
Özesmi, U., & Özesmi, S. L. (2004). Ecological models
OECD. (2005). OICT, E-business and SMEs. OECD Digi-
based on people’s knowledge: A multi-step fuzzy cogni-
tal Economy Papers, 88. doi:doi:10.1787/232475651473
tive mapping approach. Ecological Modelling, 176(1-2),
OGC. (2007). Service strategy. London, UK: TSO for 43–64. doi:10.1016/j.ecolmodel.2003.10.027
the Office of Government Commerce.
Ozkan, S., & Koseler, R. (2009). Multi-dimensional
Oh, K. Y., Cruickshank, D., & Anderson, A. R. (2009). students’ evaluation of e-learning systems in the higher
The adoption of e-trade innovations by Korean small education context: An empirical investigation. Com-
and medium sized firms. Technovation, 29(2), 110–121. puters & Education, 53(4), 1285–1296. doi:10.1016/j.
doi:10.1016/j.technovation.2008.08.001 compedu.2009.06.011
446
Page, C. K. (2000). Critical success factors for implement- Parasuraman, A., Zeithaml, V. A., & Malhotra, A. (2005).
ing a clinical information system. Nursing Economics, E-S-QUAL: A multiple-item scale for assessing elec-
18(5). Jannetti Publications. tronic service quality. Journal of Service Research, 7(3),
213–233. doi:10.1177/1094670504271156
Palmer, W. (2002). Website usability, design, and per-
formance metrics. Information Systems Research, 13(2), Paré, G., Sicotte, C., Jaana, M., & Girouard, D. (2008).
151–167. doi:10.1287/isre.13.2.151.88 Prioritizing clinical Information System project risk
factors: A Delphi study. Proceedings of the 41st Hawaii
Palvia, P. C., Perkins, J. A., & Zeltmann, S. M. (1992).
International Conference on System Sciences.
The PRISM system: A key to organisational effectiveness
at Federal Express Corporation. Management Information Pare, G., Sicotte, C., & Jacques, H. (2006). The effects
Systems Quarterly, 16(3), 277–292. doi:10.2307/249529 of creating psychological ownership on physicians’ ac-
ceptance of clinical Information Systems. Journal of
Palvia, P., Leary, D., Mao, E., Midha, V., Pinjani, P., &
the American Medical Informatics Association, 13(2),
Salam, A. F. (2004). Research methodologies in MIS: An
197–205. doi:10.1197/jamia.M1930
update. Communications of the Association for Informa-
tion Systems, 14, 526–542. Park, Y., & Gretzel, U. (2007). Success factors for desti-
nation marketing websites: A qualitative meta-analysis.
Palvia, P., Mao, E., Salam, A. F., & Soliman, K. (2003).
Journal of Travel Research, 46-63(1), 46.
Management information systems research: What’s there
in a methodology? Communications of the Association Parker, D. B. (1998). Fighting computer crime - A new
for Information Systems, 11, 289–309. framework for protecting information. New York, NY:
John Wiley & Sons.
Palvia, S. C., & Chervany, N. L. (1995). An experimental
investigation of factors influencing predicted success in Park, J. H., Lee, J. K., & Yoo, J. S. (2005). A framework
DSS implementation. Information & Management, 29(1), for designing the balanced supply chain scorecard. Eu-
43–53. doi:10.1016/0378-7206(95)00006-I ropean Journal of Information Systems, 14(4), 335–346.
Pang, A. (2009). WW ERP applications 2009-2013 fore-
cast and 2008 vendor shares (pp. 1–16). IDC Insights. Park, Y. A., & Gretzel, U. (2007). Success factor
for destination marketing websites: A qualitative
Papageorgiou, E. I. (2011). A new methodology for
meta-analysis. Journal of Travel Research, 46, 46–63.
decisions in medical informatics using fuzzy cognitive
doi:10.1177/0047287507302381
maps based on fuzzy rule-extraction techniques. Ap-
plied Soft Computing, 11(1), 500–513. doi:10.1016/j. Parnas, D. L. (1994, 16-21 May). Software aging. Paper
asoc.2009.12.010 presented at the 16th International Conference on Software
Engineering (ICSE-16), Sorrento, Italy.
Parasuraman, A. (2004). Assessing and improving
service performance for maximum impact: Insights Pather, S. (2006). E-commerce Information Systems
from a two-decade-long research journey. Perfor- (ECIS) success: A South African study. Doctral Disserta-
mance Measurement and Metrics, 5(2), 45–52. tion, Cape Peninsula University of Technology
doi:10.1108/14678040410546064
Peabody, P. (2002). Setting a new standard. Health Man-
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985). agement Technology, November.
A conceptual model of service quality and its implications
Pease, W., & Rowe, M. (2005). Diffusion of innovation -
for future research. Journal of Marketing, 49(4), 41–50.
The adoption of electronic commerce by small and medium
doi:10.2307/1251430
enterprises (SMEs) - A comparative analysis. Australasian
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). Journal of Information Systems, 13(1), 287–294.
SERVQUAL: A. Journal of Retailing, 64, 1.
447
Peffers, K., Gengler, C. E., & Tuunanen, T. (2003). Ex- Pfleeger, C. P., & Pfleeger, S. L. (2003). Security in
tending critical success factors methodology to facilitate computing (3rd ed.). Prentice Hall.
broadly participative Information Systems planning. Jour-
Phillips, N., & Brown, J. L. (1993). Analyzing communi-
nal of Management Information Systems, 20(1), 51–85.
cation in and around organizations: A critical hermeneutic
Peng, Y., Li, F., & Mili, A. (2007). Modeling the evolu- approach. Academy of Management Journal, 36(6), 1547.
tion of operating systems: An emprical study. Journal doi:10.2307/256821
of Systems and Software, 80, 1–15. doi:10.1016/j.
Phippen, A., Sheppard, L., & Furnell, S. (2004). A practical
jss.2006.03.049
evaluation of Web analytics. Internet Research: Electronic
Pérez-Mira, B. (2010). Validity of DeLone and McLean’s Networking Applications and Policy, 14(4), 284–293.
Model of Information Systems success at the website doi:10.1108/10662240410555306
level of analysis. Doctral Dissertation, Lousiana State
Pickard, R. (1989). Computer crime. Information Center,
University.
5(9), 18–27.
Perry, T., & Wallich, P. (1984). Can computer crime be
Pijpers, G. G. M., & Montfort, K. (2006). An investiga-
stopped? IEEE Spectrum, 21(5).
tion of factors that influence senior executives to accept
Perry, W. E. (1985). Management strategies for computer innovations in Information Technology. International
security. Boston, MA: Butterworth-Heinemann. Journal of Management, 23(1), 11–23.
Perschke, G. A., Karabin, S. J., & Brock, T. L. (1986). Piris, L., Fitzgerald, G., & Serrano, A. (2004). Strategic
Four steps to information security. Journal of Accountancy, motivators and expected benefits from e-commerce in
161(4), 104–113. traditional organisations. International Journal of In-
formation Management, 24(6), 489–506. doi:10.1016/j.
Peterson, R. (2004). Crafting Information Technology
ijinfomgt.2004.08.008
governance. Information Systems Management, 21(4),
7–22. doi:10.1201/1078/44705.21.4.20040901/84183.2 Pisello, T. (2004, October). Is there a business case for IT
security? Security Management. Retrieved October 18,
Petre, M., Minocha, S., & Roberts, D. (2006). Usability
2010 from http://www.securitymanagement.com/article/
beyond the website: An empirically-grounded e-commerce
there-business-case-it-security
evaluation instrument for the total customer experience.
Behaviour & Information Technology, 25(2), 189–203. Pitt, L. F., Watson, R. T., & Kavan, C. B. (1995). Service
doi:10.1080/01449290500331198 quality: A measure of Information Systems effectiveness.
Management Information Systems Quarterly, 19(2),
Petter, S., DeLone, W. H., & McLean, E. R. (2008).
173–188. doi:10.2307/249687
Measuring Information Systems success: Models, dimen-
sions, measures, and interrelationships. European Journal Pitt, L., Watson, R., & Kavan, C. (1995). Service quality:
of Information Systems, 17(3), 236–263. doi:10.1057/ A measure of Information Systems effectiveness. Man-
ejis.2008.15 agement Information Systems Quarterly, 19(2), 173–187.
doi:10.2307/249687
Petter, S., DeLone, W., & McLean, E. (2008). Measuring
information systems success: Models, dimensions, mea- Pituch, K., & Lee, Y. (2006). The influence of system
sures, and interrelationships. European Journal of Infor- characteristics on e-learning use. Computers & Educa-
mation Systems, 17, 236–263. doi:10.1057/ejis.2008.15 tion, 47(2), 222–244. doi:10.1016/j.compedu.2004.10.007
Petter, S., & McLean, E. R. (2009). A meta-analytic as- Pollard, C., & Cater-Steel, A. (2009). Justifications,
sessment of the DeLone and McLean IS success model: strategies, and critical success factors in successful ITIL
An examination of IS success at the individual level. In- implementations in U.S. and Australian companies: An
formation & Management, 46(3), 159–166. doi:10.1016/j. exploratory study. Information Systems Management, 26,
im.2008.12.006 164–175. doi:10.1080/10580530902797540
448
Poon, P., & Wagner, C. (2001). Critical success factors Pujani, V., & Xu, J. (2005). E-commerce in Indonesian
revisited: Success and failure cases of information systems SMEs: Towards a research model of website success.
for senior executives. Decision Support Systems, 30(3), Paper presented at the International Conference on Ser-
393–418. doi:10.1016/S0167-9236(00)00069-5 vices Systems and Services Management (ICSSSM05).
Poore, R. (2006). Information security governance. In Pumphrey, L. D., Trimmer, K., & Beachboard, J. (2007).
Handbook of information security management. Boca Enterprise resource planning systems and HIPAA.
Raton, FL: CRC Press. Research in Healthcare Financial Management, 11(1),
57–75.
Porter, M. (2001). Strategy and the Internet. Harvard
Business Review, 79(3), 62–78. Purao, S., & Vaishnavi, V. (2003). Product metrics for
object-oriented systems. ACM Computing Surveys, 35(2),
Post, G. V., & Diltz, J. D. (1986). A stochastic dominance
191–221. doi:10.1145/857076.857090
approach to risk analysis of computer systems. Manage-
ment Information Systems Quarterly, 10(4), 363–375. Qin, F., Tucek, J., Zhou, Y., & Sundaresan, J. (2007).
doi:10.2307/249191 Rx: Treating bugs as allergies—A safe method to sur-
vive software failures. ACM Transactions on Computer
Posthumus, S., & von Solms, R. (2004). A framework
Systems, 25(3).
for the governance of information security. Computers
& Security, 23, 638–646. doi:10.1016/j.cose.2004.10.006 Quaddus, M., & Achjari, D. (2005). A model for electronic
commerce success. Telecommunications Policy, 29(2-3),
Powers, R. F., & Dickson, G. W. (1973). MIS project
127–152. doi:10.1016/j.telpol.2004.11.009
management: Myths, opinion and reality. California
Management Review, 15(3), 147–156. Quelch, J., & Klein, L. (1996). The Internet and inter-
national marketing. Sloan Management Review, 37(3),
Praeg, C., & Schnabel, U. (2006). IT-service cachet -
60–75.
Managing IT-service performance and IT-service quality.
Proceedings of the 39th Annual Hawaii International Rababah, O., Al-Shboul, M., & Al-Sayyed, R. (2011).
Conference on System Sciences, Vol. 2. IEEE Computer A new vision for evaluating the quality of e-commerce
Society. websites. International Journal of Advanced Corporate
Learning, 4(1), 32–41.
Premkumar, G. (2003). A meta-analysis of research
on information technology implementation in small Rada, R., & Finley, S. (2004). The aging of a clinical
business. Journal of Organizational Computing and İnformation System. Journal of Biomedical Informatics,
Electronic Commerce, 13(2), 91–121. doi:10.1207/ 37(5), 319–324. doi:10.1016/j.jbi.2004.06.003
S15327744JOCE1302_2
Ragowsky, A., Stern, M., & Adams, D. A. (2000). Relat-
Price, D. J. (1965). Networks of scientific papers. Sci- ing benefits from using IS to an organisation’s operating
ence, 149, 510–515. doi:10.1126/science.149.3683.510 characteristics: Interpreting results from two countries.
Journal of Management Information Systems, 16(4),
Price, R., Neiger, D., & Shanks, G. (2008). Develop-
175–194.
ing a measurement instrument for subjective aspects of
information quality. Communications of the Association Rahman, M., Haque, T. L., & Fukui, T. (2005). Research
for Information Systems, 22(1), 49–74. articles published in clinical radiology journals: Trend of
contribution from different countries. Academic Radiol-
Pritchard, A. (1969). Statistical bibliography or biblio-
ogy, 12(7), 825–829. doi:10.1016/j.acra.2005.03.061
metrics. The Journal of Documentation, 25(4), 348–349.
Rai, A., Lang, S. S., & Welker, R. B. (2002). Assessing
Protti, D. (2002). What can the American electronic health
the validity of IS success models: An empirical test and
record (EHR) pioneers tell us about what it takes to be
theoretical analysis. Information Systems Research, 13(1),
successful? Health Management Forum, 15(2), 33–35.
50–69. doi:10.1287/isre.13.1.50.96
doi:10.1016/S0840-4704(10)60578-9
449
Rai, A., Lang, S., & Welker, R. (2002). Assessing the Raymond, L., & Brisoux, J., J., & Azami, A. (2001).
validity of IS success models: An empirical test and Marketing Information Systems practices in small manu-
theoretical analysis. Information Systems Research, 13(1), facturing firms: antecedents and consequences. Journal
50–69. doi:10.1287/isre.13.1.50.96 of Computer Information Systems, 41(3), 32–41.
Rainer, R. K., Snyder, C. A., & Carr, H. H. (1991). Risk Razzouk, N., Seitz, V., Lamude, K., & Kepekci, A. C.
analysis for Information Technology. Journal of Manage- (2005). A content analysis of value-based advertising on
ment Information Systems, 8(1), 129–147. the Internet: An empirical investigation. Journal of Website
Promotion, 1(3), 61–73. doi:10.1300/J238v01n03_05
Rajagopal, P. (2002). An innovation–diffusion view of
implementation of enterprise resource planning (ERP) Reich, B. H., & Benbasat, I. (2000). Factors that influ-
systems and development of research model. Information ence the social dimension of alignment between busi-
& Management, 40(2), 87–114. doi:10.1016/S0378- ness and information technology objectives. Manage-
7206(01)00135-5 ment Information Systems Quarterly, 24(1), 81–113.
doi:10.2307/3250980
Rajendram, R., Lewison, G., & Preedy, V. R. (2006).
Worldwide alcohol-related research and the disease Reid, I. (2005). Quality assurance, open and distance
burden. Alcohol and Alcoholism (Oxford, Oxfordshire), learning, and Australian universities. International Review
41(1), 99–106. doi:10.1093/alcalc/agh238 of Research in Open and Distance Learning, 6(1), 1–11.
Ramanathan, R. (2010). E-commerce success criteria: Richard, P. J., Devinney, T. M., Yip, G. S., & Johnson, G.
Determining which criteria count most. Electronic Com- (2009). Measuring organizational performance: Towards
merce Research, 10(2), 191–208. doi:10.1007/s10660- methodological best practice. Journal of Management,
010-9051-3 35(3), 718–804. doi:10.1177/0149206308330560
Ramayah, T., Ahmad, N., & Lo, M. (2010). The role Richins, M. L. (1997). Measuring emotions in the con-
of quality factors in intention to continue using an sumption experience. The Journal of Consumer Research,
e-learning system in Malaysia. Procedia-Social and 24(2), 127–146. doi:10.1086/209499
Behavioral Sciences, 2(2), 5422–5426. doi:10.1016/j.
Riemenschneider, C. K., Harrison, D. A., & Mykytyn, P.
sbspro.2010.03.885
P. (2003). Understanding it adoption decisions in small
Ramirez, R., Melville, N., & Lawler, E. (2010). Informa- business: Integrating current theories. Information &
tion Technology infrastructure, organizational process Management, 40(4), 269–285. doi:10.1016/S0378-
redesign, and business value: An empirical analysis. 7206(02)00010-1
Decision Support Systems, 49(4), 417–429. doi:10.1016/j.
Riemenschneider, C. K., & McKinney, V. R. (2001).
dss.2010.05.003
Assessing belief differences in small business adopters
Ranganathan, C., & Ganapathy, S. (2002). Key dimen- and non-adopters of Web-based e-commerce. Journal of
sions of business-to-consumer websites. Information Computer Information Systems, 42(2), 101–107.
& Management, 39(6), 457–465. doi:10.1016/S0378-
Ringle, C. M., & Will, A. (2005). SmartPLS 2.0 (M3).
7206(01)00112-4
Retrieved June 1, 2009, from: http://www.smartpls.de
Ratnasingam, P. (2009). Service quality management
Rivest, R. L. (1978). A method for obtaining digital sig-
applying the balanced scorecard: An exploratory study.
natures and public key cryptosystems. Communications
International Journal of Commerce and Management,
of the ACM, 21(2), 120–126. doi:10.1145/359340.359342
19(2), 127–136. doi:10.1108/10569210910967888
Roach, S. (1987). America’s technology dilemma: A profile
Rauyruen, P., & Miller, K. (2007). Relationship quality as a
of the information economy. Special economic study. New
predictor of B2B customer loyalty. Journal of Business Re-
York, NY: Morgan Stanley.
search, 60(1), 21–31. doi:10.1016/j.jbusres.2005.11.006
450
Robertson, A., Cresswell, K., Takian., A., et al. (2010). Rolstadås, A. (1998). Enterprise performance mea-
Implementation and adaption of nationwide electronic surement. International Journal of Operations
health records in secondary care in England: Qualitative & Production Management, 18(9/10), 989–999.
analysis of interim results from a prospective national doi:10.1108/01443579810225577
evaluation. BMJ Online Journal, 314.
Rosemann, M., & Wiese, J. (1999). Measuring the
Roca, J., Chiu, C., & Martínez, F. (2006). Understanding performance of ERP software – A Balanced Scorecard
e-learning continuance intention: An extension of the approach. Conference Proceeding, The 10th Australasian
technology acceptance model. International Journal of Conference on Information Systems, 8(4).
Human-Computer Studies, 64(8), 683–696. doi:10.1016/j.
Rosenthal, D. (2002). Intrusion detection technology:
ijhcs.2006.01.003
Leveraging the organization’s security posture. Informa-
Rockart, J. (1979). Chief executives define their own data tion Systems Management, 19(1), 35–44. doi:10.1201/1
needs. Harvard Business Review, 57(2), 81–93. 078/43199.19.1.20020101/31474.5
Rockart, J. F. (1979). Chief executives define their own Roses, L. K., Hoppen, N., & Henrique, J. L. (2009).
data needs. Harvard Business Review, 52(2), 81–93. Management of perceptions of İnformation Technology
service quality. Journal of Business Research, 62(9),
Rockart, J. F., & DeLong, D. W. (1998). Executive support
876–882. doi:10.1016/j.jbusres.2008.10.005
systems: The emergence of top-management computer
use. Homewood, IL: Dow Jones-Irwin. Roses, L., Hoppen, N., & Henrique, J. (2009). Manage-
ment of perceptions of Information Technology service
Rodriguez-Repiso, L., Setchi, R., & Salmeron, J. L.
quality. Journal of Business Research, 62(9), 876–882.
(2007). Modelling IT projects success with fuzzy cognitive
doi:10.1016/j.jbusres.2008.10.005
maps. Expert Systems with Applications, 32(2), 543–559.
doi:10.1016/j.eswa.2006.01.032 Rovai, A., & Downey, J. (2009). Why some distance
education programs fail while others succeed in a global
Roffe, I. (2002). E-learning: engagement, enhancement
environment. The Internet and Higher Education, 13(3),
and execution. Quality Assurance in Education, 10(1),
141–147. doi:10.1016/j.iheduc.2009.07.001
40–50. doi:10.1108/09684880210416102
Rowe, F. (2003). Faire de la Recherche en Systèmes
Rogers, E. (2003). Diffusion of innovations. Free Press;
d’Information. Vuibert, Fnege.
Original edition (August 16, 2003).
Russell, D., & Gangemi, G. T. (1991). Computer security
Rogers, E. M. (1983, 1995). Diffusion of innovation, (1st
basics. Sebastopol, CA: O’Reilly.
and 4th edition). New York, NY: The Free Press.
Ryan, S., Harrison, D., & Schkade, L. (2002). Informa-
Rogers, D. (2000). A paradigm shift: Technology in-
tion-Technology investment decisions: When do costs
tegration for higher education in the new millennium.
and benefits in the social subsystem matter? Journal of
Educational Technology Review, 19–27.
Management Information Systems, 19(2), 85–127.
Rogers, E. (1983). Diffusion of innovation (3rd ed.). New
Rycroft, S., & Tully, M. (2007). Building an information
security meta standard. BT Technology Journal, 25(1),
Rogers, E. (2003). Diffusion of innovations (5th ed.). New 37–40. doi:10.1007/s10550-007-0006-8
Saarelainen, M.-M., Ahonen, J. J., Lintinen, H., Koskinen,
Rogers, E. M. (1962). Diffusion of ınnovations. Free Press. J., Kankaanpää, I., Sivula, H., et al. (2006). Software
modernization and replacement decision making in
Rogers, E. V. (2003). Diffusion of innovation (4th ed.). ındustry: A qualitative study. Paper presented at the 10th
New York, NY: The Free Press. International Conference on Evaluation and Assessment
in Software Engineering (EASE), UK.
451
Saarinen, T. (1996). An expanded instrument for evaluating Salmeron, J. L., & Herrero, I. (2005). An AHP-based
information system success. Information & Management, methodology to rank critical success factors of executive
31(2), 103–118. doi:10.1016/S0378-7206(96)01075-0 information systems. Computer Standards & Interfaces,
28(1), 1–12. doi:10.1016/j.csi.2004.09.002
Sabherwal, R., Jeyaraj, A., & Chowa, C. (2006). Infor-
mation System success: Individual and organizational Salmeron, J. L., & Lopez, C. (2010). A multicriteria ap-
determinants. Management Science, 52(12), 1849–1864. proach for risks assessment in ERP maintenance. Journal
doi:10.1287/mnsc.1060.0583 of Systems and Software, 83, 1941–1953. doi:10.1016/j.
jss.2010.05.073
Saeed, K. A., & Abdinnour-Helm, S. (2008). Examining
the effects of Information System characteristics and per- Samiaji, S., & Zowghi, D. (2005). Information Technology
ceived usefulness on post adoption usage of Information adoption process within Indonesian SMEs: An emperical
Systems. Information & Management, 45(6), 376–386. study. Paper presented at the 16th Australian Conference
doi:10.1016/j.im.2008.06.002 on Information Systems.
Sagar, V. (2006). A digital library success model for Sampler, J. (1996). Exploring the relationship between
computer science student use of a meta-search system. Information Technology and organizational structure. In
Masters Thesis, Faculty of Virginia Polytechnic Institue Earl, M. (Ed.), Information management: The organiza-
and State. tional dimension. Oxford, UK: Oxford University Press.
Sakthivel, S. (2007). Managing risk in offshore systems Sanders, G. L., & Courtney, J. F. (1985). A field study
development. Communications of the ACM, 50(4), 69–75. of organizational factors influencing DSS success. Man-
doi:10.1145/1232743.1232750 agement Information Systems Quarterly, 9(1), 77–93.
doi:10.2307/249275
Saleh, Y. M. (2002). Information System/Information
Technology success and evaluation: An evaluation frame- Sauders, M., Lewis, P., & Thornhill, A. (2003). Research
work and general pracitioner model. PhD, University of methods for business students (3rd ed.). Harlow, UK:
Salford, Uk. Prentice Hall.
Salle, M. (2004). IT service management and IT gover- Sauer, J. (2006). Agile practices in offshore outsourcing–
nance: Review, comparative analysis and their impact An analysis of published experiences. Paper presented
on utility computing. HP Research Labs. at the 29th Information Systems Research Seminar in
Scandinavia, IRIS.
Salmerona, J. L., & Smarandacheb, F. (2006). Processing
uncertainty and indeterminacy in Information Systems Saull, R. (2000). The IT balanced scorecard – A roadmap
projects success mapping. Computational Modeling in to effective governance of a shared services IT organi-
Applied Problems: Collected Papers on Econometrics (p. zation. Information Systems Control Journal, 2, 31–38.
94). Operations Research, Game Theory and Simulation. doi:10.1023/A:1010041819361
Salmeron, J. L. (2009). Augmented fuzzy cognitive maps Saunders, C. S., & Jones, J. W. (1992). Measuring per-
for modelling LMS critical success factors. Knowledge- formance of the Information Systems function. Journal
Based Systems, 22(4), 275–278. doi:10.1016/j.kno- of Management Information Systems, 8(4), 63–82.
sys.2009.01.002
Scharl, A., Wober, K., & Bauer, C. (2004). An integrated
Salmeron, J. L. (2010). Modelling grey uncertainty approach to measure website effectiveness in the European
with fuzzy grey cognitive maps. Expert Systems with hotel industry. Information Technology and Tourism, 6(4),
Applications, 37(12), 7581–7588. doi:10.1016/j. 257–271. doi:10.3727/1098305032781157
eswa.2010.04.085
Schaupp, L., Fan, W., & Belanger, F. (2006). Determin-
ing success for different website goals. Paper presented
at the The 39th Annual Hawaii International Conference
on System Sciences (HICSS’06) Track 6.
452
Schaupp, L. C., Bélanger, F., & Weiguo, F. (2009). Ex- Scott, J. E. (1995). The measurement of informa-
amining the success of websites beyond e-commerce: An tion systems effectiveness: evaluating a measur-
extension of the IS success model. Journal of Computer ing instrument. SIGMIS Database, 26(1), 43–61.
Information Systems, 49(4), 42–52. doi:10.1145/206476.206484
Scheepers, R. (2003). Key roles in intranet imple- Scott, J. E., & Walczak, S. (2009). Cognitive engage-
mentation: The conquest and the aftermath. Jour- ment with a multimedia ERP training tool: Assessing
nal of Information Technology, 18(2), 103–119. computer self-efficacy and technology acceptance. Infor-
doi:10.1080/0268396032000101162 mation & Management, 46(4), 221–232. doi:10.1016/j.
im.2008.10.003
Scheer, A.-W., & Habermann, F. (2000). Making ERP
a success. Communications of the ACM, 43(4), 57–61. Scott, J. T., Rundall, T. G., Vogt, T. M., & Hsu, J. (2005).
doi:10.1145/332051.332073 Kaiser Permanente’s experience of implementing an elec-
tronic medical record: A qualitative study. BMJ Online
Schneider, M., Shnaider, E., Kandel, A., & Chew, G.
Journal, 331(7528), 1–5.
(1998). Automatic construction of FCMs. Fuzzy Sets
and Systems, 93(2), 161–172. doi:10.1016/S0165- Scott, J., E., & Vessey, I. (2002). Managing risks in
0114(96)00218-7 enterprise systems implementations. Communications
of the ACM, 45(4), 74–81. doi:10.1145/505248.505249
Schniederjans, M. J., & Hamaker, J. L. (2003).
A new strategic Information Technology invest- Seddon, P. B., & Kiew, M.-Y. (1994). A partial test and
ment model. Management Decision, 41(1/2), 8–17. development of the DeLone and McLean model of IS
doi:10.1108/00251740310445527 success. Proceedings of the International Conference on
Information Systems, Atlanta, GA, (pp. 99–110).
Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M., &
Spraragen, S. (2000). Measuring success. Communica- Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell,
tions of AIS, 43(8), 53–57. M. (1999). Dimensions of Information Systems success.
Communications of the AIS, 2(20).
Schubert, P., & Selz, D. (2001). Measuring the effective-
ness of e-commerce websites. In Barnes, S., & Hunt, B. Seddon, P., & Kiew, M. Y. (2007). A partial test and
(Eds.), E-commerce & v-business (pp. 83–102). Oxford, development of DeLone and McLean’s model of IS suc-
UK: Butterworth-Heinemann. cess. Australasian Journal of Information Systems, 4(1).
Schultz, E. E. (2002). A framework for understanding Seddon, P. B. (1997). A respecification and extension of
and predicting insider attacks. Computers & Security, the DeLone and McLean model of IS Success. Information
21(6), 526–531. doi:10.1016/S0167-4048(02)01009-X Systems Research, 8, 240–253. doi:10.1287/isre.8.3.240
Schwarz, A. (2003). Defining Information Technology Seddon, P., Graeser, V., & Willcocks, L. (2002). Measuring
acceptance: A human-centred, management-oriented organizational IS effectiveness: An overview and update
perspective. Unpublished doctoral dissertation, Univer- of senior management perspectives. SIGMIS Database,
sity of Huston. 33(2), 11–28. doi:10.1145/513264.513270
Schwarz, A., & Hirschheim, R. (2003). An extended Seddon, P., Staples, S., Patnayakuni, R., & Bowtell, M.
platform logic perspective of IT governance: managing (1999). Dimensions of Information Systems success.
perceptions and activities of IT. The Journal of Strategic Communications of AIS, 2(20), 1–61.
Information Systems, 12(2), 129–166. doi:10.1016/S0963-
Sedera, D., & Tan, F. (2010). User satisfaction: An
8687(03)00021-0
overarching measure of enterprise system success. Pa-
Schweitzer, J. A. (1982). Managing information security: cific Asia Conference on Information System, 7-10 July,
A program for the electronic information age. Boston, Bangkok, Thailand.
MA: Butterworth-Heinemann.
453
Sedera, D., Gable, G., & Chan, T. (2003). ERP success: Shah, S. G., & Robinson, I. (2009). Developing medical
Does organisation size matter? 7th Pacific Asia Confer- device technologies from users’ perspectives: A theoretical
ence on Information Systems, Adelaide, (pp. 1075-1088). framework for involving users in the development process.
International Journal of Technology Assessment in Health
Sedera, D., Gable, G., & Rosemann, M. (2001). A balanced
Care, 25(4), 514–521. doi:10.1017/S0266462309990328
scorecard approach to enterprise systems performance
measurement. In, G. Finnie, D. Cecez-Kecmanovic, & Shailer, D. M. (2001). Project manager’s toolkit: Prac-
B. Lo (Eds.), Proceedings of the Twelfth Australasian tical checklists for systems development. Butterworth-
Conference on Information Systems, Coffs Harbour Heinemann.
Australia, 5-7 December.
Shang, S., & Seddon, P. B. (2002). Assessing and managing
Segars, A. H. (1997). Assessing the unidimensionality the benefits of enterprise systems: The business managers
of measurement: A paradigm and illustration within the perspective. Information Systems Journal, 12, 271–299.
context of Information Systems research. Omega, 25(1), doi:10.1046/j.1365-2575.2002.00132.x
107–121. doi:10.1016/S0305-0483(96)00051-5
Shannak, R. O. (1999). An interpretive approach to
Segars, A., & Grover, V. (1998). Strategic Information analysing the role and value of Information Systems in
Systems planning success: An investigation of the con- Jordanian finacial organisations. Doctor of philosophy.
struct and its measurement. Management Information Aston University.
Shannon, C., & Weaver, W. (1969). The mathematical
Selim, H. (2007). Critical success factors for e-learning theory of communication. University of Illinois Press.
acceptance: Confirmatory factor models. Computers
Sharkey, U., Scott, M., Galway, I., & Acton, T. (2007).
& Education, 49(2), 396–413. doi:10.1016/j.compe-
The influence of quality on the success of e-commerce
du.2005.09.004
systems. National University of Ireland.
Sellitto, C., & Carbone, D. (2007). Success factors associ-
Sharma, H., & Jansen, B. J. (2005). Automated evaluation
ated with health Information Systems implementation: A
of search engine performance via implicit user feedback.
study of an Australian regional hospital. Journal of Busi-
Paper presented at the 8th Annual İnternational ACM
ness Systems, Governance and Ethics, 2(4).
SIGIR Conference on Research and Development in
Sellitto, C., Wenn, A., & Burgess, S. (2003). A review of İnformation Retrieval, Salvador.
the websites of small Australian wineries: Motivations,
Sharma, A., Krishnan, R., & Grewal, D. (2001). Value
goals and success. Information Technology Management,
creation in markets: A critical area of focus for business-
4(2), 215–232. doi:10.1023/A:1022954429432
to-business markets. Industrial Marketing Management,
Shah, S. G., Farrow, A., & Robinson, I. (2009). The 30(4), 391–402. doi:10.1016/S0019-8501(01)00153-5
representation of healthcare end users’ perspectives by
Shee, D., & Wang, Y. (2008). Multi-criteria evaluation
surrogates in healthcare decisions: A literature review.
of the Web-based e-learning system: A methodology
Scandinavian Journal of Caring Sciences, 23(4), 809–819.
based on learner satisfaction and its applications. Com-
doi:10.1111/j.1471-6712.2008.00674.x
puters & Education, 50(3), 894–905. doi:10.1016/j.
Shah, S. G., & Robinson, I. (2006). User involvement compedu.2006.09.005
in healthcare technology development and assessment:
Shih, Y. Y., & Huang, S. S. (2009). The actual usage of
Structured literature review. International Journal of
ERP systems: An extended technology acceptance per-
Health Care Quality Assurance, 19(6-7), 500–515.
spective. Journal of Research and Practice in Information
doi:10.1108/09526860610687619
Technology, 41(3), 263–276.
Shimeall, T. J., & McDermott, J. J. (1999). Software

security in an internet world: An executive summary.
IEEE Software, 16(4), 58–62. doi:10.1109/52.776950
454
Shivers-Blackwell, S. L., & Charles, A. C. (2006). Singh, Y., & Goel, B. (2007). A step towards software
Ready, set, go: Examining student readiness to use ERP preventive maintenance. ACM SIGSOFT Software En-
technology. Journal of Management Development, 25(8), gineering Notes, 32(4).
795–805. doi:10.1108/02621710610684268
Sink, D. S., & Tuttle, T. C. (1989). Planning and measure-
Shun, C., & Yunjie, X. (2006). Effects of outcome, process ment in your organization of the future. Norcross, GA:
and shopping enjoyment on online consumer behaviour. Industrial Engineering and Management Press.
Electronic Commerce Research and Applications, 5(4),
Siponen, M. T. (2001). On the role of human morality
272–281. doi:10.1016/j.elerap.2006.04.004
in Information Systems security. Information Resources
Shurville, S., O’Grady, T., & Mayall, P. (2008). Educa- Management Journal, 14(4), 15–23. doi:10.4018/
tional and institutional flexibility of Australian educational irmj.2001100102
software. Campus-Wide Information Systems, 25(2),
Siponen, M. T. (2005). Analysis of modern IS security
74–84. doi:10.1108/10650740810866576
development approaches: Toward the next generation
Shye, A., Ozisikyilmaz, B., Mallik, A., Memik, G., Dinda, of social and adaptable ISS methods. Information
P. A., Dick, R. P., et al. (2008). Learning and leveraging and Organization, 15, 339–375. doi:10.1016/j.infoan-
the relationship between architecture-level measurements dorg.2004.11.001
and ındividual user satisfaction. Paper presented at the
Siponen, M., Pahnila, S., & Mahmood, M. A. (2010).
ISCA ‘08: 35th International Symposium on Computer
Compliance with information security policies: An empiri-
Architecture.
cal investigation. Computer, 43(2), 64–71. doi:10.1109/
Siegel, D., & Griliches, Z. (1992). Purchased services, MC.2010.35
outsourcing, computers, and productivity in manufactur-
Skok, W., Kophamel, A., & Richardson, I. (2001).
ing. In Griliches, Z. (Ed.), Output measurement in the
Diagnosing Information Systems success: Importance-
service sectors (pp. 429–458). Chicago, IL: University
performance maps in the health industry. Information
of Chicago Press.
& Management, 38, 409–419. doi:10.1016/S0378-
Šimková, E., & Basl, J. (2006). Business value of IT. 7206(00)00076-8
Systems Integration Conference, Prague.
Smart, W. (2009). Information system success: Evaluation
Simon, S. J., & Paper, D. (2007). User acceptance of voice of a carbon accounting and sequestration system. Doctoral
recognition technology: An empirical extension of the Dissertation, Southern Cross University.
technology acceptance model. Journal of Organizational
Smith, H. J. (1994). Managing privacy: Information
and End User Computing, 19(1), 24–50. doi:10.4018/
Technology and corporate America. Chapel Hill, NC:
joeuc.2007010102
University of North Carolina Press.
Simonsen, J., & Hertzum, J. (2006). A regional PD strat-
Smolin, R. (2008). From program to product: Turning
egy for EPR systems: Evidence-based IT development.
your code into a saleable product. Apress.
Ninth Biennial Conference on Participatory Design, 2,
(pp. 125-128). Smyth, R. W. (2001). Challenges to successful ERP use.
The 9th European Conference on Information Systems,
Simonsen, J., & Hertzum, J. (2008). Participative design
Bled.
and the challenges of large-scale systems: Extending the
iterative PD approach. Proceedings Participatory Design Snider, B., da Silveira, G. J. C., & Balakrishnan, J.
Conference. (2009). ERP implementation at SMEs: Analysis of
five Canadian cases. International Journal of Op-
Simons, R. (1995). Levers of control: How managers use
erations & Production Management, 29(1-2), 4–29.
innovative control systems to drive strategic renewal.
doi:10.1108/01443570910925343
Boston, MA: Harvard Business School Press.
455
Snitkin, S., & King, W. (1986). Determinants of the ef- Song, J., & Zahedi, F. (2001). Web design in e-commerce:
fectiveness of personal decision support systems. Infor- A theory and empirical analysis. Paper presented at the
mation & Management, 10(2), 83–89. doi:10.1016/0378- International Conference on Information Systems, New
7206(86)90050-9 Orleans, LA.
Soares, P. G. (1992). On remote procedure call. Paper Son, S., Weitzel, T., & Laurent, F. (2005). Designing a
presented at the Conference of the Centre for Advanced process-oriented framework for IT performance manage-
Studies on Collaborative research, Toronto, Ontario, ment systems. The Electronic Journal of Information
Canada. Systems Evaluation, 8(3), 219–228.
Sobol, M., & Klein, G. (2009). Relation of CIO back- Southern, A., & Tilley, F. (2000). Small firms and informa-
ground, IT infrastructure, and economic performance. In- tion and communication technologies (ICTs): Toward a
formation & Management, 46(5), 271–278. doi:10.1016/j. typology of ICTs usage. New Technology, Work and Em-
im.2009.05.001 ployment, 15(2), 138–154. doi:10.1111/1468-005X.00070
Soh, C., & Markus, M. L. (1995.) How IT creates business Southon, G., Sauer, c., & Dampney, K. (1999). Les-
value: A process theory synthesis. Proceedings of the Six- sons form a failed information system initiative: Issues
teenth International Conference on Information Systems. for complex organizations. International Journal of
Medical Informatics, 55, 33–46. doi:10.1016/S1386-
Soh, C., Sia, K. S., Boh, W. F., & Tang, M. (2003). Misalign-
5056(99)00018-0
ments in ERP implementation: A dialectic perspective.
International Journal of Human-Computer Interaction, Spiliopoulou, M. (2000). Web usage mining for website
16(1), 81–100. doi:10.1207/S15327590IJHC1601_6 evaluation. Communications of the ACM, 43(8), 127–134.
doi:10.1145/345124.345167
Solla Price, D. d. (1963). Little science, big science. New
York, NY: Columbia University Press. Spiliopoulou, M., & Pohle, C. (2001). Data mining for
measuring and improving the success of websites. Data
Solow, R. S. (1987). We’d better watch out. New York
Mining and Knowledge Discovery, 5(1-2), 85–114.
Times Book Review.
doi:10.1023/A:1009800113571
Somanchi, V., & Dwivedula, R. (2010). Program manage-
Spiliopoulou, M., Pohle, C., & Faulstich, L. (2000).
ment approach for large-scale Information Technology
Improving the effectiveness of a website with Web us-
development projects: A case study. Journal of Project.
age mining. In Masand, B., & Spilliopoulou, M. (Eds.),
Program & Portfolio Management, 1(1), 41–54.
WEBKDD, LNAI (pp. 142–162). Berlin, Germany:
Somers, M. T., Nelson, K., & Karimi, J. (2003). Confir- Springer-Verlag.
matory factor analysis of the end-user computing satis-
Stallings, W. (1995). Network and internetwork security
faction instrument: Replication within an ERP domain.
principles and practice. Englewood Cliffs, NJ: Prentice
Decision Sciences, 34(3), 595–621. doi:10.1111/j.1540-
Hall.
5414.2003.02428.x
Standish Group. (1995). The CHAOS report. Retrieved
Somers, T. M., & Neslon, K. G. (2004). A taxonomy of
12 April, 2005, from http://www.standishgroup.com/
players and activities across the ERP project life cycle.
sample_research/chaos_1994_1.php
S0378-7206(03)00023-5 Standish Group. (1998). CHAOS: A recipe for success.
Retrieved 12 April, 2005, from http://www.standishgroup.
Sommerville, I. (2004). Software engineering (7th ed.).
com/sample_research/chaos1998.pdf
Pearson Addison-Wesley.
Stanton, J. M., Stam, K. R., Mastrangelo, P., & Jolton, J.
(2005). Analysis of end user security behaviors. Computers
& Security, 24, 124–133. doi:10.1016/j.cose.2004.07.001
456
Staples, D. S., Wong, I., & Seddon, P. B. (2002). Having Stockdale, R., & Lin, C. (2008). Evaluation of the effec-
expectations of İnformation Systems benefits that match tiveness of small and medium sized businesses websites
received benefits: Does İT really matter? Information in a business to business context. In Calero, C., Muñoz,
& Management, 40, 115–131. doi:10.1016/S0378- C., Moraga, A., & Piattini, M. (Eds.), Handbook of re-
7206(01)00138-0 search on Web Information Systems quality (Vol. 4, pp.
71–85). Hershey, PA: Information Science Reference.
Stapleton, J. (2002). DSDM: A framework for business
doi:10.4018/978-1-59904-847-5.ch004
centred development. Addison-Wesley.
Stoney, C., & Winstanley, D. (2001). Stakeholding: Confu-
Stav, J., Nielsen, K., Hansen-Nygård, G., & Thorseth,
sion or utopia? Mapping the conceptual terrain. Journal of
T. (2010). Experiences obtained with integration of
Management Studies, 38(5), 603–626. doi:10.1111/1467-
students response systems for iPod Touch and iPhone
6486.00251
into e-learning environments. Electronic Journal of e-
Learning, 8(2), 179-90. Straub, D. (1989). Validating instruments in MIS research.
Management Information Systems Quarterly, 13(2),
Steers, R. M. (1975). Problems in the measurement of
147–169. doi:10.2307/248922
organisational effectiveness. Administrative Science
Quarterly, 20(4), 546–558. doi:10.2307/2392022 Straub, D. (2006). The value of scientometric studies: An
introduction to a debate on IS as a reference discipline.
Stefanou, C. J. (2001). A framework for the ex-ante evalu-
Journal of the Association for Information Systems, 7(5),
ation of ERP software. European Journal of Information
241–245.
Systems, 10, 204–215. doi:10.1057/palgrave.ejis.3000407
Straub, D. W. (1990). Effective IS security: An empiri-
Stefanou, C. J., & Revanoglou, A. (2006). ERP integra-
cal study. Information Systems Research, 1(3), 255–276.
tion in a healthcare environment: A case study. Journal
doi:10.1287/isre.1.3.255
of Enterprise Information Management, 19(1), 115–130.
doi:10.1108/17410390610636913 Straub, D. W., Limayem, M., & Karahanna-Evaristo, E.
(1995). Measuring systems usage: Implications for IS
Steinberg, R. A. (2006). Measuring ITIL: Measuring,
theory testing. Management Science, 41(8), 1328–1342.
reporting and modeling the IT service management met-
doi:10.1287/mnsc.41.8.1328
rics that matter most to IT senior executives. Victoria,
Canada: Trafford. Straub, D. W., & Welke, R. J. (1998). Coping with systems
risk: Security planning models for management decision
Sterne, D. F. (1991). On the buzzword ‘security policy’.
making. Management Information Systems Quarterly,
In Proceedings of the IEEE Computer Society Symposium
22(4), 441–469. doi:10.2307/249551
on Research in Security and Privacy, 219-230.
Straub, D., Hoffman, D., Weber, B., & Steinfield, C.
Stockdale, R., Lin, C., & Liu, Y. (2006). The perception
(2002). Toward new metrics for Net-enhanced orga-
of website effectiveness by West Australian small business
nizations. Information Systems Research, 13(3), 227.
owners in a B2B context. Paper presented at the Pacific
doi:10.1287/isre.13.3.227.80
Asia Conference on Information Systems (PACIS06).
Straub, D., & Klein, R. (2001). E-competitive transfor-
Stockdale, R., Lin, C., & Stoney, S. (2005). The ef-
mations. Business Horizons, 44(3), 3–12. doi:10.1016/
fectiveness of SME websites in a business to business
S0007-6813(01)80029-X
context. Paper presented at the The IADIS E-Commerce
Conference. Stroes, H. J., & Egberts, M. E. (1996). Veranderen met
resultaat. Deventer, The Netherlands: Kluwer Bedrijf-
sInformatie (Dutch text).
457
Strother, J. (2002). An assessment of the effectiveness of Sun, Y., Bhattacherjee, A., & Ma, Q. (2009). Extending
e-learning in corporate training programs. International technology usage to work settings: The role of perceived
Review of Research in Open and Distance Learning, work compatibility in ERP implementation. Informa-
3(1), 1–17. tion & Management, 46(6), 351–356. doi:10.1016/j.
im.2009.06.003
Stvilia, B., Gasser, L., Twidale, M., & Smith, L. (2007).
A framework for information quality assessment. Journal Sutherland, J. (2001). Agile can scale: Inventing and re-
of the American Society for Information Science and inventing SCRUM in five companies. Cutter IT Journal,
Technology, 58(12), 1720–1733. doi:10.1002/asi.20652 14(12), 5–11.
Stvilia, B., Twidale, M., Smith, L., & Gasser, L. (2008). Sveiby, K. E. (1997). The new organizational wealth:
Information quality work organization in Wikipedia. Managing & measuring knowledge-based assets (1st ed.).
Journal of the American Society for Information Science San Francisco, CA: Berrett-Koehler Publishers.
and Technology, 59(6), 983–1001. doi:10.1002/asi.20813
Sviokla, J. (1989). Business implication of knowledge-
Stylos, C. D., & Groumpos, P. P. (1999). Mathematical based systems. In Sprague, R. H., & Watson, H. J. (Eds.),
formulation of fuzzy cognitive maps. Proceedings of the Decision support systems: Putting theory into practice
7th Mediterranean Conference on Control and Automa- (pp. 125–151). Englewood Cliffs, NJ: Prentice Hall.
tion, (pp. 2251-2261).
Symon, G., & Clegg, C. (2005). Constructing identity
Stylos, C. D., & Groumpos, P. P. (2000). Fuzzy cognitive and participation during technological change. Human
maps in modeling supervisory control systems. Journal Relations, 58(9), 1141. doi:10.1177/0018726705058941
of Intelligent and Fuzzy Systems, 8, 83–98.
Szyperski, C., Gruntz, D., & Murer, S. (2002). Component
Subramoniam, S., & Sadi, A. H. M. S. (2010). Information software: Beyond object-oriented programming (2nd ed.).
Technology Professional, Nov/Dec, 46-51. Addison-Wesley.
Sundhedsministeriet (The Ministry of Health). (1999). Tait, P., & Vessey, I. (1988). The effect of user involve-
National strategi for IT i sygehusvæsenet 2000-2002 ment on system success: A contingency approach. Man-
(national strategy for IT in the hospital sector 2000-2002). agement Information Systems Quarterly, 12(1), 91–108.
København, Denmark: Sundhedsstyrelsen. doi:10.2307/248809
Sung, T. (2006). E-commerce critical success factors: East Tallon, P., & Kraemer, K. (2002). Executives’perspectives
vs. West. Technological Forecasting and Social Change, on IT: Unraveling the link between business strategy,
73(9), 1161–1177. doi:10.1016/j.techfore.2004.09.002 management practices and IT business value. Paper
presented at the America’s Conference on Information
Sun, L., Srivastava, R. P., & Mock, T. J. (2006). An in-
Systems (ACIS2002).
formation systems security risk assessment model under
the Dempster-Shafer theory of belief functions. Journal Tallon, P. P., Kraemer, K. L., & Gurbaxani, V. (2000).
of Management Information Systems, 22(4), 109–142. Executives’ perceptions of the business value of informa-
doi:10.2753/MIS0742-1222220405 tion technology: A process-oriented approach. Journal
Sun, P., Tsai, R., Finger, G., Chen, Y., & Yeh, D. (2008).
What drives a successful e-learning? An empirical Tallon, P., Kraemer, K., & Gurbaxani, V. (2000). Execu-
investigation of the critical factors influencing learner tives’ perceptions of the business value of information
satisfaction. Computers & Education, 50(4), 1183–1202. technology: A process-oriented approach. Journal of
doi:10.1016/j.compedu.2006.11.007 Management Information Systems, 16(4), 145–173.
458
Tanaka, H., Matsuura, K., & Sudoh, O. (2005). Vulner- Tejay, G., Dhillon, G., & Chin, A. G. (2005). Data quality
ability and information security investment: An empiri- dimensions for Information Systems security: A theoretical
cal analysis of e-local government in Japan. Journal of exposition. In Dowland, P., Furnell, S., Thuraisingham,
Accounting and Public Policy, 24, 37–59. doi:10.1016/j. B., & Wang, X. S. (Eds.), Security management, integrity,
jaccpubpol.2004.12.003 and internal control in Information Systems (pp. 21–39).
New York, NY: Springer. doi:10.1007/0-387-31167-X_2
Tanenbaum, A. (1992). Modern operating systems.
Englewood Cliffs, NJ: Prentice-Hall. Telang, R., & Wattal, S. (2007). An empirical analysis
of the impact of software vulnerability announcements
Tang, C. M., & Marthandan, G. (2011). An analytical
on firm stock price. IEEE Transactions on Software
model to measure IS-enabled organizational effectiveness.
Engineering, 33(8), 544. doi:10.1109/TSE.2007.70712
International Journal of Enterprise Information Systems,
7(2), 50–65. doi:10.4018/jeis.2011040104 Tenenhaus, M., Vinzi, V. E., Chatelin, Y. M., & Lauro,
C. (2005). PLS path modelling. Computational Statis-
Tangen, S. (2004). Performance measurement from
tics & Data Analysis, 48(1), 159–205. doi:10.1016/j.
philosophy to practice. International Journal of Produc-
csda.2004.03.005
tivity and Performance Management, 53(8), 726–737.
doi:10.1108/17410400410569134 Teo, T. S. H., & Wong, P. K. (1998). An empirical study
of the performance impact of computerization in the retail
Tangen, S. (2005). Demystifying productivity and
industry. The International Journal of Management Sci-
performance. International Journal of Productiv-
ence, 26(5), 611–621.
ity and Performance Management, 54(1), 34–46.
doi:10.1108/17410400510571437 Teo, T., & Pian, Y. (2004). A model for Web adoption.
Tangpong, C. (2008). IT-performance paradox revisited:
S0378-7206(03)00084-3
Resource-based and prisoner’s dilemma perspectives.
Journal of Applied Management and Entrepreneurship, Thelwall, M. (2001). Effective websites for small and
13(1), 35–49. medium-sized enterprises. Journal of Small Business and
Enterprise Development, 7(2), 149–159. doi:10.1108/
Tang, R., & Thelwall, M. (2003). US academic departmen-
EUM0000000006836
tal Web-site interlinking in the United States disciplinary
differences. Library & Information Science Research, Thierauf, R. J. (1988). User-oriented decision support
25(4), 437–458. doi:10.1016/S0740-8188(03)00053-7 systems. Englewood Cliffs, NJ: Prentice Hall.
Tan, K. S., Chong, S. C., Lin, B. S., & Eze, U. C. (2009). Thiry, M. (2010). Program management: Fundamentals
Internet-based ICT adoption: Evidence from Malaysian of project management. Gower.
SMEs. Industrial Management & Data Systems, 109(1-2),
Thompson, R., Compeau, D., & Higgins, C. (2006).
224–244. doi:10.1108/02635570910930118
Intentions to use Information Technologies: An integra-
Tauritz, D. R., Kok, J. N., & Sprinkhuizen-Kuyper, I. G. tive model. Journal of Organizational and End User
(2000). Adaptive ınformation filtering using evolutionary Computing, 18(3), 25–46. doi:10.4018/joeuc.2006070102
computation. Information Sciences, 122(2-4), 121–140.
Thong, J. Y. L. (2001). Resource constraints and Infor-
doi:10.1016/S0020-0255(99)00123-1
mation Systems implementation in Singaporean small
Teece, D., Pisano, G., & Shuen, A. (1997). Dynamic businesses. Omega-International Journal of Manage-
capabilities and strategic management. Strategic Manage- ment Science, 29(2), 143–156. doi:10.1016/S0305-
ment Journal, 18(7), 509–533. doi:10.1002/(SICI)1097- 0483(00)00035-9
0266(199708)18:7<509::AID-SMJ882>3.0.CO;2-Z
459
Thong, J. Y. L., & Yap, C. S. (1995). CEO characteristics, Treacy, M., & Wiersma, F. (1997). The discipline of
organizational characteristics and Information Technol- market leaders: Choose your customers, narrow your
ogy adoption in small businesses. Omega-International focus, dominate your market. London, UK: HarperCollins.
Journal of Management Science, 23(4), 429–442.
Trimmer, K. J., Pumphrey, L. D., & Wiggins, C.
doi:10.1016/0305-0483(95)00017-I
(2002). ERP implementation in rural healthcare. Jour-
Thong, J. Y. L., & Yap, C. S. (1996). Information Sys- nal of Management in Medicine, 16(2/3), 113–132.
tems effectiveness: A user satisfaction approach. Infor- doi:10.1108/02689230210434871
mation Processing & Management, 32(5), 601–610.
Trompeters, C. M., & Eloff, J. H. P. (2001). A framework
doi:10.1016/0306-4573(96)00004-0
for the implementation of socio-ethical controls in in-
Thong, J. Y. L., Yap, C. S., & Raman, K. S. (1996). Top formation security. Computers & Security, 20, 384–391.
management support, external expertise and Information doi:10.1016/S0167-4048(01)00507-7
Systems implementation in small businesses. Informa-
Trotter, P. (1976). User feedback and how to get it. Paper
tion Systems Research, 7(2), 248–267. doi:10.1287/
presented at the 4th Annual ACM SIGUCCS Conference
isre.7.2.248
on User Services, Tucson.
Tilley, S., Gerdes, J., Hamilton, T., Huang, S., Müller, H.,
Tsim, Y. C., Yeung, V. W. S., & Leung, E. T. C. (2002).
& Wong, K. (2002). Adoption challenges in migrating to
An adaptation to ISO 9001:2000 for certified organisa-
Web services. Paper presented at the Fourth International
tions. Managerial Auditing Journal, 17(5), 245–250.
Workshop on Web Site Evolution.
doi:10.1108/02686900210429669
Tillquist, J., & Rodgers, W. (2005). Using asset specificity
Turban, E., & Gehrke, D. (2000). Determinants of e-
and asset scope to measure the value of IT. Communications
commerce website. Human Systems Management, 19,
of the ACM, 48(1), 75–80. doi:10.1145/1039539.1039542
111–120.
Tittle, C. R. (1980). Sanctions and social deviance: The
Turner, C., Zavod, M., & Yurcik, W. (2001). Factors that
question of deterrence. New York, NY: Praeger.
affect the perception of security and privacy of e-commerce
Tompkins, T., & Handley, D. (2003). Giving e-mail back websites. Paper presented at the Fourth International
to the users: Using digital signatures to solve the spam Conference on Electronic Commerce Research.
problem. First Monday, 8(9).
Turoff, M. (1999). Education, commerce, communica-
Tong, M., Bishop, G., Enkelmann, H., Why, Y., Di- tions: The era of competition. WebNet Journal, 1(1), 22–31.
ong, S., Khader, M., & Ang, J. (2007). Emotion and
Tyre, M., & Orlikowski, W. (1994). Windows of op-
appraisal: A study using ecological momentary as-
portunity: Temporal patterns of technological adaptation
sessment. Cognition and Emotion, 21(7), 1361–1381.
in organizations. Organization Science, 5(1), 98–118.
doi:10.1080/02699930701202012
doi:10.1287/orsc.5.1.98
Torkzadeh, G., & Dhillon, G. (2003). Measuring factors
Tzeng, S., Chen, W., & Pai, F. (2008). Evaluating the
that influence the success of Internet commerce. Infor-
business value of RFID: Evidence from five case studies.
mation Systems Research, 13(2), 187–204. doi:10.1287/
International Journal of Production Economics, 112(2),
isre.13.2.187.87
601–613. doi:10.1016/j.ijpe.2007.05.009
Torkzadeh, G., & Doll, W. J. (1993). The place and value
Udo, G. J., & Davis, J. S. (1992). Factors affecting decision
of documentation in end-user computing. Information
support systems benefits. Information & Management, 23,
& Management, 24(3), 147–158. doi:10.1016/0378-
359–371. doi:10.1016/0378-7206(92)90017-A
7206(93)90063-Y
Tracy, L., Guevara, J., Harcourt, O., & Stegman, E. (2006).

IT key metrics data: Key industry measures: Sample
measures. Gartner 2006.
460
Udo, G., Bagchi, K., & Kirs, P. (2010). An assessment Van der Meijden, M. J., Tange, H., Troost, J., & Hasman,
of customers’e-service quality perception, satisfaction A. (2003). Determinants of success of inpatient clinical
and intention. International Journal of Information Information Systems: A literature review. Journal of the
Management, 30(6), 481–492. doi:10.1016/j.ijin- American Medical Informatics Association, 10, 235–243.
fomgt.2010.03.005 doi:10.1197/jamia.M1094
Udo, G., & Marquis, G. (2001). Factors affecting e- Van der Merwe, R., & Bekker, J. (2003). A framework
commerce website effectiveness. Journal of Computer and methodology for evaluating e-commerce websites.
Information Systems, 42(2), 10–16. Internet Research: Electronic Networking. Applications
and Policy, 13(5), 330–341.
Ulrich, D., Zenger, J., & Smallwood, N. (1999). Results-
based leadership. Boston, MA: Harvard Business School van der Zee, J. T. M., & de Jong, B. (1999). Alignment
Press. is not enough: Integrating business and Information
Technology management with the balanced business
Umble, E. J., Haft, R. R., & Umble, M. M. (2002). En-
scorecard. Journal of Management Information Systems,
terprise resource planning: implementation procedures
16(2), 137–156.
and CSF. European Journal of Operational Research,
146(2), 241–257. doi:10.1016/S0377-2217(02)00547-7 Van Deursen, A., Visser, E., & Warmer, J. (2007). Model-
driven software evolution: A research agenda. Proceedings
Urbach, N., Smolnik, S., & Riempp, G. (2009). The state
of the International Workshop on Model-Driven Software
of research on Information Systems success. Business
Evolution held with the ECSMR’07.
& Information Systems Engineering, 1(4), 315–325.
doi:10.1007/s12599-009-0059-y Van Dyke, T., Kappelman, L., & Prybutok, V. (1997).
Measuring Information Systems service quality: Concerns
Uzoka, F. M. E., Abiola, R. O., & Nyangeresi, R. (2008).
on the use of the SERVQUAL questionnaire. Manage-
Influence of product and organizational constructs on
ERP acquisition using an extended technology acceptance
doi:10.2307/249419
model. International Journal of Enterprise Information
Systems, 4(2), 67–83. doi:10.4018/jeis.2008040105 Van Grembergen, W. (2000). The balanced scorecard and
IT governance. Information Systems Control Journal,
Vaidyanathan, K., & Trivedi, K. S. (2005). A comprehen-
2, 40–43.
sive model for software rejuvenation. IEEE Transactions
on Dependable and Secure Computing, 2(2), 124–137. van Grembergen, W., De Haes, W., & Amelinckx, S.
doi:10.1109/TDSC.2005.15 I. (2003). Using COBIT and the balanced scorecard as
instruments for service level management. Information
Valdés, O., St-Jean, M., Renault, A., Picard, M., Cortina, S.,
Systems Control Journal, 4, 56–62.
& Betry, V. (2009). ITSM process assessment supporting
ITIL. Amersfoot, The Netherlands: Van Haren Publishing. Van Raan, A. F. J. (1996). Advanced bibliometric meth-
ods as quantitative core of peer review based evaluation
van Bon, J., & van Selm, L. (2008). ISO/IEC 20000: An
and foresight exercises. Scientometrics, 36(3), 397–420.
introduction (1st ed.). Zaltbommel, The Netherlands: Van
doi:10.1007/BF02129602
Haren Publishing.
Vargo, S., & Lusch, R. (2004). Evolving to a new dominant
Van der Meijden, M. J., Tange, H., Troost, J., & Hasman,
logic for marketing. Journal of Marketing, 68(1), 1–17.
A. (2001). Development and implementation of an EPR:
doi:10.1509/jmkg.68.1.1.24036
How to encourage the user. International Journal of
Medical Informatics, 64, 173–185. doi:10.1016/S1386- Veerman, N. (2006). Automated mass maintenance of a
5056(01)00208-8 software portfolio. Science of Computer Programming,
62, 287–317. doi:10.1016/j.scico.2006.04.006
461
Velcu, O. (2007). Exploring the effects of ERP systems Verhoef, C. (2005). Quantitative aspects of outsourcing
on organisational performance: Evidence from Finnish deals. Science of Computer Programming, 56(3), 275–313.
companies. Industrial Management & Data Systems, doi:10.1016/j.scico.2004.08.003
107(9), 1316–1334. doi:10.1108/02635570710833983
Verville, J. (2002). Evaluating ERP Software in the
Venkatesh, V. (1998). User acceptance of information acuqisition process: A three phase activity. International
technology: A unified view. Unpublished doctoral dis- Association for Computer Information Systems Confer-
sertation, University of Minnesota. ence, (pp. 625 – 631).
Venkatesh, V. (2000). Determinants of perceived ease of Vicente, B. d., Alonso, A., & Amador, J. (1991). Dynamic
use: Integrating perceived behavioural control, computer software replacement model and its Ada implementation.
anxiety and enjoyment into the technology acceptance Paper presented at the Annual International Conference
model. Information Systems Research, 11(4), 342–365. on Ada, TRI-Ada ‘91: Today’s Accomplishments; Tomor-
doi:10.1287/isre.11.4.342.11872 row’s Expectations, San Jose, California.
Venkatesh, V., & Bala, H. (2008). Technology accep- Viedma-Del-Jesus, M. I., Perakakis, P., Muñoz, M. A.,
tance model 3 and a research agenda on interventions. López-Herrera, A. J., & Vila, J. (2011). Sketching the first
Decision Sciences, 39(2), 273–315. doi:10.1111/j.1540- 45 years of the journal Psychophysiology (1964–2008):
5915.2008.00192.x A co-word-based analysis. Psychophysiology, 48(8),
1029–1036. doi:10.1111/j.1469-8986.2011.01171.x
Venkatesh, V., & Davis, F. D. (2000). A theoretical
extension of the technology acceptance model: Four Vigna, G., & Kemmeerer, R. A. (1999). NetSTAT: A
longitudinal field studies. Management Science, 46(2), network-based intrusion detection system. Journal of
186–205. doi:10.1287/mnsc.46.2.186.11926 Computing Security, 7(1), 37–71.
Venkatesh, V., & Morris, M. G. (2000). Why don’t men Vijayaraman, B., & Bhatia, G. (2002). A framework for
ever stop to ask for directions? Gender, social ınfluence, determining success factors of an e-commerce initiative.
and their role in technology acceptance and usage behav- Journal of Internet Commerce, 1(2), 63–75. doi:10.1300/
ior. Management Information Systems Quarterly, 24(1), J179v01n02_05
115–139. doi:10.2307/3250981
Villanueva, T., & Caston, R. (1981). Welfare legislation
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. and American poverty traps: Ironies and characteristics.
D. (2003). User acceptance of Information Technology: Journal of Social and & Societal Welfare, 8, 733.
toward a unified view. Management Information Systems
Vingtoft, S., Lippert, S., Bernstein, K., Bruun-Rasmussen,
Quarterly, 27(3), 425–479.
M., Kristensen, M., & Nøhr, C. (2000). EPJ-observatoriets
Venkatesh, V., Morris, M., & Davis, G., Davis. (2003). statusrapport 2000 (EHR status report 2000)
User acceptance of Information Technology: Toward a
Vliet, H. v. (2008). Software engineering: Principles and
unified view. Management Information Systems Quarterly,
practice (3rd ed.). John Wiley&Sons.
27(3), 425–478.
von Solms, B. (2000). Information security – The
Venkatraman, N., & Ramanujam, V. (1986). Measure-
third wave? Computers & Security, 19(7), 615–620.
ment of business performance in strategy research: A
doi:10.1016/S0167-4048(00)07021-8
comparison of approaches. Academy of Management
Review, 11(4), 801–814. von Solms, R., & von Solms, B. (2004). From poli-
cies to culture. Computers & Security, 23(4), 275–279.
Verhees, F., & Meulenberg, M. T. G. (2004). Market
doi:10.1016/j.cose.2004.01.013
orientation, innovativeness, product innovation, and
performance in small firms. Journal of Small Business
Management, 42(2), 134–154. doi:10.1111/j.1540-
627X.2004.00102.x
462
von Solms, R., & von Solms, S. H. (2006). Information Wang, E. T. G., Shih, S.-P., Jiang, J. J., & Klein, G. (2007).
security governance: A model-based on the direct-control The consistency among facilitating factors and ERP
cycle. Computers & Security, 25, 408–412. doi:10.1016/j. implementation success: A holistic view of fit. Journal of
cose.2006.07.005 Systems and Software, 81(9), 1609–1621. doi:10.1016/j.
jss.2007.11.722
von Solms, S. H. (2005). Information security governance
- Compliance management vs operational management. Wang, R., & Strong, D. (1996). Beyond accuracy: What
Computers & Security, 24, 443–447. doi:10.1016/j. data quality means to data consumers. Journal of Man-
cose.2005.07.003 agement Information Systems, 12(4), 33.
Vroom, C., & von Solms, R. (2004). Towards information Wang, T. C., & Lin, Y. L. (2009). Accurately predicting
security behavioural compliance. Computers & Security, the success of B2B e-commerce in small and medium
23(3), 191–198. doi:10.1016/j.cose.2004.01.012 enterprises. Expert Systems with Applications, 36(2),
2750–2758. doi:10.1016/j.eswa.2008.01.033
Waddington, C. H. (1977). Tools for thought about complex
systems. St. Albans, UK: Palladin Press. Wang, W., & Wang, C. (2009). An empirical study of
instructor adoption of Web-based learning systems.
Wagner, N., Hassanein, K., & Head, M. (2008). Who is
responsible for e-learning success in higher education? A
compedu.2009.02.021
stakeholders’ analysis. Analysis, 11(3), 26–36.
Wang, Y. (2008). Assessing e commerce systems success: A
Wallace, L., Keil, M., & Rai, A. (2004). How software
respecification and validation of the DeLone and McLean
project risk affects project performance: An investi-
model of IS success. Information Systems Journal, 18(5),
gation of the dimensions of risk and an exploratory
529–557. doi:10.1111/j.1365-2575.2007.00268.x
model. Decision Sciences, 35(2), 289–321. doi:10.1111/
j.00117315.2004.02059.x Wang, Y. S., & Liao, Y. W. (2008). Assessing eGovernment
systems success: A validation of the DeLone and McLean
Wallace, L., Keil, M., & Rai, A. (2004). Understand-
model of information systems success. Government
ing software project risk: A cluster analysis. Informa-
Information Quarterly, 25(4), 717–733. doi:10.1016/j.
tion & Management, 42(1), 115–125. doi:10.1016/j.
giq.2007.06.002
im.2003.12.007
Wang, Y., & Ahmed, P. K. (2009). The moderating ef-
Wallace, T. F., & Kremzar, M. H. (2001). ERP: making
fect of the business strategic orientation on eCommerce
it happen – The implementer’s guide to success with en-
adoption: Evidence from UK family run SMEs. The
terprise resource planning. New York, NY: John Wiley
Journal of Strategic Information Systems, 18(1), 16–30.
& Sons.
doi:10.1016/j.jsis.2008.11.001
Walsh, M. E. (1981). Software security. Journal of Systems
Wang, Y., & Liao, Y. (2007). The conceptualization and
Management, 32(10), 6–14.
measurement of m-commerce user satisfaction. Comput-
Walton, E. J., & Dawson, S. (2001). Managers’ percep- ers in Human Behavior, 23(1), 381–398. doi:10.1016/j.
tions of criteria of organisational effectiveness. Journal of chb.2004.10.017
Management Studies, 38(2), 173–199. doi:10.1111/1467-
Wang, Y., Tang, T., & Tang, J. (2001). An instrument for
6486.00233
measuring customer satisfaction toward websites that
Wang, E. T. G. (2002). Transaction attributes and soft- market digital products and services. Journal of Electronic
ware outsourcing success: an empirical investigation of Commerce Research, 2(3), 89–102.
transaction cost theory. Information Systems Journal,
12(2), 153–181. doi:10.1046/j.1365-2575.2002.00120.x
463
Wang, Y., Wang, H., & Shee, D. (2007). Measuring Weill, P., Subramani, M., & Broadbent, M. (2002). Build-
e-learning systems success in an organizational con- ing IT infrastructure for strategic agility. MIT Sloan School
text: Scale development and validation. Computers in of Managment, Working Paper
Human Behavior, 23(4), 1792–1808. doi:10.1016/j.
Weill, P. (1992). The relationship between investment in
chb.2005.10.006
Information Technology and firm performance: A study
Warkentin, M., & Willison, R. (2009). Behavioral and of the value manufacturing sector. Information Systems
policy issues in Information Systems security: The insider Research, 3(4), 307–333. doi:10.1287/isre.3.4.307
threat. European Journal of Information Systems, 18(2),
Weill, P. (1992). The relationship between investment in
101–105. doi:10.1057/ejis.2009.12
Information Technology and firm performance: A study
Warman, A. R. (1992). Organizational computer security of the valve manufacturing sector. Information Systems
policy: The reality. European Journal of Information Research, 3(4), 307–333. doi:10.1287/isre.3.4.307
Systems, 1(5), 305–310. doi:10.1057/ejis.1992.2
Weill, P., & Ross, J. (2004). IT governance: How top
Warne, L., & Hart, D. (1996). The impact of organizational performers manage it decision rights for superior results.
politics on Information Systems project failure: A case Harvard Business Press.
study. Paper presented at the 29th Hawaii International
Weill, P., & Ross, J. (2005). A matrixed approach to de-
Conference on System Sciences (HICSS), Maui, Hawaii.
signing IT governance. MIT Sloan Management Review,
Watson, R., Pitt, L., & Kavan, C. (1998). Measuring 46(2), 26.
information systems service quality: Lessons from two
Weill, P., & Vitale, M. (2002). What IT infrastructure
longitudinal case studies. Management Information
capabilities are needed to implement e-business models.
MIS Quarterly Executive, 1(1), 17–34.
Wears, R. L., & Berg, M. (2005). Computer technology
Weischedel, B., & Huizingh, E. (2006). Website optimiza-
and clinical work. Still waiting for Godot. Journal of the
tion with Web metrics: A case study. Paper presented at the
American Medical Association, 293(10), 1261–1263.
8th International Conference on Electronic Commerce,
doi:10.1001/jama.293.10.1261
Fredericton, New Brunswick, Canada.
Webb, H., & Webb, L. (2004). SiteQual: An inte-
Welch, E., & Pandey, S. (2007). E-government and
grated measure of website quality. The Journal of
bureaucracy: Toward a better understanding of intranet
Enterprise Information Management, 17(6), 430–440.
implementation and its effect on red tape. Journal of Public
doi:10.1108/17410390410566724
Administration: Research and Theory, 17(3), 379–404.
Webler, T., Rakel, H., & Ross, R. J. S. (1992). A critical doi:10.1093/jopart/mul013
theoretical look at technical risk analysis. Industrial Crisis
Welling, R., & White, L. (2006). Website performance mea-
Quarterly, 6, 23–38.
surement: Promise and reality. Managing Service Quality,
Webster, J., & Watson, R. (2002). Analyzing the past to 16(6), 654–670. doi:10.1108/09604520610711954
prepare for the future: Writing a literature review. Man-
Welsh, J. A., & White, J. F. (1981). A small business is not
agement Information Systems Quarterly, 26(2), 13–23.
a little big business. Harvard Business Review, 59(4), 18.
Webster, J., & Watson, R. T. (2002). Analyzing the past to
Welti, N. (1999). Successful SAP R/3 implementation –
prepare for the future: Writing a literature review. Man-
Practical management of ERP project. Addison-Wesley.
agement Information Systems Quarterly, 26(2), xiii–xxiii.
Wen, H., Chen, H., & Hwang, H. (2000). E-commerce
Weick, K. (1990). Technology as equivoque: Sensemak-
website design: Strategies and models. Informa-
ing in new technologies. In Goodman, P., & Sproull, L.
tion Management & Computer Security, 9(1), 5–12.
(Eds.), Technology and organizations. San Francisco,
doi:10.1108/09685220110366713
CA: Jossey Bass.
464
Wernerfelt, B. (1984). A resource-based view of the Williamson, O. E. (1975). Markets and hierarchies: Analy-
firm. Strategic Management Journal, 5(2), 171–180. sis and antitrust implications. New York, NY: Free Press.
doi:10.1002/smj.4250050207
Williamson, O. E. (1981). The economics of organiza-
West-Knights, L. (2000). Getting IT right for govern- tion - The transaction cost approach. American Journal
ment: A review of public sector IT projects. Retrieved of Sociology, 87(3), 548–577. doi:10.1086/227496
from http://www.intellectuk.org/component/content/
Willison, R., & Backhouse, J. (2006). Opportunities for
article/139/4439-getting-it-right-for-government-report
computer crime: Considering systems risk from a crimi-
White, M. (2008). Permeating innovative technologies nological perspective. European Journal of Information
throughout the institution. Paper presented at the 36th Systems, 15, 403–414. doi:10.1057/palgrave.ejis.3000592
Annual ACM SIGUCCS Conference on User Services
Winniford, M., Conger, S., & Erickson-Harris, L. (2009).
Conference, Portland, OR.
Confusion in the ranks: IT service management practice
Whitman, M. E. (2003). Enemy at the gate: Threats to and terminology. Information Systems Management,
information security. Communications of the ACM, 46(8), 26(2), 153–163. doi:10.1080/10580530902797532
91–95. doi:10.1145/859670.859675
Wixom, B. H., & Watson, H. J. (2001). An empirical
Whitman, M. E. (2004). In defense of the realm: Un- investigation of the factors affecting data warehousing
derstanding the threats to information security. Interna- success. Management Information Systems Quarterly,
tional Journal of Information Management, 24(1), 43–57. 25(1), 17–41. doi:10.2307/3250957
doi:10.1016/j.ijinfomgt.2003.12.003
Wixom, B., & Todd, P. (2005). A theoretical integration
Whittaker, B. (1999). What went wrong? Unsuccess- of user satisfaction and technology acceptance. Infor-
ful Information Technology projects. Information mation Systems Research, 16(1), 85–102. doi:10.1287/
Management & Computer Security, 7(1), 23–30. isre.1050.0042
doi:10.1108/09685229910255160
Wold, H. (1982). Soft modelling: The basic design and
Whyte, G., & Bytheway, A. (1996). Factors affecting some extensions. Systems under indirect observations
Information Systems’ success. International Jour- (Vol. 2, pp. 1–52). Elsevier Science.
nal of Service Industry Management, 7(7), 74–93.
Wolfinbarger, M., & Gilly, M. C. (2003). eTailQ: di-
doi:10.1108/09564239610109429
mensionlizing, measuring and predicting etail quality.
Wierenga, B., & van Bruggen, G. H. (1998). Drivers Journal of Retailing, 79, 183–198. doi:10.1016/S0022-
of success for marketing management support systems: 4359(03)00034-4
towards a theory of marketing decision support. Erasmus
Wood, C. C. (1999). Information security policies made
University Rotterdam: Management Report No. 1998-8.
easy. San Rafael, CA: Baseline Software.
Wilkin, C., & Castleman, T. (2003). Development of an
Woodruf, J., & Burg, W. (2003). Satisfaction/dissatisfac-
instrument to evaluate the quality of delivered systems.
tion: are users predisposed? Information & Management,
Paper presented at the 36th Annual Hawaii International
40(4), 317–324. doi:10.1016/S0378-7206(02)00013-7
Conference, Big Island, Hawaii.
Woody, C. (2006). Applying OCTAVE: Practitioners
Willcocks, L., Hindle, J., Feeny, D., & Lacity, M. (2004).
report. Carnegie Mellon University.
IT and business process outsourcing: The knowledge
potential. Information Systems Management, 21(3), 7–15. Working Group from the Royal Academy of Engineering
doi:10.1201/1078/44432.21.3.20040601/82471.2 and the British Computer Society. (2004). The challenges
of complex IT projects. London, UK: The Royal Academy
Willcocks, L., & Lacity, M. C. (2006). Global sourc-
of Engineering.
ing of business and IT services. Palgrave Macmillan.
doi:10.1057/9780230288034
465
Wu, W. (2005). Carrying out contract successfully ends Yang, Z., Cai, S., Zhou, Z., & Zhou, N. (2005). Devel-
up with an IT outsourcing failure: Relationship manage- opment and validation of an instrument to measure user
ment cases of IT outsourcing in a cross-cultural context. perceived service quality of information presenting Web
ACIS 2005 Proceedings, (p. 23). portals. Information & Management, 42(4), 575–589.
Wu, J. H., & Wang, Y. M. (2006). Measuring KMS suc- Yang, Z., & Fang, X. (2004). Online service quality di-
cess: A respecification of the DeLone and McLean’s mensions and their relationship with satisfaction A content
model. Information & Management, 43(6), 728–739. analysis of customers reviews of securities brokerage ser-
doi:10.1016/j.im.2006.05.002 vices. International Journal of Service Industry Manage-
ment, 15(3), 302–326. doi:10.1108/09564230410540953
Wu, J. H., & Wang, Y. M. (2007). Measuring ERP success:
The key-users’ viewpoint of the ERP to produce a viable Yap, C. S., Soh, C. P. P., & Raman, K. S. (1992). Informa-
IS in the organization. Computers in Human Behavior, tion Systems success factors in small business. Omega-
23(3), 1582–1596. doi:10.1016/j.chb.2005.07.005 International Journal of Management Science, 20(5-6),
597–609. doi:10.1016/0305-0483(92)90005-R
Wu, J., Tennyson, R., & Hsia, T. (2010). A study of student
satisfaction in a blended e-learning system environment. Yeh, R. K. J., & Teng, J. T. C. (2011). Extended concep-
Computers & Education, 55(1), 155–164. doi:10.1016/j. tualization of perceived usefulness: Empirical test in the
compedu.2009.12.012 context of information system use continuance. Behaviour
& Information Technology, 30(2), 1–16. doi:10.1080/01
Wyatt, J. (2002). ERP in the consumer-driven marketplace.
44929X.2010.517272
Health Management Technology, 23(7), 24.
Yen, H. R., Li, E. Y., & Niehoff, B. P. (2008). Do organi-
Xiao, L., & Dasgupta, S. (2002). Measurement of user
zational citizenship behaviors lead to Information System
satisfaction with Web-based Information Systems: An
success? Testing the mediation effects of integration
empirical study. Paper presented at the 2002 Americas
climate and project management. Information & Man-
Conference on Information Systems, Dallas, TX.
agement, 45(6), 394–402. doi:10.1016/j.im.2008.04.004
Xiao, L., & Dasgupta, S. (Eds.). (2005). User satisfaction
Yin, R. K. (2009). Case study research: Design and
with Web portals: An empirical study. Hershey, PA: Idea
methods (4th ed.). SAGE Publications.
Group Publishing.
Yixin, D., & Bhattacharya, K. (2008). Estimating business
Xirogiannis, G., & Glykas, M. (2007). Intelligent modeling
value of IT services through process complexity analysis.
of e-business maturity. Expert Systems with Applications,
IEEE Network Operations and Management Symposium
32(2), 687–702. doi:10.1016/j.eswa.2006.01.042
(pp. 208-215).
Xu, S. (2007, 17-19 December). An accurate model of
Yi, Y. M., Fiedler, K. D., & Park, J. S. (2006). Understand-
software reliability. Paper presented at the 13th IEEE
ing the role of individual innovativeness in the acceptance
International Symposium on Pacific Rim Dependable
of IT-based innovativeness: Comparative analyses of
Computing.
models and measures. Decision Sciences, 37(3), 393–426.
Yamada, S., Osaki, S., & Tanio, Y. (1992). Software doi:10.1111/j.1540-5414.2006.00132.x
reliability measurement and assessment methods dur-
Yoo, B., & Donthu, N. (2001). Developing a scale to
ing operation phase and their comparisons. Systems
measure the perceived quality of an Internet shopping
and Computers in Japan, 23(7), 23–34. doi:10.1002/
site (SITEQUAL). Quarterly Journal of Electronic Com-
scj.4690230703
merce, 2(1), 31–47.
Yang, Z. (2001). Consumer perceptions of service quality
Yoo, Y., & Alavi, M. (2001). Media and group cohesion:
in Internet-based electronic commerce. Paper presented
Relative influences on social presence, task, participation,
at the The 30th EMAC Conference, Bergen.
and group consensus. Management Information Systems
Quarterly, 25(3), 371–390. doi:10.2307/3250922
466
Youngberg, E., Olsen, D., & Hauser, K. (2009). Determi- Zhang, P., & Von Dran, G. (2002). User expectations and
nants of professionally autonomous end user acceptance rankings of quality factors in different website domains.
in an enterprise resource planning system environment. International Journal of Electronic Commerce, 6(2), 9–33.
International Journal of Information Management, 29(2),
Zhi-Yong, G., Jian-Min, G., Fu-Min, C., & Lin, G. (2009).
138–144. doi:10.1016/j.ijinfomgt.2008.06.001
Estimation method for manufacturing information qual-
Younghwa, L., & Kozar, K. A. (2006). Investigating the ity based on material constraint relationship. Computer
effect of website quality on e-business success: An analytic Integrated Manufacturing Systems, 1, 57–72.
hierarchy process (AHP) approach. Decision Support Sys-
Zhuang, Y., & Lederer, A. (2003). An instrument for
tems, 42(3), 1383–1401. doi:10.1016/j.dss.2005.11.005
measuring the business benefits of e-commerce retail-
Yourdon, E. (1989). Modern structured analysis. Engle- ing. International Journal of Electronic Commerce,
wood Cliffs, NJ: Prentice-Hall. 7(3), 65–99.
Yusof, M., Papazafeiropoulou, A., Paul, R., & Stergiou- Zhu, K. (2004). The complementarity of Information
las, L. (2008). Investigating evaluation frameworks for Technology infrastructure and e-commerce capability: A
health Information Systems. International Journal of resource-based assessment of their business value. Journal
Medical Informatics, 77(6), 377–385. doi:10.1016/j. of Management Information Systems, 21(1), 167–202.
Zhu, K., & Kraemer, K. (2003). E-commerce metrics
Zadeh, L. A. (1965). Fuzzy sets. Information and Control, for Net-enhanced organizations: Assessing the value of
8, 338–353. doi:10.1016/S0019-9958(65)90241-X e-commerce to firm performance in the manufacturing
sector. Information Systems Research, 13(3), 275–295.
Zeithaml, V. A. (2002). Service excellence in electronic
doi:10.1287/isre.13.3.275.82
channels. Managing Service Quality, 12(3), 135–138.
doi:10.1108/09604520210429187 Zmud, R. (1983). The effectiveness of external informa-
tion channels in facilitating innovation within software
Zeithaml, V., Parasuraman, A., & Malhotra, A. (2000).
development groups. Management Information Systems
A conceptual framework for understanding e-service
Quarterly, 7(2), 43–58. doi:10.2307/248912
quality: Implications for future reseach and managerial
practice. Cambridge, MA: Marketing Science Institute. Zott, C., Amit, R., & Donlevy, J. (2000). Strategies
for value creation in e-commerce: Best practice in Eu-
Zeithaml, V., Parasuraman, A., & Malhotra, A. (2002).
rope. European Management Journal, 18(5), 463–475.
An empirical examination of the service quality-value-
doi:10.1016/S0263-2373(00)00036-0
loyalty chain in an electronic channel. Chapel Hill, NC:
University of North Caroline. Zviran, M., Chanan, G., & Itay, A. (2006). User satis-
faction from commercial websites: The effect of design
Zhang, P., Small, R., Von Dran, G., & Barcellos, S. (1999).
and use. Information & Management, 43(2), 157–178.
Websites that satisfy users: A Theoretical framework for
doi:10.1016/j.im.2005.04.002
Web user interface design and evaluation. Paper presented
at the The 32nd Annual Hawaii International Conference Zviran, M., & Erlich, Z. (2003). Measuring IS user sat-
on System Sciences, HICSS-32. isfaction: Review and implications. Communications of
the AIS, 12(5), 81–103.
Zhang, P., von Dran, G. M., Blake, P., & Pipithsuksunt,
V. (2001). Important design features in different website Zviran, M., & Haga, W. J. (1999). Password security:
domains. e-Service Journal, 1(1), 77. An empirical study. Journal of Management Information
Systems, 15(4), 161–184.
Zhang, P., & Von Dran, G. (2000). Satisfiers and dissatisfi-
ers: A two-factor model for website design and evaluation.
Journal of the American Society for Information Science
American Society for Information Science, 51(14), 1253–
1268. doi:10.1002/1097-4571(2000)9999:9999<::AID-
ASI1039>3.0.CO;2-O
467
468
About the Contributors
Zakariya Belkhamza is a Senior Lecturer at the School of Business and Economics, Universiti
Malaysia Sabah, where he teaches Management Information Systems subjects at both undergraduate
and postgraduate level. He holds a PhD in Management and a Master’s degree in Business. Beside his
teaching experience, he is also involved in many research and consultancy activities in Malaysia, Thai-
land, Turkey, and Algeria. His research interests include IT management, strategic Information Systems,
IS implementation, and IS assessment and evaluation.
Syed Azizi Wafa is a Professor of Management at the School of Business and Economics, Universiti
Malaysia Sabah. Syed has served as the Dean, School of Business and Economics, Universiti Malaysia
Sabah, Director of the Centre for Co-Curriculum and Student Development, UMS, and as Campus
Director, Labuan International Campus, Malaysia. Syed is also the founding member and currently is
the President of the Asian Academy of Applied Business (AAAB). He is actively involved in the con-
sultancy and Executive Development Programs of various companies such as Magellan Sutera Hotel,
Sabah, Penang Seagate Industries, HITACHI Manufacturing, INTEL, OTIS Manufacturing, Permodalan
Nasional Berhad, Sharp-Roxy, MARA, Federation of Malaysian Manufacturers, Franklin Porcelain,
Maxtor Manufacturing, Texchem Group of Companies, Siemens Semiconductor Sdn. Bhd., Public Bank
Bhd., and the Malaysian Airforce Training Academy. His current research interests are in cross-cultural
managerial issues, corporate strategy, corporate culture, corporate ethics, and regional business issues.
***
Ahmed Younis Alsabawy is a PhD candidate in the Faculty of Business and Law, School of Infor-
mation Systems at University of Southern Queensland. Ahmed is a Member of Academic Staff in the
Faculty of Business and Economics at the University of Mosul since 2002. He graduated from the Uni-
versity of Mosul in 1998 with a Bachelor’s degree in Business Administration. He received his Master’s
in Business Administration/Strategic Management from the University of Mosul in 2001. In 2009, he
earned a scholarship from the Iraqi government and joined the University of Southern Queensland to
complete a PhD degree.
Luis Becerra is an Associate Professor at the Universidad Industrial de Santander since 1996 and
Researcher at the Center for Technology and Innovation Management Research – INNOTEC. Prior to
that, he was Industrial Engineer at the Universidad Industrial de Santander, specialist in Technology
Management (1997) and University Education (2003) at the Universidad Industrial de Santander. He holds
MBA from TEC (ITESM) de Monterrey and Universidad Autónoma de Bucaramanga. He is currently
PhD candidate in Engineering in the area of innovation, development, and technology management at
the Universidad Industrial de Santander. He is also a CFO at the financial division at the Universidad
Industrial de Santander and a Coordinator for Latin-America in the SUMA project (Towards Sustainable
Financial Management), funded by the European Commission (ALFA 3) and researcher at the PILA-
Network project (Latin-American Intellectual Property network). His special fields of interest include
technology watching, futures and prospectives, and technology management.
George L. Benwell is Pro-Vice-Chancellor (Commerce), Professor, and Dean of the Business School
at the University of Otago. After working as a surveyor, consultant, and academic at University of
Melbourne, Benwell joined Otago’s Information Science Department as a Senior Lecturer in 1990. He
led the Department from 2001-2006 and served as the School’s Dean of Research. Between 1996 and
1999, he was the University’s Academic Director of Research. In 2007, he was appointed Dean of the
School. His research and teaching are mainly focused on spatial information processing and analysis,
and land and health-related Information Systems. He holds a Bachelor’s of Surveying from the Uni-
versity of Melbourne, MPhil from City University London, and a PhD from Melbourne. George has
been involved with research in Australia, New Zealand, Peru, Chile, Canada, Fiji, England, America,
Malaysia, and Tonga, and has published over 250 research papers and articles. In 2001, he received the
GJ Thornton-Smith Memorial Medal as a distinguished graduate of Melbourne University’s School of
Engineering Department of Geomatics.
Samo Bobek is a Professor of Information Systems at the School of Economics and Business at
University of Maribor. He teaches courses in undergraduate level (Introduction to Information Systems,
Information Systems in Finance and Banking), Masters of Science program (Information Systems Man-
agement, Management Information Systems, Information Systems in Service Organizations) and MBA
program (Information Management). His research areas are strategic Information Systems planning,
information management, and banking technology. He has published several books in Slovene language.
His bibliography includes more than 200 articles, conference papers, research reports, et cetera. Bobek
is head of Information and Organization Systems Department. He also acts as consultant to several
corporations, banks, and insurance firms in Slovenia.
Jaime Camacho is a Titular Professor at the Universidad Industrial de Santander since 1993 and
Director of the Center for Technology and Innovation Management Research – INNOTEC. He is also
Chairman at the Universidad Industrial de Santander. His special fields of interest include technology
management, science parks, university education, and innovation management. He is and Industrial
Engineer and holds Master’s of Science in Informatics from the Universidad Industrial de Santander,
specializing in production management and quality and technology. He also holds a PhD in Industrial
Engineering from the Universidad Politécnica de Cataluña, Diploma studies in Strategic Management
from the Eurocentro, innovation paradigm, technological strategy from the Scuola Superiore di Studi
Universitari e di Perfezionamento Sant’Anna.
469
Andrea Carugati holds a PhD from the Technical University of Denmark (Lyngby). He is Associate
Professor at Aarhus School of Business, Aarhus, Denmark, and IESEG School of Management at Lille,
France. His research focuses on the behaviors caused by technology and the alternative, deviating uses
of technology in organizations. His work has been published in the European Journal of Information
Systems, and Database for Advances in Information.
Aileen Cater-Steel is Associate Dean of Research in the Faculty of Business and Law, and Associate
Professor in Information Systems at University of Southern Queensland. She is the lead Chief Investigator
in two Australian Research Council Linkage Grants in the area of IT Service Management. Aileen has
presented at many international conferences and has published three edited books: Women in Engineering,
Science and Technology, IT Governance and Service Management, and Information Systems Research.
Her work has been published in many international journals including the Communications of the ACM,
Information and Software Technology, International Journal of IT Standards and Standardization Re-
search. Her research interests include IT service management and IT governance. She was awarded the
ACPHIS medal for her PhD thesis. Aileen has certificates in ITIL, ISO/IEC20000, ISO/IEC15504, and
CobiT. Prior to her academic career, Aileen worked in local government for 17 years and also in private
industry as IT support manager and IT manager.
Kenneth R. Deans is originally a Scientist, Associate Professor, and Head of the Department of
Marketing within the internationally ranked School of Business at the University of Otago, Dunedin,
New Zealand. He lectures undergraduate, postgraduate, MBA, and executives in principles of marketing,
marketing strategy, Internet marketing, and services marketing. His research, in collaboration with overseas
researchers has lead to over 70 publications in journals, books, and refereed conference proceedings as
well as a numerous seminars to businesses and organisations. He is co-author (Kotler, Brown, Burton,
Deans, & Armstrong) of the highly successful “Marketing” textbook. Ken has worked for and with a
number of SMEs as well as blue-chip companies such as Scottish & Newcastle and Pilkington. He is past
Treasurer and currently President of the Australian & New Zealand Marketing Academy (ANZMAC).
Dirk Deschoolmeester is Professor in MIS/BPM at Vlerick Management School and University of

Ghent’s Faculty of Economics and Business Administration, Dept Management, Innovation, and Entre-
preneurship (Ghent University). Since his retirement in 2006, he is still serving both institutes for their
research. He obtained his doctorate in 1976 from Faculty of Engineering Sciences (Ghent University)
on a subject in the domain of methods for evaluating Information Systems related to the implementation
of enterprise systems. He was a consultant for several companies on methods for sourcing software and
IT-infrastructure. He also served several companies on formulating computer contracts and operating as
an expert for legal courts on IT litigation. In 2003, he founded at Vlerick Management School a Research
Centre on “Business Process Management” (see www.vlerick.com/BPM). He is also Chairman of Belgian
Computer Society, called FBVI-FAIB, and represents Belgium as a Member Society in IFIP and CEPIS.
Jan Devos is an Assistant Professor of Information Systems at the Ghent University Association,
Department Howest, where he teaches and lectures in IT management, e-business, IT security, and cloud
computing. He holds a Master’s degree in Electronics and Computer Science and MBA from the Vlerick
Leuven Gent Management School. Before his academic career, Jan Devos worked as an IT practitioner
470
specialized in IT governance and IT litigation. He conducted more than seventy expert opinions on
litigation of IS failures. He has a broad experience in Information Systems management and project
management. His current research interests are IT governance in SME’s, cloud computing, e-business,
and IT security. He has published several articles on IT and SMEs and was a speaker at international
academic and business conferences. He is co-author of the Cobit QuickStart methodology.
Kimberley D. Dunkerley is a Lecturer of Cybersecurity in the Strom Thurmond Institute of Gov-

ernment and Public Affairs, Clemson University. She received a PhD in Information Systems from
Nova Southeastern University in 2011 with a doctoral focus of information security success within
organizations. Her research interests include information security and the success and measurement of
organizational information security initiatives.
Francis Gacenga is a PhD candidate at University of Southern Queensland, developing a performance

measurement framework for ITSM in an Australian Research Council linkage project. The research is
supported by Queensland Health and itSMF Australia. Francis has presented at international conferences
and published articles in academic and industry journals. His paper was awarded best paper at GITMA
2011. He has worked internationally in information systems development, support, training and manage-
ment as a systems administrator and computer programmer at the local and state government level and
as a Lecturer and Tutor at the tertiary level. In 2002, Francis was awarded a gold medal by the Institute
for the Management of Information Systems (IMIS) UK. His current research interests include IT service
management, service science, performance measurement, content analysis, and design science. Francis
has served as a reviewer in a number of international information systems journals and conferences. He
has certificates in ITIL and ISO/IEC 20000.
Ces J. Gelderman is Associate Professor of Marketing and Purchasing Management at the Open
University of the Netherlands. He studied Business Economics at the State University of Groningen
where he graduated (cum laude) in the field of Marketing and Marketing Research. His PhD research
project was on the possibilities of a portfolio approach to purchasing and supplier strategies at the Uni-
versity of Eindhoven. His research interests include buyer-supplier relationships, (safeguarding against)
opportunistic behavior, power and dependence, account management, public procurement, and purchas-
ing portfolio management. Gelderman has published in many academic journals, including Industrial
Marketing Management, the British Journal of Management, the Journal of Common Market Studies,
the International Journal of Public Sector Management, the Journal of Supply Chain Management, and
the Journal of Purchasing and Supply Management.
Ahmad Ghandour completed his BSc in Electronic Engineering from Bath University, UK, and
MSc in Management Information Systems from The Amman Arab University, Jordan. He also holds a
PhD from University of Otago, New Zealand. He has worked in both public and private sectors where
he was responsible for overseeing and maintaining high profile projects. His primary area of research
is assessment of website effectiveness and value.
471
Frank Goethals holds a PhD in Applied Economics from the K.U. Leuven in Belgium. He is currently
Assistant Professor of IS and head of the IS track at the IESEG School of Management in Lille, France.
His research focuses on B2B and B2C e-business. Frank has published in numerous journals including
Data and Knowledge Engineering, IEEE Software, Information Systems Frontiers, and Database for
Advances in Information Systems.
Anna Marie Balling Høstgaard has a solid experience within the healthcare sector as she has been
working as a dentist for a number of years before her employment at Aalborg University. She holds a
Master of Public Health (MPH) and a PhD in Technology Planning, Assessment, and Evaluation, and
she is researching and teaching within both fields. Her fields of specialization are health Information
Systems evaluation, participatory approaches to organizational and technological change of health in-
formatics, public health assessments, and evaluation.
Kieren Jamieson has a PhD in Decision Making in the adoption and selection of Information Sys-
tems. His research focuses on the elements of technology adoption that are subject to human bias and
irrational cogitative processes. Kieren has an industry background with experience in a range of tech-
nical, operational, and managerial roles in large Information Systems implementations. His synthesis
of research and industry experience underpins his interest in understanding the antecedents of positive
and negative organisational outcomes from technology implementations. He is a Senior Lecturer at
CQUniversity, Australia, and teaches a wide range of technical and IS courses, with a special interest in
infrastructure planning and management.
Stephan Kronbichler studied business informatics; he holds both Master’s degree in Business Infor-
matcis from the University of Applied Sciences Kufstein and Master’s degree in Business Law from the
University of Saarbrücken. Since 2007, he is a Junior Researcher and a PhD candidate at the University
for Healthcare Sciences, Medical Informatics and Technology (UMIT), Hall/Tyrol (Austria). His research
interests include ERP-systems and Information Systems especially in the healthcare environment. He
works as a SAP consultant and his working focus is on the logistic modules.
Rob J. Kusters obtained his Master’s degree in Econometrics from the Catholic University of
Brabant in 1982 and PhD in Operations Management at Eindhoven University of Technology in 1988.
He is Professor of ‘ICT and Business Processes’ at the Dutch Open University in Heerlen where he is
responsible for the Master Program ‘Business Process Management and IT’. He is also an Associate
Professor of ‘IT Enabled Business Process Redesign’ at Eindhoven University of Technology where he
is responsible for a section of the program in management engineering. He published over 100 papers
in international journals and conference proceedings and co-authored six books. Research focuses on
project and process performance, enterprise modelling, software quality, and software management.
Aurélie Leclercq-Vandelannoitte obtained her PhD in Management from Paris-Dauphine University

in 2008 and is Assistant Professor at IESEG School of Management, France. Her current research interests
include the interplay between IS, people and organizations, change management, and IS adoption. She
has published her research at ICIS, ECIS, AMCIS, as well as in Organization Studies, The Database for
Advances in Information Systems, and Systèmes d’Information et Management.
472
Cristina Lopez received her both MSc and PhD degrees in Business from University of Seville,
Spain. Currently, she is an Associate Professor at the School of Engineering at the University Pablo of
Olavide. In addition, she participates in various research projects at a national level. Her papers have
been published in Journal of Systems and Software, IEEE Transactions on Software Engineering, Journal
of Applied Research and Technology, among others. Her research interests include enterprise systems,
software maintenance and project risk management. She is an active participant in conferences in these
areas.
Govindan Marthandan is the Head of Department for Information Technology Management,

Quantitative Methods and Law Units in Faculty of Management Multimedia University Malaysia. He
received his Bachelor of Engineering, Master of Engineering, and PhD from University of Madras India,
and Master of Business Administration (Operations Management) from IGNOU, India. His wide area
of interest includes the Information System development, IS evaluation, ICT applications, Web data
mining, e-commerce, supply chain management, and operations management. He has a vast experience
working with IT industry, government, and academic institutions.
Hugo Martinez is an Electronic Engineer and holds Master of Science in Electronic Engineering
from the Universidad Industrial de Santander. Currently, he is a PhD candidate in Engineering in the area
of innovation, development, and technology management at the Universidad Industrial de Santander.
He is also ITIL and COBIT certified. Since 2009, he is a researcher at the Center for Technology and
Innovation Management Research – INNOTEC. His special fields of interest include innovation man-
agement, technology management and technology screening, open innovation, and relative absorptive
capacity of knowledge in strategic alliances.
Simon McGinnes is a Lecturer and Director of the Information Systems Programme in the School
of Computer Science and Statistics at Trinity College Dublin, Ireland. McGinnes obtained his Doctor-
ate in Information Systems at the London School of Economics. He has many years’ experience in the
Information Technology industry as a Consultant, Project Manager, Analyst, and Software Developer,
and has worked as an academic at universities in Australia, the UK, and Ireland. McGinnes has founded
IT companies in the USA and Ireland. His current research interests include Information Systems de-
velopment, agile outsourcing, crowdsourcing, conceptual modelling, and the application of psychology
to Information Systems.
Herwig Ostermann is the Director of the Division for Health Policy, Administration and Law at the
Department of Public Health and Health Technology Assessment at the University for Health Sciences,
Medical Informatics, and Technology, Hall/Tyrol (Austria). His research interests encompass the field of
health policy and administration, especially focusing on health care delivery and service provisioning.
He received a Master‘s degree in International Business Administration from Innsbruck University (Aus-
tria), a Master degree in Health Sciences and a PhD in Health Sciences from University for Healthcare
Sciences, Medical Informatics and Technology (UMIT), Hall/Tyrol (Austria).
473
Angela Perego is a Lecturer at SDA Bocconi, School of Management. She obtained a PhD in Infor-
mation Systems from Luiss University of Rome and a PhD in Sciences de Gestion from Paris-Dauphine
University. She is co-Director of SDA Bocconi Observatory on Business Intelligence and Business
Performance Management. She is also Member of the Faculty of Masters in Project and Information
Management and in Energy and Management of Transport, Logistics, and Infrastructure at Bocconi Uni-
versity. Main research interests include IS performance measurement and management, decision making
process and IT, data warehouse and business intelligence systems, customer relationship management
systems, knowledge management, and text mining.
Pietro Previtali is an Assistant Professor of Business Organization at University of Pavia. He ob-

tained his PhD in Management and Business Administration from Bocconi University. He teaches and
conducts research on the managerial issues surrounding the introduction of Information Technologies
in public agencies and private companies
O. Tolga Pusatli is an Assistant Professor at the Department of Mathematics and Computer Science
of the Faculty of Arts and Sciences at the Cankaya University Turkey, where he undertakes lecturing
in Information Systems. He also works as a consultant in data and information management fields. His
research scope includes software quality, health informatics, and IT usage in the industry.
Brian Regan is an IT academic in the School of Design Communication and IT at the University
of Newcastle Australia, after a career in software development in industry. He has served in a number
of administrative roles including Head of School and Assistant Dean International for his faculty. His
research interests are in health informatics, software development methods, and the social impacts of IT.
Jose Luis Salmeron is a Professor at the School of Engineering at the University Pablo of Olavide.
He has co-authored eight books and has served as a Visiting Scholar and Researcher at Texas Tech
University and several Spanish universities. At the present, he is leading research projects at a national
and international level. His papers have been published in IEEE Transactions on Software Engineering,
Expert Systems with Applications, International Journal of Approximate Reasoning, Communications of
the ACM, Journal of Systems and Software, Knowledge-Based Systems, Computer Standards & Inter-
faces, Information & Management, Interacting with Computers, Technovation, Industrial Management
& Data Systems, European Journal of Operational Research, Scientometrics, and so on. He is actively
conducting research in fuzzy cognitive maps, computational intelligence, and complex adaptive systems.
Lucie Sermanet holds MSc in Management (Information Systems) from IESEG School of Manage-
ment in Lille. She is currently working on the Information System Architecture group at PSA, Peugeot
Citroën. She has done research on website design and more specifically on the impact of 3D elements
and dynamic elements on websites.
Jeffrey Soar is a Researcher in intelligent technologies to assist ageing and aged care. He is the
Convener of the Queensland Smart Home Initiative, founder and Interim CEO of the SAIL (Smart Age-
ing & Independent Living) consortium, and founder and Secretary of Aged Care Informatics Australia.
He came to academia from a career as a senior executive in health services including state and national
474
departments of health in Australia and a regional health authority and the Ministry of Health in New
Zealand. Soar has published over 120 peer-reviewed journal articles, conference papers, and book chap-
ters. He has been awarded over 30 grants and research commissions.
Simona Sternad is Senior Lecturer of Information Systems at the School of Economics and Busi-
ness at the University of Maribor where she received a Master of Science degree in 2005. Her research
areas are business process reengineering, ERP solutions and e-business solutions, implementation of
ERP solutions, and maintenance of ERP solutions. Her bibliography includes more than 50 articles,
conference papers, research reports, et cetera. She collaborates in several courses in undergraduate
level (E-Business, Introduction to Information Systems, E-Finance and E-Banking, Enterprise Resource
Planning Solutions, E-Business Information Systems) and several courses in Master of Science Program
(Business Information Solutions, Strategic Management Issues in E-Business). She also cooperates with
some companies in Slovenia as a consultant.
Wui-Gee Tan’s IT career spans over 30 years. He spent the first half with multinational corpora-
tions in various IT roles and the second, in academia, involved in teaching, consulting, and researching
IT. He started his career as a Systems Analyst and later worked as a Project Manager and IT Auditor
before moving to a Systems Development Management role. Prior to joining the University of Southern
Queensland in 2004 Wui-Gee was a Program Director At the National University of Singapore. Wui-
Gee received his Doctorate from the Queensland University of Technology. He is a certified member of
the Project Management Institute and has certificates in ITIL, ISO/IEC 20000, and CobiT. His research
interests are: IT service management, IT governance, and IT project management.
Chun Meng Tang received his BSc. and MSc. degrees in Business Information Systems and MBA
from Utah State University, USA, and a PhD degree from Multimedia University, Malaysia. His major
areas of research include IS evaluation, IS strategy, and IS applications in the 21st century organisations.
One of the research considerations is to work closely with the industry to conduct studies that add to the
understanding in the field of information systems.
Gurvirender Tejay is an Assistant Professor of Information Security at the Graduate School of

Computer and Information Sciences, Nova Southeastern University, USA. He received a PhD in Busi-
ness from Virginia Commonwealth University USA in 2008. His doctoral work investigated strategic
information systems security initiatives in organizations. He also holds MSc in Computer Science from
the University of Chicago, and M.A. in Economics from the University of Wisconsin – Milwaukee. His
research interests include information security management, knowledge management, and technological
and organizational change.
Mark Toleman is Professor and Head of the School of Information Systems, and Chair, Academic
Board at the University of Southern Queensland since mid-2006. From 2003-2005 he was Director of the
e-Business Advisory and Research Centre at USQ, and from 2000-2006 he was Deputy Chair, Academic
Board. Professor Toleman has a PhD in Computer Science from the University of Queensland and an
MSc in Mathematics from James Cook University. He is currently President of the Australian Council
of Professors and Heads of Information Systems. Toleman has published over 130 peer reviewed articles
475
in areas ranging from Information Systems development methodologies to human-computer interaction

to agricultural science. His current research interests include community support Information Systems,
IT service management, IT governance, and systems development methodologies.
Hendrik Van Landeghem is Professor at Ghent University, following a 10-year career in industry
that started at P&G Strombeek. He was Head of the Department of Industrial Management between
1992 and 2010. He is an expert on logistic systems and their application in business processes. His
research focuses on logistic ICT systems, benchmarking methods, and in-plant logistic systems. He
also teaches and performs research in lean manufacturing. Currently, he is involved in several research
projects within the automotive industry, as well as in projects that implement Lean concepts in small
and medium sized enterprises (SME). Van Landeghem is Fellow of the European Academy of Industrial
Management, Senior Member of the Institute of Industrial Engineering and member of the European
Operations Management Association. In 2006, he was elected a Fellow of the World Confederation of
Productivity Science. He is also a fellow of the Flemish Academy of Belgium for Arts and Sciences.
476
477
Index
A compound model 89
construct reliability 31
agile methods 160 control vs. flexibility 160-161
sprints 160 corporate governance 84
timeboxes 160 critical success chains (CSC) 121
alignment issues 4 critical success factors (CSFs) 3, 121, 185, 212, 235
allocation of tasks 160, 163 in ERP projects 212
amensalism 8 customer perspective 46
analytic hierarchy process (AHP) 121, 289-290,
310, 350 D
appraisal 47
augmented FCM approach 124 Davis' TAM 148
decay 140
B DeLone and McLean Approach 299, 301
DeLone & McLean IS success model (D&M) 80-81,
Balanced Scorecard (BSC) 40, 43, 60, 103, 220, 388 84, 90, 214, 277, 343
business process perspective 46 design phase 391
customer perspective 46 development vs. maintenance 161, 165
financial perspective 45 Diagnosis Related Groups (DRG) 207
learning and growth perspective 47 Document management system (DMS) 209
based basically on frequency counting 64
benchmark approach 207, 221-222 E
bespoke system 141
best-of-breed approaches 208 ease of use 28
bibliometrics 64 e-commerce evaluation 234, 272
bibliometric techniques 63 E-Learning Quality Approach 301
business change vs. technology 161 e-learning systems 294
business process perspective 46 Electronic health record (EHR) 208, 319, 338
business process re-engineering (BPR) 3, 11, 206 engineering project 159
enterprise resource planning (ERP) systems 2, 41,
C 179-180, 200, 202
in healthcare 209
change management metrics 145 enterprise system implementations 6
change requests 139 enterprise systems (ES) 2
characterisation of work 161, 165 benefits 4
clinical information system 209 ERP acceptance model 183
Cobit framework 80, 92 ERP projects
Cognitive Maps (CM) 123 critical success factors 212
commensalism 8 ETHICS method 333
Index
ethnography 9 IS for organisational effectiveness (ISOE) model

EUPHIT method 319 39-40, 49, 51
strengths and limitations 333 background 41
evolutionary vs. revolutionary change 161, 165 IS performance management systems (PMS) 386
Extended ERP Systems Success Measurement IS PMS Project
Model 207, 216 design phase 391
external factors - See personal characteristics and objectives and boundaries 390
information literacy (PCIL). refining phase 393
IS product aging 143
F IS security
checklists and standards 344
FCM development 124 effectiveness aspects of 349
FCM inference 125 success model 342, 352, 356
FCM model technical aspects 344
development of 126 understanding success of 351
dynamic behaviour of 127 utility of success model 356
financial perspective 45 IS spending 40
FITT Model 207, 220-221 IS success
flexibility vs. control 160-161 conceptual model 27
form of contract 160, 164 IS success models 342
Fuzzy Cognitive Maps (FCM) 122 IS success research stream has been studied 65
IT governance 83
G IT/IS projects 121, 126, 131
Gable et al. (2003) Model 207, 214 IT service management (ITSM)
goodness-of-fit (GoF) coefficient 195 and Performance Measurement Frameworks
106
H performance 102
proposed model 110
healthcare industry
ERP system implementation 209 L
Healthcare Information Systems Success Measure-
ment Models 220 Laboratory information management system (LIMS)
health information technology (HIT) success 318- 209
319 learning and growth perspective 47
Long Term versus Short Term Studies 7
I
M
implementation quality 29
information systems (IS) 23-24, 35, 39-40, 60, 80- maintenance vs. development 161, 165
81, 103, 120-121, 137-138, 158-159, 263, 277, managerial perception 42
341, 385, 387 Markus and Tanis 218
success measurements 1 Mason model 343
information system success 297 Master patient index (MPI) 208
in-sourcing vs. outsourcing 160, 163 multidimensionality 243
interfaces 209 MUST method 333
intranet implementation 368 mutualism 8
IS champion 30
IS development 141
competing views 159
IS evaluation method 40
478
Index
O S
operating sponsor 30 Science Citation Index (SCI) databases 63
Operational Research (OR) 159 scope of work 161
organisational effectiveness 42 SCOT Analysis synthesis 332
organisational issues 4 Shannon and Weaver model 342
organisational level 41 Short Term versus Long Term Studies 7
organisational requirements 141 small- and medium-sized enterprises (SMEs) 81
original conceptual model 44 Social Construction of Technology Theory (SCOT)
outsourcinv vs. in-sourcing 160, 163 321, 323
overlapping vs. standalone projects 161, 166 Social Sciences Citation Index (SSCI) databases 63
owner perspective 257 Socio-Technical-Carrier-of-Technology Theory
(STCT) 321
P software engineering 145
software maturity index (SMI) 144
parasitic relationship 8 sprints 160
partial least squares (PLS) 49, 188, 202 stakeholder theory 43
payoffs 256 standalone vs. overlapping projects 161, 166
perceived ease of use (PEOU) 87, 181 steady-state development 168
perceived usefulness (PU) 87, 181 steady-state framework 170
Performance Measurement Frameworks structural equation modeling (SEM) 188, 202
and IT service management (ITSM) 106 success dimensions 240
personal characteristics and information literacy success measurement models 214
(PCIL) 185, 187, 197, 202 surrogate measure 23-24
phasing of delivery 161, 165 system and technological characteristics (STC) 185,
Picture archiving and communication system 187, 197, 202
(PACS) 209 system flexibility 28
process oriented approach 258 system integration 28
protection vs. partnership 160, 164 system quality 28
system specialisation 141
R system success 24
Radiological information system (RIS) 208 system usage 27
refining phase 393
related to co-occurrence analysis involving 64 T
relationship alignment 5 Tanis and Markus 218
relationships between projects 161, 166 Task-Technology Fit (TTF) 218
Research Key Topics: Keywords Analysis 68 technical security controls 344
resource based theory 257 technology acceptance approach 297, 300
Resource-Based View (RBV) 257 technology acceptance model (TAM) 28, 80-81, 84,
response time 28 87, 121, 142, 179, 181, 220, 297-298
revolutionary vs. evolutionary change 161, 165 Technology-Carrier Analysis 327-328
Rigour of Context 243 technology vs. business change 161
risk analysis 346 theory of planned behavior (TPB) 80-81, 84, 86
theory of reasoned action (TRA) 181, 298
timeboxes 160
top management support 29
transaction cost economics (TCE) 88
transaction cost economy (TCE) model 80-81
479
Index
U W
underlying management philosophy 160-161 website evaluation 236
user feedback 142 website investment 261
user involvement 29 website payoff 264
user involvement approach 299 website success measurement models 238
user perspective 257 website usage 263
user satisfaction 27 Website Usage Metrics 255-256
user satisfaction approach 299, 301 website visibility 263
workload reduction 320
V
value delivery 255
vanilla ERP 206
vendor lock-in 13
via the VOS -visualization of similarities technique
70
480

(Premier Reference Source) Zakariya Belkhamza, Zakariya Belkhamza, Syed Azizi Wafa - Measuring Organizational Information Systems Success - New Technologies and Practices (2012, IGI Global) PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

(Premier Reference Source) Zakariya Belkhamza, Zakariya Belkhamza, Syed Azizi Wafa - Measuring Organizational Information Systems Success - New Technologies and Practices (2012, IGI Global) PDF

Uploaded by

Copyright:

Available Formats

Measuring Organizational

Syed Azizi Wafa

Published in the United States of America by

Library of Congress Cataloging-in-Publication Data

British Cataloguing in Publication Data

Compilation of References................................................................................................................ 403

About the Contributors..................................................................................................................... 468

Compilation of References................................................................................................................ 403

About the Contributors..................................................................................................................... 468

Benbasat, I. (2001). Editorial notes. Information Systems Research, 12(2), iii–iv.

Syed Azizi Wafa

Syed Azizi Wafa

INTRODUCTION 2002; Standish Group, 1995) with the desired

Table 1. Symbiotic success categorisations

Relationship Effect on Information System Effect on Organisation

While a justification for the implementation was Reliance on Contractors

A key motivation for the move to an enterprise SUCCESS OR FAILURE?

INTRODUCTION puzzled many researchers and practitioners for

Table 1. Some recent studies aimed at explaining IS success

Source IS success Explanatory variables

Figure 1. Research design

Figure 2. Conceptual model of IS success

Most studies include user satisfaction and

Table 2. Explanatory model for IS success

Dependent Variables Cronbach’s alphas Means Standard

Table 4. Model fit: Results of the multiple regression analysis

Dependent variable IS Success

Analysis of variance d.f. Sum of squares Mean squares

Variables Unstandardized coefficients Standardized t

ADDITIONAL READING Urbach, N., Smolnik, S., & Riempp, G. (2008). A

Chun Meng Tang

INTRODUCTION Koch (2002) highlight that an evaluation model

Figure 1. The conceptual model

to a dimension of organisational performance unidimensional approach to measure IS success.

benign-positive, stressful. Irrelevant appraisals Content refers to the performance outcomes an

existence of higher-order factor structures7. The construct of IS-enabled organisational effective-

Table 1. Newly identified first-order and second-order constructs

Second-order construct First-order construct Item description

Figure 2. The IS for organisational effectiveness (ISOE) model

INTRODUCTION fast in recent decades and has impacted a variety

Figure 1. Distribution of published articles retrieved from ISI Web of Science

Country # Records Institution # Records

Table 2. Top 15 list of most productive and most cited journals

Most Productive Journal # Records Most Cited Journal # Records

Table 3. Top 15 list of most productive and more cited authors

Most Productive Authors # Records Most Cited Authors # Records

Table 4. The most used topics by country

Knowledge Project Critical Success Information

Figure 2. Knowledge structure of the IS success stream research

Figure 3. Relations among the main connected keywords

Figure 4. Tendency of the five most studied themes

ADDITIONAL READING Ho, Y. S. (2007). Bibliometric analysis of absorp-

Hendrik Van Landeghem

INTRODUCTION consistent with the existing IS network established

Figure 2. Nomological network of theory of planned behavior (Ajzen, 1991)

Figure 4. Transaction cost economics model for SMEs

Figure 6. The nomological IS Net (Benbasat & Zmud, 2003)

Table 1. The contribution of the theories to the explanation of the IS Net

Figure 7. Global schema of the framework based on the IS Net

Figure 8. Framework to explain IS Success in SMEs

EMPIRICAL VALIDATION find any contradictions in the way Cobit builds

AI1: Identify Automated Solutions

pendence on external IT expertise, are taken into REFERENCES