Project

Management
Journal
The Professional Journal of the Project Management Institute
Volume 33, Number 3 | September 2002

PAPERS

5 Project Management in the Information Systems and Information

Technologies Industries
Francis Hartman and Rafi A. Ashrafi

16 Scheduling Programs With Repetitive Projects Using Composite Learning

Curve Approximations
Jean-Pierre Amor

30 A Measure of Software Development Risk

James J. Jiang, Gary Klein, and T. Selwyn Ellis

42 A Hybrid Intelligent System to Facilitate Information System Project

Management Activities
Hamid R. Nemati, Dewey W. Todd, and Paul D. Brown

53 The Fall of the Firefly: An Assessment of a Failed Project Strategy

Bud Baker

58 The Impact of the Project Manager on Project Management Planning

Processes
Shlomo Globerson and Ofer Zwikael
 Project
Management
Journal
The Professional Journal of the Project Management Institute
Volume 33, Number 3 | September 2002

3 From the Editor

Parviz F. Rad, PhD, PMP

4 Research Report
Harry Stefanou, PhD

5 Project Management in the Information Systems and Information

Technologies Industries
Francis Hartman and Rafi A. Ashrafi

16 Scheduling Programs With Repetitive Projects Using Composite Learning

Curve Approximations
Jean-Pierre Amor

30 A Measure of Software Development Risk

James J. Jiang, Gary Klein, and T. Selwyn Ellis

42 A Hybrid Intelligent System to Facilitate Information System Project

Management Activities
Hamid R. Nemati, Dewey W. Todd, and Paul D. Brown

53 The Fall of the Firefly: An Assessment of a Failed Project Strategy

Bud Baker

58 The Impact of the Project Manager on Project Management Planning

Processes
Shlomo Globerson and Ofer Zwikael

65 Cover to Cover—Book Reviews

Kenneth H. Rose, PMP

69 Guidelines for PMJ Book Reviews

70 Calendar

71 Notes for Authors

72 Advertisers Index
 Project Management Journal
PROJECT MANAGEMENT
JOURNAL EDITOR
Parviz F. Rad, PhD, PMP
Stevens Institute of Technology
Book Review Editor
Kenneth H. Rose, PMP
PMI PUBLISHING STAFF
Publisher
Linda Cherry; linda.cherry@pmi.org
Editor in Chief
Dan Goldfischer; dan.goldfischer@pmi.org
Publishing Support Specialist
Natasha Pollard; natasha.pollard@pmi.org
EDITORIAL REVIEW BOARD
Frank T. Anbari, PMP, The George Washington
University; Bud Baker, Wright State University; Rick
Bilbro, The Innova Group, Inc.; David Christensen,
Cedar City, UT; David Cleland, University of Pittsburgh;
Helen S. Cooke, Cooke & Cooke; Dan H. Cooprider,
Creative People Management; Jeffrey Covin, Indiana
University; Steven V. DelGrosso, IBM; Deborah Fisher,
University of New Mexico; Vipul Gupta, Saint Joseph’s
University; Soren Hansen, Pennoni Associates of Ohio;
Kenneth O. Hartley, PMP, Parsons Brinckerhoff; Francis
T. Hartman, The University of Calgary; Gary C.
Humphreys, Humphreys & Associates; Lewis Ireland,
PMP, Project Technologies Corp.; Peter Kapsales,
Bellcore; Lee R. Lambert, PMP, Lambert Consulting
Group, Inc.; Alexander Laufer, Technion-Israel
Inst. of Technology; Bill Leban, Keller Graduate
School of Management; Robert Loo, The University of
Lethbridge; Kenneth G. Merkel, PMP, University
of Nebraska; James J. O’Brien, PMP, O’Brien-Kreitzberg
& Assoc.; Michael D. Okrent, Agilent Technologies
Inc.; John B. Phillips, Engineering Development
Institute; Peggy C. Poindexter, Great Falls, VA; Tzvi Raz,
Tel Aviv Unversity; Paul B. Robinson, The Eastern
Management Group; Arnold M. Ruskin, PMP, Claremont
Consulting Group; Avraham Shtub, Tel Aviv University;
Richard L. Sinatra, PMP, Potomac, MD; Larry A. Smith,
Applied Management Associates; Dwight Smith-
Daniels, Arizona State University; James Snyder,
Springfield, PA; Paul Solomon, PMP, B-2 Earned Value
Management Systems; Robert G. Staples, Monroe, VA;
Walter Taylor, Delta Airlines; Charles J. Teplitz, University
of San Diego; Veljko M. Torbica, PMP, Florida
International University; Walter A. Wawruck,
Vancouver, BC, Canada; Itzhak Wirth, Saint John’s
University; Janet K. Yates, San Jose State University
PUBLICATIONS ADVISORY BOARD
Linda C. Cherry, PMI Publisher; Greg Hutchins, Quality
Plus Engineering; Sandy Jenkins, Sanjenco; William V.
Leban Jr., PMP, Keller Graduate School of Management;
Charles J. Teplitz, PMP, University of San Diego School
of Business Administration
© 2002 Project Management Institute, Inc. All rights reserved.
“PMI” is a trade and service mark registered in the United States and other nations; “PMP” and the PMP logo are registered certification marks in the United States and other nations; “PMBOK” is a
trademark registered in the United States and other nations; and the PMI logo, “PM Network”, “Project Management Journal”, “PMI Today”, and “Building professionalism in project management.” are
trademarks of the Project Management Institute, Inc.
2 Project Management Journal
Book Development Editor
Richard Schwartz; richard.schwartz@pmi.org
Permissions Coordinator
Insuk Choe; insuk.choe@pmi.org
Bookstore Administrator
Regina Madonna; regina.madonna@pmi.org
Book Publishing Planner
Danielle Moore; danielle.moore@pmi.org
Administrative Assistant
Dotti Bobst; dotti.bobst@pmi.org
General e-mail:
pmipub@pmi.org
PUBLICATION & MEMBERSHIP
The Project Management Journal (USPS 8756-9728/
02) is published quarterly (March, June, September,
December) by the Project Management Institute. PMJ
is printed in the USA by Cadmus Magazine, Richmond,
VA. Periodical postage paid at Newtown Square, PA
19073 USA and at additional mailing offices.
Canadian agreement #40030957. Postmaster: Send
address changes to Project Management Institute,
Headquarters, Four Campus Boulevard, Newtown
Square, Pennsylvania 19073-3299 USA. Phone +610-
356-4600, fax +610-356-4647.
The mission of the PMJ is to provide information
advancing the state of the art of the knowledge of pro-
ject management. PMJ is devoted to both theory and
practice in the field of project management. Authors
are encouraged to submit original manuscripts that
are derived from research-oriented studies as well as
practitioner case studies. (See Notes for Authors in the
back of this journal.) All articles in PMJ are the views
of the authors and are not necessarily those of PMI.
Subscription rate for members is $14 per year and is
included in the annual dues.
Claims for undelivered copies must be made no
later than two months following month of publication.
The publisher will supply missing copies when losses
have been sustained in transit and when the reserve
stock will permit.
PMI is a nonprofit professional organization whose
mission is to serve the professional interests of its col-
lective membership by: advancing the state of the art
in the leadership and practice of managing projects
and programs; fostering professionalism in the man-
agement of projects; and advocating acceptance of
project management as a profession and discipline.
Membership in PMI is open to all at an annual dues of
$119 U.S. For information on PMI programs and mem-
bership, or to report change of address or problems
with your subscription, contact:
Project Management Institute, Headquarters,
Four Campus Boulevard, Newtown Square, pennsyl-
vania 19073-3299 USA; tel: +610-356-4600; fax:
+610-356-4647; e-mail: pmihq@pmi.org
EDITORIAL & ADVERTISING SERVICES
Address manuscripts and other editorial submissions,
advertising and mailing list rental inquiries, and
requests for reprints/bulk copies/reprint permission to:
PRODUCTION SERVICES PROVIDED
BY IMAGINATION PUBLISHING,
CHICAGO, IL, USA
Senior Editor
Ross Foti; rfoti@imaginepub.com
Assistant Managing Editor
Lauren Strandquist; lstrandquist@imaginepub.com
Art Director
Doug Kelly; dkelly@imaginepub.com
Associate Art Director
Tonya Weiland; tweiland@imaginepub.com
Associate Art Director
Theresa Rogulic; trogulic@imaginepub.com
Publications Services Manager
Angela Kramer; akramer@imaginepub.com
Project Management Institute, Publishing
Department, Four Campus Boulevard, Newtown Square,
Pennsylvania 19073-3299 USA. Phone +610-356-4600,
fax +610-356-4647; e-mail: pmipub@pmi.org
Unless otherwise specified, all letters and articles
sent to the PMI Publishing Department are assumed
for publication and become the copyright property of
PMI if published. PMI is not responsible for loss, dam-
age, or injury to unsolicited manuscripts or other mate-
rial.
READER SERVICES
Photocopies. PMJ has been registered with the
Copyright Clearance Center, Inc. Consent is given for
copying of articles for personal or internal use, or for
the personal use of specific clients. This consent is
given on the condition that the copier pays through the
Center the per copy fee stated in the code on the first
page of each article for copying beyond that permitted
by Sections 107 or 108 of the U.S. Copyright Law. The
appropriate fee should be forwarded with a copy of the
first page of the article to the Copyright Clearance
Center, Inc., 222 Rosewood Drive, Danvers, MA 01923
USA (tel: +508-750-8400; fax: +508-750-4744). The
fee indicated on the first page of an article in this issue
will apply retroactively to all articles published in the
journal, regardless of the year of publication. This con-
sent does not extend to other kinds of copying, such as
for general distribution, resale, advertising and promo-
tion purposes, or for creating new collective works.
Special written permission must be obtained from the
publisher for such copying.
Permissions. Requests to reprint articles pub-
lished in PMJ must be made in writing to the publisher.
Reprints. Individual articles may be purchased
through the Knowledge & Wisdom Department
(www.pmi.org/k&wc) at a cost of $10.00 per article
for members and $15.00 per article for nonmembers.
Glossy Reprints. Requests for glossy reprints of
individual articles in quantities of 100 or more can be
sent to pmipub@pmi.org.
Bulk Copies of Current Issues. Copies of the cur-
rent PMJ can be obtained in quantities of 25 or more.
Orders must be placed 40 days prior to date of issue.
The cost is $5.50 per copy plus shipping.
Back Issues. Back issues are available on request
at $20 each. Call +610-356-4600 for availability.
September 2002
 From the Editor
Parviz F. Rad, PhD, PMP
T ypically, internal projects either develop new tools and
processes or expand and enhance the existing objectives of
the organization. Enlightened project-oriented organizations
have a formalized mechanism for managing the portfolio of
these projects. If the organization installs any variation of a
project management office (PMO), the task of managing the
project portfolio falls squarely within the jurisdiction of the
PMO. Further, if the PMO is a mature and sophisticated orga-
nizational entity, the tools and processes for portfolio man-
agement are highly formalized and consistently effective.
Regardless of the nature of the organizational unit in
charge of the portfolio management and the formality by
which this task is conducted, the task includes two sets of
evaluations: the initial evaluation to determine which pending
project should receive implementation funding and the mid-
stream evaluation/audit to decide whether the project funding
should continue. If formalized procedures are used to autho-
rize project initiation and if these procedures are sufficiently
comprehensive, the same procedures and/or models can be
used for project selection and as part of the midstream audit.
The initial selection process and the midstream audit
process includes two distinct components. One part deter-
mines whether the project deliverable is in line with the cur-
rent organizational vision and, for that matter, if the project
charter is in line with organizational profitability and com-
petitiveness strategies. The other component is more focused;
it reflects the willingness of the organization to sponsor this
project. The sponsorship will be demonstrated by approval of
the estimated cost and duration of the project during the
selection phase.
The willingness to sponsor the project is repeatedly
affirmed during any subsequent updates to the values of cost
and duration. The latter usually is a sobering exercise because
the estimate of the cost and duration of the vast majority of
projects tends to increase as the details of the deliverables are
fleshed out during the project life cycle.
The metrics that determine the effectiveness of the project
team in delivering the project objectives are reasonably well
identified and extensively quantified, in most cases. These met-
rics deal with the current/updated estimate of cost, schedule,
quality, and scope in comparison with the planned and/or
expected values. Equally important to the project success but
less quantified are those metrics that assess the people-related
issues such as team morale, conflict management, client satis-
faction, and stakeholder relations.
The models that determine the initial attractiveness of pro-
ject and midstream deliverable appropriateness probably are
the most qualitative, although the indices that comprise these
September 2002 Project Management Journal
models are somewhat numerous. This list includes indices that
deal with enterprise objectives, operational implications,
financial impacts, sales and marketing interests, and those that
reflect the organizational willingness to dedicate the full range
of resources to the project. Clearly, this facet of portfolio man-
agement is more an art than a science.
Some skillful project managers formulate portfolio man-
agement models that are largely intuitive and informal. Thus,
the deliverable relevance indices become part of an implicit
and unspoken model that determines the attractiveness of the
project during the selection phase and the viability of the pro-
ject during a midstream audit. As such, only organizations that
are fortunate to employ seasoned and highly intuitive project
management professionals would do well in this area.
Thus, there is a need for a set of comparison indices that for-
malize the project evaluation process and make explicit what is
implicit in these seemingly subjective evaluations. A formal-
ized structure for portfolio management will capitalize on the
skills of the more experienced and innovative project managers
for the good of the organization. If these intuitive indices
become formalized, then managerial intuition can become the
logical basis and the structural foundation for an explicit and
standardized evaluation system to be used by all PMO personnel
during any phase of the portfolio management process.
It would be a major advancement in the best practices of
portfolio management if the analysis of the pertinent data pro-
vided guidance as to what extent project and team performance
and the deliverable relevance should be quantified with scales
and plateaus. It would be equally intriguing to refine historical
data, demonstrating to what extent the survival of a project
should depend on nonquantified judgment and professional
intuition of project management professionals.
As always, on behalf of the editorial board, I invite readers
to reflect on their professional experiences and empirical
observations and to share these observations with the project
management community via submitting articles dealing with
any facet of the project portfolio management.
3
Research Report

Knowledge Generation and Sharing—

Shaping the Future
by Harry Stefanou, PhD

G rowing and sharing the body of knowledge for the project

management profession is a critical desire of the Project
Management Institute (PMI) and a key focus of the PMI®
PMI also encourages research through its sponsorship of
research projects. Two such projects recently have been com-
pleted and are expected to be published as PMI books later this
Project Management Research Program. The PMI® Research year: Quantifying the Value of Project Management by William
Program, with the advice and counsel of its Member Advisory Ibbs and Justin Reginato and Selling Project Management to
Group, has chosen to take a broad approach toward this out- Senior Executives: Framing the Moves that Matter by Janice
come. It is committed to create and enhance opportunities to Thomas, Connie Delisle, and Kam Jugdev. Through this fund-
grow the body of knowledge and to communicate the learning. ing from PMI and separately from the PMI Educational
In this way, the program hopes to help you shape the future of Foundation, four additional external grants are under way this
the profession. year (see PMJ’s June 2002 Research Report). PMI also plans to
Clearly the recent PMI® Research Conference 2002, held in continue to advance the generation of new knowledge through
Seattle, WA, USA, in July, is one example. Almost 300 acade- funding additional research in 2003.
mics and practitioners gathered to share their interest in
project management research, to learn from each other and to
set the direction for future learning. Twenty-three countries
were represented, and the ratio of practitioners to academics
was 2 to 1. Because of invited speakers, contributing presenta-
tions, panel sessions, and networking events, new knowledge
has been generated and exchanged. One value of new knowl-
edge is its implementation or practice. Time will tell but given
the attendance at the Research Conference, we can anticipate
this practice will occur. As was brought out at the conference,
knowledge sharing is a two-way interaction. We expect, there-
fore, that the practitioners have laid the groundwork with the
researchers for the next wave of research.
Other examples of our commitment to share knowledge are
PMI’s recent publication of The Frontiers of Project Management
Research and, of course, the long history of the Project
Management Journal, which serves as a tribute to the effort of
project management researchers over the years. Yet another
example is the Research Topics Track at PMI 2002 in San
Antonio, TX, USA, this October, where, again, the results of
research will reach the minds of practitioners.
The number of opportunities and interest in project man-
agement research continues to grow. In recognition of that fact,
PMI is proud to announce the creation of the PMI Research
Achievement Award, which will be presented for the first time
in 2003. The Institute’s hope is that recognition will encourage
the quest for knowledge and the achievement of excellence in
the profession. Watch for a call for nominations in the
December issue of PM Network and on our Web site,
www.pmi.org.

4 Project Management Journal September 2002

Project Management in the Information
Systems and Information Technologies
Industries
▼ market, there is a corresponding pressure to increase productivity. To maintain
Abstract
For many enterprises, sustainable success is
closely linked to information systems (IS) and
information technologies (IT). Despite signifi-
cant efforts to improve software project suc-
cess, many still fail. Current literature indicates
that most of the software project problems
are related to management, organizational,
human, and cultural issues—not technical
problems. This paper presents results of a
survey of 36 software owners/sponsors, con-
tractors/suppliers, and consultants on 12 pro-
jects. The empirical results address answers
to questions related to success, performance
metrics, and project business drivers. A lack
of alignment on these critical issues emerge
consistently by phase as well as across
the entire project. The results of this study
also are compared with others that span
seven additional industry sectors. As a result,
the authors have developed an approach that
links project critical success factors (CSFs) to
corporate strategy, and project metrics to the
CSFs. An important finding of this study is the
critical need to identify and manage realistic
expectations of the stakeholders to achieve
perceived project success.
Keywords: information systems; information
technology; managing stakeholder expecta-
tions; critical success factors; software
project management
©2002 by the Project Management Institute
2002, Vol. 33, No. 3, 5–15
8756–9728/02/$10.00 per article + $0.50 per page
September 2002
Francis Hartman, University of Calgary, 2500 University Dr. NW, Calgary, Alberta T2N 1N4
Canada
Rafi A. Ashrafi, University of Calgary, 2500 University Dr. NW, Calgary, Alberta T2N 1N4
Canada
I nformation systems (IS) and information technologies (IT) are the fastest growing
industries in developed countries. Huge amounts of money continue to be invest-
ed in these industries (Abdel-Hamid & Madnick, 1990). Due to pressure of time-to-
a competitive edge in today’s fast-changing world, an organization’s success
depends on effectively developing and adopting IS. Literature has discussed concern
for problems related to IT/IS development and implementation.
According to Zells (1994) and other studies, approximately 85% of software pro-
jects undertaken in Europe and North America are at level one of the Software
Engineering Institute’s capability maturity model (CMM). Level one is the lowest
level of CMM. The challenges at level one are to have project planning, project man-
agement, configuration management, and software quality assurance in place—and
have them working effectively. To improve project delivery performance, a number of
organizations are adopting project management approaches and setting up project
management offices (Barnes, 1991; Butterfield & Edwards, 1994; King, 1995; Munns
& Bjeirmi, 1996; Raz, 1993; Redmond, 1991).
Current literature on software projects shows that most of the software problems
are of a management, organizational or behavioral nature, not technical (Johnston,
1995; Martin, 1994; Whitten, 1995).
A survey of high-tech firms showed that if project management improved, time and
cost could be reduced by more than 25% and profits would increase by more than 5%
(Fisher, 1991). This has since been validated by use of Stratigically Managed Aligned
Regenerative Transitional (SMART) project management, based on internal bench-
marking by the companies involved in the field trials.
Objectives of the Study
In this paper, the authors report findings on current project management practices in
the IT/IS industries. The purpose of the study was to find out what practices are
important to IT/IS industries in successfully accomplishing their projects. Do they
use proven project management practices? Whatever the IT/IS industries regard as
important for the success of their projects, do they measure it? What are the project
drivers? Are these three important elements aligned with each other? The authors
investigated these questions not only for various phases of a project, but also from
the perspective of three major stakeholders. These stakeholders include an owner or
sponsor, a major contractor or supplier, and a consultant for the same project.
Project Management Journal 5
 In the next section, the authors review the current literature,
summarizing major problems of IT/IS projects. In the fourth
section, the authors discuss their research methodology. This is
followed by a discussion of the results of the study and a sum-
mary of the findings. Finally, the authors propose an approach
for managing projects based on the SMART framework and
implemented on a number of software and other projects with
markedly improved results, followed by conclusions. The
authors hope that this study will help project managers in
understanding the state of the art of project management in
the software industry and how it might be improved.
Literature Survey
The horror stories about delay, cost overrun, and abandon-
ment of software projects are widely reported in the literature
(Bailey, 1996; Gibbs, 1994; Lucas, 1995; Martin, 1994; Ward,
1994). In other industries, causes of project failures are inves-
tigated and reports written, but in the computer industry their
causes are covered up or ignored. As a result, the IT/IS industry
keeps making the same mistakes over and over again
(Johnston, 1995).
There are differences in the opinions of experts as to
whether software project management is similar or different to
project management in other industries. The authors believe
that the principles are the same across industries, but the ter-
minology and some applications are specific to each industry
and sometimes to each company or physical location. But
many believe that software management is very different
(Otto, Dhillon, & Watkins, 1993; Raybould,1996; Roetzheim,
1993; Samuels, 1996). However, in Duncan’s (1991) view,
software projects are not different from other projects. In the
authors’ opinion there are both differences and commonalties
in all types of projects, let alone software projects. Any two pro-
jects from one industry sector can be unique, and we can ben-
efit from other industries’ experiences.
In summary, the most commonly reported causes of soft-
ware project failure are as follows (based on a content analysis
of the cited literature):
■ Misunderstood requirements (business, technical, and social)
(King, 1995; Lane, Palko, & Cronan, 1994; Lavence, 1996);
■ Optimistic schedules and budgets (Martin, 1994);
■ Inadequate risk assessment and management
(Johnston, 1995);
■ Inconsistent standards and lack of training in project man-
agement (Jones, 1994; O’Conner & Reinsborough, 1992;
Phan, Vogel, & Nunamaker, 1995);
■ Management of resources (people more than hardware and
technology) (Johnston, 1995; Martin, 1994; Ward, 1994);
■ Unclear charter for a project (Lavence, 1996);
■ Lack of communication (Demery, 1995; Gioia, 1996;
Hartman, 2000; Walsh & Kanter, 1988).
The authors of this paper believe that these are symptoms of
the disease and not the root causes of the disease.
Main Reasons for Failures of IT/IS Projects
Before looking into the main causes of project failures in the
IT/IS industry, we must define critical success factors (CSFs)
6 Project Management Journal
and review the importance of metrics. The CSFs are the ele-
ments that make a project a success. These include trust,
effective communication, top management support, etc. Key
result areas (KRAs) are specific results that are needed to
deliver a successful project. CSF methodology has been
highly successful in identifying KRAs crucial for the success
of a project (Atkinson, 1999; Baccarini, 1999; Belassi &
Tukel, 1996; Byers & Blume, 1994; Clarke, 1999; Cooke-
Davies, 2002; Fisher & L’Abbe, 1994; Forsberg & Mooz,
1996; Fowler & Walsh, 1999; Johnston, 1995; Levene,
Bentley, & Jarvis, 1995; Lim & Mohamed, 1999; Martin,
1982; Pinto & Kharbanda, 1995; Raz, 1993; Shank,
Boynton, & Zmud, 1985; Tan, 1996; Wateridge, 1999;
Whitten, 1995; Zahedi, 1987; Zells, 1991).
With changing business conditions, half-century-old
project performance metrics are no longer effective for the
monitoring and control of today’s projects. Proper mea-
surement tools and metrics are necessary for effective con-
trol of projects (Hartman & Jearges, 1996; Kiernan, 1995;
Simmons, 1992; Thamhain, 1994).
Based on both consulting and earlier research, the
authors found that the main reasons for most of these
problems are:
■ Major stakeholders generally do not have a clear idea of
project success or have differing views of what success
constitutes. If a clear vision exists, it is not effectively com-
municated or the project team does not understand it.
This leads to scope creep, inappropriate measurement,
churn in developments, specification changes, delays, and
other issues;
■ Generally there is a problem in identifying KRAs and
CSFs and linking them to the stakeholders’ business strate-
gy. This leads to lack of support by senior management;
■ The project team and major stakeholders are not very
clear on what the performance and control metrics should
be. Normally the focus is on time, cost, performance, and
quality. But this focus is not consistent between stakehold-
ers or over time. Some have recognized the importance of
customer and end-user satisfaction;
■ Project control and performance metrics are not linked
to KRAs and CSFs. This means we measure the wrong
things and distract the team from what is important to suc-
cess. It looks like inadequate or ineffective project control;
■ Generally, there is very little or sometimes no alignment
among major stakeholders on success criteria, KRAs, CSFs,
performance metrics, project drivers, and on the dynamics
of change for these elements over the project life cycle. This
leads to inappropriate decision-making and inconsistency
in management style and focus.
Current literature also supports these views, albeit piece-
meal in many cases, as the focus of many papers is on spe-
cific aspects. A number of researchers have commented on
the lack of project success criteria and on a lack of proper
project metrics (Adams, Sarkis, & Liles, 1995; Demery,
1995; Ingram, 1994; Jiang, Klein, & Balloun, 1996;
Johnston, 1995; Peters, 1996; Pinto & Slevin, 1988;
Raybould, 1996; Stevens, 1991; Turner, McLaughlin,
September 2002
Thomas, & Hastings, 1994; Wateridge, 1995). Hartman and
Ashrafi (1996) reported an overview on CSFs, project drivers,
and metrics of various industries. Some of the results of the
current study were reported in Hartman and Ashrafi (1998).
As a first step to collecting empirical evidence to test the
hypotheses, the authors decided to collect data on the cur-
rent state of affairs for these aspects of project manage-
ment. This included but was not limited to:
■ Were the criteria for success clearly defined at the begin-
ning of the project? Were KRAs and CSFs identified?
■ Was there any alignment between major stakeholders
on these CFSs?
■ What project metrics were used for monitoring project
performance during various phases of the project?
■ Was there an alignment of major stakeholders on what
these metrics should be?
■ Were these metrics linked to KRAs and CSFs?
■ Were the project priorities set at the beginning of the
project? Did the priorities change during various phases of
the project life cycle?
■ Were the KRAs, CSFs, metrics, and project priorities con-
sistent with each other?
■ Were the CSFs, metrics, and project priorities changed
during various phases of the project?
■ Was there any alignment between major stakeholders
on the dynamics of such change across the various phases
of the project?
The first of these aspects is to identify what KRAs would
be crucial to the successful accomplishment of the project.
This allows the project team to keep a focus on them and
not get led astray by the everyday fire fighting on project
management problems. The second aspect is to link these
KRAs and CSFs to corporate strategy and to get buy-in
of all the major stakeholders. This linkage validates the pro-
ject and helps senior management see its relevance and,
thus, provide appropriate support to the project.
The third aspect is to monitor, control, and measure those
elements regarded as critical for project success. In other
words, once we know what is important for success, project
elements that contribute to this success are the ones
we should be measuring to monitor performance during
implementation. The fourth aspect is to identify project
business drivers. This helps make project priorities very
clear to everyone. The fifth aspect is to align all major stake-
holders and the project team on KRAs, CSFs, project
drivers, and metrics. Finally, it is important to have
an understanding of the dynamics of these elements over
various phases of the project.
The authors strongly believe that if project success crite-
ria are defined at the beginning of a project, KRAs are iden-
tified and related to corporate strategy through a clear pro-
ject mission, metrics are linked to these KRAs, project pri-
orities are made clear, and buy-in is obtained by all major
project stakeholders on all these aspects, most of the prob-
lems reported in the literature could be avoided. As a
result, efficiency and success of projects could be signifi-
cantly improved.
September 2002 Project Management Journal
Research Methodology
The authors developed a survey instrument to collect data on
all the stated aspects of project management. The survey was
divided into five sections. The first section collected project-
related and demographic information such as industry sector,
experience of project manager, project value, duration, loca-
tion, completion date, purpose of the project, role of the
respondent, etc. The second section provided a list of 33 items
identified by the authors as potential CSFs. These CSFs were
synthesized from the extensive literature on this subject. The
respondents were asked to rate these factors in terms of their
importance on a scale of 0 to 5 on each of the four project
phases (5 = very important; 1 = not important; 0 = not applic-
able). These four phases were definition, planning, execution,
and termination.
The third section of the survey dealt with project met-
rics. A list of 20 different project metrics were provided to
the respondents and they were asked to rank the impor-
tance of these metrics on the scale of 0 to 5 over the four
phases of a project. These metrics were drawn from
standard project management texts and were guided by the
Project Management Institute’s A Guide to the Project
Management Body of Knowledge (PMBOK® Guide) (PMI
Standards Committee, 2000). In the fourth section, a list
of six project priorities was given and the respondents were
asked to rank the importance of these project drivers at
each of the four phases. Last, several open-ended questions
were asked. Was this project successful? If so, on what
basis? Other relevant information the respondent wanted
to add was recorded here.
Data was collected on 12 projects in Canada through
personal interviews of 36 project owners/sponsors, con-
tractors/suppliers, and consultants—three people per pro-
ject. This was part of a much larger study spanning eight
industry sectors and more than 100 projects. A brief sum-
mary of projects is included in Appendix A.
First, an owner/sponsor of a suitable project was con-
tacted and interviewed. With permission of the
owner/sponsor, a major contractor or a supplier and a con-
sultant to the same project were identified and inter-
viewed. The respondents were asked to reply in the context
of actual project management practices and not in terms of
company policy or their personal opinions or preferences.
The sample used in the study was small and based in
Calgary, Alberta, Canada. However, based on correlation
with other findings and observations from the literature,
the authors believe these results have broad application.
Results of Survey Analysis
One of the main goals of this study was to identify KRAs and
CSFs and to find out if project metrics were linked to these
KRAs and CSFs. The authors also wanted to establish project
priorities during various phases of the project life cycle. In
addition to these, the authors wanted to answer several ques-
tions including:
■ Is there a change in the CSFs, metrics, and priorities over the
life of the project?
7
 Stakeholders
Minimum
Scope Changes
Change Management
Technology and
Expertise
Project Plan
Success Factors
Business Purpose
Top Management
Support
Project Mission
Communication
Owner’s Consultation
Owner’s Approval
0 1 2 3
Importance
Figure 1. Project Success Factors by Phases
■ How consistent are the three major stakeholders (owner,
contractor, and consultant) in their perceptions of CSFs, met-
rics, and project priorities?
■ Are the perceived CSFs consistent with the metrics used and
the project priorities identified by these stakeholders?
An average of all scores of the responses in the appropriate
survey groups was calculated. The most important characteristics
then were defined as those that had the highest average score:
■ CSFs by Phases. Figure 1 shows the 10 most important
CSFs over the four phases of projects;
■ CSFs by Stakeholders. The 10 most important CSFs by
stakeholders group are shown in Figure 2;
■ Project Metrics by Phases. Figure 3 shows the 10 most
important project metrics for four phases of the projects inves-
tigated;
■ Project Metrics by Stakeholders. The most important pro-
ject metrics according to each of the stakeholders are shown in
Figure 4;
■ Project Priority Ranking by Phase. Figure 5 shows project
priority ranking during four phases of the projects studied;
■ Project Priority Ranking by Stakeholders. Figure 6 shows
the most important project priority rankings by stakeholders.
From the results, it was concluded that the value of metrics as
a predictive tool was not fully exploited by the project teams. It
may have been possible to place more importance on key met-
rics earlier on in the project. This could have been done to ensure
8 Project Management Journal
Termination
Execution
Planning
Definition
4 5
that things did not get out of hand by the time the execution
stage rolled around because, at that point, the project has enough
momentum that it becomes quite difficult to get it back on track.
Table 1 shows the most important overall CSFs and project
metrics as identified by all the three stakeholder groups over
four project phases. The respondents showed inconsistencies
between what they identified as the project success factors and
what were used as project metrics. It was observed that in some
cases that respondents in the same project agreed on the
importance of certain CSFs, but they did not agree on how the
CSFs were measured. In other cases, respondents agreed on the
importance of these factors, but they indicated that they did
not have a metric established for measuring them.
It also was observed that project owners, contractors, and
consultants do not have a clear understanding of the methods
that are used on their projects to measure how well project
goals are met. Although there is some agreement as to which
factors are important to the success of the project, there also
should be agreement on how to measure success. If project
metrics are not clearly understood, it is difficult to determine
the level of success of the project. Each individual involved in
the project may have a different opinion as to how successful
the project is, depending on his or her own measurement.
Another important point is that success factors considered to
be important to a project should be measured in some
way during execution to determine ahead of time whether
September 2002
 Stakeholders

Minimum
Scope Changes
Change Management

Technology and
Expertise

Project Plan Contract

Success Factors

Business Purpose
Consultant
Top Management
Support
Owner
Project Mission

Communication

Owner’s Consultation

Owner’s Approval

0 1 2 3 4 5
Importance
Figure 2. Success Factors by Stakeholders

New Technology
Accepted
Responsibilities
Assigned

Resources Supplied

Within Budget Termination

Completion Defined Execution

Metrics

Activities Schedule
Planning
Scope Defined

Definition
Deliverables Identified

Milestones Met

On Time

0 1 2 3 4 5
Importance
Figure 3. Project Metrics by Phases

September 2002 Project Management Journal 9

 New Technology
Accepted
Responsibilities
Assigned
Resources Supplied
Within Budget
Completion Defined
Metrics
Activities Schedule
Scope Defined
Deliverables Identified
Milestones Met
On Time
0 1
Figure 4. Project Metrics by Stakeholders
performance objectives will be met. If the project stakeholders
and the team do not formally measure the factors that they
deem to be most important, they cannot hope to predict
its success and take corrective action as required. The project
stakeholders and the team may be spending their time on mea-
suring less important factors that will lead it to an incorrect
ongoing measurement of whether or not the project is a success.
Summary of Findings
Based on the results, the authors found:
■ The ratings for a particular project success factor did not
change very significantly between different phases;
■ Throughout all project phases, there was general agreement
among survey participants that a project mission, consultation
with the project owner, good communication, and the avail-
ability of resources are important factors for project success;
■ Participants on each project agreed on certain project suc-
cess factors, but they tended to either disagree on how the suc-
cess factor should be measured, or they did not attempt to
measure the success factor at all;
■ Project metrics were not fully utilized as a predictive tool
but rather as a measure of how well the project performed to
that point in time. This often is too late to allow effective cor-
rective action;
■ The owners of the projects agreed unanimously that it is very
important for the project to meet the needs of the end user;
10 Project Management Journal
Contract
Consultant
Owner
2 3 4 5
Importance
■ Responsibility breakdown structures, work breakdown
structures, and CSFs were not well utilized;
■ The owners did not have control, monitoring or feedback
systems independent of those used by the contractors and/or
consultants;
■ Time taken to align stakeholders on what is important to
the project probably would help improve communication,
reduce rework, and enhance the possibility of success;
■ The alignment of project metrics with project success factors
and priorities appears to be an opportunity for improvement
in the software industry.
Recommendations
It is widely accepted that there is room for improvement in the
delivery of software projects including new software develop-
ment, upgrades, or implementation. Many of the specific stud-
ies in this area suggest either what the problems may be or what
needs to be in place for success. While this is useful informa-
tion, it does not help the practitioner with the question: “How
do I achieve greater success?” This study set out to link the
symptoms for success or failure with what constructive action
may be needed to achieve such success. These recommenda-
tions, which have been tested on live projects to validate them,
make that critical link. Based on internal benchmarks in the test
companies, savings in time and cost of between 10% and 30%
were matched by improved quality and end-user acceptance.
September 2002
 Team Development

Career Development
and Training

Termination
End-User Satisfaction
Priorities

Execution

Cost Planning

Definition
Performance

Time

0 1 2 3 4 5
Importance
Figure 5. Project Priorities by Phase

Team Development

Career Development
and Training

End-User Satisfaction Contract

Priorities

Consultant
Cost

Owner

Performance

Time

0 1 2 3 4 5
Importance
Figure 6. Project Priorities by Stakeholders

September 2002 Project Management Journal 11

 Rank order Critical success factors
1 Owner is informed of the project status and
his/her approval is obtained at each stage
2 Owner is consulted at all stages of develop-
ment and implementation
3 Proper communication channels are estab-
lished at appropriate levels in the project team
4 The project has a clearly defined mission
5 Top management is willing to provide the
necessary resources (money, expertise,
equipment)
6 The project achieves its stated business
purpose
7 A detailed project plan (including time
schedules, and milestones) with a detailed
budget in place
8 The appropriate technology and expertise
are available
9 Project changes are managed through
a formal process
10 The project is completed with minimal
and mutually agreed scope changes
Table 1. Overall 10 Most Important Critical Success Factors and Metrics
The recommendations that follow represent the four most
significant elements identified and tested in this study:
■ Link your project to corporate business strategy;
■ Align major stakeholders on key issues;
■ Simplify project controls and metrics;
■ Make sure effective communication and expectation
management is maintained throughout the project life.
Greater detail on how these aspects are implemented can be
found in Hartman (2000).
Conclusions
Although the projects surveyed were rated as successes, some
projects lacked defined goals or defined metrics to measure this
success. If the owner, contractor, and consultant on a project all
have different ideas of what success is and how success will be
measured, it is unlikely that everyone (or possibly anyone) will
be satisfied when the project is completed. There are many tools
that can be utilized to ensure a successful project. For the
12 Project Management Journal
Project metrics
Project completed on time or ahead
of schedule
Milestones are identified and met
Deliverables are identified
The scope of the project is clearly
defined and quantified
Activities and logical sequences
are determined and scheduled (CPM)
Project completion is precisely defined
The project is completed within
a predetermined budget
Resource requirements are identified
and supplied as needed
Responsibilities are assigned
A specific new technology is adopted and
accepted by end users
software industry, it may just be a matter of learning what tools
are available and how to use them properly to raise the number
of successful software projects to an acceptable level.
The authors hope that this study will help in:
■ Considering a holistic approach for the project;
■ Understanding what is important for success;
■ Understanding the dynamics of project drivers and priori-
ties and that these may shift over time;
■ Getting and maintaining alignment of major stakeholders
including the immediate project team on all important strate-
gic and tactical issues;
■ Realizing better planning and more effective control;
■ Accomplishing a successful project with satisfied stake-
holders, project teams, and customers.
Some general guidelines for how this may be achieved
have been offered. The suggested approaches to achieving
project success have been tested on live projects with consis-
tently successful outcomes.
September 2002
Acknowledgments
The authors would like to thank all the students of the
fundamentals of project management class of the project man-
agement specialization at the University of Calgary who
conducted the interviews reported in this paper. Thanks also
are due to all industry personnel who gave their time and
participated in the interview surveys. The authors also would
like to thank the Natural Sciences and Engineering Research
Council of Canada (NSERC), the Social Sciences and the
Humanities Research Council of Canada (SSHRC), and indus-
try partners who support the research program of the project
management specialization at the University of Calgary. In
addition, the authors thank referees of this paper for their use-
ful comments and constructive suggestions to improve it.
References
Abdel-Hamid, T., & Madnick, S. (1990). The elusive silver
lining: How we fail to learn from software development fail-
ures. Sloan Management Review, 32 (1), 39–47.
Adams, S.M., Sarkis, J., & Liles, D. (1995). The develop-
ment of strategic performance metrics. Engineering
Management Journal, 7 (1), 24–32.
Atkinson, R. (1999). Project management: Cost, time,
and quality, two best guesses and a phenomenon, its time to
accept other success criteria. International Journal of Project
Management, 17 (6), 337–342.
Baccarini, D. (1999). The logical framework method for defin-
ing project success. Project Management Journal, 30 (4), 25–32.
Bailey, R. (1996). Approving systems projects: Eight ques-
tions an executive should ask. PM Network, 10 (5), 21–24.
Barnes, M. (1991). Innovation—Why project manage-
ment is essential to successful business. International Journal
of Project Management, 4 (4), 207–209.
Belassi, W., & Tukel, O.I. (1996). A new framework for
determining critical success/failure factors in projects.
International Journal of Project Management, 14 (3), 141–151.
Butterfield, L., & Edwards, R. (1994). PM software devel-
opment using PM techniques. Proceedings of the Project
Management Institute’s 25th Annual Symposium, Vancouver,
Canada. Upper Darby, PA: PMI, 522–526.
Byers, R., & Blume, D. (1994). Tying critical success fac-
tors to systems development. Information and Management,
26 (1), 51–61.
Clarke, A. (1999). A practical use of key success factors to
improve the effectiveness of project management. International
Journal of Project Management, 17 (3), 139–145.
Cooke-Davies, T. (2002). The ‘real’ success factors on
projects. International Journal of Project Management, 20 (3),
185–190.
Demery, K. (1995). Magic schedules delivering consumer
software on time. Proceedings of the Project Management
Institute’s 26th Annual Symposium, New Orleans, LA. Upper
Darby, PA: PMI, 662–667.
Duncan, W.R. (1991). Concern of project managers coun-
terpoint vive la difference. PM Network, 5 (6), 33–34.
Fisher, F., & L’Abbe, M. (1994). PM implementation
through organizational change. Proceedings of the Project
September 2002 Project Management Journal
Management Institute’s 25th Annual Symposium, Vancouver,
Canada. Upper Darby, PA: PMI, 277–283.
Fisher, K.J. (1991). Project management will drive revenue
recognition for software providers. PM Network, 5 (6),
25–28.
Fowler, A., & Walsh, M. (1999). Conflicting perceptions
of success in an information systems project. International
Journal of Project Management, 17 (1), 1–10.
Gibbs, W. (1994). Software’s chronic crisis. Scientific
American, 271 (3), 86–95.
Gioia, J. (1996). Twelve reasons why programs fail. PM
Network, 10 (11), 16–19.
Hartman, F. (2000). Don’t park your brain outside: A
Practical Guide to Improving Shareholder Value with SMART
Management. Newtown Square, PA: PMI.
Hartman, F., & Ashrafi, R. (1996). Failed successes and
successful failures. Proceedings of the Project Management
Institute’s 27th Annual Symposium, Boston, MA. Upper Darby,
PA: PMI, 907–911.
Hartman, F., & Ashrafi, R. (1998). Project management in
the IT/IS industry. Proceedings of the Project Management
Institute’s 29th Annual Symposium, Long Beach, CA. Newtown
Square, PA: PMI, 706–709.
Hartman, F., & Jearges, G. (1996). Simplifying project suc-
cess metrics. Proceedings of the 40th Meeting of the American
Association of Cost Engineers (AACE) International, Vancouver,
B.C., Canada. Morgantown, WV: AACE International,
PM.7.1–PM.7.4.
Ingram, T. (1994). Managing client/server and open sys-
tem projects: A 10-year study of 62 mission-critical projects.
Project Management Journal, 25 (2), 26–36.
Jiang, J., Klein, G., & Balloun, J. (1996). Ranking of sys-
tem implementation success factors. Project Management
Journal, 27 (4), 49–53.
Johnston, A.K. (1995). A hacker’s guide to project manage-
ment. Oxford, U.K.: Butterworth-Heinemann.
Jones, C. (1994). Assessment and control of software risks.
Burlingtion, MA: Yourden Press.
Kiernan, M. (1995). Get innovative or dead. Vancouver,
Canada: Dougas & Mcintyre.
King, J. (1995). ‘Tough love’ reins in IS projects.
Computerworld, 29 (23), 2.
Lane, P., Palko, J., & Cronan, T. (1994). Key issues in the
MIS implementation process: An update using end user
computing satisfaction. Journal of the End User Computing, 6
(4), 3–13.
Lavence, D. (1996). Project management in IT/MIS: An
ever increasing challenge. Proceedings of the Project
Management Institute’s 27th Annual Symposium, Boston, MA.
Upper Darby, PA: PMI, 464–466.
Levene, R.J., Bentley, A.E., & Jarvis, G.S. (1995). The scale
of project management. Proceedings of the Project Management
Institute’s 26th Annual Symposium, New Orleans, LA. Upper
Darby, PA: PMI, 500–507.
Lim, C.S., & Mohamed, M.Z. (1999). Criteria of project
success: An exploratory re-examination. International Journal
of Project Management, 17 (4), 243–248.
13
 Lucas, J.J. (1995). Work management: Why can’t infor-
mation managers manage? Proceedings of the Project
Management Institute’s 26th Annual Symposium, New Orleans,
LA. Upper Darby, PA: PMI, 304–310.
Martin, E.W. (1982). Critical success factors of chief
MIS/DP executive. MIS Quarterly, 6 (2), 1–9.
Martin, J.E. (1994). Revolution, risk, runaways: Three Rs
of IS projects. Proceedings of the Project Management Institute’s
25th Annual Symposium, Vancouver, Canada. Upper Darby,
PA: PMI, 266–272.
Munns, A.K., & Bjeirmi, B.F. (1996). The role of project
management in achieving project success. International
Journal of Project Management, 14 (2), 81–87.
O’Conner, M.M., & Reinborough, L.M. (1992). Quality
projects in the 1990s: A review of past projects and future
trends. International Journal of Project Management, 10 (2),
107–114.
Otto, R.A., Dhillon, J., & Watkins, T. (1993).
Implementing project management in large-scale
information-technology projects. In P.C. Dinsmore (Ed.).
AMA Handbook of Project Management (pp. 352–361).
New York: Amacom.
Peters, L. (1996). The master’s touch in project success-
where, how, when, what, of leading projects to excellence.
Proceedings of the Project Management Institute’s 27th Annual
Symposium. Boston, MA. Upper Darby, PA: PMI, 806–812.
Phan, D., Vogel, D., & Nunamaker, J.F., Jr. (1995).
Empirical studies in software development projects: Field
survey on OS/400 study. Information and Management, 28
(4), 271–280.
Pinto, J., & Slevin, D. (1988). Critical success factors
across the project life cycle. Project Management Journal, 19
(3), 67–75.
Pinto, J.K., & Kharbanda, O.P. (1995). Successful project
managers: Leading your team to success. New York: Van
Nostrand Reinhold.
PMI Standards Committee. (2000). A guide to the project
management body of knowledge. Newtown Square, PA: Project
Management Institute.
Raybould, M. (1996). Is project management of software
projects a special case. Proceedings of the Project Management
Institute’s 27th Annual Symposium. Boston, MA. Upper Darby,
PA: PMI, 549–554.
Raz, T. (1993). Introduction of the project management
discipline in a software development organization. IBM
Systems Journal, 32 (2), 265–277.
Redmond, W.F. (1991). The evolution of project manage-
ment in corporate systems development projects. Proceedings
of the Project Management Institute’s 22nd Annual Symposium,
Dallas, TX. Drexel, PA: PMI, 129–134.
Roetzheim, W. (1993). Managing software projects:
Unique problems and requirements. In P.C. Dinsmore (Ed.).
AMA Handbook of Project Management (pp. 347–352). New
York: Amacom.
Samuels, R. (1996). Managing software programs: A dif-
ferent kind of animal. Proceedings of the Project Management
14 Project Management Journal
Institute’s 27th Annual Symposium, Boston, MA. Upper Darby,
PA: PMI, 627–633.
Shank, M., Boynton, A., & Zmud, R. (1985). Critical suc-
cess factor analysis as a methodology for MIS planning. MIS
Quarterly, 9 (2), 121–129.
Stevens, W.M. (1991). Concerns of project managers this
& that “Yes, but …”. PM Network, 5 (2), 14–18, 22.
Simmons, D.B. (1992). A win-win metric based software
management approach. IEEE Transactions on Engineering
Management, 39 (1), 32–41.
Tan, R. (1996). Success criteria and success factors for
external technology transfer projects. Project Management
Journal, 27 (2), 45–56.
Thamhain, H. (1994). Designing modern project man-
agement systems for a radically changing world. Project
Management Journal, 25 (4), 6–7.
Turner, J.R., McLaughlin, J.J., Thomas, R.D., & Hastings,
C. (1994). A vision of project management in 2020:
Interactive session. Proceedings of the Project Management
Institute’s 25th Annual Symposium, Vancouver, Canada. Upper
Darby, PA: PMI, 634–635.
Walsh, J.J., & Kanter, J. (1988). Towards more successful
project management. Journal of Systems Management, 39
(319), 16–21.
Wateridge, J. (1995). IT projects: A basis for success.
International Journal of Project Management, 13 (3), 169–172.
Wateridge, J. (1998). How can IS/IT projects be measured
for success? International Journal of Project Management, 16
(1), 59–63.
Ward, J.A. (1994). Productivity through project manage-
ment: Controlling the project variables. Information Systems
Management, 11 (1), 16–21.
Whitten, N. (1995). Managing software development projects
(2nd ed.). New York: John Wiley & Sons.
Zahedi, F. (1987). Reliability of information systems
based on the critical success factors—formulation. MIS
Quarterly, 11 (2), 187–204.
Zells, L. (1991). Balancing trade-offs in quality, cost,
schedule, resources, and risk. Proceedings of the Project
Management Institute’s 25th Annual Symposium, Dallas, TX.
Drexel, PA: PMI 112–118.
Zells, L. (1994). World-class software practices.
Proceedings of the Project Management Institute’s 25th Annual
Symposium, Vancouver, Canada. Upper Darby, PA: PMI,
249–253.
September 2002
 Appendix 1. Project Details
Data on 12 software projects was collected. A brief description of the projects follows for the interest of the readers of this paper.

Project Value Duration

Facilities information and reporting management system $1 million 15 months
Data transmission security system $4 million Two years
Network management software $4 million One year
Financial systems $14 million Two years
Software project for a major defense project Not reported Two years
Photo and driver’s license information system $2 million One year
Flip-Chip implementation Not reported Not reported
Business process control system Not reported Not reported
Implementation of a new corporate reserve database $6.5 million One year
Development of a new version of software $1.5 million One year
Accounting system implementation $0.5 million Three months
Design and implementation of a software system to $1.2 million Two years
manage customer contract information

Francis Hartman, PhD, PMP, is a professor of project

management at the University of Calgary and holder of
the Natural Sciences and Engineering Research Council
of Canada (NSERC) and Social Sciences and Humanities
Research Council of Canada (SSHRC) Chair in Project Management.
Before accepting this position in 1991, he gained more than 30 years
of experience in the industry on more than $30 billion worth of diverse
projects. His industrial experience spans all phases of projects from
selection to decommissioning. He is the principal researcher behind the
development and testing of Strategically Managed Aligned Regenerative
Transitional (SMART) project management, which is used to enhance the
effective management of a growing number of projects, programs, and busi-
nesses. Through Quality Enhanced Decisions Inc., he offers consulting ser-
vices related to use of SMART management at the project, program, and
corporate levels to Fortune 100 companies, merging enterprises, and gov-
ernment agencies at the local, national, and global levels.

Rafi Ashrafi, PhD, PMP, obtained his master’s degree

in computer science and a PhD in project management
from the University of Bradford, U.K. He has more than
20 years of experience in academia and business in
the United Kingdom, Middle East, and Canada. Ashrafi is a project
management consultant and has worked in the information technology
(IT), telecommunications, energy and utility industries. He also is an
adjunct professor at the University of Calgary and an instructor at the
PMI South Alberta Chapter Project Management Professional (PMP®)
preparation workshop. His research interests include project manage-
ment maturity models, project management office, CSFs, and project
management issues in IT/information systems and e-Business/e-
Commerce. He has published 25 research papers in global journals
and conference proceedings.

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.
Scheduling Programs With Repetitive
Projects Using Composite Learning
Curve Approximations
▼ sulting, the upgrade of existing equipment, the change of software, or the intro-
Abstract
Programs that require executing a modest
number of similar projects arise in several
industries, such as aerospace, construction,
and defense. The scheduling of such programs
often involves deciding how many projects will
run simultaneously (in parallel) to “optimally”
trade-off resource utilization and penalty costs.
One approach for scheduling such programs is
to perform a quick approximation and follow-
ing it later with a “full-blown” procedure. By
assuming a common learning rate for all pro-
ject activities, the quick approximation,
although computationally inexpensive, is not
sufficiently accurate to eliminate the need for
the full procedure, which requires so much
data tracking and so many calculations that it
is discouraging to practitioners. The approxi-
mation proposed in this paper is significantly
more accurate than the quick approximation,
while requiring only slightly more computation,
making it more valuable for program managers.
Keywords: schedule development; learning
curves; multiproject planning; estimating
©2002 by the Project Management Institute
2002, Vol. 33, No. 2, 16–29
8756–9728/02/$10.00 per article + $0.50 per page
16
Jean-Pierre Amor, University of San Diego, School of Business Administration, 5998 Alcala
Park, San Diego, CA 92110–2492 USA
P rograms that deliver a relatively small number of similar products arise in a variety
of industries. In the defense industry, the budgetary reductions that followed the
end of the Cold War and the increasing complexity and cost of major weapon sys-
tems have led to the procurement of shrinking quantities of combat ships and air-
craft from the prime contractors. In the aerospace industry, the nature of space mis-
sions and their related activities imply fairly small orders for spacecraft, satellites, and
rocket boosters. And in the housing industry, it has long been a common practice to
build the more expensive homes in relatively small tracts. These types of pro-
grams also occur in the provision of certain services, such as management con-
duction of a process improvement or of a new monitoring system.
Frequently, these types of programs consist of a one-time order for a product that
the contractor has never produced before, and due to the complexity of the product,
each “unit” requires the execution of a distinct project. Thus, the scheduling of such
programs—that is programs with repetitive projects—is very complicated. It also is
very important because the cost per unit produced can be quite high. The program
manager, who is responsible for providing an accurate delivery schedule to all parties,
must make the development of this schedule the most important planning activity.
What the Problem Entails
The scheduling of a program is used to make an a priori estimate of its overall dura-
tion and cost. This enables the manager to set a completion date, to budget resources,
and to allocate resource usage. Scheduling also permits control over the timing of
ongoing activities to ensure the timely completion of each project.
The scheduling of programs with repetitive projects often involves balancing
two opposite tendencies. On the one hand, when the contractor has little or no
previous experience with the product, it is imperative to take advantage of the
learning phenomenon and execute as many projects as possible with the same
resources, e.g., individuals, crews, teams, materials, or equipment. This lessens the
cost of the resources used because the project performance time decreases as the
number of repetitions increases. In the extreme case, all the projects would be per-
formed in a single sequence—all in series. On the other hand, the need to deliver
each unit by its contracted due date to avoid a penalty cost often requires that sev-
eral projects be executed simultaneously. In the extreme case, all the projects
would be performed simultaneously—all in parallel. Thus, a major challenge
Project Management Journal September 2002
to the program manager in order to minimize the total cost
of the program is to determine:
■ How many parallel sequences to operate;
■ How many projects to assign to each sequence.
How the Problem Has Been Addressed
This important scheduling problem has been addressed
before, albeit with some restrictions. In 1996, Shtub,
LeBlanc, and Cai developed an integer programming prob-
lem, which seeks the least cost assignment of repetitive
projects to a given number of available teams. Thus, in their
formulation, the number of parallel sequences is fixed, while
the optimal assignment of projects to these sequences must
be determined. Their objective function is defined as the
sum of the production and penalty costs, less any incentives
for early project completion. Although their function is not
closed due to the function’s ability to handle any type of
learning curve and of penalty/incentive cost structure, the
authors efficiently obtained near-optimal solutions using
the pair-wise swap algorithm (1996).
In 1991, Shtub offered a heuristic search procedure for solv-
ing a somewhat different formulation of this scheduling prob-
lem. His goal was to find the number of parallel project
sequences, which minimizes the sum of the production and
penalty costs, less any early completion incentives. However, a
constraint implied by his search procedure is that the
sequences must be of equal or nearly equal length. That is, the
numbers of projects assigned to the parallel sequences/teams
can differ by, at most, one. For example, with eight projects and
three sequences, the assignments would be as follows:
■ Sequence/Team 1: Project 1, Project 4, Project 7;
■ Sequence/Team 2: Project 2, Project 5, Project 8;
■ Sequence/Team 3: Project 3, Project 6.
Thus, in this formulation, the assignment of projects to par-
allel sequences is a fixed process while the optimal number of
sequences must be determined. While many feasible projects-
to-sequence assignments are ignored by this simple, partial
enumeration procedure, Shtub does obtain satisfactory
solutions to his formulation (1991).
Limitations of the Previous Approaches
These two formulations suffer from a common limitation.
They do not explicitly address the contents of the projects such
as the activities involved, their precedence relationships, and
the learning rates of the resources that perform these activities.
This omission can cause substantial inaccuracies when fore-
casting the program schedule and costs.
Focusing on heuristic algorithms that might efficiently solve
their optimization problem, Shtub, Leblanc, and Cai (1996)
ignore the project details altogether. Essentially, they assume
that a project can be viewed as a macro-activity performed by
a single, possibly very large, resource entity (team). Hence, the
only learning rate required for their model’s time and cost cal-
culations is that of a projectwide resource (the project learning
rate). Because this rate is typically unknown at the start of a
new program, the authors assume its value is a priori. However,
because a project’s activities usually are defined in terms of the
September 2002 Project Management Journal
specialists who perform them and because these specialized
activities are learned at rates that can vary considerably (the
activity learning rates), this assumption can lead to substantial
errors when estimating the project completion dates and cer-
tain costs associated with the overall program. Consequently,
errors also can be made when searching for a combination of
sequences and projects, which would minimize the total cost
of the program. To address these issues, the authors recom-
mend extending their model by breaking down the work
content of each project into specific tasks, i.e., by explicitly
addressing the activities.
In his earlier article, Shtub (1991) does model the various
program costs down to the level of the project activities. He
also assumes that each activity is, under normal conditions,
performed by one unit, i.e., worker or crew, of a single type of
resource. However, to reduce the extensive computations
associated with repeated applications of the critical path
method (CPM), Shtub further assumes that all the activities/
resource types learn at the same rate. This quick approxima-
tion is tantamount to assuming a priori knowledge of the
overall project-learning rate, as is done later (1996). It also is
subject to the same possibilities for error.
Admittedly, Shtub advises the reader utilize this restrictive
assumption/approximation “… during the early stages of the
program (when data availability is limited) to analyze the
tradeoffs between alternative schedules” (1991, p. 53). He sub-
sequently recommends relaxing that assumption and using the
full-blown procedure “… for fine tuning of the delivery sched-
ule when accurate data … are accumulated” (1991, p. 53). At
that point, however, many of the extensive computations that
were avoided by the quick approximation still must be per-
formed. Thus, it is doubtful that practitioners would ever carry
out the full-blown procedure.
Proposed Solution
To improve the scheduling of programs with repetitive pro-
jects, the analysis must be conducted at the level of project
activities. However, the data tracking and the computational
workload associated with this level of detail quickly become
overwhelming. Thus, it is imperative to make the schedule and
cost estimation processes very efficient—sufficiently accurate
and relatively inexpensive.
In a 1993 article, Amor and Teplitz do explicitly address the
project contents in the scheduling of programs with repetitive
projects by incorporating detailed (task level) learning effects
into the CPM (1993). However, soon thereafter, several practi-
tioners expressed concern over the heavy computational
requirements of that approach. As a result, they developed an
efficient approximation method (Amor & Teplitz, 1998) by
extending Badiru’s composite learning rate approximation in
critical resource diagramming (1995). Their approximation
method dramatically reduces the computational workload,
while still providing accurate estimates of project delivery
dates. In both the 1993 and the 1998 articles, however, con-
tractual due dates and program costs are not included. Thus, in
the absence of penalties, the projects always are conducted in a
single sequence—all in series.
17

Number of Number of
Scheduling CPMs learning curves
procedures (number of (number of reps.)
activity)
Full-Blown (FB) N (NM) M (N)
Tangent approx. 1 (M) 1 (N)
Streamlined FB N (M) M (N)
Secant approx. 2 (M) M (2)
Table 1. Required Calculations for Various Scheduling
Procedures
In this research effort, an efficient program-scheduling tool
is developed by implementing the Amor and Teplitz approxi-
mation method for project composite learning curves (Amor &
Teplitz, 1998) in the context of Shtub’s full-blown procedure
for scheduling programs with repetitive projects (Shtub, 1991).
Evaluation of Shtub’s Scheduling Approach
The Basic Tradeoff. The scheduling of programs with repetitive pro-
jects is characterized by two potentially conflicting considerations:
■ The need to complete each unit by its due date (according
to the contract schedule);
■ The learning phenomenon. Frequently, the contractor has
little or no experience with the product, and therefore, sub-
stantial reductions in time and cost per unit (as the number of
repetitions increases) are anticipated.
Because of the first consideration, there is a tendency to set
up operations all in parallel, so that meeting the due dates is
assured. However, the second consideration encourages pro-
ducing the units all in series, so that the learning effects are
maximized. Of course, there are many other less extreme
options. The delivery schedule depends on how many parallel
sequences are utilized and how many projects are assigned to
each sequence. Thus, to make a sound scheduling decision, it
is essential to compare a variety of parallel sequencing options,
which trade-off the penalty associated with late deliveries with
the savings due to learning and to any incentive payments for
early completion. Also, the analysis should be conducted at the
level of project activities to ensure sufficient accuracy.
In the articles previously discussed, the risk of overshooting
the contracted due dates is not considered. Hence, a correct
trade off between penalty and resource utilization costs occurs
when the total cost of the program is minimized. In the
context of Shtub’s approach, this means when the optimal
number of (equal/nearly equal) sequences to be operated in
parallel has been found (Shtub, 1991). A slightly generalized
version of Shtub’s cost model, which was used in this research
effort, is presented in Appendix 1. In this model, the total program
cost is the sum of the resource utilization, hiring, firing, and
penalty costs, minus the value of any incentives resulting from
early project completions. Although the cost equations are
18 Project Management Journal
fairly complex, the following discussion is based primarily on
the number of projects in the program (N), the number of
activities in each project (M), the assumed common learning
rate (r), and the number of parallel sequences to be used (x).
Computational Workload. Because Shtub’s numerical
search heuristic for identifying an optimal number of parallel
sequences (1991) is quite straightforward, the core of his con-
tribution resides in his program cost calculations. However, as
mentioned earlier, his equal learning rates assumption essen-
tially bypasses the analysis at the project level and can lead to
appreciable errors in program schedule and cost calculations.
This assumption does yield a quick approximation to the full-
blown procedure. Shtub makes this approximation because, as
he calculates it, the full-blown procedure requires applying the
CPM to N program networks (one for each possible value of
x), each with NM activities, and evaluating the M activity learn-
ing curves, each for N repetitions. Thus, even for moderate
values of M and N, the original procedure can represent an
enormous amount of data-tracking and computations.
With his quick approximation (which the author refers to
as tangent approximation) on the other hand, Shtub shows
that the work simply consists of executing the CPM on one
project network (that of the first project) with M activities and
evaluating one learning curve, i.e., the common one, for N rep-
etitions using the critical path duration of the first project as
the duration of the first repetition. This work covers the case of
a single sequence (x = 1). Following this, Shtub applies his
fixed project-to-sequence assignment process (illustrated earlier)
to “pick off” the estimated project delivery dates in the various
parallel sequence options (x = 2 through x = N). Although the
quick approximation yields a tremendous computational
advantage, it is not quite as dramatic as it appears.
The computational reduction due to the equal learning
rates assumption appears larger than it actually is because, to
perform the full-blown procedure, it really suffices to deal with
the all in series (x = 1) situation; that is, to apply the CPM to N
project networks, each with M activities, and evaluating the M
activity learning curves, each for N repetitions. Following this,
the fixed projects-to-sequence assignment process can be used
to “pick off” the estimated project delivery dates for the other
options (x = 2 through x = N). Thus, there is no need to apply
the CPM to entire program networks, and the computational
workload associated with the full-blown procedure really is far
less than reported by the author. The author calls this compu-
tational simplification the “streamlined full-blown procedure”
and, of course, it leads to exactly the same solution as Shtub’s
full-blown procedure.
For the all-in-series situation, the computational require-
ments of the streamlined full-blown procedure are similar in
nature to those addressed by Teplitz and Amor (1993).
Nevertheless, as mentioned earlier, such requirements still are
too much for most practitioners. For each repetition of the
project, each activity time must be adjusted for the learning
effect, using appropriate parameters in the learning curve
equation (typically of the log-linear form) before applying the
CPM. Hence, there is a need to approximate the streamlined
full-blown procedure in such a way that the data-tracking and
September 2002
 Task Description Immediate Time @ Rate of improvement
predecessor SRQ (days)

A Excavate and pour footers — 4 90%

B Pour concrete foundation A 2 95%

C Erect wooden frame, including rough roof B 4 85%

D Lay brickwork C 6 85%

E Install basement drains and plumbing B 1 90%

F Pour basement floor E 2 70%

G Install rough plumbing E 3 85%

H Install rough wiring C 2 85%

I Install heating and ventilating C, F 4 90%

J Fasten plaster board and plaster G, H, I 10 90%

K Lay finish flooring J 3 85%

L Install kitchen fixtures K 1 90%

M Install finish plumbing K 2 85%

N Finish carpentry K 3 80%

O Finish roofing and flashing D 2 85%

P Fasten gutters and downspouts O 1 90%

Q Lay storm drains for rain water B 1 90%

R Paint L, M 3 95%

S Sand and varnish floors N, R 2 70%

T Finish electrical work R 1 80%

U Finish grading P. Q 2 90%

V Pour walks and complete landscaping U 5 90%

W Finish S, T, V 0 —

Note. From “Case and Readings in Production and Operations Management,” by J.C. Latona and J. Nathan (1994), Boston, MA: Allyn and Bacon; and
“Improving CPM’s Accuracy Using Learning Curves,” by C.J. Teplitz and J.P. Amor, 1993, Project Management Journal, 24, pp. 15–19.

Table 2. Tasks, Precedences, Times and Improvement Factors for Housing Example

September 2002 Project Management Journal 19

computational requirements are dramatically reduced.
180
Obviously, such an approximation must be sufficiently accu-
rate to obviate the need for a follow-up with the (even stream-
lined) full-blown procedure, as suggested by Shtub. For this 160

purpose, Shtub’s quick approximation will not do. One

Accuracy. Basically, Shtub’s quick approximation requires 140
very few computations (one CPM and N learning curve calcu-
Two
lations), but its accuracy is not very predictable. By assuming, 120 Parallel
a priori, a common learning rate, r, for all the project activities,

Calendar Days
two types of errors arise, which eventually affect the schedule
100 Three
and cost estimations.
Parallel
First, the composite learning curve of the M activities [which
800
is not log-linear even if, as is usually assumed, the individual
Four
activity learning curves are log-linear (Amor & Teplitz, 1998)]
Parallel
is approximated by a log-linear curve, which coincides with the 600

composite curve only at one point—corresponding to the first

project. The amount and direction of the divergence between
the composite curve and Shtub’s “tangent approximation” is
unknown and totally dependent on the value of r, the assumed
common learning rate. There is no guidance regarding how to
select a “good” value for r.
Because the composite learning curve of the M activities is at
the heart of the resource utilization cost calculations (see
Appendix 1), it follows that the amount and direction of the
error in the approximated resource costs are likewise unknown
and unpredictable. Strictly speaking, Shtub’s quick approxima-
tion is not necessarily a tangent to the composite learning curve.
It simply coincides with that curve at only one point and tends
to diverge rapidly from that curve as the number of projects
increases. Nonetheless, from this point on, the author shall
refer to that approximation as the tangent approximation.
Second, the project composite learning curve [which also is
not log-linear even if, as is usually assumed, the individual
activity learning curves are log-linear (Amor & Teplitz, 1998)]
is approximated by a log-linear curve, which coincides with the
composite curve at only one point—corresponding to the first
project. This composite curve reflects the fact the critical path
G M
R T
E F I J K L
A B C H N S W
D O P follow-up with the streamlined full-blown procedure.
Q U V
Note. From “Cases and Readings in Production and Operations
Management,” by J.C. Latona and J. Nathan (1994), Boston, MA: Allyn and
Bacon, and “Improving CPM’s Accuracy Using Learning Curves,” by C.J.
Tepliz and J.P. Amor (1993), Project Management Journal, 24, pp. 15–19.
Figure 1. Activity-on-Node Network for Housing Example
Five
400
Parallel
200 Six
Parallel
0
1 3 5 7 9 11 1 3 15 17 19 21 23 25 27 29
Project Number
Figure 2. Completion Times: IDEAL – Up to Six Parallel
Sequences
may change from repetition to repetition (Amor & Teplitz,
1998). Again, the amount and direction of the divergence
between the composite curve and Shtub’s “tangent approxima-
tion” is unknown and totally dependent on the value of r, the
assumed common learning rate. Again, there is no guidance
regarding how to select a “good” r.
Because the project composite learning curve is the basis for
determining the delivery dates, and because these dates are at
the heart of the penalty/incentive cost calculations (see
Appendix 1), it follows that the amount and direction of the
error in the approximated penalty/incentive costs are likewise
unknown and unpredictable.
Essentially then, for any given number of parallel sequences,
the project delivery dates and the total program cost cannot be
reliably and accurately estimated with Shtub’s tangent approxi-
mation. Hence, the optimal number of sequences, provided by
his search heuristic, also may be in error. Consequently, there is
a need for a more predictable approximation method that is
economical and sufficiently accurate to obviate the need for a
An Efficient Approximation Procedure
The efficient approximation method mentioned earlier (Amor &
Teplitz, 1998) constructs a log-linear secant to a project’s com-
posite learning curve. This approximation requires calculating the
critical path times of the first and last projects of interest—using
appropriate task times from the individual activity learning
curves—and estimating the slope of the log-linear curve passing

20 Project Management Journal September 2002

 $2,000,000
Resource
$1,500,000
Hire
Fire
Dollars
$1,000,000
Penalty
Incentive
$500,000
TOTAL
1 3 5 7 9 11 13 15
Number of Parallel Sequences
Figure 3. Cost Curves: IDEAL – All Types of Costs
through these two points. Thus, the “secant approximation” gen-
erates, a postiori, an approximate learning rate, rp, for the project.
The computational requirements of this method (two CPMs,
each with M activities, and two M learning curve calculations) are
not very different from those of Shtub’s “tangent approxima-
tion.” For the composite learning curve of the M activities, the
secant approximation generates, a postiori, another approximate
learning rate, rs, which can be viewed as that of an all-in-series
network of the project’s M activities. This learning curve is used in
the resource utilization cost calculations (see Appendix 1).
The direction of the divergence between a composite
learning curve and its “secant approximation” is always the
same. Because the composite curve is convex (Amor &
Teplitz, 1998), the secant approximation always overesti-
mates it. The amount of divergence cannot be bounded the-
oretically, but Amor and Teplitz demonstrate, via examples
from several industries, that the accuracy of the approxima-
tion is well within 4%. In fact, the accuracy is within 2% in
most cases (Amor & Teplitz, 1998).
For comparison purposes, the numbers of required cal-
culations for the various procedures are summarized in
Table 1. Note that the number of calculations associated
with the “secant approximation” is independent of N, the
number of projects in the program—an attractive feature
from a practitioner’s point of view.
Therefore, an efficient program-scheduling tool is developed
in this research effort by implementing the Amor and Teplitz
(1998) approximation method in the context of Shtub’s (1991)
September 2002 Project Management Journal
200
180
Tgnt (0.90)
160
IDEAL
Resource Days
140
Scnt (1–30)
120
Tgnt (0.85)
100
Tgnt (0.80)
80
60
1 4 7 10 13 16 19 22 25 28
Project Number
Figure 4. Learning Curves: Composite – All Activities
full-blown procedure. This tool combines the CPM, individual
task learning curves, and the “secant approximation” with
Shtub’s cost model and search heuristic. The resulting package
dramatically reduces the computational workload required by
Shtub’s approach (quick/tangent approximation followed by
the streamlined full-blown procedure), while providing suffi-
ciently accurate and robust results. This last point is illustrated
in the next section, which compares the results from the two
approximation methods (for those practitioners who might
consider using only the quick approximation and ignore the
follow-up) to the “ideal” results of the full-blown procedure.
Results
A spreadsheet is used to integrate the various elements of the
author’s approximation procedure for scheduling sequences of
similar projects. An example from the house construction
industry, developed by Amor and Teplitz (1998), is used to:
■ Compare the accuracies of the tangent and secant approxi-
mation methods;
■ Examine the robustness of these methods over a range of
plausible scenarios.
This example is used because it is conservative; it generated the
largest error (3.3%) among the various examples reported. The
author’s results indicate that the secant approximation method is
consistently more accurate than Shtub’s tangent approximation.
House Construction Example. The author uses the multi-
unit construction program presented by Amor and Teplitz
(1998). This example had been drawn from Latona and
21
 Description Notation Value (s)

Number of projects N 30

Contract schedule excursions

Contract schedule A Dj 120 days j = 1 through j = 10

180 days j = 11 through j = 20
240 days j = 21 through j = 30

Contract schedule B Dj 180 days j = 1 through j = 10

270 days j = 11 through j = 20
360 days j = 21 through j = 30

Contract schedule C Dj 270 days j = 1 through j = 10

405 days j = 11 through j = 20
540 days j = 21 through j = 30

Contract schedule D Dj 360 days j = 1 through j = 10

540 days j = 11 through j = 20
720 days j = 21 through j = 30

Penalty cost excursions

Daily penalty cost pj $0 j = 1 through j = 30

Daily penalty cost pj $50 j = 1 through j = 30
Daily penalty cost pj $125 j = 1 through j = 30
Daily penalty cost pj $250 j = 1 through j = 30
Daily penalty cost pj $500 j = 1 through j = 30

Daily incentive cost gj $0 j = 1 through j = 30

Number of activities M 23 (the twenty-third represents

project completion — 0 days)

Acitivity first performance tj 8.1; 2.8; 11.8; 17.7; 2.0; 21.4; 8.8;
time (days) 5.9; 8.1; 20.1; 8.8; 2.0; 5.9; 13.2;
5.9; 2.0; 2.0; 4.2; 21.4; 4.4; 4.0;
10.1 (based on data from Table 2)

Daily resource cost ki $300/crew i = 1 through i = 22

Hiring cost hi $300/crew i = 1 through i = 22

Firing cost fi $400/crew i = 1 through i = 22

Initial number of crews ci 1 i = 1 through i = 22

Table 3. Input Data for Base Case and Excursions

22 Project Management Journal September 2002

 $1,800,000
Tgnt (0.90)
$1,600,000
IDEAL
$1,400,000
Dollars
Scnt (1–30)
$1,200,000
Tgnt (0.85)
$1,000,000
Tgnt (0.80)
$800,000
1 3 5 7 9 11 13 15
Number of Parallel Sequences
Figure 5. Resource Cost
Nathan (1994), with the learning curve data extracted from
Teplitz and Amor (1993) and adapted to that situation.
Table 2 lists the tasks, precedence, and times at standard
reference quantity (SRQ) and rates of improvement associated
with the construction of a single home. The SRQ normally rep-
resents a unit beyond which a fully experienced worker is con-
sidered to demonstrate no perceptible future improvement.
In this example, the SRQ is assumed to be unit 100. Although
there are 23 tasks listed in the table, only the first 22 are time-
consuming activities; the twenty-third task simply represents
the project completion. Based on their precedence relation-
ships, the tasks are arranged in a moderately complex network
(see Figure 1). Coincidentally, there are 22 possible paths
through the network, all which could be a critical path at any
time throughout this repetitive program.
The author examined the construction of a 30-house subdi-
vision and, as done earlier (Amor & Teplitz, 1998), made two
simplifying assumptions. The first assumption is that all work-
ers and crews are new hires with no previous experience with
their upcoming tasks. This assumption serves two purposes:
■ If all workers are equally experienced/inexperienced, it is
not necessary to maintain learning curves depicting the actual
repetitive position of each worker;
■ The “early stage” of a learning curve is the most demanding
on the accuracy potential of the approximations.
The second assumption is that no house in a parallel sequence
is started until the previous house has been completed, i.e.,
only x houses are under construction at any given time in the
September 2002 Project Management Journal
120
Tgnt (0.90)
100
IDEAL
Resource Days
80
Scnt (1–30)
60
Tgnt (0.85)
40
Tgnt (0.80)
20
1 4 7 10 13 16 19 22 25 28
Project Number
Figure 6. Learning Curves: Composite – Project
program. Normally, of course, a given task is performed on
more than one house at a time (within any given sequence),
resulting in a shorter program duration. While this assumption
avoids the complexity of dealing with “overlapping” projects, it
does tend to overstate the beneficial impact of incorporating
learning curves (Amor & Teplitz, 1997).
Because, in this paper, the house construction example is
used with a cost model, additional data are obtained from
Shtub’s paper (1991) and from informal conversations with a
building contractor. These data then were adapted to the cur-
rent example. Table 3 lists the values of all the inputs for the cost
model defined in Appendix 1. It includes the values that are
used later in the robustness excursions. For the base case, ana-
lyzed in the next section, Contract Schedule B (first 10 houses
due on day 180, next 10 houses due on day 270, and last 10
houses due on day 360) and a penalty cost of $50/day are used.
This table presents the information in the same order as it is
introduced in Appendix 1 under “Assumptions and Notation.”
Comparative Accuracy Results. In this section, the author
compares the accuracy of the tangent and secant approxima-
tions methods for the base case in the house construction
example. The streamlined, full-blown procedure is used to
obtain the ideal results, to which the approximated results were
compared. In the Figures 2, 3, 4, 5, 6, 7, 9 and 10, the curves cor-
responding to the full-blown procedure are labeled “IDEAL.”
To use the tangent approximation method, an a priori value
for the common learning rate, r, must be selected. This value is
used for approximating:
23
 Method and Optimal number of Minimum total Error Percentage
parameter parallel sequences (x*) cost (TC*) error

IDEAL 4 $1,421,277 N/A N/A

Scnt (1–30) 5 $1,454,832 $33,554 2.4%

Tgnt (0.85) 5 $1,479,731 $58,454 4.1%

Tgnt (0.80) 4 $1,328,746 ($92,532) –6.5%

Tgnt (0.90) 6 $1,619,996 $198,719 14.0%

Table 4. Comparative Optimization Results for the Base Case

■ The composite learning curve of all the activities, which is
required for calculating the resource utilization cost;
■ The project composite learning curve, which is needed for
calculating the delivery dates and the penalty costs.
Shtub furnishes no guidance for selecting a good value of r.
In his example, he uses 0.85 and, because the activity learning
rates are not provided, one can only assume that 0.85 is a rel-
atively central value for the individual rates he has in mind. In
the author’s housing example, the largest and smallest learning
rates are 0.95 and 0.70 respectively; their mean is 0.861, their
median is 0.875, their mode is 0.90, and their Delionback
weighted average (NASA, 1975)—at the first unit—is 0.831.
From among the activities that are on the critical path of the
first project, the largest and smallest learning rates still are 0.95
and 0.70, respectively; their mean is 0.844, their median is
0.90, their mode is 0.90, and their Delionback weighted aver-
age (NASA, 1975)—at the first unit—is 0.804. Which one of
these numbers would make a good, a priori estimate for r?
The initial estimate for r can affect greatly the computations
in the tangent approximation method and, hence, the com-
parative accuracy results. Consequently, the author was reluc-
tant to select only one value for r. Based on the potential choic-
es listed, the author used three values: 0.80, 0.85, and 0.90.
Because they are extreme among the calculated measures of
centrality and to develop empirical bounds on the error of the
tangent approximation method, the values of 0.80 and 0.90
were selected. The value of 0.85 also was used because it is
Shtub’s choice and because it is halfway between 0.80 and
0.90. In the Figures 4, 5, 6, 7, 9, 11, 12, 13, and 14, the curves
corresponding to these three estimates are labeled “Tgnt
(0.80),” “Tgnt (0.90),” and “Tgnt (0.85).”
As mentioned earlier, the secant approximation method pro-
duces a postiori approximate learning rates, rs and rp, for the all-in-
series and project networks. These rates are derived from the
slopes of the log-linear secants (to the composite learning curves)
“passing through” projects/houses 1 and 30. In this example,
these rates turn out to be 0.844 and 0.831, respectively. In the
Figures 4, 5, 6, 7, 9, 11, 12, 13, and 14, the curves corresponding
to a secant approximation are labeled “Scnt (1–30).”
Ideal Completion Times and Costs. Figure 2 displays the
ideal house completion times for up to six parallel sequences of
projects as obtained with the full-blown procedure. Note that this
figure illustrates the rapid increase in completion times and thus
in the likelihood of penalty costs as the number of sequences
decreases. Figure 3 displays the various program cost curves as
functions of the number of parallel sequences used. Because
there are not daily incentive “costs” in this example, the incentive
“cost” curve lies completely on the horizontal axis. Because the
initial number of crews is one for all the activities, the hire and
fire cost curves simply increase linearly. Note how the resource
utilization cost and penalty cost curves “pull” in opposite direc-
tions and, essentially, determine the U shape of the total cost
curve. For this base case, the optimal number of sequences to
operate is four, which yields a minimum total cost of $1,421,277.
Comparative Resource Utilization Costs. Figure 4 dis-
plays the composite learning curves for all M activities. Note
that the secant method approximates the ideal curve quite
well. However, the tangent method can yield a broad spec-
trum of approximations between Tgnt (0.80) and Tgnt
(0.90). As expected, the Tgnt (0.85) curve diverges from the
ideal curve less than the other two tangents, but it does not
fit the ideal curve as well as Scnt (1–30). Figure 5 displays
the resource utilization cost curves as functions of the num-
ber of parallel sequences used. Again, the ideal curve is well
approximated by the secant method, and the Tgnt (0.80)
and Tgnt (0.90) curves bound a broad spectrum of approx-
imations. Note, however, the convergence of the tangent
curves as the number of sequences increases. This is due to:
■ The inverse relationship between the number of projects
in a sequence and the number of sequences used;
■ The fact that increasingly shorter portions of the early
part of the composite learning curve (of all the activities) are
used as the number of sequences increases.
Comparative Penalty Costs. Figure 6 displays the project
composite learning curves. Although the vertical scale is dif-
ferent from that of Figure 4, the general observations are
similar, as should be expected, for both sets of curves. Figure
7 displays the penalty cost curves as functions of the

24 Project Management Journal September 2002

 CS (A)
Scnt (1—30) 2.4%
Tgnt (0.85) 3.9%
Tgnt (0.80) –14.8%
Tgnt (0.90) 25.9%
Table 5. Comparative Maximum Percentage Errors
number of parallel sequences used. Again, the ideal curve is
well approximated by the secant method, and the Tgnt
(0.80) and Tgnt (0.90) curves bound a broad spectrum of
approximations, especially when the number of sequences
is small. As with Figure 5, there is a convergence of the tan-
gent curves. This is due to similar, though slightly more
complex, reasons as for the resource utilization cost curves.
Comparative Hire and Fire Costs. Figure 8 displays the
hire and fire cost curves. These curves do not depend on
either of the composite learning curves (all M activities or
projects). Consequently, they are the same for all the meth-
ods being compared.
Comparative Total Costs. Figure 9 displays the total cost
curves as functions of the number of parallel sequences used.
As expected from the results, the ideal curve is well approxi-
mated by the secant method. However, the tangent method
yields a broad range of possible approximations between Tgnt
(0.80) and Tgnt (0.90). Of course, Tgnt (0.85) provides a bet-
ter fit, but it is not a good as that of Scnt (1–30).
Comparative Optimization Results. For each method and
parameter value discussed, the optimal number of parallel
sequences (x*), its associated minimum total cost (TC*), and
the error and percentage error from the full-blown procedure
are presented in Table 4.
Clearly, the secant approximation method results in much
smaller cost estimation errors than the tangent approximation
method. Although Scnt (1–30) would recommend using five
sequences rather than four, the error associated with following
that recommendation would result in an error of only $173
($1,421,277 with four sequences in IDEAL vs. $1,421,450 with
five sequences in IDEAL) after the fact, i.e., after program
implementation and assuming that the ideal curve turns out to
be a perfect forecast. Thus, for the base case example, the secant
method provides an approximation to Shtub’s full-blown pro-
cedure, which is superior to almost any reasonable application
of the tangent approximation method. However, what about
the robustness of these results?
Comparative Robustness Results:
Excursions from the Base Case
The base case is derived from the example in Amor and Teplitz
(1998) that turned in the largest error (3.3%) when they
September 2002 Project Management Journal
CS (B) CS (C) CS (D)
2.5% 3.8% 2.9%
4.1% 6.5% 9.3%
–14.8% –14.8% –14.8%
25.9% 25.9% 25.9%
estimated the duration of several programs (in various indus-
tries), each one using a single sequence of projects.
Consequently, it does not seem beneficial to work with their
other examples. Instead, to investigate the robustness of these
approximation methods, the author conducted a series of
excursions from the base case by varying two key parameters
over relevant ranges of values. These two parameters are the
contract schedule and the daily penalties. The various other
parameters are far less significant.
Four contract schedules are examined, including the one
that is used in the base case. From the tightest to the loosest
schedule, they are labeled A, B, C, and D; their details are
shown in Table 3. For each contract schedule, five daily penal-
ties are examined; they range from $0 to $500. (As mentioned
earlier, the base case uses Contract Schedule B and a $50/day
penalty.) Because, when there are no penalty and incentive
costs the total program cost is independent of the contract
schedule, there are 17 data points/excursions (including the
base case) for use in the robustness analysis.
Ideal Minimum Total Cost. Figure 10 displays the ideal mini-
mum total cost, TC*, for each of the 17 excursions mentioned, as
obtained with the streamlined full-blown procedure. It also indi-
cates in parentheses under each data point the optimal number of
parallel sequences, x*, which yields the plotted value of TC*.
As expected, for any given penalty rate, the tighter the contract
schedule, the more sequences are needed to minimize the total
cost—because there is a greater likelihood of incurring penalties.
Naturally, the tighter the schedule, the larger the minimum total
cost. Also as expected, for any given contract schedule, as the
penalty rate increases, the number of sequences needed to mini-
mize the total cost tends to increase. The larger the penalty rate,
the larger the minimum total cost—until x* becomes greater
than the number of sequences beyond which there would be no
more late projects. This turns out to be the case for Contract
Schedule D, when x* becomes 3, because for that relatively loose
schedule there are no late projects when x is greater than 2.
Comparative Errors Under
the Various Contract Schedules
Figures 11, 12, 13, and 14 display the percentage error of each
approximation method for each daily penalty under Contract
Schedules A, B, C, and D. That is, for each excursion, the
25
 $1,600,000 $140,000

$1,400,000
$120,000
Tgnt (0.90) Hire

$1,200,000
$100,000
IDEAL Fire
$1,000,000

$80,000
Dollars

Dollars
Scnt (1–30)
$800,000

$60,000
$600,000 Tgnt (0.85)
$40,000
$400,000
Tgnt (0.80)
$20,000
$200,000

$0 $0
1 3 5 7 9 11 13 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Parallel Sequences Number of Parallel Sequences

Figure 7. Penalty Costs Figure 8. Hire and Fire Costs

approximated minimum total costs are compared to the ideal Conclusion

minimum total cost, and the percentage errors are reported.
Clearly, under contract Schedule A, Scnt (1–30) outperforms the
tangent approximations examined—its error never exceeding
2.4% ($40,143).
Likewise, under Contract Schedule B, Scnt (1–30) outper-
forms the tangent approximations examined—its error never
exceeding 2.5% ($37,524). Again, under Contract Schedules C
and D, Scnt (1–30) outperforms the tangent approximations
examined—its error never exceeding 3.8% ($51,429) and 2.9%
($34,506), respectively. Also note that, as the contract schedule
loosens, the percentage error of the approximations tends to
increase, regardless of the daily penalty values.
Summary
Based on the results shown in Figures 11, 12, 13, and 14, Table
5 presents the maximum percentage error in the minimum total
cost for each contract schedule and penalty cost combination
examined. Clearly, the secant approximation method is consis-
tently more accurate than the tangent approximation method.
The only way that the tangent might outperform the secant is
with an extremely fortunate and highly unlikely a priori estimate
for r. However, it is also clear that, given the activity learning rates
of the housing example, a choice of 0.85 (as Shtub made in his
paper) does turn in sufficiently accurate results. Nevertheless,
why gamble with the unpredictable size and direction of its error
when the secant approximation errs minimally and consistently
(typically less than 4% and always as an overestimation).
An existing two-stage approach for scheduling programs with repet-
itive projects recommends performing an initial quick (tangent)
approximation and following it later with a full-blown scheduling
procedure. By assuming a common learning rate for all project
activities, the tangent approximation, although computationally
inexpensive, is not sufficiently accurate to obviate the need for the
full-blown follow-up. The full-blown method, however, requires so
much data-tracking and so many calculations that it is discouraging
to practitioners. The secant approximation presented in this paper is
significantly more accurate and reliable than the tangent approxi-
mation, while requiring only slightly more computation. Because
the secant approximation comes sufficiently close to the full-blown
(ideal) follow-up, the latter can be avoided altogether, making the
secant approximation a sufficiently accurate and extremely econom-
ical procedure, which should be attractive to most practitioners.
The scheduling of programs with repetitive projects discussed
in this paper (and in the associated references) does not consid-
er two important factors: risk management and project overlap.
For example, the activity times are assumed to be known with
certainty, whereas they could be treated as random variables and
dealt with using risk analysis techniques. Likewise, the projects
in any given sequence are assumed to begin only upon comple-
tion of their predecessor, whereas they could be overlapped and
analyzed using a newly developed technique (Amor & Teplitz,
1997). These considerations, however, are beyond the scope of
the present work and are left as suggestions for further research.
If they can be incorporated into the secant approximation

26 Project Management Journal September 2002

 $2,800,000 $2,000,000

$2,600,000
Tgnt (0.90) $1,800,000 A: 120/180/240
$2,400,000

IDEAL $1,600,000 B: 180/270/360

$2,200,000

Scnt (1–30)
Dollars

Dollars
$2,000,000 $1,400,000
C: 120/405/540

Tgnt (0.85)
$1,800,000 D: 360/540/720
$1,200,000

$1,600,000 Tgnt (0.80)

$1,000,000
$1,400,000

$1,200,000 $800,000
1 3 5 7 9 11 13 15 1 100 200 300 400 500

Number of Parallel Sequences Daily Penalty ($)

Figure 9. Total Costs Figure 10. Min TC – IDEAL – Contract Schedules

and Daily Penalties

method, that procedure will make the scheduling of sequences
of similar projects even more valuable for program managers.
References
Amor, J.P., & Teplitz, C.J. (1993). Improving CPM’s accuracy
using learning curves. Project Management Journal, 24 (4), 15–19.
Amor, J.P., & Teplitz, C.J. (1997). On the relative impacts
of learning and overlap on the scheduling of programs with
repetitive projects. Proceedings of the Twenty-Sixth Annual
Meeting of the Western Decision Sciences Institute, Kamuela, HI.
Madison, WI: Omnipress, 729–732.
Amor, J.P., & Teplitz, C.J. (1998). An efficient approxima-
tion procedure for project composite learning curves. Project
Management Journal, 29 (3), 28–42.
Badiru, A.B. (1995). Incorporating learning curve effects
into critical resource diagramming. Project Management
Journal, 2 (2), 38–45.
NASA. (1975). Guidelines for application of learning/cost
improvement curves (Report TM X-64968). Washington, DC:
Delionback, L.M.
Latona, J.C., & Nathan, J. (1994). Cases and readings in produc-
tion and operations management. Boston, MA: Allyn and Bacon.
Shtub, A. (1991). Scheduling of programs with repetitive
projects. Project Management Journal, 22 (4), 49–53.
Shtub, A., LeBlanc, L.J., & Cai, Z. (1996). Scheduling pro-
grams with repetitive projects: A comparison of a simulated
annealing, a genetic and a pair-wise swap algorithm.
European Journal of Operational Research, 88, 124–138.
Appendix 1. The Cost Model and its Characteristics
Assumptions and Notation
For the program:
■ The program consists of N identical projects, each result-
ing in the completion of one unit of the product;
■ Each project, j, refers to one product unit; j = 1 through j =
N. And “J” represents the set of all projects within the program;
■ A due date, Dj, is specified for each project according to
the contract schedule;
■ A delivery date, Tj, eventually occurs for each project as a
result of executing the program;
■ A penalty rate, pj, applies to each project completed
beyond schedule, e.g., in $/day;
■ An incentive rate, gj, applies to each project completed
ahead of schedule, e.g., in $/day.
For the projects:
■ Each project consists of a network of M activities;
■ Each activity, i, is performed by one unit of the specific
resource type required for that activity, e.g., one worker, one
crew; i = 1 through i = M. And “I” represents the set of all
activities within a project;
■ A duration, ti, applies to each activity when performed for
the first time (by one unit of its associated resource type);
■ A learning curve exponent, bi, corresponds to the learning
rate, ri, of resource type i. (bi = log ri / log 2);
■ A resource cost rate, ki, applies to each resource unit when
it is in use, e.g., in $/day;

September 2002 Project Management Journal 27

 30 30

Tgnt (0.85) Tgnt (0.85)

20 20

Scnt (1–30) Scnt (1–30)

Error in Min TC (%)

Error in Min TC (%)

10 10
Tgnt (0.80) Tgnt (0.80)

0 Tgnt (0.90) 0 Tgnt (0.90)

–10 –10

–20 –20
1 100 200 300 400 500 1 100 200 300 400 500

Daily Penalty ($) Daily Penalty ($)

Figure 11. Percent Error in Min TC – Contract Schedule A Figure 12. Percent Error in Min TC – Contract Schedule B
(120/180/240) (180/270/360)

■ Each resource type has an associated hiring cost, hi, e.g., The Cost Model
in $/crew;
■ Each resource type has an associated firing cost, fi, e.g., TC(x) = RC(x) + HC(x) + FC(x) + PC(x) – IC(x)
in $/crew;
■ Each resource type is available in a certain quantity, e.g., N/x

number of crews, ci, at the beginning of the program, e.g., ci where: RC(x) = x∑ ∑ ki ti (j)bi if N/x is an integer or
∋
j=1 i I
= 0 through ci = N, and must returned to that level at the end
of the program. [N/x]+

= (x – [N/x]+ x + N) ∑ ∑ ki ti (j)b ∋
i

j=1 i I
For the cost functions and the optimization process:
■ x represents the number of (equal/near equal) parallel [N/x]+–1

sequences under consideration, x = 1 through x = N; + ([N/x]+ x – N) ∑ ∑ ki ti (j)b if N/x is not an integer

∋
i

j=1 i I
■ Tj(x) represents the delivery date of project j when using
x parallel sequences;
■ RC(x) represents the resource utilization cost when using HC(x) = ∑ hi Max{x – ci,0}
∋
i I
x parallel sequences;
■ HC(x) represents the hiring cost when using x
parallel sequences;
■ FC(x) represents the firing cost when using x FC(x) = ∑ fi Max{x – ci,0}
∋
parallel sequences; i I

■ PC(x) represents the penalty cost when using x

parallel sequences;
■ IC(x) represents the incentive “cost” when using x PC(x) = ∑ pj [Tj (x) – Dj]
∋
j J
parallel sequences; Tj(x) < Dj
■ TC(x) represents the total cost when using x
parallel sequences;
■ [N/x]+ represents the next integer value of N/x, whenever IC(x) = ∑ gj [Dj – Tj(x)]
∋
j J
N/x is not an integer. Tj(x) < Dj

28 Project Management Journal September 2002

 30 30

Tgnt (0.85) Tgnt (0.85)

20 20

Scnt (1–30)
Scnt (1–30)
Error in Min TC (%)

Error in Min TC (%)

10 10

Tgnt (0.80)
Tgnt (0.80)

0 0 Tgnt (0.90)
Tgnt (0.90)

–10 –10

–20 –20
1 100 200 300 400 500 1 100 200 300 400 500

Daily Penalty ($) Daily Penalty ($)

Figure 13. Percent Error in Min TC – Contract Schedule C Figure 14. Percent Error in Min TC – Contract Schedule D
(270/405/540) (360/540/720)

Characteristics of the Model

The total cost of the program, TC(x), which is to be mini-
mized by choice of an appropriate number of (equal/nearly
equal) parallel sequences, x, is simply the sum of the resource
Jean-Pierre Amor, PhD, is associate professor of deci-
utilization, hiring, firing, and penalty costs minus the value
sion sciences in the School of Business Administration at
of any incentives resulting from early project completions.
the University of San Diego. He received his PhD in oper-
Although there is but a single variable to optimize, x, the cost
ations research from the University of California at Los
functions are relatively complex.
Angeles. During his prior career with the U.S. Air Force, Colonel Amor was
The resource utilization cost function, RC(x), has one of
commander of a high-technology laboratory, which managed research
two forms, depending on whether or not the sequences are of
and development projects associated with the Strategic Defense
equal length. In both forms, the inside summation represents
Initiative. While in the Air Force, Amor also planned and analyzed military
the composite learning curve of the M activities; and the vari-
operations at the Pentagon, NATO, and in Southeast Asia. At the universi-
able of interest, x, appears in several locations, including the
ty, he teaches courses in operations management and management
upper limit of the outside summation. The hiring and firing
science. His current research interests include project management,
costs functions are essentially linear.
work in the 21st century, and complexity issues in organizations.
The penalty and incentive costs functions are perhaps the
most complex, because they depend in two ways on the delivery
date, Tj(x), of each late/early project. First, in the summation lim-
its, Tj(x) determines whether or not a project is included in the
calculations; second, in the individual terms, Tj(x) helps deter-
mine the size of the penalty/incentive for the included projects.
Furthermore, a project’s delivery date is itself based on
that project’s critical path as well as on the critical paths of
all the projects preceding it in the (parallel) sequence to
which it belongs. Hence, ultimately, Tj(x) depends on the
project composite learning curve. Therefore, any analytical
optimization clearly is out of the question and the search
process must be numerical.

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.
A Measure of Software
Development Risk
▼ 1999). Government agencies completed only 18% of all IT projects on time and on
Abstract
Risks to software development are present
throughout the creation of information systems
(IS). The ability of researchers and practitioners
to consider risk within their models and project
management methods has been hampered by
the lack of a rigorously tested instrument to
measure risk properties. An instrument to
measure software development risk based
on properties described in the literature is pro-
posed. A survey of 152 IS project participants
was collected to examine the reliability and
validity of the instrument, both which appear
strong. The instrument then was used to
empirically demonstrate a strong anticipated
relationship between risk and project success.
Keywords: information systems implemen-
tation; risk analysis; survey research
©2002 by the Project Management Institute
2002, Vol. 33, No. 3, 30–41
8756–9728/02/$10.00 per article + $0.50 per page
30
James J. Jiang, University of Central Florida, College of Business Administration, Department of
Management Information Systems, P.O. Box 160000, Orlando, FL 32816–1400 USA
Gary Klein, The University of Colorado at Colorado Springs, College of Business and
Administration, 1420 Austin Bluffs Parkway, Colorado Springs, CO 80933–7150 USA
T. Selwyn Ellis, Louisiana Tech University, College of Administration and Business, Department
of Computer Information Systems, Ruston, LA 71272–0001 USA
T he development of information technology (IT)-related projects has become a
critical aspect of most businesses because of organizational dependence on com-
puter-based systems to remain competitive. Even with the widespread use
of tools such as prototyping, data modeling, structured design, computer-assisted
software engineering (CASE), and other project management tools, software devel-
opment still suffers extremely high failure rates (Meyer, 1998). In a recent study of
100 companies, only 37% of major IT projects were completed on time and only
42% were completed on budget (Gordon, 1999).
Data from The Standish Group International suggests that the problem is not
going away. Results from a survey of more than 7,000 IT projects shows that in 1998,
40% were canceled prior to completion and more than 45% of the projects were late
or over budget. Leading the way in late or over-budget projects is the government
sector with more than half of their projects falling into this category. These failed pro-
jects cost U.S. companies and government agencies $145 billion annually (Gordon,
budget (Davis & Wilder, 1998).
In a recent survey of 150 IT managers, poor project management was cited as the
leading reason for IT project failure (Davis & Wilder, 1998). Risk management can
have a positive impact on the selection, scope, schedules, and budgets of projects
(Schwalbe, 2000). Two areas of risk management are risk identification and risk
quantification. Risk identification has been the topic of many research endeavors
(Anderson & Narasimhan, 1979; Balloun, Jiang, & Klein, 1996; Barki, Rivard, &
Talbot, 1993; Boehm, 1989; Cafasso, 1994; McFarlan, 1981; Nidumolu, 1995;
Zmud, 1980). Given the volume of research addressing the risks associated with
software development projects, risk identification has been properly considered.
Risk quantification involves evaluating risks and satisfactorily measuring the risks
associated with a project. Very little progress in the way of theoretical development
has been made in this area. Barki, Rivard, and Tallard (1993) made an early effort at
the development of an instrument to measure software development risk. The instru-
ment, which consisted of 34 variables and 144 items, was empirically tested using
120 projects. This was an important step toward current risk management research.
With the availability of a reliable and valid instrument to measure software develop-
ment risks, the area of project management could be strengthened and expanded. In
this study, the authors tested and refined previous software risk measurement con-
structs. A short form risk measurement has been proposed and empirically tested
Project Management Journal September 2002
with 152 projects. The results indicate a negative relationship
between the software risks and project success.
Software Development Risk Literature
According to McFarlan (1981), failure to assess individual pro-
ject risk is a major source of the information systems (IS) devel-
opment problem. Many researchers have attempted to identify
these various risk variables threatening successful software devel-
opment. Anderson and Narasimhan (1979) identified eight
such factors: unwilling users; multiple implementers; turnover
among users, developers, or sponsors; inability to specify
requirements; inability to cushion impact on others; lack of
management support; lack of development experience; and
technical or cost-effectiveness problems. Cafasso (1994) identi-
fied user involvement, executive management support, proper
planning, realistic expectations, and a clear statement of require-
ments as the top five factors influencing project success. An addi-
tional set of factors including personnel change, technological
newness, technological change, top management support, team
expertise, novelty of applications, and users involvement signifi-
cantly affect system development success (Jiang, Klein, & Pick,
1995; Pressman, 1992; Saleem, 1996; Zmud, 1980).
Although, many risks have been identified, their impacts
on performance have not been empirically tested. The authors
suspect two critical reasons for this lack of empirical evidence
in the literature. First, development risks have been conceptu-
alized in a variety of ways and lack a sound measurement con-
struct (Boehm, 1989; Charette, 1989). Secondly, development
risks, similar to system success measures, are elusive to define.
Different researchers have addressed different aspects of system
success, making comparisons difficult and the prospect of
building cumulative tradition for risk research similarly elusive
(Barki, Rivard, & Talbot, 1993). Without such measurement, it
is not surprising that one has difficulty finding reports of soft-
ware risk effects in the literature.
Recognizing the importance of a risk measurement con-
struct, Barki, Rivard, and Talbot (1993) proposed an initial
measure for software development risks. Based upon a com-
prehensive review of the IS risk literature, a 144-item software
risk measurement instrument pertaining to various characteris-
tics of a software development project was developed. The 11
multiple-item risk dimensions include:
■ Application complexity;
■ Technical complexity/acquisition;
■ Extent of changes;
■ Resources insufficiency;
■ Application size;
■ Team’s lack of expertise with task;
■ Team’s lack of general expertise;
■ Lack of user experience;
■ Lack of user support;
■ Intensity of conflicts;
■ Lack of clear of role definitions.
Reliability and structure validation were examined through
traditional factor analysis.
This type of measurement can be applied in a variety of
ways. In application settings, it may be used by IS project
September 2002 Project Management Journal
managers to diagnose project risks before a project starts. It also
may be used during the software development process to
manage the potential risks more effectively. Similarly, a risk
scale may be used in organizations to make selections between
contending software development projects. Researchers may
choose to use these measures to better understand how risk fac-
tors influence the success of IS. More generally, it is likely to be
used by researchers who are interested in understanding the
diffusion of various software development risks in organiza-
tions and determining its impacts on various project outcomes,
e.g., system performance, information quality, and user satisfac-
tion. The relationships between software risks, project manage-
ment practices, and project success can be established clearly.
Given the potential wide usage of a software risk measure
by both IS practitioners and academicians, it is important to
conduct studies that further test the psychometric properties of
this instrument and examine its relationship to project success.
Unfortunately, no major study was found that re-examined
this proposed instrument. In addition, one limitation of apply-
ing this instrument is its relatively large number of items.
An extremely large sample is needed to establish meaningful
reliability and validity. The authors believe that a shorter
version of the original scale may be more applicable.
Research Methodology
In an attempt to respond to these issues, the authors
developed a short-form instrument for measuring soft-
ware development risk by modifying the original mea-
surement model (Barki, Rivard, & Talbot, 1993). Factor
analysis was selected to examine the dimensions of the
original construct (five dimensions were found), but the
relationships between these factors were not clear. Overall
software development risk is an aggregate of various unde-
sirable events (Nidumolu, 1995). While each of these
dimensions is distinct, the overall software development
risk is a combination—and interaction—of the various
dimensions. Are these dimensions independent of each
other? In other words, is software project risk a higher-
order phenomenon evidenced through the extent of risks
across multiple dimensions?
Such a measure is valid only if there is a link between risk
and project success. Researchers and practitioners believe
that there is a significant relationship between software risk
and project success, but there is little empirical evidence to
support this conventional wisdom. In fact, Barki, Rivard,
and Talbot (1993) specifically call for studies examining the
relationships between the risk and project success.
Therefore, another question addressed was whether there
exists a significant relationship between the risk and project
success. The authors examined two issues:
■ Are the proposed measurements of various software
development risk factors (first-order) independent of each
other and, thus, must be measured as such? Or, can an inte-
grated overall software project risk (second-order) be
obtained in the proposed measurement?
■ Are software development risk factors significantly associ-
ated with project success?
31
Sampling and Data Collection
In the pilot study, 300 questionnaires were mailed to IS
managers in the Midwestern region of the United States. The
items were derived from the set provided by Barki, Rivard, and
Talbot (1993) and reduced to accommodate the factors
identified in their study. Self-addressed return envelopes for
each questionnaire were enclosed, and a total of 67 responses
were obtained. This data was analyzed with factor analysis to
determine agreement with the expected structure. The ques-
tionnaire was modified by removing items not identified with
a particular factor. The modified measurement was reviewed by
five IS managers and three IS researchers for clarity and
appropriateness. Minor changes were made for instruction
clarity. Due to the modification of the questionnaire, these 67
responses were not included in further analysis.
In the main study, the revised questionnaires were mailed
to 500 IS project managers in the United States who were
members of the Project Management Institute (PMI). The sam-
ple was chosen because IS members of PMI typically have
expertise on software project management and represent a vari-
ety of organizational settings. PMI membership has been used
widely in project management research. Self-addressed return
envelopes for each questionnaire were enclosed. All the
respondents were assured that their responses would be kept
confidential. A total of 86 questionnaires were returned. To
increase the response rate, a second set of 500 questionnaires
was sent three months later. A total of 66 questionnaires
were returned in this round, for a combined total of 152
responses. To examine the potential bias between the first-
round respondents and the second-round respondents, Chi-
square statistical tests were conducted on the demographic
variables. No significant difference on the respondents’
demographic backgrounds was found. These two rounds of
respondents were combined for data analysis as described in
Appendix 1. A summary of the demographic characteristics
of the sample is presented in Table 1. The demographic
characteristics of this sample were similar to other studies
sampling PMI members (Larson, 1997).
Short Form of Software
Development Risk Assessment
The remaining items contained in the applied instrument are
shown in Table 2. The major dimensions of the original instru-
ment with multiple items were included. The resulting ques-
tionnaire consisted of 45 items. The questionnaire asks respon-
dents about the presence of complexity and problems in their
most recently completed IS projects as understood in the early
stages of development. Each item was presented using a five-
point scale. All the items were scaled so that the greater the
score, the greater the presence of the risk in question.
Exploratory factor analysis is appropriate when the number
and nature of the underlying measurement structures are not
certain. Therefore, an exploratory factor analysis first was con-
ducted as briefly described in Appendix 1. The results yielded
six risk dimensions (technical acquisition, project size, role
definition, user experience, user support, and team expertise),
which were retained for confirmatory factor analysis (CFA) to
32 Project Management Journal
further examine dimensionality, reliability, and validity (see
Appendix 1).
Project Success Measurement
Project success often is measured in terms of its efficiency,
effectiveness, and timeliness (Henderson & Lee, 1992).
Efficiency is the ratio of outputs to inputs, the subjective
perception of efficiency in team operations, and the adherence
to allocated resource, e.g., time and budget. Effectiveness is the
quality of the system produced. The instrument used in this
study to measure project success was adopted from Henderson
and Lee (1992). The items of project success include meeting
project goals, conducting the required quantity and quality of
work, adhering to schedules and budget, and speeding
operations. Project success is the focus of this study because
documented reports on software development failures often
are related to delay of schedules, budget overrun, and lack of
system quality.
Self-evaluation of performance has been widely adopted in
the area of organizational behavior. Bandura’s (1986) work on
self-efficacy suggests that self-appraisals are a valid predictor of
success. When detailed measurements are made, efficacy
assessments and success are highly correlated (Schunk, 1989).
Individuals often are the best judges of their own project suc-
cess. In addition, by becoming involved in a project, they may
become more motivated to improve their project success
(Campbell & Lee, 1988). Based upon the theoretical and
empirical research in areas of organizational behavior, the
authors expect that IS project managers’ perceptions of their
project success will be a useful indicator of IS project success.
Specifics may be found in Appendix 1.
Assessment Question 1. Are the proposed measurements
of various software development risk factors (first-order) inde-
pendent of each other and should thus be measured as such?
Or, can an integrated overall software project risk (second-
order) be obtained in the proposed measurement?
The initial measurement model for software development
risks (see Appendix 1) implies that technical acquisition, project
size, user support, user experience, team experience on applica-
tion, and team members’ role definitions are associated but not
governed by a common phenomenon. On the other hand, an
alternative model posits a second-order factor model governing
the correlations among the six factors. The theoretical interpre-
tation of this second-order factor is an overall trait of software
development risks. While each of these dimensions is distinct,
overall software risk is an integration of each distinct risk factor.
Previous research notes this operational perspective represents a
theoretically strong basis for capturing complex software risk
measures (Boehm, 1989; Barki, Rivard, & Talbot, 1993).
This second-order model explains the covariance among
first-order factors in a more parsimonious way, i.e., one that
requires fewer degrees of freedom. Even when the second-
order model is able to explain the factor covariance, the good-
ness-of-fit of the second-order model can never be better than
the corresponding first-order model. In this sense, the
first-order model provides a target or optimum fit for the high-
er-order model. However, because the second-order model
September 2002
 Characteristics First Round Second Round Total

Position

IS executive 8 6 14
IS manager 12 10 22
IS project leader 44 39 83
Other IS professionals 18 5 23

Work experience

1–5 years 5 3 8
6–10 years 12 9 21
11–15 years 20 5 25
16–20 years 20 9 29
21 or above 25 33 58

Gender

Male 65 48 113
Female 18 12 30

Organization size (number of employees)

Under 100 employees 7 5 12

100–500 employees 12 13 25
500–1,000 employees 7 6 13
1,000–2,500 employees 10 6 16
2,500–5,000 employees 6 3 9
5,000–10,000 employees 22 9 31
10,000 or more employees 20 18 38

Average number of members of IS project teams

2–3 members 1 1 2
4–5 members 13 3 16
6–10 members 22 21 43
11–15 members 12 9 21
16–20 members 15 14 29
21–25 members 18 13 31

First round sample size = 86

Second round sample size = 66
Total sample size = 152

Table 1. Sample Demographics

represents a more parsimonious representation of observed co-
variances, it should be accepted over the baseline as a “true”
representation of model structure.
Assessment Question 2. Are software development risks
significantly associated with project success?
A correlation between software development risk and project
success was found to be –0.22 using a confirmatory factor
analysis model (see Appendix 1). This negative relationship was
anticipated due to the diametrically arranged adjectives between
the software development risks and project success dimensions.
Essentially, it confirms that software risk does dampen project
success. In fact, the strength of the correlation shows that risk
should be measured and controlled to improve success and that
the measure proposed is a good indicator of total risk. The
results confirm the authors’ anticipation that software develop-
ment risk adversely affects the outcomes of a project.

September 2002 Project Management Journal 33

 Technological acquisition
1. Need for new hardware
2. Need for new software
3. Large number of hardware suppliers
4. Large number of software suppliers
Application size
5. Large number of people on team
6. Large number of different stakeholders
on team, e.g., information systems staff, users,
consultants, suppliers, customers
7. Large project size
8. Large number of users will be using this system
9. Large number of hierarchical levels occupied by
users who will be using the system, e.g., office
clerks, supervisors
Lack of team’s general expertise
10. Ability to work with uncertain objective
11. Ability to work with top management
12. Ability to work effectively as a team
13. Ability to understand human implications
of a new system
14. Ability to carry out tasks effectively
Lack of team’s expertise with the task
15. In-depth knowledge of the functioning of user
department
16. Overall knowledge of organizational operations
17. Overall administrative experience and skill
18. Expertise in the specific application area
of the system
19. Familiarity with this type of application
Lack of user support
20. Users have a negative opinion about the system
meeting their needs
21. Users are not enthusiastic about the project
22. Users are not an integral part of the development
team
23. Users are not available to answer the questions
24. Users are not ready to accept the changes the
system will entail
25. Users slowly respond to development team request
26. Users have negative attitudes regarding the use
of computers in their work
27. Users are not actively participating in
requirement definition
Table 2. Initial Items for Software Development Risks
34 Project Management Journal
Intensity of conflicts
28. Great intensity of conflicts among team members
29. Great intensity of conflicts between users and
team members
Extent of changes brought
30. The system requires a large number of users’
tasks to be modified
31. The system will lead to major changes in the
organization
Resources insufficient
32. In order to develop and implement the system,
the scheduled number of people per day is
insufficient
33. In order to develop and implement the system,
the dollar budget provided is insufficient
Lack of clarity of role definitions
34. Role of each member of the team is not
clearly defined
35. Role of each person involved in the project is not
clearly defined
36. Communications between those involved in the
project are unpleasant
Application complexity
37. Technical complexity, e.g., hardware, software,
database
38. Large number of links to existing systems
39. Large number of links to future systems
Lack of user experience
40. Excessive requirements specifications
41. Users are not very familiar with system
development tasks
42. Users have little experience with the activities
to be supported by the future applications
43. Users are not very familiar with this type
of application
44. Users are not aware of the importance of their
roles in successfully completing the project
45. Users are not familiar with data processing
as a working tool
September 2002
Conclusion and Implications
The purpose of this study was to examine software develop-
ment risk measurement in light of the importance of project
risk management on system success. The relationship of soft-
ware project development risk to the traditional project success
also was examined. This was accomplished by administering
both the proposed software risk and project success survey
instruments to the same sample of IS project leaders and then
deriving unique dimensions of software development risks,
which contribute to the prediction of project success. It was
found that the dimensions are integrated to a higher-order of
overall risk, and risk is adversely related to project success.
This study represents an important step in the develop-
ment of valid and reliable measures of software development
risk. It contributes to the IS literature with a measure that pro-
vides both IS researchers and practitioners with more specific
information concerning software development risk effects on
system success. In particular, it suggests that IS project leaders
should obtain measures of such project risk dimensions as
project complexity, user experience, user support, and project
team expertise, which may be overlooked when beginning a
project. In practice, the importance of these dimensions
points to the need for stronger project management emphasis
on estimating project development risks and a demonstration
of how specific risks may affect the outcomes of project
delivery, e.g., budget, schedule and system quality.
Although the results of this study provide insight into the rela-
tionship of software development risks and project success, one
obvious limitation is the single dimension of system success.
Additionally, only one stakeholder is considered, the IS project
manager. Examination of other interest groups may indicate the
presence of other risk categories that would provide a more com-
prehensive picture of the development process, including the
group interaction items dropped during analysis. The proposed
short version of software development risk assessment may not
be comprehensive enough to capture other dimensions of risk,
especially those that could best be measured directly, such as
budget size and number of impacted existing systems.
Nevertheless, the proposed short form of risk measurement pro-
vides a starting point to establish such a comprehensive model.
References
Anderson, J.C. (1987). An approach for confirmatory mea-
surement and structural equation modeling of organizational
properties. Management Science, 33 (4), 525–541.
Anderson, J.C., & Gerbing, D.W. (1988). Structural equa-
tion modeling in practice: A review and recommended two
step approach. Psychological Bulletin, 103 (3), 411–423.
Anderson, R.E., Hair, J.F., & Tatham, R.T. (1992).
Multivariable data analysis with readings (3rd ed.). New York:
MacMillan Publishing.
Anderson, J., & Narasimhan, R. (1979). Assessing imple-
mentation risk: A methodological approach. Management
Science, 25 (6), 512–521.
Bagozzi, R.P., Yi, Y., & Phillips, L.W. (1991). Assessing con-
struct validity in organizational research. Administrative Science
Quarterly, 36 (3), 421–458.
September 2002 Project Management Journal
Balloun, J., Jiang, J., & Klein, G. (1996). Ranking of system
implementation success factors. Project Management Journal, 27
(4), 50–55.
Bandura, A. (1986). Social foundation of the thought
and action: A social cognitive theory. Englewood Cliffs, NJ:
Prentice-Hall.
Barki, H., Rivard, S., & Talbot, J. (1993). Toward an assess-
ment of software development risk. Journal of Management
Information Systems, 10 (2), 203–225.
Boehm, B.W. (1989). Software risk management.
Washington, DC: IEEE Computer Society Press.
Bollen, K.A. (1989). Structural equations with latent variables.
New York: John Wiley & Sons Inc.
Cafasso, R. (1994). Few IS projects come in on time, on
budget. Computerworld, 28 (50), 20–21.
Campbell, D.T., & Cook, T.D. (1979). Quasi-experimentation.
Boston: Houghton Mifflin Co.
Campbell, J.P., Ghiselli, E.E., & Zedeck, S. (1981).
Measurement theory for behavioral science. San Francisco:
W.H. Freeman.
Campbell, J.P., & Lee, C. (1988). Self-appraisal in perfor-
mance evaluation: Development versus evaluation. Academy of
Management Review, 13 (2), 302–314.
Charette, R.N. (1989). Software engineering risk analysis and
management. New York: McGraw-Hill.
Cronbach, L.J. (1951). Coefficient alpha and the internal
structure of tests. Psychometrika, 16, 297–334.
Davis, B., & Wilder, C. (1998). False starts, strong finishes.
InformationWeek, n711, 41–53.
Fornell, C., & Larcker, D.F. (1981). Evaluating structural
equation models with unobservable variables and measure-
ment error. Journal of Marketing Research, 18, 39–50.
Gordon, P. (1999). To err is human, to estimate, divine.
InformationWeek, n711, 65–72.
Hayes, F. (1997). Managing user expectation.
Computerworld, 31 (44), 8–9.
Henderson, J.C., & Lee, S. (1992). Managing I/S design
teams: A control theory perceptive. Management Science, 38 (6),
379–387.
Hocevar, D., & Marsh, H. (1985). Application of confir-
matory factor analysis to the study of self-concept: First and
higher order factor models and their invariance across
groups. Psychological Bulletin, 97 (3), 562–582.
Jiang, J., Klein, G., & Pick, R. (1995). Are your sure you want
to use that data in your study? IEEE Transactions on Systems,
Man, and Cybernetics, 25 (2), 378–380.
Joreskog, K.G. (1993). Testing Structural Equation Models. In
K.A. Bollen and L.S. Long (Eds.). Newbury Park, CA: Sage
Publications.
Joreskog, K.G., & Sorbam, D. (1989). LISREL 7: A guide to the
program and applications (2nd ed.). Chicago: SPSS Inc.
Larson, E. (1997). Partnering on construction projects: A
study of the relationship between partnering activities and pro-
ject success. IEEE Transactions on Engineering Management, 44
(2), 188–195.
Long, J.S. (1983). Confirmatory factors analysis: An intro-
duction to LISREL. Sage University Paper Series On
35
 Factor Loaded items Loading

Technical acquisition 1. Need for new hardware 0.64

2. Need for new software 0.65
3. Large number of hardware suppliers 0.66
4. Large number of software suppliers 0.65

Project 6. Large number of different stakeholders on team 0.71

size 7. Large project size 0.67
8. Large number of users will be using the system 0.77
9. Large number of hierarchical levels occupied by users
who will be using this system 0.63

Lack of clarity of 34. Role of each member of the team is not clearly defined 0.78
role definition 35. Role of each person involved in the project is not clearly defined 0.79
36. Communications between those involved in the project
are unpleasant 0.77

Lack of user 41. Users are not very familiar with system development 0.67
experience on 42. Users have little experience with the activities to be supported
systems development by the future system 0.81
43. Users are not very familiar with this type of application 0.77
45. Users are not familiar with data processing as a work tool 0.71

Lack of 20. Users have a negative opinion about the system meeting their needs 0.77
user support 21. Users are not enthusiastic about the project 0.79
23. Users are not available to answer the question 0.72
24. Users are not ready to accept the changes the system will entail 0.79
25. Users slowly respond to development team requests 0.69

Lack of team 13. Ability to understand human implications of a new system 0.69
expertise 15. In-depth knowledge of the functioning of user department 0.70
16. Overall knowledge of organizational operation 0.75
17. Overall administrative experience and skill 0.75
18. Expertise in the specific application area of the system 0.71

Table A3. Summary Results of Exploratory Factor Analysis

Quantitative Application in the Social Science. Thousand Oaks,
CA: Sage Publications.
MacCallum, R.C. (1986). Specification searches in covariance
structural modeling. Psychological Bulletin, 100 (1), 107–120.
McFarlan, F.W. (1981). Portfolio approach to informa-
tion systems. Harvard Business Review, 59 (5), 142–150.
Meyer, R.L. (1998). Avoiding the risks in large software
system acquisitions. Information Strategy, 14 (4), 18–33.
Nidumolu, S. (1995). The effect of coordination and
uncertainty on software project performance: Residual per-
formance risk as an intervening variable. Information
Systems Research, 6 (3), 191–219.
Pressman, R.S. (1992). Software engineering: A practitioner’s
approach (3rd ed.). New York: McGraw-Hill.
Saleem, N. (1996). An empirical test of the contingency
approach to user participation in information system
development. Journal of Management Information Systems, 13
(1), 145–166.
Schunk, D.H. (1989). Self-efficiency and cognitive skill
learning. In C. Ames & R. Ames (Eds.). Research on Motivation
in Education (Vol. 3). San Diego: Academic Press.
Schwalbe, K. (2000). Information technology project
management. Cambridge, MA: Course Technology.
Zmud, R.W. (1980). Management of large software
development efforts. MIS Quarterly, 4 (1), 45–55.
Appendix 1
Determination of Factor Structure
First, in an exploratory factor analysis using principal compo-
nent analysis (PCA), if a given item has a meaningful loading,
i.e., loading greater than 0.45 on more than one, that item is
dropped. Likewise, items with less than a 0.45 loading were

36 Project Management Journal September 2002

 1 0.80

2 0.77
Technology
Acquisition
3

0.50 0.23 0.27 0.32 0.22

4
6
0.62

7 Project
0.79 Size
8
0.83

0.31 0.29 0.20 0.09

13 9

15 0.78

0.97
Lack of Team
16 Experience
0.56

17
0.43
0.84
18 20
0.83
21
Lack of User
23 Support
0.84

0.22 0.46
24
0.72 0.43 0.42

25
34 0.97

Lack of Role
0.94
35 Definition
0.54
36
0.30
41

42 0.85
Lack of User
0.92 Experience
43
0.62

45

Figure A1. Software Risk Measurement Model

September 2002 Project Management Journal 37

 Construct Standardized T-value Composite Variance Cronbach
indicators loading reliability extracted alpha
estimate

F1. Technological acquisition (0.77*) 0.62 0.76

1 0.80 8.74 0.64

2 0.77 8.45 0.59

F2. Project size (0.79*) 0.57 0.78

6 0.62 7.19 0.38

8 0.79 9.56 0.63
9 0.83 10.04 0.69

F3. Lack of team expertise (0.83*) 0.62 0.80

15 0.78 9.75 0.61

16 0.97 12.96 0.94
17 0.56 6.62 0.32

F4. Lack of user support (0.88*) 0.66 0.88

20 0.84 11.38 0.71

21 0.83 11.12 0.69
24 0.84 11.36 0.71
25 0.72 9.03 0.52

F5. Lack of clarity of role definition (0.87*) 0.70 0.85

34 0.97 14.15 0.94

35 0.94 13.38 0.88
36 0.54 6.58 0.29

F6. Lack of user experience on system development (0.86*) 0.71 0.85

42 0.85 11.32 0.72

43 0.92 12.63 0.85
45 0.67 8.25 0.55

Note. * Indicates composite reliability of the corresponding construct.

Table A4. Properties of the Revised First-Order Software Development Risk Model

dropped. The variables remaining after the PCA are shown in
Table A3. All the remaining items loaded as expected, except
that the two a priori dimensions “the lack of team’s general
expertise” and “lack of team’s expertise with the task” indicat-
ed being combined. Four dimensions of the software develop-
ment risk did not have a single item with a significant load-
ing—including intensity of conflicts, extent of changes
brought, resource insufficiency, and application complexity—
and were not considered in further analysis. Application
complexity may best be measured by direct observation rather
than psychometric scales (Jiang, Klein, & Pick, 1995).
Subsequent confirmatory factor analysis (CFA) determined
construct validity (MacCallum, 1986). The expectation was
that each of the developed scales in Table A3 uniquely measures
its associated factor and that the system of factors measures
different aspects of software development risks. The
Covariance Analysis of Linear Structural Equations (CALIS)
procedure of Statistical Analysis Systems (SAS), version 6.12,

38 Project Management Journal September 2002

 Construct Standardized T-value
indicators loading
Project success
P1. Meet project goals 0.87 12.96
P2. Amount of work 0.82 11.78
P3. Quality of work 0.75 10.30
P4. Adhere to schedules 0.88 13.14
P6. Speed of operations 0.83 12.10
P7. Adhere to budgets 0.68 9.09
Table A5. Properties of the Project Success Measurement
was utilized as the analytical tool for testing the measurement
and structural equation models.
One important assumption of confirmatory factor modeling
is multivariate normality. Because multivariate normality is diffi-
cult to test, it is recommended that univariate normality among
variables be initially tested (Anderson, Hair, & Tatham, 1992).
Such testing was accomplished through examination of the
moments around the mean of each variate’s distribution (Bollen,
1989). Among the variables of this study, analysis of these statis-
tics suggests no serious departures in univariate normality.
When a CFA model provides a reasonably good approxi-
mation to reality, it accounts for the observed relationships in
the data set. Four fit indices are typically used to identify over-
all goodness of fit in a CFA:
■ Comparative fit index (CFI);
■ The Bentler-Bonett non-normed fit index (NNFI);
■ Adjusted goodness-of-fit index (AGFI);
■ The Chi-square divided by the degrees of freedom
(< 3 preferred).
Values greater than 0.90 are desirable for CFI and NNFI and
greater than AGFI should be equal or greater than 0.80.
The CFA measurement model is shown in Figure 1. The
items correspond to those of Table A3. Table A4 provides a
summary of the analysis. In the initial phase of model estima-
tion, several items (7, 13, 17, 18, and 23) were deleted due to
a significant cross-loading with other constructs. In the subse-
quent tests for discriminant validity (discussed in the following
section), items 3 and 4 were deleted due to a lack of significant
reliability. The final model, Chi-square/degrees of freedom =
1.41 < 3, AGFI = 0.81, CFI = 0.95, and NNFI = 0.94, indicates
a good fit between the model and data. In addition, the
significance of all parameter estimates, e.g., t-value > 3, shown
in Table A4 support the revised model.
September 2002 Project Management Journal
Composite Variance Cronbach
reliability extracted alpha
estimate
(0.92) 0.65 0.90
0.76
0.67
0.56
0.77
0.69
0.46
A second-order factor model captures correlations among
the first-order constructs. The efficacy of such a structure can
be tested using a comparative methodology for higher-order
factor models (Bollen, 1989; Joreskog, 1993). A second
order analysis also was conducted using a CFA over the six
identified factors. The fit indices showed a good fit with
AGFI = 0.81, CFI = 0.94, NNFI = 0.93, and the Chi-
square/degrees of freedom = 1.5. Further empirical support
for acceptance of the higher-order factor structure is found in
the significance of estimated parameters as well as the
amount of variance explained by the structural equations. All
structural equation parameters exhibit significantly high t-
values (all > 3.18, p = 0.01). Specifically, the CFA path
between overall software risks and its underlying first-order
dimensions are 0.61 for technical acquisition, 0.58 for pro-
ject size, 0.69 for team expertise, 0.53 for user support, 0.44
for role definition, and 0.40 for user support.
In the project success measurement model initial
estimation, item 5, due to a lack of significant loading, was
deleted from the model. The final model described indicat-
ed a good fit AGFI = 0.87, CFI = 0.97, NNFI = 0.95, and the
Chi-square/degrees of freedom = 2.6. Internal consistency
(reliability) was assessed using a measurement model and
reported in Table A5 (Joreskog & Sorbam, 1989). The com-
posite reliability (0.92), Cronbach (1951) alpha (0.90), and
variance extracted estimate (0.65 > 0.50) indicate strong
internal consistency of this construct. In summary, the
project success measure has strong validity and reliability.
It has been suggested that the efficacy of second-order
models be assessed through examination of the target (T)
coefficient [T = Chi-square (baseline model)/Chi-square
(alternative model)] (Hocevar & Marsh, 1985). This coeffi-
cient has an upper bound of 1.0 with higher values implying
39
 the responses were generally from more (or less) risky projects.
Test Construct T-value Chi-square The mean of software risk items (Table 2) ranged from 2.87 to
correlations difference 3.92, median from 2.67 to 4.00, skewness from –0.04 to 0.78,
and kurtosis from –1.23 to 0.18. The responses had good dis-
Technology acquisition (F1) tribution on project risks since the mean and median were
similar, skewness was less than 2, and kurtosis was less than 5
F2 0.50 5.54* 53.39* (Campbell, Ghiselli, & Zedeck, 1981). Overall, project risk-
F3 0.23 2.32* 73.25* related bias seemed unlikely because of the considerable varia-
F4 0.27 2.70* 71.54* tion shown on the distributional characteristics.
F5 0.32 3.37* 70.64* Additional threats to external validity occur if the sample
F6 0.22 2.16* 74.07* showed other systematic bias in the relation of demograph-
ics to the project measures. An analysis of variance (ANOVA)
Project size (F2) was conducted by using project success (as the dependent
variable) against each demographic category (independent
F3 0.31 3.30* 103.89* variables). Results did not indicate any significant relation-
F4 0.29 3.01* 103.69* ships. Similar results held for each software risk factor as the
F5 0.20 2.13* 118.65* dependent variables. In summary, no significant threats to
F6 0.09 0.89 124.44* external validity were discovered in the data.
Item reliability (indicator reliability) is defined as the
Team expertise (F3) square of the correlation between a latent factor and that
item. This reliability indicates the percent of variation in the
F4 0.43 5.36* 124.56* indicator that is explained by the item that it is supposed to
F5 0.22 2.47* 147.40* measure (Long, 1983). The reliability of an indicator can be
F6 0.46 5.78* 123.01* computed by squaring the standardized factor loading
obtained in the analysis model. However, this test is quite
User support (F4) conservative; very often item reliability will be below 0.36,
even when reliabilities are acceptable (Fornell & Larcker,
F5 0.43 5.41* 225.81* 1981). For low reliability items in any factor, the composite
F6 0.42 5.08* 148.48* reliability for the factor must be examined to determine
whether the low reliability items should be eliminated. A
Role Definition (F5) 0.70 acceptable level of composite reliability is strongly
desired. The results of the item reliability are shown in Table
F6 0.30 3.40* 170.62* A4 with values of 0.77 or greater.
Composite reliability represents the internal consistency
of the items measuring a given factor. Computationally,
Note. * Indicates significant at p < 0.05 level. composite reliability is the square of the sum of the stan-
dardized factor loadings for that factor divided by the sum of
Table A6. Discriminant Validity Tests: Software
the error variance of the individual indicator variables and
Development Risk Constructs
the square of the sum of the standardized factor loadings
(Fornell & Larcker, 1981). The composite reliability for each
that the relationship among first-order factors is sufficiently risk factor is shown in Table A4. In addition, the Cronbach
captured by the higher-order factor. In this study, the alpha scores were calculated and are reproduced in Table A4.
observed Chi-square for the baseline model was 192.54 The Cronbach alpha and composite reliability scores for
degress of fiction = 137; p = 0.001) and the second-order fac- each construct are acceptable.
tor model is 216 (degrees of freedom 146; p = 0.0001). If all factor loadings for the indicators measuring the same
Adjusting for degrees of freedom, the normed value of Chi- construct are statistically significant (greater than twice their
square = 1.40 for the baseline model and Chi-square = 1.48 standard error) there is evidence supporting convergent validi-
for the higher-order model, indicating good model fit and ty of those indicators (Anderson & Gerbing, 1988). The fact
no evidence of over-fitting. The calculated target coefficient that all t-tests are significant shows that all indicators are effec-
between the baseline and hypothesized model is a very high tively measuring the same construct. The t-values for each indi-
0.95. This value suggests that the addition of the second- cator loading are shown in Table A4. The constructs demon-
order effect does not significantly increase Chi-square. strated acceptable convergent validity.
Reliability, Convergent Validity, and Discriminant Validity. Evidence of discriminant validity is obtained through the
External validity refers to the extent to which the findings can comparison of an unconstrained model that estimates the cor-
be generalized to or across times, persons, and settings relation between a pair of constructs and a constrained model
(Campbell & Cook, 1979). External validity of the findings is that fixes the value of the construct correlation to unity. The
threatened if the sample itself is systematically biased, e.g., if resulting Chi-square difference also is a Chi-square variate with

40 Project Management Journal September 2002

 one degree of freedom. A significant difference means the
unconstrained model is a better fit. This condition supports the
existence of discriminant validity (Anderson, 1987; Bagozzi,
Yi, & Phillips, 1991). Table A6 contains the results of the pair-
wise difference tests among constructs. As shown, all differ-
ences are significant at p < 0.001 levels. Hence, each scale
seems to capture a construct that is significantly unique from
other constructs, providing evidence of discriminant validity.

James J. Jiang, PhD, is a professor of management

information systems (IS) at the University of Central
Florida. He obtained his PhD in IS from the University
of Cincinnati. His research interests include IS project
management and IS personnel management. He has published more
than 70 academic articles in journals such as the IEEE Transactions
on Systems, Man, and Cybernetics, Decision Support Systems, IEEE
Transactions on Engineering Management, Decision Sciences, Journal
of Management Information Systems (JMIS), Communications of the
ACM, Information & Management, Journal of Systems & Software,
Data Base, and Project Management Journal. He is a member of the
Institute of Electronic and Electrical Engineering, Association for
Computing Machinery, Association for Information Systems, and
Decision Science Institute.

Gary Klein, PhD, is the Couger Professor of information

systems (IS) at the University of Colorado at Colorado
Springs. He obtained his PhD in management science
from Purdue University. He served with Arthur Andersen
& Company in Kansas City and was director of the IS department for a
regional financial institution. His interests include project management,
knowledge management, system development, and mathematical
modeling with more than 60 academic publications in these areas.
He has made professional presentations on decision support systems
in the United States and Japan, where he served as a guest professor
to Kwansei Gakuin University. He is a member of the Institute of
Electronic and Electrical Engineering, Association for Computing
Machinery, Institute for Operation Research and the Management
Sciences, Society of Competitive Intelligence Professionals, Decision
Science Institute, and Project Management Institute.

T. Selwyn Ellis is an assistant professor of computer

information systems (IS) at Louisiana Tech University.
He earned a BS in computer science and mathemat-
ics and an MBA from Mississippi College and a DBA
from Louisiana Tech University. He has publications in Journal of
Computer Information Systems, Data Base, Computer Personnel, and
others. His current research interests include investments in technol-
ogy, electronic commerce, and other IS/information services topics
related to corporate strategy.

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.
A Hybrid Intelligent System
to Facilitate Information System
Project Management Activities
▼ Project management has long been viewed as a domain where automation is lim-
Abstract
In most large companies, management
has long recognized a deficiency in project
management and quality assurance (QA)
concerning information systems development
and maintenance projects. This paper provides
a practical, prototypical artificial intelligence-
based approach to solving these ill-structured
problems. The authors show how a hybrid
intelligence system with expert system and
artificial neural network components can be
used to aid in project estimate validation and
quality prediction of the deliverables. The
output from such a hybrid system then can
be used in a traditional project management
timeline tool for collecting the estimate infor-
mation, tracking the project through its com-
pletion. The authors also provide guidelines
for large-scale development of such systems.
Keywords: artificial intelligence; expert sys-
tems; hybrid systems; information systems;
neural networks; project management;
quality assurance
©2002 by the Project Management Institute
2002, Vol. 33, No. 3, 42–52
8756–9728/02/$10.00 per article + $0.50 per page
42
Hamid R. Nemati, Department of Information Systems and Operations Management,
The University of North Carolina at Greensboro, 440 Bryan Building, P.O. Box 26165,
Greensboro, NC 27014–6165 USA
Dewey W. Todd, Department of Decision Sciences, Georgia State University, University Plaza,
Atlanta, GA 30303–3083 USA
Paul D. Brown, Department of Decision Sciences, Georgia State University, University Plaza,
Atlanta, GA 30303–3083 USA
I n most large companies, management has long recognized a deficiency in
project management and quality assurance (QA) concerning information sys-
tems development and maintenance projects. Traditional “seat of your pants”
management practices and scattered innovative project management techniques
have been limiting at times. This paper provides a practical, prototypical artificial
intelligence (AI)-based approach to solving these ill-structured problems. It covers
two key areas of need for most large businesses and their information technology
(IT) organizations: a hybrid network of systems proposed to facilitate project
management and IT-associated activities of project estimation, tracking, develop-
ment, testing and implementation, and QA.
ited to some type of timeline tool, if that much. When starting this project, the
authors asked, “Why?” It appears that this domain has resisted AI for the very reason
that AI was developed. The newness of AI sometimes makes potential recipients of
AI systems skeptical of their possible benefits. Project management requires so much
human reasoning, and each project typically bears so much uniqueness, that on the
surface, developing a system to mimic that process seems impossible. To show the
potential power that AI could bring to this vital arena, this paper will present the
development of a prototypical hybrid intelligence system. This system consists of the
expert system (ES) to aid in project estimate validation and two artificial neural net-
work (ANN) models to be used for delivery and quality prediction.
This paper first will provide an overview of project management and QA. Next,
the research process for the ES component of this hybrid system to aid in project esti-
mate validation will be described. The authors also will describe the database and
research process for developing the neural network models used for predicting the
delivery time and quality of the final deliverable. The authors also will provide a
breakdown of the project management process and analysis of how AI systems can
be used to develop a hybrid network of systems to support this process. The last part
of this paper is a detailed description of the total project management picture and
how these they can be supported through the development of an architecture of AI
systems similar to the one described in the paper.
Project Management: A Historical Perspective
Project management is the planning, organizing, directing, and controlling of
company resources for a relatively short-term objective that has been established to
Project Management Journal September 2002
complete specific goals and objectives (Kerzner, 1995).
Producing high levels of productivity and quality and low lev-
els of uncertainty also are objectives of project management. It
is the management of all the factors that surround and enable
the technical work to be accomplished.
Project management is an organized or structured approach
for managing a variety of independent and interdependent
events and activities leading toward a common outcome. The
objectives of the approach are to:
■ Complete planned activities according to a stated schedule;
■ Deliver the completed project or product on time, with
minimal slippage;
■ Manage the costs of the process to ensure attainment or
reduction of budgetary projections;
■ Monitor the results of the process to ensure that the purpose
and benefits of the system or project have been accomplished.
Project management aims to bring a project to completion
on time, within budgeted cost, and to meet the planned per-
formance by synthesizing all resources assigned to the project
effectively and efficiently (Simpson, 1987). Project manage-
ment assists management in defining the proper level of qual-
ity and then acts to ensure the project delivers a product of that
quality. A plan is based on estimates of the size and duration
of activities, and estimates are based on probabilities
(Kitchenham, 1991).
Making estimates is necessary to predict cost and time
requirements for the project (Ellwood & Maurer-Williford,
1994). However, estimating the effort or costs required for a
project is difficult because there are always many unknowns
(Jeffrey & Low, 1990). According to Finnie and Wittig
(1997), estimating software development remains a
complex problem and improving estimation techniques
available to project managers would facilitate more effective
control of time and budgets.
There are five bases for estimation (McDermid, 1990).
They are professional judgment, historical database, and the
use of experts, standard times, and formulae. Very few com-
panies have bothered to store previous project data in a way
that can be used later; therefore, a historical database often
cannot be used. But if such a historic database were available,
project estimation would be enhanced greatly (Jeffrey & Low,
1990). Standard times are factors that have been constructed
by analyzing many previous project times so that a standard
time is obtained.
Research has shown that the reliability and accuracy of exist-
ing models and tools is very limited (Bennatan, 1995; Ellwood
& Maurer-Williford, 1994; Finnie & Wittig, 1997; Mcleod &
Smith, 1996; Srinivasan & Fisher, 1995). Results from cost-esti-
mating models do not correlate very well with each other when
tried on a common problem (Macro, 1990). Lines of code, a
metric used for estimating software development efforts, has a
number of problems (Albrecht & Gaffney, 1983). One prob-
lem is the difficulty of language dependence. It is not possible
to directly compare projects developed by using different lan-
guages (Jeffrey & Low, 1990). A second problem is that it also
is difficult to estimate the number of lines of code that will be
needed to develop a system.
September 2002 Project Management Journal
The project plan is the primary tool used for any project
undertaken. The purpose of the project plan is to provide the
foundation and framework for the project. The main elements
of a standard project plan are project tasks and activities, a
responsible person, weeks required, beginning date, target and
actual completion dates, and status. Project plans ensure that
all activities are recorded and accounted for. When developing
the outline of activities to be done, the project manager and
team must consider the scope of the project to know the
boundaries within which to define what must be accom-
plished. The development of this portion of the project plan
will involve the identification of activities, events, and mile-
stones. Activities relate to the physical work that must be
undertaken to complete the desired outcome. Events are cate-
gories of activities that summarize the result of the activities
into a final planned outcome, and a milestone is an event very
important to the project.
Once these objectives have been defined, the beginning
date and target completion date must be determined. Both
dates are targets or planned events. They must be planned
and recorded to manage the entire process. A third date is
the actual completion date. The actual completion date is
used to compare the planned timing with the actual, thus
enabling closure and a measure of performance as to slip-
page (Taff, Borchering, & Fisher, 1995). Slippage shows
the time by which the actual completion date exceeded the
planned target date.
Planning entails the future, and in dealing with the future,
project managers deal with uncertainty (Kitchenham, 1991;
Matson & Mellichamp, 1994). Often, estimates are quite
rough because what is supposed to be done has never before
been done in precisely the same way (Taff, Borchering, &
Fisher, 1995). It is important that project teams recognize
how uncertainty bears on the planning effort. The level of
uncertainty of the proposed project largely determines the
character of the plan. Good planning may mean phased
planning, in which planning is divided into phases over the
duration of the project.
One reality of project management is the threat that project
costs will be exceeded. To cope with this threat, project man-
agers commonly build some “fat” into their cost estimates.
Project costs typically are composed of four components:
direct labor costs, overhead, fringe benefits, and auxiliary
costs. In most service projects, which are not capital intensive,
direct labor costs are the largest single component of project
costs. Two approaches to cost estimating are the bottom-up
cost estimation procedure and parametric cost estimating. In
bottom-up cost estimation, if project managers know the
labor costs, they can make good estimates of total project
costs. Here, they estimate how much labor is needed to carry
out project tasks.
A Hybrid Intelligence System
for Project Management
SunTrust Bank Inc. is one of the largest regional banks in the
United States with total assets in excess of $80 billion.
Like many other large businesses, it maintains a large IT
43
 Similar Project
Done in the Past? How Comfortable
Yes No
With Estimates?

Very Fairly Slightly

Any Tasks
> 80 Hours?
Enter Percentage
of Total Estimate
Yes
for Which Tasks are
No > 80 Hours

What is the
Expected Duration
< 2 Weeks of the Project?
2–4 1–3
> 3 Months
Weeks Months

How Often Will

it be Reviewed by
At Least Management?
Once/
Once Every Rarely
Month
2 Weeks

What is the What is the

Experience Level of Experience Level of
the Estimator the Programmer
(Scale of 0 to 100)? (Scale of 0 to 100)?

Acceptable Estimate
Acceptable Estimate Highly Questionable
Add 15% Management Acceptable Estimate
Adjust Management Estimate – Needs
Overhead Adjust Contingency
Overhead to be Reviewed
Add 10% Contingency

Note. The flow diagram begins with the box labeled “Similar project done in the past?” It then flows through the qualifiers and variables and finally to
the choices. The flow diagram does not attempt to display or determine any weights used in the decision process. All alternatives are treated equally
and displayed as such.

Figure 1. Flow Diagram Showing the Influences and Relationships Among Qualifiers and Variables with the Choices

44 Project Management Journal September 2002

organization called Application Systems Division (ASD). ASD
is a part of SunTrust Service Corp. (STSC), which is a wholly
owned subsidiary of Sun Trust Bank Inc. In the past, organi-
zations such as ASD found it much easier to maneuver
through IT requirements than is the case today with fast-
moving and complex technical environments. Continual
change combined with large size means that having control
over all IT efforts is vital. This control is not possible without
the previously mentioned standardization. It means tactical-
ly utilizing available tools and maintaining a strategic vision
for additional tools and methods.
One of the problems with managing projects at ASD was
a lack of consistency. Even if standards were implemented
for estimating, logging, and tracking projects, differences in
managers’ leadership styles would lead to misinformation
and a general breakdown in comparing and planning future
projects. In 1996, ASD assembled a team consisting of man-
agement and project management experts. The ultimate goal
for the team was to build a strategy of cost reduction through
process improvement. It focused on standardization, faster
development times, higher quality systems delivery, and on-
time delivery.
This team, whose establishment was mandated by senior
executives at STSC, was a mixture of internal employees and
contract consultants and was charged with the development of
an efficient and effective project management methodology for
all STSC projects. The resulting methodology was labeled
Vision Methodology (STSC, 1997) and mostly consists of doc-
umented methodologies, system standardization (using
Microsoft Project®), and periodic hiring of external experts to
help guide major projects. Vision was a documented method-
ology whereby STSC could create more consistency within IT
departments surrounding the management and implementa-
tion of IT projects.
There are six major phases in the methodology: definition,
requirements, analysis, design, development, and implementa-
tion. Each of these major phases is subdivided into major
activities. Part of the process, for example, is definition. Within
this phase, all projects must be defined along the same steps
and activities. This allows for common entry into the appro-
priate project management programs (in the current case,
Microsoft Project®). It also streamlines management reporting,
project tracking, and statistical analysis. As steering committees
and managers review requested projects, the standardized for-
mat gives them a common point of reference with which to
compare the benefits and costs.
Upon reviewing this methodology, the authors sensed a
potential application for AI solutions. The idea of AI is not
new. In fact, there are numerous examples available where AI
solutions successfully have applied to practical problems.
However, in the current situation, the ultimate objective of this
AI-based approach was to facilitate project management activ-
ities as outlined in the Vision document. The approach pre-
sented in this paper for creating a hybrid network of systems
can provide support for achieving the overall goals of the
desired methodologies requested by management. However,
the AI-based approach presented also can facilitate managing
September 2002 Project Management Journal
projects in situations lacking the overall development method-
ologies similar to Vision (STSC, 1997).
For example, at first glance, one can envision using an ES
for estimating projects. However, this approach may not be
realistic. The rules involved would be extremely numerous and
complex. This would hamper development of the system and
make ongoing maintenance unattractive to a potential user
organization. Instead, the authors classified a smaller portion
of the process, namely, validation of provided estimates, as
suitable for an ES. Others, including a separate ES designed to
analyze and profile team members, support even that system.
Yet another ES is envisioned for determining project quality.
The total AI system approach, though, is much bigger.
The authors feel that the addition of a neural network to
analyze the database of project information over time and pre-
dict delivery time and quality is more viable than the results
that could be expected from an ES. This approach makes per-
fect sense as an “umbrella” system under which to develop the
overall project management schema. This intelligent tool is
exceptional in managing domains where records in a database
are similar, yet typically different in many ways, and where
new, better solution is preferred over a near-match alone. It has
that capability. The authors believe an effective, maintainable
project management and quality assessment environment can
be properly administered and applied to the typical IT shop.
There usually is a wealth of general domain knowledge
available to help with estimating. This almost certainly could
be developed into some sort of rule set for validation of esti-
mates and, perhaps, for the estimating itself. For example, an
ES can be used to allow management to develop profiles on
each employee making up project teams, regarding experience
and knowledge levels, and a different ES can allow a manager
to evaluate an estimate done by an employee and validate the
estimate based on certain domain knowledge rules. A neural
network also could support this function once the database of
completed projects is large enough. A traditional project man-
agement tool for collecting the estimate information, tracking
the project through to completion, and recording project sta-
tistics into a database can be an integral part of such system. An
ES will allow some task force, e.g., a user group, to evaluate and
score a delivered application for quality. Again, a neural net-
work also could support this function once the database of
completed projects is large enough.
This paper demonstrates how AI tools could be used to
facilitate project management activities. This paper
also describes the development of a prototype hybrid intel-
ligence system as proof of concept. For this prototypical
implementation, the authors developed a hybrid
intelligence system that consists of an ES to aid in project
estimate validation and two ANN models to be used for
delivery time and quality prediction.
Description of Expert System Model to Aid in Project
Estimate Validation. An ES is a system that uses human
knowledge captured in a computer to solve problems that
ordinarily require human expertise (Turban & Aronson, 2001).
ESs are used today by most large- and medium-sized organiza-
tions as a tool for improving decision-making as well as
45
 Similar project The choices are yes or no. The letters “NC” indicate that this choice will not affect
the weighting. A negative value means this selection reduces the associated
“preferable choice” by that number (scale of 1 to 10) and a positive number
means it increases a “non-preferred” choice by that number.

Comfort level These weights only come into play if “Similar Project” is No. The numbers 1 through
4 correspond to the four selections.

Tasks > 80 This selection by itself does not impact the weights.

Percentage > 80 If there are tasks greater than 80 hours, the choice from the 0 to 100 range
is divided into four discrete weights and applied.

Duration This selection by itself does not impact the weights.

Review frequency These weights only come into play if “duration” is greater than two weeks.
The numbers 1 through 4 correspond to the four selections.

Experience of estimator The choice from the 0-to-100 range is divided into five discrete weights and applied.

Experience of programmer The choice from the 0-to-100 range is divided into five discrete weights and applied.

Table 1. Summary of Each Possible Choice and How It Would be Influenced by Each of the Qualifiers and Variables

increasing productivity and quality. The development of an ES
involves the construction of a problem-specific knowledge
base by acquiring knowledge from experts or documented
sources. The most widely used form of representing knowledge
in the knowledge base is using rules. The system then uses the
rules in the knowledge base to draw conclusions similar to
those drawn by the experts.
The first component of this intelligence system is an ES. It is
designed to help management validate estimates of the deliv-
ery time and quality of a given project. After extensive review
of project management activities and conducting interviews
with a number of project leaders, a set of rules was compiled.
This was accomplished according to steps:
1. The qualifiers used to decide the accuracy of the estimate
were written up;
2. The variables that would be used with the qualifiers were
determined;
3. The possible choices were determined;
4. A flow diagram was developed to show how these elements
would interact to move from qualifying to choosing (Figure 1);
5. A table was developed (using Lotus 1-2-3® ) as a matrix of
the possible effects of each qualifier’s choice and the range of
values in the variables and how they would affect the certainty
factor for each choice (Table 1 and Table 2);
6. The elements were all entered into EXSYS Professional®
according to the matrix;
7. The functional screens were developed using EXSYS Editor
and an editor called Brief®;
8. Multiple iterations of sample estimates were tested to vali-
date the system.
Rule Set Definition. The elements that make up the ES for
project estimate validation and contingency are shown in
Figure 1. The qualifiers for this exercise are:
1. Was the estimate developed from a similar past project
(similar implies that at least 40% to 60% of the tasks would
be roughly similar in nature)? A yes to this question takes the
user to Qualifier 3, otherwise to Qualifier 2;
2. How comfortable are you with the estimates considering
the lack of familiarity with the tasks? The user has four
“fuzzy” options here ranging from “very comfortable” to
“uncomfortable.” These responses have a weighted effect on
the outcome;
3. Are any of the tasks in the project estimated to be more
than 80 hours? There is an “80 hour rule” that basically says
that any estimate more than 80 hours is wrong (imprecise).
The authors’ domain expert recommended that tasks greater
than 80 hours be broken down, if possible. If no, the user
continues to Qualifier 4, otherwise to Variable 1 to determine
a percentage;
4. What is the expected duration of the project? This question
is asked to determine the need for management review meet-
ings. It has been shown conclusively that frequent reviews pos-
itively influence the overall development time because delays
are caught and corrected early. If the project duration is less
than two weeks, the user proceeds to Variable 2. There are three
other choices ranging from “two to four weeks” to “greater than
three months.” These values do not currently affect the weight-
ing but may in the future. The user proceeds to Qualifier 5;
5. How often will the project be reviewed by management?
If once a week is selected, there is no weighting, but the other

46 Project Management Journal September 2002

three choices, ranging from “every two weeks” to “rarely” do
weigh accordingly.
The variables that affect ES estimate validation and con-
tingency are:
1. Enter the percentage of the total estimate, which the tasks
over 80 hours represent. This is a slide bar, which ranges from
0 to 100. The range is divided into four discrete points and
weighted appropriately;
2. What is the experience level of the estimator (scale from
0 to 100)? This is a slide bar that allows the user to evaluate
how closely he or she feels the estimator would come to the
true estimate. This is broken down into five discreet ranges and
is used to weight the outcome appropriately;
3. What is the experience level of the programmer (scale
from 0 to 100)? This is a slide bar that allows the user to
evaluate how closely he or she feels the programmer will
come to the estimated development time. This is broken
down into five discreet ranges and is used to weight the out-
come appropriately.
The choices that go into ES estimate validation and con-
tingency are:
1. Acceptable Estimate. This implies that the estimate seems
accurate, with a certainty factor ranging from 1 to 10;
2. Add 15% management overhead and 10% contingency.
These values are standard within ASD for an accurately esti-
mated project. The assumption is that direct supervisors and
middle management also will charge time to the project for
management, and the contingency allows for typical slow-
downs and interruptions. This choice always will accompany
Choice 1, but its certainty factor may differ;
3. Adjust management overhead to 20%. This, too, will
accompany Choice 1, but weighting factors surrounding the
previously selected qualifiers may introduce the need to
increase the predicted management time;
4. Adjust management overhead to 25%. Same as Choice 3;
5. Adjust management overhead to 30%. Same as Choice 3;
6. Adjust contingency to 15%. This, too, will accompany
Choice 1, but weighting factors surrounding the previously
selected qualifiers may introduce the need to increase the con-
tingency provision;
7. Adjust contingency to 20%. Same as Choice 6;
8. Adjust contingency to 25%. Same as Choice 6;
9. Highly questionable estimate; needs to be reviewed. This
choice suggests that the weighting factors are not in favor of
this estimate being accurate. Its lack of accuracy may dictate the
need to re-estimate the project with better experience and man-
agement involvement.
Rule Set Matrix. A rule set matrix was developed for this ES.
This matrix is used to define each of the possible choices and
how each qualifier and variable would influence their weight-
ing. The weighting is explained individually in Table 1.
The Neural Network Models for Predicting
Delivery Time and Level of Quality
Artificial neural networks recently have attracted the attention
of researchers from various disciplines including computer
science, psychology, mathematics, physics, and operations
September 2002 Project Management Journal
research/management science. They have been applied suc-
cessfully to solve many practical problems, such as times
series predictions, classification, pattern recognition and
mathematical optimization, and many more. During the past
decades, a number of paradigms of ANNs have been intro-
duced to carry out a wide variety of computational tasks, such
as the feed-forward ANNs (Chauvin & Rumelhart, 1995;
Rumelhart, Hinton & Williams, 1986), Hopfield neural net-
works (Hopfield & Tank, 1985), the Boltzmann Machine
(Aarts & Korst, 1989; Hinton, & Sejnowski, 1983), and the
Self-Organizing networks (Kohonen, 1988). In this section,
the authors provide an overview of ANNs as applied to this
project. For a more comprehensive and detailed description
of ANNs, their learning algorithms, and different network
topologies, the authors recommend Rumelhart, Hinton, and
Williams (1986) and Wasserman (1989).
ANNs are biologically inspired systems of highly intercon-
nected simple processing elements that achieve learning. These
system are composed of many simple processing elements,
called neurons, operating in parallel whose function is deter-
mined by network structure or the way neurons are connected
to each other, connection strengths between neurons, and the
information processing that is performed at each neuron. Each
neuron is connected to other neurons by means of intercon-
nections or links with associated weights. Signals are passed
between neurons over these interconnecting links. The net
input to a neuron is the weighted sum of all the signals it
receives from all other neurons to which it is connected. The
vector set of inputs to neuron, j, is (X1j through Xnj), where Xij
is the signal sent from neuron i to neuron j. This input vector
is multiplied by the weight vector of interconnections (W1j
through Wnj), where Wij is the strength of connection between
neuron i and neuron j. The resulting Cartesian product is
referred to as the action potential. The action potential or the
weighted sum of the inputs to the neuron is transformed using
a mathematical function called the activation function. When
the activation function reaches a given level, the neuron fires
and sends a message to the other neurons connected to it.
Memories are stored or represented in a neural network in
the pattern of interconnection strengths among the neurons.
Learning is achieved by changing the strengths of these inter-
connecting weights. Therefore, to train a neural network, a
method is needed to allow the network to modify its weights
such that a desired outcome is achieved.
The training method is a mathematical function that
updates the strength of the interconnecting weights such that
the difference between the desired output of the network and
its actual output is minimized. To train a neural network, a ran-
dom set of weights is assigned as the initial set of the intercon-
necting weights. A training set of data consisting of the inputs
and the corresponding outputs then are presented to the
network. The network is allowed to generate its output based
on its current set of weights. The output from the network then
is compared to the actual output and the sum of squared of
errors is calculated.
In each succeeding iteration, the network attempts to
minimize the sum of squared errors by adjusting its weight
47
 Choice Accept Add 15% mgmt Adjust Adjust Review
overhead and management contingency completely
10% contingency overhead

Similar project? Y = NC Y = NC Y = NC Y = NC Y=2

N = –3 N = –3 N=4 N=5 N=3

Comfort level 1 = NC 1 = NC 1 = NC 1 = NC 1 = NC
2 = –2 2 = –2 2=3 2=3 2=3
3 = –6 3 = –6 3=8 3=8 3=8

Tasks > 80 NC NC NC NC NC

Percentage > 80 0 – 33 = –1 0 – 33 = –1 NC 0 – 33 = 2 0 – 33 = 2
34 – 66 = –3 34 – 66 = –3 34 – 66 = 4 34 – 66 = 4
> 66 = –6 > 66 = –6 > 66 = 8 > 66 = 8

Duration NC NC NC NC NC

Review frequency 1 = NC 1 = NC 1 = NC 1 = NC 1 = NC
2 = –2 2 = –2 2=3 2=3 2=3
3 = –6 3 = –7 3=7 3=8 3=8

Experience of estimator 0 – 33 = –7 0 – 33 = –7 0 – 33 = 7 0 – 33 = 7 0 – 33 = 8
34 – 66 = –3 34 – 66 = –3 34 – 66 = 3 34 – 66 = 3 34 – 66 = 4
> 66 = NC > 66 = NC > 66 = NC > 66 = NC > 66 = NC

Experience of programmer 0 – 33 = –7 0 – 33 = –7 0 – 33 = 7 0 – 33 = 7 0 – 33 = 8
34 – 66 = –3 34 – 66 = –3 34 – 66 = 3 34 – 66 = 3 34 – 66 = 4
> 66 = NC > 66 = NC > 66 = NC > 66 = NC > 66 = NC

Table 2. Impact on Choices

set using the training algorithm. The adjustment to the
weight set can be done in either a single update, in which the
weights are changed every time the network is presented
with a new case data or an epoch update, in which the net-
work changes the weights only after it receives the whole
training data set. For faster training, it is possible to specify
an epoch size. For example, an epoch size of five means that
the network adjusts the weights only after it has been pre-
sented the entire training set data five times. The training
continues until a prespecified condition is reached. Training
a neural network is an iterative process. The developer goes
through the designing system, training the systems and eval-
uating the system repeatedly.
In a neural network, neurons are grouped into layers, or
slabs, that include neurons of the same type. There also are dif-
ferent types of layers. The input layer consists of neurons that
receive input from the external environment. The output layer
consists of neurons that communicate to the user or external
environment. The hidden layer consists of neurons that only
communicate with other layers of the network. The authors
define a multilayer neural network as a network in which the
interconnections between more than the two I/O layers of the
network change in the learning process.
The network has additional layers, hidden from the envi-
ronment. The most widely multilayer neural network used
network today is the Backpropagation neural network
(Chauvin & Rumelhart, 1995; Rumelhart, Hinton, &
Williams, 1986). The basic idea in a Backpropagation net-
work is that the neurons of a lower layer send their outputs
up to the next layer. It is a neural network that is very good
in generating input-to-output mappings based on computa-
tions of interconnection of its nodes. It is a fully connected,
feed forward, hierarchical multilayer network, with hidden
layers and no intraconnections.
Figure 2 shows a typical Backpropagation neural network.
The network shown has four input neurons, four output neu-
rons, and a hidden layer with three neurons. The weights
between output layer and the hidden layer are updated based
on the amount of error at the output layer, using generalized
delta rule (an iterative steepest descent procedure, minimizing
the least-mean-square error measure between the desired and
actual outputs). The error of the output layer propagates back

48 Project Management Journal September 2002

to the hidden layer via the backward connections. The connec-
tion weights between the input layer and the hidden layer are
updates based on the generalized delta rule as well.
There is a forward flow of outputs and a backward update
of the weights based on the error on the output layer. During
the training process, the network seeks weights that mini-
mize the sum of the square of output layer errors. There is a
backward connection from the output layer to the hidden
layer, from the hidden layer to the next lower hidden layer,
and from the lowest hidden layer to the input layer. The
backward connections between each pair of layers have the
same weight values as those of the forward connections and
are updated together.
Taking the parameters from the ES component described
earlier along with the additional factors that may influence
project delivery time and quality level of the resulting deliver-
able from the project, a database was constructed. This data-
base was used to train two ANN models. This database con-
sisted of relevant data from 200 previously completed projects.
The fields contained in the database are derived from project
information that is typically gathered and/or available on any
given project. The data set initially contained 57 variables.
Nonparametric methods, i.e., neural networks, tend to gen-
eralize better from fixed-sized data sets if the dimensionality of
the data set is lowered without losing significant information
as reflected by the relationships in the data set. A common
method used to reduce the dimensionality of the data set with-
out significant loss of information is principle component
analysis (PCA). In essence, PCA reduces the input set from n to
m by identifying an m-dimensional subset of the n-dimen-
sional input space that seems to be the most significant. Using
PCA, the authors reduced the initial 57 variables to a final set
of 15. The model consisted of 15 input variables and one out-
put variable for each model. The inputs are designed to pro-
duce an estimate of a project score given a set of new inputs.
For each of the 200 records in the database, 15 fields were
eventually selected:
■ Project Number. This field identifies each record with a
four-digit project number that is typically used by SunTrust to
identify projects for tracking and billing purposes. It is not used
in the neural network (ANN);
■ Project Manager. This is the last name of the application
manager assigned to the project. This person is responsible for
estimation, approval, development, testing, and implementa-
tion and follow up. For the ANN, only four fictional managers
were used, and their names were substituted for numbers (1 =
Smith, 2 = Jones, 3 = Carlisle, 4 = Free);
■ Estimate Date. This is the date the estimate was completed;
it is not used in the ANN;
■ Estimated Hours. This is the total estimated actual hours,
including documentation, development, testing, and imple-
mentation;
■ Management Overhead. This value (entered as a percent-
age) is an additional percentage of the overall project estimate
that will be used for managers’ hours that are not directly
involved but are directly responsible for the project team
assigned to the project. This includes the project manager. This
September 2002 Project Management Journal
value is typically set to 15%, however, it may be set to a higher
figure if the project manager feels the project will demand
more time than normal, i.e., an inexperienced lead program-
mer, and on some rare occasions may be set to less;
■ Contingency. This value (entered as a percentage) is an
additional percentage of the overall project estimate that is
set aside for normal project additions, i.e., unplanned pro-
ject meetings, requirements’ discrepancies, etc. It is typically
set at 10%;
■ Adjusted Estimate. This is the estimated hours plus the
management overhead and contingency factored hours;
■ Comfort Level. This field takes on a value from 1 to 3, with
1 indicating “Very Comfortable,” 2 indicating “Comfortable,”
and 3 indicating “Slightly Comfortable;”
■ Percent > 80. This is a value representing what percentage
of the project tasks have been estimated to be greater than 80
hours. The lower the percentage the better, because any task
greater than 80 hours automatically can be assumed to be
wrong to a certain extent;
■ Number of Reviews. This is a value representing how often
the project will undergo reviews: 1 = At least every two weeks,
2 = At least monthly, and 3 = Rarely;
■ Estimator Experience. This is a percentage value represent-
ing the estimator’s experience level. It ranges from 0 to 100%,
with 100% being the most experience possible;
■ Programmer Experience. This is a percentage value repre-
senting the lead programmer’s experience level. It ranges from
0 to 100%, with 100% being the most experience possible;
■ Actual Hours. This is the total number of hours charged to
the project, including management overhead. It is factored in
as part of the output for the neural network;
■ Quality Rating. This is a rating that the authors arbitrarily
assigned for this project. It, too, is factored in as part of the pre-
dictive output for the neural network. It is initially entered as a
normalized value from 0 to 1 but is converted to a value from
0 to 100;
■ Score. This is a compilation score of the overall project and
represents the output for the ANN. It is constructed using the
following formula:
[(Adjusted Hours / Actual Hours) x Quality Rating x 100] (1)
Of these fields, comfort level, percent > 80, number of
reviews, estimator experience, and programmer experience
are directly from the ES project. Actual hours and quality
rating are follow-up values entered after the project’s
completion. The quality rating is conceptual, but an ES for
evaluating and determining a quality rating is quite neces-
sary to valid project estimation.
The authors constructed neural network models to predict
the delivery time and quality for a given project. The neural
network models developed for this project were
Backpropagation neural networks consisting of an input layer,
a hidden layer with 10 neurons, and output layer. The authors
used one hidden layer with multiple neurons in this study
because this configuration is sufficient to approximate any
continuous function to an arbitrary precision (Hornik,
49
 Input
Layer
Hidden
Layer
Figure 2. Typical Backpropagation Artificial Neural Network
Stinchcombe, & White, 1989). To build the prediction model
for “Score,” building two prediction models actually was
necessary. The first, which in theory was based on the vari-
ables “manager” through “programmer experience,” was
developed to predict actual hours, because this is, of course,
important in predicting how close the actual hours will
come to the estimate. For example, a project that completed
exactly on time and achieved a perfect quality rating would
have a score of 100. A score of less than 100 would indicate
a project that suffered from quality of less than 100%, actu-
al hours exceeding its estimate, or a combination of both. A
score higher than 100 indicates a project with high quality or
with actual hours that were less than estimated, or some
combination of both.
The authors experimented with using ANN models to
predict the actual hours required for a project. This neural
network model consisted of the 15 input variables and one
output variable. In training the network, the authors
employed the BackPack Neural Network software package
using its default settings. The network had one hidden layer
with 10 neurons, an epoch size of five, and 20,000 as the
maximum number of iterations. Training the network
using the training set and testing it using the testing data
set, the authors obtained a high correlation between the
model and the predicted actual hours [correlation coeffi-
cient was 0.9895 and root mean squared errors (MSE) was
0.03350]. The results indicates that the neural network
model is a powerful predictive tool.
50 Project Management Journal
Output
Layer
Because this produced a satisfactory network for predicting
actual hours, the authors then developed the second neural
network, using the same database. This one was called “Score”
and contained a neural network for predicting the score of a
given project based on all of the previous variables plus the
actual hours. Here, quality is an input from the user, because
no ES has yet been developed for this function. The network
using the training set and testing it using the testing data set,
the authors obtained a high correlation between the model
and the predicted actual hours (correlation coefficient was
0.738 and root MSE was 0.0898).
Future Development of Hybrid Systems
In this section, the authors provide a vision for future devel-
opment of intelligence hybrid systems for project manage-
ment and QA of information systems projects. The authors
provide a breakdown project management/QA processes
and analyze the details of development of individual com-
ponents of an AI-based hybrid system that can be used to
support these processes.
The steps that are typically taken in an IT development
project and the associated recommended AI systems for each
piece are:
■ Team Member Profiles. Recommended AI Approach:
Expert System. This is a crucial step in developing a tactical
estimating system. In studying project management, one
does not have to delve too deeply to realize how much influ-
ence the make-up of a team has on the completion of the
September 2002
project. If a highly experienced individual develops the esti-
mates for the tasks, these estimates may be very different from
the actual hours of development done by a very inexperi-
enced programmer. This process can be very limited (as
shown in the ES example the authors developed as part of
this project) or very in-depth under which managers’ and
team members’ cognitive evaluations, experience levels, time
availabilities, work progress ratings, etc. are taken into
account. The result is an overall standardized score for the
team that helps to evaluate the estimate;
■ Problem Sensing. Recommended AI Approach: This is
the origination of the idea for a system or system change; no
AI solution is recommended. Someone must discover the
need for a new system or enhancement, calling for a potential
project. This is not a function that realistically can be relegated
to an AI system;
■ Ballpark Estimate. Recommended AI Approach: This
would typically be done by a human, because little is known
about realistic requirements or potential team members.
This is typically done for initial funding approvals. Because
user requirements probably have not been developed by this
point, this responsibility would best be left to humans. This
estimate usually is not used to limit funding or measure per-
formance, but is merely a source for making a tentative deci-
sion to go ahead with a project. Once the actual estimate is
made, then funding decisions usually are finalized (this is
sometimes true only if the final estimate is beyond some
preapproved tolerance level from the ballpark);
■ Requirements. Recommended AI Approach: No AI solu-
tion initially, however, an ES could be developed to aid in
this process. This is usually an in-depth process involving both
development staff, end users, training sections, etc. It is, as with
the preceding steps, not usually considered a part of “project
management,” so probably would be handled manually. It is
covered because an ES could be developed to facilitate this
process by developing a shell based on the many similar func-
tions between projects;
■ Detailed Project Estimate. Recommended AI Approach:
This initially would be done manually, however, the eventu-
al development of an ES can be used here too. Although this
would initially have to be done manually while the base case
is being developed, it could be supported eventually by an ES
estimating tool using past projects that are similar in nature.
■ Project Validation. Recommended AI Approach: Expert
System (as discussed in this paper). Although an ES was used
here, it may be that another AI tool, e.g., a neural network,
could be better used, especially once data have been accumu-
lated from prior projects.
■ Long-term Base Case Development. Recommended AI
Approach: Neural Network (discussed in this paper),
however, the eventual development of the case-based
reasoning (CBR) case base and search tool eventually
could be used. In developing the base case for estimating, a
neural network could be valuable for establishing trends
and predictive models.
■ Quality Assessment. Recommended AI Approach:
Expert System. This may be an appropriate platform for an ES.
September 2002 Project Management Journal
A decision tree approach could be used to guide end-users,
trainers, support personnel, management, and even the devel-
opment team members in evaluating quality points, such as:
■ How smooth was the implementation?
■ How close did the deliverable meet your expectations?
■ Is the system relatively “bug” free?
■ Was training handled well considering the level of changes
or degree of complexity?
■ Prediction Case-Base. Recommended AI Approach: Case-
based reasoning system. This CBR case-base would become
the primary platform for estimating future projects.
Conclusion
In this paper, the authors note that, in most large companies,
management has long recognized a deficiency in project man-
agement and QA concerning information systems-develop-
ment and maintenance projects. The authors outlined the idea
of using a hybrid network of systems for supporting project
management and QA, which is an integral part of any success-
ful project management processes related to developing infor-
mation system projects.
Given the nature of the tasks involved in project manage-
ment, a hybrid intelligence approach may be a more realistic
solution than an AI-based method consisting of a single com-
ponent. The authors discussed specific project management
activities that can be supported using this hybrid approach.
These activities include project estimation, tracking, develop-
ment, testing and implementation.
The authors’ hope that the guidelines provided in this paper
can be used to harness the potentially powerful AI to support
this vital IS area. To show the potential uses of the AI techniques
in project management, the authors presented the development
of a prototypical hybrid intelligence system. This system con-
sists of the ES to aid in project estimate validation and two ANN
models to be used for delivery and quality prediction.
Acknowledgment
This research was partially supported by a Faculty Development
Research Grant from Bryan School of Business and Economics,
The University of North Carolina at Greensboro.
References
Aarts, E., & Korst, H. (1989). Simulated annealing and
Boltzmann machines: A stochastic approach to combinatorial opti-
mization and neural computing. New York: John Wiley & Sons Inc.
Albrecht, A., & Gaffney, J., Jr. (1983). Software function,
source lines of code, and development effort prediction: A soft-
ware science validation. IEEE Transactions on Software
Engineering, 9 (4), 639–648.
Bennatan, E.M. (1995). On time, within budget software pro-
ject management practices and techniques. New York: John Wiley
& Sons Inc.
Chauvin, Y., & Rumelhart, D. (1995). Backpropagation:
Theory, architectures, and applications. Hillsdale, NJ: Erlbaum.
Ellwood, C., & Maurer-Williford, M. (1994). Rethinking
project management for systems. International Association of
Systems Analysts, 4 (1), 111–117.
51
 Finnie, G., & Wittig, G. (1997). A comparison of software
effort estimation techniques: Using function points with
neural networks, case-based reasoning and regression
models. Journal of Systems Software, 5 (2), 281–289.
Jeffrey, D., & Low, G. (1990). Calibrating estimation tools Hamid Nemati, PhD, is an assistant professor of
for software development. Software Engineering Journal, 7 (1), information systems at the Information Systems and
215–221. Operations Management Department of The University
of North Carolina at Greensboro. He holds a doctorate
Hinton, G., & Sejnowski, T. (1983). Optimal perceptual from the University of Georgia and a master’s degree from The
inference. Proceedings of Institute of Electrical and Electronics University of Massachusetts. His research specialization is in the areas
Engineers Conference on Computer Vision and Pattern Recognition, of decision support systems, data warehousing, data mining and
Washington, D.C. New York: IEEE, 448–453. knowledge management. He has presented nationally and internation-
Hopfield, J., & Tank, D. (1985). Neural computation of ally, and his papers have appeared in number of leading journals. He
is currently finishing work on two books on organizational data mining
decisions in optimization problems. Biological Cybernetics, 52 and global knowledge management.
(1), 141–152.
Hornik, K., Stinchcombe, M., & While, H. (1989).
Multilayer feedforward networks are universal approximators.
Neural Networks, 2 (3), 359–366.
Dewey W. Todd is a doctoral candidate in decision
Kerzner, H. (1995). Project management: A systems approach to sciences at Georgia State University’s J. Mack Robinson
planning, scheduling, and controlling (5th ed.). New York: Van College of Business. He teaches courses in statistics and
Nostrand Reinhold. decision-making/problem-solving, and his research areas
Kitchenham, B. (1991). Making process predictions, software include group problem-solving, decision-making biases, organizational
psychology, organizational learning, cognitive style, and conflict manage-
metrics: A rigorous approach. New York: Chapman and Hall.
ment. He also has 18 years experience in banking, management
Kohonen, T. (1990). Self-organization and associative memory, information systems, and information systems project management.
(2nd ed.). Berlin, Germany: Springer-Verlag.
Macro, A. (1990). Software engineering: Concept and manage-
ment. Upper Saddle River, NJ: Prentice Hall.
Matson, J.B., & Mellichamp, J. (1994). Software develop- Paul Brown is a doctoral student in decision sciences at Georgia
ment cost estimation using function points. IEEE Transactions State University. His research interest is data mining, supply chain
of Software Engineering, 20 (4), 212–231. management, and electronic procurement. He is a member of
McDermid, D.C. (1990). Software engineering for information Decision Sciences Institute, Institute for Operations Research and the
systems. Oxford, UK: Blackwell Scientific Publications. Management Sciences, and National Association of Project Managers.
Mcleod, G., & Smith, D. (1996). Managing information tech-
nology projects. Cambridge, MA: Course Technology.
Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning
internal representations by error propagation. In D.E.
Rumelhart and J.L. McClelland (Eds.). Parallel distributed pro-
cessing, explorations into the microstructure of cognition I. (pp.
318–362). Cambridge, MA: MIT Press.
Simpson, W. (1987). New techniques in software project man-
agement. New York: John Wiley & Sons Inc.
Srinivasan, K., & Fisher, D. (1995). Machine learning
approaches to estimating software development effort. IEEE
Transactions on Software Engineering, 21 (2), 114–131.
SunTrust Service Corporation (STSC). (1997). Vision
methodology. Atlanta, GA: STSC Information Systems Division.
Taff, L., Borchering, J., & Hudgins, R. (1995). Estimating:
Development estimates and a front-end process for a large
project. IEEE Transactions on Software Engineering, 17 (8),
178–201.
Turban, E., & Aronson, J. (2001). Decision support system and
intelligent systems. Upper Saddle River, NJ: Prentice Hall.
Wasserman, P. (1989). Neural computing: Theory and practice.
New York: Van Nostrand Reinhold.

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.

52 Project Management Journal September 2002

The Fall of the Firefly: An Assessment
of a Failed Project Strategy
▼ pilots. The results proved to be different: a loss of the more than $40 million invest-
Abstract
Choices made early in a project determine
future success. Missteps in early phases will
cause trouble later in the project’s life cycle.
The U.S. Air Force’s acquisition of the T-3A
“Firefly” trainer was just such a troubled
project. Rather than develop a new aircraft,
the Air Force decided to save time and money
by buying a commercial off-the-shelf (COTS)
trainer. But significant aircraft modifications
undermined the integrity of the COTS strategy.
This paper suggests four project lessons: Any
project must be managed as a system of inter-
related parts; a project strategy must be flexi-
ble to accommodate changing circumstances;
testing must be done in realistic environ-
ments; and concurrency carries with it benefits
and dangers.
Keywords: troubled execution; project
strategy; risk management; commercial off-
the-shelf acquisition, government projects
©2002 by the Project Management Institute
2002, Vol. 33, No. 3, 53–57
8756–9728/02/$10.00 per article + $0.50 per page
September 2002
Bud Baker, Wright State University, Department of Management, 270 Rike Hall, Dayton, OH
45435–0001 USA
E valuating the success of projects is rarely a precise science. Examples of project
ambiguity abound, including the Hubble Telescope, which started its opera-
tional life as a national joke and a case study of project failure. Yet, today it contin-
ues to reveal never-before-seen views of the heavens, views unobtainable from any
other source. At its completion, the Sydney Opera House was seen by most as a stu-
pendous failure: a music hall with poor acoustics, stunningly over cost and behind
schedule. Decades later, that same structure is a unique national treasure, its massive
cost and schedule overruns long ago forgotten.
On occasion, though, a project ends in such a manner that there can be no doubt
about its failure. The U.S. Air Force’s effort to acquire the T-3A Firefly trainer aircraft
is such a case. The Firefly was to improve the screening process for pilot candidates,
saving the government money while helping the Air Force produce more skilled
ed, a reduction in the Air Force’s ability to select pilots, a significant damage to the
Air Force’s reputation, and, worst of all, the deaths of six young men.
Birth of the Firefly Project
The Air Force has used a variety of small aircraft to screen pilot candidates since at
least 1952. The rationale for such screening was based primarily on economic effi-
ciency: Not all pilot candidates had the necessary motivation, aptitude, and skills to
fly high-performance military aircraft. Given that fact, if such candidates could be
identified earlier and eliminated from training, less time, money, and resources
would be wasted on them (Secretary of the Air Force Inspector General, 1998). In the
mid-1960s, the Air Force chose a single-engine Cessna model 172, dubbed it the T-
41, and made it the primary flight screening aircraft. The reliable-but-never-glam-
orous T-41 did its job well for the next 30 years: Despite the inability and inexperi-
ence of thousands of student pilots over three decades, not a single fatality occurred
in T-41 operations.
By the late 1980s, though, some in the Air Force argued that the T-41 needed to
be replaced. The old Cessna design could not handle—nor was it ever designed to
handle—the high stresses of aerobatic flight. And such aerobatics were deemed, by
Air Force leadership, to be necessary as “a means of evaluating a candidate’s ability
to react quickly and accurately while flying more complex maneuvers representative
of follow-on trainers and operational USAF aircraft” (Secretary of the Air Force
Inspector General, 1998, p. 3).
Project Management Journal 53
 The Air Force Chief of Staff at the time, himself a former
fighter pilot, was a strong proponent of replacing the T-41. In a
pithy remark that would become more widely reported when
fatalities began to occur, he claimed, “The T-41 is your grand-
mother’s airplane. Our mission is to train warrior-pilots, not
dentists to fly their families to Acapulco” (Thompson, 1998,
electronic version).
Not everyone shared the general’s sentiments. Most
Air Force pilot training graduates don’t move on to highly
maneuverable fighter/attack aircraft, but instead to heavier,
more stable platforms—bombers, aerial tankers, or cargo
jets—where the spins, loops, and rolls of aerobatic flying are
not exactly commonplace. Furthermore, others thought that
USAF leaders were losing sight of the mission: The purpose of
these aircraft was only to screen prospective pilots, not to train
them, per se. The training would come later, in other aircraft,
after the initial screening. In the words of one instructor pilot:
A common question at the time was “Why are we
spinning students during flight screening? The plane
was simply a screener to determine who qualified to
enter Undergraduate Pilot Training ... We wondered
why spinning and advanced acrobatics were involved in
an aircraft designed to screen applicants. Some of us
had a philosophy that functioning in a flying pattern
and being able to land an aircraft solo was enough cri-
teria to determine who should progress to pilot training
(“The making of a trainer,” 1998).
The Failed Project Strategy:
“Commercial Off-the-Shelf ... Sort of ...”
With the strong support of Air Force leadership and in spite of
the misgivings of others involved in the project, the acquisition
of the new aircraft, called the Enhanced Flight Screening
Program (EFSP), proceeded. One of the first decisions involved
acquisition strategy.
A number of aerobatic-capable flight trainers existed
throughout the world, so the Air Force decided to select one of
these, rather than to develop a new aircraft from scratch. This
strategy, generically known as commercial off-the-shelf (COTS)
is approved and required by the U.S. Department of Defense:
Market research and analysis shall be conducted to
determine the availability and suitability of existing
commercial and nondevelopmental items prior to the
commencement of a development effort ... Preference
shall be given to the use of commercial items ... The
overriding concern is to use the most cost-effective
source of supply throughout a system’s life cycle (U.S.
Department of Defense, 2000).
One reason that commercial items tend to cost less is that
there is not so great a need to do extensive testing and evalua-
tion, because such presumably already was done when the
product was introduced commercially. Still, the EFSP moved
ahead rapidly, by Air Force acquisition standards. The program
management directive authorizing the project was released in
54 Project Management Journal
July 1990, with initial flight demonstration by seven compet-
ing manufacturers held during the next month.
One of the competitors was the Firefly. Offered by
Slingsby Aviation Ltd. of England, it was judged to be under-
powered, slow to climb, and had the lowest cruising speed of
any of the competitors. Brake effectiveness was poor, seating
adjustments difficult, and visibility was limited, both over
the nose and over the low-mounted wings. But handling
earned the Firefly higher marks, with overall stability and
responsiveness both judged to be very good (Secretary of the
Air Force Inspector General, 1998).
Over the next year, a critical thing happened: The slow rate
of climb and sluggish cruise performance of the Firefly in the
1990 tests caused Slingsby to replace the original 200-hp
engine with a much larger, stronger, and heavier powerplant,
generating 260 hp. While this solved the power problem, it
created a host of other difficulties.
In the summer of 1991, when the new, higher-powered
Firefly again was evaluated, along with the other competitors,
the problems were apparent. Spins, which were cited as easy to
enter and correct a year earlier, were now cited as a problem,
especially for a low-time pilot. The brakes were even more
problematic than before. Perhaps most ominously, the Firefly’s
new engine showed a tendency to just quit, both on the
ground and in flight. In seven missions, the engine stopped
four times, once in the air (during a spin) and three times on
the ground. The engine stoppages were attributed to vapor
locks in the fuel system feeding the new engine (Secretary of
the Air Force Inspector General, 1998).
In hindsight, a larger problem is clear. An aircraft is a
system where everything affects everything else. Perhaps the
most critical of all those elements is the engine. When Slingsby
replaced the original four-cylinder, 200-hp engine with the
much larger six-cylinder, 260-hp motor, changes rippled
throughout the entire system. The fuel pump had to be moved
and fuel lines repositioned. The exhaust system moved closer to
the fuel filter/screener, causing fuel to overheat and suggesting
that the vapor locks were systemic problems, not mere anom-
alies. The new engine weighed 80 pounds more than the origi-
nal, which pushed the Firefly’s critical center of gravity beyond
the forward limit: Even more repositioning and rerouting of
other systems were required to get the aircraft’s weight distribu-
tion back in balance (“The making of a trainer,” 1998). The
brakes, marginal to begin with, were now incapable of holding
the re-engined aircraft in place on the ground (Secretary of the
Air Force Inspector General, 1998).
The deeper problem is equally clear: The engine change,
along with the ripple effect through the rest of the aircraft,
effectively destroyed the integrity of the COTS acquisition strat-
egy. The whole project approach was undermined: The aircraft
being bought still may have been “commercial,” but there was
no longer anything “off the shelf” about it. In its brief life, the
Air Force’s Firefly was the subject of wholesale changes, gener-
ating 131 service/modification bulletins, an average of about
two per month (Secretary of the Air Force Inspector General,
1998). The re-engined Firefly had become—or at least should
ave become—an experimental aircraft.
September 2002
Pressing On
Following the evaluation of all competing aircraft (an assess-
ment which lasted only 12 days, from 26 July to 7 August 1991
[Secretary of the Air Force Inspector General, 1998]), the
Aeronautical Systems Center issued a request for proposal in
September 1991. The Slingsby Firefly with the larger engine, by
now designated the T-3A, was selected on 29 April 1992, a
decision immediately protested by some of the losing bidders.
Following a review by the General Accounting Office, the con-
tract award was upheld.
The first prototype was delivered to the Air Force on 15 June
1993 (Secretary of the Air Force Inspector General, 1998). The
total buy was set at 113; 56 would be at the Air Force Academy
in Colorado Springs, CO, USA, and 57 would go to a flying
training squadron at Hondo, TX, USA (Secretary of the Air
Force Inspector General, 1998).
Testing
Because the T-3A was at least nominally a COTS acquisition,
testing and evaluation was very much abbreviated. From 23
September through 1 October 1993—a period of just eight
days—test pilots evaluated the T-3A. Surprisingly, most of the
testing was done by the contractor, with the Air Force in only a
supporting role: “Slingsby primarily conducted the test, with
participation by the 4950th Test Wing ... Slingsby’s final report
stated that the T-3A demonstrated full compliance with system
specifications” (Secretary of the Air Force Inspector General,
1998, p. 16).
While some might question allowing a contractor’s own
employees to assess the degree of that contractor’s own
compliance, there is another troubling point here: The pilots
chosen for such work likely are to be highly skilled and experi-
enced test pilots. Yet the people who would fly the T-3A
operationally were 20-year-old college juniors, supervised by
instructor pilots of widely varying backgrounds. This concern
was to be addressed in another phase of testing, called
Qualification Operational Test and Evaluation (QOT&E).
QOT&E was designed to get the aircraft out of the hands of
test pilots and into the hands of pilots who would have more
typical qualifications. The goal was to see how the aircraft
would behave in its operational environment. QOT&E took
place at Hondo, TX, USA (at an elevation of 930 feet, Hondo
was not representative of the Air Force Academy, at whose
6,572-foot elevation airfield half of the T-3A’s would operate)
(Secretary of the Air Force Inspector General, 1998).
QOT&E was to be in two phases. Phase I, scheduled for 14
weeks, was slashed to just five weeks because the test aircraft
were delivered late. Phase II had to be cut short because of
“extended grounding of the fleet due to uncommanded
engine stoppages during the test” (Secretary of the Air Force
Inspector General, 1998, p. 17). Still, the testing agency rated
the T-3A as “operationally effective but not suitable,” noting
in November 1994 that the criteria for aircraft availability
was 81%, yet the T-3A was fully mission capable only 15.8%
of the time (Secretary of the Air Force Inspector General,
1998, p. 17).
September 2002 Project Management Journal
The Fall of the Fireflies
But by now the pipeline was open. The official “acceptance cer-
emony” for the T-3A had taken place at Hondo in October
1994, the month before the “operationally effective but not
suitable” assessment. In January 1995, the first T-3As arrived at
the Air Force Academy, and it was the very next month that the
first disaster occurred.
On 22 February 1995, an instructor pilot and student were
killed instantly when their T-3A plummeted into a Colorado
pasture in a spin. Investigators concluded that the young stu-
dent inadvertently had put the Firefly into a spin from which
the instructor pilot could not recover. Following the accident,
spins—a major justification for the T-3A in the first place—
were banned from the flight screening program (“The making
of a trainer,” 1998). Morale in the flight training squadrons
dropped (Secretary of the Air Force Inspector General, 1998).
In September 1996, a second T-3A crashed, again in
Colorado. The pilots had been practicing simulated forced
landings, an especially prudent part of the curriculum given
the Firefly’s propensity for engine trouble. As in the first crash,
both pilots died, and simulated forced landings were soon
banned as a result (“The making of a trainer,” 1998).
Just nine months later, the third and last fatal accident
occurred. While executing an approach to the Air Force
Academy’s airfield, the Firefly fell into a stall and spin, impact-
ing the ground before the crew could recover. Again, both pilots
died. A few days later, another Firefly lost power on landing, and
the entire fleet was grounded (“The making of a trainer,” 1998).
The Firefly’s Last Days
Following attempts by the Air Force, Slingsby, and others to
find the cause of the Firefly’s engine stoppages, the Air Force
hired defense contractor Science Applications International
Corp. (SAIC) to evaluate the problem. While SAIC never was
able to isolate a single cause for the failures, it proposed a list
of changes to the fuel system. Ten of those changes were incor-
porated, tested, and approved by the FAA (Scott, 1998).
In 1998, the year following the last of the fatal crashes and
the grounding of the fleet, the Air Force Flight Test Center final-
ly was tasked to test four Fireflies, both with and without the
SAIC-recommended modifications. At last, the Firefly was sub-
jected to the rigorous testing that it should have undergone
years before. For 15 months, Air Force test pilots flew 417
flights for a total of 604 hours, subjecting the Fireflies to inten-
tional mishandling and even “abusive conditions.” Their con-
clusions were that the Firefly was “safe for training,” although
they recommended 27 different changes to the aircraft, flight
procedures, and training curricula. Most of the 27 recommen-
dations were related to just two areas: aircraft handling and
control and the fuel system (United States Air Force, n.d.).
Ultimately the tests again were cut short. On 9 October
1999, the Air Force decided to ground the fleet permanently
(United States Air Force, n.d.). Following unsuccessful
negotiations to sell the remaining 110 Fireflies back to Slingsby,
the Air Force planned to scrap the entire fleet, selling the
aircraft for parts (Diedrich, 2001).
55
Lessons for Project Managers
If the Firefly program is to be of any value, it is in the lessons it
holds for future project managers.
Lesson One. Like an aircraft, a project is a total system, in
which all parts must fit together. This was not the case with the
T-3A. All three accidents, for example, occurred at the Air Force
Academy, with no crashes at the contractor-operated flight
school at Hondo, TX, USA. At least two systemic differences
existed between the Academy T-3A operation and its Texas
counterpart: First, the Academy airfield was a mile higher, with
the thinner air causing a significant drop in the T-3A’s perfor-
mance. Second, the Air Force Academy instructor pilots tended
to be experienced in large jet aircraft, not small aerobatic
planes such as the Firefly. Nor were most of them full-time
instructor pilots: Almost half held full-time jobs as academic
faculty members, flying only a few hours per week (Secretary of
the Air Force Inspector General, 1998). In contrast, the com-
mercial flying school instructors at Hondo flew full time, and
on average had seven times as much single-engine experience as
the Air Force Academy instructors (Secretary of the Air Force
Inspector General, 1998).
The testing that was eventually done by the Air Force Flight
Test Center suggests that in the highly skilled hands of expert
pilots, the flaws of the Firefly were not necessarily fatal ones.
But replace those pilots with the less experienced Academy
instructors, and the integrity of the T-3A flight screening system
appears to have been compromised in a deadly manner.
Lesson Two. The COTS acquisition strategy proved to be
inappropriate for the T-3A, given the substantial modifications
that the aircraft required. It is true that a well-executed COTS
strategy can save time and money by reducing both develop-
ment and testing effort. But as soon as the Firefly’s engine was
changed, with all the other resultant modifications to the air-
craft, the COTS strategy was no longer feasible.
This lesson is acknowledged in a Department of Defense pol-
icy statement issued well after the grounding of the Firefly fleet:
A commercial off-the-shelf (COTS) item is one
that is sold, leased, or licensed to the public; offered
by a vendor trying to profit from it: ... available in
multiple, identical copies; and used without modi-
fication of the internals [emphasis added]
(Commercial Item Acquisition, 2000, p. 3).
Certainly, the scores of changes made and/or recom-
mended to the Firefly qualify as pretty substantial “modifi-
cation of the internals.” But there are other references in the
Department of Defense policy statement that seem specifi-
cally tailored to prevent another T-3A-style failure. For exam-
ple, when considering COTS items, the Department of
Defense acknowledges that COTS is not a panacea, and there
likely will be differences between what the user/client wants
and what already is on the market:
A gap will exist between DoD and commercial
use—and the gap may be large ... Modifying the com-
mercial items is not the best way to bridge the gap ... If
56 Project Management Journal
the gap is too great, commercial items may not be
appropriate … Don’t modify the commercial item
(Office of the Secretary of Defense, 2000, pp. 7–8).
The adage about “not having one’s cake and eating it, too”
applies here: It is unwise to choose a strategy, accept the bene-
fits of that strategy, and then try to ignore its inherent penalties.
Lesson Three. A project must be tested in the environment
in which it will actually operate and with the people who will
actually operate it. As obvious as that statement is, it is impor-
tant to note that it never really happened with the Firefly. The
initial testing was done largely by the contractor’s own test
pilots, and the Air Force Flight Test Center evaluation, per-
formed after the fatal accidents began, also was carried out by
highly skilled professional test pilots. Even the brief opera-
tional testing that was scheduled at Hondo was cut short by the
Firefly’s engine problems. The testing done there was per-
formed by the vastly more experienced commercial instructor
pilots, not the full-time professors/part-time pilots prevalent at
the Air Force Academy.
Lesson Four. Concurrency kills. Normally, concurrency
refers to overlap between project stages: to start testing while
designing or to start producing before testing is complete.
Here, one could argue that the stages of the project actually
were reversed: Purchase the plane, change the design, deliver
and operate it, learn its shortcomings, and only afterwards,
subject it to thorough and rigorous testing.
Concurrency can be, of course, a necessary project tactic,
often for competitive reasons: Without concurrency, the three-
year product development cycles common to the automotive
industry or the much shorter cycles of high-tech firms would
be impossible.
In this case, though, one wonders: What was the rush? Why
was such a high degree of concurrency necessary? This was no
wartime emergency, no crisis response. The previous screener
aircraft was performing safely, reliably, and—except in the eyes
of some senior Air Force leaders—effectively. If there really was
a need to move to an aerobatic airplane, there was time to do
the job in a careful, measured manner: a rigorous source selec-
tion with thorough and operationally representative testing.
Peter Drucker used to tell his graduate students that when
intelligent, moral, and rational people make decisions that
appear inexplicable, it’s because they see a reality different than
the one seen by others. With that in mind, what reality did the
T-3A project managers see?
Meredith and Mantel in their book, Project Management,
offer a possible answer. They refer to a project model that they
call “The Sacred Cow”:
In this case, the project is suggested by a senior and
powerful official in the organization. Often the project
is initiated with a simple comment such as “If you
have a chance, why don’t you look into ...,” and there
follows an undeveloped idea for a new product … The
immediate result is the creation of a “project” to
investigate whatever the boss has suggested (Meredith
& Mantel, 2000, p. 45).
September 2002
 What was the fatal flaw in the Firefly tragedy? Was it a lack
of consideration for the “systems approach?” A COTS buy that
evolved into a developmental program? Or, possibly, was it
inadequate testing in a true-to-life operational environment,
too much concurrency, or a rushed favorite project of senior Bud Baker, PhD, is a professor of management at
leaders? The most likely answer is that the Firefly failed as a Wright State University, where he also directs the
result of a combination of these issues. In years to come and as MBA program in project management. He holds an
MBA from the University of North Dakota and mas-
the trauma caused by the project subsides, more complete
ters and doctoral degrees from the Peter Drucker Center of the
assessments may be able to explain the seemingly inexplicable Claremont Graduate School. Prior to 1991, Baker spent more than two
causes of the Firefly’s failure. decades as a U.S. Air Force officer, including a number of years as a
project manager. He is a contributing editor the PM Network magazine,
References a member of the editorial review board for the Project Management
Journal, and a charter member of the Department of Defense
The making of a trainer: The Air Force's acquisition of the
research integrated product team.
hapless Slingsby T-3A [Electronic version]. (1998, February)
Light Plane Maintenance, 20 (2), 5–11; 22–23.
Diedrich, J. (2001). Air Force might sell troubled T-3 fireflies
for scrap. Air Force Times, 61 (26), 10.
Meredith, J.R., & Mantel, S.J., Jr. (2000). Project manage-
ment: A managerial approach (4th ed.). New York: John Wiley
& Sons Inc.
Office of the Secretary of Defense. (2000). Commercial item
acquisition: Considerations and lessons learned [CD-ROM].
Retrieved January 8, 2001, from http://www.deskbook.osd.mil
Secretary of the Air Force Inspector General. (1998). Broad
area review of the enhanced flight screening program. Retrieved
December 16, 2002, from http://www.af.mil/lib/misc/
t3bar.html
Scott, W.B. (1998). USAF modifying Slingsby trainers to
correct inflight engine shutdowns [Electronic version]. (1998).
Aviation Week and Space Technology, 148 (2), 38.
Thompson, M. (1998). The deadly trainer [Electronic ver-
sion]. Time, 151 (1), 42–45.
United States Department of Defense. (2000). Mandatory
procedures for major defense acquisition programs (MDAPS) and
major automated information system (MAIS) acquisition programs
(DoD 5000.2-R). Retrieved January 8, 2001, from
http://www.deskbook.osd.mil
United States Air Force. (n.d.) T-3A System improvement
program final report executive summary. Edwards Air Force Base,
CA: 412th Test Wing.

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.

September 2002 Project Management Journal 57

The Impact of the Project Manager
on Project Management Planning
Processes
▼ For proper project completion, both planning and execution must be properly
Abstract
If a project is to be successfully completed,
both planning and execution must be properly
implemented. Poor planning will not allow
appropriate execution and control processes
or achievement of the project’s targets. The
objective of the study reported in this paper
is to evaluate the impact of the project manag-
er on the quality of project planning processes
within the nine knowledge areas defined by
A Guide to the Project Management Body of
Knowledge (PMBOK® Guide) and to determine
ways of increasing the effectiveness of the
manager’s intervention. Participants in the
study evaluated their use of the 21 processes
that relate to planning, out of the 39 process-
es required for proper project management.
The results of the study reveal risk manage-
ment and communications as the processes
with the lowest planning quality. Poor quality in
these areas results when project managers lack
the formal tools and techniques for dealing with
communications and the functional managers
are not equipped with the tools and techniques
that will allow them to effectively contribute to
the risk management process. Improving quality
planning processes requires the development
of new tools in areas such as communications,
as well as organizational training programs
designed for the functional managers.
Keywords: project manager; functional
manager; planning; impact
©2002 by the Project Management Institute
2002, Vol. 33, No. 3, 58–64
8756–9728/02/$10.00 per article + $0.50 per page
58
Shlomo Globerson, School of Business Administration, Tel Aviv University, Ramat Aviv,
P.O. Box 39010 Tel Aviv 69978 Israel
Ofer Zwikael, Technology Management Department, Holon Academic Institute of Technology,
25 Golomb St., Holon 58102 Israel
P roject success is measured as the ability to complete the project according to
desired specifications and within the specified budget and the promised time
schedule, while keeping the customer and stakeholders happy.
implemented. Control is the monitoring mechanism that ensures each of the two
phases is properly implemented, with corrective actions being introduced where
there are undesired discrepancies between the project’s plan and its execution.
Much has been written about control (Cleland, 1994; Fleming & Koppelman,
1994; Kimmons, 1990; Shtub, Bard, & Globerson, 1994; Wysocki, Beck, &
Crane,1995; Zwikael, Globerson, & Raz, 2000). However, most of this literature
relates to the use of control during the execution phase, the plan being used as the
baseline for evaluating progress during the execution phase.
The main reason for the scarcity of literature on planning control is the difficulty
in defining a baseline for monitoring progress during the planning phase. One may
say that stakeholders’ requirements should be used as the baseline for evaluating
planning. However, requirements are expressed in terms of functional needs, where-
as planning is expressed by technical parameters. As these two areas use different
“units of measurement,” they are difficult to compare. Despite the evaluation and
control difficulties, it is of the utmost importance to verify that planning is properly
done and to develop tools that will improve its quality. Poor planning will result in
poor execution.
The purpose of this paper is to evaluate the actual impact of the project manager
on the quality of project planning processes and to determine how intervention can
be made more effective. The methodology used in this study is based on A Guide to
the Project Management Body of Knowledge (PMBOK® Guide) (PMI Standards
Committee, 2000). According to the PMBOK® Guide, a project manager is concerned
with nine different knowledge areas, in which he or she has to properly manage 39
different processes. The processes are grouped into four life-cycle phases: initiation,
planning, execution, and closure. Because this study concentrates on planning, Table
1 identifies the processes that support just the planning phase.
Out of the 39 processes listed, 21 are identified by the PMBOK® Guide as related
to planning. If a project is to be properly planned, these 21 processes must be prop-
erly executed. To evaluate the quality of planning process implementation, the prod-
ucts of each single process must be evaluated. Although each process may have mul-
tiple sets of outputs and each set may have multiple products, one major product can
Project Management Journal September 2002
 Knowledge area Planning processes Other processes

Integration Project plan development Project plan execution

Integrated change control

Scope Scope planning Initiation

Scope definition Scope verification
Scope change control

Time Activity definition Schedule control

Activity sequencing
Activity duration estimating
Schedule development

Cost Resource planning Cost control

Cost estimating
Cost budgeting

Quality Quality planning Quality assurance

Quality control

Human resources Organizational planning Team development

Staff acquisition

Communications Communications planning Information distribution

Performance reporting
Administrative closure

Risk Risk management planning Risk monitoring and control

Risk identification
Qualitative risk analysis
Quantitative risk analysis
Risk response planning

Procurement Procurement planning Solicitation

Solicitation planning Source selection
Contract administration
Contract closeout

Table 1. Separating the 39 Processes Belonging to the Nine Knowledge Areas by Planning Processes and Other Processes

be identified for each planning process. For example, the major
product of the scope definition process is the work breakdown
structure (WBS). Table 2 lists the major products for all plan-
ning processes.
The Study
A field study was conducted to evaluate the extent of the pro-
ject manager’s involvement in the planning processes and to
evaluate their quality. A major problem in designing this study
was to establish a way to evaluate the extent to which planning
processes were used in projects and their quality level. For this
purpose, the following assumption was made: The quality of a
process is a function of the frequency with which it is used to
obtain the major product of the process. This assumption is
based on the learning curve theory, which has proved ongoing
improvement as a function of the number of repetitions
(Griffith, 1996; Snead & Harrell, 1994; Yiming & Hao, 2000;
Watson & Behnke, 1991).
Participants in the study were project managers and others
who are involved in project management activities. The 282
participants came from different project management work-
shops administered as part of internal or external training sem-
inars. The portion of the questionnaire that they were asked to
complete, which served as the database for this study, appears
in Appendix 1.
The following scale was used for evaluating the intensity of
use of the different products:
■ 5 = The product is always obtained;

September 2002 Project Management Journal 59

 Knowledge area Planning processes Major product

Integration Project plan development Project plan

Scope Scope planning Project deliverables

Scope definition Work breakdown structure

Time Activity definition Project activities

Activity sequencing PERT or Gantt chart
Activity duration estimating Activity duration estimates
Schedule development Activity start and end dates

Cost Resource planning Activity required resources

Cost estimating Resource cost
Cost budgeting Time-phased budget

Quality Quality planning Quality management plan

Human resources Organizational planning Role and responsibility assignments

Staff acquisition Project staff assignments

Communications Communications planning Communications management plan

Risk Risk management planning Risk management plan

Risk identification Risk list
Qualitative risk analysis Project overall risk ranking
Quantitative risk analysis Prioritized list of quantified risks
Risk response planning Risk response plan

Procurement Procurement planning Procurement management plan

Solicitation planning Procurement documents

Table 2. Major Product of Each Planning Process

■ 4 = The product is obtained quite frequently;
■ 3 = The product is obtained frequently;
■ 2 = The product is seldom obtained;
■ 1 = The product is hardly ever obtained;
■ 9 = The product is irrelevant to the projects I am involved in;
■ 0 = I do not know whether the product is being obtained.
Table 3 summarizes the results. As may be seen from Table
3, the quality of the processes ranges from a low of 2.0 for
qualitative risk analysis up to 4.2 for activity duration estimat-
ing. In other words, qualitative risk analysis is hardly practiced.
There is a high correlation among the quality of products
belonging to the same knowledge area; that is, they are either
low or high. For example, the quality of all risk processes is
below 2.9, whereas the quality of all the scope processes is
above 3.5. The quality of each knowledge area is calculated by
the average quality of the processes belonging to it, as present-
ed in Table 3, and charted in descending order in Figure 1.
The three groups of knowledge areas identified in Figure 1 are:
■ High quality areas, to which integration, scope, time, and
human resources belong. The score for this group is around 4;
■ Medium quality areas, to which cost, quality, and procure-
ment belong. The score for this group is around 3;
■ Poor quality areas, to which risk and communications
belong. The score of both is around 2.3.
Analysis and Discussion
Assuming that for successful completion of a project, all
processes in the nine knowledge areas should be high quali-
ty, we should discuss ways to improve the poor performance
areas. Performance in a specific area is a function of the pro-
ject manager’s know-how of that area, the know-how of
other professionals, such as functional managers, who are
involved in the specific process, and with the project manag-
er’s ability to affect the area and its attendant processes.
In general, functional managers are accountable for the
proper execution of the specific work packages assigned to
them, whereas the project manager is responsible for integra-
tion and infrastructure-related work packages and activities. For
example, a project manager is heavily involved with the process
of establishing the WBS for the whole project and is directly

60 Project Management Journal September 2002

 Knowledge area Processes Major product

Name Quality Average STD

Integration Project plan development 4.0 4.0 1.1

Scope Scope planning 4.1 3.8 0.9

Scope definition 3.6

Time Activity definition 4.1 3.9 0.7

Activity sequencing 3.4
Activity duration estimating 4.2
Schedule development 4.0

Cost Resource planning 3.7 3.3 1.0

Cost estimating 3.0
Cost budgeting 3.2

Quality Quality planning 2.9 2.9 1.2

Human resources Organizational planning 3.8 3.7 0.9

Staff acquisition 3.6

Communications Communications planning 2.3 2.3 1.1

Risk Risk management planning 2.2 2.3 1.2

Risk identification 2.8
Qualitative risk analysis 2.0
Quantitative risk analysis 2.3
Risk response planning 2.3

Procurement Procurement planning 3.3 3.3 1.2

Solicitation planning 3.3

Table 3. Quality of Major Planning Processes Within the Different PMBOK® Guide Knowledge Areas

accountable for it. However, a process such as quantitative risk
analysis, which must be done separately for each individual
work package, requires heavy involvement of the functional
manager and those of his employees who are responsible for
executing the work package (assuming a matrix organization).
Therefore, to perform the process properly, all the involved par-
ties should possess know-how of risk management methods.
Process know-how means being familiar with the required
inputs, the tools and techniques used, and the desired outputs
of the process. If this is not the case, then risk management or
any other relevant processes can’t be effectively handled on a
work package level or on the integrated level. A project man-
ager functioning in a matrix environment gets work packages
done by “contracting out” to internal suppliers, who are the
functional managers in the organization, and purchasing other
work packages from external sources via procurement process-
es. Because procurement requires the signing of formal docu-
ments, as compared to working with functional managers
within the same organization, much care is taken to include all
relevant expectations in the contract, to ensure that the external
suppliers follows all requirements. The combination of employ-
ees working on a work package without the relevant know-how
in relevant risk management processes, and a project manager
without the ability to endow them with this knowledge makes
it unrealistic to expect effective implementation of project man-
agement processes. In other words, if a certain process is to be
carried out in an appropriate manner, the people involved in the
process need the relevant know-how.
Table 4 divides the planning processes according to the
individual who expends most effort in properly executing a
process (the project manager or the functional manager), keep-
ing in mind that the project manager is held responsible for all
processes. In general, in a matrix organization, a functional
manager is responsible for carrying single work package relat-
ed activities, whereas a project manager deals with work pack-
ages that are integrated in nature. For example, a functional

September 2002 Project Management Journal 61

 5

4.0
3.9
4 3.8
3.7

3.3 3.3 3.3

3 2.9
Planning Quality

2.3 2.3

0
s

ns
n

st

ity

sk

e
t
en
ce

ag
io

op

Co

tio
al

Ri
at

em
ur
Ti

er
Sc

Qu

ica
gr

so

Av
ur
te

un
Re

oc
In

m
Pr
an

m
Co
m
Hu

Knowledge Areas

Figure 1. Planning Quality of the Nine PMBOK® Guide Knowledge Areas

manager carries the major responsibility for activity definition,
whereas the project manager has to integrate all project activi-
ties into activity sequencing, and so on.
Keep in mind that a project manager is regarded as a func-
tional manager when he or she works on the professional work
packages that fall within his or her direct area of accountabili-
ty. For example, the project manager is directly accountable for
the scope-definition process, the major output of which is the
WBS. However, the project manager will not be able to execute
this process out without the cooperation and the know-how of
functional managers. The same holds true for all processes,
regardless of the manager who is directly accountable.
Further, the two areas that belong to the “poor quality” group,
that is, risk management and communications, displayed poor
results. Because risk management is part of the PMBOK® Guide
and, therefore, also part of the Project Management Professional
(PMP®) examination, one may assume that project managers are
familiar with it. However, functional managers do not have any
formal risk management training, and one may assume that they
suffer from lack of the requisite know-how with regard to risk
management processes. It is therefore not surprising that func-
tional managers are of little help in performing risk management
processes. The project manager is thus left alone to struggle with
it, with limited success according to the results of this study and
others (Ibbs & Kwak, 2000; Mullaly, 1997).
The situation is not the same for the communications
area, because its only planning process—communications
planning—is an integrated work package with which the pro-
ject manager is the most heavily involved. Although other
professionals, such as functional mangers, must be involved
in the communications planning process, the project manag-
er should lead the overall effort and have a strong command
of the relevant tools and techniques. Therefore, lack of know-
how on the part of the functional manager is not the main
reason for poor performance in this area. The explanation has
to be sought elsewhere.
According to the PMBOK® Guide, “Communications
planning determines the information and communications
needs for the stakeholders: Who needs what information,
when they will need it, and how it will be given to them”
(p.119). This area is probably the most difficult for the
project manager to plan, because it requires getting present
and future information needs from all the stakeholders. Very
few formal tools and techniques are available to the project
manager for supporting the communications area. PMBOK®
Guide offers only one unstructured tool, namely, stakeholder
analysis. The lack of proper and easily accessible tools cou-
pled with the difficult task of identifying future information
needs of stakeholders makes communication a very difficult
process to plan.

62 Project Management Journal September 2002

 Knowledge area
Knowledge area Project manager Functional manager
quality

High Integration Project plan development

Scope Scope planning

Scope definition

Time Activity sequencing Activity definition

Schedule development Activity duration estimation

Human resources Organizational planning Staff acquisition

Medium Cost Cost budgeting Resource planning

Cost estimating

Procurement Procurement planning

Solicitation planning

Quality Quality planning

Poor Communications Communications planning

Risk Risk management planning Risk identification

Qualitative analysis
Quantitative analysis
Risk response planning

Table 4. Mapping of the 21 PMBOK® Guide Planning Processes According to the Individual (Project or Functional
Manager) Who Performs Most of the Activities Required to Execute the Planning Process

The areas that belong to the middle quality group are cost,
quality, and procurement. From Table 4, we can identify
the manager who needs most of the know-how required
to properly execute each of the processes. While the project
manager leads the process in the quality area and the func-
tional manager in the procurement area, both are involved in
the cost area. Therefore, the key to success in these areas is
different tailor-made training modules for the project and the
functional manager.
Conclusion
As the person who is fully accountable for the success of the
project as a whole, the project manager is responsible for
overcoming the difficulties encountered in guaranteeing
that all planning processes are properly executed. To resolve
the problems, the project manager should identify the
events that have a negative impact on the successful com-
pletion of the project and develop explicit mitigating plans
to accommodate them.
In some areas, such as communications and quality, the
project management community should develop better tools
and techniques to support the project manager’s efforts. In
other areas, such as risk and cost, more emphasis should be
placed on the training of functional managers in the use of
the relevant tools and techniques. In other words, the func-
tional manager also should get intensive but adapted project
management training. The agent for such a fundamental
change in the organizational culture can’t be the project man-
ager alone. It is essential that it be sponsored at a high level
of the organization and even treated as a project by itself.
References
Cleland, D.I. (1994). Project management: Strategic design
and implementation. New York: McGraw-Hill.
Fleming, Q.W., & Koppelman, J.M. (1994). The earned
value concept: Back to the basics. PM Network, 8 (11), 27–29.
Griffith, T.L. (1996). Negotiating successful technology
implementation: A motivation perspective. Journal of
Engineering & Technology Management, 13 (1), 29–53.
Ibbs, C.W., & Kwak, Y.H. (2000). Assessing project man-
agement maturity. Project Management Journal, 31, 32–43.
Kimmons, R.L. (1990). Project management basics.
Houston, TX: Kimmons-Asaro Group Ltd. Inc.
Mullaly, M. (1998). 1997 Canadian project management
baseline study. Proceedings of the Project Management
Institute’s 29th Annual Symposium, Long Beach, CA. Newtown
Square, PA: PMI, 375–384.
PMI Standards Committee. (2000). A guide to the project
management body of knowledge. Newtown Square, PA: Project
Management Institute.

September 2002 Project Management Journal 63

 Shtub, A., Bard, J.F., & Globerson, S. (1994). Project manage-
ment: Engineering, technology and implementation. Englewood
Cliffs, NJ: Prentice Hall.
Snead, K.C., & Harrell, A.M. An application of expectancy
theory to explain a manager’s intention to use a decision sup- Shlomo Globerson, PMP, PhD, a professor at the
Graduate School of Business Administration, Tel Aviv
port system. Decision Sciences, 25 (4), 499–513.
University, is an internationally known researcher, educa-
Watson, W.E., & Behnke, R.R. (1991). Application tor, and consultant in the fields of project management
of expectancy theory and user observations in identifying and operations management. As a researcher and writer, he has
factors which affect human performances on computer published more than 70 refereed articles and seven books and is a
projects. Journal of Educational Computing Research, 7 (3), frequent contributor to PMI® Seminars & Symposium.
363–376.
Wysocki, R.K., Beck, R., & Crane, D.B. (1995). Effective
project management. New York: John Wiley & Sons Inc.
Yiming, C., & Hau, L. (2000). Toward an understanding of Ofer Zwikael is a lecturer at the faculty of manage-
the behavioral intention to use a groupware application. ment at Tel Aviv University and a faculty member at
the Technology Management Department at the Holon
Proceedings of the 2000 Information Resource Management Academic Institute of Technology. He also acts as
Association International Conference, Anchorage, AK. Hershey, academic counselor and senior lecturer of project management at
PA: Idea Group Publishing, 419–422. John Bryce Training.
Zwikael, O., Globerson, S., & Raz, Z. (2000). Evaluation of
models for forecasting the final cost of a project. Project
Management Journal, 31 (1), 53–57.

Appendix 1. Project Planning Assessment Questionnaire

For each planning product listed, please mark the most suitable answer regarding the projects you are involved in,
according to the following scale.
5 = The product is always obtained. 1 = The product is hardly ever obtained.
4 = The product is quite frequently obtained. 9 = The product is irrelevant to the projects I am involved in.
3 = The product is frequently obtained. 0 = I do not know whether the product is obtained.
2 = The product is seldom obtained.

Planning product Never Always Irrelevant Do not know

Project plan 1 2 3 4 5 9 0
Project deliverables 1 2 3 4 5 9 0
Work breakdown structure 1 2 3 4 5 9 0
Project activities 1 2 3 4 5 9 0
PERT or Gantt chart 1 2 3 4 5 9 0
Activity duration estimates 1 2 3 4 5 9 0
Activity start and end dates 1 2 3 4 5 9 0
Activity required resources 1 2 3 4 5 9 0
Resource cost 1 2 3 4 5 9 0
Time-phased budget 1 2 3 4 5 9 0
Quality management plan 1 2 3 4 5 9 0
Role and responsibility assignments 1 2 3 4 5 9 0
Project staff assignments 1 2 3 4 5 9 0
Communications management plan 1 2 3 4 5 9 0
Risk management plan 1 2 3 4 5 9 0
Risk list 1 2 3 4 5 9 0
Project overall risk ranking 1 2 3 4 5 9 0
Prioritized list of quantified risks 1 2 3 4 5 9 0
Risk response plan 1 2 3 4 5 9 0
Procurement management plan 1 2 3 4 5 9 0
Procurement documents 1 2 3 4 5 9 0

This article is copyrighted material and has been reproduced with the permission of PMI. Unauthorized reproduction of this material is strictly prohibited.
Cover to Cover
Book Review Editor, Kenneth H. Rose, PMP
Successful Information System Implementation:
The Human Side, Second Edition
by Jeffrey K. Pinto and Ido Millet
T his book is a cautionary tale:
Don’t try to install an informa-
tion system (IS) unless you’re will-
ing to suffer through all the stages of
getting it right. It’s affirmative pain
on behalf of a good result. Successful
Information System Implementation:
The Human Side first tells you what,
then shows you how.
The most succinct statement of
Jeffrey K. Pinto’s and Ido Millet’s
worthy thesis comes on p. 43: “One
of the most important points that
past research and experience have taught us is that the client is
the ultimate determinant of successful system implementa-
tion. This lesson, so fundamental to practicing managers, is
one that continually escapes the attention of researchers and IS
theoreticians and must be continually relearned.”
The authors may be forgiven for beating this horse so
often (on almost every one of the book’s 196 pages),
because apparently the horse isn’t dead! The preposterous
idea that information systems themselves (whatever that
may mean) are the key factor in their own success has
apparently been the guiding lack-of-light for decades.
Failure rates “continue to hover at 67%, an astounding fig-
ure when one considers that companies have spent enor-
mous amounts of money in the past decade not only trying
to install these systems, but also trying to understand what
went wrong with the previous failures.”
The first two chapters, The Problem With Information
System Projects and Implementation Theory: What the Past
Has Taught Us, detail the dismal history of IS implementation
failures—and their costs in dollars, human frustration, and
business failures. Pinto and Millet make a compelling case for
the need to see the territory differently and provide a map to
take managers and designers from problem definition to suc-
cessful resolution.
The foundation for this map-making comes in the book’s
crucial Chapter 3, Defining Implementation Success and
Failure. If system designers, managers, and users don’t know—
and agree on—what an answer should look like, how will any-
one know if we have or have not found it?
September 2002 Project Management Journal
Reviewing earlier partial definitions, Pinto and Millet add
what’s needed for a full picture of implementation success.
Their three-part definition starts with system traits, under
which come two measures: system quality (“The system
adheres to satisfactory standards in terms of its operational
characteristics.”) and information quality (“The material pro-
vided by the system is reliable, accurate, timely, user friendly,
concise, and unique.”).
Second, under characteristics of data usage comes use (“The
material provided by the information system will be readily
employed by the organization in fulfillment of its opera-
tions.”) and user satisfaction (“Clients making use of the sys-
tem will be satisfied with the manner in which it influences
their jobs, through the nature of the data provided.”).
Last, under impact assessment comes individual impact
(“Members of the departments using the information system
will be satisfied with how the system helps them perform their
jobs, through positively impacting both efficiency and effec-
tiveness.”) and organizational impact (“The organization as a
whole will perceive positive benefits from the information sys-
tem, through making better decisions and/or receiving cost
reductions in operations.”).
I quote the definition in full both because it seems to me
to be correct—both necessary and sufficient—and also
because, like much of the prose in the book, it is avoidably
ugly. That is, it has been very well written down, but it has
not been well written up. The ideas are too good and too
needed to be so unpleasant to take in. This is the editor’s
fault and should be corrected if a third edition is needed.
Judging from the magnitude and pervasiveness of the prob-
lems the authors discuss, I’m afraid that need will indeed
arise. While the editor’s at it, how about an index? I hate
books without indexes, and I know I’m not alone. The lack
is easy to fix and the benefit substantial.
The remainder of the book is, appropriately, a detailed
how-to, starting with critical success factors in information
system projects and moving through techniques for project
selection, planning, scheduling, the politics of implementa-
tion, team-building and cross-functional cooperation,
implementation champions, and finishing the project on
a high note.
The final chapter, Conclusions: Quo Vadis?, stresses the
importance of realizing that no matter how complete and
comprehensive the process for IS design and implementa-
tion, it always will be “characterized by enormous difficulty,
ambiguity, and personal challenge.” The human side of IS
65
implementation is the hard part, the stumbling block, the rea-
son for the incredible, intolerable, but true 67% IS failure rate.
Project Management Institute, 1999, ISBN: 1880410664,
paperback, 196 pp., $32.95.
Reviewed by Louis I. Middleman, ICF Consulting, who
specializes in facilitating communication and organiza-
tional development workshops for federal agencies.
World Class Contracting: How Winning
Companies Build Successful Partnerships in the
e-Business Age
by Gregory A. Garrett, PMP, CPCM,
F inally, a book on project procure-
ment management written by
a Project Management Professional
(PMP®)! World Class Contracting:
How Winning Companies Build
Successful Partnerships in the e-
Business Age by Gregory A. Garrett
is not a book that will gather dust
as it sits unopened on your book-
shelf, but a ready reference source
that you will want to keep within
arm’s reach as you manage your
project’s procurement process.
Today’s project managers face increased organizational flat-
tening, downsizing, and outsourcing. They must manage crit-
ical contractors and suppliers and manage the risks that come
with establishing long-term business relationships. Critical
new skills include creating, negotiating, and administering
contracts—the contract management process. World Class
Contracting meets this new need by putting the language of
project procurement management at the reader’s fingertips.
A certified professional contracts manager and PMP®,
Garrett has extensive experience as a program manager and
contract manager on a wide range of high-technology prod-
ucts, services, and customized solutions for multinational
organizations.
The book provides both new and well-seasoned project
managers a thorough understanding of the contract man-
agement process. Garrett expands on the project procure-
ment management section of the Project Management
Institute’s A Guide to the Project Management Body of
Knowledge (PMBOK® Guide)—2000 Edition and traces the
contract management process through each of the six steps
from the perspectives of both buyer and seller. This is a
unique benefit from the book. Traditional procurement and
contract management books are written from the perspective
of the buying organization. Because contracts are about
developing and maintaining business partnerships between
buyers and sellers, Garrett’s book provides excellent insight
into both parties of the contracting process. It goes to the
66 Project Management Journal
essence of contract management: the business partnerships,
people, processes, and tools to be successful in the e-busi-
ness age. The author discusses more than 100 best practices
from leading global companies involved in contracting for a
wide range of goods and services.
Garrett begins with a discussion of how companies go
about building successful partnerships and the importance of
these partnerships in the e-business age and then delves into
the process of how companies build trust with their vendors
and customers by managing expectations and honoring com-
mitments. His checklist of 20 techniques for managing expec-
tations and building trust between partners provides practical
and functional information to the reader.
Garrett then dives straight into the contract manage-
ment process, breaking down the process into the three
general phases: pre-award, award, and post-award. He
spends the majority of his book discussing the major steps
that constitute the three phases of the contracting process,
from both a buyer and seller perspective. His coverage of
the buyer’s activities (procurement planning, solicitation
planning, solicitation, source selection, contract adminis-
tration, and close-out/termination) is compared and con-
trasted with the seller’s activities (presales activity, bid/no-
bid decision, bid/proposal preparation, contract negotia-
tion/formation, contract administration, and contract
close-out/termination).
This coverage not only is unique to a text on contract
management but also provides valuable insight for the pro-
ject manager and contract manager in building trust, man-
aging expectations, and honoring commitments in the pro-
ject environment. Garrett’s standard format of describing
each step in terms of inputs, tools/techniques, and outputs
makes this book easy to understand and apply to real-
world projects. His in-depth analysis of the contract man-
agement process and his treatment of the buyer and seller
activities provide both practical and most valuable contri-
butions to the contract management body of knowledge.
His use of figures and diagrams also helps to simplify the
complex contracting process.
The chapter on teamwork focuses on the intricate
relationship between the project manager and the contract
manager and the importance of establishing “who’s in
charge.” Garrett’s treatment of basic contracting concepts
and principles as well as his coverage of e-procurement
methods provides a concise-yet-complete treatment of this
critical area of contract management from competitive
contracting methods to noncompetitive methods including
single-source and sole-source negotiations.
Garrett closes his book with a discussion of common mis-
conceptions about contract management and then provides
a summary of 15 of the most important best practices. His
appendix of more than 20 templates, checklists, matrixes,
and forms, ranging from a sample proposal compliance
matrix to a contract close-out checklist, help to streamline
and demystify the contract management process. This, along
with an enriching glossary of contract management terms
and extensive bibliography of resource materials, makes his
September 2002
book a valuable contract management desktop reference and
guide that should be kept within arm’s reach of any serious
contract manager.
CCH Inc., 2001, ISBN: 0-8080-0563-4, hardcover, 341 pp.,
$55.00.
Reviewed by U.S. Air Force Lieutenant Colonel Rene G.
Rendon, PMP, director of contracts for the Air Force’s
Space-Based Infrared Systems program and a member of
the PMI Los Angeles Chapter.
Project Management: Strategic Design and
Implementation, Fourth Edition
by David I. Cleland and Lewis R. Ireland
“ T here is nothing permanent
except change,“ advised
Heraclitus of Greece in 513 B.C.
Project Management: Strategic Design
and Implementation, Fourth Edition,
by David I. Cleland and Lewis R.
Ireland provides contemporary evi-
dence that both change and improve-
ment are the natural order of things
in project management literature.
This new edition contains much
of the content of the previous edi-
tion, but puts forth, in the parlance
of electronic Web site developers, a whole new look and feel.
Its design follows the format of the authors’ previous collabo-
rative success, Project Manager’s Portable Handbook, with deci-
mally numbered paragraphs for structure and bullet lists for
detail. This form of layout and presentation results in content
that is modularly organized and easily accessible for readers.
The authors have improved individual chapters by adding a
brief introduction paragraph that outlines the central points of
the chapter and warms-up readers for what follows. Each chap-
ter now concludes with four additional sections: a listing of
additional sources of information in the form of a generously
annotated bibliography; a listing of project management prin-
ciples that summarize chapter content in pithy statements of
enduring, universal value; a project management situation—a
brief, descriptive case study that illuminates chapter content by
way of a practical example; and a student/reader assignment
that offers food for thought, discussion, or investigation.
In whole, the book comprises 22 chapters divided into
seven parts that march progressively forward from history
through current practice to a view of the future. This new edi-
tion provides a guiding graphic that appears at the head of
each chapter as a structural map showing readers where they
are and maintaining a sense of global reference throughout
their journey through the text.
In a very general way, the content of this edition follows
closely that of the previous edition, as most sequential editions
September 2002 Project Management Journal
do. But this is no mere tweaking of text to generate a more cur-
rent publication date and perhaps rejuvenate sagging sales. It is
a significant improvement in both form—described above—
and substance. The authors have rearranged technical material
in places to make it more logical and readable and updated ref-
erences throughout to make them more relevant to readers and
reflective of recent research.
The chapter on strategic issues in project management is a
good example. The old material is still there, but newly
designed with headings, bullets, and graphics that are more
appealing in appearance and more facilitating in function.
Bringing things up to date, the authors have added a timely dis-
cussion of project portfolio management and references to arti-
cles as recent as June 2001.
Cleland and Ireland have combined previous separate
chapters on project organization charting and authority into a
single chapter that presents a complete integrated view of these
interrelated topics. They also have recast the previous chapter
on working with project teams as a discussion of effective pro-
ject teamwork that is more action oriented and that prescribes
specific steps to take when building project teams.
A new and much needed chapter addresses project man-
agement maturity. The authors briefly discuss the history
of maturity models, specific models developed by the
Software Engineering Institute and by the Federal Aviation
Administration, and two general approaches to model devel-
opment. They provide a tool for assessing project manage-
ment maturity and describe the contributing roles of bench-
marking and business intelligence. While no cookbook solu-
tion to maturity management currently exists, the informa-
tion in this chapter leaves readers well informed and well
armed to deal with this important emerging issue on their
own as individual project needs demand.
Another welcome addition is a separate chapter on
earned value management systems (EVMS). Unlike texts that
limit discussion of earned value to syntax and formulas,
Cleland and Ireland address concepts, meaning, and appli-
cation concisely and completely. They temper the unique
and essential utility of EVMS with the candid and practical
observation that it is not a tool for all projects. It requires a
rigorous detailed plan and disciplined management process-
es. A poorly defined project will condemn an EVMS effort to
frustration and failure.
Project Management: Strategic Design and Implementation,
Fourth Edition, is not the last word on project management
theory and practice. Given the evolutionary nature of the
domain, such a book probably will never be written. But it
significantly raises the bar for books of its kind and sets a
standard against which others may be measured now and in
the future.
Mcgraw-Hill, 2002, ISBN: 0-07-139310-2, hardcover, 656 pp.,
$70.00.
Reviewed by Ken Rose, PMP, an instructor for ESI
International residing in Hampton, VA, USA, and vice
president for programs, PMI Hampton Roads Chapter.
67
Cultivating Communities of Practice:
A Guide to Managing Knowledge
by Etienne Wenger, Richard McDermott,
and William M. Snyder
A persistent barrier to project
management’s evolution as a
discipline has been the distribution
of knowledge. There’s no shortage
of innovative solutions or solid
methodologies, but unless they are
shared within a company, industry,
or profession, their value never will
be realized fully. At best, these
practices will be disseminated
through formal channels such as
publications or conferences and
may slowly enter the mainstream;
at worst, they will be lost—to be reinvented time and time
again. Similarly, most problems in this domain could benefit
from the collective brainpower of the greater project manage-
ment community. While professional collaboration opportu-
nities are more numerous now than in recent years, it’s likely
that many project managers still lack adequate sounding
boards for their ideas.
The Project Management Institute (PMI) and technological
innovations such as the Internet have advanced project man-
agement by quantum leaps by lessening these knowledge
transfer and collaboration impediments. By examining why
efforts similar to this are successful, the book Cultivating
Communities of Practice: A Guide to Managing Knowledge creates
a framework that can be used to leverage and expand an
intellectual capital. Neither cookbook nor textbook,
Cultivating Communities of Practice distills knowledge
management (KM)—a topic fraught with abstraction and
high-minded ideas—into a series of effective principles and
guidelines illustrated through case studies at companies such
as Hewlett-Packard, McKinsey, and DaimlerChrysler. While the
book is not targeted at the project management community,
there are numerous examples of project-related KM practices.
Authors Etienne Wenger, Richard McDermott, and William
Snyder define three fundamental concepts—community,
domain, and practice—upon which their methodologies are
based. A community can range from an informal gathering of
professionals to an institutionalized group to just short of a
functional department or division. The common bond is the
domain, which is the purpose of the community or the topic
that it was organized to address, such as project management
for PMI. Last, as the authors state, the practice defines “a set of
socially defined ways of doing things … that create a basis for
action, communication, problem-solving, performance and
accountability.” Even within communities that share domains
and practices, there are variances in missions that include
helping members solve everyday problems; recognizing and
publicizing best practices; institutionalizing knowledge; and
encouraging innovation.
68 Project Management Journal
Armed with this basic model, Wenger and his co-authors
explore how communities of practice are born, live, and some-
times die. Seeking to increase the efficiency of deepwater explo-
ration projects, Shell Oil created networks of like-minded pro-
fessionals to share ideas, regardless of their associated project
or functional unit. One group consisted of geoscientists who
studied turbidite rock formations in the Gulf of Mexico, and it
consequently earned the colorful moniker “Turbodudes.” The
informal meetings are marked by members posing questions,
and the group responding with observations or suggestions. A
coordinator shepherds the discussion, ensures that knowledge
is captured, and works outside of meetings to keep members
engaged while fine-tuning the community’s approach. “With
so many meetings that aren’t relevant to your work, it’s nice to
go to one where we talk about rocks,” one geologist mused.
Over their multi-year life, the Turbodudes have increased the
return of Shell’s exploration efforts and added more than $100
million annually to the company’s bottom line.
So where’s the revolutionary thinking in groups of profes-
sionals gathering to talk about common problems? While
communities such as the Turbodudes may emerge organically
within companies or society, work is required to nurture them,
as is true with any meaningful relationship. In some situations,
more formal efforts are required to launch communities of
practice, which sometimes run up against conflicting corporate
cultures. The book’s value comes in its detailed dissection of
how this is done. Drawing upon learning theory, industrial
psychology, and organizational change management, the
authors instruct the reader how to build strength into commu-
nities and how to combat the pitfalls. There is no “one-size-fits-
all” way of creating communities, and a number of hybrid
approaches are explained for different types of organizations.
Especially relevant for modern project managers is the discus-
sion on how to build geographically distributed communities
of practice. (Hint: It takes more than having an e-mail address
and a home page.) In a few instances, the authors can be
accused of overreaching by alluding to how communities of
practice can cure a wide variety of societal ills, but their other
advice is mostly pragmatic.
Project managers must be attuned to project-related quality
assurance, risk management, and knowledge-transfer issues, as
well as how to increase their personal skills and efficiency with-
in the discipline. Easy to do but hard to do well, communities
of practice offer a proven method to accomplish these goals.
This book provides both a comprehensive overview and sug-
gested steps on how to create effective communities.
Harvard Business School Press, 2002, ISBN: 1-57851-330-8,
hardcover, 284 pp., $29.95.
Reviewed by Joseph Galarneau, PMP, a senior manager in
the media practice of KPMG Consulting in New York.
September 2002
Notes for Authors Guidelines for PMJ Book Reviews

Selecting Books for Review

PMJ welcomes recommendations from project managers and others regarding books
that may be of professional value to fellow PMI associates. Areas of potential interest
include: new ideas about the theory, concepts, and techniques of project management;
new approaches to technology and management; getting business results; competing
in today’s complex workplace; and global changes. Recommendations should include
the title, author, and publisher, and a brief statement as to why the book should be
considered for review. PMJ will select books for review and identify a reviewer.
Individuals recommending books for review also may volunteer to write the review.
However, individuals should not submit a review before PMJ has selected the book.
PMJ receives many books from publishers and authors and cannot review them all.

Guidelines for Writers

Reviews should begin with a strong, brief opening paragraph that identifies the book
and author, and tells the reader why the book is important. The review should not only
describe the content of the book, but also what the content means; that is, why it is a
contribution to the project management body of knowledge. Reviewers may include the
following elements:
■ A summary of key or unique concepts;
■ Favorite quote, graphic, chart, etc.;
■ Important tips or guidelines;
■ New terms or phrases, such as “knowbots” or “teamocracy;”
■ Message from the book that should be remembered for future use, or should have
been disclosed years ago.
Reviews should include the book’s strong points and any weak points if this information
will be useful to the reader. Reviews should be written in a conversational style that
maintains academic rigor. Reviewers should avoid use of the first person (“I”) and focus
on the book and its contents. Reviewers should also avoid use of extensive lists as a
means of describing or duplicating content. Instead, focus on what the content means
to readers. Reviews should be no longer than 750 words in length (please use your
computer word count to verify length of the review).

Reviews should include complete publishing information, if possible: title, author(s),

publisher (city and state), year published, ISBN number, total pages, and price in U.S.
dollars. PMJ will add any information that is not available to reviewers.

Reviews should be prepared using MS Word and may be submitted by e-mail (preferred)
or on disk. Submissions should include the name, title, company, address, phone/fax/
e-mail, and brief (one or two sentence) biosketch of the reviewer. Reviews should be
submitted to:

Book Review Editor

PMI Publications
natasha.pollard@pmi.org

PMI reserves the right to edit all material submitted for publication.

September 2002 Project Management Journal 69

 Calendar of Events

September 16–17 Successful Project Management for IT Professionals. Atlanta, GA, USA. For more information,
visit www.TLTraining.com or call +908-789-2800.

September 18–19 Successful Project Management for IT Professionals. Ft. Lauderdale, FL, USA. For more informa-
tion, visit www.TLTraining.com or call +908-789-2800.

September 24–27 Third International NAISO Symposium on Engineering of Intelligent Systems and 2002
Workshop on Information Systems for Mass Customization. Sponsored by the Natural and
Artificial Intelligence Systems Organization. University of Malaga, Malaga, Spain. For more infor-
mation, visit www.icsc-naiso.org/conferences/eis2002/index.html.

September 25–27 NORDNET 2002—International Project Management Conference. Reykjavik, Iceland.

Sponsored by the Project Management Association of Iceland (VSF). For more information,
visit www.vsf.is/nordnet2002.

October 2–4 First UK International Performance Management Symposium. Bristol, U.K. For more informa-
tion, visit www.mtc.aust.com/symposium/uk2002/main.html.

October 3–10 PMI® 2002 Annual Seminars & Symposium. Sponsored by the Project Management Institute.
Henry B. Gonzales Convention Center, San Antonio, TX, USA. For more information,
visit www.pmi.org/pmi2002.

October 10–11 Managing People in Projects. Princeton, NJ, USA. For more information, visit www.kepner-
tregoe.com or call +609-921-2806.

October 10–11 Managing People in Projects. San Francisco, CA, USA. For more information, visit www.kepner-
tregoe.com or call +609-921-2806.

October 14–16 Technical Project Management. New York, NY, USA. Sponsored by the American Management
Association. For more information, visit www.amanet.org.

October 16–18 Technical Project Management. San Francisco, CA, USA. Sponsored by the American
Management Association. For more information, visit www.amanet.org.

October 18 Project Management: The Journey, Not the Destination. PMI South Florida Chapter
Professional Development Seminar 2002. Fort Lauderdale, FL, USA. For more information,
visit www.southfloridapmi.org.

October 18 Puerto Rico Chapter Project Management Annual Symposium. Tropimar Convention Center,
Isla Verde, Carolina, Puerto Rico. For more information, e-mail j_rodriguez@actpr.com.

November 11–15 Project World. Navy Pier, Chicago, IL, USA. For more information, visit www.projectworld.com.

November 13–16 Professional Development Days. Minneapolis, MN, USA. Sponsored by the PMI Minnesota
Chapter. For more information, visit www.pmi-mn.org.

November 14 PMI Information Technology & Telecommunications SIG member teleconference. For more
information, visit www.pmi-ittelecom.org.

November 16 Project Management in the Government. Ritz Carlton, San Juan, Puerto Rico.

70 Project Management Journal September 2002

 Notes for Authors
Scope
The Project Management Journal is the profes-
sional journal of the Project Management
Institute (PMI®). The mission of PMJ is to
advance the state of the art of the knowledge
of project and program management. PMJ
presents useful information on both theory
and practice in the field of project manage-
ment. Authors are encouraged to submit the
following types of original manuscripts:
descriptions of innovative practices; sum-
maries of research results; reviews of current
literature; surveys of current practices; critical
analyses of concepts, theories, or practices;
developments of concepts, theories, or prac-
tices; analyses of failure. Manuscript length
should not exceed 12,000 words. The selec-
tion of manuscripts for publication is based
on the extent to which they advance the
knowledge and understanding of project
management. PMI® neither approves nor dis-
approves of any data, claims, opinions, or
conclusions presented.
Manuscript Review
PMJ uses a double-blind review process. The
first review of every manuscript is performed
by two or three anonymous referees (mem-
bers of the PMJ Editorial Review Board). The
manuscript is then either accepted, rejected,
or returned to the author for revision (with
reviewer comments furnished to the
author). Revised manuscripts are sent to the
Editor, who makes a final disposition. PMJ
strives to respond to all authors within three
months of the date the manuscript is
received at the PMI® Publishing Division.
Accepted manuscripts are subject to editori-
al changes. The author is solely responsible
for all statements made in the manuscript,
including editorial changes.
Original Publication
It is the policy of PMI® to be the sole, original
publisher of manuscripts. Manuscripts that
have been submitted simultaneously to other
magazines or journals will be rejected outright
and will not be reconsidered. Republication
of a manuscript, possibly revised, that has
been disseminated via conference proceed-
ings or newsletter is permitted if the Editor
judges there are significant benefits to be
gained from publication.
Copyright
Upon acceptance of a manuscript, authors will
be asked to transfer copyright of the article to
the publisher. This transfer will ensure the
widest possible dissemination of information.
This transfer of copyright enables PMI® to pro-
tect the copyrighted material for the authors,
but does not relinquish the author’s
September 2002
proprietary rights. The copyright transfer gives
PMI® the exclusive rights to republish or reprint
the manuscript in any form or medium, as well
as to grant or refuse permission to third parties
to republish all or parts of the manuscript.
Short Items
Short items do not need rigorous academic
scrutiny and are not refereed. Upon receipt,
however, these items become the copyright
property of PMI®.
■ Opinion presents thoughtful discussion of
project management issues.
■ Correspondence pertains to the project and
program management profession, including
references to literature, practice, and scholar-
ship as well as discussion and replies related
to articles published in PMJ.
■ Book Reviews express opinion about books
related to the project management profession,
or about general management or technical
books that cover topics of particular value to
the project manager.
■ Calendar of Events offers notices of forth-
coming meetings, conferences, and calls
for papers.
Submissions
Send manuscripts to: Editor, Project
Management Journal, Four Campus
Boulevard, Newtown Square, PA, 19073-3299
USA. Submit five copies of the manuscript on
8 1/2 x 11 inch paper, double spaced through-
out, and printed on one side only; and/or send
an electronic copy via e-mail to natasha.
pollard@pmi.org. Manuscripts should
include the following in the order listed:
■ A title page that includes the title of the
manuscript and each author’s name, affilia-
tion, mailing address, and phone, fax, and e-
mail address. Correspondence will be directed
only to the first author listed.
■ An abstract of 100 words or less that out-
lines the purpose, scope and conclusions of
the manuscript, and selected keywords.
■ Text (use headings and no more than
two levels of subheadings). To permit objec-
tive reviews by two referees, the abstract and
first page of the text should not reveal the
authors and/or affiliations, but only the
manuscript title.
■ References.
■ Figures and Tables (titled, numbered in
Arabic, with captions, each on a separate
sheet, preferred location indicated within the
body of the text).
■ Biographical details of each author.
Upon manuscript acceptance, authors
must also provide a final electronic version
with the information above, a black-and-
white passport-style professional photograph,
and a signed copyright agreement.
Project Management Journal
Computer-Generated
Text and Illustrations
Authors are requested to submit the final text
and illustrations on 3.5" IBM/compatible disks
or via e-mail. As with the requirements for man-
uscript submission, the main text, list of refer-
ences, table and figure captions, and author
biographies should be stored in separate text
files with clearly identifiable file names. Keep
the layout of the text as simple as possible and
save text in their original applications and/or
Rich Text Format (RTF). It is essential that the
name and version of the word processing pro-
gram and format of the text files are clearly indi-
cated (example: Word for Windows 95 doc).
Upon acceptance of the manuscript
for publishing, authors will also be asked to
provide illustrations in computer format.
Preferred formats are CorelDraw, MacroMedia
Freehand, Freelance Graphics, PowerPoint,
Harvard Graphics, Harvard Chart, or Canvas.
If one of these formats is unavailable, submit
in Windows Metafile (WMF), Adobe
Illustrator (AI or EPS) or Encapsulated
PostScript (EPS). Please use default/standard
extensions. If illustrations are available in
paper form only, they will be recreated elec-
tronically for publication. Contact PMI®
Publishing Division for further details.
Style of Text
You should write in clear and concise English.
Spelling should follow Webster’s New World
Dictionary. Authors whose native tongue is not
English are assured that in-house editorial
attention to their manuscript will improve
clarity and acceptability to readers. For ques-
tions regarding style and format of text, refer
to the Publication Manual of the American
Psychological Association, Fifth Edition.
References
For questions regarding reference format, refer
to the Publication Manual of the American
Psychological Association, Fifth Edition,
Bibliographic Forms for Journal Articles.
References used in the text should be identified
by author name and publication date in paren-
theses, e.g., (Cleland & King, 1983), and listed
alphabetically at the end of the manuscript.
Page numbers should be cited for all quota-
tions. Follow the format example shown below:
Baker, B. (1993). The project
manager and the media: Some
lessons from the stealth bomber pro-
gram. Project Management Journal, 24
(3), 11–14.
Cleland, D.I., & King, W.R. (1983).
Systems analysis and project management.
New York: McGraw-Hill.
71
 Hartley, J.R. (1992). Concurrent Float Checklist
engineering. Cambridge, MA: Productivity Funding ■ 5 copies of manuscript or sent via e-mail
Press. Human Resource Management ■ IBM/PC compatible disk (upon acceptance)
Information Systems ■ 100-word abstract
Please ensure that references are complete, Integration Management ■ Illustrations
that they include, where relevant, author’s Large Project ■ Author biographies
name, article or book title, volume and issue Leadership ■ Black-and-white passport-style professional
number, publisher, date and page reference. Life-cycle Costing author photographs (upon acceptance)
The use of page footnotes should be kept Manufacturing ■ Signed copyright agreement (upon
to a minimum. Footnotes should be num- Management Skills acceptance)
bered consecutively and listed at the end of Matrix Organization
the text as endnotes. Milestones Proofs
Mitigation Correspondence and proofs for correction
Keywords Monte Carlo Analysis will be sent to the first-named author unless
Keywords categorize your manuscript. They Multiproject Planning otherwise indicated. Copyediting of manu-
cover project management methodologies Negotiating scripts is performed by PMI® staff. The authors
and processes, tools and techniques, PMBOK® Networking are asked to check proofs for typographical
Guide knowledge areas, industries, types of New Product Development errors and to answer queries from editors. To
projects, geography. Please list three or four Organizational Planning improve publication times it is important that
keywords that best categorize your manu- Organizational Structure proofs be returned within three days. Authors
script. Choose from the following list of sug- Parametric Modeling may be charged for extensive corrections at
gested keywords (this is not a comprehensive Performance Reporting the proofing stage.
list) or you may use your own. Pharmaceuticals
Procurement Management Copies and Reprints
Accounting Productivity Authors will receive ten copies of the Journal
Activity Duration Estimating Project Life Cycle free of charge. Additional copies of the Journal
Agriculture Project Management Software and/or article reprints can be ordered at any
Arrow Diagramming Method Project Plan Development time from the PMI® Publishing Division.
Baselines Quality Assurance
Benchmarking Quality Management Project Management
Benefit/Cost Analysis Reengineering Institute Publishing Division
Budgeting Resource Planning Four Campus Boulevard
Change Control Responsibility Newtown Square, PA 19073-3299
Communications Management Risk Management USA
Concurrent Engineering Risk Response Development Tel: +610-356-4600
Configuration Management Schedule Development Fax: +610-356-4647
Conflict Resolution Schedule Control E-mail: natasha.pollard@pmi.org
Constraints Scope Management
Construction Scope Definition
Contingency Planning Scope Change Control
Contract Closeout Simulation
Cost Estimating Staff Acquisition
Cost Management Stakeholders
Critical Path Standards
Delegation Statistical Sampling
Deliverables Team Development
Design Time Management
Documentation Tools
Earned Value Training
Engineering Transportation Variance
Environment Utilities
Estimating Virtual Organization
Fast-Tracking Work Breakdown Structure
Feedback Work Packages
Finance

Index of Advertisers

Mindjet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C2

Primavera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C4

72 Project Management Journal September 2002

 CALL FOR PAPERS
Project Management Journal solicits unpublished papers
in project management and allied fields

The editor of the Project Management Journal is actively

seeking submissions of previously unpublished
research papers, commentaries, and dissertations in project management
as well as a number of related disciplines, including:

Organizational behavior

Organizational development

Software engineering

Construction engineering

Human resource management

Communications

General management

Papers from researchers and practitioners in allied fields should have a project management slant.
For more information on publishing in PMJ, please see the Notes for Authors on the PMI Web site at www.pmi.org.
Questions about submissions may be addressed to the PMJ Editor at natasha.pollard@pmi.org or via mail to:
PMJ Editor
PMI Publishing Dept.
Four Campus Boulevard
Newtown Square, PA 19073 USA