The Journal of Systems and Software 66 (2003) 7–21

www.elsevier.com/locate/jss

A quantitative and qualitative analysis of factors affecting

software processes
Austen Rainer *, Tracy Hall
Department of Computer Science, University of Hertfordshire, Hatfield Campus, Hatfield, Hertfordshire AL10 9AB, UK
Received 10 March 2001; received in revised form 12 November 2001; accepted 2 March 2002

Abstract

Despite the growing body of research on software process improvement (SPI), there is still a great deal of variability in the
success of SPI programmes. In this paper, we explore 26 factors that potentially affect SPI. We also consider the research strategies
used to study these factors. We have used a multi-strategy approach for this study: first, by combining qualitative and quantitative
analysis within case studies; second, by comparing our case study results with the results of a previously conducted survey study.
Seven factors relevant to SPI (i.e. executive support, experienced staff, internal process ownership, metrics, procedures, reviews, and
training) were identified by the case studies and the survey study. Two factors (reward schemes and estimating tools) were found, by
both the case studies and the survey study, not to be relevant to SPI. Three additional factors (people, problems and change) were
identified by the case studies. The frequency with which people, problems and change are discussed by practitioners suggests that
these three factors may be pervasive in SPI, in a way that the other factors are not. These factors, however, require further in-
vestigation.

2002 Elsevier Science Inc. All rights reserved.

1. Introduction disillusionment over the lack of resultsÕ, and another

A growing number of studies have examined the fac-
tors that affect software development projects and soft-
ware process improvement (SPI) programmes. These
studies include surveys of practitionersÕ opinions (Kitson
and Masters, 1993; Brodman and Johnson, 1994;
Herbsleb et al., 1994; Goldenson and Herbsleb, 1995;
Hayes and Zubrow, 1995; Herbsleb and Goldenson,
1996; El Emam et al., 1999; Paulk et al., 2000; Rainer and
Hall, 2002), case study reports of SPI in particular
companies (see Paulk (2000) for a bibliography of case
studies), and reviews of previous studies (e.g. Stelzer and
Mellis, 1998; Krasner, 1999).
Despite the number of studies, there is still a great
deal of variability in the success of SPI programmes. As
Cattaneo et al. (2001) point out, while Herbsleb and
Goldenson (1996) report quite positive results for
CMM-based SPI, 49% of their respondents had Ôa lot of
*
Corresponding author. Tel.: +44-1707-284-763; fax: +44-1707-284-
303.
E-mail address: a.w.rainer@herts.ac.uk (A. Rainer).
26% report that Ônothing much has changedÕ. Cattaneo
et al. also point out similar results from the SPICE trials
(SPICE, 1998, p. 159): 54% of respondents believe that
SPI is costing more than they had anticipated; 64% do
not believe that the assessment has had a major impact
on the organisation; and only 28% disagree with the
statement that Ônothing much has changed since the
assessmentÕ.
A number of explanations for the mixed results of
process assessments and SPI are available. For example,
the authors reporting the results of the SPICE trials
suggest two explanations: first, that the expectations of
those involved in SPI may not have been managed
properly; second, that insufficient resources were made
available. By contrast, Cattaneo et al. suggest a third
explanation: that most SPI efforts limit their attention to
the technical and engineering aspects of software de-
velopment, and do not consider the relationships be-
tween these engineering factors and organisational and
market factors.
We would like to consider two issues that relate to the
variable success of SPI: the factors that affect software
processes, and the research strategies that are used to

0164-1212/03/$ - see front matter
2002 Elsevier Science Inc. All rights reserved.
doi:10.1016/S0164-1212(02)00059-6
8 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
investigate these factors. We are interested in research
strategies because they influence the way in which a
phenomenon is investigated, and therefore the kinds of
findings that can be made, and the reliability of those
findings (Yin, 1994). As an example, Herbsleb and
Goldenson (1996) appear to rely exclusively on a survey
strategy. By contrast, the SPICE trials appear to rely
principally on a survey strategy, but supplement this
data with observation reports, problem reports, and
Ôfield experimentsÕ. A multi-strategy approach, such as
that used in the SPICE trials, can compensate for the
inherent weaknesses within any particular research
strategy (Yin, 1994).
As part of our investigations of SPI, we have adopted
an approach similar to that of the SPICE trials. We have
conducted both a survey study (Rainer and Hall, 2002)
and case studies of particular companies (Hall et al.,
2000b, 2001; Rainer et al., 2001). This allows us to ex-
plore the relative benefits of survey studies and case
studies for SPI.
In this exploratory paper, we quantitatively and
qualitatively analyse the words used by practitioners in
our case study group interviews to gain insights into the
factors that these practitioners think affect software
processes. We compare these analyses with the findings
from our survey study. This multi-strategy approach
should reveal insights into both software processes and
into research strategies.
To focus our study, we investigated the following
research questions:
RQ1 What factors, as identified by our case studies, af-
fect software processes?
RQ2 What factors, as identified by our case studies, do
not affect software processes?
RQ3 Are the factors identified from our survey study
consistent with the factors identified from our case
studies?
RQ4 Regardless of research strategy, how do the fac-
tors relate to each other?
RQ5 Is a multi-strategy approach a potentially useful
approach to investigating software development
and SPI?
The remainder of the paper is organised as follows. In
Section 2, we review previous work, including our sur-
vey study, to justify our multi-strategy approach. In
Section 3, we discuss the design of this study. Section 4
presents the results of our quantitative analysis of 18
ÔcoreÕ factors. Section 5 compares the results of our
quantitative analysis with the results of the survey study.
Section 6 presents the results of our quantitative analysis
of eight additional factors. Section 7 provides a quali-
tative analysis of some of the major factors that emerged
from the quantitative analysis. Section 8 evaluates our
multi-strategy approach. Section 9 presents conclusions.
2. Background
For our survey study (previously reported in Rainer
and Hall (2002)), we conducted three sets of analysis (i.e.
a general analysis, a maturity-based analysis, and a
success-based analysis) against three data sets (compa-
nies with appraised maturity, companies with self-rated
maturity, and companies with no rating of maturity).
We conducted this analysis to identify those factors that
have a major impact, or no impact, on implementing
SPI. Factors were identified as having a major impact
(or no impact) on SPI if at least 50% of the respondents
in the sample(s) reported that the factor had a major
impact (or no impact). Factors which the survey study
identified as having a major impact on SPI are presented
in Table 1. There was some evidence that two factors
(reward schemes and estimating tools) did not have an
impact on SPI.
As part of our survey investigation, we also reviewed
three previous studies, each of which has investigated
process standards that are recognised internationally:
• Goldenson and HerbslebÕs (1995) survey of CMM-
appraised organisations in North America.
• El Emam et al.Õs (1999) survey of companies involved
in the first phase of the SPICE trials.
• Stelzer and MellisÕ (1998) review of experience reports
and case studies of SPI and quality systems initia-
tives.
Factors identified by these three studies as being as-
sociated with successful SPI programmes are presented
in Table 2. Goldenson and Herbsleb also identified a
number of factors associated with unsuccessful SPI
programmes: organisational politics, turf guarding,
discouragement and cynicism from previous unsuccess-
ful improvement experience, belief that SPI gets in the
way of real work, and the need for more guidance on
how, not just what, to improve. As noted earlier, we
found in our survey study that reward schemes and es-
timating tools may not have an impact on SPI pro-
grammes.
There are some similarities between the findings of
the four studies. In particular, all four studies recognise
the contribution of senior management and the in-
volvement of staff in the improvement effort (which we
have termed internal process ownership in our survey
study). Some of the factors identified by Stelzer and
Mellis suggest ways in which other factors may be im-
plemented. For example, encouraging communication
and collaboration, and recruiting and promoting change
agents and opinion leaders from the technical staff, may
be ways to involve staff in SPI.
There are also a number of differences between the
findings, mainly due to the fact that different studies
have investigated different factors. These differences
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21 9

Table 1
Factors having a major impact on SPI programmes (Rainer and Hall, 2002)
Factor Generala Maturityb Success
High Low Appraised Self-rated Not-rated
Training and mentoring Yes Yes Yes Yes Yes
Reviews Yes Yes Yes
Standards and procedures Yes Yes
Experienced staff Yes Yes
Inspections Yes Yes
Internal process ownership Yes Yes
Internal leadership Yes
Executive support Yes
Metrics Yes
Total 4 8 2 2 2 1
A blank cell indicates that the factor was not identified as having a major impact by the respective sample.
a
The general analysis makes no distinction between appraised, self-rated and not-rated data.
b
The high-maturity and low-maturity samples include appraised and self-rated data.

Table 2
Factors affecting SPI, as identified by other researchers (Rainer and Hall, 2002)
Factor Goldenson El Emam Stelzer
Senior management commitment Yes Yes Yes
Clear and relevant SPI goals Yes Yes Yes
Clear, compensated assignment of responsibility for SPI Yes Yes
Staff involvement Yes Yes Yes
SPI people highly/well respected Yes Yes
Staff time and resource. Yes Yes
Creating process action teams Yes
Change agents and opinion leaders Yes
Encouraging communication and collaboration Yes
Managing the SPI project Yes
Providing enhanced understanding Yes
Stabilising changed processes Yes
Tailoring improvement initiatives Yes
Unfreezing the organisation Yes
An ÔemptyÕ cell indicates that the factor was not studied by the respective researchers.

indicate that SPI researchers are still seeking to identify
the key factors that affect SPI programmes. Dyb

(2000), for example, claims to have identified nearly 200
prescriptions for success, from an extensive review of
quality management, organisational learning and SPI
literature. Our investigations are a contribution to this
search.
All four studies use a survey strategy, although in
different ways. Goldenson and Herbsleb, El Emam
et al., and our study are all questionnaire surveys. By
contrast, Stelzer and Mellis survey previous literature.
In addition to these survey studies, there are many case
studies of SPI. Table 3 summarises 39 case reports, or-
ganised according to the companies (cases) investigated.
(We consider these to be important publications because
the 11 organisations include all five of the organisations
that have been awarded the IEEE Computer Society
Award for Process Achievement. In addition, 12 of 14
sites within these companies have been assessed at
CMM Level 3 or higher. We have examined these 39
cases in more depth in Rainer and Hall (2001)).
As already noted above, the body of case studies and
a survey studies on SPI is suggesting a large number of
factors that might affect the success of SPI. It is possible
to compare these studies (e.g. Stelzer and Mellis, 1998)
in an effort to abstract more reliable and more widely
applicable findings. One inevitable problem with such
comparisons, however, is that there is no coherent,
overall design to the studies being reviewed (e.g. because
they were conducted by different researchers, at different
times, in different settings, with different objectives). A
multi-strategy design (such as that of the SPICE trials)
allows a comparison of results from different research
strategies within a coherent, overall research design.
3. Study design
3.1. An overview of the design
We conducted group interviews with three types
of practitioners (i.e. developers, project managers and
10 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21

Table 3
Publications relating to 11 key organisations in SPI
Organisation Publications
Advanced Information Services Ferguson et al. (1999)
Boeing Fowler (1997), Kness and Satake (1997), Wigle and Yamamura (1997), Yamamura and Wigle (1997),
Yamamura (1999) and Raman (2000)
Hughes Humphrey et al. (1991) and Willis et al. (1998)
Motorola Daskalantonakis (1992, 1994), Diaz and Sligo (1997) and Fitzgerald and OÕKane (1999)
NASA SEL McGarry et al. (1994), NASA (1994), SEMATECH (1995), NASA (1996) and Pajerski and Basili (1997)
Oklahoma City Air Logistics Butler (1995, 1997) and Butler and Lipke (2000)
Center
Raytheon Dion (1992, 1993), Haley et al. (1995), Haley (1996), Oshana (1998) and Raytheon Corporate Communi-
cations (1998)
Schlumberger Wohlwend and Rosenbaum (1993), Wohlwend and Rosenbaum (1994) and Schlumberger (2000)
Siemens Paulish and Carleton (1994), Gonauser et al. (1998) and Mehner et al. (1998)
Space Shuttle Software Project Billings et al. (1994), Krasner et al. (1994), Paulk et al. (1994) and Florac et al. (2000)
Telcordia Technologies Ferrera and Timko (2000) and Pitterman (2000)

senior managers) in four telecommunications compa-
nies. This is part of a wider study, involving group in-
terviews with 13 companies and a survey of over 80
companies. (See Rainer et al. (2001) for more detail.)
Each group interview was attended by between four and
six practitioners of the same type i.e. four to six devel-
opers, or four to six project managers, or four to six
senior managers. We used a template of questions to
direct the group discussion. Each interview was taped
and subsequently transcribed. In quantitatively analy-
sing the transcriptions, we chose to investigate 26 fac-
tors, and used sets of words to represent each factor.
Our criterion for a factor relevant to SPI was that one or
more of the factorÕs associated words should occur in
50% or more of the transcriptions. We compared those
factors identified in the case studies with those factors
identified from our survey study. We also qualitatively
analysed some of these factors to gain some general idea
of how they might relate to each other.
3.2. Factors investigated
This study has investigated the presence of 26 factors
within software process and SPI. In referring to ÔfactorsÕ,
we are not making a distinction between causes and
effects, but are referring to the presence of 26 issues. For
example, two of the factors that we investigate are ex-
perienced staff and quality. It may be that there is a
causal relationship between experience (perhaps tradi-
tionally known as a factor) and quality (an outcome).
We are not, however, investigating such relationships
within this study. The results of this study may be useful
in suggesting a candidate set of factors within which to
subsequently search for cause–effect relationships.
3.2.1. Core factors
Our survey study examined the impact of 16 factors.
Two of these factors (standards and procedures, and
training and mentoring) were separated into four fac-
tors, to allow for a more detailed examination. This gave
us 18 ÔcoreÕ factors to investigate.
3.2.2. Additional factors
To more thoroughly understand the factors that im-
pact SPI programmes, and in recognition that SPI
programmes vary in their success, we also examined
eight additional factors. These factors were chosen be-
cause they relate to recurring issues in the software
process research community. SPI is directed at reducing
costs and schedule and improving quality, so we wanted
to see whether practitioners discussed these issues. SPI is
also concerned with identifying and resolving problems
within the development process, and SPI responds to,
requires, and also initiates, change. Change may come
from the market, so we investigated markets. Finally,
there are relationships between process and people e.g.
the creation, implementation and conformance to pro-
cesses depend on people.
3.3. Sample
We collected survey data from 84 companies, and
group interview data from 13 companies (these 13
companies being a subset of the 84 companies). Focus-
ing on companies that develop a particular type of
software system allowed us to control for some of the
organisational and commercial issues surrounding soft-
ware development. Our subsample still allowed a variety
of maturity levels, and a reasonable Ôsample sizeÕ of
interviews (15 in total). We chose to concentrate on
telecommunications companies because telecommuni-
cations is a leading commercial market, resulting in the
requirement for, and development of, Ôleading edgeÕ
technology.
Table 4 provides some characteristics of the four
companies, taken from responses to the survey study.
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
Table 4
Characteristics of the four cases
Company CMM Level CMM assessment method
1 1 Self-estimate
2 1 Self-estimate
3 4 Formal appraisal
4 3 Formal appraisal
(See Appendix D for further information on some of the
responses reported in Table 4.) All four companies are
ISO 9001 certified and have had SPI programmes for
over 5 years. Companies 1 and 2 are sites within one
larger company, which is over 50 years old, and the two
companies each develop both hardware and software.
By contrast, companies 3 and 4 are within separate
larger organisations, both of these organisations being
between 10 and 20 years old.
While companies 1 and 2 claim to have formal SPI
programmes, and use the CMM in full, they have not
formally assessed their maturity levels. This may be
because they are using the CMM as preparation for an
assessment, or it may be that the companies do not think
that an assessment is necessary. Another possibility is
that informal assessments suggest that the company has
not matured to CMM Level 2, and so a formal assess-
ment would not be considered worthwhile. It is not
possible to tell from the data whether companies 3 and 4
have been using the CMM for the duration of their SPI
programmes, or have only recently started to use the
CMM. Their maturity levels (4 and 3 respectively)
would suggest that they have been using the CMM for
some time (cf. Hayes and Zubrow, 1995).
Company 4 has been formally assessed at CMM
Level 3 and yet their SPI programme is only marginally
successful (according to the respondentÕs answer to a
particular survey question). Perhaps company 4Õs ob-
jectives are to attain Level 4 or Level 5 and so Level 3 is
judged as ÔonlyÕ marginally successful. The variability of
SPI success in our four telecommunications companies
ÔmapsÕ well to the variability of SPI, as reported by
Herbsleb and Goldenson, and the SPICE trials.
Responses from the questionnaire also indicate that
company 4 makes full use of an alternative model to the
CMM. Companies 1, 2 and 3 make partial use of an
alternative model to the CMM.
3.4. Analysis
As we have conducted group interviews with practi-
tioners, we are particularly interested in the analysis of
natural language, While there are many case studies of
the software process, and SPI, few provide detailed in-
formation on the techniques used to analyse the data
collected.
11
Formal approach to SPI Successful SPI Number of group interviews
Yes Unknown 5
Yes Unknown 5
Yes Yes 3
Yes Marginally 2
Natural language analysis has been discussed in the
context of software development for a number of years.
For example, almost two decades ago, Abbott (1983)
provided advice on using nouns to identify data types.
Others having developed such ideas further (e.g. Chen,
1983; Saeki et al., 1989; Carley, 1997; Misic and Russo,
2000). Because the analysis of natural language is a
relatively novel approach to investigating the software
process, we have looked outside of the software engi-
neering research literature to gather advice on this ap-
proach. In particular, we have drawn upon: BromleyÕs
(1977) account of analysing ordinary language descrip-
tions of personality; HolstiÕs (1969) guide to content
analysis as an approach to documentary research;
StraussÕs (1987) handbook for qualitative analysis for
social science; and Miles and HubermanÕs (1994)
sourcebook of qualitative data analysis.
3.4.1. Units of analyses
We have used two main units of analyses: the group
interview (transcript) and the company. The group in-
terviews provide a finer-grained unit of analysis, and we
have aggregated interviews to establish a company unit
of analysis. For the comparison of the results from the
survey study with these results, we have used a third unit
of analysis: the study.
3.4.2. The interviews
The researchersÕ questions for the group interviews
asked about various aspects of the software develop-
ment process and SPI. These questions were open-ended
and intended to give practitioners freedom to discuss
issues within the context of a particular aspect of soft-
ware development. (See Appendix B for more informa-
tion.) Given that these were group interviews, efforts
were made to compensate for undesirable group dy-
namics. For example, in some interviews, particular in-
dividuals attempted to dominate the group discussions.
The researchers compensated for this by encouraging
other practitioners to contribute.
3.4.3. Factors and words
We assume that a practitioner is more likely to discuss
those issues that they consider relevant, rather than those
issues that they consider not relevant. To simplify our
analysis, we also assume that a set of words can be used
12 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
to represent a factor, so that if a word relating to a factor
was used by at least one member of a group then the
group was considered to have discussed that factor.
(Word frequency then provides an indication of how
important a factor is to a group of practitioners.)
Clearly, this second assumption is disputable because,
for example, a word can mean different things in different
contexts. Given that one of our units of analyses is the
group interview, provided at least one occurrence of a
word in a transcript is within a suitable context, then the
practitioners in that group interview have discussed the
respective factor that the word represents. Furthermore,
the questions asked by the research should help to nar-
row the possible contexts, increasing the likelihood that
the words occur in an appropriate context and thus in-
creasing the likelihood that the words represent the fac-
tors. We discuss this issue in more depth in Section 3.4.5.
We began with 161 words that were used to identify
the 26 factors in our study (see Appendix A for more
information). In assessing the preliminary word fre-
quencies we noted that many words were either not used
or very rarely used. These words did not really con-
tribute to the analysis. In response to this, and to
sharpen our analysis, we chose to focus on those words
that were used in at least 50% of the transcripts. We
found as a result of this refinement that:
• 32 of the original 161 words accounted for almost
80% of the citations.
• Only eight of the 18 core factors were cited in 50% or
more of the transcriptions.
• Seven of the eight additional factors were cited in
50% or more of the transcriptions.
The choice of a 50% ÔthresholdÕ is consistent with our
previous study, in which factors were only identified as
having a major impact or no impact on SPI if 50% or
more of the survey respondents considered that the
factor had either a major impact or no impact.
3.4.4. Qualitative analysis of the transcripts
In order to gain a more thorough understanding of
the factors identified by the quantitative analysis, we
used NVivo (QSR International Pty Ltd., 1999) to ex-
tract each and every occurrences of a word, together
with the surrounding paragraph, from every transcript.
For pragmatic reasons we could not investigate all fac-
tors. Therefore, we focused on the factor people, and our
investigations of this factor revealed connections be-
tween this factor and a number of the other 25 factors.
We identified these connections by noting that words
relating to different factors were occurring within the
same paragraphs. The qualitative analysis allowed us to
take account of context and allowed us to ensure that
co-occurrence of words did suggest a meaningful con-
nection between factors.
3.4.5. Threats to the study
The main threat to this study is the problem of using
a set of words to represent a factor. This problem occurs
because, for example, the same word can ÔhaveÕ different
meanings in different contexts, and different words can
ÔhaveÕ the same meaning in the same or different con-
texts. The lack of context when quantitatively analysing
words introduces some further problems:
• There is little indication of the direction of impact of
a factor e.g. whether a factor has a major positive im-
pact or a minor negative impact.
• A word may be used in a context that is irrelevant to
this study. This can bias the results.
• The occurrence of a word may be conditional on the
previous occurrence of a word. For example, one re-
spondent may use a word to refer to one of our fac-
tors and this may ÔtriggerÕ other people to discuss
that factor and to use that word. We alluded to this
earlier, where a dominant individual may discourage
the discussion of some factors, or encourage the dis-
cussion of other factors.
• The frequency of occurrence of a word may actually
be an indication of the complexity of the factor,
rather than an indication of the relevance of that fac-
tor to SPI.
We have taken the following steps to compensate for
these problems.
• We used sets of words to represent a factor. The
problem still remains, however, that the sets of words
do not completely and comprehensively ÔcaptureÕ the
factor. Consequently, we conducted some prelimi-
nary and informal analysis to check whether the sets
of words do represent the factors. As noted earlier (in
Section 3.4.3) we also refined the words that we used
in the analysis.
• We chose to focus on the frequency of occurrence of
a factor across transcripts, rather than the frequency
of occurrence of a factor within each transcript. Pro-
vided that one or more words relating to a factor oc-
curred in a transcript, we considered that the
practitioners in that group interview had discussed
the factor. With this criterion, we only need one word
to genuinely refer to the factor.
Word frequencies within a transcript may provide
some indication of relevance, but this is together with
some indication of complexity. We cautiously present
word frequencies, and cautiously interpret them in
our discussion of the additional factors, in Section 8.
• We compared the results of the analysis of words
with results from the survey study.
Another threat to this study may be the lack of op-
erational definitions of the factors. This paper is at-
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21 13

tempting to identify, in a general way, issues that prac-
titioners consider relevant to SPI. Having identified
general issues, the next step might be to Ôsharpen upÕ our
understanding of these issues. This is where operational
definitions would be developed. In the qualitative anal-
ysis for this paper (Section 7) we have started to clarify
our understanding of the people factor. Further work on
this factor and other factors like change and problems
are clearly required. Therefore, we see operational defi-
nitions as being a future outcome of this research rather
than a present outcome.
4. Quantitative analysis of the core factors
This section reports our investigations of the first and
second research questions:
RQ1 What factors, as identified by our case studies, af-
fect software processes?
RQ2 What factors, as identified by our case studies, do
not affect software processes?
4.1. Factors identified by the four companies
Table 5 summarises the eight factors that were cited
by the majority of types of practitioner groups, in each
company. Of the eight factors, seven are cited by prac-
titioners in all four companies, and two (executive sup-
port and experienced staff) are cited by all 15
practitioner groups.
The fact that only eight of the 18 core factors are cited
by the majority of practitioners is surprising, given some
of the findings of our previous study. For example, in our
previous study we identified training and mentoring (a
combined factor in the previous study) as the most
commonly cited factor having an impact on SPI. Table 5
indicates, however, that mentoring is not discussed by
the majority of practitioner groups. (Indeed, the word
frequencies indicate that this factor was not discussed by
any of the practitioner groups. See the Appendix C for
information on word frequencies.) The differences be-
tween the case studies and the survey study suggest that
different strategies reveal different factors and that a
multi-strategy approach is necessary for a more reliable
understanding of factors affecting SPI.
A more detailed examination of the citations by
practitioner groups reveals that the factor metrics is not
discussed by any of the practitioner groups in company
2. Yet metrics is discussed by at least one instance of
each type of practitioner group in companies 1 and 3,
and by project managers in company 4. Of the eight
factors, metrics is the only factor that might distinguish
the lower maturity companies (companies 1 and 2) from
the higher maturity companies (companies 3 and 4). It is
also the only factor that might distinguish those com-
panies that can make a judgement on the success of their
SPI programmes from those that cannot make a
judgement. (Company 3 reports a successful SPI pro-
gramme, whilst company 4 reports a marginally suc-
cessful SPI programme. Both companies 1 and 2 report
that they did not know about the success of the pro-
gramme.) It may be, however, that companies 1 and 2
cannot judge the success of their SPI programmes be-
cause they are at a low level of maturity. This is a point
recognised by, for example, Catteneo et al.:
In immature companies, it is difficult and some-
times impossible to base an improvement action
on quantitative evaluation. (Cattaneo et al., 2001,
p. 17)
We have further explored practitionersÕ opinions of
metrics in (Hall et al., 2000a, 2001).
5. Comparison with the survey study
This section reports our investigations of the third
research question:

Table 5
Factors identified by each of the four companies
Factor Company Total number Normalised Word freq.
1 2 3 4 of transcripts total

Executive support Yes Yes Yes Yes 15 11 176

Experienced staff Yes Yes Yes Yes 15 11 176
Reviews Yes Yes Yes Yes 14 10 100
Internal process ownership Yes Yes Yes Yes 13 10 39
Training Yes Yes Yes Yes 10 9 30
Procedures Yes Yes Yes Yes 10 8 64
Stringent control Yes Yes Yes Yes 9 8 12
Metrics Yes No Yes Yes 8 7 34
The normalised total takes account of differences in the actual number of practitioner groups in each company. For example, there were actually three
developer groups in company 1, but only one developer group in company 4. If one or more of the developer groups in company 1 cited the factor
executive support, then these citations are counted just once in the normalised total. The normalised total is, therefore, a count of citations of each
factor per type of practitioner group, per company. The normalised total provides a more reliable indicator of practitionersÕ opinions on each factor.
14 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21

RQ3 Are the factors identified from our survey study One factor (stringent control) was identified as rele-
consistent with the factors identified from our case vant in the case studies. There was no clear indication, in
studies? the survey study, as to whether stringent control had a
major impact or no impact on SPI programmes. It may
Table 6 compares the factors identified in the survey be that stringent control has a minor impact. The minor
with the factors identified in the case studies. As the impact of a factor was not, however, investigated in the
table indicates, there are a number of similarities be- survey study.
tween the two studies. Seven factors are identified by
both studies as having some relevance to SPI. Two
factors (reward schemes and estimating tools) are con- 6. Quantitative analysis of additional factors
sidered, in both studies, to have no relevance to SPI.
(For reward schemes, Goldenson and Herbsleb (1995) Because over 50% of our core factors were not cited
drew a similar conclusion from their study.) in at least 50% of the transcriptions, and in an effort to
Four factors (inspections, internal leadership, stan- more thoroughly understand the factors that impact SPI
dards and mentoring) were identified in the survey, but programmes, we also examined the presence of eight
were not identified in the case studies. The most sur- additional factors. To be consistent with our analysis of
prising of these is mentoring, because this emerged (to- the core factors we also refined the word list for these
gether with training) as the most cited factor in the additional factors. Table 7 summarises the additional
survey study (cf. Table 1). factors that were cited by the majority of types of
practitioner groups, in each company.
While the factors quality and schedule are cited by all
companies, and a majority of texts and practitioner
Table 6 groups, the frequencies for these two factors suggest that
Comparison of factors from the survey study with factors from the they are not discussed in much detail. This may be be-
case study
cause the discussions focused on the implementation of
Factor Survey study Case studies SPI rather than the outcomes (see the interview ques-
Executive support Yes Yes tions in Appendix B).
Experienced staff Yes Yes
We recognised, in Section 3, that there are difficulties
Internal process ownership Yes Yes
Metrics Yes Yes interpreting the word frequencies from our analysis.
Procedures Yes Yes Accepting these difficulties, we are interested in the fre-
Reviews Yes Yes quency with which the additional factors are cited. The
Training Yes Yes factors change, people, problems and SPI are cited
Inspections Yes No much more frequently than the other additional fac-
Internal leadership Yes No tors. Indeed, these four factors are cited more frequently
Standards Yes No than any of the ÔcoreÕ factors, although the core fac-
Mentoring Yes No
tors executive support, experienced staff and reviews
Stringent controla Uncertain Yes have frequencies of the same order of magnitude (see
Reward schemes No No Table 5).
Estimating tools No No It is unsurprising that the factor SPI is mentioned so
a
For stringent control, it was not clear whether the factor had a frequently, given that the focus of the interview ques-
major impact or no impact. See Rainer and Hall (2002) for more in- tions was on SPI. For the other three factors, their high
formation. frequency of citation suggests that these factors may be

Table 7
Additional factors cited by the four companies
Factor Company Normalised Total number Word freq.
1 2 3 4 total of transcripts

Change Yes Yes Yes Yes 11 15 214

People Yes Yes Yes Yes 11 15 415
Problems Yes Yes Yes Yes 11 15 321
S. process improvement Yes Yes Yes Yes 11 15 256
Quality Yes Yes Yes Yes 10 13 98
Market Yes Yes Yes Yes 10 12 31
Schedule Yes Yes Yes Yes 9 11 30
Costa No No No No
a
The factor cost was not retained in the refined word list. It is shown here for completeness. See Table 5 for an explanation of normalised totals.
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
particularly interesting and relevant to SPI. It may be
that these factors are much more important to success-
ful SPI than the ÔcoreÕ factors, or that these three factors
are more abstract versions of some of the ÔcoreÕ factors.
The importance of people has been repeatedly raised
by researchers and practitioners within software engi-
neering and SPI (e.g. Kellner et al., 1991; Cugola and
Ghezzi, 1998; Baddoo et al., 2000; van Solingen et al.,
2000). Less attention has been directed explicitly at
change, but such disciplines as SPI, management learn-
ing and organisational development are fundamentally
about change. Similarly, there appears to be little re-
search that takes a broad perspective on the range of
actual problems faced by practitioners. Most research
appears to begin with a particular problem, and then
develops the investigation from that problem. An obvi-
ous implication of these discussions is that addressing
people, problems and change may lead to more effective
SPI. This implication requires further investigation.
One implication of the frequency of the factor people
is that although researchers and practitioners recognise
the importance of people to software engineering, and to
SPI, they may not recognise just how important people
are. As de Bono states:
Most of our thinking is not to do with puzzles or
games. Most thinking is to do with other people.
(De Bono, 1994, p. 77)
This suggests that people pervade many of the other
factors in SPI. This relates to our second interpretation:
that people, change and problems may be more abstract
factors than the ÔcoreÕ factors. For example, executive
support and experienced staff clearly relate to people.
Training relates to change. Reviews and inspections are
intended to find certain types of problems. This inter-
pretation requires further investigation.
7. Qualitative analysis of the factors
In this section, we investigate the fourth research
question:
RQ4 Regardless of research strategy, how do the fac-
tors relate to each other?
During the qualitative analysis of the transcriptions,
two general insights about the factors emerged. First,
some factors appear to be closely linked with each other.
Second, factors in addition to the 26 that we have in-
vestigated (but that relate to some of these 26 fac-
tors) emerge from the analysis. The nature of our
analysis, and of the data, make specific or comprehen-
sive descriptions of these links, and of the new factors,
difficult. We offer two ÔsketchesÕ of some of the rela-
15
tionships between factors, and use these ÔsketchesÕ to
indicate the kinds of issues that need to be addressed in
further research.
One implication of the close relationships between the
factors is for survey research. Many of the studies, in-
cluding our own, of factors affecting SPI present a list of
factors, perhaps ranking the importance of these factors.
These lists imply (although not necessarily intentionally)
that these factors can be treated independently of each
other. Our qualitative analysis suggests that attention
should be directed at how such factors relate to each
other.
7.1. Experienced staff, people, training, reviews and
inspections
There appears to be relationships between the factors
experienced staff, people, training, reviews and inspections.
Clearly, one reason that people are so important is for
their expertise and experience. This is recognised by the
need for training, as one method of increasing, or im-
proving, expertise. One developer, in company 2, stated:
We can whinge now that we have lost all our exper-
tise and that people only want experienced staff but
itÕs our fault that the less experienced staff did not
gain more experience when we had the opportunity
to train them.
This may also be an example of the need for men-
toring, a factor which was not explicitly mentioned in
any group interview.
The issue of experienced staff also relates to a po-
tentially new, unexplored factor: recruitment policy. A
senior manager from company 1, stated:
Another problem is that historically we mainly re-
cruit people straight from university, people with
very little experience. We get very few people who
have experience from elsewhere.
Reviews or inspections also provide an opportunity
for sharing expertise:
I think [that] the important thing about a review
inspection process is that a senior, experienced
designer can make a contribution into many differ-
ent pieces of development but in a managed and
controlled way. . . (Project manager)
This suggests an alternative perspective: one contri-
bution of training, reviews, inspections, standards,
procedures, and mentoring is that they all share exper-
tise. This also appears to be one of the functions of
processes and standards i.e. to ÔcaptureÕ, store and dis-
seminate knowledge.
16 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
7.2. People, procedures, change and ownership
Comments in the transcriptions suggest that under-
standing the importance of the factor people is, at least
partly, about understanding:
• A personÕs ability to do work. This relates to experi-
ence and expertise, and to shared expertise.
• A personÕs ability to change, which involves a recog-
nition and willingness to accept that their knowledge
and work habits are no longer effective, but also a
recognition and willingness to accept that the pro-
posed changes will be effective.
For example, a developer from company 1 stated:
In our local process we have our own review proce-
dure, which we developed about 4 to 5 years ago
because we weren’t happy with what was there at
a corporate level. (emphases added)
In this example, the developer indicates that they
were prepared to change (to develop a review procedure)
but that they were not prepared to accept the corporate
process. This example further suggests that part of the
problem of corporate standards is that they do not
(maybe they cannot) take account of all local situations.
This affects a personÕs willingness to change when they
do not believe that corporate standards will be effective
in their own situations. Ownership of procedures may
occur when people recognise that the procedure is useful
for them in their local situation. The provision of a
corporate-wide procedure, without evidence of how this
procedure will help in the local situation, may be one
reason people are reluctant to change.
8. Evaluation of our multi-strategy approach
8.1. An evaluation of our study design
In this section we investigate the final research
question:
RQ5 Is our approach a potentially useful approach to
investigating software processes?
There is a great deal of variability in the success of
SPI programmes, and there are a number of explana-
tions for this variability. We examined the factors that
affect software processes, and examined the research
strategies used to investigate these factors. We examined
the research strategies because the use of any particular
type of research strategy will influence the kinds of data
that can be collected from practitioners. We have used a
multi-strategy approach for this study: first, by com-
bining qualitative and quantitative analysis within our
case studies; second, by comparing our results with our
previous survey study results. We found that:
• Many of the factors identified in the survey study were
also identified in the case studies. Also, both types of
studies found these factors to have similar kinds of rel-
evance. The similarity of results is encouraging be-
cause they come from two different types of study.
• There were a small number of factors where the two
types of study produced opposing results. In one
instance, mentoring, the differences were dramatic.
The survey study found training and mentoring, as a
combined factor, to be the factor most obviously rel-
evant to SPI. (This does not mean that training and
mentoring was the factor with the greatest impact.)
Within the case studies, however, training was rec-
ognised by the majority of practitioner groups,
whereas mentoring was not recognised by any of
the practitioner groups. This absence of mentoring
contradicts other research (Billings et al., 1994; Kras-
ner et al., 1994; Paulk et al., 1994). It may be that, due
to the problems of using individual words to repre-
sent a factor, certain factors were not properly repre-
sented by a set of individual words.
A related issue is that an alternative case study design
(e.g. one that used a different set of words to repre-
sent a factor) may also ÔdetectÕ different factors. The
different results of the two studies suggests potential
design problems for both the survey study and the
case studies. For the survey study, there was a design
problem with investigating a combined factor. For
the case studies there is the problem of using words to
represent factors. (This was discussed in Section
3.4.5.)
• Although the two types of study are identifying sim-
ilar factors, the case studies may have identified more
fundamental factors i.e. people, change and prob-
lems. While a survey study could be designed to in-
vestigate these factors, the nature of survey studies
limits the ways in which these factors could be stud-
ied. The very high frequencies with which people,
change and problems are cited suggests that these fac-
tors are complex and that they relate to many issues
within software development. The high frequencies
further suggest that practitioners do not realise how
often they refer to these issues.
• The case studies reveal potentially relevant factors
and relationships between factors in ways that the sur-
vey studies could not.
9. Conclusions
Despite the growing body of research on SPI, there is
still a great deal of variability in the success of SPI
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21 17

Table 8
Factors and words used in the analysis
Factor Original words Revised words Factor Original words Revised words
Automated tools Automation Problems Difficult Difficult
Automate Difficulty
Automated Issue Issue
Automatic Issues Issues
Cost Cost Obstacle
Costs Obstacles
Change Change Change Problem Problem
Changed Changed Problems Problems
Changes Changes Procedures Framework
Changing Changing Guidance
Reorganisation Guide
Estimating tools Estimate Guideline
Estimates Guidelines
Estimating Procedure Procedure
Overestimate Procedures Procedures
Overestimated Project post-mortem Mortem
Overestimates Mortems
Underestimate Quality Defect
Underestimated Defects
Underestimates Quality Quality
Executive support Executive Reviews Review Review
Executives Reviewed
ExecutiveÕs Reviewing
Management Management Reviews Reviews
ManagementÕs Reward schemes Incentive
Manager Manager Incentives
Managers Managers Reward
ManagerÕs Rewarded
Experienced staff Experience Experience Rewarding
Experienced Rewards
Experiences Risk assessment Risk
Expert Risked
Expertise Risks
Experts Schedule Date
Knew Deadline
Know Know Deadlines
Knowing Deliver Deliver
Knowledge Delivered
Skill Delivering
Skilled Delivers
Skills Delivery Delivery
External consultants Consultant Finish
Consultancies Finished
Consultancy Finishes
Consultants Milestone
Inspections Inspections Milestones
Fagan Schedule
Fagen SPI Improve Improve
Inspect Improved Improved
Inspected Improvement Improvement
Inspecting Improves
Inspection Improving
Inspects PI PI
Internal leadership Leader SPI
Leaders Standards Standards
Leadership Standard
Internal process ownership Ownership Standardise
Own Own Standardised
Owns Standardised
Market Industries Standarize
Industry Training Educate
IndustryÕs Educated
Market Market Educates
(continued on next page)
18 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21

Table 8 (continued)
Factor Original words Revised words Factor Original words Revised words
Markets Educational
Market’s Taught
Mentoring Mentor Teach
Mentored Train
Mentoring Training Training
Mentors Tutorial
Metrics Measure Tutorials
Measured Stringent control Control Control
Measurement Controlled
Measures Controlling
Measuring Controls
Metric Stringent
Metrics Metrics
People Culture
Cultures
Human
Humans
People People
Peoples
People’s
Person Person

programmes. We have considered two issues that relate
to this variability of success: the factors that affect
software processes, and the research strategies that are
used to investigate these factors.
In terms of the factors, we found that the relevance of
several factors to SPI was repeated across a survey study
and a multiple case study. These factors are: executive
support, experienced staff, internal process ownership,
metrics, procedures, reviews and training. We also
found two factors, reward schemes and estimating tools,
which the survey study and multiple case study found
was not relevant to SPI.
The case studies also investigated additional factors
not investigated by the survey study. Three of these
additional factors appear to be pervasive in SPI. They
are: people, change and problems. The possible rela-
tionships between these factors can be expressed with
two conjectures: people change in response to problems;
and, problems occur because people do not change.
In terms of the research strategies, the comparison of
results from the survey study and case study suggest that
a multi-strategy approach is necessary for a more com-
prehensive and reliable understanding of SPI.
Acknowledgements
We are sincerely grateful to all the companies and
practitioners for their participation in our Managing
Practitioner Impact on Process and Product (PPP) pro-
ject. The PPP project is funded by the UKÕs Engineering
and Physical Science Research Council, under grant
number EPSRC GR/L91962. We are grateful to Nathan
Baddoo for his helpful comments on an earlier draft of
this paper, and to Nathan and Prof. David Wilson for
their help in collecting the data. We also thank the re-
viewers for their helpful comments.
Appendix A. Factors and words
Table 8 presents the original list of 161 words and the
revised list of 32 words, as used in the word frequency
analysis of the group interview transcripts. No distinc-
tions were made between upper-case and lower-case
spellings.
Appendix B. Interview questions
We used the following questions to guide the group
interviews.
What do you think SPI is?
What do you think the problems and issues in soft-
ware development at company X are?
How will SPI address these issues?
What are the strengths of your current software de-
velopment process?
What are the weaknesses of your software develop-
ment process?
What do you think the aims and objectives of SPI at
company X should be?
What do you think managementÕs aims and objec-
tives of SPI at company X are?
What are the potential obstacles to SPI at company
X?
What are the potential motivators for SPI at com-
pany X?
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21 19

Table 9
Factors identified in the original and revised word lists
Factor Original word list Revised word list
Number of transcripts Freq. of citation Number of transcripts Freq. of citation
Executive support 15 179 15 176
Experienced staff 15 235 15 176
Reviews 14 113 14 100
Internal process ownership 13 46 13 39
Metrics 13 60 8 34
Training 12 40 10 30
Procedures 11 75 10 64
Stringent control 11 17 9 12
Inspections 10 67 0 0
Estimating tools 7 24 0 0
Internal leadership 7 17 0 0
Standards 7 27 0 0
Risk assessment 4 10 0 0
Reward schemes 3 5 0 0
Project post-mortem 2 3 0 0
Automated tools 1 1 0 0
External consultants 1 1 0 0
Mentoring 0 0 0 0
Total 920 631
% citations retained N/A 69
People 15 471 15 415
Problems 15 336 15 321
S/w process improvement 15 276 15 256
Change 15 218 15 214
Schedule 12 108 11 30
Quality 13 106 13 98
Market 12 48 12 31
Cost 9 25 0 0
Total 1588 1365
% citations retained N/A 86

Appendix C. Factors identified by the majority of prac- 2. How successful has your improvement programme
titioners been so far?

Table 9 summarises two sets of word frequencies for

Response Company
the 26 ÔcoreÕ and additional factors. The first set is based
on the original 161 words. The second set is based on the Not successful
refined 32 words. Marginally successful 3
Successful 4
Very successful
Appendix D. Survey questions and responses on SPI Not able to assess 1, 2
success Do not know

Some of the data in Table 4 was taken from responses

to the following two survey questions: References
1. In relation to your companyÕs attempts to improve
its approach to software development, how formal is Abbott, R.J., 1983. Program design by informal English descriptions.
your companyÕs current approach Communications of the ACM 26 (11), 882–894.
Baddoo, N., Hall, T., Wilson, D.N., 2000. Implementing a people
focused SPI programme. In: 11th European Software Control and
Metrics Conference and The Third SCOPE Conference on Soft-
Response Company ware Product Quality, Munich, 18–20 April.
Formal (documented process) Billings, C., Clifton, J., Kolkhorst, B., Lee, E., Wingert, W.B., 1994.
Journey to a mature software process. IBM Systems Journal 33 (1),
Informal (undocumented process) 1, 2, 3, 4
46–61.
20 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
Brodman, J.G., Johnson, D.L., 1994. What small businesses and small
organizations say about the CMM. In: 16th International Confer-
ence on Software Engineering (ICSE16), Sorrento, Italy, 16–21
May, pp. 331–340.
Bromley, D.B., 1977. Personality description in ordinary language.
John Wiley & Sons, London.
Butler, K.L., 1995. The economic benefits of software process
improvement. CrossTalk 8 (7), 1–5.
Butler, K.L., 1997. Process lessons learned while reaching level 4.
CrossTalk 10 (5), 1–6.
Butler, K.L., Lipke, W., 2000. Software process achievement at Tinker
Air Force Base. Software Engineering Institute, Carnegie Mellon
University, Oklahoma, Pittsburgh, CMU/SEI-2000-TR-014.
Carley, K.M., 1997. Extracting team mental models through tex-
tual analysis. Journal of Organizational Behavior 18 (S1), 533–558.
Cattaneo, F., Fuggetta, A., Sciuto, D., 2001. Pursuing coherence in
software process assessment and improvement. Software Process––
Improvement and Practice 6 (1), 3–22.
Chen, P.P.-S., 1983. English sentence structure and entity-relationship
diagrams. Information Sciences, 29.
Cugola, G., Ghezzi, C., 1998. Software processes: A retrospective and
a path to the future. Software Process––Improvement and Practice
4 (3), 101–123.
Daskalantonakis, M.K., 1992. A practical view of software measure-
ment and implementation experiences within Motorola. IEEE
Transactions on Software Engineering 18 (11), 998–1010.
Daskalantonakis, M.K., 1994. Achieving higher SEI levels. IEEE
Software 11 (4), 17–24.
De Bono, E., 1994. De BonoÕs thinking course. BBC Worldwide Ltd.,
London.
Diaz, M., Sligo, J., 1997. How software process improvement helped
motorola. IEEE Software 14 (5), 75–81.
Dion, R., 1992. Elements of a process-improvement program. IEEE
Software 9 (4), 83–85.
Dion, R., 1993. Process improvement and the corporate balance sheet.
IEEE Software 10 (4), 28–35.
Dyb
a, T., 2000. An instrument for measuring the key factors of success
in software process improvement. Empirical Software Engineering
5 (4), 357–390.
El Emam, K., Fusaro, P., Smith, B., 1999. Success factors and barriers
for software process improvement. In: Tully, C., Messnarz, R.
(Eds.), Better Software Practice for Business Benefit: Principles and
Experience. IEEE Computer Society, pp. 355–371.
Ferguson, P., Leman, G., Perini, P., Renner, S., Seshagiri, G., 1999.
Software process improvement works! Software Engineering Insti-
tute, Carnegie Mellon University, Pittsburgh, PA, CMU/SEI-99-
TR-027.
Ferrera, L., Timko, C., 2000. The Telcordia Technologies road to
quality. Cutter IT Journal 13 (2), 28–34.
Fitzgerald, B., OÕKane, T., 1999. A longitudinal study of software
process improvement. IEEE Software 16 (3), 37–45.
Florac, W.A., Carleton, A.D., Barnard, J.R., 2000. Statistical process
control: Analyzing a space shuttle onboard software process. IEEE
Software 17 (4), 97–106.
Fowler, K.M., 1997. SEI CMM level 5: A practitionerÕs perpsective.
CrossTalk 10 (9).
Goldenson, D.R., Herbsleb, J.D., 1995. After the appraisal: A
systematic survey of process improvement, its benefits, and factors
that influence success. Software Engineering Institute, Carnegie
Mellon University, CMU/SEI-95-TR-009 (alternatively ESC-TR-
95-009).
Gonauser, M., Paulisch, F., V€ olker, A., 1998. SiemensÕ experience with
people/process/technology: Lesson learned. 3rd Annual European
Software Engineering Process Group (ESEPGÕ98) Conference,
London, June 1998, C307a 1-36.
Haley, T.J., 1996. Software process improvement at raytheon. IEEE
Software 13 (6), 33–41.
Haley, T.J., Ireland, B., Wojtaszek, E., Nash, D., Dion, R., 1995.
Raytheon electronic systems experience in software process im-
provement. Software Engineering Institute, Carnegie Mellon Uni-
versity, Pittsburgh, PA, CMU/SEI-95-TR-017.
Hall, T., Baddoo, N., Wilson, D.N., 2001. Measurement in software
process improvement programmes: An empirical study. In: Dumke,
R., Abran, A. (Eds.), New approaches to software measurement
Proceedings of IWSM 2000. Springer-Verlag, pp. 73–83.
Hall, T., Baddoo, N., Wilson, D.N., Rainer, A.W., 2000a. Optimising
software measurement programmes using practitioner input. In:
Australian Conference on Software Measurement, Sydney, 1st–3rd
November.
Hall, T., Wilson, D.N., Baddoo, N., 2000b. Towards implementing
successful software inspections. In: International Conference on
Software Methods and Tools (IEEE Computer Society), Wollon-
going, Australia, 6–10 November.
Hayes, W., Zubrow, D., 1995. Moving on up: Data and experience
doing CMM-based process improvement. Software Engineering
Institute, CMU/SEI-95-TR-008.
Herbsleb, J., Goldenson, D., 1996. A systematic survey of CMM
experience and results. In: 18th International Conference on
Software Engineering (ICSE-18), Berlin, Germany, 25–26
March.
Herbsleb, J., Carleton, A.D., Rozum, J., Siegel, J., Zubrow, D., 1994.
Benefits of CMM-based software process improvement: Initial
results. Software Engineering Institute, Carnegie Mellon Univer-
sity, Pittsburgh, CMS/SEI-94-TR-013.
Holsti, O.R., 1969. Content analysis for the social sciences and
humanities. Addison-Wesley, London.
Humphrey, W.S., Snyder, T.R., Willis, R.R., 1991. Software pro-
cess improvement at Hughes Aircraft. IEEE Software 8 (4), 11–
23.
Kellner, M.I., Curtis, B., DeMarco, T., Kishida, K., Schlumberger,
M., Tully, C., 1991. Panel: Non-technological issues in software
engineering. In: 13th International Conference on Software Engi-
neering (ICSE-13), Austin, Texas, 13–17 May, pp. 144–
155.
Kitson, D.H., Masters, S.M., 1993. An analysis of SEI software
process assessment results: 1987–1991. In: 15th International
Conference on Software Engineering, Baltimore, Maryland.
Kness, S.P., Satake, M.S., 1997. A level 5 organization looks at the
personal software process. CrossTalk 10 (10).
Krasner, H., 1999. The payoff for software process improvement:
What it is and how to get it. In: El Emam, K., Madhavji, N.H.
(Eds.), Elements of software process assessment & improvement.
IEEE Computer Society, Washington.
Krasner, H., Pyles, J., Wohlwend, H., 1994. A case history of the space
shuttle onboard systems project. SEMATECH Inc., Austin, TX,
94092551A-TR.
McGarry, F., Pajerski, R.S., Page, G., Waligora, S., Basili, V.R.,
Zelkowitz, M., 1994. Software process improvement in the NASA
software engineering laboratory. Software Engineering Insti-
tute, Carnegie Mellon University, Pittsburgh, PA, CMU/SEI-94-
TR-22.
Mehner, T., Messer, T., Paul, P., Paulisch, F., Schless, P., V€ olker,
A., 1998. Siemens process assessment and improvement ap-
proaches: Experiences and benefits. In: The Twenty-Second Annual
International Computer Software and Applications Conference
(Compsac Õ98). IEEE Comput. Soc, Los Alamitos, CA, USA.
Miles, M.B., Huberman, A.M., 1994. Qualitative data analysis. SAGE
Publications, London.
Misic, M.M., Russo, N.L., 2000. Reading between the lines: An
examination of systems analysis and design texts. Journal of
Systems and Software 50 (1), 65–73.
NASA, 1994. An overview of the software engineering laboratory.
NASA Goddard Space Flight Center, Greenbelt, Maryland, SEL-
94-005.
 A. Rainer, T. Hall / The Journal of Systems and Software 66 (2003) 7–21
NASA, 1996. Software process improvement handbook (revision 1),
NASA Goddard Space Flight Center, Greenbelt, Maryland,
NASA-GB-001-95.
Oshana, R.S., 1998. Tailoring Cleanroom for industrial use. IEEE
Software 15 (6), 46–55.
Pajerski, R.S., Basili, V.R., 1997. The SEL adapts to meet changing
times. In: 22nd Software Engineering Workshop, NASA/Goddard
SEL, Greenbelt, Maryland, December 1997.
Paulish, D.J., Carleton, A.D., 1994. Case studies of software-process-
improvement measurement. Computer 27 (9), 50–57.
Paulk, M.C., 2000. Software CMM case study bibliography. Software
Engineering Institute. Available from <http://www.sei.cmu.edu/
cmm/docs/roi.html>.
Paulk, M.C., Weber, C.V., Curtis, B., Chrissis, M.B., 1994. A high-
maturity example: Space shuttle onboard software. In: Paulk,
M.C., Weber, C.V., Curtis, B., Chrissis, M.B. (Eds.), The capability
maturity model: Guidelines for improving the software process.
Addison-Wesley, Harlow, England.
Paulk, M.C., Goldenson, D., White, D.M., 2000. The 1999 survey of
high maturity organizations. Software Engineering Institute, Car-
negie Mellon University, CMU/SEI-2000-SR-002.
Pitterman, B., 2000. Telcordia technologies: The journey to high
maturity. IEEE Software 17 (4), 89–96.
QSR International Pty Ltd., 1999. QSR NVivo. http://www.qsr.co-
m.au/.
Rainer, A.W., Hall, T., 2001. An analysis of some Ôcore studiesÕ of
software process improvement. Software Process––Improvement
and Practice 6 (4), 169–187.
Rainer, A.W., Hall, T., 2002. Key success factors for implementing
software process improvement: A maturity-based analysis. Journal
of Systems and Software.
Rainer, A.W., Hall, T., Baddoo, N., Wilson, D.N., 2001. An overview
of the practitioners, processes and products project. In: 6th Annual
Conference of the UK Academy of Information Systems (UK-
AIS2001), 18–20 April, University of Portsmouth, UK.
Raman, S., 2000. It is software process, stupid. IEEE AES Systems
Magazine (June).
Raytheon Corporate Communications, 1998. RaytheonÕs command
and control division awarded software engineering instituteÕs level
five rating. Press release, 7th January, Raytheon, Arlington,
Virginia.
Saeki, M., Horai, H., Enomoto, H., 1989. Software development
process from natural language specification. In: 11th International
Conference on Software Engineering, Pittsburgh, PA, 15–18 May.
IEEE Computer Society, pp. 64–73.
Schlumberger, 2000. Schlumberger joins the worldÕs software elite––
award confirms quality of processes and people. Press release, 27th
November, Schlumberger.
21
SEMATECH, 1995. A case history of process improvements at the
NASA software engineering laboratory, Sematech, Austin, TX,
94122662A-TR.
SPICE, 1998. SPICE Phase 2 trials interim report version 1. 17th June
1998. http://www.iese.fhg.de/SPICE.
Stelzer, D., Mellis, W., 1998. Success factors of organizational change
in software process improvement. Software Process––Improvement
and Practice 4 (4), 227–250.
Strauss, A.L., 1987. Qualitative analysis for social scientists. Cam-
bridge University Press, Cambridge.
van Solingen, R., Berghout, E., Kusters, R., Trienekens, J., 2000.
From process improvement to people improvement: Enabling
learning in software development. Information and Software
Technology 42 (14), 965–971.
Wigle, G.B., Yamamura, G., 1997. Practice of an SEI CMM level 5
SEPG. CrossTalk 10 (11). Available from <http://www.stsc.hill.
af.mil/CrossTalk/1997/nov/cmm5sepg.asp>.
Willis, R.R., Rova, R.M., Scott, M.D., Johnson, M.I., Ryskowski,
J.F., Moon, J.A., Shumate, K.C., Winfield, T.O., 1998. Hughes
AircraftÕs widespread deployment of a continuously improving
software process. Software Engineering Institute, Carnegie Mellon
University, Pittsburgh, PA, CMU/SEI-98-TR-006.
Wohlwend, H., Rosenbaum, S., 1993. Software improvements in an
international company. In: 15th International Conference on
Software Engineering (ICSE), Baltimore, Maryland, pp. 212–220.
Wohlwend, H., Rosenbaum, S., 1994. SchlumbergerÕs software
improvement program. IEEE Transactions on Software Engineer-
ing 20 (11), 833–839.
Yamamura, G., 1999. Head to head: Process improvement satisfies
employees. IEEE Software 16 (5), 83–85.
Yamamura, G., Wigle, G.B., 1997. SEI CMM level 5: For the right
reasons. CrossTalk 10 (8). Available from <http://www.stsc.hil-
l.af.mil/crosstalk/1997/aug/seicmm5.asp>.
Yin, R.K., 1994. Case Study Research: Design and Methods. SAGE
Publications, London.
Austen Rainer is currently Research Fellow in the Department of
Computer Science at the University of Hertfordshire. He received a
B.Sc. (Hon) in Applied Psychology and Computing from Bourne-
mouth University and, subsequently, a Ph.D. in Software Engineering
Management in collaboration with IBM Hursley Park. He has previ-
ously worked in the software industry.
Tracy Hall is currently Principal Lecturer in the Department of
Computer Science at the University of Hertfordshire. Previously she
was Head of the Centre for Systems and Software Engineering at
South Bank University. She has published widely in the area of soft-
ware quality and measurement. Following B.A. and M.Sc. degrees she
was awarded her Ph.D. from City University.