CHAPTER 2

LITERATURE REVIEW

This chapter explores the literature that is relevant for understanding the
development and interpreting the results of this research. This chapter provides an
overview of previous research on statistical process control, software metrics and CMM.

2.1 Statistical Process Control

The software business’s significant impact on today’s economy generates
considerable interest in making software development more cost effective and producing
higher quality software (Shull et al., 2006). An Information Week survey conducted in
2003 found that 62 percent of their respondents feel that the software industry has trouble
producing good quality software (Hayes, 2003). Losses due to inefficient development
practices lead to inadequate quality that cost the US industry approximately $60 billion
per year (Thibodeau and Rosencrance, 2002).

Over the past decade, the concepts, methods and practices associated with process
management and continual improvement have gained wide acceptance in the software
community (Florac et al., 2000). These concepts, methods and practices embody a way of
thinking, acting and understanding the data generated by processes that collectively result
in improved quality, increased productivity and competitive products.

The acceptance of these approaches in software industry requires a method for

measuring software processes in terms of effectiveness, efficiency, timelines,
predictability, improvements and product quality. Traditional software measurement and
analysis methods, which provide status at a point in time and compare it against the plan,
are not sufficient for determining past process performance or for predicting process
performance.

Statistical Process Control is a statistical based approach, which can determine

whether a process is stable or not by discriminating between the presence of common
cause variation and assignable cause variation (Baldassarre et al., 2004). Control charts

30

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 are used to check for process stability (Woodall, 2000). It is a well-established technique,
which has shown to be effective in manufacturing processes but not yet in software
process contexts. Managers and engineers in various disciplines have used control charts
for many years and have decided that control charts provide the basis for making process
decisions and predictions. SPC and Control Charts have received little attention in most
software organizations, but this scenario is changing steadily. Software companies have
started realizing the benefits of the empirical methods associated with SPC and are now
concentrating more on their potential for improving software products and services.
Owing to this increasing interest in SPC in software industry, this section focuses on
providing a complete understanding of SPC in software processes and reviews some
work done in this field.

2.2 Application of SPC in Software Industries

The utilization of SPC in software development is studied by a number of
researchers. Statistical process control is a powerful technique for managing, monitoring,
analyzing and improving the performance of a process through the use of statistical
methods (Mahanti and Evans, 2012). Some of these studies are based on process
improvement models and focus on SPC as a way of achieving high process maturities
(Sargut and Demirors, 2006; Jalote and Saxena, 2002). These model-based studies mostly
represent Capability Maturity Model understanding on the utilization of Statistical
Process Control for software process improvement. One of these studies belong to
Humphrey (1989), who describes a framework for software process management,
outlines the actions to provide higher maturity levels and acts as a basic guide to improve
processes in a software organization. Florac and Carleton (1999a) focus on the principles
and methods for evaluating and controlling the process performance depending on
Shewhart’s (1939) principles.

While the benefits of SPC are well founded for manufacturing companies, there
have been many debates (Card, 1994; Kan, 1995; Keller, 1995) about its application in
software industry. It is frequently argued that the inherent characteristics of software
cause difficulties to apply SPC techniques. Nevertheless, the interest to apply SPC
techniques in the software industry has grown as more organizations improve their

31

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 processes by utilizing models such as Capability Maturity Model (Paulk et al., 1993),
Capability Maturity Model Integration (CMMI Product Team, 2001) and SPICE
(ISO/IEC 15504-4:1998).

Keller (1995) debates the applicability of SPC to software processes. A seven-step

guideline for successful SPC implementation in a software organization was provided
and this study reveals four important points for the application of SPC to software
processes:
 Metrics should correlate to the quality characteristics of the products that are
defined by the customer
 Metrics should be selected for the activities that produce tangible items
 SPC should be applied only to critical processes
 The processes should be capable of producing the desired software product

Kan (1995) discusses the theory and application of metrics in software

development. The study found out that control charts can be used as a supplementary tool
during software development to improve quality during construction of defect and
reliability models. However, the author also cautions that, it is not possible to provide
control as in manufacturing since the parameters being charted are usually in-process
measures instead of representing the final product quality. The final product quality can
only be measured at the end of a project as opposed to the production in manufacturing
industry, so that on-time control on processes becomes impossible. The author also
underlines the necessity of maturity for achieving process stability in software
development. Finally, the work is concluded by stating that the processes can be regarded
in control when the project meets in-process targets and achieves end-product quality
goals.

Radice (1998) describes SPC techniques constrained within software domain and
gives a detailed tutorial based on theoretical knowledge with practical experiences. The
author states that all SPC techniques may not be applicable for software processes and
gives XmR and u-charts as possible techniques. The study also explains the relevance of
SPC for CMM Level 4 and regards backing-off control charts in Level 4 as a mistake.

32

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 The work states five problems with control charts: too much variation; unnecessary use
of control charts; lack of enough data; lack of specification limits from the clients; the
idea that control charts cannot be used with software processes.

The literature also includes studies that present practical experiences of SPC in
software domain. Weller (2000) provides one of the rare practical examples by presenting
the SPC implementation details from a software organization. The researcher uses X and
moving range charts for the lines of code inspected per hour for each inspection and
achieves a stable inspection process after removing the outliers from the dataset. Then u-
chart is drawn for the defect density data for each inspection. By using these findings,
reliable estimations are drawn for inspection effectiveness and it gains an insight on when
to stop testing.

Card (1994) discusses the utilization of SPC for software by also considering
some of the objections and mentioning about possible implementation problems. A
control chart example is given for testing effectiveness measure and concludes that SPC
principles can be beneficial for a software organization even if formal statistical control
techniques are used.

Florac et al., (2000) present their analysis findings from a collaborative effort
between SEI and Space Shuttle Onboard Project. They first emphasize the importance of
selecting key processes, providing operational definitions, addressing issues of data
homogeneity and rational subgrouping, using the correct control charts, understanding
multiple-cause and mixed-cause systems, finding and testing trial limits and recalculating
limits. Then they perform an SPC study on package review data by implementing each of
these factors. Finally they summarize the benefits of applying SPC to software processes.

Jalote and Saxena (2002) move ahead on the idea of 3 - sigma control limits and
propose a model for the calculation of control limits to minimize the cost of type 1 and
type 2 errors. The foundation of this study is a pioneering one as it questions an accepted
practice for control charts and the results of the example studies are encouraging.

33

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 However, the study is far from being practical one as it includes too many parameters and
assumptions.

The usability of SPC to software development is also discussed in a panel named

“Can Statistical Process Control be Usefully Applied to Software” in European Software
Engineering Process Group (SEPG) conference in 1999. In the panel, Keller (1995)
mentions the importance of SPC on management decision making and forecasting. The
author also outlines the observations and results for a specific SPC implementation.
Hirsch (1999) states that SPC charts should be used to gather desired benefits and the
managers should be trained to ensure use of SPC. The necessity of embracing the value
of metrics, having a defined and repeatable process, having a defined and repeatable
metric program and having curiosity about metrics for usefully applying SPC to software
is also mentioned. Meade (1999) demonstrates a synopsis from the SPC utilization as part
of Level 4 implementation in Lockheed Martin Company. The importance of
understanding the data is specified and the work also reveals that it is not possible to
apply SPC to all software metrics. Finally the results of SPC study in the company are
portrayed, which shows that smaller programs are better able to perform SPC. Wigle
(1999) questions SPC implementation in software organizations and stresses the
importance of applying SPC at a level where decision making occurs. Heijstek (1999) has
experience on statistical studies in Ericsson and shows the lack of data quality as the
main analysis problem. As referred by Paulk and Carleton (1999), Card emphasizes the
importance of process stability and delineates lack of well-defined business objectives to
guide collection and analysis as the major problem.

Card et al., (2008) analyzes the usage of SPC in software development, to find the
phase of application and why SPC in that particular phase. From this study it is found that
under right conditions, SPC can be used as another toolkit that can be mixed or integrated
with other software tasks.

2.3 CMM and Software Process

A standard is a rule or basis for the comparison that is used to assess the size,
content, value or quality of an object or activity. Thus, it defines what characteristics a

34

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 product or a process shall possess. On the other hand, a guideline is a suggested practice,
method or procedure, typically issued by some authority (Humphrey, 1989). Therefore,
they lead process owners in performing their work although the rules are not mandatory.
In practice, today’s software standards include both mandatory rules and suggested
actions which are prepared by very experienced people in the related domain in long
periods of time. Moreover, they evolve over time with the accumulation of new practices,
tools, techniques and methodologies.

Because of the complexity of processes and the diversity of products, standards

play a significant role for software development. And measurement is one of the
important processes for which specific regulations are defined within software standards.
The horizon of measurement is continuously growing day by day. Accordingly, the newer
versions of software standards and guidelines begin to direct software organizations to
utilize statistical techniques like reliability analysis, 6 sigma analysis and SPC.

Quantitative Process Management is one of the Level 4 key process areas of

CMM (Paulk et al., 1993), which involves establishing goals for the performance of the
project's defined software process, taking measurements of the process performance,
analyzing these measurements and making adjustments to maintain process performance
within acceptable limits. In the description of Commitment 1, the term “quantitative
control” is referred to as any quantitative or statistically-based technique appropriate to
analyze a software process, identify special causes of variations in the performance of the
software process and bring the performance of the software process within well-defined
limits. Although the control chart is given only as an example for statistical analysis (not
mandatory), CMM implicitly supports its utilization by mentioning about calculation of
mean and variances, identifying control limits and establishing process performance
baselines. CMM also necessitates the collection of data from the projects to characterize
the process capability in the organizational level. Therefore, the software organizations
are trying to use control chart in order to meet the CMM Level 4 requirements.

On the other hand, CMMI (2001) analyzes process performance in two process
areas:

35

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

  Organizational Process Performance
 Quantitative Project Management

Organization process performance deals with establishment of process

performance baselines for the organization’s standard software processes and quantitative
project management aims to quantitatively manage the project’s defined process to
achieve the project’s established quality and process performance objectives. Actually
these two process areas work in harmony. The organizational process performance
baselines, which are established by the data from all the projects, are used in order to
control project level process performances. The project data are then added to the data
pool in order to update organizational level process performance baselines.

Quality Process Management (QPM) process area of CMMI aims to statistically

manage a project’s defined software processes (Sub-goal 2). CMMI differs from CMM in
that it gives more detail about the statistical management. It gives more practical
information to the software organizations as it is based on the realization that statistical
techniques are not appropriate for all the processes. It necessitates identification of tasks
and appropriate measures for those tasks to perform statistical analyses; it defines criteria
for selecting the measures and for determining whether historical data are comparable.
Similar to CMM, control chart is given as an example for potential statistical analysis
techniques. However, the requirements can best be met by implementing control chart
analysis.

SPICE is an assessment model, which implicitly provides software organizations

a framework to improve their processes (ISO/IEC TR 15504-5, 1998). In SPICE, the
analysis is performed in two dimensions: Process Dimension and Capability Dimension.
Capability Dimension is divided into 6 levels and Level 4 (Predictable Process) has
measurement (PA 4.1) and process control (PA 4.2) attributes. Measurement attribute
requires a software firm to analyze trends in process performance and maintain process
capability within defined limits across the organization. Similarly, Process Control
attribute necessitates control of process performance to maintain control or implement
improvement.

36

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 The importance of quantitative methodologies is mainly because good decision
making depends on how much you know about the process and the product. As defined in
metrics guidebook of National Aeronautics and Space Administration (NASA) (Pajerski
and Sova, 1995), knowing what an organization does and how it operates is a
fundamental requirement for any attempt to plan, manage or improve. On the other hand,
Tracey O’Rourke, CEO of Allen-Bradley, says “without the right information, you’re just
another person with an opinion”. Thus information is essential for understanding of
processes and this leads to better management as far as the information is valid.

A detailed description on the need for measurement in project management is

given in Project Management Body of Knowledge (PMBOK) (IEEE Computer Society,
1998). It defines project management as: “the application of knowledge, skills, tools and
techniques to project activities in order to meet project requirements”. It investigates
project management in nine knowledge areas and five process groups. PMBOK describes
inputs, outputs and relevant tools and techniques for the major processes in the
knowledge areas in order to aid as a guideline for project managers. Careful analysis of
PMBOK reveals that control charts, Pareto diagrams, statistical sampling and trend
analysis are referred as techniques for quality control process. As PMBOK is a guideline,
it does not have any obligatory nature for the software firms. However, it provides useful
practices for good project management and SPC techniques are also recommended within
this guide.

CMM in its new version CMMI and SPICE implicitly direct software companies
to implement SPC as a crucial step for achieving higher process maturities. They suggest
control charts for both project level process control and organizational level process
improvement purposes. The traces of SPC are also found in software process guidelines
such as PMBOK and NASA measurement guidelines. Actually, it is possible to utilize
SPC methodologies to comply with some other software standards such as (International
Standard Organization (ISO) 9000-3, which require determination of applicable statistical
methods to monitor measure and analyze the processes. This trend shows that software
community has high expectations from statistical applications of SPC techniques. As

37

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 seen, the results of practical SPC applications in the industry will be designative for its
place in the future.
Basili (1989) has made a statement about software development as “we have
begun to understand software development and it is not an easy task. There is no simple
set of rules and methods that work under all circumstances”.

There are many software process models introduced already. Some of the models
mention here are: Waterfall, revised waterfall, prototyping, incremental, Rapid
Application Development (RAD), spiral and eXtreme Programming (XP) are some of the
major process models to mention.

Software development is not an easy task for all the existing models to help them
master their problems. As the software evolves, the existing models has to be modified
even then, the set modifications cannot help much to find the solution of important
problems like cost, budget, over running of schedules or effort deviations, poor quality
software, etc.,

Davis, (1995) has discussed towards this briefly as “the software community
redefines the problem by shifting its focus from product issues to process issues. While
the natural tendency of a pendulum is to come to rest at a point midway between two
extremes, the software community’s focus constantly shifts”

Assessment and its pre-requisite, measurement, may be the weakest point of any
software organization. Brady and DeMarco, (1994) have made the following statement
about measurement:

“Only in software do people cling to the illusion that it’s OK to come up with
estimates of the future, even though you’ve never measured anything in the past.”

Successful management of any process requires planning, measurement and

control. There must be some means of measuring completeness of the product at any
point of its development by inspections or testing. Finally, the measured data must be
used for controlling the process.

38

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 The process work should be measured or assessed but above all, it should be
improved. Existing theories couldn’t solve the problems so new solutions have to be
introduced.

Software process improvement has become the primary approach to improve

software quality and reliability, employee and customer satisfaction and return on
investment (Mathiassen et al., 2005).

Based on the software CMM, ISO/IEC-9126 (ISO/IEC TR 9126-3, 2000) and

similar norms, SPI aims to build infrastructure and culture that support effective methods,
practices and procedures and integrate into the ongoing way of doing business. This goal
has proven difficult to reach, however, because SPI involves organizational change on a
scale and of a complexity that requires commitment, resources and skills at all
organizational levels and it carries a large risk of failure (Aaen, 2003).

Over the past decades, software companies have spent vast resources on SPI.
Although there have been a lot of studies aiming at better software, the results have not
changed dramatically since those studies and now research has begun (Aaen, 2003).

It is very hard to make changes from developers to managers and to make the
situation worse, many managers lack experience. SPI efforts need commitment at all
levels. The managers should know the organization elements tactics and contexts that
facilitate successful change for improvement.

If SPI used for project management process, the new process may result in faster
cycle times, shorter time to market, higher customer satisfaction, accurate time and
budget accounting and better cost and schedule performance (Rico, 2004).

To change the whole process SPI accesses the existing process. These points are
the vital data for improvement or to make changes in the whole process. Small software
companies do not have the accurate measurements and hence successful implementation
is not possible, all the time and some of them even do not have a system for
measurement. Till 1994, 75 percent of the industry did not collect metrics and hence
could not be measured (Yorden, 1994).

39

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 Two impressive success stories among others are from the Systems Engineering
and Analysis Support (SEAS) Center of the Computer Sciences Corporation (CSC), US,
(Putnam and Myers, 1996) and a Danish company which has not been revealed
(Mathiassen et al., 2005). SEAS from US, by initiating an SPI program in 1995, had
become the sixth organization in the world to attain the Software Engineering Institute’s
CMM Level 5, back in 1998. Although SEAS has been a company with a long
experience with processes and carrying out SPI programs, it is still interesting that they
have achieved this in just three years. Just a year after SEAS had kicked off its SPI
program, it was predicted that at the average rate of increase it would take 10 years to
advance from CMM Level 2 to Level 3. The Danish company, which was a medium-
sized software house (140 employees), in 1997, had aimed for reaching CMM Level 3 in
less than three years. Although the organization had not reached its goal in three years, it
was formally assessed at CMM Level 3 in 2002 and at CMMI Level 4 in 2004.

SPI carefully analyzed and it got results from the industry it seems to be the
ultimate approach but it is not compulsory, that it will help other organization to reach its
improvement goals. There was not enough examples provided and every organization has
its own culture, staff, etc., It cannot be guaranteed that what SPI did for one organization
could applicable for others also. The current process should be assessed before making an
SPI attempt. It serves as a base and guide.

The ISO/IEC 9126 standard is one of the available models that can be used to
assess a software process or product. Using the ISO/IEC 9126 model it is possible to
evaluate the product or process from quality point of view.

Software product quality can be evaluated by measuring internal attributes

(typically static measures of intermediate products) or by measuring external attributes
(typically by measuring the behavior of the code when executed) or by measuring quality
in use attributes. The objective is for the product to have the required effect in a particular
context of use.

ISO/IEC 9126 is a software product evaluation model for high quality software
products and is used to evaluate every relevant quality by using validated and widely
accepted metrics. ISO/IEC 9126 categorizes software quality attributes into six

40

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 characteristics as functionality, reliability, usability, efficiency, maintainability and
portability, which are further sub-divided into sub-characteristics. The sub-
characteristics can be measured by internal or external metrics (Güceglioglu, 2006).

Process quality (the quality of any of the lifecycle processes) contributes to

improving product quality and product quality contributes to improving quality in use.
Therefore, assessing and improving a process is a means to improve product quality and
evaluating and improving product quality is one means of improving quality in use.
Similarly, evaluating quality in use can provide feedback to improve a product and
evaluating a product can provide feedback to improve a process.

An SPI approach for evaluating metrics based on static descriptions of software

development processes, rather than dynamic observation and evaluation of the processes
as applied, has recently been proposed by Güceglioglu, (2006). Using the analogy of
software product evaluation via product metrics such as cyclomatic complexity,
interoperability and testability etc., Güceglioglu, (2006) suggests that an organization can
benefit from product based models and also process quality based measurements for
selecting the most suitable alternative. The model proposed can also be used by itself in
the process improvement studies. By means of the model organizations can measure
impacts of the process improvement studies on their process quality (Güceglioglu, 2006).

This model includes quality metrics and its metrics can be defined and authorized
to develop new metrics if needed and it is not listed in the existing one. This metrics
helps to measure the quality of activities by which the quality of process can be predicted.
The user can detect the quality issue and take corrective measures during the early stages
of development.

The users can measure the extent to which the process meets quality
considerations (Güceglioglu, 2006). Güceglioglu’s approach brings predictability to some
extent and mention about adopting ISO 9126-3 to an organization’s needs which the
author also agrees.

The viability of SPI method is found by Seckin, (2006) though more

comprehensive improvement effort is needed to bring tangible benefits.

41

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.

 To have a high quality product, first and foremost the development process should
be defined, then it should tailor the process to itself with the industry standard and the
process should be assessed and measured. Because poor-quality work is not predictable,
quality is a prerequisite to predictability (Humphrey, 2005).

Only after making the assessment and analyses of the measurements, improving
the process can be started and it will create the desired outcome and the advantage of
existing guidelines like CMM or CMMI for SPI program can be used.

After going through the literature it is realized that the current trends in software
industry designate a movement towards utilization of SPC techniques. In CMM and even
more in CMMI, SPC is depicted as a challenge for high maturity organizations. They are
also the most detailed resources in terms of providing a guideline for SPC
implementation in a software organization. Apart from the software standards, the studies
of researchers in this area also constitute an important part of the literature. One
important common point among the researchers is that they all accept the difficulties of
using SPC techniques in software domain and a high majority of them regard SPC as a
beneficial tool for controlling processes. While supporting their ideas, they usually refer
to de-facto experiences in manufacturing industries.

2.4 Chapter Summary

This chapter highlighted the findings from the literature review conducted as part
of the first stage of the research. The finding supported the view of the global initiatives
on the importance of software quality and applicability of SPC in software industries to
achieve high level process maturity. The following chapter will further discuss in details
the suitable methodology options to investigate the key research objectives of this
research.

42

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.