Software Qual J (2006) 14: 135–157

DOI 10.1007/s11219-006-7599-x

Utilization of statistical process control (SPC) in
emergent software organizations: pitfalls and suggestions

K. U. Sargut · O. Demirörs 

C Springer Science + Business Media, Inc. 2006

Abstract Common wisdom in the domain of software engineering tells us that companies
should be mature enough to apply Statistical Process Control (SPC) techniques. Since reach-
ing high maturity levels (in CMM or similar models such as ISO 15504) usually takes 5–10
years, should software companies wait years to utilize Statistical Process Control techniques?
To answer this question, we performed a case study of the application of SPC techniques
using existing measurement data in an emergent software organization. Specifically, defect
density, rework percentage and inspection performance metrics are analyzed. This paper
provides a practical insight on the usability of SPC for the selected metrics in the specific
processes and describes our observations on the difficulties and the benefits of applying SPC
to an emergent software organization.

Keywords Statistical process control · Control chart · Defect density · Rework percentage ·
Inspection performance

1. Introduction

SPC (Statistical Process Control) has been widely used in manufacturing industries to control
and improve processes (Sutherland et al., 1992). While the benefits of SPC are well founded
for manufacturing companies, there have been many debates (Card, 1994; Kan, 1995; Lantzy,
1992) about its application in the software industry. Some of the inherent characteristics of
software such as invisibility and complexity (Brooks, 1987) usually entail subjective judg-
ment in collecting and interpreting process and product measures, even when these measures

K. U. Sargut ()
Computer & Information Sciences & Engineering Department,
University of Florida, Gainesville, FL, US

O. Demirörs
Informatics Institute, Middle East Technical University, Ankara, Turkey

136 Software Qual J (2006) 14: 135–157

are well defined. This causes variation of data and makes it difficult in applying SPC tech-
niques in software. Nevertheless, the interest to apply SPC techniques in the software industry
has grown as more organizations improve their processes by utilizing models such as Capa-
bility Maturity Model (CMM) (Paulk et al., 1993), Capability Maturity Model Integration
(CMMI) (CMMI product team, 2001) and SPICE (ISO/IEC 15504-4, 1998). Process im-
provement models implicitly direct software companies to implement SPC as a crucial step
for project level process control and organizational level process improvement purposes. In
CMM, Quantitative Process Management key process area at level 4 requires establishing
goals for the performance of the project’s defined software process, taking measurements of
the process performance, analyzing these measurements, and making adjustments to maintain
process performance within acceptable limits. Similarly in CMMI, Organizational Process
Performance and Quantitative Project Management process areas require establishing process
performance baselines for the organization’s standard software processes and quantitatively
managing the project’s defined process to achieve the project’s established quality and pro-
cess performance objectives. The SPICE Measurement attribute requires a software firm
to analyze trends in process performance and maintain process capability within defined
limits across the organization. Likewise, a level 4 process attribute called Process Control
necessitates control of process performance to maintain control and implement improvement.
Although these models direct software organizations to apply SPC techniques, the exis-
tence of SPC related requirements in high maturity levels (level 4 and above) and the common
sense belief that the SPC can only be performed after achieving level 4 almost prohibits or-
ganizations implementing SPC techniques earlier. The critical issues for successful SPC
implementation are process stability, measurement capability and data reliability. In other
words, if the process is performed consistently, the right metrics are selected and a reliable
data collection mechanism is established, it may be possible to benefit from implementing
SPC techniques.
In this study we focused on the question “Can emergent software organizations—
organizations which are small to medium size and have CMM maturity levels three and
below—utilize SPC techniques and benefit from the results?” To answer this question we
performed a case study in an emergent organization (Sargut, 2003). During the case we col-
lected the existing metric data related to company-specific procedures. By utilizing control
charts, we observed practical evidence of the benefits and difficulties of applying SPC. In
this paper we demonstrate the findings of this case study.
In the second section, we review the studies related to the utilization of SPC in the software
industry. In the third section, we describe major problems in implementing SPC and provide
possible solution methods related to the specific metrics that we analyzed during the case
study. In the fourth section, we give details of the case study and demonstrate our practices
on defect density, rework percentage and review performance metrics. In the fifth and sixth
sections, we discuss our findings and present our conclusions.

2. Utilization of SPC in software development

The utilization of SPC in software development has been studied by a number of researchers.
Some of these studies are based on process improvement models and focus on SPC as
a way of achieving high process maturities (Burr and Owen, 1996; Florac and Carleton,
1999; Carleton and Florac, September 1999; Florac et al., 2000; Florac et al., 1997; Jakolte
and Saxena, 2002; Humphrey, 1989). These model-based studies mostly represent CMM
understanding of SPC for software process improvement. One of these studies belongs to

Final product quality can only be measured at the end of a project as opposed to the manufacturing industry. Burr and Owen provide such a guideline by describing the statistical techniques currently available for managing and controlling the quality of software during specification. SPC was one of the important themes in the 2001 High Maturity Workshop (Paulk and Chrissis. he brings a relaxed understanding by stating that the processes can be regarded in control when the project meets in-process targets and achieves end-product quality goals. r The processes should be capable of producing the desired software product. Florac and Carleton. Florac and Carleton describe the utilization of SPC in the context of CMM for software process improvement (Florac and Carleton 1999). They provide detailed technical information about SPC and a roadmap of SPC implementation. 1996). r SPC should be applied only to the critical processes. Radice describes SPC techniques constrained within the software domain and gives a detailed tutorial by supporting his theoretical knowledge with practical experiences Springer . He mentions that the use of control charts can be helpful for a software organization especially as a supplementary tool to quality engineering models such as defect models and reliability models. 1999). He also underlines the necessity of maturity for achieving process stability in software development. production and maintenance (Burr and Owen. This study reveals some important points for the application of SPC to software processes: r Metrics should correlate to the quality characteristics of the products that are defined by the customer. With all these various dimensions. design. outlines the actions to provide higher maturity levels and acts as a basic guide to improve processes in a software organization (Humphrey. it is not possible to provide control as in manufacturing since the parameters being charted are usually in-process measures that do not represent final product quality. 1992). 1989). It emphasizes the difficulty in achieving process capability in the software domain and is cautious about SPC implementation.Software Qual J (2006) 14: 135–157 137 Humphrey. They also discuss issues related to the application of control charts in software development and incorporate the experience gained from the manufacturing industries to the software processes. They mostly focus on the principles and the methods for evaluating and controlling the process performance depending on Shewhart’s principles (Shewhart. so that on-time control of processes becomes impossible. One of the workshops in this CMM-based study demonstrated that level three measures are not appropriate for implementing SPC. Lantzy presents one of the earliest studies on the debate of applying SPC to software pro- cesses (Lantzy. 1995). He outlines a seven-step guideline for successful SPC implementation in a software organization. It also showed that the processes for establishing the process performance baselines should be maintained at low levels so that the organization-wide baselines can be useful in different projects. They focus on control charts as beneficial SPC tools and provide guidelines for measurement. process improvement and process management within software domain. Kan provides a comprehensive study of the theory and application of metrics in software development (Kan. Some of the studies act as guidelines for using SPC by converting general SPC principles specifically to software development (Burr and Owen. However. 1939). 1996. the study can be regarded as a CMM-based SPC guideline for software development. SPC appears as a means of data analysis for level 4 organizations. 2002). who describes a framework for software process management. Thirty five high maturity (CMM level 4 and 5) organizations took part in the workshop that was performed to investigate the practices characterizing CMM for Software level 4 and 5 organizations. r Metrics should be selected for the activities that produce tangible items. Finally. In his work.

1999). She finally portrays the results of the SPC study in the company showing that the smaller programs are better able to perform SPC. addressing the issues of data homogeneity and rational sub-grouping. (Meade.. lack of enough data. He states five problems with control charts: too much variation. He uses XmR charts for the lines of code inspected per hour for each inspection and achieves a stable inspection process after removing the outliers from the dataset. He also explains the relevance of SPC for CMM level 4 and regards backing-off of control charts in level 4 as a mistake. 1999). She also mentions the necessity of embracing the value of metrics. the study is far from being practical as it includes too many parameters and assumptions. By using these findings.. lack of specification limits from the clients. The usability of SPC in software development was also discussed in a panel named “Can Statistical Process Control be Usefully Applied to Software” in the European SEPG conference in 1999.138 Software Qual J (2006) 14: 135–157 (Radice. Card discusses the utilization of SPC for software by also considering some of the ob- jections and mentioning possible implementation problems (Card. Keller mentions the importance of SPC on management decision making and forecasting (Keller. Hirsh states that SPC charts should be used to gather desired benefits and the managers should be trained to ensure the use of SPC (Hirsch. unnecessary use of control charts. and the idea that control charts cannot be used with software processes. 2002.. Then they perform an SPC study on the package review data by implementing each of these factors. having a defined and repeatable metric program and having curiosity about metrics for usefully applying SPC to software. using the correct con- trol charts. 2000. Meade demonstrates a synopsis from the SPC utilization as part of level 4 implementation in the Lockheed Martin Corp. Florac et al. Jakolte and Saxena move ahead on the idea of 3 sigma control limits and propose a model for the calculation of control limits to minimize the cost of type 1 and type 2 errors (Jakolte and Saxena. 2000). 1994. Then he draws the u-chart for the defect density data for each inspection. 1994). 1999) referencing their experiences on the Space Shuttle Onboard Project (Florac et al. providing operational definitions. understanding multiple-cause and mixed-cause systems. 2002). However. She specifies the importance of understanding the data and reveals that it is not possible to apply SPC to all software metrics. He states that all SPC techniques may not be applicable for software pro- cesses and gives XmR (X and moving range) and u-charts as possible techniques. Jakolte and Saxena. Florac et al. having a defined and repeatable process. Finally they summarize the benefits of applying SPC to software processes. This is a pioneering study as it questions an ac- cepted practice for control charts and the results of the sample studies are encouraging. He gives a control chart example for testing the effectiveness measure and concludes that SPC principles can be beneficial for a software organization although formal statistical control techniques may not be used. he makes reliable estimations for inspection effectiveness and gains an insight on when to stop testing. He also outlines the observations and results for a specific SPC implementation. finding and testing the trial limits and recalculating the limits. 1998). 2000). 2000). The literature also includes studies that present practical experiences with SPC in the software domain (Card. 2000). Weller. 1999). Barnard and Carleton emphasize the mixed behavior of processes (Barnard and Carleton. present their analysis findings from a collaborative effort between SEI (Soft- ware Engineering Institute) and the Space Shuttle Onboard Project (Florac et al. They first emphasize the importance of selecting key processes. Wigle questions SPC implementation Springer . Weller provides one of the rare practical examples by presenting the SPC implementation details from a software organization (Weller.

r The processes should be well-defined and stable so that we can apply SPC techniques successfully. data analysis guidelines. 3. how efficient our inspection processes are. but without practical evidence. Then we carried out theoretical research to establish a firm knowledge on these measures. 1999). the existing practical evidence represents only the state of high level organizations. Heijstek shares his experiences on statistical studies in Ericsson and shows the lack of data quality as the main analysis problem (Heijstek.Software Qual J (2006) 14: 135–157 139 in the software organizations and stresses the importance of applying SPC at a level where decision making occurs (Wigle. As referred by Paulk and Carleton (Paulk and Carleton. we decided on the metrics for which we were going to make an SPC analysis. and which components of the system are error prone. but it is advisable to have the basic measurement practices before using SPC (Carleton and Flo- rac. Therefore. On the other hand. Moreover. defect density and review performance metrics after discussions with the software engineering process group. Selected metrics At the beginning of the case study. We outlined the general characteristics of each measure including its definition. r SPC should only be applied to the critical processes in a software organization. Springer . Nevertheless. most of these studies contain very lit- tle information about implementation details. Radice supports utilization of SPC at lower levels once the process is executed consistently within the definitions and sufficient data are provided (Radice. how much rework will be performed. and examples are usually restricted to de- fect density and inspection effectiveness measurements. September-1999). Although the current resources do not provide practical ev- idence for our case study. we established a good understanding of SPC in a software organization by reviewing the available literature. difficulties in the measurement processes. but also on the quality of the related processes. which processes need im- provement. importance on software development.1. r Not all SPC techniques are applicable to software processes. In this regard. Florac and Carleton state that it is possible to benefit from SPC at low levels. our survey revealed that: r SPC might not be applicable to all software processes. We observe that the difficulties of using SPC techniques in the software domain are frequently discussed by researchers. and possible uses of the control charts. 1999). various implications of metric information. We first analyzed the company status and selected rework effort. Card emphasizes the importance of process stability and delineates lack of well-defined business objectives to guide collection and analysis as the major problem. Defect density The number of defects in a work product is an important measure as it provides intuition about how well the customer will be satisfied (from post-release defects). 3. 1999). 1998). SPC utilization in emer- gent organizations is discussed by very few researchers. collection principles. defect counts provide evidence not only on the quality of the product.

Florac and Carleton. As we desire to find most of the existing defects in a successful inspection. the u-chart is suggested for tracking the defect density data. 1995). The resulting metric is defect density. the width of control limits for the u-chart is inversely proportional to the square root of the sample size (product size). it is very important to check for the validity of this assumption by analyzing metric data. nor so low that the results become meaningless. or a very successful inspection process. we established the necessary background to provide consistency among measurements. having 10 defects in a sample of size 20 is not equivalent to having 100 defects in a sample of size 200.2. Therefore. r The charting technique should be determined very carefully. Rework percentage Rework is the total hours spent that was caused by unplanned changes or errors (Pajerski and Sova. on average. Radice. we have a certain expectancy of defect count for unit size of a software artifact. (IEEE Std 1044-1993. On the other hand. The level of detail should be neither so high that the analysis becomes too difficult.1-1995)). Observations that exceed the upper limit may indicate a highly defective component. Weller. There- fore. 1999. which is described as Springer . The amount of rework is a good indicator of the quality of processes as it shows the magnitude of effort spent due to previous errors. 1992. 2000). The determination of causes requires a detailed root-cause analysis. 3. r The defect data should be categorized into specific groups (in terms of defect type. the u-chart depends on the assumption that the defect data follows a Poisson distribution. it is possible to produce a product without any rework by doing the processes right the first time (Crosby. Thus. 1998. it is usually more appropriate to use XmR charts in the other cases.140 Software Qual J (2006) 14: 135–157 In order to provide a basis for comparison between different software products. a low defect density measure may be due to a high quality product with very few defects. we can comment on the reasons for unusual behaviors. which is not reasonable for code and document defects. this measure is also an indicator for the effectiveness of the related inspection processes. Likewise. Although the chart may be useful for code defects if SLOC (software lines of code) is counted for software components with similar sizes. or a poor inspection process. which should be carried out apart from the SPC analysis. Before starting to measure defect density. 1980). defect data are normalized with the addition of size dimension to the analysis. IEEE Std 1044. Once the chart (u-chart or XmR chart) is drawn. r The defect origins (cause codes) should be recorded for each defect. However. We found the following issues to be important concerns for a successful defect density analysis: r A precise definition should be documented for each type of defect. In most of the resources (Sutherland. which is defined as: number of defects defect density = product size The analysis and interpretation of the defect density metric lies on the assumption that. Rework increases project costs without adding any value to the product. r The size measures should be precisely defined and measured for different work products separately. r The critical products for SPC analysis should be determined. priority. severity etc.

For instance. However.” Even though eliminating all the rework is not quite possible in real life. or a very long project phase (We know that most of the rework is done towards the end of project phases after inspections. configuration management etc. On the other hand. If a project phase lasts too long and the inspections have not been performed yet within the measured period. may be utilized for the related process: rework effort Rework Percentage = total effort Rework percentage provides an understanding of the relative amount of rework with respect to the total effort. superior process performance. The analysis can be performed by comparing the rework percentages among different projects/project components at the end of each project phase. XmR charts can be drawn for SPC analysis. r Metric data. such an analysis necessitates the determination of defect causes. more em- phasis should be given to the ongoing inspections as an aid to avoid future deficiencies. r The enhancement effort should not be regarded and measured as part of the rework. the quality of activities in a certain project phase or the processes during the preparation of a work product can all be examined by looking at rework percentage values. Springer .) comprising particular tasks. However.Software Qual J (2006) 14: 135–157 141 “Cost of Quality. the performance in a software domain (i. poor performance of processes. r For each rework process instance. However. Obviously. an increasing trend in the rework percentage may be due to problems in the earlier project phases or the previous inspection processes. this theoretical limit provides a threshold for assessing the processes.e. It is also possible to analyze the rework from a different perspective by finding the amount of rework related to a certain process or project phase. this analysis should be based on the assumption that the expected rework percentage amount is the same in the different periods within which the measurement is performed. In this analysis. A decreasing trend may be the reason for bad performance in current inspections. In order to provide a baseline to make a comparison among rework efforts of different process instances. quality assurance.e. such as rework percentage. as well as the rework data. As soon as the mentioned issues are resolved and the relevant measurement data are collected. high inspection performance or a long period of testing and inspection processes. projects. monthly) ba- sis independent of the project phases. the rework will accordingly be low). a successful and meaningful analysis requires the consideration of some preliminary issues: r Rework should be defined precisely. the charts will indicate the performance of a project component at a particular project phase. Thus. The points under the lower limit might be due to low inspection performance. it enables us to analyze any group of tasks representing a specific domain. or might be indicative of high product quality. Thus. the values exceeding the upper limit may indicate deficiencies in project planning. should not be used besides its intended goal like for judging the individual performance. In this case. and/or builds. the analysis may be carried out separately for different CSCIs (Computer Software Configuration Item). Depending on the data on hand. the rework percentage and the defect den- sity charts should be analyzed concurrently so that more reliable interpretations will be possible. a normalized measure. Alternatively the rework percentage may be calculated on a periodic (i. the related process/project phase that causes the rework should be defined.

the type of defects. final reviews. the reviewers. design documents. We performed the case study in an organization which was established in 1998 Springer . it is best to perform separate analyses for different products or product groups (e. requirements doc- uments. On the other hand. It is also possible that the product actually contains very few defects. the points below the lower control limit may indicate low inspection process performance.3.142 Software Qual J (2006) 14: 135–157 3. change reviews. Utilization of SPC The goal of the case study was to investigate the applicability of SPC by using an emergent company’s existing measurement data and to identify the difficulties and the benefits of the technique. In either case. the number of defects found by each reviewer. when there are many defects left undetected in the product. r The inspection type. Thus. joint reviews). 4. The importance of inspection on the product quality and the project costs necessitates tracking the performance of the inspection process in a software company. and the related trouble report should be recorded for each inspection. the points above the upper control limit show the instances in which the number of defects found per unit effort exceeds the process perfor- mance limits. One of the useful measures reflecting the inspection process effectiveness is the inspection performance: number of defects found Inspection Performance = inspection effort The interpretation of the data depends on the idea that the number of defects found during an inspection is directly proportional to the inspection effort. and to complete a formal task (Humphrey. SPC analysis can be carried out by drawing XmR charts for the individual inspection effectiveness measures. the inspection date. Humphrey (Humphrey. the inspected artefact. As soon as the above issues are considered and the required metric data are collected. Inspection performance Inspection is the formal review of a product to identify defects. r An effective defect measurement mechanism should be constructed (see section 1. some pre- liminary issues should be considered. It is also possible to use u-charts to track effort in terms of the minutes spent on the inspection. This may be because of high inspection effectiveness or low product quality. However. As the inspection outcomes depend on the product. In order to have a correct interpretation of the inspection performance metric. 1989). In the XmR chart.g.). to verify the product acceptance criteria. code etc. the defect density measure may be used in parallel to reach the correct understanding while interpreting the results.1) r The inspections should be categorized considering their points in the life cycle and their purposes (such as draft reviews. 1989) shows that inspection is a powerful tool for improving the product quality and for reducing rework amounts. the inspection effort (including assessment and meeting times). the applicability and utility of such a sensitive analysis should be in- vestigated by using the existing metric data. It enables software specialists to figure out and correct errors before the product is released. It is also used to provide an agreement among the appropriate parties. we expect to find more errors if the inspection lasts longer. We left implementation of u-charts on inspection performance data as a future study item.

Then we mapped the collected metrics and the selected processes. we presume that the measurement processes and metric data are precise enough to enable SPC analysis. The raw data are presented by means of bar and Pareto charts. IEEE Std 828-1998 and IEEE Std 830-1998). we identified the most significant issues for process improvement in the company. Organizational level metric data are analyzed in a similar way by the quality assurance department. the data would be multiplied by a constant factor. Finally. defects. and the company level met- ric data are collected by the assigned individuals from the relevant departments. Similarly. A statement of work is prepared and signed by both parties. requirement stability and SLOC. and the quality assurance is provided extensively via periodic and milestone based audits. which are analyzed by the project managers and the relevant technical staff in the project. Therefore. The project level metric data are periodically collected by the assigned project individuals. Moreover. As the limits of a u-chart are dependent on the size measure. we had the relevant data to derive them. the company would be given a copy of all research documentation including process descriptions and analysis results. Before starting to work.Software Qual J (2006) 14: 135–157 143 and is certified CMM level three in August 2002. By using such a bottom- up approach. the actual data would not be presented on the charts. a similar change would cause errors while identifying the outliers. an agreement is made with the company to outline the boundaries of our study. the existing process descriptions are mostly based on the definitions from the CMM and various IEEE standards (including IEEE Std 1044-1993. the results are discussed in the periodic project meetings. Although some of these measures were not directly collected. Similarly. After a more detailed analysis. we decided to disregard the Test Performance metric as we could not gather the effort data for different tests separately. the same trend would be observed and the same outliers would be detected after this modification despite the changes in the mean and the variance values. As a result of this mapping. At the end of the study. All the metrics used in the case study have been previously collected in the projects (before and after achieving CMM level three). but the numbers on the y-axes would be hidden. The analysis is performed by detecting any irreg- ular behavior through subjective judgment. Then we prioritized the remaining metrics with respect to their significances and started to work on the high priority metrics (metrics 1 through 6 in the Table 1). the selected metrics had been collected for years. the data represents the company status while the processes were in transition for CMM level three. we selected the metrics among the ones that were already being collected in the company. we prepared a list of metrics collected in the company and worked on each measure considering its meaningfulness and feasibility in terms of performing SPC analysis. In the organization. we constructed a list of measures as the initial candidates for our analysis (see Table 1). Standard metric datasheets are used to collect more than 20 metrics in- cluding effort. and the results are discussed in the periodic managerial level meetings. this study would serve as an appealing experience for the company for future improvement programs to achieve higher maturity levels. For this reason actual data would be used to draw u-charts. we decided to exclude Requirements Stability and Productivity metrics since the number of data points was not sufficient for Springer . Therefore. All the processes are standardized by the documented procedures. IEEE Std 1044.1-1995. However. Moreover. On the other hand. It is decided that the name of the company would not be mentioned in any part of the study considering the confidentiality of the utilized data. Based on this foundation we started our study in two dimensions. On the one hand. After an initial evaluation. Therefore.1- 1995. IEEE Std 730. we decided to eliminate some of the candidate measures which were not deemed critical enough to necessitate a thorough SPC analysis. Before drawing Individuals (XmR) charts. rework effort. However. a proposal is prepared to document the general objectives of the study. IEEE Std 730-1998. change can be regarded natural since continuous improvement is already part of being a high level organization.

application domains (management information systems. embedded systems. By working on the metric data. Now we will go over the results of the case study for each metric in detail. Springer . the data of all defects found during a review. the basic defect information such as the subject work product. It was not appropriate to directly use the collected raw metric data for our SPC study.144 Software Qual J (2006) 14: 135–157 Table 1 Candidate Measures No Measure Description 1 Rework percentage Percentage of rework effort to the total effort 2 Review performance How effective is the review process (in terms of the defects found per review time) 3 Defect density Relative number of defects in a product 4 Productivity Production amount (product size) per effort 5 Test performance Number of defects found per testing effort 6 Requirements stability Percentage of added. We then collected actual metric data from seven projects with various characteristics regarding their sponsors (inter- nal or external). 4. However. test or audit have been col- lected and tracked through Problem Reports (PR—for code defects) and Document Change Requests (DCR—for document defects) since the foundation of the company. market places (commercial or military) and some others. Our study revealed various difficulties of using metric data that were not specifically defined for SPC analysis. Defect density In the company. In the end. Although the trouble reports evolved in time. we provided practical evidence on the utilization of Statistical Process Control in software domain. Moreover. For each selected metric. we had to make some derivations by using some additional data from different sources. command and control systems). we re- stricted our analysis only to the tests of upper and lower control limits instead of investigating the trends and other tests for detecting the outliers. our scope has been drawn by Rework Percentage. After organizing the data in the relevant format. utilized the relevant SPC techniques and interpreted the analysis results. Therefore. deleted and changed requirements 7 Customer support time Time passed from receiving an accepted customer problem until corrected code is deployed to the target environment 8 Backlog management index (BMI) Trouble report creation rate relative to the closure rate 9 Audit effort percentage The amount of audit effort relative to the total effort 10 Audit performance The number of nonconformities detected during audits relative to the total effort 11 Peer review effort index The amount of the peer review effort relative to the work product size 12 Total defect containment Number of pre-release defects relative to all defects found effectiveness (TDCE) in a product 13 Trouble report aging Time elapsed between the initiation and the resolution of a trouble report a thorough analysis.1. Defect Density and Review Performance metrics. carried out the relevant normalization procedures. we first performed a detailed analysis to establish a good un- derstanding of the metric basics and the data analysis mechanisms. detected outliers and investigated them to understand the variation. we determined the company-specific parameters. we drew Individuals and u-charts.

the defect density value gradually increases as more defects are detected. and defined size measures as the following: 1. the size of the artifact that is active at the initiation date of the latest DCR written for the related phase is used. Now the initial defect density count is updated and the updated information is used for the analysis. we decided to perform separate analyses by comparing the defect density values at the end of each project phase for the two document groups. We observed that the system level IRS of Project 4 (point 4) is very close to the upper control limit on the chart. and the size of the last version which is already released at the end of the project phase is used for the defect density measurement. For this reason. the division of data among different priorities. We also realized that different builds of a CSCI have the same documentation.Software Qual J (2006) 14: 135–157 145 the related project phase. For instance assume that 30 defects are found for 30 requirements (size of an SRS) at the end of requirements analysis phase of build 1. For instance. Unfortunately. non-blank Source Lines of Code. we realized that there was not enough data from each priority category for a successful SPC analysis. 3 requirements have been deleted and 1 added). In such cases.e. the defect priority. Design Documents (Software Design Description (SDD) and Interface Design Description (IDD)): The number of pages is used to compute size. in some of the projects the components are further divided into builds and different phases occur at the same time because of evolutionary design approaches. Consequently. an SRS document prepared for build 1 is updated for build 2 with required changes. Let 17 more defects be found since the end of build 2 and the size become 28 (i. project phases and document groups highly reduced the affectivity of the SPC analysis as the number of samples became insufficient. medium. we decided to restrict our analysis to the requirements and the design documents. we restricted our analysis to the implementation and the maintenance phases. after collecting the data. however. with different versions. The code size is collected for each CSCI in terms of non-comment. The Individuals chart for the group 2 defects at the end of implementation phase for the requirements documents is shown in Figure 1. As the size of a document remains almost the same throughout a software project. we decided to use the latest version by counting all the defects for the previous and the current versions. The document size is gathered for each version. After talking to the project manager and looking at the IRS document we understood that the IRS requirements included interface details regarding Springer . very high or other) is assigned for each defect on a trouble report. Thus we combined 5 priority categories within 3 groups: Group 1: high and very high Group 2: medium Group 3: low and other While making this categorization. On an ongoing project. The cumulative number of defects is computed for each document. However. However. An individual priority (low. 2. data from the seven projects were not sufficient to perform SPC analysis for code defects. these data are used to calculate the defect density amount for the requirements analysis phase. high. the initiation and closure dates are recorded for all projects. Therefore. Requirements documents (Software Requirements Specification (SRS) and Interface Re- quirements Specification (IRS)): The number of requirements is used to compute size. we made an assumption that we could pay similar attention to the defects in priorities low and other. We drew Individuals charts for each project phase-priority combination for the require- ments and the design documents after putting the data into time order. Therefore. However.

a separate analysis is performed by combining all the defects without considering their priorities. The relaxed behavior of the customer during the requirements analysis phase resulted in future updates in the requirements document.5 0.0 LB=0 1 3 5 7 9 11 13 15 17 Observation Fig. this may cause data insufficiency as the number of data points per group decreases.631 2.g. Although the variability can be reduced by further dividing the document groups (e. a very high variability is observed among defect density values of different documents. this incident indicates that the nature of some processes could make them very difficult to stabilize.634 0. the data shows a similar trend for the code defects.5 2. Assuming it as an outlier. Considering that the processes are defined and defects are correctly recorded. This finding showed that SPC would not be applied for all categorized metric data. and reduced the sensitivity of control charts. Moreover. we removed the point from the dataset and calculated the con- trol limits again. the project was open to any proposed changes being an internal research-and-development project. an abnormal situation has already been recognized for this project (section 4. Although the graphs are not depicted in this paper. apart from group 2 and group 3 priority analyses.0 UCL=2. A common observation for requirements and design documents was the scarcity of data points for the group 1 defects.0 _ X=0. Consequently. The control charts revealed some important concerns about defect density analysis. Thus a high defect rate is observed in the IRS document. The customer was a partner organization and the company was more tolerant of the changes and the additional requirements.2). The high variability also caused control limits to be too wide. which makes SPC more difficult to apply. Thus. During the rework percentage analysis. any change to one of the CSCIs affected the IRS document and caused updates. This time Project 5 SRS (point 3) appeared to be an out-of-limit instance (Figure 2). Moreover.0 Defect Density 1. 1 Defect density group 2 implementation (requirements documents) all the CSCIs. As it was not possible to draw separate control charts for the group 1 defects. Springer . separate analyses for SRS and IRS). there is a trade-off between the number of data points and the depth of analysis.146 Software Qual J (2006) 14: 135–157 Defect Density Group 2 Implementation 3.5 1. The detection of the same outlier by two independent control charts has been a good sign for the efficacy and the reliability of our analysis.

which corresponds to the rework effort in terms of person- hours. Problem Reports and Document Change Requests are used to document defects that are detected after an internal or formal baseline is established. Although it shows the variable behavior of the process. we prepared a list of lessons learned which provided a baseline for the future improvement studies in the organization.0 LB=0 1 3 5 7 9 11 13 15 Observation Fig.0 Defect Density 1.571 2. The u-chart showing priority group 2 defect density values for the design documents at the end of the implementation phase is given in Figure 3. rework is defined as any modification to configuration items after the final peer review of the first release and changes to the internal/external baselines. it becomes part of an external baseline. Rework percentage In the company. Springer . it is possible to calculate the total rework effort by summing up the efforts on trouble reports within the dates corresponding to the related 1A document/code becomes part of an internal baseline when it is put into configuration control. we also drew u-charts.1 Any change before the product is put into the configuration control is regarded as part of the development. Moreover. By working on the reasons for these outliers. the u-charts were over- sensitive for the defect data and captured too many data points outside the limits. the total problem resolution time. Such nonconformities are recorded on and tracked through Action Item Lists.Software Qual J (2006) 14: 135–157 147 Defect Density Group 2 Implementation (Updated) 2.0 _ 0. 4. the control charts gave us an understanding of the status of different products allowing us to make a comparison with respect to the control limits. any defect recorded on a PR or DCR form is believed to cause rework. If the document/code is also submitted to the customer.2.5 UCL=2.5 1. In order to attain a second dimension on the process.510 0. However. We also captured extreme points that demonstrated poor performances and poor product quality as a result. For this reason. On the other hand. it proves to be useless in the current conditions. 2 Defect density group 2 implementation updated (requirements documents) Nevertheless.5 X=0. Therefore. but not as rework. is recorded on these forms.

Nevertheless. As different defects on a form would have different origins.3 percent of the total effort is counted for week 2. the project phases are overlapping for the different CSCIs and the builds.6 percent for week 1 (4 days from Thursday to Sunday. CL: central line. A trouble report related to a document includes many defect items on the same form. we assumed that the effort amount is uniformly distributed among different work days. the exact rework percentages for different project phases cannot be derived. UCL: upper control limit. the defect fixing (including analysis. four consecutive weekly rework and total effort amounts are summed up separately and the rework percentage amount is calculated for each of these four-week periods. As the rework effort is calculated for more than three years in the company and the results are utilized for process improvement studies. the rework effort and the total effort data are divided into weekly units for the analysis. correction and relevant reviews) effort is recorded as one measure for all the defects. For this reason. However. 3 u-chart (defect density for design documents) (DD: defect density. we can assume that there is a firm understanding of the meaning of measurement. Therefore. where the weight is determined by the number of days that the trouble report stayed open in the corresponding week. The timesheet data are also collected daily for each individual and thus the effort amounts can be obtained at the project level.148 Software Qual J (2006) 14: 135–157 Fig. In order to smooth out the effect of this natural variation. 33. as the related project phase is not recorded on the trouble reports. we calculated a weighted effort for the weeks. we were unable to make an analysis on the defect origins. Thus. the study has been limited to measure the rework percentages within specific time intervals. As the existing data collection mechanism is not sufficient to obtain the CSCI and the build based effort amounts. 2 days from Monday to Tuesday). For the trouble reports that stayed open for more than one week (7 days). In the company. For instance if the initiation date of a trouble report is 22nd of May—Thursday and closure date is 27th of May—Tuesday. and 66. As the rework amounts increase during the inspection and the testing periods. LCL: lower control limit) project phases. we performed SPC analysis on a four-week period through which the rework percentage is assumed to be within certain limits. The analysis Springer . the variation with respect to the different points in the life cycle can be considered natural.

The project manager relates this occurrence to the nature of the project and to the behavior of the customer.2002 and 24.05.Software Qual J (2006) 14: 135–157 149 Project 5 Coding Rework 0. company-wide monthly rework percentage amounts are calculated combining data from all projects.07. Moreover. the occurrence of high documentation and coding rework is a natural result of the defects in the IRS document as mentioned in section 4. Points 11.4 1 0.1883 0.0 LB=0 1 3 5 7 9 11 13 15 17 Period Fig.6 1 0. In order to find out the organizational level control limits.0553 0. Springer . 4 Project 5 coding rework is performed separately for the documentation and the coding rework as they would possess different trends.2002 to 30. Moreover. In fact.2002 to 29. From our project-specific analysis we learned that the customer started to use the product actively after May 2002. point 12 makes a peak between the dates 27.08. the rework data for the periods before the implementation phase for the coding rework are disregarded although some coding has been done during throw-away prototyping in some of the projects.2002. The customer was a partner firm of the company and this increased its power to act on the requirements. The chart makes a peak at point 14 and then drops down to normal values. The Individuals chart for the documentation rework in the same project does not show any out-of-control situation (Figure 5). 12 and 14 demonstrate high rework amounts covering a period from 27.09.5 Re w or k Pe r ce nt a ge 0.1. The high rework trend after the 27th of May is a reason for the problems detected by the customer during product usage.05. The Individuals chart for Project 5 coding rework is shown in Figure 4. This flexible mood caused the customer to be less concerned about the stability of the requirements and the modifications became unavoidable after the product installation.3 0.2002. the programmers were not experienced with the programming language and this caused more rework than expected during the maintenance phase. Finally Individuals charts are drawn to analyze each project separately with the organizational upper and lower control limits.1 _ X=0. showing a parallelism to the coding rework in the same period.2 UCL=0. However.06.2002 and from 26.

08 0. Nevertheless.16 0. final and change. the product is released. Change peer review is performed to verify the appropriateness of the proposed changes to a product that is previously released. Therefore. Draft peer review is per- formed as a first check before the product is ready for final peer review.14 UCL=0. IDD and code are inspected during the peer reviews. SDD.02 0. 5 Project 5 documentation rework SPC analysis for the rework percentage demonstrated that control charts act as means of viewing process performances and detecting various abnormal situations in the projects.10 0. IRS.04 0.150 Software Qual J (2006) 14: 135–157 Project 5 Documentation Rework 0. the reviewers will most Springer . the inspection process is carried out by the independent reviewers and a summary report is prepared at the end of each review. it was usually difficult to comment on the reasons of the process deviations without having a chance to take on-time corrective actions. the product is expected to have fewer defects before entering the final peer review. 4.12 Rework Percentage 0. Being able to obtain this outcome despite the limitation of the case study is a promise for the value of SPC implementation in a software organization. the results demonstrate the success of control charts in detecting out-of-control situations that cause high rework in the projects. Final peer review is performed with the customer just before the release to verify the appropriateness of the product. There are three types of peer reviews: draft. their review and preparation times. As the study relied upon the historical data and past events. If no defects are found during this review.3. As some of the defects are fixed during a draft peer review.00 LB=0 1 3 5 7 9 11 13 15 17 19 Period Fig.06 _ X=0. Critical documents such as SRS. the number of problems found and the review results. Although the findings were not some hidden facts or surprising deductions. this study provided an opportunity to observe issues within the pro- cesses which lead the company to further improvement actions. Inspection performance In the company.1424 0. The summary report for a peer review includes details such as the names of the reviewers.0545 0. The aim of performing this early review is to minimize the number of the defects before the product is submitted to the customer.

This finding has been a good feedback for future studies. it was difficult to understand the causes of outliers for all cases. For this reason. a regression analysis is performed. the reviews for code and different document types are analyzed separately. This major change caused many updates in the requirements as well as in the design. SRS. As soon as the independence is provided. The chart for SDD draft peer reviews is shown in Figure 6. However. Therefore. there is not enough evidence to assume that the decrease in the number of defects found is proportional to the decrease in the effort. Moreover. the design is found to be inferior than the other CSCIs’ and many changes are regarded significant defect removals. A number of causes are proposed for this trend and one of the most important causes is believed to be the high staff turnover rate. change peer review is left aside in our analysis. different trouble reports are used to record the defects that are found during draft and final peer reviews. Finally. UITD (unit integration test de- scription). the reviews could not be investigated just after their occasions. Point 20. Although the effort will also be low. any available and skilled personnel may be included in the review team. On the other hand. most of the reviews were performed by only one reviewer and the remaining data could not capture any relationship. First of all. Thus we worked on the individual review data within the whole project. which represents the SDD document of CSCI 3 in Project 1. As the aim of using this measure is not to evaluate the product but the review process. and the review effectiveness will seem to be worse. The investigation of the corresponding review summary report and the DCR form depicted some critical issues about this CSCI. As the structures and the review orientations are different for a code component and a document. two CSCIs were combined into one within CSCI 3 a few months ago. As the defects found during a draft review are not categorized with respect to their priorities. it may be natural to observe different distributions for review effectiveness for these products.Software Qual J (2006) 14: 135–157 151 probably find fewer defects during a final in contrast to a draft peer review. Springer . the reviewers are inclined to focus on the changed sections rather than the whole product when the review is performed to verify the changes. In order to investigate whether the metric is statistically related to the number of reviewers. is out of the upper limit. Finally the Individuals charts are drawn for SDD. Moreover. SPC application to peer review performance measure can be regarded beneficial as the charts enabled us to notice outliers that represent special causes. it will not be rational to judge a change peer review as ineffective although no defect has been found. For this reason. the number of defects found during the draft peer reviews turned out to be more than the ones detected during the final peer reviews. In parallel to our assumption. the review data are put into time order for each review type-product combination. Nevertheless. the analysis is separated for the draft and the final peer reviews. Similarly. this update caused the CSCI to be quite large and this caused critical modifications during the implementation of the subsequent builds. The reviews are performed by independent individuals within a project. The charts demonstrated that the draft and the final peer reviews have different trends and behaviors in the organization. The defect density chart also shows a similar peak for this CSCI and supports our deduction (point 14 in Figure 7). the reviewers will not find even a single defect in most of the change peer reviews. As observed in the figure. Then the review effectiveness values are calculated by dividing the number of defects by review time (minutes) for each review. This structure makes it irrelevant to restrict the analysis to the CSCI level since the performance of a specific review will not be indicative of the CSCI for which the review is performed. As the data was historical. UTD (unit test description) documents and for code. In the light of these decisions.

152 Software Qual J (2006) 14: 135–157 SDD Draft Peer Review 0.00005 0. 7 Defect density group 2 implementation (design documents) Springer .0000648 0.93 0 LB=0 1 3 5 7 9 11 13 15 17 Observation Fig.81 8 Individual Value 6 4 _ 2 X=1. 6 SDD draft peer review individuals chart Defect Density Group 2 Implementation 10 1 1 UCL=8.0001613 0.00010 _ X=0.00025 1 0.00015 0.00020 Peer Review Performance UCL=0.00000 LB=0 1 3 5 7 9 11 13 15 17 19 Observation Number Fig.

Therefore. Considering the fact that the company already collected and analyzed the metric data. and peer review are refined. Most significantly. Nevertheless. collection and interpretation. planning the improvement program. As a result metrics were frequently used to validate decisions rather than to determine them. having well-defined process descriptions and a collection of metric data was not sufficient to apply SPC techniques suc- cessfully. the rest of the work was to draw control charts and analyze the results. Similarly. we needed some other measures to normalize the metric data.Software Qual J (2006) 14: 135–157 153 5. problem resolution. it is not possible to calculate the control limits without the data. the existing shortcomings are eliminated and the data became more meaningful for interpretation. For this reason. Although the industrial averages for different metrics are available. A considerable amount of effort is spent on constructing a company specific foundation before actually starting to use SPC. and normalizing the metric data. processes like measurement. the additional effort of implementing SPC. We also observed that each metric has particular characteristics and complexities related to its definition. Previously. identifying the processes that need close control and selecting the right metrics for our analysis. One of our initial observations has been the importance of pre- implementation activities. Although the processes Springer . Not all the data were stored in a computerized system. organizing the metric database to determine the initial process performance baselines. Even if the data are available. and visualized by bar/Pareto charts. Moreover. the quantity of the metric data becomes one of the major problems for the small-sized organizations and can limit the utilization of SPC techniques to a few metrics. We saw that the existence of the process documentation and the periodic status reports eased our work in di- agnosing the company status. The interpretations were based on subjective criteria of the managers who determined their actions based on their intuition. Most of these difficulties arise due to the unsuitability of the metric definitions for detailed statistical analysis. Therefore. For this reason. After SPC implementation. During the case study we also achieved benefits of SPC implementation. the process capability values of the other organizations cannot construct a baseline for a specific organization as it depends on various factors such as the procedures. the data was put in a tabular form. we had to reconfigure the database and manually collect some information from the datasheets. we also had a convenient database. The measurement data hides some inherent characteristics of the processes which are unique for each software organization. we had to put more effort on collecting those measures. metrics became more supportive and assistive for decision making. As the metric data had been collected and stored for more than 4 years. had been insignificant. The difficulty increases further when the existing metric data are used for SPC implementation. This study showed that the quantity (the number of data items) and the quality (how precise and detailed) of the metric data are very critical for the reliability of control limits. and some of the related data items were stored in independent sheets. it enabled us to detect more specific occurrences of outliers together with the use of control limits. the existing metric data are reconfigured. apart from the initial work. the preliminary study helped to reveal problems in the related processes and the metrics. Moreover. That is. the metric definitions and the analysis approaches. the definition of the metrics should have considered the intent to apply SPC techniques and the number of projects these metrics are applied should be large enough. Discussion This case study gave us insight on the difficulties and effort required to implement SPC in a software organization. Once this preliminary study is finished and a foundation is established. project management. The basic work items in this phase include diagnosing the current state of the organization. selecting the metrics. defining the new processes.

After this case study. insufficient detail and amount of data cause additional work to provide required settings. the data turns out to be insufficient for the analysis. but also improves all related processes. The study reveals many preliminary issues that should be resolved before actually utilizing SPC. inappropriate organization of data. The level of detail for some metrics was not sufficient to classify the data as expected and perform an in-depth analysis. and track of process improvement actions. the benefits are better process control. Finally. so we had to manually classify and reorganize the data. more effective detection of outliers. If the analysis is performed for specific processes. 6. normalize and analyze the data. the cost of using SPC can be regarded negligible after this initial effort. and the criticality of the process. using well-defined measurement processes are positive aspects that reduce the amount of work during this period. Another finding of this study is that it might be difficult to construct a general logic that will be used as a standard guideline for implementing SPC for software measures.154 Software Qual J (2006) 14: 135–157 are stable and various metrics are collected. For this reason. We considered some data that would have been useful for SPC analysis. Thus. The only additional effort is for drawing and analyzing the control charts. During the study. it is very important to make a tradeoff analysis before starting an improvement program. In contrast. the company management decided to standardize SPC analysis for technical and managerial decision making. Moreover. we came up against various difficulties while trying to use the existing metric data. The data are collected in various time intervals. separate approaches might be required for each metric to be analyzed. It shows how important it is to define the metrics and the measurement processes considering future needs. Thus. Our case study has been a good reference for showing the required effort and short term benefits of SPC in an emergent software organization. However. the level of detail should be determined very carefully based on the amount and the quality of the metric data on hand. the payoff is high especially in the long term. the data set may not be meaningful since a small sample size is used. the variability increases and it becomes difficult to detect the outliers. preventive action and project management processes are updated Springer . inherent characteristics and normalization techniques. we observed that the complexity of the processes and the existence of the large number of external parameters for the value of the metric data inflate variability and make it difficult to apply SPC to the software production. Therefore. problem resolution. the lack of some data items. maintaining a large metric database. but they were not collected as we assumed. Thus. this effort not only provides SPC utilization. the lack of awareness about the specific attributes of the metrics that are vital for detailed comparisons and statistical analyses creates difficulties for SPC utilization. analyses are performed in different points in the life cycle and different charts should be used for depiction. if SPC is applied to generic process data. Conclusions Process improvement initiatives usually face resistance until the management is convinced that the projected benefits are beyond possible costs. it takes more time to collect. On the other hand. Each metric has its own dynamics. Having documented procedures. Accordingly. In effect. we can conclude that the level of process maturity is an influencing factor on the amount of the effort spent before using SPC. The collection periods of some data were different from our needs. On the other hand. process definitions for measure- ment.

Pearson Education. this study has been a step towards understanding practical implications of SPC in a software organiza- tion. A. these costs can be justified by the associated improvements in the processes. defect type.W. and Florac. A. Statistical Methods for Software Quality. Florac. Card. It gives an opportunity to detect the problems and improve the software processes. 1999. One possible extension is to investigate the effect of different parameters (like program- ming language.Software Qual J (2006) 14: 135–157 155 to incorporate SPC implementation for rework percentage. Our study had some limitations due to time and data constraints. Carleton. Moreover. Therefore. ISBN: 0-451-62585-4. 97–106. and Carleton. we have evidence that it is possible to implement and benefit from SPC in an emergent software organization. ISBN 1-85032-171-X. a new study has been initialized to expand SPC analysis to include some other metrics (these are kept confidential by the organization). Another direction for future work is to try variations in SPC imple- mentation regarding the measures used. A. 1994. To sum up we see that SPC implementation is not a straightforward task in an emergent software organization.. Penguin Book USA Inc. J. Software Engineering Institute. Quality is Free: The Art of Making Quality Certain. Nevertheless. Burr. Carnegie Mellon University. A. A.D. A. and Carleton. As a further step. Similarly. The related data collection forms are also refined to collect some additional data for use in SPC. F. A. it may be easier to collect a sufficient number of data points by avoiding excessive classification.). Better results may be obtained by using different metric definitions (i. SLOC etc. 95–97.e. peer review type.. and the control limits applied. Carleton. 1999. Florac. with proper implementation a control chart acts as an auditor by demonstrating existing nonconformities about a process/product provided that the necessary preliminary actions are taken. IEEE Computer Mag- azine.1). ISBN 0-201-60444-2. Relatively low-maturity processes may require some additional effort before the actual implementation. J. Thomson Publishing Company. D. 2000. CMMI Product Team 2001. new process definitions are written to describe SPC details (the normalization of data. Nevertheless. pp. defect priority. IEEE Software pp. Brooks.) and actions to take in various situations.B. M. and Barnard. and Owen. Measuring the Software Process: Statistical Process Control for Software Process Improvement. number of people in the review for review performance) on SPC outcomes. Carnegie Mellon University. Springer . CMMI S M for Systems Engineering. Statistical Process Control for Software?.D. and Integrated Product and Process Development (CMMI-SE/SW/IPPD. V1. 1996. 1987.D. It is also an effective tool as it provides a visual interface with a scientific foundation. Statistical Process Control: Analyzing a Space Shuttle Onboard Software Process. charts drawn. No Silver Bullet: Essence and Accidents of Software Engineering. Classifying data points with respect to the relevant parameters provides more sensitive and effective results. Further studies are also needed to compare implementation of SPC for various metrics among different software organizations with different characteristics in terms of maturity and size. Analyzing Mature Software Inspection Process Using Statistical Process Control (SPC). Statistically Controlling the Software Process. drawing different statistical charts and applying limits other than 3-sigma for certain data. Crosby. for product size. P.D. 1980. The European SEPG Conference. The 99 SEI Software Engineering Symposium. Continuous Representation. IEEE Software. 1999. creation and interpretation of control charts etc.P. A.W. This will help us to have more expertise on SPC implementation in the software domain. SLOC definition for defect density. References Barnard. defect density and inspection per- formance metrics. Software Engineering.W. Moreover.R.

Addison-Wesley Publishing Company. Software Engineering Program. Park. 1999. Lockheed Martin Mission Systems. WADAS ’92.. A. Practical Applications of Statistical Process Control. T. The European SEPG Conference. T.156 Software Qual J (2006) 14: 135–157 Florac. During his 2-year work experience. and Chang. S. R. Paulk. (CMU/SEI 2001-SR-014). 2003. and Sova. 1939. pp.B. A.C. and Bush. Can Statistical Process Control be Usefully Applied to Software? The 11th Software Engineering Process Group (SEPG) Conference. E. Reading.C. Applying SPC Techniques to Software Development: A Management Perspective. and Saxena. 2002. Std 1044-1993. 1992. Optimum Control Limits for Employing Statistical Process Control in Software Process. he started to work as a software engineer in Milsoft Software A. Carnegie Mellon University. 1998.U. S. Kan. Cur- rently he is a PhD student in the University of Florida Computer and Information Sciences and Engineering Department.V. Umut Sargut He received his BS degree from Bilkent University In- dustrial Engineering department.. Quantitative Management in Software Engineering. Shewhart.1. 1999.M. After graduation. Std 1044. R. Paulk. B.R. NASA GB-001-94. Middle East Technical University (Master’s Thesis). Software Engineering Institute. 1993. IEEE Guide to Classification for Software Anomalies. M. Chrissis. Devor. 10th Software Engineering Process Group Conference. Sargut. Application of Statistical Process Control to Software Development Processes via Control Charts. 1999. Humphrey. Lancaster Press Inc. Paulk. Statistical Process Control for Software Projects. pp. Statistical Quality Design and Control. 2000. Mass. SPC in Ericsson. Carnegie Mellon University.B. 1997. Lantzy. Managing the Software Process.1-1995. 1995. D. Software Engineering Institute. Carnegie Mellon University. 1992.: Addison-Wesley Publishing Company. Garcia.C. C. Wigle. Practical Software Measurement: Measuring for Process Management and Improvement (CMU/SEI-97-HB-003). The 2001 High Maturity Workshop.. S. He received his master’s degree in Information Systems from Middle East Technical University. R. 1999. Software Measurement Guidebook. 1995.W.. Heijstek. The Euro- pean SEPG Conference.. ISBN 0-201-18095-2. Proceedings of the Ninth Washington Ada Symposium on Empowering Software Users and Developers.A.H. M. P. IEEE Transactions on Software Engineering 28(12): 1126–1134. Version 1. 1999. K. Jakolte. Sutherland. 1989. and Chrissis. IEEE Software. Statistical Method: From the Viewpoint of Quality Control.D.. G. W. Information Technology—Software Process Assessment—Part 4: Guide to Per- forming Assessments. ISBN 0-201-63339-6. 1999. Meade.A. M. he participated in process improvement studies for measurement and problem resolution processes. Metrics and Models in Software Quality Engineering. 2002. M. 48–55. A. Prentice Hall Publishing Company. M. IEEE Standard Classification for Software Anomalies. W. E.. Radice. and Carleton.D. Pajerski.. Weber. Keller. Application of Statistical Process Control to Software Processes. and witnessed a suc- cessful CMM Level 3 assessment. ISBN: 002329180X. Can Statistical Process Control be Usefully Applied to Software? The European SEPG Conference. A. M. A. The European SEPG Conference. Springer . Hirsch. and Carleton.S. 113–123. M. Key Practices of the Capability Maturity Model. J. The European SEPG Conference. ISO/IEC 15504-4:1998(E). Weller.B.

metu.ii. software engi- neering education.Sc. He managed a number of research and devel- opment projects on software process improvement. He has over 50 papers published in various books. His work focuses on software process improvement. ISO 15504 and CMM. He is currently working for Middle East Technical University as the head of the department of Information Systems – www.Sc. He worked as a consultant for a number of software developing companies to improve their processes based on ISO 9001. software project management. and M. and organizational change management. journals and conferences and over 20 students have completed their graduate degrees under his supervision.Software Qual J (2006) 14: 135–157 157 Onur Demirörs He has business process modeling and large scale software intensive system specification/acquisition. Springer .D. software engineering standards. degrees in Computer Science from Southern Methodist University and B. He has been working in the domain of software engineering as an academician. researcher and consultant for the last 15 years. degree in Computer En- gineering from Middle East Technical University.

Further reproduction prohibited without permission.Reproduced with permission of the copyright owner. .