Professional Documents
Culture Documents
• A difficult defect that was fixed at great expense Force Base in Montgomery, Ala. Gunter’s first
suddenly reappears. assessment was conducted in 1987 by Watts
• A developed and tested feature is mysteriously Humphrey. The base was supplying MIS software sup-
missing. port for the entire Air Force. They had about 500 out-
• A fully tested program suddenly does not standing Trouble Reports and so much pressure to get
work. the defects fixed that they were sending out a release
per month and patches in between. Based on the sta-
• The wrong version of the code was tested.
tistic that for every four defects one detects and
• There is no traceability between the software
“fixes” a new defect is introduced, one can imagine
requirements, documentation, and code.
the chaos. While they did not keep statistics, they
• Programmers are working on the wrong version were probably introducing two to three errors for
of the code. every four that they fixed. After the assessment, they
• The wrong versions of the configuration items focused on configuration management. When one of
are being baselined. the authors conducted the follow-up assessment in
• No one knows which modules comprise the 1989 while working at the SEI, the Air Force base was
software system delivered to the customer. still not at CMM level 2, but they had made such a dis-
Most of these problems were revealed during soft- tinct business difference in configuration management
ware process assessments conducted by the authors they told the whole world how well they had done. It
(Kasse 1995). Projects, under great pressure to meet was not surprising that the number of trouble reports
difficult deadlines, found their scarce time resource went from 500 to 50 and product releases went from
constantly under attack because of the rework caused once per month to once every six months. The results
by these reasons. One developer stated that he had were impressive for any organization.
“fixed” a problem three times, and three months after A key role of SCM is to control changes actively in
each delivery the problem recurred. Project leaders order to answer the following questions (Babich
cannot afford to have their software developers redoing 1986): What is the current software configuration?
what was already done. What is the status of the modules? What changes have
The cost of rework adds greatly to the cost of been made to the software? Do anyone else’s changes
developing software and can be reduced by imple- affect the software? SCM provides visibility into the
menting an effective SCM program. The principles status of the evolving software product. SCM answers
behind the modern cost-of-quality concept were who, what, when, and why. Who made the changes?
derived for manufacturing applications and can be What changes were made to the software? When were
found in the works of J. M. Juran (Juran and Gryna the changes made? Why were the changes made?
1988). In 1988 Raytheon Electronic Systems began Project leaders must be able to obtain answers to
using a cost of software quality model derived from these questions in order to manage the project’s
Philip Crosby (1984) and process improvement meth- technical activities and determine actual product
ods to increase efficiency. According to Herb Krasner evolution.
(1997), one of the leading researchers in the field of
the cost of quality, by 1991, Raytheon moved from
CMM level 1 to level 3; by 1994 it decreased rework SCM VS. CHANGE CONTROL
costs from 41 percent to 20 percent of project costs. SCM is often equated to change control. Indeed change
In 1995 it reached level 4. Between 1988 and 1994, control is a critical component of SCM, but it is only one
Raytheon’s cost of rework from both internal and of many. Following is a brief look at the components of
external nonconformances was reduced to less than SCM and how they connect to supporting a project
10 percent of development cost, and the productivity leader’s ability to manage and control the project.
of the development staff increased by a factor of 170
percent. Additional material on this topic can be
found at (Krasner 1998; Dion 1993; Haley 1996). Configuration Identification
Another example of how a strong SCM program Configuration identification supports the identifying
can reduce the cost of rework is that of Gunter Air of the structure of the software system and identifying
the related life-cycle work products. It provides a to modify the technical approach, and management
unique identifier for each work product, and it sup- wants to modify the project approach. Modification is
ports traceability between the requirements and all necessary, because, as time passes, all parties know
other related software products. more about what they need, which approach would be
Two structures that SCM is concerned with best, and how to get it done and still make money.
directly affect a project: The additional knowledge becomes the driving force
• Problem-solving structure. The system concept behind most changes.
evolves through the life cycle by successive The fundamental success of any development
refinement and elaboration of the resulting effort depends on well-defined reference points
work products. against which to specify requirements, formulate a
• Product system structure. The system is design, and specify changes to these requirements and
composed of subsystem components, which the resultant designs. The term baseline is normally
are themselves composed of subsystem used to denote such a reference point. A baseline is an
components. approved snapshot of the system at appropriate points
Figure 1 shows a V-Software life cycle (McDermid in the development life cycle. A baseline establishes a
1994) out of which comes a number of predefined formal base for defining subsequent change. Without
work products. Examples of work products that this line or reference point, the notion of change is
should be considered for placement under configura- meaningless. A baseline could be:
tion control include:
• A specification (for example, requirements
Source code modules Quality plans specification, design specification)
System data files Configuration management plans • A product that has been formally reviewed
System build files/scripts Compilers and agreed upon
Requirements specification Linkers/loaders
• A partial system
Interface specifications Debuggers
Design specifications Operating systems A baseline is a record of a contract and serves as
Software architecture specification Shell scripts the basis for further development. It should be changed
Test plans Third-party tools (STSC 1994) only through an agreed-upon change procedure. A
Test procedures Other related support tools baseline helps a project to control change without
User documentation Procedure language descriptions seriously impeding justifiable change. It will help a
Software development plan Development procedures &
project to control the identified configuration items
standards but not constrain early development excessively from
the aspects of time, money, or resources. Before a
Figure 2 is a simple example of a software product baseline is established, change may be made quickly
system composed of subsystems and modules and informally. Once a baseline is established, change
(Walker 1996). Each system or subsystem compo- can be made but a specific, formal procedure must be
nent may have associated with it an “include” file for applied to evaluate and verify each change. The items
code or data and a “make” file for creating compiled in the baseline are the basis for the work in the next
and linked systems or subsystems. A discussion phase of the software development cycle. The items of
between the project and the SCM representative can the next baseline are measured and verified against
help the project leader look critically at the software previous baselines before they become baselines
architecture and plan for evolutionary builds that themselves.
can be controlled and tested at the developmental Figure 3 illustrates both the types of baselines
and system level.
that are typical and the quality functions that may
be used to ensure that the work products are of the
Baselining highest quality before they are baselined (Bersoff,
Change is a fact of life in software development: Cus- Henderson, and Siegel 1980; Bryan and Siegel 1988;
tomers want to modify requirements, developers want Humphrey 1990). The functional baseline, allocated
Feasibility
study Project
Operation
phaseout
Requirements
definition V-Model Software
Development Life Cycle
Statement of Operational
requirements software
© 2000, ASQ
Code reading Code
• The software environment changes in a way record would be kept of what the change was, how-
that necessitates change. ever, why the change was requested, who approved
In each case, a new version of a configuration item the change, who made the change, and who verified
is needed that supersedes the earlier version. Without the change (IEEE 1997; Whitgift 1991). In addition,
change control, a software engineer could make an it would be hard to find answers to the following
important change to a configuration item or its questions:
interfaces without a lot of extra work and red tape. No • “Why doesn’t my software link this morning?
It linked last night!”
FIGURE 2 Product structure • “Why does this regression test fail now? It
worked yesterday!”
A software product system is composed of subsystems, which • “Why does the product behave this way now?
are in turn composed of subsystems, which are composed of
modules, which are composed of lines of code. It didn’t before!”
• “Why are we running out of memory now? We
System
did not have that problem yesterday!”
All changes made to the configuration management
Subsystem A Subsystem B Subsystem C baselines or baselined software configuration items
should be done according to a documented change
control process. The change control process should
Module A1 Module A2 specify:
Module C1
• Who can initiate the change requests
Subsystem B.1 Subsystem B.2 • The individuals, group, or groups who are
responsible for evaluating, accepting, and
tracking the change proposals for the various
© 2000, ASQ
baselined products
Module B11 Module B12
• What the criteria are for placing the software
components under formal change control
• The “change impact” analysis expected for that decisions to make changes are decided by the
each requested change project leader. If a change to a software module results
• How revision history should be kept in a required change to an interface module to a
• The check-in/check-out procedures hardware device, the project leader may share the
approval responsibility with the appropriate hardware
• The process the software configuration control
manager. When the software passes to the integration
board (SCCB) follows to approve changes
and test stage, the project leader may share approval
• How change requests will be linked to the authority to make changes with the integration and
trouble-reporting system test manager. When the product is ready to be shipped
• How change requests are tracked and resolved to the customer and a product baseline is established,
• The reviews and/or regression tests that must the SCCB again becomes the approval authority. What
be performed to ensure that changes have not baselines, when they are created during the software
caused unintended effects on the baseline life cycle, and who the approval authorities are
• The procedure that will be followed to update become part of each project’s SCM plan.
all affected software life-cycle components to
reflect the approved changes
To control the organizational or system baselines
Configuration Management
or contracts with the external customers, many organ- Status Accounting
izations establish one or more SCCBs. The purpose of Configuration management status accounting involves
the SCCB is to ensure that every change is properly maintaining a continuous record of the status and
considered and coordinated. This SCCB may include history of all baselined items and proposed changes to
members from program management, systems engi- them. It includes reports of the traceability of all
changes to the baseline throughout the software life
neering, software engineering, software quality
cycle, and it should identify what changes have been
assurance, SCM, independent test, documentation,
made to the system and what changes remain to be
hardware engineering, and may even include a customer
implemented.
representative. The SCCB is responsible for receiving
Configuration management status accounting
and initially evaluating the change requests that come
provides visibility into the system evolution by
from all sources (that is, customer, engineering, recording and reporting the status of all items and the
marketing, trouble reports, program management) status of all requests for change. Questions that con-
and performing triage to get the most critical or signif- figuration management status accounting should be
icant change requests to the right people for impact able to answer include (IEEE 1997; Whitgift 1991):
analysis. Following the impact analysis, the SCCB
• What is the status of an item? A programmer
ensures that all affected groups are able to recommit
may want to know whether a specification has
to the new requirements. The SCCB can make the been fully approved. He or she may want to
decision to implement the change request, defer it to know whether a subsystem has been tested so
the next release, or discard it altogether. It is also the programmer can test his or her modules
possible that the SCCB will have to seek additional that interface with that subsystem. A project
information before a decision can be made. leader will wish to track the progress of a
project as items are developed, reviewed,
tested, and integrated.
Project Leader Approval • Has a change request been approved or
of Baseline Changes rejected by the SCCB?
Once the allocated baseline is created and the customer • Which version of an item implements an
has accepted the software requirements specification approved change request? Once a requested
and interface specification, the control of the work enhancement of a library routine is imple-
products and system components comes under devel- mented, the originator and other developers
opmental configuration control. Normally this means will want to know which version of the routine
Product as
Requirements Consistent built Consistent Documentation
© 2000, ASQ
Requirements
traceability
matrix
• An audit of the formal test documentation In-process audits In-process audits are held during
against test data the design and development phases prior to the FCA
• An audit of the verification and validation and PCA to verify the consistency of the design as it
reports to ensure their accuracy evolves through the development process. In-process
• A review of all approved changes (problem audits are performed to determine:
reporting and corrective actions) to ensure • Are the hardware and software interfaces
they have been correctly technically incorpo- consistent with the design requirements?
rated and verified • Is the code fully tested to ensure that the
• A review of the updates to previously deliv- functional requirements are satisfied?
ered documents to ensure their accuracy and • Is the design of the product, as it is evolving,
consistency satisfying the functional requirements?
• A sampling of the design review outputs to • Is the code consistent with the detailed
ensure that all findings were completed and design?
incorporated
Traceability audits Traceability auditing is neces-
• A comparison of the code with the docu- sary to be able to verify throughout the software
mented software requirements to ensure that development process:
the code addresses all and only the docu-
• Can the software requirements be traced
mented requirements
back to the system requirements allocated to
• A review to ensure that all testing was accom-
software?
plished with appropriate test documentation
• Can the architectural design be traced back to
and approved test data to ensure that all
the software requirements?
configuration items meet the established
performance criteria • Can the detailed design(s) be traced back to
the architectural requirements?
Physical configuration audits The objective of the
• Can the code modules be traced back to
physical configuration audit (PCA) is to provide an
detailed design modules?
independent evaluation of the system configuration
items to confirm that each item that makes up the “as • Can the module tests, integration tests, and
built” system maps to its specifications. The audit is system tests be traced back to the software
held to verify that the software and its documentation requirements?
are internally consistent and ready for delivery to the Traceability audits are also necessary to be able to
customer or end user. Appropriate customer deliverable answer the following questions:
documentation includes installation manuals, operating • Can it be proven that what one is doing is
manuals, maintenance manuals, and release notes or required?
version description documents. A PCA includes:
• How does one know that what he or she is
• An audit of the system specification for delivering is what the customer asked for?
completeness • Are the software life-cycle work products
• An audit of the FCA report for discrepancies consistent as the system evolves and change
and actions taken requests are processed?
• A comparison of the architectural design with Software configuration audits check that the config-
the detailed design components for consistency uration management system and practices are effective
• A review of the module listing for compliance and efficient for the project and the organization.
with approved coding standards
• An audit of the manuals for format com-
pleteness and conformance to systems and Software Library
functional descriptions. Such manuals include The software library should contain the items that are
user’s manuals, programmer’s manuals, and important to a software project, including source
operator’s manuals code, user documentation, system documentation,
requests have been implemented completely and Krasner, H. 1997. Accumulating the body of evidence for the payoff of soft-
correctly. Once established, FCAs would be the next ware process improvement. URL document www.utexas.edu/coe/sqi/archive .
step to ensure that the system matches what was Krasner, H. 1998. Using the cost of quality approach for software.
approved, and nothing more. Then, a physical audit Crosstalk, The Journal of Defense Software Engineering (November) (or
see URL www.stsc.hill.af.mil/crosstalk/1998/nov/krasner.asp).
can be added to ensure that the documentation
Paulk, M. C., B. Curtis, M. B. Chrissis, and C. V. Weber. 1993. Capability
matches the changes.
Maturity Model for Software, version 1.1. (CMU/SEI-93-TR-24).
It is important to implement requirements trace- Pittsburgh: Software Engineering Institute, Carnegie Mellon University.
ability from the beginning through systems testing,
Paulk, M. C., C. V. Weber, S. M. Garcia, M. B. Chrissis, and M. Bush.
implementing life-cycle work-product consistency 1993. Key Practices of the Capability Maturity Model, version 1.1.
checks. This means that if a requirements change (CMU/SEI-93-TR-25). Pittsburgh: Software Engineering Institute,
request has been accepted, one must go through life- Carnegie Mellon University.
cycle phases to determine if it is necessary to make a STSC. 1994. Software configuration management technology report,
corresponding change. Without the ability to trace Software Technology Support Center. (This document provided by the
through the life-cycle process, it cannot be done. STSC is a checklist of criteria to help organizations develop their require-
Once one has the design document, he or she needs ments for selecting configuration management tools.)
the ability to go backward and forward to look at the Walker, G. 1996. What is configuration management? Alcatel Alsthom
other life-cycle work products. internal presentation, Paris, France.
After expanding to multisite, multicountry, and mul- Whitgift, D. 1991. Methods and tools for software configuration
ticultural projects, one may be looking at implementing management New York: John Wiley & Sons.
Methodist University and a bachelor’s degree in systems engineering from McQuaid has a doctorate and a master’s degree in computer science and
the University of Arizona. He is a member of IEEE and has participated in engineering from Auburn University, an MBA from Eastern Michigan Uni-
the development of the IEEE standard on reviews and audits. Kasse can versity, and an undergraduate degree in accounting from Case-Western
be reached at Kasse Initiatives, LLC., 30 W. Sheffield Ave., Gilbert, AZ Reserve University. She is a Senior member of ASQ, and a member of the
85233, or by e-mail at kassetc@aol.com . Association for Computing Machinery, IEEE, IEEE Computer Society, the
Patricia McQuaid is an associate professor of management information International Function Point Users Group, the Information Systems Audit
systems at California Polytechnic State University. She has taught a wide and Control Association, and the Decision Sciences Institute. She can be
range of courses in both the college of business and college of engineer- reached at California Polytechnic State University-MIS, College of Busi-
ing. She has industry experience in information systems auditing in both ness, San Luis Obispo, CA 93407, or by e-mail at pmcquaid@calpoly.edu .
the banking and manufacturing industries, and is a certified information
systems auditor (CISA). Her research interests include software quality, in
particular, the areas of software process improvement, software testing,
and complexity metrics. McQuaid is serving as the chair for the Americas
for the Second World Congress for Software Quality, to be held in Japan
in September 2000.