Edited by Foxit Reader
Copyright(C) by Foxit Software Company,2005-2008
For Evaluation Only.
ELSEVIER
Maturity, Models, and Goals:
How to Build a Metrics Plan
Shari Lawrence Pfleeger
Centre for Software Reliabilily, City University London, England
Many software project managers want to implement a
process or product improvement program for their
software development or maintenance project. To be
effective, the improvement activities must be accom-
panied by measurements to support them. Thus, man-
agers must select metrics for a measurement plan
that will start small and address key project needs.
This article explains what a metrics plan is and de-
scribes an approach that combines a goal-question-
metric analysis with process maturity assessment and
process modeling. This combination allows an organ-
ization to tailor metrics collection and analysis to
the needs and characteristics of the project and organ-
ization that will use the metrics. The goal-question-
metric technique ensures that each measurement is
useful to someone in the organization. The process
maturity framework is used to ensure that what needs
to be measured is visible in the process, and the
process model helps the project manager understand
when and where metrics will be collected and how
they will be used. In concert, these techniques enable
the project manager to migrate from a small, initial set
of key indicators to a larger, more comprehensive
measurement program. The article lists example goals,
questions, and metrics that have been identified for a
defense organization and a reuse project manager,
and describes how the techniques discussed have
actually been used on real metrics programs to tailor
metrics plans to individual project needs.
INTRODUCTION
Many software project managers acknowledge that
measurement helps them understand and guide the
development of their projects. They want to select
particular metrics, perhaps as part of an overall
Address correspondence to Shari Lawrence pfleeger, Centre for
Sofhvare Reliabili& City Uniuersity, Northampton Square, London
EC1 V OHB, United Kingdom.
J. SYSTEMS SOFTWARE 1995; 31:143-l%
0 1995 by Elsevier Science Inc.
655 Avenue of the Americas, New York, NY 10010
development plan, but they don’t know how to be-
gin. The software engineering literature describes
dozens of metrics that can be applied to a wide
variety of project, process, and product attributes
(Conte et al. 1986; Pfleeger, 1991; Fenton, 1991).
Which metrics should be chosen for a given project?
How should the metrics be used? What actions
should be taken in response to patterns indicated by
the measurements? All of these questions should be
answered in the project’s metrics plan so that man-
agers and developers know what to collect, when to
collect it, and how the data relate to management
decisions. This article describes how to build such a
metrics plan so that managers can establish a flexi-
ble and comprehensive metrics program as part of a
larger process or product improvement program.
The approach to metrics planning combines three
techniques: the goal-question-metric paradigm (Bas-
ili and Rombach, 1988), the process maturity frame-
work for metrics (Pfleeger and McGowan, 1990),
and models of an organization’s software develop-
ment or maintenance process (McGowan and Boh-
ner, 1993). This combination of methods has been
used successfully in different situations.
We begin by describing the contents of a metrics
plan and outlining the types of information th’at such
a plan should include. Then we explain how to
derive measurements from project goals, and from
them, key questions that need to be answered. That
is, a manager can decide what measurements are
needed, not just what measurements are possible.
Next, we show how to use notions of process matu-
rity and visibility for selecting metrics. Maturity can
help managers decide what is visible enough to
measure. Then maturity is combined with the goals
and questions to produce a list of candidate metrics
for a given project or organization. Once a subset of
the candidates is chosen as a starter set by the
0164.1212/95/$9.50
SSDI 0164-1212(94)00094-4
144 .I. SYSTEMS SOFTWARE
1995: 31:143-m
project manager, process models can be used to help
determine when selected metrics will be collected
and analyzed and how the results will be applied to
improve the process. Throughout the discussion, we
present examples of how these techniques have been
used in real projects to decide how to begin metrics
collection and analysis.
WHAT IS A METRICS PLAN?
A metrics plan is much like a newspaper article: it
must describe the who, what, where, when, how, and
why of metrics. With answers to all of these ques-
tions, whoever reads the plan knows exactly why
metrics are being collected, how they are used, and
how metrics fit into the larger picture of software
development or maintenance.
The plan usually begins with the why. It is in this
introductory section that the plan lays out the goals
or objectives of the project, describing what ques-
tions need to be answered by project members and
project management. For example, if reliability is a
major concern of the developers, then the plan
discusses how reliability will be defined and what
reporting requirements are imposed on the project;
later sections of the plan can then discuss how
reliability will be measured and tracked.
Next, the plan addresses what will be measured.
In many cases, the measurements are grouped or
related in some way. For instance, productivity may
be measured in terms of two component pieces, size
and effort. So the plan will explain how size and
effort are defined and then how they are combined
to compute productivity.
At the same time, the plan must lay out where and
when during the process the measurements will be
made. Some measurements are taken once, whereas
others are made repeatedly and tracked over time.
The time and frequency of collection are related to
the goals and needs of the project, and those rela-
tionships should be made explicit in the plan.
How and who address the identification of tools,
techniques, and staff available for metrics collection
and analysis. It is important that the plan discuss not
only what measures are needed but also what tools
will be used for data capture and storage and who is
responsible for those activities. Often, metrics plans
ignore these crucial issues, and project members
assume that someone else is taking care of metrics
business; the result is that everyone agrees that the
metrics are important, but no one is actually as-
signed to get the data. Likewise, responsibility for
analysis often slips through the project cracks, and
data pile up but are never used in decision making.
S. L. Pfleeger
The plan must state clearly what types of analysis
will be carried out with the data, who will do the
analysis, how it will be conveyed to the decision
makers, and how it will support decisions.
Thus, the metrics plan paints a comprehensive
picture of the measurement process, from initial
definition of need to analysis and application of the
results. The remainder of this article discusses tech-
niques that aid in answering the questions addressed
in the metrics plan.
WHY MEASURE?
We measure because we need to know about an
attribute or characteristic of something that is im-
portant to us. A metric is a vehicle for that under-
standing, providing a standard unit to be used as a
basis for comparison among like items or character-
istics. For example, in our everyday life, meters,
liters, and grams are used as three basic metrics
for comparing measurements of the length, capac-
ity, and weight of many things. In the same way,
software metrics provide a basis for comparing the
various aspects of software development and main-
tenance that are needed for understanding and eval-
uating the development and maintenance processes.
One way that metrics promote understanding is by
making aspects of development or maintenance more
visible, so we can see into the ways in which pro-
cesses, products, resources, methods, and technolo-
gies relate to one another. Aided by metrics, we can
answer questions about the effectiveness of tech-
niques or tools, the productivity of activities (such as
testing or configuration management), the quality of
products, and more. In addition, metrics allow par-
ticipants in the process to define a baseline for
understanding the nature and impact of proposed
changes. Finally, metrics allow both managers and
developers to monitor the effects of activities and
changes on all aspects of development, so that ac-
tion can be taken as early as possible to control the
final outcome should actual measurements differ
significantly from plans. Thus, measurement is use-
ful for
l visibility and understanding
l establishing a baseline for improvement
l planning, monitoring, and controlling products,
processes, and resources
Although a measurement program is an essential
part of any development program, it is useful only in
the larger context of assessment and improvement.
Choosing metrics, collecting data, analyzing the re-
How to Build a Metrics Plan
suits, and taking appropriate action require time and
resources; these activities make sense only if they
are directed at specific improvement goals.
GOALS, QUESTIONS, AND METRICS
A manager or developer may find a particular mea-
surement useful when it promotes understanding of
a process, its resources, or one of its resultant prod-
ucts. However, the utility derives from knowing how
the process, product, or resource relates to overall
project goals. Thus, recognizing improvement can
occur only when the project has clearly defined goals
for process and products. That is, you can’t tell if
you are going in the right direction until you know
where you started and where you want to go. The
goal-question-metric (GQM) approach to process
and metrics, first suggested by Basili and Rombach
(1988), provides a structure from which managers
and developers can derive a set of crucial project
goals, plus the questions that must be answered in
order to tell if the goals have been met. The tech-
nique has proven to be an effective approach to
implementing metrics at NASA’s Goddard Space
Flight Center, Contel Corporation, and elsewhere
(Grady and Caswell, 1987; Rifkin and Cox, 1991;
Pfleeger, 1993). The technique works as follows.
First, the overall goals of the organization are delin-
eated. From each goal, questions are identified, the
answers to which help managers know if progress is
being made toward reaching the goal. Then, each
question is analyzed in terms of what measurements
are needed to help determine the answer to each
GOAL: Evaluate effectiveness of coding standard
QUESTIONS: Who is using
stanxm
METRICS: Proportion
of coders
- using
standard
- using
language
Figure 1. Example of deriving metrics from goals and
questions.
J. SYSTEMS SOFTWARE 145
1995; 31:143-155
question. In this way, an understanding of goals
is used to generate suggested metrics for a given
project.
The GQM templates provided by Basili and Rom-
bath (1988) also encourage managers to consider
the context in which a question is being asked, as
well as the viewpoint of the questioner. For example,
a programmer is likely to ask different questions
about productivity or quality than a corporate vice
president. The programmer may want to know pro-
ductivity to be able to project when he or she will
finish designing or coding a module, whereas the
vice president may want to know if productivity has
been improved by a recent investment in tools. The
viewpoint can affect the way a metric is defined (by
the programmer on a module-by-module basis or by
the vice president on a project-by-project basis) and
used.
By deriving measurements from goals and ques-
tions, it is clear how the resulting data points will be
related to one another and used in the larger con-
text of the project. Figure 1 illustrates a simple
example of how several metrics might be generated
from a single goal. The figure shows that the overall
goal is to evaluate the effectiveness of using a coding
standard. To determine if the standard is effective,
several key questions must be asked. First, it is
important to know who is using the standard, so that
the project manager can compare the productivity of
the coders who use the standard with the productiv-
ity of those who do not. Likewise, the manager
wants to compare the quality of standard-compliant
code with the quality of nonstandard code. To ad-
What is coder What is code
Experience of Code Effort Errors ...
coders Size
- with standard (lines of
- with language code, function
_ with environ- points, etc.)
ment,
etc.
 Edited by Foxit Reader
Copyright(C) by Foxit Software Company,2005-2008
For Evaluation Only.
146 J.SYSTEMS SOFlWARE
1995: 31:143-m
dress these issues, it is important to ask question
about productivity and quality.
Once these questions are identified, each question
is analyzed to determine what must be measured in
order to answer the question. For example, to un-
derstand who is using the standard, it is necessary to
know what portion of coders uses the standard.
However, it is also important to have an experience
profile of the coders that explains how long they
have worked with the standard, the environment, the
language, and other factors that might affect code
quality. The productivity question requires a defini-
tion of productivity, which is usually some measure
of effort divided by some measure of product size.
As shown in Figure 1, the measurement can be
expressed in terms of lines of code, function points,
or any other metric that will be useful to the project
manager. Similarly, quality may be measured in terms
of the number of errors found in the code, failure
rates, or other quality measurements that the pro-
ject manager would like to use.
In this way, only those measurements that are
related to the goal are generated. Notice that, in
many cases, several measurements may be needed to
answer a single question. Likewise, a single mea-
surement may apply to more than one question. The
goal provides the purpose for collecting the data,
and the questions tell the project how to use the
data. What is not evident from the GQM tree is the
model needed to combine the measurements in a
sensible way so that the questions can be answered.
For example, coder productivity may be measured in
terms of effort per line of code, but that relationship
is not explicit in the tree. Thus, the GQM approach
must be supplemented by one or more models that
express the relationships among the metrics.
Similarly, the metrics are not always clear and
unambiguous. The tree does not tell the project
manager how to measure a line of code or a func-
tion point, but only that some measurement of code
size is needed to answer a question about productiv-
ity. Additional work is needed to define each metric.
In cases where no objective measurement is avail-
able, subjective measurements must be identified.
To see how the GQM paradigm can be applied on
a broad scale, consider MITRE’s Metrics Advisor
tool (J. Heimberger, 19941, which uses goals, ques-
tions and metrics to help a project manager tailor a
metrics program to the needs and goals of a devel-
opment project. MITRE Corporation was asked by
the U.S. Department of Defense (DOD) to assist in
designing a metrics program for a large number of
DOD organizations. Rather than perform a GQM
analysis for each organization and use it to suggest
S. L. Pfleege
metrics, the MITRE group chose instead to tab
advantage of the commonalities among the organi
zations needing a metrics plan. The MITRE tean
developed a prototype tool that uses hypermedia tc
guide a project manager through choices for goal:
and questions. Such an approach allows a projec
manager to select those goals that are most critica
first, and the tool then suggests measurements im
mediately useful on the manager’s project. Afte:
consultation with DOD representatives, it appearec
that any DOD project is likely to have at least one o
the following high-level goals:
l Improving productivity
l Improving quality
l Reducing risk
Each of these high-level goals can then be exam
ined for its implications for resources, products, am
process. For example, the goal of improving produc
tivity can involve several subgoals for resources:
l Ensuring adequate staff skills
l Ensuring adequate managerial skills
l Ensuring adequate software engineering technol
ogY
Similarly, improving productivity with products in
volves three tasks:
l Identifying problems early in the process
l Using appropriate technology
0 Leveraging reuse
Interpreting productivity in terms of process mean
ensuring both appropriate technology and effective
process activities. Table 1 shows an expanded hierar
thy of goals and subgoals derived in this way. Once
the manager selects the goals and subgoals that art
most important to the project, the tool asks him OI
her to select questions (related to each goal) tha
reflect the areas of deepest concern. For example, i
the manager has chosen
l Improving productivity
and then
l Ensuring adequate staff skills
he or she can then select at least one of the follow
ing questions:
1. Does project staffing have the right skills mix fol
the job?
2. Does the staff have adequate experience?
How to Build a Metrics Plan J. SYSTEMS SOFTWARE I47
1995: 31:143-155

Table 1. Goals and Subgoals for a ‘Apical DOD Project their development projects. For instance, users in
ImprovingProductivity the Army must collect standard metrics, called STEP
Improvingproductivityusingresources (Software Test and Evaluation Plan) metrics, in ad-
Ensuring adequate staff skills dition to the ones generated by the goals and ques-
Ensuring adequate managerial skills
Ensuring adequate host software engineering technology tions.
Improving productivity using products However, specific metrics recommendations can-
Identifying problems early not yet be offered, because there is a missing piece
Using appropriate technology
Leveraging reuse to the metrics puzzle. Measurements can be made
Improving productivity using process only when the item to be measured is visible enough
Using appropriate technology in the deveIopment or maintenance process to be
Ensuring that the process activities are effective
Improving Quality understood and controlled. To see how visibility
Improve quality using resources relates to measurement, we turn to the notion of
Ensuring adequate staff skills process maturity.
Ensuring adequate managerial skills
Ensuring adequate target software engineering technology
Improve quality using products
Improving requirements quality APPROPRIATE MEASUREMENT: METRICS
Improving design quality AND PROCESS MATURITY
Improving code quality
Improving test quality Some development processes are more mature than
Improving documentation quality others, in that some processes are open and easily
Improving quality using process
Detecting and eliminating errors early in the process controlled, whereas others are arcane and depen-
Ensuring that process activities are effective dent on a small set of individual programmers. The
Reducing risk using resources notion of process maturity, first developed at IBM,
Ensuring staff stability
Ensuring that hardware and software are adequate
has been expanded and built on by the Software
Reducing risk using products Engineering Institute’s (SE11 reports and activities
Complying with budget and schedule constraints on process maturity and capability maturity (Hum-
Estimating and forecasting accurately
phrey, 1989; Paulk, 1991, 1993a and b). A key dis-
Maintaining requirements stability
Maintaining design stability criminator among process maturity levels is the
Maintaining code stability amount of visibility developers and managers have
Reducing risk using process
into the overall development process. At the lowest
Preventing errors
levels of maturity, the process is not well understood
at all; as maturity increases, the process is better
understood and better defined. Measurement and
Similarly, if the manager has chosen visibility are closely related: a developer can mea-
l Improving quality sure only what is visible in the process, and measure-
ment helps to enable and increase visibility. In other
and then words, what we can see is what we can measure.
l Improving requirements quality Thus, as noted by Pfleeger and McGowan (19901,
the five-level maturity scale used by the SE1 is a
then the related questions are as follows: convenient context for determining what to measure
first and how to plan an increasingly comprehensive
1. Is each requirement clear and understandable?
measurement program.
2. Is each requirement testable?
Table 2 presents an overview of the types of
3. Is each requirement reusable?
metrics suggested by each maturity level. Level 1
4. Is each requirement maintainable?
metrics provide a baseline for comparison as im-
5. Is each requirement correct?
provements are sought. Level 2 metrics focus on
After navigating through the hierarchy and identi- project management, whereas level 3 metrics mea-
fying the key goals and questions, the manager is sure the intermediate and final products produced
directed to describe several organizational charac- during development. The metrics at level 4 capture
teristics that affect the way in which the tool will characteristics of the development process itself to
suggest metrics. For example, the manager should allow control of the individual activities of the pro-
identify where the DOD the organization is located. cess. A level 5 process is mature enough and man-
This information can be used to expand the scope of aged carefully enough to allow metrics to provide
the suggestions, because some military groups have feedback for dynamically changing the process dur-
very specific measurement requirements for all of ing a particular project’s development.
148 J. SYSTEMS SOFTWARE
1995; 31:143-155
Table 2. Overview of Process Maturity and Measurement*
Maturity Level Characteristics
5. Optimizing Improvement fed back to the process
4. Managed Measured process
3. Detined Process defined and institutionalized
2. Repeatable Process dependent on individuals
1. Initial Ad hoc
*Pfleeger and McGowan, 1990.
The initial level of process maturity is character-
ized by an ad hoc approach to the software develop
ment process. That is, inputs to the process are ill
defined, the outputs are expected, and the transition
from inputs to outputs is undefined and uncon-
trolled. Similar projects may vary widely in their
productivity and quality characteristics because of
lack of adequate structure and control. For this level
of process maturity, it is difficult even to write down
or depict the overall process; the process is reactive
and ill defined, depending on particular individuals
and their capabilities rather than on clear and ex-
plicit activities and controls. For this reason, visibil-
ity into a level 1 process is nil and comprehensive
measurement difficult. Managers may have goals
relating to improved quality and productivity, but
they do not know what current levels of quality and
productivity are. For this level of process maturity,
baseline metrics should be collected to provide a
starting point for measuring improvement as matu-
rity increases. Developers at level 1 should concen-
trate on imposing more structure and control on the
process, in part to enable more meaningful measure-
ment.
The second maturity level, called repeatable, iden-
tifies the inputs and outputs of the process, the
constraints (such as budget and schedule), and the
Control
Resources
Staff
Tools
S. L. Pfleeger
Type of Metrics to Use
Process plus feedback for changing the process
Process plus feedback for control
Product
Project management
Baseline
resources used to produce the final product. The
process is repeatable in the same sense that a sub-
routine is repeatable; proper inputs produce out.
puts, but there is no visibility into how the outputs
are produced. Asked to define and describe the
process, the manager can draw no more than a
diagram similar to Figure 2. This figure shows a
repeatable process as a structure analysis and design
technique (SADT) diagram, with input on the left!
output on the right, constraints at the top, and
resources on the bottom. For example, requirements
may be input to the process, with the software
system as output. The control arrow represents such
items as schedule and budget, standards, and man
agement directives; the resources arrow can include
tools and staff.
Because visibility suggests measurement, Figure 2
suggests that project management metrics make the
most sense for a repeatable process. That is, because
all that is visible are the arrows, it is possible tc
associate metrics with each arrow in the process
diagram. Thus, for a repeatable process, measures ot
the input might include the size and volatility of the
requirements. The output may be measured in terms
of system size (functional or physical), the resources
as overall staff effort, and the constraints as cost and
schedule in dollars and days, respectively.
Figure 2. A repeatable process (level 2).
How to Build a Metrics Plan
Inspection
Design Method
s Critefia
+
Require- Define,
ments+ Design
t
Tools,
Staff
Figure 3. A defined process (level 3).
The defined level of maturity (level 3) differs from
level 2 in that a defined process provides visibility
into the “construct the system” box. At level 3,
intermediate activities are defined, and their inputs
and outputs are known and understood. This addi-
tional structure means that the input to and output
from the intermediate activities can be examined,
measured, and assessed, because these intermediate
products are well defined and visible. Figure 3 shows
a simple example of a defined process with three
typical activities. However, different processes may
be partitioned into more distinct functions or activi-
ties.
Because the activities are delineated and distin-
guished from one another in a defined process,
product measurement should begin no later than
level 3. Defects discovered in each type of product
can be tracked, and the defect density of each prod-
uct can be compared with planned or expected val-
ues. In particular, early product measurements can
be useful indicators of later product measurements.
For example, many changes to the requirements may
Rep rting requirements from senior management
Figure 4. A managed process (level 4).
J. SYSTEMS SOFIWARE 149
1995; 31:143-IS.5
Test
Plans
System
Software
Tools, Tools,
Staff Staff
suggest instability of the requirements or design,
which can affect the quality of the code. By measur-
ing and tracking changes and their impact, charac-
teristics of the requirements or design can be mea-
sured and used to predict the quality of the code.
Such measurements use process visibility to provide
more control over development: if requirements
quality is unsatisfactory, additional work can be ex-
pended on the requirements before the design activ-
ity begins. This early correction of problems helps
not only to control quality but also to improve
productivity and reduce risk.
A managed process (level 4) adds to a defined
process some visible degree of management over-
sight, as shown in the example process of Figure 4.
Here, feedback from early project activities (e.g.,
problem areas discovered in design) can be used to
set priorities for current activities (e.g., redesign)
and later project activities (e.g., more extensive re-
view and testing of certain code and a changed
sequence for integration). Because activities can be
compared and contrasted, the effects of changes in
f
I 1 I 1
150 J. SYSTEMS SOFTWARE
1995; 31:143-155
one activity can be tracked by others. By level 4, the
feedback determines how resources are deployed;
the basic activities themselves do not change. At this
level, the effectiveness of process activities can be
evaluated: how effective are reviews? configuration
management? quality assurance? defect-driven test-
ing? The measurements collected are used to stabi-
lize the process, so that productivity and quality will
match expectations.
A significant difference between levels 3 and 4 is
that level 4 measurements reflect characteristics of
the overall process and the interaction among major
activities. Management oversight relies on a metrics
data base that can provide information about such
characteristics as distribution of defects, productivity
and effectiveness of tasks, allocation of resources,
and the likelihood that planned and actual values
will match.
An optimizing process is the ultimate level of
process maturity; it is depicted in Figure 5. Here,
measurements from activities are used to improve
the process, possibly by removing and adding process
activities and changing the process structure dynam-
ically in response to measurement feedback. Thus,
the process change can affect the organization and
project as well as the process. Results from one or
more ongoing or completed projects may also lead
to a refined, different development process for fu-
ture projects. The spiral model is an example of such
a dynamically changing process, responding to feed-
back from early activities in order to reduce risk in
later ones.
This tailoring of the process to the situation is
indicated in Figure 5 by the collection of process
boxes labeled To, T,, . . . , T,,. At time To, the process
is as represented by box T,,. However, at time T,,
management has the option of revising or changing
the overall process. For example, the project man-
ager may begin development with a standard water-
S. L. Pfleeger
fall approach. As requirements are defined and de-
sign is begun, metrics may indicate a high degree of
uncertainty in the requirements. Based on this infor-
mation, the process may change to one that proto-
types the requirements and the design, so that some
of the uncertainty can be resolved before substantial
investment is made in implementation of the current
design. In this way, an optimizing process gives maxi-
mum flexibility to development. Metrics act as sen-
sors and monitors, and the process is not only under
control but can change significantly in response to
warning signs.
The five categories of metrics described here-
baseline, project management, product, process for
feedback, and process for control-are meant to be
nested. That is, at a given maturity level, the metrics
for that level and all levels below it can be collected.
However, it is important to note that process matu-
rity does not involve a discrete set of five possible
ratings. Instead maturity represents relative loca-
tions on a continuum from 1 to 5. An individual
process is assessed or evaluated along many dimen-
sions, and some parts of the process can be more
mature or visible than others. For example, a repeat-
able process may not have well-defined intermediate
activities, but the design activity may indeed be
clearly defined and managed. The process visibility
diagrams presented here are meant only to give a
general depiction of typical processes. It is essential
to begin any measurement program with an exami-
nation of the process model and determination of
what is visible. The figures and tables should not
proscribe metrics simply because the overall matu-
rity level is a particular integer; if one part of a
process is more mature than the rest, then metrics
can enhance the visibility of that part and help to
meet overall project goals, at the same time bringing
the rest of the process up to a higher level of
maturity. Thus, in a repeatable process with a well-
J Figure S. An optimizing process (level 5).
How to Build a Metrics Plan
defined design activity, design quality metrics may be
appropriate and desirable, even though they are not
generally recommended for level 2.
Moreover, it is important to remember that suc-
cessful metrics programs start small and grow ac-
cording to the goals and needs of a particular pro-
ject (Rifkin and Cox, 1991). A metrics program
should begin by addressing the critical problems or
goals of the project, viewed in terms of what is
meaningful or realistic at the project’s maturity level.
The process maturity framework then acts as a
guideline for how to expand and build a metrics
program that not only takes advantage of visibility
and maturity but also enhances process improve-
ment activities. The next section illustrates how pro-
cess maturity can be combined with GQM to suggest
particular metrics.
COMBINING GQM WITH PROCESS MATURITY
Suppose a manager has done a GQM analysis of an
upcoming project for customer X and decides that
quality is the most important project goal. Further-
more, suppose the manager has worked with cus-
tomer X before and knows that X likes to add
requirements to the project as development pro-
gresses. Thus, the manager decides that a critical
question to be answered is this: Are the require-
ments growing?
Consider how process maturity affects the way in
which this question can be answered. If the project’s
process is at level 1, then the requirements are likely
to be ill defined. Measuring characteristics of the
requirements is difficult at this level, so measuring
what is visible means that at best the number of
requirements can be counted in some way and estab-
lished as a baseline. This may be a simple measure-
ment, such as number of sentences or number of
“shahs.” At level 2, the same question can be an-
swered more easily. The requirements are well de-
fined, and the user can not only count requirements
but also capture additional information about each
requirement. For instance, the type of each require-
ment (data, interface, and so on) can be very useful
in answering the question and understanding what is
happening on the project. Here, the additional visi-
bility into the requirements offered by type informa-
tion provides a finer granularity to the measurement
and more insight into what is happening. At level 1,
the manager can show X that requirement growth is
severe and is likely to affect the schedule or budget.
But at level 2, the manager has a lot more to say. By
knowing growth within types of requirements, the
manager can pinpoint the fact that growth is occur-
J. SYSTEMS SOFTWARE 151
1995; 31:143-155
ring mainly in the number of interface requirements,
for example, and much more specific action can be
taken to limit the problems such growth might cause.
Likewise, at level 3, visibility is improved even
more, and intermediate activities are defined, with
entry and exit criteria for each activity. Here, re-
quirements number and type can be augmented with
the traceability of each requirement to the corre-
sponding design, code, and test components. By this
level, the manager can begin to assess the impact of
each new requirement on particular parts of later
system products. This new information enables the
project team, including the customer, to take action
early in development that will head off potential
problems later.
Thus, GQM and maturity work hand in hand to
suggest measurement related to key project goals.
This combination offers several advantages. First, it
allows even immature projects to start small, estab-
lishing not only a baseline for measurement but an
understanding of which parts of the process need to
be made more explicit and controllable. That is, the
process of choosing metrics is itself a process im-
provement activity, because it highlights areas where
more information and control are needed. Second,
the combination provides a migration plan from a
starter set of measurements to a larger, more com-
prehensive measurement program. The expansion
happens either when new goals must be addressed
by measurement or when process improvement and
increased visibility enable richer or expanded mea-
surement. Finally, the use of GQM and maturity
encourage the application of measurement back to
process, product, or resource improvement. Some-
times, projects collect metrics information to meet
an externally imposed requirement: “thou shalt col-
lect and report metrics.” But metrics derived from
maturity and goals are metrics derived from a clear
and visible project need; they are more like to be
used to support important project decisions and
actions.
WHERE, WHEN, WHO AND HOW: THE
PROCESS MODEL
The approach described in the previous section helps
the project manager to decide
l what to measure
l why to measure it
However, a metrics plan is neither complete nor
useful until the developers and maintainers know
l who is responsible for collecting data
152 J. SYSTEMS SOFTWARE
1995; 31:143-155
l when the data will be collect and with what fre-
quency
l where in the process the data will be collected
l how the data will be collected
Suppose a manager has selected a small set of
measurements to answer key questions for managing
the project. To understand where and when the data
will be collected, the manager needs a model of the
development or maintenance process. The model
can be expressed in any of several ways, graphical or
textual. Whatever way the process is described or
depicted, the model must have the following charac-
teristics:
1. The process model must include all of the major
process activities that involve measurement.
2. The model must be detailed enough to show the
products, resources, and subprocesses that will be
measured.
3. The model must display the inputs and outputs of
each process activity that will be measured.
4. The model must identity the constraints on the
process activities (such as budgets, schedules, and
applicable standards).
5. The model must identify the resources that will
be used in each process activity (such as hard-
ware, software, and personnel).
When the model is complete, the manager can
examine each of the candidate measurements to
determine where in the process the measuring will
be done. Then, the metrics plan records who will
collect that data, how the data will be collected (e.g.,
manually, with a particular tool, and so on), and how
often. The picture of the process can be annotated
with where in the process the measuring will be
done. Sometimes, the activity of examining the pro-
cess in this way leads the project manager to revise
the development or maintenance process, thus mak-
ing things more visible so that they are easier to
measure and assess. So again, measurement and
process improvement go hand in hand. To see how a
process model, coupled with GQM and maturity, is
generated and used to support measurement, the
remainder of this section turns to the problem of
measuring the effectiveness of reuse.
Software reuse refers to the organized and man-
aged generation, control, and use of software arti-
facts, so that artifacts produced in one maintenance
or development project can be used again in other
projects. Artifacts can include any output of devel-
opment or maintenance, including requirements, de-
sign components, code, documentation, test data,
S. L. Pfleeger
test scripts, models, architectures, and more. A reuse
management program can enable management to
l understand the reuse process
l evaluate the quality of the artifacts of develop-
ment and maintenance and the library that con-
tains the artifacts
l control the activities of the reuse process
l forecast future reuse needs based on current
trends
The measurements needed to address these goals
fall into two classes: technical and managerial. The
technical measurements address the needs of soft-
ware developers and maintainers as they analyze a
problem and generate a software-based solution that
incorporates existing artifacts. The managerial mea-
surements address the questions of understanding,
control, and forecasting, including management of
one or more libraries of reusable components. To-
gether, the measurements allow an organization to
establish a reuse program, provide guidelines to
developers, evaluate the quality of candidate and
stored components, and assess the effectiveness of
the reuse program as a whole.
Theofanos and Pfleeger (1993) documented the
typical goals of a reuse process in the DOD. Then,
using a GQM approach, they identified likely ques-
tions to be answered and the measurements that will
aid in answering them. For example, storing and
managing components is a major goal of reuse. The
goal is viewed from five different perspectives: that
of the software developer (the staff who build and
maintain the library), project manager (the person
who oversees a particular library), program manager
(the manager of the libraries related to a single
domain), department manager (who oversees a col-
lection of distributed libraries), and corporate execu-
tive (who is interested in the costs and benefits of
reuse from a corporate perspective). Within each
perspective, questions are listed that reflect the con-
cerns of that perspective. Thus, the developer asks
questions such as
l How much effort is involved in certifying, main-
taining, and cross-referencing components?
l How much effort is involved in supplying compo-
nents to requesters?
while the project manager asks
l How efficient is the library system in terms of
speed and disk space?
How to Build a Metrics Plan
The program manager wants to know
l What is the usage profile of the components,
how can it be improved?
l What criteria are used to decide whether a com-
ponent should be removed from the library?
The department manager is interested in
l What trends of component characteristics
mote reuse across domains?
whereas the corporate executive asks
l What are the costs and benefits of reuse for this
corporation?
Each of these questions is accompanied by a list
of measurements that will help to answer the ques-
tion. For instance, to understand what trends of
component characteristics promote reuse across do-
mains, the authors recommend collecting
size attributes
quality attributes
whether or not a component is the product analy-
sis
domain name of the component
number of examinations of the component
library user
number of extractions of the component from the
library
number of uses of the component in a develop-
ment or maintenance project
number of component dependencies on other
components
names of libraries where the component resides
In all, almost 100 measurements are identified and
numbered. However, it is impractical to expect a
project to begin its reuse measurement program by
collecting all 100 measurements. Instead, it is up to
the reuse managers to decide which goals are the
most critical; then, the measurements associated with
those goals are collect first. Once the measurements
are selected, the managers must identify who will
collect them, when, and how. To do that, the man-
agers need a reuse process model.
The process model is composed of a process’s
major activities or subprocesses. In the example
described above, Theofanos and Pfleeger (1993) view
reuse in terms of
1. domain analysis: identifying areas likely to pro-
duce reusable components
2. reusable software production: creating reusable
components
J. SYSTEMS SOFTWARE 153
1995; 31:143-155
3. reusable software consumption: using reusable
and components
4. library management: assessing components and
maintaining them in a repository
Then, each of the four activities is modeled as an
SADT diagram. The process for an activity is exam-
ined in light of each recommended measurement.
pro- The model is annotated to show where in the pro-
cess the measurement would be taken. An example
is shown in Figure 6, in which the library manage-
ment subprocess includes references to the number
of each measurement that will be made as part of
that process. The actual metrics plan requires addi-
tional information, including who will collect the
data, in what way, and how often. This information
can be captured in the process model directly or in
accompanying documentation. In total, the measure-
ment plan and process model(s) present a complete
picture of measurement for the reuse process. The
project manager can use the goals and questions to
decide what to measure first. Then, the metrics
program can be expanded to include additional mea-
surements as time and need permit. Thus, the met-
rics plan can include a description of the starting
point for the metrics program as well as the scheme
by a
for expanding to other measurements.
SUMMARY AND CONCLUSIONS
This article has described a combinational approach
to measurement plans and programs. By blending a
top-down analysis of goals to be met and questions
to be answered with a process maturity framework
that allows more to be measured as the process
elements are made more visible, the approach allows
a project to start small (with key goals) and grow
a metrics program as time and resources permit.
Moreover, goals allow the metrics to grow broad
across the project, and process maturity aliows the
metrics to grow deep. That is, the process maturity
suggests ways to enrich the meaning and utility of a
metric with the additional information that becomes
visible as the process matures. In this way, the
framework supports both initial metrics selection
and a migration path to a larger, more comprehen-
sive measurement and process improvement pro-
gram.
Unlike previous approaches to measurement, the
approach described here uses a process model and
multiple viewpoints to help ensure that the metrics
program will be practical. The modeling forces the
project manager ‘r) understand the component activ-
ities of the development or maintenance process and
154 J. SYSTEMS SOFTWARE
1995; 31:143-l%
I Excluded
components
Figure 6. Library management process, annotated with
metrics references.
thereby learn where measurement will fit in. The
model also helps managers to identify who will col-
lect and analyze the measurements. Then, the multi-
ple viewpoints ensure that all who need to under-
stand the project will be able to do so. Too often, a
measurement plan explains what to measure but not
where, when, and by whom, and the measurements
are useful only to some of the interested parties.
The synthesis of techniques described here is not
just theoretical; it has been used repeatedly and
successfully in major organizations. For example, the
Metrics Advisor tool (Heimberger, 1994) and the
AM1 project (Rowe, 1994) base their approaches on
combining GQM with process maturity; there are
hundreds of members of the AMI user group alone
who find the approach appealing. The hard work of
deriving a goals tree, examining maturity, and devel-
oping a process model is rewarded when the result is
a measurement program tailored to project needs
and clearly mapped to day-to-day project activities.
The examples given here confirm what has been
reported in the literature (Rifkin and Cox, 1991);
Pfleeger, 1993; Fenton, 191; Grady and Caswell,
19871, namely, that
Measurements must be tied to the needs of the
organization that will collect and use them.
Measurement programs should start small and
grow as their needs change and resources permit.
Measurements are closely related to what is visi-
S. L. Pfleeger
I component to requester Candidates
ble in the process, and they can be used to make
aspects of the process more visible.
l Measurement is a necessary part of process im-
provement.
REFERENCES
Basili, V. R., and Rombach, H. D., The TAME Project:
Towards Improvement-Oriented Software Environ-
ments, IEEE Trans. Software Eng. 14,758-773 (1988).
Conte, S. D., Dunsmore, H. E., and Shen, V. Y., Software
Engineering Metrics and Models, Benjamin-Cummings,
Menlo Park, California, 1986.
Fenton, N., Software Metrics: A Rigorous Approach, Chap-
man and Hall, London, 1991.
Grady, R., and Caswell, D., SojIware Metrics: Establishing
a Company-Wide Program, Prentice-Hall, Englewood
Cliffs, New Jersey, 1987.
Humphrey, W., Managing the Sofhyare Process, Addison-
Wesley, Reading, Massachusetts, 1989.
McGowan, C. L., and Bohner, S., Model-based process
assessment, in Proceedings of the 15th International Con-
ference on Sofhvare Engineering, IEEE Computer Soci-
ety, 1993.
Paulk, M. C., Curtis, B., Chrissis, M. B., Averill, E. L.,
Bamberger, J., Kasse, T. C., Konrad, M., Perdue, J. R.,
Weber, C. V., Witney, J. V., Capability Maturity Model
for Software, SE1 Technical Report SEI-CMU-91-TR-
24, Software Engineering Institute, Pittsburgh, Pennsyl-
vania, 1991.
Paulk, M. C., Curtis, B., Chrissis, M. B., Weber, C. V.,
Capability Maturity Model for Software, Version 1.1,
How to Build a Metrics Plan
SE1 Technical Report SEI-CMU-93-TR-24, Software
Engineering Institute, Pittsburgh, Pennsylvania, 1993a.
Paulk, M. C., Curtis, B., Chrissis, M. B., Weber, C. V., Key
Practices of the Capability Maturity Model for Soft-
ware, Version 1.1, SE1 Technical Report SEI-CMU-93-
TR-25, Software Engineering Institute, Pittsburgh,
Pennsylvania, 1993b.
Pfleeger, S. L., Sofiware Engineering: The Production of
Qualily Software, 2nd edition, Macmillian, New York,
1991.
Pfleeger, S. L., Lessons Learned in Building a Corporate
Metrics Program, IEEE Sofbvare lo(3) (1993).
J. SYSTEMS SOFTWARE 155
1995; 31:143-155
Pfleeger, S. L., and McGowan, C. L., Software Metrics in a
Process Maturity Framework, J. Syst. Software 12,
255-261 (1990).
Rifkin, S., and Cox, C., Measurement in Practice, SE1
Technical Report SEI-CMU-91-TR-16, Software Engi-
neering Institute, Pittsburgh, Pennsylvania, 1991.
Theofanos, M. F., and Pfleeger, S. L., A Framework for
Creating a Reuse Measurement Plan, Technical Report,
Martin Marietta Energy Systems-Data Systems Re-
search and Development Division, Oak Ridge, Ten-
nessee, 1993.