See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/255607711

Software Testing Product Metrics - A Survey

Article · January 2007

CITATIONS READS
5 288

3 authors, including:

Arvinder Kaur Bharti Suri

Guru Gobind Singh Indraprastha University Guru Gobind Singh Indraprastha University
58 PUBLICATIONS   1,051 CITATIONS    56 PUBLICATIONS   322 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Regression testing View project

Software Design Defects View project

All content following this page was uploaded by Bharti Suri on 22 July 2014.

The user has requested enhancement of the downloaded file.

 Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.

Software Testing Product Metrics - A Survey

Dr. Arvinder Kaur1,#Mrs. Bharti Suri2,#Ms. Abhilasha Sharma3
1
Reader, University School of Information Technology,
GGS Indraprastha University, Kashmere Gate, Delhi, arvinderkaurtakkar@yahoo.com
2
Lecturer, University School of Information Technology,
GGS Indraprastha University, Kashmere Gate, Delhi, bhartisuri@gmail.com
3
M.Tech Fellow (Gate scholar), University School of Information Technology,
GGS Indraprastha University, Kashmere Gate, Delhi, abhilasha_sharma87@yahoo.com

Abstract 2001; N.Nagappan[10] in 2005 proposed metrics that deals with

Metrics are gaining importance and acceptance in corporate
sectors as organizations grow, mature and strive to improve
enterprise qualities. Measurement of a test process is a
required competence for an effective software test manager for
designing and evaluating a cost effective test strategy. A
number of metrics are proposed by various researchers in the
area of software testing. They enable us to detect trends and
anticipate problems with their probable solutions in effective
cost control, quality improvements, time and risk reduction.
Thus, facilitates to ensure and achieve optimal business
objectives in global competitive market.
This paper, focusing on Software Testing Product
Metrics, surveys, classifies and also systematically analyzes the
various proposed metrics during the last decades by the
researchers. Advantages or disadvantages for each product
metrics along with its need/purpose, suitability, effect, data
calibration and interpretation etc., have also been discussed in
the present paper.
Keywords – Software Testing, Software Testing Metrics,
Software Testing Product Metrics.
1. INTRODUCTION
In recent years software testing technologies have emerged as a
dominant software engineering practice which helps in effective
cost control, quality improvements, time and risk reduction etc.
The growth of testing practices has required software testers to
find new ways for estimating their projects. A key research area
in this field has been ‘measurement of and metrics for’ the
software testing. Measurement since plays a critical role in
effective and efficient software development, making
measurements of the software development and test process is
very complex [2]. Researches therefore, in proposing, applying,
validity and extending metrics for software testing has reached
a critical mass – as also evident from various publications,
presentations, products[2][9][16]. However, without the benefit
of a birds-eye perspective, researchers in different groups were
continued to discover and propose metrics. For instance,
Norman [15] in 1994; Chen ET. Al [2] in 2004 proposed
metrics that deal with product level whereas Kan et. Al [7] in
process level in software testing.
The aim of this paper, therefore, is to survey
categorically on proposed product metrics for software testing
so as to understand the researches conducted so far in this area
which may be overviewed in the Section 2 of this paper.
Section 3 deals with the perspective on software testing and
demonstrates the need and importance of measurement and
usefulness of software metrics whereas Section 4 identifies the
requirement of metrics in software testing and emphasize,
classify systematically the various product level metrics in
software testing. The complete set of metrics is captured from a
survey of the literature. Conclusions drawn from literature
survey, classification and their analysis are being highlightened
in the Section 5 of this paper
2. OVERVIEW OF RELATED RESEARCH AND
PROJECT
A number of useful related measurements of testing product
level metrics have been reported in the literature.
George, Robert and Tammy [17] presented a set of metrics
during the testing of NASA’s MCCU. Their usage of the metric
set throughout the testing effort leads to identify risks and
problems early in the test process, minimizing the impact of
problems. Further, by having a metric set with good coverage,
managers were provided with more insight into the causes of
problems, improving the effectiveness of response.
Rosenberg and Huffman [14] use an automated tool to track
requirements and their test cases has opened the door to the use
of new requirements and testing measures. The tool can be used
to point out requirements that may be subject to testing
problems. Based on their research work, they conclude that
metrics are available in the requirements phase to assess test
plans. Their work discusses the efforts to evaluate testing for
the requirements.
Norman emphasize over a unified product evaluation [15]. His
conclusion was based on both predictive and retrospective use
of reliability, risk and test metrics, that is feasible to measure
and assess product quality.

1
Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.
Yanping Chen, Robert and Robenson[2] presented a real life
project conducted by the IBM ECD test team to evaluate a set
of metrics. They analyze the effectiveness of a set of
complementary metrics for the cost, time and quality to
measure the quality of test process based on the results of the
project. Their future work includes the evaluation of new
metrics to measure performance of individual test phases,
giving suggestions to test teams and support teams for
necessary changes in the test process, and implementing the
whole set of metrics in a production test environment.
3. SOFTWARE TESTING AND SOFTWARE
METRICS
A. Software testing Guidelines
Software testing is the technical kernel of software quality
engineering and to develop critical and complex software
systems which not only requires a complete, consistent and
unambiguous design, and implementation methods, but also a
suitable testing environment that meets certain requirements to
face the complexity issues [3].Traditional methods are not
effective in understanding the software’s overall complex
behavior as they are time consuming and insufficient for the
dynamism of modern business. But today’s innovative software
testing technologies enable the creation of more so
sophisticated quality assurance.
We create tests, run them and report the progress and results of
our work. New version of the product comes, we might reuse
some old tests, check whether bugs have been fixed, and (or)
try new tests. Eventually, the product is either cancelled or put
into production.
Testing is the process of executing a program with the intention
of finding faults [1]. Manages traditionally have evaluated test
progress using programmer and test team intuitions [14]. The
IEEE/ANSI defines testing as: “The process of operating a
system or component under specified conditions, observing or
recording the result and making an evaluation of some aspect of
the system or component” and as “the process of analyzing a
Software item to detect the difference between existing and
required conditions and to evaluate the features of the software
items”.
The question arises: Why do we test? The two main reasons
are: to make a judgment about quality or acceptability and to
discover problems [6]. Although it is an expensive activity, yet
launching of software without testing may lead to cost
potentially much higher than that of testing [1]. The primary
advantage of testing is that the software being developed can be
executed in its appropriate environment and the results of theses
executions with the test cases provide confidence that the
2
software will perform as intended and must satisfy the user’s
requirements [12].
B. Measurement –Need and Importance
An eminent physicist, Lord Kelvin said “when you can measure
what you are speaking about, and can express it in numbers,
you know something about it; but when you can not measure it,
when you can not express it in numbers, your knowledge is of
meagre and unsatisfactory kind; it may be the beginning of
knowledge, but you have scarcely in your thoughts advanced to
the stage of science”. Hence, it is clear that everything should
be measurable. If it is not measurable, we should make an effort
to make it measurable [1].
Measurement lies at the heart of many systems that govern our
lives. It plays a critical role on effective software development
and provides the scientific basis for software testing.
Measurement could be defined as the process of empirical,
objective, assignment of numbers to properties of objects or
events of the real world in such a way as to describe them [5]. It
is the process by which numbers or symbols are assigned to
attributes of entities in the real world in such a way as to
characterize them according to clearly defined rules [4].
Fundamental measurement is a means by which numbers can be
assigned according to natural laws to represent the property,
and yet does not presuppose measurement of any other
variables than the one being measured [18]. An effective
measurement activity should be able to evaluate the current
process and provide suggestion to the manager for future
improvement [2].
C. Software Metrics and their Usefulness:
In applying measurements to software engineering, several
types of metrics are available called software metrics. Metric
can be defined as “The continuous application of measurement
based techniques to the software development process and its
products to supply meaningful and timely management
information, together with the use of those techniques to
improve that process and its products” [1].
Schulmeyer defines a metric as “A quantitative measure of the
degree to which a systems, component or process possesses a
given attribute” [16].
Software metrics are used to evaluate the software
development process and the quality of the resulting product
[17]. Software metrics aid evaluation of the testing process and
the software product by providing objective criterion and
measurements for management decision making. Their
association with early detections and correction of problems
make them important in software. Software metrics are all
about measurements which, in turn, involve numbers, the use of
numbers to make things better, to improve the process of
developing software and to improve all aspects of the
management of that process.
Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.
Software metrics are used to provide quantitative checks on Test metrics are powerful risk management tool, help us to
software design. They include information about productivity, measure current performance. Importance of having test metrics
quality and product or process effectiveness. Use of software [13]:
metrics does not ensure survival, but it improves the probability (i) Provides a basis for estimation and facilitates planning
of survival. for closure of the performance gap.
(ii) Provides a means for control/status reporting.
Table 1: Classification of metrics (iii) Identify risk areas that require more testing.
Category Description Example (iv) Quickly identifies and helps to resolve potential
Product Describe the Size, problems and identifies areas of improvement.
Metrics characteristics Performance, (v) Test metrics provide an objective measure of the
of the project Efficiency etc. effectiveness and efficiency of testing.
Process Describe the Effort, Time,
A. Software Testing Product Metrics
Metrics effectiveness No. of defects
Software testing product metrics are used for measurement of
and quality of found during
testing parameters at product level. As per the literature survey
the process that testing
[2][7][8][10][13]and in general, the criteria for the
produce the
classification of testing product metrics are as follows:
software
(i) High quality product
product
(ii) Time-to-market
Project Describe the No. of software (iii) Cost-to-market
Metrics project’s developers, (iv) Parameters for specific test phase
characteristics Cost, Schedule (v) Test session efficiency and
and execution (vi) Test Focus
The salient features for each type of the metrics based on the
Some metrics belong to multiple categories like quality metric above criteria are also brief in Table 2.
may belong to all three categories.
5. CONCLUSION
4. SOFTWARE TESTING METRICS
We have presented a survey of existing metrics proposed for
Software metrics are applicable to the whole development life software testing focusing on product metrics. In particular, we
cycle from initiation, when cost must be estimated to have analyzed the metrics with respect to the coverage of
monitoring the reliability of the end product in the field, and the effective cost control, quality improvement risk and time
may that product changes over time with enhancement. reduction.
Software metrics hold importance in testing phase, as software The survey does not concentrate over the specific concerns of
testing metrics acts as indicators of software quality and fault the individual metric as one metric is not sufficient to gain in
proneness. Testing metrics exhibit trends and characteristics the quintessence of testing efficiency. Hence, the survey results
over time that would be indicative of the stability of the process indicate that carefully chosen testing metric suite can be very
[11]. beneficial in software product measurement and can provide
The essential step is establishing test metrics is to identify the useful information to test managers for decision making and
key software testing processes that can be objectively future improvements.
measured. The survey provides a historical view and uncovers gaps in
A. Test Metrics – Need and Importance existing research. The presented literature is based on a uniform
A major percentage of software projects suffer from quality representation of existing metrics and is aided by storing the
problems, which in turn requires new testing metrics to measure information in a structured repository for ease of processing.
test processes effectively. Test metrics are key “facts” that
project managers can use:
(i) to understand their current position
(ii) to prioritize their activities to reduce the risk of
schedule over-runs on software releases.

3
Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.
Table 2: Comparison of Software Testing Product Metrics
Year and Test Metric Definition/ Purpose Formula Effect
Size of
project
2004 Quality code It captures the relation between the WTP+WF/KCSI The lower the value of QC, indicating fewer
(QC) number of weighted defects and the defects or less serious defects found, the higher
size of product release. Purpose is to the quality of code delivered.
deliver a high quality product.
2004 Quality of the It shows the relation between the WF/KCSI A low number of QP indicates fewer defects, or
Product (QP) number of weighted defects shipped to less serious defects were shipped, implying a
customers and size of the product higher quality of the code delivered by the test
release. Purpose is to deliver a high teams.
quality product.
2004 Test It shows the relation between the WTTP/KCSI The higher the number, indicating more defects
Improvement number of weighted defects detected or more imp. Defects were detected the higher
(TI) by the test team during and the size of the improvement to the quality of the product
the product release. Purpose is to which can be attributed to the test teams.
deliver a high quality product.
2004 Test It shows the relation between the WT/(WTP+WF)*10 The higher the TE, indicating a higher ratio of
effectiveness number of weighted defects detected 0% defects or imp. Defects were detected before
(TE) during testing and the total number of release, the higher is the effectiveness of the test
weighted defects in the product. organization to drive out defects.
Purpose is to deliver a high quality
product.
2004 Test time (TT) It shows the relation between tie spent TT/KCSI The lower this number, the less time required by
on testing and the size of the product the test teams to test the product.
release. Purpose is to decrease time-
to-market.
2004 Test time over It shows the relation between time TT/TD*100% The lower this number, the lower is the amount
development spent on testing and the time spent on of time required by the test teams to test the
time(TD) developing. Purpose is to decrease product compared to the development team.
time-to-market.
2004 Test cost It shows the relation between resource CT/KCSI The lower this number, the lower is the cost
normalized to or money spent on testing and the size required to test each thousand lines of code.
product size of the product release. Purpose is to
(TCS) decrease cost-to-market.
2004 Test cost as a It shows the relation between testing CT/CD*100% The lower this number, the lower is the cost
ratio of cost and development cost of the required by the test teams to test the product
development product. Purpose is to decrease cost- compared to development team.
cost (TCD) to-market.
2004 Cost per It shows the relation between money CT/WT The lower this number, the lower is the cost of
weighted defect spent by test team and the number of finding one defect unit, and the more cost-
unit (CWD) weighted defects detected during effective is the test process.
testing. Purpose is to decrease cost-to-
market.
2004 Test It shows the relation between thi WP/KCSI The higher this number, the higher is the
improvement in number of weighted defedts detected improvement of the quality of the product
product quality. and the size of the product release. contributed during this test phase.
2004 Test time It shows the relation between time TTP/KCSI The lower this number, the less time required
needed spent on testing and the size of the for the test phase relatively.
normalized to product release.
size of product.
2004 Test cost It shows the relation between resource CTP/KCSI The lower this number, the lower is the cost
normalized to or money spent on the test phase and required to test each thousand lines of code in
size of product. the size of the product release. the test phase.
2004 Cost per It shows the relation between money CTP/WT The lower this number, the lower is the cost of
weighted defect spent on the test phase and the number finding one defect unit in the test phase, and the
unit. of weighted defects detected. more cost-effective is this test phase.
2004 Test It shows the relation between the WD/(WD+WN)*10 The higher this number, indicating a higher ratio
effectiveness number of one type of defects 0% of defects or important defects was detected in
for driving out detected in one specific test phase and the “appropriate” test phase, the higher is the
defects in each the total number of this type of defect effectiveness of this test phase to drive out its
test phase. in the product. target type of defects.
815,362 Software size Measured by a count of source lines of __ Large increases in software size late in the
SLOC of code (SLOC). Goal is to show the development cycle often result in increased

4
Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.
NASA’s risks to the system over time. testing and maintenance activities.
MCCU
project,
1992
815,362 Software It is the probability of failure free Z(t) = (h)exp(- The decreasing failure rate represents the
SLOC of reliability operation of a computer program for a ht/N) growth in the reliability of software.
NASA’s specified time in a specified
MCCU environment.
project,
1992
815,362 Test session Goal of the test session efficiency SYSE= Both should be greater than 80%.
SLOC of efficiency metric is to identify trends in the Active Test
NASA’s scheduled test time’s effectiveness Time/Scheduled
MCCU Test Time
project,
1992 TE= Total no. of
good runs/Total
runs
815,362 Test focus Goal is to identify the amount of effort TF = No. of DRs In an ideal case TF approaches unity as testing
SLOC of spent finding and fixing real faults closed with a proceeds.
NASA’s versus the effort spent either software fix/Total
MCCU eliminating false defects or waiting for no. of DR
project, a hardware fix.
1992
815,362 Software Goal is to quantify the relative T = Total no. of Graph of T versus H should begin with a near
SLOC of maturity stabilization of a software subsystem DRs changed to a infinite slope and approach a zero slope.
NASA’s (ii) To identify any possible over- subsystem/1000 Otherwise a low quality subsystem is indicated
MCCU testing or testing bottlenecks by SLOC and should be investigated. Graph of O versus H
project, examining the fault density of the is an indication of low rapidly faults are being
1992 subsystem over time. Three O = No. of fixed. It should begin with a positive slope, then
components are: T, O, H. currently open as debuggers begin to correct the faults, the
subsystem slope should become negative
DRs/1000 SLOC

H = Active test
hours per
subsystem/1000
SLOC
Subprogram Goal is to identify the complexity of % if functions A positive trend indicates the needs to re-
complexity each function and to track the progress with a complexity evaluate the software change philosophy,
of function with a relatively high greater than a possibly resulting in some re-design.
complexity as they represent the recommended A negative trend indicates the changed function
highest risk. threshold. had been redesigned to reduce complexity and
increase maintainability.
Test coverage Goal of the metric is to examine the % of code
efficiency of testing over time. branches that have __
been executed
during testing
Computer Goal is to estimate the utilized
resource capacity of the system prior to __ __
utilization operations to ensure that sufficient
resources exist.

WTP – no. of weighted defects found in the product under test
(before official release)
WF – no. of weighted defects found in the product after release.
KCSI – no. of new or changed source lines of code in
thousands.
WTTP – is the no. of weighted defects found by the test team in
the test cycle of the product.
WT – no. of weighted defects found by the test team during the
product cycle.
TT – no. of business days used for product testing.
TD – no. of business days used for product development.
CT – total cost of testing the product in dollars.
CD – total cost of developing the product in dollars.
WP – no. of weighted defects found in one specific test phase.
TTP – no. of business days used for a specific test phase.
CTP – total cost of a specific test phase in dollars.
WD – no. of weighted defects of this defect type that are
detected after the test phase.
WN – no. of weighted defects of this defect type (any particular
type) that remain uncovered after the test phase (missed defects)

5
Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007)
RIMT-IET, Mandi Gobindgarh. March 23, 2007.
Z(t) – instantaneous failure rate.
h – Failure rate prior to the start of testing
N – no. of faults inherent in the program prior to the start of
testing.
SYSE – System Efficiency
TE – Tester efficiency
DR – Discrepancy Report
TF – Test Focus
T – Total Density
O – Open Density
H – Test Hours
REFERENCES
[1] K. K. Aggarwal & Yogesh Singh “Software
Engineering Programs Documentation Operating
Procedures (Second Edition)” New Age International
Publishers, 2005.
[2] Yanping Chen, Robert L. Probert, Kyle Robenson
“Effective Test Metrics for Test Strategy Evolution”
Proceedings of the 2004 Conference of the centre for
Advanced Studies on Collaborative Research
CASCON’04
[3] P. Dhavachelvan, G. V. Uma, V.S.K.
Venkatachalapathy “A new approach in development
of distributed framework for automated software
testing using agents”
[4] N. E. Fenton and S. L. Pfleager “Software Metrics: A
Rigorous and Practical Approach”, Second Edition
Revised ed. Boston: PWS Publishing, 1997.
[5] L. Finkelstein “Theory and Philosophy of
Measurement” , in Theoretical Fundamentals, Vol. 1,
Handbook of measurement Science, P.H. Sydenlam,
Ed. Chichester : John Wiley & Sons, 1982, pp. 1-30.
[6] Paul C. Jorgensen “Software Testing - A Craftsman’s
Approach Second Edition” CRC Press, 2002.
View publication stats
[7] S.H. Kan, J. Parrish and D.Manclove “In-process
metrics for software testing” IBM Systems Journal,
Vol. 40. no. 1,2001.
[8] Stephen H. Kan “Metrics and Models in Software
Quality Engineering”, Second Edition, 2002.
[9] Cem Kaner “Software Engineering Metrics: What do
they measure and how do we know?” 10th International
Software Metrics Symposium, Metrics 2004.
[10] N.Nagappan “Toward a software Testing and
Reliability Early warning Metric Suite” Proceedings of
the 26th International Conference on software
engineering (ICSE’04), 2004.
[11] Hideto Ogasawara, Atsushi Yamada, Michiko Kojo
“Experiences of software Quality Management Using
Metrics through Life cycle”, Proceedings of ICSE-18,
1996..
[12] E. Osterweil “Strategic Directions in Software
Quality”, ACM Computing Surveys 4, 1996, pp 738-
750.
[13] Ramesh Pusala “Operational Excellence through
efficient Software Testing Metrics” Infosys, 2006.
[14] Linda H. Rosenberg, Theodore F. Hammer, Lenore L.
Huffman “Requirements, Testing, and Metrics” NASA
GSFC, 1998.
[15] Norman F. Schneidewind “Measuring and Evaluating
Maintenance Process Using Reliability, Risk, and Test
Metrics” IEEE Transaction on Software Engineering,
Vol. 25, No. 6, 1999.
[16] G.Gordon Schulmeyer, James I. Mcmanns, “Handbook
of Software Quality assurance, 3rd edition, Prentice
Hall PTR, Upper Saddle River, NJ, 1998.
[17] George E. Stark, Robert C. Durst, Tammy M. Pelnik
“An Evaluation of Software Testing metrics for
NASA’s Mission Control Center” 1992.
[18] W. Torgerson, S., “Theory and Methods of Scaling”
New York: John Wiley & Sons, 1958.