Software Metrics for Fault Prediction Using Machine

Learning Approaches
A Literature Review with PROMISE Repository Dataset

Meiliana1, Syaeful Karim2 Harco Leslie Hendric Spits Warnars1,

1,2
Computer Science Department, School Ford Lumban Gaol2, Edi Abdurachman3
of Computer Science Computer Science Department, BINUS Benfano Soewito
1
Computer Science Department, BINUS Graduate Program - Doctor of Computer Computer Science Department, BINUS
Graduate Program - Doctor of Computer Science Graduate Program – Master of Computer
Science Bina Nusantara University Science,
1,2
Bina Nusantara University Jakarta, Indonesia 11480 Bina Nusantara University
1
Jakarta, Indonesia 11480 shendric@binus.edu, 2fgaol@binus.edu, Jakarta, Indonesia 11480
1
meiliana@binus.edu, 2karim@binus.ac.id 3
edia@binus.edu bsoewito@binus.edu

Abstract— Software testing is an important and critical phase
of software development life cycle to find software faults or
defects and then correct those faults. However, testing process is
a time-consuming activity that requires good planning and a lot
of resources. Therefore, technique and methodology for
predicting the testing effort is important process prior the testing
process to significantly increase efficiency of time, effort and cost
usage. Correspond to software metric usage for measuring
software quality, software metric can be used to identify the
faulty modules in software. Furthermore, implementing machine
learning technique will allow computer to “learn” and able to
predict the fault prone modules. Research in this field has
become a hot issue for more than ten years ago. However,
considering the high importance of software quality with support
of machine learning methods development, this research area is
still being highlighted until this year. In this paper, a survey of
various software metric used for predicting software fault by
using machine learning algorithm is examined. According to our
review, this is the first study of software fault prediction that
focuses to PROMISE repository dataset usage. Some conducted
experiments from PROMISE repository dataset are compared to
contribute a consensus on what constitute effective software
metrics and machine learning method in software fault
prediction.
Keywords—software metric; machine learning algorithm;
software defect prediction; software testing
I. INTRODUCTION
Software testing is an important phase of software
development life cycle that verify and validate software
quality. One way to monitor software quality is to find
software faults or defects and then correct those faults.
However, testing process is a time-consuming activity that
makes it impossible to find all faults within given resources.
Therefore, methodologies and techniques for predicting the
testing effort is important process prior the testing process to
significantly increase efficiency of time, effort and cost usage.
Correspond to software metric usage for measuring
software quality, software metric can be used to identify the
faulty modules in software. Deriving a fault prediction model
from software metrics provides a prediction in the early phase
of software development process. Thus, software construction
and testing can be focused on the fault prone module to
improve the quality. The most widely used methods are
statistical and machine learning, or the combination between
both methods. But in this paper, we will focus on machine
learning implementation to predict the model. Applying
machine learning technique will allow computer to “learn”
and able to predict the fault prone modules. Significant
development of machine learning methods on the last decades
contributes significant research in this field, such as Least
Squares Support Vector Machine (LSSVM) [1] or
Conventional radial basis function neural network (RBFNN)
[2].
Public dataset such as Open Source “Apache POI” [3],
Rhino (an open-source implementation of JavaScript written
in Java) [4] are used as experiment data to examine the model.
Therefore, most dataset used in this field are NASA dataset
from PROMISE repository that will be a focus on this review.
PROMISE repository is design for predictive models and data
analytics in software engineering [5]. The rest of this paper are
organized as follows. The second section will discuss about
software metrics and possibilities to use software metrics for
fault prediction. Subsequent section describes some machine
learning approaches used to predict software faults. The 4th
section presents several conducted researches regarding
software fault prediction based on software metrics using
machine learning approaches. Conclusions and future
development gaps are given in the last section.
II. STATE OF THE ART
A. Software Metrics
Software metrics is a quantitative measurement that
assigns numbers or symbols to attributes of the measured
entity [6]. An attribute is a property/feature of an entity that
can be measured, e. g. length, age and cost. An entity can be
source code of application or a software development process
activity. Measures obtained from entities which have been
associated with risk factor can be used to create defect

19
prediction models. There are many types of available software
metrics for different purposes that will be discussed on the
next section.
1) Software Metric Classification
Software metrics can be classified static code metrics and
process metrics. Static code metrics can be directly extracted
from source code, such as Lines of Code (LOC), Cyclomatic
Complexity Number (CCN), or other static code metrics that
include compiler instruction and data declaration counts.
Object oriented metrics is a subcategory of static code metrics,
such as Depth of Inheritance Tree (DIT), Coupling Between
Objects (CBO), Number of Children (NOC), and Response for
Class (RFC). Aggarwal et al [7] observed that import coupling
metrics are strongly associated with fault proneness and
predict faulty classes with high accuracy in object oriented
system.
Process metrics can be extracted from Source Code
Management system based on historic changes on source code
overtime. Moser et al [8] concluded that change (process)
metrics are more efficient than static code metrics for defect
prediction because process data contains more discriminatory
information about defect distribution than the source code
itself. Other research by Nagappan et al [9] used metrics
concerning code churn, a measure of changes in source code
over a specified period of time. They conclude that the use of
relative code churn metrics predict the defect per source file
better than other metrics and that code churn can be used to
discriminate between faulty and non-faulty files.
Metrics can also be classified based on development phase
of software life cycle, into source code level metrics, detailed
design level metrics or test level metrics.
2) Minimum Number of Software Metrics for Fault
Prediction Model
Excessive software metrics during software development
life cycle often collected for different reasons and stored in
software project repositories. Therefore, selection of software
metrics is another critical point to build a software defect
prediction model. Empirical study conducted by [10]
investigated five threshold-based feature selection techniques
to remove irrelevant and redundant software metrics. The
experiments demonstration able to remove 98.5% available
software metrics to construct a model of defect prediction with
better performance. The main conclusion of that study is that
three combination of software metrics are sufficient. On
average, this number will create effective software prediction
models. This number can be used as a reference for selecting
minimum number of software metrics with regard to fault
prediction
B. Machine Learning Approaches for Fault Prediction
Software fault prediction is the process to predict
software’s parts that are fault prone. By focusing on fault
prone files that are detected, testers can save testing effort. As
mentioned before, it is important to find software fault/defect
as early as possible to reduce testing cost. Therefore, using
software metric that is produced on the early stage of software
development process is a superiority of this research. The idea
behind software fault prediction is to use measurement
extracted from development process e.g. source code that can
be found in software metrics. Many approaches have been
studied to conduct software fault prediction, start with simple
equation, statistical analysis, expert estimation, and machine
learning. Among all approaches, machine learning has been
proved as the most successful approaches based on research of
[11].
To predict the fault proneness module automatically, a
large variety of machine learning algorithms is used. Machine
learning algorithm is known as classification. Classification
procedure basically means a class labelling to a new sample
based on a given set of samples that have been labeled.
Several machine learning algorithms will be discussed in this
subsection.
Support Vector Machine is a machine learning technique
for classifying unseen data correctly by constructing N-
dimensional hyperplane. Finding optimal hyperplane that
separate cluster from vector is the main task of Support Vector
Machine modeling. Therefore, cases with one category of
dependent variable will separate side by side from cases with
another category of the plane. The term support vectors refer
to all vectors near hyperplane. SVM modelling finds oriented
hyperplane to maximize margin between the support vectors.
SVM able to handle nonlinear separator between points by
using kernel function to map the data into different space, and
then hyperplane will be used to separate the space.
Random forest is another machine learning type that
consist of many classification trees. The classification trees are
decision trees that represents major advance in knowledge
discovery and data mining. Random forest classifier offers
prediction with high level accuracy. The classification
algorithm classifies a new object from an input vector. The
input vector is put down to each tree in the forest, where each
tree gives a classification result. The tree votes for that class,
then the forest will choose most votes of classification.
The next machine learning method is bagging algorithm or
known as bootstrap aggregating. Bootstrap aggregating is a
technique that will repeatedly samples from dataset based on a
uniform probability distribution. The size of each sample of
bootstrap and the original data is same. It is possible for an
instance to occur several times during the same training set
because the sampling is done with replacement. On the other
hand, other instance could be omitted from the training set.
Each sample has probability of 1-(1-1/N)N of being selected.
Therefore, a bootstrap sample may contain around 63%
original training data in sufficiently large data of N. An
instance will be assigned to a class after training K classifiers
if the class receives the highest number of votes.
Artificial Neural Network is being popular lately, such as
Multilayer Perceptron (MLP). MLP could be used to solve
almost any problems, such as pattern recognition,
interpolation, etc. Multilayer Perceptron contains of one or
two hidden layers as advancement to the perceptron neural
network model. Feedforward neural networks is trained with
back propagation algorithm that consists of two passes
(forward and backward pass). Forward pass presented an input
20
to the neural network that will propagated through the
network.
An instance-based classifier, K-star, used similarity
function to determine class of test instance according to the
class that has similar instance. By using an entropic measure,
this method retrieves the nearest stored example.
Naive Bayesian is a classification method that quite
efficient and easy to implement. Most used and well-known
classifier method that need smaller data quantity to estimate
the parameters. The basic algorithm is a probabilistic
according to theorem of Bayes, that assumes class feature
existence does not depend on another feature existence.
C. PROMISE repository dataset
PROMISE repository aims to provide verifiable and
repeatable models that can be used for software engineering
research. It’s an online repository to share dataset among
software engineering researchers and practitioners. A
researcher or practitioner can store their dataset or access
another model share by others. Thus, user will be able to
compare their result and make conclusion, sustain from
previous works [5]. PROMISE was started to as a research to
predict effort and defect of a software and then developed to
be model based requirements engineering and value based
software engineering. The repository contains a lot of dataset
that can be refute, confirm, repeat, or improve from previous
result.
III. PREVIOUS RESEARCH
Considerable research has been performed on software
metrics for fault prediction process using machine learning
algorithm. Most of the research used NASA MDP (Metric
Data Program) repository as the experiment dataset. Each
dataset list all discovered faults of the system and the number
of modules containing the fault, while each module is
described by a set of code-level or design level attribute.
Catal et. al [12] used algorithm of Artificial Immune
Recognition System (AIRS) to create defect model based on
method-level metrics and Chidamber-Kemerer metrics suite.
The experiments showed that AIRS algorithm obtain the best
fault prediction model by using combination of CK metrics
and Line of Code metrics. Catal et al. [13] on other research
work examined nine classifiers for each of the five public
NASA dataset. According to the research, Naïve Bayes
algorithm provides the best prediction performance for small
datasets, while Random Forest is the best prediction algorithm
for large datasets.

TABLE I. COMPARISON TABLE FOR MACHINE LEARNING ALGORITHM AND METRIC FROM SEVERAL RESEARCH

Authors Data source Metrics Algorithm/Method Result

Catal et. al 5 public NASA datasets 13 method level metrics Random Forest Naïve Bayes algorithm is better for smaller
[13] Naive Bayes dataset, while Random forest algorithm
J48 perform best for larger dataset
Immunos 1 & 2
CLONALG
AIRS1 & 2
AIRS 2 Parallel
Catal et. al 6 OO metrics of WMC (Weighted Methods per AIRS - Artificial The best fault prediction result is
[12] Chidamber-Kemerer Class) Immune Recognition combination between CK metrics and the
metrics, DIT (Depth of Inheritance Tree) System algorithm lines of code (LOC) metric
4 metrics from KC1 RFC (Response for a Class)
projects, and NOC (Number of Children)
Halstead & McCabe CBO (Coupling Between Object
metric Classes)
LCOM (Lack of Cohesion in
Methods)
Percent Pub Data
Access To Pub Data
Dep On Child
Fan In
Shanthini [14] Public domain KC1 21 method-level metrics Naïve Bayes Precision, recall and accuracy of SVM show
NASA data set, method proposed by Halstead and SVM better result compare to other machine
level and class level McCabe. K-Star learning methods for both class and method
metrics 10 class-level Random Forest level metrics.
oriented metrics are used
Mundada et al PROMISE repository: Object Oriented CK metrics Artificial Neural ANN show better accuracy compare to
[15] JM1/Software Defect Network and Resilient previous research
Prediction Back Propagation
Lines of Code (LoC) metric,
Unique Operator (UOp),
Cyclomatic Complexity
AR3, AR4, AR5 from Quad K-Means algorithm more efficient in the term
Bishnu et al. (CC), Total
Promise data Tree-Based K-Means of iterations except for AR5, and
[16] Operator (TOp), Unique
Iris dataset Clustering Algorithm the SSE.
Operand (UOpnd), Total
Operand (TOpnd)

21
 Authors Data source
An open source Eclipse
Dejaeger et al
Foundation & NASA
[17]
IV&V facility dataset
Okutan et al
Promise data repository
[18]
Kumar et al 45 real-life datasets from
[19] the PROMISE repository
four NASA datasets ( two
of them are based on SCM
Wondaferaw Object Oriented Metrics (OOM)
and the rest two
et al [20] and Static Code Metrics (SCM)
datasets are based on
OOM)
Five datasets from the
Kumudha et NASA data program
al [2] repository (CM1, JM1,
KC1, KC2, and PC1)
Gupta et al CK metrics: DIT, WMC, CBO,
PROMISE data repository
[21]
Ant, jEdit, Camel, Xalan,
Erturka et al Log4j and Lucene projects
[22] from the PROMISE
repository
Alighardashi NASA and PROMISE
et al [23] with ten datasets
Metrics
Halstead, Line Of Code, and
McCabe complexity metrics
nine data sets
from Promise data repository,
NOD for the number of
developers and LOCQ for the
source code quality
Chidamber and Kemerer Java
Metrics
NECM is employed as a key
performance metric
NOC, RFC, LCOM
CBO, WMC, and RFC
20 object oriented metrics
Algorithm/Method Result
Fifteen Bayesian Compared to common Naïve Bayes classifiet,
Network (BN) Augmented Naive Bayes classifiers show
classifiers better performance.
LOC, LOCQ, and RFC are the most effective
Bayesian networks
metrics.
MVE approach performs the best result. Fault
Majority Voting
prediction model developed using MVE
Ensemble (MVE)
method consume less fault removal cost as
method
compare to other techniques
combination of Random Forts (RF) with
8 classifiers: NB, NN,
Information Gain (IG) FS yields the highest
SVM, RF, KNN,
Receiver Operating Characteristic (ROC)
DTr, DTa, and RTr
curve value.
Conventional radial
basis function neural
Proposed ADBBO-RBFNN classifier
network and the novel
approach show more effective result compare
adaptive dimensional
to early existing predictors.
biogeography based
optimization model.
derive threshold values
The statistical comparison showed better fault
and fault classification
classification after redundancy removal and
using the derived
log-transformed data than otherwise
metrics
Fuzzy Inference
Fault prone module can be located
Systems (FISs) and
automatically with iterative software fault
Adaptive Neuro Fuzzy
prediction
Inference System are
employed
The results show the effectiveness of the used
feature selection method method. Therefore, our proposed WF method
by using combination of can find the best features with the highest
filter feature selection speed for the improvement of the fault
methods prediction accuracy. In this research, we use
the five filter methods

Shanthini [14] evaluated 21 method-level metrics and 10
class-level metrics from the same dataset (public domain KC1
NASA data set) using four different classifiers such as Naïve
Bayes, K-star, Random Forest and SVM. Based on conducted
experiments, Random Forest classifier showed better result
(precision, recall F-Measure, and accuracy) for method level
metric. The best performance for class level metrics is
produced from SVM method.
In object oriented software system, object oriented metrics
are useful in predicting the fault proneness of classes, such as
Object Oriented CK metrics and QMOOD metrics used by
Ruchika [3]. The research apply statistical method and six
machine learning methods; Random Forest, Adaboost,
Bagging, Multilayer Perceptron, SVM, Genetic Programming.
Prediction models to estimate fault proneness are validated by
using dataset of Open Source “Apache POI” (pure Java library
for manipulating Microsoft documents). The best result is
showed by random forest and bagging algorithm.
Table 1 present better comparison of all related works
discuss in this paper. The comparison table identified software
metrics used and machine learning algorithm applied for each
research with the result.
IV. CONCLUSION
In fault prediction studies, the most widely used dataset is
NASA datasets from PROMISE repository that provide many
metrics to be investigated. The usage of public dataset is
recommended to make fault prediction models repeatable,
refutable and verifiable. Moreover, the use of same public
dataset will simplify comparison study from one research to
another research. From several research observed in this
paper, software metric is proved as one efficient source to
provide fault predictive model. In despite of every type of
metric can be used to estimate fault proneness module, class
level metric show better prediction performance compare to
method level metric. Average three software metrics are
sufficient to build effective software prediction models.

22
Machine learning algorithm is applied to automatically
construct fault predictive model by classifying module into
defective and non-defective one. Based on performance
(accuracy, sensitivity, specificity, and receiver operating
characteristic analysis) that is showed from previous
experiments, SVM and Random Forest algorithm provide best
prediction model for two different datasets. These two
algorithms can be used as a recommendation machine learning
algorithm for fault prediction. Most of these studies have
proved the usage of software metric for fault prediction using
most machine learning algorithms. However, research in this
field remains open for implementation of deep learning
algorithm as part of broader family of machine learning to get
better prediction model as started by Yang et. al [24]
REFERENCES
[1] L. Kumar, S. Krishna, A. Sureka, and S. Ku, “Effective fault
prediction model developed using Least Square Support,”
vol. 0, 2017.
[2] P. Kumudha and R. Venkatesan, “Cost-Sensitive Radial
Basis Function Neural Network Classifier for Software
Defect Prediction,” vol. 2016, 2016.
[3] R. Malhotra and A. Jain, “Fault prediction using statistical
and machine learning methods for improving software
quality,” J. Inf. Process. Syst., vol. 8, no. 2, pp. 241–262,
2012.
[4] H. M. Olague, L. H. Etzkorn, S. Member, S. Gholston, and
S. Quattlebaum, “Empirical Validation of Three Software
Metrics Suites to Predict Fault-Proneness of Object-Oriented
Classes Developed Using Highly Iterative or Agile Software
Development Processes,” vol. 33, no. 6, pp. 402–419, 2007.
[5] G. Boetticher, T. Menzies, T. Ostrand, PROMISE
Repository of Empirical Software Engineering Data, West
Virginia University, Department of Computer Science,
2007.
[6] Fenton, N. E., & Pfleeger, S. L. (2009). Software metrics: a
rigorous and practical approach. Methods, 5, 26.
[7] K. K. Aggarwal, Y. Singh, A. Kaur, and R. Malhotra,
“Investigating effect of design metrics on fault proneness in
object-oriented systems,” J. Object Technol., vol. 6, no. 10,
pp. 127–141, 2007.
[8] Moser, R., Pedrycz, W., & Succi, G. (2008, May). A
comparative analysis of the efficiency of change metrics and
static code attributes for defect prediction. In 2008
ACM/IEEE 30th International Conference on Software
Engineering (pp. 181-190). IEEE.
[9] Nagappan, N., & Ball, T. (2005, May). Use of relative code
churn measures to predict system defect density. In
Proceedings. 27th International Conference on Software
Engineering, 2005. ICSE 2005. (pp. 284-292). IEEE..
[10] H. Wang, “How Many Software Metrics Should be Selected
for Defect Prediction ?,” no. Mi, pp. 69–74, 2005.
[11] Catal, C., & Diri, B. (2009). A systematic review of software
fault prediction studies. Expert systems with applications,
36(4), 7346-7354
[12] Catal, C., Diri, B., & Ozumut, B. (2007, June). An artificial
immune system approach for fault prediction in object-
oriented software. In Dependability of Computer Systems,
2007. DepCoS-RELCOMEX'07. 2nd International
Conference on (pp. 238-245). IEEE..
[13] Catal, C., & Diri, B. (2009). Investigating the effect of
dataset size, metrics sets, and feature selection techniques on
software fault prediction problem. Information Sciences,
179(8), 1040-1058.
[14] A. Shanthini, “Applying Machine Learning for Fault
Prediction Using Software Metrics,” vol. 2, no. 6, pp. 274–
278, 2012.
[15] D. Mundada, A. Murade, O. Vaidya, and J. N. Swathi,
“Software Fault Prediction Using Artificial Neural Network
And Resilient Back Propagation,” Int. J. Comput. Sci. Eng.,
vol. 5, no. 03, pp. 173–179, 2016.
[16] P. S. Bishnu. and V. Bhattacherjee, “Software Fault
Prediction Using Quad Tree-Based K-Means Clustering
Algorithm,” vol. 24, no. 6, pp. 1146–1150, 2012.
[17] K. Dejaeger, T. Verbraken, and B. Baesens, “Prediction
Models Using Bayesian Network Classifiers,” vol. 39, no. 2,
pp. 237–257, 2013.
[18] A. Okutan and O. Taner, “Software defect prediction using
Bayesian networks,” no. 2, pp. 154–181, 2014.
[19] Kumar, L., Rath, S., & Sureka, A. (2017). Using Source
Code Metrics and Ensemble Methods for Fault Proneness
Prediction. arXiv preprint arXiv:1704.04383.
[20] C. W. Yohannese and T. Li, “A Combined-Learning Based
Framework for Improved Software Fault Prediction,” vol.
10, pp. 647–662, 2017.
[21] S. Gupta and D. L. Gupta, “Fault Prediction using Metric
Threshold Value of Object Oriented Systems,” vol. 7, no. 6,
pp. 13629–13643, 2017.
[22] E. Erturk and E. Akcapinar, “Iterative software fault
prediction with a hybrid approach,” Appl. Soft Comput. J.,
vol. 49, pp. 1020–1033, 2016.
[23] F. Alighardashi, M. Ali, and Z. Chahooki, “The
Effectiveness of the Fused Weighted Filter Feature Selection
Method to Improve Software Fault Prediction,” no. 8, 2016.
[24] X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, “Deep
Learning for Just-In-Time Defect Prediction,” no. 1.
23