The Journal of Systems and Software 80 (2007) 1349–1361

www.elsevier.com/locate/jss

Predicting object-oriented software maintainability

using multivariate adaptive regression splines
Yuming Zhou, Hareton Leung *

Department of Computing, Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, PR China

Received 21 February 2006; received in revised form 26 October 2006; accepted 29 October 2006
Available online 8 December 2006

Abstract

Accurate software metrics-based maintainability prediction can not only enable developers to better identify the determinants of soft-
ware quality and thus help them improve design or coding, it can also provide managers with useful information to help them plan the
use of valuable resources. In this paper, we employ a novel exploratory modeling technique, multiple adaptive regression splines
(MARS), to build software maintainability prediction models using the metric data collected from two different object-oriented systems.
The prediction accuracy of the MARS models are evaluated and compared using multivariate linear regression models, artificial neural
network models, regression tree models, and support vector models. The results suggest that for one system MARS can predict main-
tainability more accurately than the other four typical modeling techniques, and that for the other system MARS is as accurate as the
best modeling technique.
Ó 2006 Elsevier Inc. All rights reserved.

Keywords: Object-oriented; Maintainability; Prediction; Multiple adaptive regression splines

1. Introduction
The largest cost associated with any software product
over its lifetime is the software maintenance cost. One
approach to controlling maintenance costs is to utilize soft-
ware metrics during the development phase (Bandi et al.,
2003). Studies examining the link between OO software
metrics and maintainability have found that in general
these metrics can be used as predictors of maintenance
effort (van Koten and Gray, 2006; Fioravanti and Nesi,
2001; Li and Henry, 1993; Briand et al., 2001; Bandi
et al., 2003; Misra, 2005; Thwin and Quah, 2005). In Fio-
ravanti and Nesi (2001), Li and Henry (1993), and Misra
(2005), linear regression techniques are used to build main-
tainability prediction models. In Thwin and Quah (2005),
neural networks are used to build maintainability predic-
*
Corresponding author. Tel.: +86 852 27667252; fax: +86 852
27740842.
E-mail address: cshleung@inet.polyu.edu.hk (H. Leung).
tion models. In Fenton et al. (2002) and Fenton and Neil
(1999, 2000), Fenton et al. state that the Bayesian Belief
Networks (BBNs) is the most promising technique for soft-
ware quality prediction because of their ability in handling
uncertainties, incorporating expert knowledge, and model-
ing the complex relationships among variables. The intro-
duction of the BBNs to software quality prediction is
certainly a positive step forward. However, the limitations
of the BBNs have also been recognized by researchers
(Ma et al., 2006; Weaver, 2003; Yu et al., 2002). First,
for software quality prediction, the links in the BBNs rep-
resent casual relationships between variables that are gen-
erally software metrics, but not all related software
metrics have causal relationships. Second, it is not easy
to express uncertainty in terms of probabilities. Third,
the inference of the BBNs requires a (subjective) prior
probability distribution, which may not be always available
or reasonable. The prediction accuracy may be greatly
decreased if the subjective prior are made incorrectly.
Forth, it is difficult to identify the correct structure of the
network that one has never explicitly described in detail

0164-1212/$ - see front matter Ó 2006 Elsevier Inc. All rights reserved.
doi:10.1016/j.jss.2006.10.049
1350 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
before. In practice, whether a modeling technique is suit-
able for maintainability prediction not only depends on
its ability in capturing causal relationships but also
depends on the ease of building prediction models.
Although the BBNs are strong in casual modeling, the lim-
itations mentioned above might make it difficult for practi-
tioners to develop a maintainability model with high
prediction accuracy. In van Koten and Gray (2006), Koten
et al. make the first use of the BBNs in building software
maintainability prediction models. They use a special type
of Bayesian networks called Naı̈ve–Bayes classifier, which
assumes no expert knowledge about the prior probability
distribution but learns it from data by batch learning.
The results show that the prediction accuracy of the BBN
model is more accurate than regression-based models for
one system but is less accurate than regression-based mod-
els for another system. Accurate software metrics-based
maintainability prediction is desirable first because it
reduces future maintenance efforts by enabling developers
to better identify the determinants of software quality
and thereby improve design or coding, and second because
it provides managers with information for more effectively
planning the use of valuable resources. Although a number
of maintainability prediction models have been developed
in last decade, they have low prediction accuracies accord-
ing to the criteria suggested in the literature (Conte et al.,
1986). Therefore, it is necessary to explore new techniques,
which are easy to use, for building maintainability predic-
tion models with high prediction accuracy.
This paper investigates the applicability to software
maintainability prediction of a novel exploratory multivar-
iate analysis technique, MARS. MARS differs from other
techniques in that its approach to model building does
not require the specification in advance of a functional
form. Rather, it attempts to adapt to the unknown func-
tional form using a series of piecewise regression splines.
This makes it very suitable for modeling complex relation-
ships that other modeling techniques find difficult, if not
impossible, to reveal. The power of MARS for building
prediction models has been demonstrated in many applica-
tions such as biomedical analysis (Deconinck et al., 2005),
network intrusion detection (Peddabachigari et al., 2007),
credit scoring (Lee and Chen, 2005), and cancer diagnosis
(Chou et al., 2004). However, MARS has been little
used in software engineering (Briand et al., 2002, 2004).
In this paper, we describe its first use in building software
maintainability prediction models with Li and Henry’s
metric data collected from two different object-oriented
systems (Li and Henry, 1993). The maintainability of a
software system can be measured in different ways. In past
studies, maintainability has been defined as ‘‘time required
to make changes’’ (Gibson and Senn, 1989) and ‘‘time to
understand, develop, and implement modification’’ (Ris-
ing, 1992). In this paper, it is measured as the number of
changes made to code during a maintenance period. To
evaluate the benefits of using MARS over typical modeling
techniques, we also build classical multivariate linear
regression models, artificial neural network models, regres-
sion tree models, and support vector regression models. We
compare the MARS models with these four models in
terms of their prediction performance using leave-one-out
cross-validation. The results suggest that for one system
MARS can predict maintainability more accurately than
the other four typical modeling techniques, and for the
other system MARS is as accurate as the best modeling
technique.
The rest of this paper is organized as follows. Section 2
introduces four typical prediction modeling techniques.
Section 3 describes the main modeling technique used: mul-
tivariate adaptive regression splines (MARS). Section 4
discusses the prediction accuracy measures, cross-valida-
tion method, and significance test technique employed in
evaluating prediction models. Section 5 presents the OO
software data sets used in our study. Section 6 reports
the results of MARS models and compares them with other
models. Section 7 concludes the paper and outlines direc-
tions for future work.
2. Typical modeling techniques
In this section, we first introduce three commonly used
modeling techniques: multivariate linear regression, artifi-
cial neural network, and regression tree. Then, we describe
the recently proposed support vector regression, which is
regarded as an excellent technique for producing prediction
models with good generalization.
2.1. Multivariate linear regression
Multivariate linear regression (MLR) is the most com-
monly used technique for modeling the relationship
between two or more independent variables and a depen-
dent variable by fitting a linear equation to observed data.
The main advantages of this technique are its simplicity
and that it is supported by many popular statistical pack-
ages (Heiat, 2002). The general form of a MLR model
can be given by:
^y i ¼ a0 þ a1 xi1 þ    þ ak xik
y i ¼ a0 þ a1 xi1 þ    þ ak xik þ ei
where xi1 ; . . . ; xik are the independent variables, a0, . . . , ak
the parameters to be estimated, ^y i the dependent variable
to be predicted, yi the actual value of the dependent vari-
able, and ei is the error in the prediction of the ith case
(Khoshgoftaar and Seliya, 2003).
When building such an MLR model, the use of a vari-
able selection method ensures that only important indepen-
dent variables are included in the model. Three commonly
used variable selection methods are forward elimination,
backward elimination, and stepwise selection. The forward
elimination method starts with a model that includes the
intercept only. Based on certain evaluation criteria, inde-
pendent variables are then selected for inclusion in the
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
model, until a stopping criterion is fulfilled. The backward
elimination method starts with a model that includes all
independent variables. A number of these independent
variables are then deleted one at a time until a stopping cri-
terion is fulfilled. Neither the forward nor the backward
elimination method is guaranteed to find the subset with
the highest evaluation criterion. In contrast, stepwise selec-
tion selects an optimal subset of independent variables for
the model. At each step of the stepwise selection process,
independent variables are either added or deleted from
the regression model.
Once the significant independent variables have been
determined, i.e. the model is selected, the parameters
a0, . . . , ak are then estimated using the least squares meth-
ods. More specifically, the values P of the parameters are
selected in order to minimize ni¼1 e2i , where n is the number
of observations in the fit data set.
2.2. Artificial neural network
An Artificial Neural Network (ANN) is an information
processing paradigm that is inspired by the way biological
nervous systems, such as the brain, process information.
The key element of this paradigm is the novel structure
of the information processing system. It is composed of a
large number of highly interconnected processing elements
(neurons) working in unison to solve specific problems
(Belsley et al., 2004; Rosenblatt, 1962).
Feed-forward supervised-learning neural networks are
the most commonly used model-building technique as an
alternative to MLR for software quality or effort prediction
(Heiat, 2002; Srinivasan et al., 1995; Karunaithi et al.,
1992; Khoshgoftaar and Lanning, 1995). Fig. 1 shows a
fully connected feed-forward ANN, which consists of four
layers of nodes: one input layer, two hidden layers, and one
output layer. Each line between nodes has a corresponding
and distinct weight. The feed-forward ANN allows signals
to travel one way only, from input to output. There is no
feedback (loops), i.e. the output of any layer does not affect
that same layer. The output of each node is determined by
its inputs and the weights associated with the lines between
these inputs and this node. In other words, each node is
hidden layer 1
input layer hidden layer 2
x1 output layer
y
x2
x3
Fig. 1. A feed-forward neural network.
1351
associated with a transfer function. The behavior of an
ANN depends on both the weights and the transfer func-
tions that are specified for the nodes.
In order to train an ANN to perform some task, we
must adjust the weight of each line in such a way as to
reduce the error between the expected output and the
actual output. The back-propagation algorithm is the most
widely used training method for multilayer neural net-
works. It initializes the network with a random set of
weights, and the network is trained from a set of input–out-
put pairs. Given an input–output pair, the error between
the actual output and the expected output is ‘‘back-propa-
gated’’ through the network and guides the adjustment of
weights in a way that reduces the collective error between
actual and expected outputs on the training set (Minsky
and Paper, 1969). The details of this algorithm can be
found in Hochman et al. (2003).
2.3. Regression tree
Regression tree (RT) is a variant of decision trees that
predicts values of continuous variables instead of labels
of classification (Breiman et al., 1993). Fig. 2 shows an
example regression tree over 209 different computer config-
urations, adapted from Witten and Frank (2000).
A regression tree is built through a recursive partitioning
process. This is an iterative process of splitting the data
into partitions, and then splitting it up further on each of
the branches. Initially all of the records in the training set
are put together in one node. The algorithm chooses an
independent variable with values that minimize the sum
of the squared deviations from the mean in the separate
parts. More specifically, assume that the values of each
independent variable IVi partition the entire training data
set T into subsets Tij, where every observation in Tij takes
on the same value, say
P Vj for IVi. The independent variable
IVi that minimizes j MSEðT ij Þ is selected to divide the
tree, where MSE(Tij) is the squared deviations from the
mean in Tij for the dependent variable y, i.e.,
P
y k 2T ij ðy k  y Þ2
MSEðT ij Þ ¼
jT ij j
CHMIN
≤ 7.5 > 7.5
CACH MMAX
≤ 8.5 > 8.5 ≤ 28000 > 28000
MMAX 88.7 (5) 157 (21) 454.3 (23)
≤ 4250 > 4250
25.3 (65) CACH
≤ 0.5 (0.5, 8.5]
32.2 (26) 59.3 (24)
Fig. 2. A regression tree model for CPU performance data.
1352 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361

where y is the mean of the yk values in Tij. This partitioning X

l

is then applied to each of the new branches. The process
continues until each node reaches a user-specified mini-
mum node size and becomes a terminal node. Note that
if the sum of squared deviations from the mean in a node
is zero, then that node is considered a terminal node even
if it has not reached the minimum size. In practice, the
most commonly used regression trees are binary trees.
2.4. Support vector regression
Support vector machine (SVM) was originally devel-
oped for solving the classification problems (Vapnik,
1995; Cortes and Vapnik, 1995) but recently it was
extended to the domain of regression problems (Vapnik
et al., 1997; Smola, 1996; Smola et al., 1998). Since SVM
employs the structural risk minimization (SRM) principle
rather than the empirical risk minimization (ERM) princi-
ple as adopted in ANN, it can, unlike ANN approaches,
produce prediction models with excellent generalization
performance. As a result, SVM is gaining in popularity in
the machine learning community.
Suppose we are given a set of l training data
{(x1, y1), . . . , (xl, yl)}, where xi 2 Rd denotes the ith input
pattern from the d dimension input space and has a corre-
sponding target value y i 2 R for i = 1, . . . , l, where R is
the set of real number. The goal of support vector regres-
sion (SVR) is to find a function that approximates the actu-
ally obtained targets yi for all the training data, and has a
minimum generalization error. The general form of a SVR
function can be given by:
f ðxÞ ¼ w  /ðxÞ þ b
where w 2 Rn , b 2 R, Æ denotes the dot product in Rn , and
/ is a non-linear transformation from Rd to the high-
dimensional space Rn (i.e., n > d). Our goal is to determine
the value of w and b such that f(x) can be determined by
minimizing the regression risk
X
l
1 2
Rreg ðf Þ ¼ C Cðf ðxi Þ  y i Þ þ jjwjj
i¼0
2
f ðxÞ ¼ ðai  ai Þð/ðxi Þ  /ðxÞÞ þ b
i¼1
X
l
¼ ðai  ai Þkðxi ; xÞ þ b
i¼1
where k(xi, x) is called the kernel function, which enables
the dot product to be performed in high-dimensional fea-
ture space using low-dimensional space data input without
the need to know the transformation /. Common kernel
functions include the linear, polynomial, and radial basis
functions (Smola et al., 1998). The most widely used cost
function is the so called e-insensitive loss function, which
takes the form

jf ðxÞ  yj  e if jf ðxÞ  yj P e
Cðf ðxÞ  yÞ ¼
0 otherwise
In this loss function, errors are not an issue as long as they
are less than e, but any deviation larger than this is not
acceptable. Such deviations are penalized in a linear
fashion.
When applying SVR in real applications, we need to
give a kernel function, the penalty C, and the radius e
which determines the data inside the e tube to be ignored
in regression.
3. Multivariate adaptive regression splines
Multivariate adaptive regression spline (MARS), pro-
posed by Freidman (1991), is a non-parametric regression
technique which models complex relationships that are dif-
ficult, if not impossible, for other modeling methods to
reveal. In a sense, MARS is based on a divide-and-conquer
strategy, partitioning the training data sets into separate
regions, each of which gets its own regression equation.
This makes MARS particularly suitable for problems with
high input dimensions.
Fig. 3 shows a simple example of how MARS would use
piece-wise linear regression splines to attempt to fit data, in
a two dimension space (where Y is the dependent variable
30

where C is a cost function, C is a constant that represents 25

penalties for estimation error (a large value for C means er-
rors are penalized heavily whereas a small value for C 20
means errors are penalized only lightly). A heavier penalty Y 15
trains the regression to minimize errors by making fewer
generalizations. w can be written in terms of data points as 10

X
l
5
w¼ ðai  ai Þ/ðxi Þ
i¼1
0
16 24
0 10 20 30 40
a*
where a and are Lagrange multipliers with the properties
X
a, a*P 0. Therefore, the general SVR function can be refor-
mulated as Fig. 3. Example knots in MARS.
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
and X is the independent variable). A key concept is the
notion of knots, which are the points that mark the end
of a region of data where a distinct regression equation is
run, i.e. where the behavior of the modeled function
changes. Fig. 3 shows two knots: 16 and 24. They delimit
three intervals where different linear relationships are
identified.
MARS makes no assumption about the underlying
functional relationship between the dependent and inde-
pendent variables. It builds flexible regression models by
fitting separate splines (or basis functions) to distinct inter-
vals of the independent variables. Both the variables to be
used and the end points of the intervals for each variable
(i.e. knots) are found through a fast but intensive search
procedure. In addition to searching variables one by one,
MARS also searches for interactions between independent
variables, allowing any degree of interaction to be consid-
ered as long as the model that is built can better fit the data.
The general MARS model can be represented using the
following equation:
_ X
M Y
Km
y ¼ c0 þ cm bkm ðxvðk;mÞ Þ
m¼1 k¼1
_
where y is the dependent variable predicted by the MARS
model, c0 is a constant, bkm ðxvðk;mÞ Þ is the truncated power
basis function with v(k, m) being the index of the indepen-
dent variable used in the mth term of the kth product, and
Km is a parameter that limits the order of interactions (the
resulting model will be an additive for Km = 1, and pairwise
interactions are allowed for Km = 2). The splines bkm are
defined in pairs:
 of the most important variable.
ðx  tkm Þq if x > tkm
bkm ðxÞ ¼ ðx  tkm Þqþ ¼
0 otherwise
and
 q
q ðtkm  xÞ if tkm > x
bkmþ1 ðxÞ ¼ ðtkm  xÞþ ¼
0 otherwise
for m an odd integer, where tkm, one of the unique values of
xv(k, m), is known as the knot of the spline, q P 0 is the
power to which the splines are raised in order to manipu-
late the degree of smoothness of the resultant regression
models. When q = 1, simple linear splines are applied.
The optimal MARS model is built in two stages: a for-
ward stepwise selection process followed by a backward
‘‘pruning’’ process. The forward stepwise selection of the
basis function starts with the constant basis function. At
each step, from all the possible splits in each basis function,
the process chooses the split that minimized some ‘‘lack of
fit’’ criterion. This search continues until the model reaches
some predetermined maximum number of basis functions.
In the backward ‘‘pruning’’ process, the ‘‘lack of fit’’ crite-
rion is used to evaluate the contribution of each basis func-
tion to the descriptive abilities of the model. The base
functions contributing the least to the model are eliminated
stepwise. The optimal model in the stepwise sequence is
1353
then selected. Here, the lack of fit measure used is based
on the generalized cross-validation criterion (GCV),
defined as
, 2
1X n
_ 2 CðMÞ
GCVðMÞ ¼ ðy  y Þ 1
n i¼1 i n
where n is the number of observations in the data set, M
the number of non-constant terms in the model, and
C(M) is a complexity penalty function. The purpose of
C(M) is to penalize model complexity, to avoid overfitting,
and to promote the parsimony of models. It is usually de-
fined as
CðMÞ ¼ M þ cd
where c is an user-defined cost penalty factor for each base
function optimization, and d is the effective degrees of free-
dom, which is equal to the number of independent basis
functions in the model. The higher the factor c is, the more
basis functions will be excluded. In practice, c is increased
during the pruning step in order to obtain smaller models.
Once the model is built, it is possible to estimate, on a
scale between 0 and 100, the relative importance of a vari-
able in terms of its contribution to the fit of the model. To
calculate the relative importance of a variable, we delete all
terms containing the variable in question, refit the model,
and then calculate the reduction in fit. The most important
(and highest scoring) variable is the one that, when deleted,
most reduces the fit of the model. Less important variables
receive lower scores. These scores correspond to the ratio
of the reduction in fit produced by these variables to that
4. Model evaluation
In this section, we first introduce the criteria for evaluat-
ing the prediction accuracy of a model. Then, we discuss
the cross-validation method used in this study to obtain
nearly unbiased estimators of prediction error. Finally,
we describe the significant test method for comparing the
prediction capability of different models.
4.1. Prediction accuracy
An important question that needs to be asked of any
prediction model is ‘‘How accurate are its predictions?’’
Researchers have used various measures of accuracy, the
most popular being residual, magnitude of relative error,
and Pred. All of these are based on only two terms, the
actual and the predicted values.
Suppose the training set consists of n observations.
Given an observation i, the corresponding residual is the
difference between the actual value and the predicted value
(Maxwell, 2002), that is,
Resi ¼ y i  ^y i
1354 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
where yi is the ith value of the dependent variable as ob-
served in the data set and ^y i is the corresponding predictive
value from the prediction model. Thus, positive numbers
mean underestimates and negative numbers mean overesti-
mates. The absolute value of this residual is called absolute
residual error (ARE), that is,
AREi ¼ jResi j
To characterize the distribution of AREs, we use the
sum of AREs (SumARE), the median of AREs (MedARE),
and the standard deviation of AREs (SDARE). SumARE
measures the total residuals over the data set, MedARE
measures the central tendency of the ARE distribution,
and SDARE measures the dispersion of the ARE distribu-
tion. These measures can be used to compare the prediction
performance of different models. In Pickard et al. (1999),
however, Pickard et al. pointed out that it is often more
useful to look at the boxplots of the residuals, which allow
different models to be compared visually. Moreover, resid-
ual boxplots show whether or not the predictions are
biased (i.e. whether the median differs from zero) and
whether the model has a tendency to underestimate or
overestimate.
Although ARE can be used to compare models on the
same data set, it cannot be used as an indicator alone
(De Lucia et al., 2005). For this reason, a normalized mea-
sure magnitude of relative error (MRE) is often used. Given
an observation i, MREi is defined as
jy i  ^y i j
MREi ¼
yi
Similar to van Koten and Gray (2006), this paper uses
the maximum value of MRE (MaxMRE) to compare the
prediction performance of different models. The MaxMRE
measures the maximum relative discrepancy, which is the
maximum error relative to the actual value in the predic-
tion. The mean magnitude of relative error (MMRE) is cal-
culated with the following formula:
1Xi¼n
MMRE ¼ MREi
n i¼1
where n is the number of observations in the data set.
MMRE is regarded as a versatile assessment criterion
and has a number of advantages: it can be used to make
comparisons across data sets and all kinds of prediction
model types; and it is independent of measurement units
and is scale independent (Walkerden and Jeffery, 1999;
Conte et al., 1986; Strike et al., 2001). Although some
research has shown that MMRE may produce misleading
results (Myrtveit et al., 2005; Foss et al., 2003), it has been
the de facto standard as an accuracy measure for prediction
models because it is easy to understand and meaningful for
the final users of prediction models.
Pred is a measure of the proportion of the predicted
values that have an MRE less than or equal to a specified
value, given by
k
PredðqÞ ¼
n
where q is the specified value, k the number of observations
whose MRE is less than or equal to q, and n is the total
number of observations in the data set. In this paper,
Pred(0.25) and Pred(0.30) are used because they are com-
monly used in the empirical software engineering literature
(De Lucia et al., 2005; Kitchenham et al., 2002). The for-
mer reports the percentage of the estimates with an MRE
of 25% or less, and the latter reports the percentage of
the estimates with an MRE of 30% or less.
4.2. Cross validation
When a model is built, we should perform some kind of
cross-validation. Cross-validation is a way of obtaining a
realistic estimate of the predictive power of a model when
it is applied to data sets other than those from which the
model was derived. In general, a data set is divided into
two subsets: a training set and a test set. The training set
is used to fit the model, and the test set is used to validate
the model. This is called split-sample validation. Since data
sets used in effort prediction are usually of limited size,
split-sample validation is often difficult. As a remedy, v-fold
cross-validation is a way of obtaining nearly unbiased esti-
mators of prediction error. For a data set with n observa-
tions, a v-fold cross-validation divides the data set into v
approximately equal partitions, and each in turn is used
for testing while the remainder is used for training.
Leave-one-out (LOO) is simply n-fold cross-validation,
where n is the number of observations in the data set. We
choose to use LOO cross-validation in this study for three
reasons. First, as Myrtveit et al. pointed out, LOO cross-
validation is a widely used variant of v-fold cross-valida-
tion. More importantly, from a practitioner’s standpoint
it is closer to a real world situation than k-cross validation
(k < n) (Myrtveit et al., 2005). Second, unlike k-fold cross-
validation (k < n), LOO is deterministic, i.e. no sampling is
involved (Witten and Frank, 2000). Third, LOO ensures
that the greatest possible amount of data is used for train-
ing for each case, which presumably increases the chance of
getting as accurate an estimate as can possibly be obtained
(Witten and Frank, 2000). The disadvantage of LOO is
that it is computationally intensive, for the entire learning
procedure must be executed n times. However, this is not
a problem for our study because of the relatively small
sample size.
4.3. Significance test
We are interested in observing whether MARS is supe-
rior to MLR, ANN, RT, and SVR for building maintain-
ability prediction models. As such, the Wilcoxon signed-
rank test for matched samples is used to test and compare
the MARS models and the other models in terms of the
two types of error, ARE and MRE. This test was chosen
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
because: (a) it is a distribution-free technique that does not
require any underlying distributions in the data; (b) it deals
with the signs and ranks of the values and not with their
magnitude (thus not influenced by outlier data points).
The test procedure first calculates the differences
between the paired observations, ranks them from the
smallest to largest by absolute value (the rank assigned to
tied observations is the mean of ranks that would have
been assigned to the observations had they not been tied),
and then affixes the sign of each difference to the corre-
sponding rank (Zar, 1984). The sum of the ranks having
a plus sign is called T+ and the sum of the ranks having
a minus sign is called T. When the sample size n is larger
than 25, the distribution of T (where either T+ or T may
be used for T) is closely approximated by a normal distri-
bution with a mean of
nðn þ 1Þ
lT ¼
4
and a standard error of
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
nðn þ 1Þð2n þ 1Þ
rT ¼
24
Thus, the test statistic can be calculated from
jT  lT j  0:5
Z¼
rT
where for T we may use, with identical results, either T+ or
T. Then, for a two-tailed test, if Z is greater than or equal
to the critical value Za(2), then the null hypothesis H0
(l1 = l2) is rejected, where l1 and l2 are two population
means of matched pairs. That is, the alternative hypothesis
H1 (l15l2) is accepted. In particular, if Z P Za(2) and
T+ < T, we conclude that l1 < l2; if Z P Za(2) and
T+ > T, we conclude that l1 > l2. The level of statistical
significance a provides an insight into the accuracy of test-
ing. The selection of a may depend on the analyst and the
property of the investigated problem. In empirical software
engineering literature, it is common to set a = 0.05 or 0.1.
In this study, the null and alternative hypotheses using
ARE are as follows:
H0 : AREMARS ¼ AREX
H1 : AREMARS 6¼ AREX
where X denotes MLR, ANN, RT, or SVR. Similarly, the
null and alternative hypotheses using MRE are as follows:
H0 : MREMARS ¼ MREX
H1 : MREMARS 6¼ MREX
The minimum significance level for rejecting a null hypoth-
esis was set at a = 0.05.
5. Data sets
In this section, we describe the data set used for this
study. We first introduce the metrics under study and then
1355
give the distribution and correlation analysis results of the
metrics that were investigated.
5.1. Studied metrics
This study makes use of two OO software data sets pub-
lished by Li and Henry (1993). The first data set, UIMS,
contains the metric data of 39 classes collected from a user
interface management system. The second data set, QUES,
contains the metric data of 71 classes collected from a qual-
ity evaluation system. Both systems were implemented in
Ada.
The metric data of both the UIMS and QUES data sets
consists of eleven metrics: nine object-oriented metrics, one
traditional size metric, and one maintainability metric.
Among these object-oriented metrics, WMC, DIT, RFC,
NOC, and LCOM were proposed by Chidamber and
Kemerer (1994), MPC, DAC, NOM and SIZE2 were pro-
posed by Li and Henry (1993), and SIZE1 is the traditional
lines of code size metric. The object-oriented metrics cap-
ture the concept of cohesion, coupling, inheritance, and
size. Maintainability was measured with the CHANGE
metric by counting the number of lines in the code that
were changed per class over a 3-year maintenance period.
A line change could be an addition or a deletion. A change
in the content of a line is counted as a deletion followed by
an addition.
Table 1 provides definitions of these metrics. Essentially,
all the object-oriented metrics measure some aspect of the
complexity of software. WMC is used to measure the com-
plexity of an individual class attributable to control flow
structures. DIT and NOC determine the complexity of a
class attributable to inheritance relationships. RFC,
MPC, and DAC measure the complexity of a class caused
by coupling, while LCOM quantifies the complexity of a
class caused by cohesion. NOM, SIZE1, and SIZE2 mea-
sure the complexity of a class caused by its size. It is
believed that the complexity of an object-oriented system
strongly affects its maintainability (van Koten and Gray,
2006; Li and Henry, 1993; Thwin and Quah, 2005). There-
fore, it is reasonable to use these object-oriented metrics to
predict software maintainability.
5.2. Characteristics of data sets
In the following we describe the results of the distribu-
tion and correlation analysis of the eleven metrics as they
appear in our data set. Tables 2 and 3 present some com-
mon descriptive statistics from the UIMS and QUES data
sets, respectively. As can be seen, both the UIMS and
QUES data sets have low medians and means for DIT.
This indicates that inheritance was not used a lot in both
systems. Similar medians and means of NOM and SIZE2
are found in the UIMS and QUES data sets, suggesting
that the class sizes at the design level in both systems are
similar. However, the medians and means of SIZE1 in
the QUES data set are significantly larger than those in
1356 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361

Table 1
Definitions of metrics
Metric
WMC (Weighted methods per class)
DIT (Depth of inheritance tree)
RFC (Response for a class)
NOC (Number of children)
LCOM (Lack of cohesion in methods)
MPC (Message-passing coupling)
DAC (Data abstraction coupling)
NOM (Number of methods)
SIZE1 (Lines of code)
SIZE2 (Number of properties)
CHANGE (Number of lines changed in the
class)
Definition
The sum of McCabe’s cyclomatic complexity of all local methods in a given class
The length of the longest path from a given class to the root in the inheritance hierarchy
The number of methods that can potentially be executed in response to a message
being received by an object of a given class
The number of classes that directly inherit from a given class
The number of pairs of local methods in a given class using no attribute in common
The number of send statements defined in a given class
The number of abstract data types defined in a given class
The number of methods implemented within a given class
The number of semicolons in a given class
The total number of attributes and the number of local methods in a given class
Insertion and deletion are independently counted as 1, change of the contents is counted as 2

Table 2
Descriptive statistics of the UIMS data set
Metric Maximum 75% Median 25% Minimum Mean Standard deviation Skewness Kurtosis
WMC 69 12 5 1 0 11.38 15.90 2.03 3.98
DIT 4 3 2 2 0 2.15 0.90 0.54 0.09
RFC 101 30 17 11 2 23.21 20.19 2.00 4.94
NOC 8 1 0 0 0 0.95 2.01 2.24 4.28
LCOM 31 8 6 4 1 7.49 6.11 2.49 6.86
MPC 12 6 3 1 1 4.33 3.41 0.731 0.70
DAC 21 3 1 0 0 2.41 4.00 3.33 12.87
NOM 40 13 7 6 1 11.38 10.21 1.67 1.94
SIZE1 439 131 74 27 4 106.44 114.65 1.71 2.04
SIZE2 61 16 9 6 1 13.97 13.47 1.89 3.44
CHANGE 289 39 18 10 2 46.82 71.89 2.29 4.35

Table 3
Descriptive statistics of the QUES data set
Metric Maximum 75% Median 25% Minimum Mean Standard deviation Skewness Kurtosis
WMC 83 22 9 2 1 14.96 17.06 1.77 3.33
DIT 4 2 2 2 0 1.92 0.53 0.10 5.46
RFC 156 62 40 34 17 54.44 32.62 1.62 1.96
NOC 0 0 NA 0 0 0 0.00 NA NA
LCOM 33 14 5 4 3 9.18 7.34 1.35 1.10
MPC 42 21 17 12 2 17.75 8.33 0.88 1.17
DAC 25 4 2 1 0 3.44 3.91 2.99 12.82
NOM 57 21 6 5 4 13.41 12.00 1.39 1.40
SIZE1 1009 333 211 172 115 275.58 171.60 2.11 5.23
SIZE2 82 25 10 7 4 18.03 15.21 1.71 3.42
CHANGE 217 85 52 35 6 64.23 43.13 1.36 2.17

the UIMS data set. This may suggest that the complexities
of the problems processed by the two systems are rather
different. Moreover, the medians and means of RFC and
MPC in the QUES data set are far higher than the corre-
sponding medians and means in the UIMS data set. This
suggests that the coupling between classes in the quality
evaluation system is higher than those in the user interface
management system. In contrast, the median and mean of
LCOM in the QUES data set are similar to the median and
mean of LCOM in the UIMS data set, implying that these
two systems have similarly cohesion. It can also be seen
that the mean of CHANGE in the QUES data set is larger
than that in the UIMS data set.
As stated in Briand et al. (2000), metrics that vary little
do not differentiate classes very well and therefore are not
likely to be useful predictors. Only measures with more
than five non-zero data values will be considered for subse-
quent analysis. From Table 2, for all metrics except NOC,
DIT, and DAC, there are large differences between the
lower 25th percentile, the median, and the 75th percentile,
thus showing strong variations across classes. For NOC,
DIT, and DAC, the number of their non-zero data points
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361 1357

Table 4
Correlations between the metrics in the data sets UIMS (upper triangle) and QUES (lower triangle)
WMC DIT RFC NOC LCOM MPC DAC NOM SIZE1 SIZE2 CHANGE
* * * * * * *
WMC 1 0.22 0.91 0.23 0.80 0.63 0.44 0.84 0.97 0.77 0.65*
DIT 0.13 1 0.23 0.47 0.19 0.06 0.43* 0.36* 0.19 0.41* 0.43*
RFC 0.74* 0.11 1 0.21 0.79* 0.74* 0.62* 0.93* 0.91* 0.89* 0.64*
NOC NA NA NA 1 0.13 0.03 0.32* 0.23 0.17 0.27 0.56*
LCOM 0.57* 0.12 0.82* NA 1 0.50* 0.36* 0.75* 0.82* 0.68* 0.57*
MPC 0.14 0.02 0.33* NA 0.10 1 0.44* 0.55* 0.67* 0.55* 0.45*
DAC 0.57* 0.39* 0.64* NA 0.56* 0.02 1 0.75* 0.52* 0.87* 0.63*
NOM 0.70* 0.13 0.81* NA 0.88* 0.11 0.81* 1 0.87* 0.98* 0.64*
SIZE1 0.89* 0.01 0.80* NA 0.54* 0.37* 0.64* 0.69* 1 0.82* 0.63*
SIZE2 0.69* 0.20 0.81* NA 0.84* 0.08 0.89* 0.99* 0.71* 1 0.67*
CHANGE 0.43* 0.09 0.39* NA 0.05 0.46* 0.08 0.14 0.64* 0.15 1
*
Correlation is significant at the 0.01 level (2-tailed).

is larger than five (Briand et al., 2000). Therefore, all these
metrics from the UIMS data set will be used in the follow-
ing analysis. However, from Table 3, we can see that there
is no non-zero data point for NOC. Thus, the metric NOC
in the QUES data set is removed from the following
analysis.
Table 4 shows the results of a linear Pearson’s correla-
tions analysis for the data sets. More specifically, the upper
triangular matrix represents the correlations between the
metrics in the UIMS data set, and the lower triangular
matrix represents the correlations between the metrics in
the QUES data sets. Assuming a reasonably sized data
set, Hopkins calls a correlation value of less than 0.1 trivial,
0.1–0.3 minor, 0.3–0.5 moderate, 0.5–0.7 large, 0.7–0.9 very
large, and 0.9–1 almost perfect (Hopkins, 2003). From
Table 4, it can be seen that almost all metrics except DIT
and NOC are related to each other in both data sets. In
particular, in the UIMS data set, all the object-oriented
metrics are statistically related to CHANGE. However,
in the QUES data set, only four metrics are statistically
related to CHANGE. From Tables 2–4, we conclude that
the characteristics of the UIMS dataset are different from
the QUES dataset. Therefore, to some extent, the UIMS
and QUES datasets are heterogeneous.
6. Empirical results
In this section, we analyze the results of MARS models
using the UIMS and QUES data sets. The tool used is
MARS 2.0 developed by Salford Systems.1 We use the
default setting provided by this tool to build MARS mod-
els. To assess the benefits of using MARS, we compare the
prediction performances of MARS models with those of
MLR models, ANN models, RT models, and SVR models
with LOO cross-validation. Furthermore, we employ the
Wilcoxon signed-rank test between MARS models and
other models to check whether their differences are
significant.
In this study, we use the stepwise selection procedure to
build the MLR prediction models. The software used is
SPSS 13.0.2 The entry criterion used in the stepwise selec-
tion is the p-value of the F statistic being smaller than or
equal to 0.05. The eliminating criterion used is the p-value
of the F statistic being larger than or equal to 0.10. In par-
ticular, the multi-collinearity between the predictor vari-
ables is tested using the condition index of the correlation
matrix of the covariates in the model (Belsley et al.,
2004). On the other hand, we use the tool WEKA 3.4.63
to build backpropagation trained feed-forward ANN
models, RT models, and SVR models. The default settings
provided by this tool are used to build these models.
6.1. Results from UIMS data set
Table 5 shows the optimal MARS model from the
UIMS data set. The columns show, from left to right, the
basis functions selected as significant covariates in the
model, the coefficients estimated, the associated standard
error, the t-ratio, and the p-value telling us about the signif-
icance of these coefficients. In this study, the interaction
effects between basis functions are not examined since we
use the default setting provided by MARS 2.0. Therefore,
this model is actually a ‘‘main-effects’’ model and its
adjusted R2 is 0.656.
Table 6 provides a ranking of the independent variables
by order of importance. Variables having no impact at all
are not shown. The loss in GCV is denoted as ‘‘gcv’’ in
Table 6. The column ‘‘Importance’’ shows the relative
importance of variables in terms of percentage of the high-
est gcv, that is the highest reduction of goodness of fit
among all variables.
Table 7 shows the prediction accuracy measures
achieved by the MARS model, the MLR model, the SVR
model, the ANN model, and the RT model for the UIMS
data set within LOO cross-validation. Although these
results do not satisfy the criteria for an accurate prediction

2
www.spss.com.
1 3
www.salford-systems.com. www.cs.waikato.ac.nz/ml/weka.
1358 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361

Table 5 contrast, the RT model has a median residual that is far

The MARS model for UIMS data set below zero. Thus, there is tendency to significantly overes-
Metric Coefficient Standard t- p- timate in the RT model. We also find that the MLR model
error Ratio Value tends to slightly underestimate while the ANN model tends
Intercept 2.184 8.637 0.253 0.802 to slightly overestimate. The MARS model also has the
BF2 = (NOC  0.131E  08)+ 12.957 3.586 3.613 0.009 narrowest box and the smallest whiskers (i.e. the lines
BF3 = (WMC  0.337E  6)+ 1.888 0.481 3.928 0.004
BF14 = (DAC  1.000)+ 6.316 2.033 3.106 0.004
above and below from the box) but at the same time, it
has the most outliers. In summary, comparing these box-
plots does not clearly show whether the MARS model is
Table 6 better than the other models.
Variable importance in the UIMS data set Table 8 presents the Z statistic values of the two-tailed
Metrics Importance gcv Wilcoxon signed-rank test for ARE and MRE values for
WMC 100.000 3277.827
the five models, where the numbers in parentheses are the
NOC 85.673 3123.369 corresponding p-values. As can be seen, the MARS model
DAC 59.394 2902.030 is not significantly different from the SVR and ANN mod-
els. However, although the test of ARE values indicates
that the MARS model is not much different from the
model, it is reported that the prediction accuracy of soft- MLR model (p = 0.364), the test of MRE values does show
ware maintenance effort prediction models is often low strong evidence that the MARS model outperforms the
and thus it is very difficult to satisfy the criteria (De Lucia MLR model (p-value = 0.009). Moreover, both the Wilco-
et al., 2005). As can be seen, the MARS model has the xon signed-rank tests of ARE and MRE values show that
smallest MedARE while its MaxMRE, MMRE, the MARS model is significantly better than the RT model.
Pred(0.25), and Pred(0.30) are inferior only to those of
the SVR model.
Fig. 4 shows the residual boxplots of all five prediction 6.2. Results from QUES data set
models for the UIMS data set, allowing a visual compari-
son. The line in the middle of each box represents the med- Table 9 shows the optimal MARS model for the QUES
ian of the residual. As can be seen, both the MARS model data set. The adjusted R2 of this model is 0.837, which is
and the SVR model have a median residual close to zero. In much larger than that (0.656) of the model for the UIMS
data set. Table 10 shows the relative importance of inde-
pendent variables for the QUES data set. Interestingly,
both the results from the UIMS and QUES data sets sug-
gest that WMC and DAC are important indicators of
object-oriented software maintainability.
Table 11 shows the prediction accuracy measures
achieved by the MARS model, the MLR model, the SVR
model, the ANN model, and the RT model for the QUES
data set within LOO cross-validation. Of all the models,
the MARS model has the best prediction accuracy.
Fig. 5 shows the residual boxplots of all five prediction
models for the QUES data set. As can be seen, the MARS
model, the SVR model, and the RT model all have a med-
ian residual close to zero. In contrast, the MLR and the
ANN models both have a median residual that is slightly
below zero causing them to have a tendency to slightly
overestimate. Of all the models, the MARS model has
Fig. 4. Residual boxplots of MARS and commonly used modeling the narrowest box and the smallest whiskers, as well as
techniques for UIMS data set. the fewest outliers. Based on these boxplots, we can say

Table 7
Prediction accuracy for UIMS data set (within LOO cross-validation)
Method MaxMRE MMRE Pred(0.25) Pred(0.30) SumARE MedARE SDARE
MARS 14.06 1.86 0.28 0.28 1532.78 9.26 59.76
MLR 18.88 2.70 0.15 0.21 1457.26 13.64 54.66
SVR 9.13 1.68 0.31 0.36 1242.59 11.03 48.66
ANN 19.63 1.95 0.15 0.15 1473.14 10.84 55.50
RT 24.57 4.95 0.10 0.10 1988.96 29.54 62.87
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361 1359

Table 8
Wilcoxon signed-rank test for UIMS data set
Modeling method MLR SVR ANN RT
ARE MRE ARE MRE ARE MRE ARE MRE
MARS 0.907a 2.610a 0.809b 0.777b 0.112b 0.334b 2.344a 2.686a
(0.364) (0.009) (0.418) (0.437) (0.911) (0.739) (0.019) (0.007)
a
T+ < T.
b
T+ > T.

Table 9
The MARS model for QUES data set
Metric Coefficient Standard error t-Ratio p-Value
Intercept 70.649 6.982 10.250 0.000
BF1 = (WMC  13.000)+ 3.663 0.885 4.137 0.001
BF2 = (13.000  WMC)+ 4.863 0.629 7.734 0.000
BF3 = (SIZE1  252.000)+ 0.786 0.166 4.728 <0.001
BF4 = (252.000  SIZE1)+ 0.612 0.066 9.229 0.000
BF5 = (DAC  0.122050E  06)+ 3.719 0.791 4.702 <0.001
BF6 = (WMC  26.000)+ 5.714 1.121 5.096 <0.001
BF9 = (SIZE1  310.000)+ 0.651 0.179 3.626 0.006
BF11 = (MPC  18.000)+ 1.982 0.449 4.418 <0.001

Table 10
Variable importance in the QUES data set
Metrics Importance gcv
SIZE1 100.000 1471.475
WMC 82.008 1171.803
DAC 36.684 679.524
MPC 33.426 658.626

that the MARS model is clearly the best of the five models.
This result is in line with the findings in Table 11.
Table 12 presents the Z statistic values of the two-tailed
Wilcoxon signed-rank test for ARE and MRE values
between the MARS model and the other four models,
where the numbers in parentheses are the corresponding
p-values. As can be seen, both the test of MRE values
Fig. 5. Residual boxplots of MARS and commonly used modeling
and the test of ARE values show that the MARS model
techniques for QUES data set.
is significantly better than the other four models.

7. Conclusions of the Li and Henry’s data sets, UIMS and QUES,

In this paper, we presented an empirical study that
sought to build object-oriented software maintainability
prediction models using a novel exploratory modeling tech-
nique, MARS. To build the MARS models, we made use
obtained from two different object-oriented systems. The
prediction performances of the MARS models were
assessed and compared with those of the multivariate linear
regression models, the artificial neural network models, the
regression tree models, and the support vector models. The

Table 11
Prediction accuracy for QUES data set (within LOO cross-validation)
Method MaxMRE MMRE Pred(0.25) Pred(0.30) SumARE MedARE SDARE
MARS 1.91 0.32 0.48 0.59 1153.10 13.45 14.66
MLR 2.03 0.42 0.37 0.41 1566.08 16.71 19.90
SVR 2.07 0.43 0.34 0.46 1563.67 15.57 19.79
ANN 3.07 0.59 0.37 0.45 2396.85 18.48 49.34
RT 4.82 0.58 0.41 0.45 1983.88 16.67 28.87
1360 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
Table 12
Wilcoxon signed-rank test for QUES data set
Modeling method MLR SVR
ARE MRE ARE
MARS 3.217a 3.134a 3.226a
(0.001) (0.002) (0.001)
a
T+ < T.
results show that the MARS models can effectively predict
the maintainability of OO software systems. For the UIMS
data set, the MARS model is as accurate as the best predic-
tion model. For the QUES data set, the MARS model has
achieved significantly better prediction accuracy than the
other four prediction models. Thus, overall, it is concluded
that the MARS model is better than the other models built
using typical modeling techniques and it can be a useful
modeling technique for software maintainability pre-
diction.
One limitation of our study is that the metric data was
collected from two systems implemented with a single lan-
guage. Future work will replicate this study across pro-
gramming languages. Such studies would allow us to
further investigate the capability of MARS in software
maintainability prediction. Another interesting work in
the future is to develop more accurate prediction models
by combining MARS with other prediction techniques.
Acknowledgements
We are very grateful to Prof. Wei Li for giving us a
scanned copy of their paper, and thank Dr. C. van Koten
for providing us with an electronic version of Li and
Henry’s data sets. We also wish to thank the anonymous
reviewers of this paper for their helpful comments and
suggestions.
References
Bandi, R.K., Vaishnavi, V.K., Turk, D.E., 2003. Predicting maintenance
performance using object-oriented design complexity metrics. IEEE
Transactions on Software Engineering 29 (1), 77–87.
Belsley, D., Kuh, E., Welsch, R., 2004. Regression Diagnostics: Identi-
fying Influential Data and Sources of Collinearity. Wiley, New York.
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J., 1993. Classifica-
tion and Regression Trees. Chapman & Hall, London.
Briand, L.C., Wust, J., Daly, J.W., Porter, D.V., 2000. Exploring the
relationships between design measures and software quality in object-
oriented systems. Journal of Systems and Software 51 (3), 245–273.
Briand, L.C., Bunse, C., Daly, J.W., 2001. A controlled experiment for
evaluating quality guidelines on the maintainability of object-oriented
designs. IEEE Transactions on Software Engineering 27 (6), 513–530.
Briand, L.C., Melo, W.L., Wüst, J., 2002. Assessing the applicability of
fault-proneness models across object-oriented software projects. IEEE
Transactions on Software Engineering 28 (7), 706–720.
Briand, L.C., Freimut, B., Vollei, F., 2004. Using multiple adaptive
regression splines to support decision making in code inspections.
Journal of Systems and Software 73 (2), 205–217.
Chidamber, S.R., Kemerer, C.F., 1994. A metrics suite for object-oriented
design. IEEE Transactions on Software Engineering 20 (6), 476–493.
ANN RT
MRE ARE MRE ARE MRE
3.155a 3.037a 3.158a 3.134a 2.854a
(0.002) (0.002) (0.002) (0.002) (0.004)
Chou, S.M., Lee, T.S., Shao, Y.E., Chen, I.F., 2004. Mining the breast
cancer pattern using artificial neural networks and multivariate
adaptive regression splines. Expert Systems with Applications 27 (1),
133–142.
Conte, S.D., Dunsmore, H.E., Shen, V.Y., 1986. Software Engineering
Metrics and Models. Benjamin/Cummings, Menlo Park, Calif.
Cortes, C., Vapnik, V., 1995. Support-vector networks. Machine Learning
20 (3), 273–297.
Deconinck, E., Xu, Q.S., Put, R., Coomans, D., Massart, D.L., Vander
Heyden, Y., 2005. Prediction of gastro-intestinal absorption using
multivariate adaptive regression splines. Journal of Pharmaceutical
and Biomedical Analysis 39 (5), 1021–1030.
De Lucia, A., Pompella, E., Stefanucci, S., 2005. Assessing effort
estimation models for corrective maintenance through empirical
studies. Information and Software Technology 47 (1), 3–15.
Fenton, N.E., Neil, M., 1999. A critique of software defect prediction
models. IEEE Transactions on Software Engineering 25 (5), 675–689.
Fenton, N.E., Neil, M., 2000. Software Metrics: Roadmap. In: Proceed-
ings of International Conference on Software Engineering. Future of
Software Engineering Track, Limerick, Ireland, pp. 357–370.
Fenton, N.E., Krause, Paul, Neil, M., 2002. Software measurement:
uncertainty and causal modeling. IEEE Software 19 (4), 116–122.
Fioravanti, F., Nesi, P., 2001. Estimation and prediction metrics for
adaptive maintenance effort of object-oriented systems. IEEE Trans-
actions on Software Engineering 27 (12), 1062–1084.
Foss, T., Stensrud, E., Kitchenham, B., Myrtveit, I., 2003. A simulation
study of the model evaluation criterion MMRE. IEEE Transactions on
Software Engineering 29 (11), 985–995.
Freidman, J., 1991. Multivariate adaptive regression splines. Annals of
Statistics 19, 1–141.
Gibson, V.R., Senn, J.A., 1989. System structure and software mainte-
nance performance. Communication of ACM 32 (3), 347–358.
Heiat, A., 2002. Comparison of artificial neural network and regression
models for estimating software development effort. Information and
Software Technology 44 (15), 911–922.
Heiat, A., 2002. Comparison of artificial neural network and regression
models for estimating software development effort. Information and
Software Technology 44 (15), 911–922.
Hochman, R., Khoshgoftaar, T.M., Allen, E.B., Hudepohl, J.P., 2003.
Improved fault-prone detection analysis of software modules using an
evolutionary neural network approach. In: Khoshgoftaar, T.M. (Ed.),
Software Engineering with Computational Intelligence. Kluwer Aca-
demic Publishers, pp. 69–100.
Hopkins, W.G., 2003. A New View of Statistics. SportScience, Dunedin,
New Zealand.
Karunaithi, N., Whitley, D., Malaiya, Y.K., 1992. Prediction of software
reliability using connectionist models. IEEE Transactions on Software
Engineering 18 (7), 563–574.
Khoshgoftaar, T.M., Lanning, D.L., 1995. A neural network approach for
early detection of program modules having high risk in the mainte-
nance phase. Journal of Systems and Software 29 (1), 85–91.
Khoshgoftaar, T.M., Seliya, N., 2003. Fault prediction modeling for
software quality estimation: comparing commonly used techniques.
Empirical Software Engineering 8 (3), 255–283.
Kitchenham, B., Pfleeger, S.L., McColl, B., Eagan, S., 2002. An empirical
study of maintenance and development estimation accuracy. Journal
of Systems and Software 64 (1), 57–77.
 Y. Zhou, H. Leung / The Journal of Systems and Software 80 (2007) 1349–1361
Lee, T.S., Chen, I.F., 2005. A two-stage hybrid credit scoring model using
artificial neural networks and multivariate adaptive regression splines.
Expert Systems with Applications 28 (4), 743–752.
Li, W., Henry, S., 1993. Object-oriented metrics that predict maintain-
ability. Journal of Systems and Software 23 (2), 111–122.
Ma, Y., Guo, L., Cukic, B., 2006. A Statistical Framework for the
Prediction of Fault-pronenessAdvances in Machine Leaning Applica-
tion in Software Engineering. Idea Group Inc.
Maxwell, K.D., 2002. Applied Statistics for Software Managers. Prentice
Hall.
Minsky, M., Paper, S., 1969. Perceptrons. MIT Press.
Misra, S.C., 2005. Modeling design/coding factors that drive maintain-
ability of software systems. Software Quality Journal 13 (3), 297–320.
Myrtveit, I., Stensrud, E., Shepperd, M., 2005. Reliability and validity in
comparative studies of software prediction models. IEEE Transactions
on Software Engineering 31 (5), 380–391.
Peddabachigari, S., Abraham, A., Grosan, C., Thomas, J., 2007.
Modeling intrusion detection system using hybrid intelligent systems.
Journal of Network and Computer Applications 30 (1), 114–132.
Pickard, L.M., Kitchenham, B.A., Linkman, S.J., 1999. An investigation
of analysis techniques for software test data sets. Technical Report
TR99-05, Department of Computer Science, Keele University, Staf-
fordshire, UK.
Rising, L.S., 1992. Information hiding metrics for modular programming
languages. PhD dissertation, Arizona State University.
Rosenblatt, F., 1962. Principles of Neurodynamics: Perceptrons and the
Theory of Brain Mechanisms. Spartan Press.
Smola, A.J., 1996. Regression estimation with support vector learning
machines. Master’s thesis, Technische Universitat Munchen.
Smola, A.J., Scholkopf, B., 1998. A tutorial on support vector regression.
Technical Report, NC2TR -1998-030, NeuroCOLT2.
Srinivasan, K., Fisher, D., 1995. Machine learning approaches to
estimating software development effort. IEEE Transactions on Soft-
ware Engineering 21 (2), 126–137.
1361
Strike, K., El-Emam, K., Madhavji, N., 2001. Software cost estimation
with incomplete data. IEEE Transactions on Software Engineering 27
(10), 890–908.
Thwin, M.M.T., Quah, T.S., 2005. Application of neural networks for
software quality prediction using object-oriented metrics. Journal of
Systems and Software 76 (2), 147–156.
van Koten, C., Gray, A.R., 2006. An application of Bayesian network for
predicting object-oriented software maintainability. Information and
Software Technology 48 (1), 59–67.
Vapnik, V., 1995. The Nature of Statistical Learning Theory. Springer,
New York.
Vapnik, V., Golowich, S., Smola, A., 1997. Support Vector Method for
Function Approximation, Regression Estimation, and Signal Process-
ing. Advances in Neural Information Processing Systems 9. MIT Press,
pp. 281–287.
Walkerden, F., Jeffery, R., 1999. An empirical study of analogy-based
software effort estimation. Empirical Software Engineering 4 (2), 135–
158.
Weaver, R.A., 2003. The safety of software – constructing and assuring
arguments. Ph.D. thesis. Department of Computer Science, University
of York.
Witten, I.H., Frank, E., 2000. Data Mining: Practical Machine Learning
Tools and Techniques with Java Implementations. Morgan Kaufman
Publishers.
Yu, Y.Y., Johnson, B.W., 2002. Modeling COTS systems for safety-
critical applications using system safety standards by Bayesian
Belief Networks. In: Proceedings of the 6th International Confer-
ence on Probabilistic Safety Assessment and Management (PSAM-6),
June.
Zar, J.H., 1984. Biostatistical Analysis. Prentice Hall, Englewood Cliffs,
New Jersey.