See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/4175543

What should you optimize when building an estimation model?

Conference Paper · October 2005

DOI: 10.1109/METRICS.2005.55 · Source: IEEE Xplore

CITATIONS READS
34 65

1 author:

Chris Lokan
Australian Defence Force Academy
99 PUBLICATIONS 1,963 CITATIONS

SEE PROFILE

All content following this page was uploaded by Chris Lokan on 14 February 2016.

The user has requested enhancement of the downloaded file.

 What Should You Optimize When Building an Estimation Model?

Chris Lokan
School of Information Technology and Electrical Engineering
UNSW@ADFA, Australian Defence Force Academy
Canberra ACT 2600, Australia
c.lokan@adfa.edu.au

Abstract
When estimation models are derived from existing data,
they are commonly evaluated using statistics such as mean
magnitude of relative error. But when the models are de-
rived in the first place, it is usually by optimizing something
else — typically, as in statistical regression, by minimizing
the sum of squared deviations. How do estimation models
for typical software engineering data fare, on various com-
mon accuracy statistics, if they are derived using other “fit-
ness functions”? In this study, estimation models are built
using a variety of fitness functions, and evaluated using a
wide range of accuracy statistics. We find that models based
on minimizing actual errors generally out-perform models
based on minimizing relative errors. Given the nature of
software engineering data sets, minimizing the sum of ab-
solute deviations seems an effective compromise.
Keywords: effort estimation, genetic programming, ac-
curacy statistics, fitness functions.
1. Introduction
Building an estimation model from existing data in-
volves finding a line of best fit for that data. This involves
minimizing some error criterion, or “fitness function”. One
model is better (“fitter”) than another if it yields a lower
value for the chosen fitness function. For example, the er-
ror criterion used in ordinary least squares regression is the
sum of squared deviations.
Though models are commonly built by minimizing
squared errors, they are not generally evaluated that way.
Numerous accuracy statistics have been proposed. In the
software engineering literature it is common to evaluate an
estimation model in terms of its mean magnitude of relative
error (MMRE), or the fraction of estimates within a given
percentage of the actual value (Pred(l)).
This leads to the question: how do estimation models for
typical software engineering data fare, on various common
accuracy statistics, when they are derived using different fit-
ness functions e.g. minimizing the sum of absolute devia-
tions, minimizing MMRE, maximizing Pred(0.25)?
Models that minimize squared deviations naturally give
greatest weight to large systems, where the scope for large
estimation errors is greatest. Models that minimize relative
errors naturally give greatest weight to the most common
types of systems. If large systems were also the most com-
mon, these two approaches would coincide. But software
engineering data sets are generally skewed towards smaller
systems, so there is a trade-off to be evaluated.
In this study, estimation models are built using a variety
of fitness functions, and evaluated using a wide range of ac-
curacy statistics. Data is sourced from the ISBSG data set.
Genetic programming is used to derive the models, because
it can work with arbitrary fitness functions.
The aim is observe how the models compare across a
range of fitness functions, and thus to gain some under-
standing of which fitness functions are generally most ef-
fective with software engineering data.
The rest of this paper is organized as follows. Section 2
discusses related work. Section 3 gives an outline of ge-
netic programming (GP) and its use in software estimation,
and explains why it is a suitable vehicle for this study. Sec-
tion 4 describes the accuracy statistics used here to evaluate
models, and the fitness functions used to derive the models.
Section 5 describes the research method used. Results are
presented and discussed in Section 6. Section 7 discusses
threats to the validity of this work. Conclusions and com-
ments on possible future work are given in Section 8.
2. Related work
There are two types of work that are related to this study:
investigations of the use of GP for software estimation, and
studies concerning what might be termed the “infrastruc-
ture” of software estimation models.
Investigations of GP for software estimation are de-
scribed in section 3.2, following a description of GP.

11th IEEE International Software Metrics Symposium (METRICS 2005)

1530-1435/05 $20.00 © 2005 IEEE
 The last few years have seen several studies related to the
development and evaluation of software estimation models.
Issues investigated include data imputation methods to han-
dle missing data [3]; the number of repetitions needed to
draw reasonable inferences [8]; the use of simulated data
sets to investigate how different analysis methods influence
the results [13, 16]; and what different accuracy statistics
actually measure [9].
MMRE has received particular attention, since it is so
commonly used as an accuracy statistic. Miyazaki noted
that MMRE is lower for models that underestimate [12].
Kitchenham et al studied MMRE theoretically to identify
what it really means, and noted that it is a measure of the
spread of z=(estimate/actual) [9], rather than a measure of
prediction accuracy. Foss et al used an example and sim-
ulations to show that MMRE can lead to counter-intuitive
results [6], and concluded that it is unsafe to use MMRE as
an accuracy statistic.
Some authors have commented on different fitness func-
tions. It has been noted that the choice of fitness function
affects the results [2, 11]; and that the choice of fitness
function will bias the search towards similar accuracy mea-
sures [11]. Dolado [5] observed that correlation coefficient
is not effective as a fitness function, and preferred mean
squared error; this is the only direct reference I am aware
of concerning the relative merit of different functions.
3. Genetic programming
3.1. Outline of GP
Genetic programming is a technique for tackling opti-
mization problems. A brief outline is presented here; for
more information see [10].
The idea is based on the theory of evolution (“survival
of the fittest”). Genetic operations on chromosomes lead to
fitter individuals, which are more likely to survive. Over
time, the population as a whole improves.
In this context, a “chromosome” is a symbolic expres-
sion that calculates one output value (an estimate of soft-
ware project effort) from the values of some input param-
eters (measurements of various project characteristics). A
population of individuals means a collection of different es-
timation functions, some of which are “fitter” (give better
estimates) than others.
The GP process starts with the random generation of
some estimation functions. Probably none of them is much
good, but some will be fitter than others. They provide a
starting point for evolution.
The fitness of an individual is evaluated using a “fitness
function”. A fitter individual is closer to the optimal solu-
tion — in this case, one that gives more accurate estimates
11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
of effort. The choice of fitness function is one of many con-
trol parameters to be set when preparing for a GP run.
A new generation is formed from an existing generation
through the application of three genetic operations. Repro-
duction involves identifying fitter individuals from the cur-
rent population, for inclusion in the next population. There
are several ways in which they might be selected; tourna-
ment selection was used here. Crossover involves taking
two chromosomes, snipping out a sub-expression from each
one, and interchanging them. Mutation involves making a
random change somewhere within a chromosome.
The aim is that over many generations, functions will
evolve that estimate effort well. The process continues until
the fitness of the best individual converges, or a specified
number of generations is reached. The fittest individual in
the final generation is taken as the best solution for that run.
The whole process is normally repeated several times, using
different seed values for the pseudo-random number gener-
ator.
Key issues in GP include defining the fitness function,
defining the forms that the chromosomes can take, and
maintaining enough diversity in the population to allow
eventual convergence to a global optimum rather than set-
tling on a local optimum. The first of these is the point of
investigation here; the second is handled here by defining a
grammar for chromosomes; the last depends largely on the
settings of control parameters.
3.2. Application of GP in software estimation
Dolado [5] was the first to investigate the suitability of
GP as a method for deriving software estimation models.
He analyzed several small published data sets, finding mod-
els to estimate effort from a single independent variable,
project size. Mean squared error was used as the fitness
function, and MMRE and Pred(0.25) as the accuracy statis-
tics.
Burgess and Lefley [2] studied Desharnais’ data set of
81 projects, in which 9 independent variables were avail-
able for use. They minimized MMRE as a fitness function,
and used several accuracy statistics. They found that GP
performed well compared to other methods, and that it mer-
ited further investigation, particularly to explore the effects
of various parameter settings.
Shan et al [15] studied 423 projects, from Release 7 of
the ISBSG data set. MSE was used as both fitness function
and accuracy statistic. In a small experiment, using separate
training and testing sets, they found that GP out-performed
OLS regression. They used Grammar Guided Genetic Pro-
gramming [17], experimenting with different grammars to
describe the allowed forms of the chromosomes. Writ-
ing a grammar supplies both a way to represent syntactical
constraints on the chromosomes, and a way to incorporate
 background knowledge to guide the search process.
Lefley and Shepperd [11] studied the “Finnish data set”
of 407 projects. Their fitness function was root average
mean squared error. They considered several accuracy
statistics. They used a holdout sample to evaluate the ac-
curacy of their models, with chronological splitting as the
basis for forming the training and evaluation sets: projects
up to a given date formed the training set, and projects after
that date the testing set.
Though these studies vary in data set size, independent
variables, fitness function, accuracy statistics, and valida-
tion procedures, they all indicate that GP has promise as a
method for finding estimation models.
A particular advantage of GP is its ability to explore a
vast search space of possible equations [5]. Another ad-
vantage is that no assumptions need to be made about the
distributions of the data; GP can make its own decisions
about dealing with outliers and the selection of relevant at-
tributes [2].
A challenge with GP is the need to determine appropriate
settings for numerous control parameters; this can take con-
siderable experimentation and may have a significant effect
on the results [2]. The fitness function is one of those pa-
rameters. Once sensible settings are known for the other pa-
rameters, it is easy to experiment with various fitness func-
tions — hence the adoption of GP in this research.
The point here is not to assess the suitability of GP as
a vehicle for building estimation models, but rather to use
it as a mechanism for creating models that are optimized
towards different things.
4. Fitness functions and accuracy statistics
This section describes the accuracy statistics used to
evaluate models in this study, and the fitness functions used
to build the models.
4.1. Accuracy statistics
These accuracy statistics were used to evaluate estima-
tion models in this study:
ʾ : The square of the coefficient of correlation between
estimated and actual values.
MSE: Mean squared error — used as an accuracy statistic
by [15].
StDev: Standard deviation of error. Recommended by [6]
as an accuracy statistic; almost identical to root mean
squared error, used as an accuracy statistic in [11].
MMRE: Mean magnitude of relative error. Widely used as
an accuracy statistic since its description in [4]. Me-
dian MRE is also reported here.
11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
MMER: Mean magnitude of error relative to the estimate.
Proposed in [9], argued as intuitively preferable to
MMRE since at the time of estimation one wants to
know what errors to expect relative to the estimate.
Median MER is also reported here.
Mean Þ : Kitchenham et al [9] recommended that accuracy
should be measured in terms of z=(estimate/actual).
It is asymmetric and favours models that minimize
overestimates (which could be a problem since over-
estimates are usually less serious than underesti-
mates). The full distribution of z should be consid-
ered when comparing prediction systems. Median
of Þ is also reported here, as well as the mean of
1/z=(actual/estimate).
Pred(l): The proportion of estimates that are within a given
percentage of the actual value. Widely used as an ac-
curacy statistic since its description in [4], particularly
in conjunction with MMRE. Pred(0.25) and Pred(0.50)
are reported here.
Means and medians of absolute errors are also reported.
Foss et al [6] recommend LSD and RSD as accuracy
statistics. These are intended as estimates of the standard
deviation of the error term. Neither could be used here.
LSD involves logarithms of estimated values, so cannot be
computed for any model that generates any negative esti-
mates; several such models emerge in this study. RSD can
only be used if estimates are based on a single predictor
variable.
4.2. Fitness functions
The fitness functions chosen for study were selected ei-
ther because they are commonly used in building estimation
models, or they are commonly used in evaluating estimation
models.
The functions chosen were:
¯ MSE: since it underpins ordinary regression, and is re-
lated to ʾ and SD.
¯ LAD (least absolute deviation): an obvious alterna-
tive to ordinary regression, proposed centuries ago but
overshadowed by least-squares regression because it
was harder to compute and lacked theoretical under-
pinnings — both addressed now [1].
¯ MRE: because it is so common as an accuracy statis-
tic, and has also been used elsewhere as a fitness func-
tion [2].
¯ MER and Z: since they are proposed as better alterna-
tives to MRE.
 ¯ Pred(l), since it is so widely used as an accuracy statis-
tic.
Writing a fitness function consists of specifying how the
error term is calculated for a single data point, given its
known and estimated values. The GP software used here
always tries to minimize the overall fitness value, so the fit-
ness function needs to be written so that smaller values are
better.
Functions are easily writen for differences between esti-
mated and actual values:
MSE: error = (estimate - actual)¾
LAD: error = abs (estimate - actual)
MRE: error = abs ( (estimate - actual) / actual )
MER: error = abs ( (estimate - actual) / estimate )
is trickier, since its optimum value is 1, not zero. Com-
puting and subtracting 1 does not work, since negative val-
ues are possible and minimization will do the wrong thing.
abs(z-1) is no help, as that is MRE. As an approximation,
the following function was used:
Z: error = max(estimate, actual)/min(estimate, actual)
This is always at least 1, and it makes sense to minimize it:
the closer it is to 1, the closer the estimate and actual values
are to each other.
Pred(l) is a blunt instrument. It is insensitive to the de-
gree of inaccuracy of estimates that fall outside the desired
range. Rather than simply optimizing Pred(l) without regard
to the inaccurate estimates, functions were investigated that
would focus first on maximizing Pred(l), but would resolve
ties by considering a second criterion: MSE, LAD, or MRE.
As noted above, the GP software used here assumes that
small fitness values are better than large values: the fitness
function needs to be minimized, not maximized. Thus max-
imizing Pred(l) needs to be re-cast as minimizing the num-
ber of projects for which the absolute relative error is not

. This is done by assigning a very high penalty (10½¼ )
to estimates where the error exceeds
. The penalty value
overwhelms the second criterion; hence the first priority in
optimization is to minimize the number of estimates that re-
ceive that penalty. Estimation equations that have the same
Pred(l) value are distinguished by the second criterion.
The following functions were used:
Pred/LAD: error = abs(diff) + 10½¼ if MRE 

error = abs(diff) if MRE

Pred/MSE: error = diff¾ + 10½¼ if MRE 

error = diff¾ if MRE

11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
Pred/MRE: error = MRE + 10½¼ if MRE 

error = MRE if MRE

where diff = estimate - actual
and MRE = abs(diff)/actual
5. Research method
5.1. Data set description
The analysis presented in this paper was based on soft-
ware projects from Release 8 of the ISBSG data set1 .
Release 8 had data on 2,027 projects, with over 50 data
attributes recorded for each project. These attributes cap-
ture the nature of the project itself, the development envi-
ronment and techniques used, and strengths and weaknesses
perceived in individual projects. Most are known early in a
project, which is important for estimation.
Not all of the 2,027 projects were appropriate for inclu-
sion in our analysis:

Projects were excluded if they were not assigned a high
data quality rating (A or B) by ISBSG.

Projects were excluded if their size was measured in
anything except IFPUG function points.

Projects were excluded if their effort values covered
more than just the development team only.

Projects were excluded if they had any missing data on
four key ratio-scaled attributes: size, effort, team size,
and duration.

Projects whose normalized effort differs from recorded
effort by more than 50% were excluded (normalized
effort is ISBSG’s estimate of effort over an entire
project, if reported effort only covers some phases of
the life cycle).

Projects were excluded if their size was below about 50
function points, or their effort below 100 hours. The
experience of practitioners is that projects below these
sizes are too variable to be included in data sets for
effort estimation models.

Projects were excluded if they were identified as out-
liers, or as high leverage points, on one or more of the
ratio-scaled attributes. The rationale for this was to
avoid the accuracy statistics being unduly influenced
by a small number of extreme values. Outliers were
identified using box plots; leverage points by the in-
fluence.measures function in the statistical lan-
guage R.
1 www.isbsg.org
 Variable Scale Description Variable Min Median Mean Max StDev
Effort Ratio Normalized project effort Size 46 233 385 3120 435
in person hours Effort 107 1943 3506 22050 4115
Size Ratio Application size in IFPUG Team size 1 5 6.85 43 6.20
adjusted function points Duration 1 4 7.58 30 5.02
Team size Ratio Maximum team size
Duration Ratio Project duration in elapsed Table 2. Summary of ratio-scaled variables
calendar months
OrgType Nominal Organization type (eg Banking,
Insurance, Manufacturing)
BusArea Nominal Business area to which the 5.2. Experimental design
application contributes
AppType Nominal Application type (eg MIS) The experiment took the form of a complete block de-
DevType Nominal Type of development (eq New sign [7, 18]. There is one factor, with 8 different treatments:
development, enhancement)
the fitness function, with the 8 functions being compared.
Platform Nominal Mainframe, midrange, PC
There are 634 objects: the 634 projects. Each treatment is
LangType Nominal Language type used (eg 3GL)
Language Nominal Specific main language used applied to each of the 634 objects; that is, an estimation
Year Ordinal Year of project completion model obtained by optimizing each of the fitness functions
is applied to each one of the projects. Thirty repetitions are
Table 1. Variables used in this study made for each treatment.
Applying every treatment to every object is inappropri-
ate if there is any question of learning, whereby participants
learn during earlier treatments and this affects later treat-
634 projects remained for study. These had high data ments. There is no issue of that here, when the objects are
quality ratings, consistent and comparable definitions, and inanimate — software development projects. Randomiza-
were not at the extreme high or low ends of the range. tion is not important in the experiment design for this rea-
Twelve variables were retained for each project. They son.
are described in Table 1. Normalized project effort is the Applying every treatment to every object, instead of us-
dependent variable. The independent variables are the other ing separate training and testing samples or a cross- valida-
three ratio-scale variables (size, team size, duration), and tion process, would also be inappropriate if absolute esti-
eight categorical variables (year of implementation, though mates were sought for the performance of each treatment.
a number from 1989 to 2002, is treated as an ordinal-scale The estimated accuracy would be too high. But in this situ-
value rather than a number on which arithmetic might be ation we seek relative comparisons, and the same situation
performed.) The independent variables are those believed applies for every treatment, so the complete block design
likely to have some explanatory value for project effort, seems appropriate.
which had not too many missing values.
Intuitively, one would expect some relationships to ex- 5.3. Experimental method
ist between size, team size, and duration. This could be
a problem, because if “independent” variables are strongly The GP software used was DCTG-GP [14]. Each sepa-
correlated they should not be included together in statisti- rate GP run generated one estimation model, based on one
cal estimations. However, in this data set the correlations fitness function.
between the independent variables are not strong enough to The parameters used for each run are given in Table 3.
be a problem (the highest is 0.38, between size and dura- The parameter values were chosen on the advice of expe-
tion); they are all correlated much more strongly with the rienced colleagues; where possible they match the settings
dependent variable. Thus it is reasonable to treat them as used in [2, 11].
independent. Thirty repetitions were made for each of the 8 treatments.
Summary statistics for the numeric variables are pre- In other words, thirty GP runs were made with each of the
sented in Table 2. 8 fitness functions, making 240 runs in total.
Raw numeric values are used, not log-transformed val- For each of the eight fitness functions, the estimated ef-
ues. None of the variables is normally distributed, as can fort for each project was calculated as the average of the
be seen in Table 2 where the mean values are all well above 30 individual estimates for that project from the 30 differ-
the medians. This would be a problem for ordinary statisti- ent estimation models built using that fitness function. Eight
cal regression, but is not a problem here since GP makes no vectors result, each of 634 elements, containing the overall
assumptions about the distributions of the data. estimated effort for each project from each fitness function.

11th IEEE International Software Metrics Symposium (METRICS 2005)

1530-1435/05 $20.00 © 2005 IEEE
 Parameter Value
Population size 1000
Number of generations 500
Number of runs 30
Maximum initial tree depth 5
Maximum tree depth 9
Probability of crossover 0.1
Probability of mutation 0.1
Tournament size 3 Pred(0.25). Other fitness functions, particularly LAD and
Table 3. Parameters used for GP system
Each vector was compared with the vector of actual ef-
fort values, and a variety of accuracy statistics calculated.
Using the individual estimation models, analysis of vari-
ance was used to identify significant differences between
the accuracy from each fitness function on each accuracy
statistic.
6. Results and discussion
6.1. Average performance
Table 4 summarizes the actual and estimated effort val-
ues. Table 5 presents the accuracy obtained with each fit-
ness function on each accuracy statistic.
From Table 4 we see that the distribution of effort esti-
mates from each fitness function broadly reflects the distri-
bution of actual effort.
MER are MRE are effectively reciprocals, so inevitably
they work in opposition. When MRE is used as the fitness
function, MMRE is best and MMER worst, and vice versa.
MER tends to underestimate significantly (as noted by
[2] and [12]), and MER to overestimate significantly. This
is at least partly an artefact of their definitions. One would
hope that the goal of minimizing MER is achieved by mini-
mizing the numerator (the difference between the estimated
and actual values). But the denominator is also relevant.
Underestimates produce smaller denominators than overes-
timates, so MER penalizes underestimates and favours over-
estimates. In the case of MRE, MRE is capped at 1 if the
estimate is too low, but is unbounded for an overestimate,
so MRE is biased towards underestimates.
The summary values for the Pred(l) fitness functions
look similar to those from the other fitness functions, and
the performance of those functions appears quite reasonable
in Table 5, but this is more good luck than genuine accuracy.
The generated estimation models include some that grossly
underestimate effort for the 63% or so of projects that do not
fall within the 25% target, and some that grossly overesti-
mate effort. It happens that the number of underestimates
roughly balances the number of overestimates. Attempting
to optimize Pred(l) produces unstable results.
11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
Table 5 shows some expected behaviour: ʾ , MSE and
Standard deviation of error are all best when MSE is the
fitness function. MMRE is best when MRE is the fitness
function. MMER is best when MER is the fitness function.
Mean Þ is best when an approximation to Þ is the fitness
function.
This does not apply to the fitness functions based on
Z, do better. This may be because Pred(l) is a step function
rather than a continuous function, which makes it harder to
evolve an optimum solution. The output from the individ-
ual GP runs suggests that convergence on a local optimum
often happened early, and the system was unable to move
away from it to a global optimum. Combining this with the
observation above that results from the Pred(l) fitness func-
tions were unstable, it seems that Pred(l) should not be used
as a fitness function.
The first row of Table 5 supports Dolado’s observation
that correlation coefficient is not an effective fitness func-
tion [5]. There is very little difference between any of the
ʾ values, even when MSE, StDev, and absolute errors vary
by large amounts.
There is nothing to choose between any of the fitness
functions when accuracy is measured in terms of Median
MER and Median MRE (except for MER penalizing MRE
and vice versa).
There is little to choose between the fitness functions
when Pred(0.25) and Pred(0.50) are used to measure accu-
racy. No model does well, and all achieve roughly the same
low level of poor performance.
LAD and Z do well across the board. They do almost as
well as MSE on accuracy measures based on squared errors.
They have the best absolute errors, and their Þ values are
closest to 1.0. Their values for MRE are not as good as
for the function designed to minimize MRE, but they do
notably better than the MSE fitness function.
It appears that improvements to MSE occur at the ex-
pense of MRE, and vice versa. This is not surprising in
software engineering data, where the most common projects
are not the large projects.
Comparing the absolute fitness functions (MSE and
LAD) with the relative fitness functions (MER and MRE),
on almost every accuracy statistic the absolute fitness func-
tions perform as well or better. The exceptions are the accu-
racy statistics for which the relative fitness functions were
specifically designed. This suggests that absolute fitness
functions should be preferred to relative fitness functions —
a nice outcome, given that regression using MSE and LAD
is easily done with statistical tools nowadays.
Another explanation could be merely that GP is better
at minimizing direct values than ratios. There is certainly
more variability in the models built with MRE and MER,
and more frequent evidence of premature convergence to
 Accuracy statistic MSE LAD MRE MER Z P(.25)/ P(.25)/ P(.25)/
MSE LAD MRE
Adjusted Ê 0.681 0.679 0.653 0.650 0.662 0.655 0.651 0.657
MSE (x 10
) 5.388 5.602 8.243 8.808 6.040 5.584 6.134 5.846
StDev of error 2323 2369 2873 2936 2460 2421 2479 2420
Mean abs error 1413 1368 1596 1694 1385 1410 1435 1416
Median abs error 752 640 708 870 620 691 698 711
Mean Þ 1.489 1.287 0.863 1.753 1.215 1.424 1.493 1.432
Median Þ 1.157 1.019 0.684 1.344 0.956 1.117 1.154 1.123
Mean 1/Þ 0.976 1.115 1.676 0.817 1.174 1.012 0.969 1.011
Mean MRE 0.702 0.580 0.466 0.877 0.545 0.655 0.702 0.665
Median MRE 0.354 0.343 0.406 0.422 0.347 0.353 0.352 0.354
Mean MER 0.431 0.470 0.832 0.401 0.495 0.438 0.423 0.439
Median MER 0.352 0.351 0.532 0.356 0.356 0.352 0.345 0.348
Pred(0.25) 0.383 0.371 0.282 0.341 0.361 0.363 0.371 0.360
Pred(0.50) 0.610 0.663 0.640 0.555 0.686 0.633 0.612 0.631

Table 5. Accuracy statistics

the wrong optimum, so perhaps the relative fitness functions

are being judged unfairly. More experimentation is needed
MSE LAD MRE
to understand GP’s behaviour with these fitness functions.
Minimum 308 190 178
1
quartile 1253 1080 681 6.2. Statistical significance
Median 2298 1987 1352
3 quartile 4426 3962 2783 The observations above apply to the overall average
Maximum 25220 24120 19510
model found from the 30 repetitions for each fitness func-
Mean 3530 3159 2250
StDev 3369 3136 2428
tion. To test the statistical significance of the differences,
analysis of variance was used. For example, MMRE was
MER Z P(.25)/ computed separately for 150 estimation models (models
MSE built with Pred(l) were excluded from this stage of the anal-
Minimum 318 223 261 ysis), and ANOVA used to test the hypothesis that fitness
1
quartile 1476 1017 1253 function has no effect on MMRE; this approach was taken
Median 2659 1894 2220 for all 14 accuracy statistics. In every case the null hypoth-
3 quartile 5126 3734 4191 esis was decisively rejected (p 10 ½½).
Maximum 22990 24560 24050 Multiple comparisons were then made to identify which
Mean 4347 3103 3364 fitness functions produced results that differed significantly
StDev 4730 3052 3208 (0.05 chosen as the level of significance). The results are
summarized in Table 6. In each row, the five fitness func-
P(.25)/ P(.25)/ Actual
tions are listed in order of performance, from best to worst;
LAD MRE effort
Minimum 323 347 107
functions whose performance is not significantly different
1
quartile 1299 1282 917 at the 0.05 level are bracketed together.
Median 2253 2194 1943 As noted in the observations based on overall averages,
3 quartile 4258 4101 4115 the performance of the models found with MRE and MER
Maximum 28740 23760 22050 as fitness functions is universally worst, except on the ac-
Mean 3599 3337 3506 curacy statistics for which they were specifically designed.
StDev 3810 3174 4115 MRE and MER may have their place as accuracy statistics,
but not as fitness functions. It appears clear that fitness func-
Table 4. Effort estimates (Hours) with each tions based on MSE, LAD and Z perform better across a
fitness function wide range of accuracy statistics.
Considering only MSE, LAD, and Z, look again at Ta-
ble 6. The rows are ordered in decreasing order of how
much difference there is between the performance of the

11th IEEE International Software Metrics Symposium (METRICS 2005)

1530-1435/05 $20.00 © 2005 IEEE
 Accuracy statistic Performance of fitness functions
Mean MRE MRE, Z, LAD, MSE, MER
Median abs error Z, LAD, MRE, MSE, MER
Mean Þ Z, LAD, MSE, MRE, MER
Mean MER MER, MSE
, LAD, Z
, MRE
Median Þ LAD, Z, MSE, MRE, MER
Mean 1/Þ MSE, LAD, Z
, MER, MRE
MSE (x 10 ) MSE, LAD, Z, MRE, MER
Pred(0.50) Z, LAD, MRE, MSE, MER
StDev of error MSE, LAD, Z, MRE, MER
Median MRE MSE, LAD, Z
, MRE, MER
Adjusted Ê
LAD, MSE
, MRE, Z
, MER
Mean abs error LAD, MSE, Z
, MRE, MER
Median MER LAD, MSE
, Z, MER
, MRE
Pred(0.25) LAD, MSE
, Z, MER, MRE
Table 6. Performance of fitness functions
three fitness functions. For mean MRE the difference be-
tween the average performance of Z (best of the 3) and MSE
(worst of the 3) is 0.16. In a range from 0.55 to 0.71, that
is a variation of 22%. At the other end of the table the dif-
ference between the average performance of LAD and MSE
(best) and Z (worst) represents a variation of about 4%.
This ordering suggests two things. One is a rough indi-
cation of which accuracy statistics provide real discrimina-
tion. The variation is over 10% down to MSE, a bit below
10% for Pred(0.5) and StDev, and 5% or less for the rest.
The other is that Z looks like the best of MSE, LAD and
Z on accuracy statistics with large variation, and the worst
of the three on accuracy statistics with small variation. In
other words, when it outperforms other fitness functions it
is by a large margin, but when it loses it is not by much.
This implies that Z might be the best fitness function to use
for building estimation models, as well as being argued [9]
to be the best statistic to use for assessing models.
6.3. Practical implications
It is all very well to suspect that certain fitness functions
are better than others, but you need to be able to apply that
knowledge in practice to build useable estimation models.
One approach is to use GP as outlined here to generate a
collection of estimation models from past data, using your
preferred fitness function. These models can be applied to
a new project, giving a set of estimates on which an overall
estimate can be based. You would have no way of knowing
whether the particular collection of models happened to be
a very good, or very bad, or quite typical set. However,
inspection of the output from GP runs based on MSE, LAD,
and Z showed the results to be very stable, so these fitness
functions should give trustworthy behaviour. The best and
worst performed models built using these fitness functions
11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
were very close to the average on most accuracy statistics
(in fact, in many cases the average of 30 models performs
better than the best single model, as errors cancel each other
out). GP runs based on MRE, MER and Pred(l) were less
stable.
Any fitness function can be used like this if a GP system
is available. If one is not, some algorithm is required en-
abling an estimation model to be built directly instead of by
search. In that light, MSE and LAD are more useful than
Z, since software is readily available to perform regressions
based on MSE and LAD.
The nature of software engineering data sets is that there
are usually many more small projects than large ones. MSE
gives greater weight to large projects, where there is more
scope for large estimation errors. By not squaring the er-
ror term, LAD gives less weight to large projects and rela-
tively more weight to smaller projects. Perhaps this makes
it more suitable than MSE for building estimation models
from software engineering data sets, and explains its better
performance on most accuracy statistics. On the same ba-
sis, generalized linear models, such as Huber’s glm in R
and S-Plus, may be worth investigating.
Naturally, though, each user needs to decide which ac-
curacy statistics matter most to them in their particular sit-
uation, which might lead to another fitness function being
preferred.
6.4. Not surprising?
In one sense it is not surprising that LAD outperforms
MSE as a fitness function on heavily skewed data. Statis-
ticians note that “The criterion of minimizing the sum of
absolute deviations is preferable to that of least squares in
the presence of large disturbances (outliers) or . . . heavy
tails” [19]. The reason is that confidence intervals for the
regression coefficients are smaller.
That reason is not directly relevant to a software esti-
mator, who cares really about the performance of the equa-
tion on various accuracy statistics. We have provided ex-
perimental evidence that the theoretical advantage of LAD
translates to better performance on most of the accuracy
statistics commonly used in software estimation.
7. Threats to validity
ISBSG data is a convenience sample, not necessarily rep-
resentative of software engineering data in general. This
may mean these findings do not generalize. But it is typical
of software engineering data in its skewed nature, and it is
that aspect of the data that is most important here, so that
risk should not be a problem.
A greater problem could be the reliability of some of the
data values themselves. In particular, normalized effort is
 an estimate of full life-cycle effort, for projects that do not
include all phases. Where normalisation is done, it is an
estimate. Where it is not done, it may not mean that all
phases are covered: it might mean that no information about
phases was available. Thus there is some uncertainty in the
effort values used. One could choose to use only projects for
which full phase data was available in the first place; I chose
not to do that because it leaves a much smaller data set.
Projects whose effort was changed the most were excluded
from the data set; perhaps a smaller threshold should be
used to improve the reliability of the effort data. This can
only be determined by further experimentation.
Different settings of the GP control parameters can influ-
ence behaviour. Other settings might lead to different mod-
els being found. However, it is likely that this would affect
speed of convergence, rather than accuracy. 500 generations
were used here; the accuracy gain after about 150-200 gen-
erations was generally small, so I believe that convergence
is reasonable and the risk here is low. Sometimes, partic-
ularly with the relative fitness functions, it seemed that GP
was unable to escape from a local optimum. This behaviour
may be improved with more experimentation with control
parameters.
GP was used here to find non-linear models, using un-
transformed data. But the usual approach to regression with
software engineering data is to use log-transforms to obtain
normally distributed data, and then to derive linear mod-
els. We may see different behaviour if the fitness functions
studied here were applied to log-transformed data. This is
an area for future work.
Using the same sample (the complete data set) for build-
ing and evaluating the models could affect the reliability of
the accuracy statistics. It is possible that models are over-
fitted to the data. In particular, this could affect Section 6.2,
which drew inferences from the actual differences observed
between model performances. But all 8 treatments should
be equally affected if that is the case, so this should not in-
validate the results.
8. Conclusions
Building a model from data involves minimizing some
criterion (fitness function). When building estimation mod-
els from software engineering data sets, this has tradition-
ally been done by least-squares regression, which mini-
mizes MSE — even though MSE is unrelated to most of
the accuracy statistics used to assess the models.
We have investigated the performance of models built
using a variety of fitness functions, across a range of com-
monly used accuracy statistics. The models were evolved
using genetic programming, using data from the ISBSG
data set. Untransformed data was used directly.
Fitness functions based on Pred(l) should not be used.
11th IEEE International Software Metrics Symposium (METRICS 2005)
1530-1435/05 $20.00 © 2005 IEEE
They are too unstable. Pred(l) is insensitive to how bad the
errors are in estimates that fall outside the target accuracy,
so they run the risk of generating hugely inaccurate esti-
mates.
Fitness functions based directly on differences between
estimates and actual values, rather than on relative errors,
appear to perform better across a range of accuracy statis-
tics. The exception is a fitness function based on an approx-
imation to Þ , the ratio of estimated to actual effort; in many
respects it was the best-performed fitness function of all. Its
disadvantage is that no algorithm is available to generate an
estimation model directly that optimizes this statistic.
This leaves MSE and LAD as two fitness functions that
can be easily used in practice and perform well across sev-
eral accuracy statistics. In software engineering data sets,
where small projects generally outnumber large ones, this
research suggests that LAD generally out-performs MSE.
One direction for future work is to extend this work to
look at log-transformed data, as is common with software
engineering models. Another is to use separate subsets for
building and evaluating models, to get more reliable esti-
mates of actual differences between their performance.
Acknowledgements
I would like to thank ISBSG for the use of its data for this
research. Bob McKay and Yin Shan helped with writing
the grammar, setting up the GP system and setting control
parameters. The anonymous reviewers provided valuable
comments and suggestions.
References
[1] D. Birkes and Y. Dodge. Alternative Methods of Regression.
Wiley, 1993.
[2] C. Burgess and M. Lefley. Can genetic programming im-
prove software effort estimation? A comparative evaluation.
Information and Software Technology, 43(14):863–873, De-
cember 2001.
[3] M. Cartwright, M. Shepperd, and Q. Song. Dealing with
missing software project data. In Proceedings of 9th Interna-
tional Software Metrics Symposium, pages 154–165. IEEE,
September 2003.
[4] S. Conte, H. Dunsmore, and V. Shen. Software Engineering
Metrics and Models. Benjamin-Cummings, 1986.
[5] J. Dolado. On the problem of the software cost function.
Information and Software Technology, 43(1):61–72, January
2001.
[6] T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit. A
simulation study of the model evaluation criterion MMRE.
IEEE Transactions on Software Engineering, 29(11):985–
995, November 2003.
[7] N. Juristo and A. Moreno. Basics of Software Engineering
Experimentation. Kluwer, 2001.
 [8] C. Kirsopp and M. Shepperd. Making inferences with small
numbers of training sets. IEE Proceedings – Software,
149(5):123–130, October 2002.
[9] B. Kitchenham, L. Pickard, and S. MacDonell. What accu-
racy statistics really measure. IEE Proceedings – Software,
148(3):81–85, June 2001.
[10] J. Koza. Genetic Programming: On the Programming of
Computers by Means of Natural Selection. MIT Press, Cam-
bridge MA, USA, 1992.
[11] M. Lefley and M. Shepperd. Using genetic programming
to improve software effort estimation based on general data
sets. In Proceedings of GECCO 2003, volume 2724 of
LNCS, pages 2477–2487. Springer-Verlag, September 2003.
[12] Y. Miyazaki, M. Terakado, K. Ozaki, and H. Nozaki. Ro-
bust regression for developing software estimation models.
Journal of Systems and Software, 27(1):3–16, October 1994.
[13] L. Pickard, B. Kitchenham, and S. Linkman. Using sim-
ulated data sets to compare data analysis techniques used
for software cost modelling. IEE Proceedings – Software,
148(6):165–174, December 2001.
[14] B. Ross. Logic-based genetic programming with definite
clause translation grammars. New Generation Computing,
19(4):313–337, 2001.
[15] Y. Shan, R. McKay, C. Lokan, and D. Essam. Software
project effort estimation using genetic programming. In
Proceedings of ICCCAS’02 (International Conference on
Communications, Circuits and Systems), pages 1108–1112,
Chengdu, China, July 2002.
[16] M. Shepperd and G. Kakoda. Using simulation to evaluate
prediction techniques. In Proceedings of 7th International
Software Metrics Symposium, pages 349–358. IEEE, 2001.
[17] P. Whigham. Grammatically-based genetic programming.
In J. Rosca, editor, Proceedings of the Workshop on Ge-
netic Programming: From Theory to Real-World Applica-
tions, pages 33–41, Tahoe City, California, USA, July 1995.
[18] C. Wohlin, P. Runeson, M. Host, M. Ohlsson, B. Regnell,
and A. Wesslen. Experimentation in Software Engineering:
an Introduction. Kluwer, 2000.
[19] S. Zanakis and J. Rustagi. Introduction to contributions in
regression and correlations. In S. Zanakis and J. Rustagi, ed-
itors, Optimization in Statistics, pages 7–9. North-Holland,
1982.

11th IEEE International Software Metrics Symposium (METRICS 2005)

1530-1435/05 $20.00 © 2005 IEEE
View publication stats