Polish J. of Environ. Stud. Vol. 16, No.

5B (2007), 72-76

Explaining Performance of the Threshold

Accepting Algorithm for the Bin Packing
Problem: a Causal Approach
J. Pérez Ortega1, L. Cruz Reyes2, V. Landero Najera1, R. Pazos Rangel1,
V. Pérez Rosas2, G. Zarate Rivera2, G. Reyes Salgado1
1
Departamento de Ciencias Computacionales, Centro Nacional de Investigación y Desarrollo
Tecnológico (CENIDET), AP 5-164, Cuernavaca, México
2
División de Estudios de Posgrado e Investigación y Departamento de Sistemas y Computación,
Instituto Tecnológico de Ciudad Madero (ITCM), Tamaulipas, México

Abstract

This paper presents an algorithm performance characterization and its analysis in order to find
explanations through causal models. A systematic development of a causal model for the Threshold
Accepting algorithm performance is shown, when it solves the bin packing problem. Observation of
factors that intervene in its behavior and the analysis of solutions space from the solved instances were
used to create a causal model. A detailed model development and its interpretation are presented. This
kind of modeling could offer elements for a better understanding of algorithms performance and as
consequence their redesign.

Keywords: causal modeling, performance characterization, performance explanation

Introduction computer sciences there are a few works that incorporate

The majority of works related to performance of
optimization algorithms is focused on it demonstration
through comparative experimental results, however it is
better to explain performance beside of showing it. The
performance explanation depends on analyzing and
understanding factors that can affect it. The analysis of
single factors can show some influences in instances for
which the performance was adequate or inadequate. The
problem of this analysis is that it does not show
accurately interactions between factors, these are needed
to provide explanations that permit a major knowledge
of how the founded relations are. Causal modeling is
a generalized form to represent knowledge.
The construction of causal models has been studied
mainly in biological and social sciences. Nevertheless, in
this kind of analysis. In this sense, this paper attempts to
contribute to the growth of this area presenting a systematic
development of a causal model for performance of opti-
mization algorithms. The approximation algorithms are the
most promised for optimization problems. They start with
an initial solution and move to another by modifying their
values through random rules based on heuristics.
Performance of Approximation Algorithms
The algorithms performance characterization may be
experimental or theoretical. In the first, the amount of
necessary resources for an algorithm is determined as
a function of the problem size. The randomness associated
to approximation algorithms makes this analysis comp-
licated. The second approach uses the experimentation to
Explaining Performance of the Threshold…
characterize the algorithm with specific instances, for this
reason it is mainly used with approximation algorithms.
The research about methodologies for computational
experimentation is growing so fast and has as an objec-
tive to promote experiments to be important, correct,
replicable and knowledge producers. [1] No matter
which method is selected to characterize the algorithms,
the goal is to understand how the performance depends
on a set of factors that influence it. The acquired under-
standing may lead to better predictions of algorithms
performance in new situations and the discovering of
improved algorithms.
Explaining Algorithm Performance
The explanation of an observed behavior is made
through a transition between three stages: description,
prediction and causality [2]. In the case of explanations for
approximation algorithms behavior, the transition has been
done mainly through the next focuses. At the beginning the
works were focused on the study of performance by the
demonstration of the algorithm superiority on a set of
standard instances [3]. The following works incorporated an
algorithm characterization and some observations about the
performance [4]. Subsequent works added the prediction
possibility based on algorithm and problem characterization
[5]. Recently, some works are trying to create more
complete and complex models that explain which cha-
racteristics affect the performance and how they are related
[6]. The presented paper follows the latest approach,
contributing to the explaining of performance through the
generation of causal models.
Creation of Causal Models for Algorithm
Performance
A causal model is a generalized representation of
knowledge that is obtained by finding dependences
through the data that implies cause-effect relations [6].
One of the most common representations for these
relations is using directed acyclic graphs. The process of
generating causal models is non trivial; Chickering
proved that this problem is NP-hard [7].
Causal modeling generally have four stages: specifi-
cation, estimation, interpretation and evaluation [8]. The
first indicates which variables are causes and which are
effects, the second determines the intensity of founded
causal relations, in the third the results are analyzed and
interpreted, and in the fourth the model is proved to know
its accuracy.
Different methods are used in the estimation phase; it
depends on how the relations are represented during the
specification stage. If they are presented by a graph, the
most common to determine their magnitudes, is to find
the probability distributions of the given graph [9].
In the interpretation phase the most important
relations with the highest magnitude are analyzed. The
73
final explanations are given by the researcher analyzing
the meaning of each variable and their interactions, in
the context of previous knowledge of the problem do-
main.
Related Works
Esposito presented a work using algorithms of induc-
tive learning and techniques of causal inference to disco-
ver causal rules for attributes of relational databases
[10]. The result of this work was a system called
CAUDISCO, in which the process consists of two pha-
ses: inferring the causal structure of the data studying the
associated conditional independences, and to generate
a rules set for relevant dependences using C4.5 classi-
fication algorithm [11].
Lemeire and Dirkx presented a causal modeling applied
to the analysis of parallel algorithms performance, for
detecting the causes of communication process anomalies
[12]. The causal model was generated applying the Tetrad
software for variables related with performance. The
obtained causal rules were given in an informal way.
Maes presented a paradigm to treat the causal multia-
gent models [13]. The investigation treats the effect of
variables obtained by the combination of observational
data and some theoretical suppositions. The authors
presented an algorithm for the identification of causal
effects in contexts in which an agent does not have
complete access to the whole dominion.
The mentioned investigations are focused on the
algorithms performance study, but they do not present a
complete treatment that includes the modeling of charac-
teristics that affect the performance, the evaluation of the
acquired knowledge, and the formal explanation of the
observed behavior. The work of Esposito, although not
focused on the analysis of algorithm, evaluates the
knowledge and incorporates explanations. So, we have
considered combining these proposals to enrich the
algorithm performance analysis.
Causal Model of the Threshold Accepting
Algorithm (TA)
The procedure used to build a causal model for the
approximation algorithm threshold accepting (TA) [14],
applied in the solution of the bin packing problem, is
presented now. The procedure incorporates the main
ideas of Cohen and Spirtes [2, 15].
Step 1: Specification of Causal Model
Identification of Explanatory Variables
Variables derived of the measurement of the
parameters of the TA algorithm and others referring to
the space of solution were analyzed to identify those that
had some effect in the algorithm performance. For this
purpose, questions about the relation between problem
74 Pérez Ortega J., et al.

characteristics and algorithm performance were for-
mulated: Is there a difference in the number of feasible
solutions from space solution for instances in which TA
performs better? How is the variability of the feasible
solutions space of solutions for these instances? Is there
a difference related to the instances which in which TA
performs worst? Are they factors related with the algo-
rithm performance?
TA algorithm was analyzed to identify which aspects
were feasible of measuring during execution time and if
these could offer an answer to the outlined questions.
Also, a sample of the space of solution associated to the
instances was analyzed, to identify those aspects that
could be measured. In such way the following variables
were created, the first five corresponding to algorithm
execution, and the remainder to solution space of the
problem. 1) Average of the initial temperature Ti, 2)
Average of the final temperature Tf, 3) Average of the
number of temperatures Tav, 4) Number of times in
which the algorithm stops by non finding solution Nfs, 5)
the execution number in which TA obtained the best
solution Br, 6) Number of feasible solutions Fs, 7)
Variance of 100 random feasible solutions Vo.
The variables related to TA execution were measured
from 30 executions of the algorithm with each instance
from a set of 1226 bin packing standard instances taken
from OR library [16], each instance has as parameters
the number of items n and their associated weights. The
analyses, so four levels were established for each
explanatory variable.
Graphical Analysis
In the graphical analysis the distributions of proposed
explanatory variables related to response variable perfor-
mance were verified to identify if each variable showed
differences in frequency distribution between it levels.
The result of this analysis showed that variables Fs, Vo,
Tav and Nfs have clear differences related to levels of
variable performance (1: win, 0: lost); while in variables
Ti, Tf and Mc the distributions for both levels were
almost the same, for this reason they were discarded of
future analysis.
Statistical Analysis
Relations between explanatory variables Fs, Vo, Tav
and Nfs with the variable ratio as a response variable were
analyzed. In Figure 1a the relation between the levels of
analyzed variables and the response variable can be
observed. It can be noted that the variables that appear
related with performance are: Fs and Tav. Figure 1b shows
the interactions of all the variables between their respective
levels; there apparently exists evidence of relations between
values of Tav and Fs as well as Nfs with Fs.
Main Effects Plot (data means) for ratio
Fs Vo

variables related to space solution of the problem were 0.45

measured from a sample of random solutions from the 0.40

0.35
space solution of each instance. 0.30
Mean of ratio

0.25

Identification of Response Variables 1 2

Tav
3 4 1 2
Nfs
3 4

Theoretical radius: ratio of a bin packing solution was 0.45

selected as a measure of performance for TA algorithm. 0.40

This is a common measure to evaluate a bin packing

0.35

0.30
solution and it indicates the distance from the solution given 0.25

by the algorithm related to the best solution calculated 1 2 3 4 1 2 3 4

theoretically [17]. For each of 30 executions of the

algorithm the ratio was calculated, and the lower value was Fig. 1a. Graphic of main effects.
taken as ratio value of each of the 1226 instances.
To establish the performance of the algorithm
a comparison with a greedy algorithm [18] was carried out. Interaction Plot (data means) for ratio
It was made by means of two criterions, first the obtained 1 2 3 4 1 2 3 4 1 2 3 4

ratio in the instance solution and second the required time

Fs
Fs 1
0.45

for the instance execution when both algorithms obtained

2
0.35 3
4
the same radius. TA algorithm was labeled as “won” when 0.25

in the same instance TA got lower value of radio or time Vo

0.45
Vo
1

than greedy algorithm; in another case TA was labeled as 0.35

2
3

“lost”. For further analysis this information was coded as a

4
0.25

variable named performance with two levels: 1 y 0 values Tav

0.45
Tav
1

for “won” and “lost” respectively. 0.35

2
3
4
0.25

Data Preparation Nfs

Preprocessing of the data consistent in discretization

of proposed variables was carried out to facilitate further Fig. 1b. Graphic of interactions.
Explaining Performance of the Threshold…
An analysis of variance (MANOVA) was carried out
using MINITAB software. The dependent variable was
ratio and the independent variables were Fs, Vo, Tav,
Nfs. The tested hypothesis was “variables Fs, Vo, Tav,
Nfs are closely related with ratio” and it was accepted
with a confidence level of 95%. The residuals were
analyzed and they looked closely adjusted to a normal
distribution and the constant variation assumption was
rejected in a Levene test [8]. However as the ANOVA
test is very robust to deviations of normality assumption
and constant variation, the F test is slightly affected in
a fixed effects model for balanced [8], which is the case.
Step 2: Estimation of Causal Order
The construction of the causal model was carried out
using the PC [15] algorithm from TETRAD [19] causal
inference software with a confidence level of 0.95%. The
data used to get the model were Fs, Vo, Tav, Nsf and
performance variables in a discrete way. In Fig. 2 it is
appreciated that variables Fs, Vo and Tav have direct
relation with the performance node, and this fact
confirms the suppositions made in the graphical and
statistical analysis.
Variables
Fs. Number of feasible solutions
Vo. Variance of 100 random solutions
Tav. temperatures number average
Nfs. Times that TA stops by non find
solution
Performance. Indicate if TA algorithm won
or lost the instance
Fig. 2. Causal model for proposed variables
Step 3: Estimation of the Model
Conditional probability and support (which indicates
how many observations accomplish with a given de-
75
pendence) permit us the differentiation between variable
values associated to the values of the performance node.
The obtained relations with greater probability and
support are: P(performance =1 | Fs =4, Vo =1, Tav =1),
P (performance | Fs = 1, Vo = 3, Tav = 4) and
P(performance = 2| Fs = 4, Vo = 1, Tav = 1).
Model Interpretation
Causal relations which showed greater values of
conditional probability and support were interpreted; the
following explanations were inferred from these relations.
P(performance =1| Fs =1, Vo =2,3, Tav =4 ). The
TA algorithm wins if the number of random feasible
solutions of a representative sample from the solutions
space of the problem is small (0), the variability among
fitness function values in a sample of 100 random
solved feasible solutions is between 0.292-1.685 and
the number of temperatures in execution time is big
(23-57).
P(performance =2| Fs =4, Vo =1, Tav =1 ). The TA
algorithm loses if the number of random feasible solutions
of a representative sample from solutions space of the
problem is big 70.978%-100%, there is a little variability
among fitness function values in a sample of 100 random
solved feasible solutions (0.012-0.092) and the number of
temperatures in execution time is small (2-13).
The obtained explanations indicate that a particular
combination of values for the variable Fs, Vo, and Tav
characterizes an instance as won or lost against greedy
algorithm. We observed that the approximation algo-
rithm wins over greedy algorithm when the search of
solutions is intensified because it is not easy to find
feasible solutions. Referring to the explanation of why
the algorithm TA lost we observed that the solution va-
lues are too similar generating a flat trajectory in which
greedy algorithm takes advance.
Conclusions
The analysis of algorithms by means of causal mo-
dels allows to study the nature of complex algorithms
deeply and to contribute to the generalization of the
knowledge about them. However, the generation of
causal models is a task that belongs to a class of dif-
ficult problems for the computational sciences. This
paper presented the systematic construction of an
algorithm performance causal model of the TA regar-
ding a greedy algorithm. The procedure consisted of
three main phases: specification, estimation and ex-
planation. The generated model was inferred from the
observation and analysis of variables related to algo-
rithm performance. To increase the trust in the model,
the causal analysis was supplemented with a graphic
and statistical analysis.
As a future work the validation of the generated causal
model is proposed, using it as prediction mechanism and
evaluating its accuracy. Besides extending the results, a
76
performance causal model of a set of approximate algo-
rithms is being built having as objective to contrast its
performance with different configurations.
Acknowledgements
This research was supported in part by CONACYT
and DGEST.
References
1. MCGEOCH C., Experimental Analysis of Algo-
rithms, Pardalos, Romeijn, H.E.: Handbook of Glo-
bal Optimization, 2, 489, 2002.
2. COHEN P., Empirical Methods for Artificial Intel-
ligence, The MIT Press Cambridge, Massachusetts,
London England, 1995.
3. HOOKER J., Needed: An empirical science of algo-
rithms, Operations Research, 42, 1994.
4. HOOS H.H., Stochastic Local Search -Methods,
Models, Applications, PhD Thesis, Department of
Computer Science from Darmstadt University of
Technology, Germany, November, 1998.
5. PÉREZ J., PAZOS R., FRAUSTO J., RODRÍGUEZ
G., ROMERO D., CRUZ L., A Statistical Approach
for Algorithm Selection WEA: pp. 417-431 2004
6. LEMEIRE J., DIRKX E., Causal Models for Per-
formance Analysis, 4th PA3CT Symposium,
Edegem, Belgica, 2004.
7. CHICKERING D., A transformational Characte-
rization of Equivalent Bayesian Network Structures,
11th Conference on Uncertainly AI. San Francisco,
pp. 87-98, 1995.
8. MONTGOMERY D., Diseño y Análisis de Expe-
rimentos, Limusa Willey. Segunda Edición 2004.
Pérez Ortega J., et al.
9. HECKERMAN D., A Bayesian Approach to Lear-
ning Causal Networks. Technical Report, MSR-TR-
95-04. Microsoft Research. Advanced Technology
Division. Microsoft Corporation, 1995.
10. ESPOSITO F., MALERBA D., RIPA V.,
SEMERARO G., Discovering Causal Rules in Rela-
tional Databases, Applied Artificial Intelligence, 11,
71, 1997.
11. QUINLAN J., C4.5: Programs for machine learning,
Morgan Kaufmann, San Mateo, Calif., 1993
12. LEMEIRE J., DIRKX E., Causal Models for Per-
formance Analysis, 4th PA3CT Symposium, Ede-
gem, Belgica, 2004.
13. MAES S., MEGANCK S., MANDERICK B., Iden-
tification of Causal Effects in Multi-agent Causal
Models, Proceedings Artificial Intelligence and
Applications, 2005.
14. PÉREZ J., PAZOS R, FRAUSTO J., RODRÍGUEZ
G., CRUZ L., FRAIRE H., Comparison and Selec-
tion of Exact and Heuristic Algorithms, Lectures
Notes in Computer Science, Vol. 3045. Springer
Verlag, Berlin Heidelberg New York, pp. 415-424,
2004.
15. SPIRTES P., GLYMOUR C., SCHEINES R., Cau-
sation, Prediction, and Search, MIT Press, 2nd
edition 2001.
16. BEASLEY J. E., OR-Library. Brunel University.
http://people.brunel.ac.uk/~ mastjjb/jeb/ orlib/
binpackinfo. html 2006.
17. FALKENAUER E., A Hybrid Grouping Genetic
Algorithm for Bin Packing, Journal of Heuristics, 2,
5, 1996.
18. MICHALEWICZ Z., FOGEL D. B., How to Solve
It: Modern Heuristics, Springer Verlag, 1999.
19. Carnegie Mellon’s University, Open Learning
Initiative (OLI), http://www.cmu.edu/oli/