2006 International Joint Conference on Neural Networks
Sheraton Vancouver Wall Centre Hotel, Vancouver, BC, Canada
July 16-21, 2006
Using Accuracy and Diversity to Select Classifiers to Build
Ensembles
Rodrigo G F Soares, Alixandre Santana, Anne M P Canuto and Marcilio C P de Souto
Informatics and Applied Mathematics Department –
Federal University of Rio Grande do Norte (UFRN)
Natal, RN - BRAZIL, 59072-970
1
Abstract—Ensemble of classifiers is an effective way of
improving performance of individual classifiers.
However, the task of selecting the ensemble members is
often a non-trivial one. For example, in some cases, a
bad selection strategy could lead to ensembles with no
performance improvement. Thus, many researchers
have put a lot of effort in finding an effective method
for selecting classifier for building ensembles. In this
context, a Dynamic Classifier Selection (DCS) method
is proposed, which takes into account both the accuracy
and the diversity of the classifiers.
I. INTRODUCTION
I N an attempt to improve recognition performance of
individual classifiers, a common approach is to combine
multiple classifiers, forming Multi-Classifier Systems
(MCSs). MCSs, also known as ensembles or committees,
exploit the idea that a pool of different classifiers, referred
individually as to experts or recognition modules, can offer
complementary information about patterns to be classified,
improving the effectiveness of the overall recognition
process [11].
In the literature, MCSs has been widely used in several
pattern recognition tasks. In the last decade, for instance, a
large number of papers have proposed the combination of
classifiers for designing high performance classification
systems, mainly in terms of classification accuracy, in
areas such as alphanumeric character recognition [4], face
recognition [6,14,24], among others.
In the context of MCSs, another aspect that has been
acknowledged as very important is the diversity of MCSs
[12,18,19,21]. For example, there is clearly no accuracy
gain in an ensemble that is composed of a set of identical
classifiers. Thus, if there are many different classifiers to
be combine, one would expect an increase in the overall
accuracy when combining them, as long as they are diverse
(i.e., the errors produced by these classifiers are
uncorrelated). In addition, in some real world applications
the number of classifiers required to form an ensemble
1
0-7803-9490-9/06/$20.00/©2006 IEEE
with a reasonable accuracy could be enormously large
[1,20]. In this case, one could use a method for selecting
classifiers in order to decrease the number of ensemble
members and, at the same time, keeping the diversity
among the selected members.
Currently, most of the existing classifier selection
methods for MCSs use as choice criterion either the
accuracy or the diversity. In contrast, in this paper, the
criterion selection proposed will be based on both the
accuracy and diversity of the classifiers. Also, the current
methods are often applied only during the training phase:
once the member classifiers are chosen, they are used to
build a given ensemble, which will be always the same
during the test (use) phase. Differently, in this paper, a
dynamic classifier selection procedure is used: the member
classifiers to form the ensemble are chosen at the test (use)
phase. That is, different testing patterns can be classified
by different ensemble configurations.
In order to do so, two different versions of the dynamic
selection procedure are presented. The first one uses a
clustering algorithm (k-means) to group patterns of a
validation set. Likewise, in the second version, a k-NN ( k
Nearest Neighbor) classifier is built with the validation set.
In both cases the diversity and the accuracy of each
classifier is calculated based on the results provided by
these methods. For example, for the k-means case, as a
testing pattern is received as input, the distances of such a
pattern is calculated to the centroids (Euclidean distance)
in the clusters produced by the k-means. The pattern will
be assigned to the cluster with the closest centroid. Then,
the ensemble is formed based on the most accurate and
diverse classifiers associated with the cluster chosen. That
is, the ensemble is formed dynamically according to the
test pattern. A similar reasoning is used in the case of the
k-NN – given a testing pattern the k closest patterns
(Euclidean distance) in the validation set are recovered in
the accuracy and diversity are calculated based on them.
This paper is divided into seven sections and organized
as follows. Section 2 describes some related works. Multi-
classifier systems are briefly described in Section 3,
focusing on the combination methods and diversity of the
ensembles. Section 4 presents the two versions of method
proposed. The experimental analysis is introduced in
1310
Section 5, while a discussion of the results obtained is
presented in Section 6. Section 7 describes the final
remarks of this work.
II. RELATED WORKS
There have been a great deal of work dealing with
methods for producing a pool of classifiers and, then,
selecting the classifiers that are most diverse and accurate
[1,2,7,13,15,20,23]. The selection procedure is, in general,
based on either accuracy [13,15,23] or diversity [2,7,20].
For instance, when using diversity, classifiers could be
clustered based on the diversity that they produce [7]. In
this case, as classifiers that belong to the same group tend
to make correlated errors, one classifier of each group is
selected to be a member of the ensemble. Differently, in
[2], an ensemble diversity procedure based on uncertain
points (patterns) is introduced. Uncertain patterns are the
ones in which the proportion of correct votes of a pool is in
between 10 and 90. These uncertain points are considered
to deliver diversity to the ensembles, since there is no
general agreement among the classifiers about the correct
output to these points. In this context, the classifiers that
have higher accuracy for the uncertain points (diversity)
are chosen to be part of the ensemble.
For the selection process based on accuracy, for
example, a new algorithm based on Bagging and genetic
algorithm was proposed [23]. Still in this context, a novel
method for pruning classifiers from ensembles, based on a
clustering approach, was presented in [13]. In order to so, a
method of distributing the voting weights of the classifiers
is implemented. In addition, a depth-first technique for
building a tree with the most independent classifiers is
used. Finally, in [1], six selection criteria were evaluated in
the setting of combining classifiers for isolated handwritten
character recognition. However, only one of them, the
exponential error count criterion, uses accuracy and
diversity to choose the ensemble members.
Most of the works reviewed in the previous paragraph
use a selection procedure which is performed priori to the
testing phase. In other words, all testing patterns will be
classified according to the same ensemble configuration.
However, the testing patterns can be very diverse,
requiring different configurations of ensembles to classify
them more accurately. This kind of problem could be
minimized, as previously mentioned, by means of a
Dynamic Classifier Selection (DCS) procedure in that the
member classifiers to form the ensemble are chosen at the
test (use) phase. That is, different testing patterns can be
classified by different ensemble configurations. In general,
the selection criterion of this method is implemented based
on accuracy and only a single classifier is chosen
[9,10,22]. In contrast, in this paper, the DCS method
proposed uses both accuracy and diversity to choose not
only the best classifier, but a set of member classifiers to
form the ensemble to classify the testing pattern. As in the
other DCS methods in the literature, in this paper, the k-
1311
means and k-NN will be used as aid to calculate the
accuracy and diversity [11].
III. MULTI-CLASSIFIER SYSTEMS
The study of Multi-Classifiers Systems (MCSs), also
known as ensembles or committee of classifiers, has
emerged as a need to have a computational system that
works with pattern recognition on an efficient way [4,17].
The goal of using MCSs is to improve the performance of
a pattern recognition system in terms of better
generalization and/or in terms of increased efficiency and
clearer design. There are three main choices in the design
of a MCS: the organization of its components, the system
components and the combination methods that will be
used. In terms of the organization of its components, a
MCS can be defined as modular and ensemble. In the
modular approach, each classifier becomes responsible for
a part of the whole system and they are usually linked in a
serial way. In contrast, in the ensemble approach, all
classifiers are able to answer to the same task in a parallel
or redundant way. Moreover, there exists a combination
module that is responsible for providing the overall system
output [11]. In this paper, the kind of MCS analyzed will
be of the ensemble type. Thus, hereafter, the terms
Ensembles and MCSs will be used interchangeably.
With respect to the choice of the components of the
MCS, the correct choice of the set of classifiers is
fundamental to the overall performance of a multi-
classifier system. As already mentioned, the main aim of
combining classifiers is to improve their generalization
ability and recognition performance. However, the
combination of a set of identical classifiers will not
outperform the individual members. The ideal situation
would be a set of classifiers with uncorrelated errors - they
would be combined in such a way as to minimize the effect
of these failures. The set of distinct classifiers can be
achieved by varying the classifier’s structure (topology and
parameters), varying the data and/or using different types
of classifiers as components of an ensemble. In terms of
using different types of classifiers, the ensembles can be
classified as hybrid and non-hybrid. When an ensemble is
composed of classifiers of the same type, it is called non-
hybrid or homogeneous ensembles. On the other hand,
ensembles composed of classifiers of more than one type
of classifiers, it is called hybrid or heterogeneous
ensembles.
Once a set of classifiers has been created and the
strategy for organizing them has been defined, the next
step is to choose an effective way of combining their
outputs. There are a great number of combination methods
reported in the literature [6,11,17,18,24]. According to
their functioning, three main strategies of combination
methods are discussed in the literature on classifier
combination: selection-based, combination-based and
hybrid methods. The assumption in selection-based
methods is that each classifier has expertise in some local
area of the feature space. When an input pattern is
submitted for classification, the classifier responsible for
the vicinity of the input pattern is given the highest
authority to label it. Combination-based methods assume
that all classifiers are equally experienced in the whole
feature space and the decisions of all of classifiers are
taken into account for any input pattern. There are some
combination models halfway between these two extremes,
which are the hybrid methods.
A. Combination-based Methods
There are a vast number of combination-based methods
reported in the literature. They could be classified
according to their characteristics as Linear or Non-linear.
• Linear combination methods. Currently, the
simplest ways to combine multiple neural
networks are the sum and average of the neural
networks’ outputs [11].
• Non-linear methods. This class includes rank-
based combiners, such as Borda Count [17],
majority voting strategies [11], the Dempster-
Shafer technique [16], fuzzy integral [4], neural
networks [4] and genetic algorithms [11].
B. Selection-based Methods
Unlike the combination-based methods, only one
classifier is needed to correctly classify the input pattern in
selection-based methods. In order to do so, it is important
to define a process to choose a member of the ensemble to
make the decision, which is usually based on the input
pattern to be classified. The choice of a classifier to label is
made during the operation phase. This choice is typically
based on the certainty of the current decision. Preference is
given to more certain classifiers. One of the main methods
in classifier selection was proposed by [22], which is called
Dynamic Classifier Selection (DCS).
Dynamic Classifier Selection: Woods et al. [22] use
local analysis of competence to nominate a classifier to
label an input. According them, the main steps to calculate
the Local Class Accuracy (LCA) for a test pattern (xt) can
be defined as follows:
1. Take the class labels provided by all classifiers,
2. For each classifier (Di, i=1,..,L) find k (k=10 is
recommended) points closest to xt for which Di has
provided the same label.
3. Calculate the proportion of points in which Di has
provided the true label and make it be the local class
accuracy of this classifier.
4. Choose the classifier with the highest LCA. Three
main situations may occur, which are:
4.1. If there is only one winner, let it label xt.
4.2. If two classifiers are tied, choose a third classier
with the second highest LCA.
4.3. If all classifiers are tied, pick a random class
label among the tied labels.
1312
It is important to emphasize that the selection of the
most suitable classifier is made during the test phase and it
is performed in case that there is a disagreement among the
classifiers.
C. Hybrid Methods
Hybrid methods are the ones in which selection and fusion
techniques are used in order to provide the most suitable
output to classify the input pattern. Usually, there is a
criterion process to decide whether to use the selection or
combination method. Also, the main idea is to use selection
only and if only the best classifier is really good to classify
the testing pattern. Otherwise, a combination method is
used. Two main examples of hybrid methods are: Dynamic
Classifier Selection based on multiple classifier behavior
(Dcs-MCS) [9] and Dynamic classifier selection using also
Decision Templates (Dcs-DT) [10].
D. Diversity in Ensembles
As already mentioned, there is no gain in the
combination (MCS) that is composed of a set of identical
classifiers. The ideal situation, in terms of combining
classifiers, would be a set of classifiers that present
uncorrelated errors. In other words, the ensemble must
show diversity among the members in order to improve the
performance of the individual classifiers. Diversity can be
reached in three different ways:
• Variations of the parameters of the classifiers
(e.g., varying the initial parameters, such as the
weights and topology, of a neural network
model [21]).
• Variations of the training dataset of the
classifiers (e.g., the use of learning strategies
such as Bagging and Boosting [11]).
• Variations of the type of classifier (e.g., the use
of different types of classifiers, such neural
networks and decision trees, as members of an
ensemble - hybrid ensembles).
There are different diversity measures available from
different fields of research. In [12,18], for instance, ten
diversity measures are defined. In this paper, one widely
applied will be used.
The double-fault measure [8]. This measure uses
the proportion of the cases that has been misclassified by
both classifier and it is defined as follows:
N 00
DFi ,k = 11 (1)
N + N 10 + N 01 + N 00
Where: N00 is the number of patterns that both
classifiers wrongly classified; N01 is the number of patterns
that the first classifier classified correctly and the second
classified wrongly; N10 is the number of patterns that the
first classifier classified wrongly and the second classified
correctly and N11 is the number of patterns that both
classifiers wrongly classified.
 IV. THE PROPOSED METHOD
As already mentioned, the classifier selection method
proposed in this paper is based on both the accuracy and
the diversity of the classifiers. Such a method uses a
dynamic classifier selection procedure, in which different
testing patterns can be classified by different ensemble
configurations. Two different versions of this method are
proposed and they will be described in the next two
sections.
A. Cluster and select version
The selection procedure of this version is similar to the
clustering and selection method proposed in [10]. The
main differences between the method in [10] and the one
proposed in this paper are the following. In method
proposed a set of classifiers is chosen based on both
accuracy and diversity, whereas in [10] only a single
classifier is selected based only on the accuracy. Also, in
[10] the best classifier is only chosen if it is statistically
better than the other classifiers. Otherwise, a fusion method
is used with all the classifiers.
The clustering and selection procedure proposed can be
described as follows.
1. Cluster the validation patterns into k groups. This can
be accomplished by using the k-means [11].
2. For each cluster produced, rank the classifiers in a
decreasing order of accuracy and increasing order of
diversity.
2.1. In order to rank the classifiers in an increasing
order of diversity any pairwise diversity measure
can be used. In this paper, a double fault
diversity measures is used (Eq. 1);
3. For each testing pattern, do:
3.1. Assign the testing pattern to the cluster with the
respective nearest centroid (Euclidean distance).
3.2. Choose the N most accurate classifiers of this
cluster.
3.3. From the N most accurate classifiers, select the J
most diverse classifiers to compose the ensemble
members to classify the testing pattern, where (J
<= N);
3.4. Classify the testing pattern using a combination-
based method.
B. k-NN and selection version
In this version of the method, instead of a clustering
method, a k-NN (k Nearest Neighbor) classifier is built
with the validation set. In this version, the selection
procedure can be described as follows for each testing
pattern.
1. Based on the k-NN built with the validation set, find
the k neighbors of the testing patterns.
2. Based on the k neighbors, rank the classifiers in a
decreasing order of accuracy and increasing order of
diversity.
1313
2.1. In order to rank the classifiers in an increasing
order of diversity any pairwise diversity measure
can be used. In this paper, a double fault
diversity measures is used (Eq. 1);
3. Select the N most accurate classifiers of the k
neighbors;
4. From the N most accurate classifiers, select the J most
diverse classifiers to compose the ensemble members
to classify the testing pattern, where (J <= N);
5. Classify the testing pattern using a combination-based
method.
As already mentioned, the main difference of this
version and the previous one is the method used to
calculate similarity of the validation patterns, k-NN or
clustering method. The other difference is when the
selection procedure is performed. When using the
clustering method, all the rankings and selection are
calculated before the testing phase. For each group, the
best configuration is chosen a priori. On the other hand, in
the k-NN method, the choice of the closest validation
patterns, as well as the ranking and selection processes are
done during the testing phase. This may lead to different
configurations to classify all the testing patterns. However,
this means a more complex processing than using the
clustering method, since these processes have to be
calculated to all testing patterns.
Intuitively, Step 3.3 of the algorithm for both versions
works as follows.
1. Pick the two most diverse classifiers and assign them
to the set of selected classifiers (I=2);
2. While I >= J
2.1. The next classifier to be selected is the one which
is most diverse to all classifiers in the set of
selected classifiers;
2.2. I = I + 1.
An example to illustrate the functioning of step 3.3 is as
follows. Suppose an ensemble composed of five classifiers
(A,B,C,D and E). In this example, suppose that the first
five elements of the decreasing rank of diversity is D(B,C),
D(A,E), D(B,E), D(C,E), D(C,D), D(A,B) and D(A,C).
Based on this, choose the three most diverse classifiers,
which would be B, C and E. The choice of B and C is
because they have the highest diversity measure. Although
the second highest diversity is between A and D, the third
chosen classifier is E because it is the most diverse to both
B and C. If a fourth classifier was chosen, it would be
classifier A.
V. EXPERIMENTAL WORK
In this investigation, an empirical comparison of the
proposed method is performed. For this investigation, ten
different classifiers will be used, which are: two MLP
(Multi-Layer Perceptron) neural networks, two RBF
(Radial Basis Function) neural networks, two naive
bayesian classifeers, two SVM (Support Vector Machine)
and two JRIP (Optimized IREP). The choice of the
aforementioned classifiers was due to their different
learning bias. JRIP, for instance, is a propositional rule
learner, Repeated Incremental Pruning to Produce Error
Reduction (RIPPER), which was proposed by William W.
Cohen as an optimized version of IREP [5].
All the learning methods used in this study were
implemented using Java Language and were exported from
the Weka machine learning visual package
(http:\\www.cs.waikato.ac.nz/~ml/weka/). The values for
parameters of a classifier were chosen as follows. For
example, for an algorithm with only one parameter, an
initial value for the parameter was chosen followed by the
run of algorithm. Then, experiments with a larger and
smaller value were also performed. If with the initially
chosen value, the classifier obtained has the best results (in
terms of validation error), then no more experiments were
performed. Otherwise, the same process was repeated for
the parameter value with the best result so far. As a
consequence, this procedure becomes more time
consuming with the increasing in the number of parameters
to be investigated.
Also, four different ensemble methods were used. The
first one is the first version of the proposed method (cluster
and select), using a sum combination-based method to
combine the output of the classifiers. The second method is
the second version proposed method (K-NN and select),
also using a voting method.
The third one is the static selection method in which a
classifier selection method based on accuracy and diversity
is performed, followed by the sum combination method.
The fourth method is the original DCS (dynamic classifier
selection) method [22], using a K-NN method to define the
neighbors of the testing pattern and choosing the classifier
with the highest accuracy of the neighbors.
The selection procedure used in the third method has
been used in a static way, in which the ensemble members
are chosen before the testing phase starts and it is based in
the whole validation set. It is aimed to investigate whether
the use of a dynamic selection is valuable for the accuracy
of a classifier selection method. Additionally, the use of
the original DCS method aims to investigate the
importance of the decision based both on accuracy and
diversity, choosing more than one classifier and applying a
combination method.
For the classifier selection methods, three classifiers are
chosen out of the ten classifiers that compose the pool of
classifiers. In both versions of the proposed method, six
classifiers are chosen based on their accuracy on the
chosen group. After that, the three most diverse classifiers
are chosen.
A. Databases
Two different databases are used in this investigation,
which are described as follows.
• Database A: an image database, where instances
were drawn randomly from a database of 7
outdoor images (segmentation dataset from the
1314
UCI repository [3]). The images were hand-
segmented to create a classification for every
pixel. Each instance is a 3x3 region. Nineteen
continuous attributes were extracted from the
region and there are 7 different classes of regions,
which are: brickface, sky, foliage, cement,
window, path, grass;
• Database B: Proteins: It is a protein database
which represents a hierarchical classification,
manually detailed, and represents known
structures of proteins. They are organized
according to their evolutionary and structural
relationship. The main protein classes are all-α,
all-β, α/β, α+β e small. It is an unbalanced
database, which has a total of 582 patterns, in
which 111 patterns belong to class all-α, 177
patterns to class all-β, 203 patterns to α/β, 46
patterns to class α+β e 45 patterns to class small.
B. Cross-Validation
In order to evaluate the robustness of the classifier, a
common methodology is to perform cross validation on the
classifier. Ten fold cross validation has been proved to be
statistically good enough in evaluating the performance of
the classifier [16]. In ten fold cross validation, the training
set is equally divided into 10 different subsets. Nine out of
ten of the training subsets are used to train the learner and
the tenth subset is used as the test set. The procedure is
repeated ten times, with a different subset being used as the
test set. In fact, in this investigation, each method was run
with a 10 replication of 10-fold cross validation
(generating 100 classifiers). Applying the distinct
algorithms to the same folds with k at least equal to 10, the
statistical significance of the differences between the
methods can be measured, based on the mean of the error
rate from the test sets (paired t-test [16]).
VI. AN ANALYSIS OF THE RESULTS
A. Individual Classifiers
Before starting the investigation of the performance of the
ensembles, it is important to analyze the performance of
the individual classifiers. Table I shows the correct mean
(CM) and standard deviation (SD) of the individual
classifiers employed in the ensembles for databases A and
B. Configurations of all classifiers are chosen taking into
account the highest correct mean and the lowest standard
deviation of them.
TABLE I: Correct mean and standard deviation of the
individual classifiers applied to databases A and B
Classifier Database A Database B
CM SD CM SD
JRIP-1 95.48 4.10 77.92 6.29
JRIP-2 95.29 4.30 76.23 5.50
 MLP-1 95.67 4.12 80.38 3.90 mean of both versions of the proposed method was
MLP-2 94.33 3.85 79.06 5.22 statistically significant for database B (p-value = 0.019 for
Naive-1 87.90 3.86 77.55 4.66 cluster/selection version and p-value = 0.031 for the k-
Naive-2 89.10 5.57 78.49 3.80 NN/Selection version). On the other hand, for database A,
RBF-1 95.33 3.48 76.23 6.55 there is no statistical evidence to state that the
RBF-2 93.71 2.95 76.79 4.79 improvement in the correct mean of the proposed methods
SVM-1 95.76 3.43 81.51 4.94 is significant.
SVM-2 95.19 3.11 81.13 5.26 In the second test, the hypothesis tests (t-test) is
According to the correct mean provided by all performed to compare the correct mean of both versions of
classifiers, one can see that all classifiers have delivered a the proposed method with the original DCS, using a
similar pattern of performance for both databases. confidence level of 95% (α = 0.05). It could be seen that
Considering the correct mean of the classifiers, for both the correct mean of both versions of the proposed method
databases, SVM-1 had delivered the highest correct mean was statistically significant for database B (p-value = 0.029
for all methods. In relation to the lowest standard for cluster/selection version and p-value = 0.048 for the k-
deviation, for database A, RBF-2 had presented the lowest NN/Selection version). On the other hand, for database A,
standard deviation for all classifiers, while Naïve-2 had there is no statistical evidence to state that the
provided the lowest standard deviation for database B. The improvement in the correct mean of the proposed methods
highest difference of the classifiers (between the highest is significant.
and the lowest correct mean) is lower for database B (5.25) As it could be seen, for database A, the improvement
than for database A (7.86). reached by the proposed method was not statistically
significant to any of the other two methods. This is because
B. Ensembles
the correct means of the individual classifiers and the
ensembles are really high. In this sense, an improvement
Table II shows the correct mean and standard deviation for that is statistically significant is difficult to reach. On the
all four ensemble methods applied to databases A and B. other hand, for database B, which is an unbalanced
For the cluster version of the proposed method, a k-means database, the improvement reached by both versions of the
clustering algorithm was used, in which k = 6 for database proposed method was statistically significant, when
A and k = 10 for database B. Differently, for the k-NN comparing with the static selection and the original DCS
version of the proposed method, a k-NN method was used, methods.
with k = 10 for database A and k = 18 for database B. For
the original DCS, a k-NN method has also been used, using VII. FINAL REMARKS
k = 10 for both databases.
In this paper, a DCS (dynamic classifier selection)
procedure was proposed, which takes into account
TABLE II: Performance of the ensemble methods
accuracy and diversity of the classifiers in order to choose
applied to databases A and B
the ensemble members. Moreover, two different versions
Classifier Database A Database B
of the proposed method were presented. The first one used
CM SD CM SD
a clustering algorithm to group patterns of a validation set.
K-NN/Selection 95.90 3.00 83.77 5.42
In the second version, a k-NN method was used to
Cluster/Selection 96.81 2.68 84.52 5.81
calculate similarity among patterns of a validation set.
Static 96.14 2.77 79.24 4.71
In order to evaluate the performance of the proposed
O-DCS 95.79 4.21 80.00 4.10
method, an empirical comparison was performed. Four
different ensemble methods were analyzed, which are: both
As it can be observed from Table II, all ensemble
versions of the proposed method, a static classifier
methods, apart from the static method for database B, have
selection method, using also diversity and accuracy, and
delivered a higher correct mean than all individual
the original DCS method. These ensembles were applied to
classifiers. Moreover, all two versions of the proposed
two different databases.
method have delivered the highest and second highest
Through this analysis, it could be observed that both
correct mean of all methods. For database B, for instance,
versions of the proposed method reached an improvement
the improvement of the proposed method was of 5.28%,
in the correct mean, when compared with the other two
when compared with the static classifier selection method.
methods. Nevertheless, this improvement was statistically
In order to verify whether the improvement in the
significant only for database B. It is believed that this is
correct mean obtained by the proposed methods is
because the correct mean of the classification systems was
statistically significant, the hypothesis tests (t-test) is
really high for database A. In this sense, an improvement
performed, using a confidence level of 95% (α = 0.05). In that is statistically significant is difficult to reach.
the first test, both versions of the proposed methods were The results obtained in this paper are very encouraging
compared with the static selection procedure. Based on the since it shows that the use of a dynamic classifier selection
hypothesis test, it is possible to conclude that the correct procedure which takes into account diversity and accuracy

1315
is positive to the performance of the ensembles. [18] Shipp C.A. and Kuncheva L.I.. Relationships between
combination methods and measures of diversity in
ACKNOWLEDGMENT combining classifiers, Inf Fusion, 3 (2), 2002, 135-148.
[19] A Tsymbal, M Pechenizkiy and P Cunningham. Diversity in
This work has the financial support of CNPq (Brazilian search strategies for ensemble feature selection. Inf Fusion,
Research Council), under process number 471309/2004-3 Special issue on Diversity in Multiple Classifier System. 6
and 307236/2003-0 (1), pp. 83-98, 2005.
[20] G. Valentino. An Experimental Bias-Variance Analysis of
SVM Ensembles Based on Resampling Techniques. IEEE
REFERENCES Transactions on System, Man and Cybernetics- Part B:
[1] M. Askela. “Comparison of classifier selection methods for Cybernetics, Vol. 35, No. 6, pp.1252-1271, 2005
improving committee performance”. In in Multiple [21] T. Windeatt, Diversity measures for multiple classifier
Classifier Systems—4th Int.Workshop, MCS 2003, vol. system analysis and design. Inf Fusion, Special issue on
2709, ser. Lecture Notes in Computer Science, T. Windeatt Diversity in Multiple Classifier System, pp. 21-36, 6 (1),
and F. Roli, Eds., Guilford, U.K., 2003, pp. 306–316. 2005.
[2] R. Banfield, L. Hall, K. Bowyer, andW. Kegelmeyer, “A [22] Woods, K. Kegelmeyer, W and Bowyer, K. Combination of
new ensemble diversity measure applied to thinning Multiple Classifiers using Local Accuracy estimates, IEEE
ensemble,” in Multiple Classifier Systems—4th Trans on Patt Analysis and Mach Intelligence, 19(4), 405-
Int.Workshop, MCS 2003, vol. 2709, ser. Lecture Notes in 410, 1997
Computer Science, T. Windeatt and F. Roli, Eds., Guilford, [23] X. Wu and Z. Chen. “Recognition of Exon/Intron
U.K., 2003, pp. 306–316. Boundaries Using Dynamic Ensembles”. Proceedings of the
[3] Blake, C.L. and Merz, C.J.: UCI Repository of machine Computational Systems Bioinformatics Conference (CSB),
learning databases 2004
[http://www.ics.uci.edu/~mlearn/MLRepository.html]. [24] Zhou,D, J. and Zhang, "Face recognition by combining
Irvine, CA: University of California, Department of several algorithms" Pattern Recognition 3(3):497-500, 2002.
Information and Computer Science. (1998)
[4] A Canuto.. Combining neural networks and fuzzy logic for
applications in character recognition. PhD thesis, University
of Kent, 2001
[5] W W. Cohen. “Fast Effective Rule Induction", 'Machine
Learning: Proceedings of the Twelfth International
Conference'(ML), pp.115-123. 1995.
[6] J Czyz and M Sadeghi and J Kittler and L Vandendorpe.
Decision fusion for face authentication, Proc First Int Conf
on Biometric Authentication, 686-693, 2004.
[7] G. Giacinto anf F Roli. “An Approach to the Automatic
Design of multiple classifier systems. Pattern Recognition
Letters, 22: 25-33, 2001.
[8] G. Giacinto and F. Roli, "Design of effective neural network
ensembles for image classification", Image and Vision
Computing Journal , 19(9-10), pp. 697-705, 2001.
[9] Giacinto, G. and Roli, F. “Dynamic Classifier Selection
based on Multiple Classifier Behaviour”. Pattern
Recognition, vol. 34, 1879-1881, 2001
[10] Kuncheva, L. Switching Between Selection and Fusion in
Combining Classifiers: An Experiment. IEEE Trans on
Systems, Man and Cybernetics – Part B Vol. 32, N.2, 146-
155, 2002.
[11] Kuncheva L.I. Combining Pattern Classifiers. Methods and
Algorithms, Wiley, 2004.
[12] Kuncheva L, Whitaker, C. Measures of diversity in classifier
ensembles, Mach Learning, 51, 181-207, 2003.
[13] A. Lazarevic, and Z Obradovic. Effective pruning of neural
network classifier ensembles. Proc. Of the International Joint
conference on Neural Networks, vol. 2, pp.796-801, 2001.
[14] Lemieux, A and Parizeau, M "Flexible multi-classifier
architecture for face recognition systems". The 16th
International Conference on Vision Interface, 2003
[15] D. Marginenatu and T. Dietterich, “Prununig adaptive
boosting,” in Proc. 14th Int. Conf. Machine Learning, San
Francisco, CA, 1997, pp.378–387.
[16] T Mitchell. Machine Learning. McGraw-hill, 1997
[17] A J C Sharkey. Multi-net System. In Combining Artificial
Neural Nets: Ensemble and Modular Multi-net Systems,
(Ed) A. J. C. Sharkey. Spring-Verlag, 1-30, 1999.

1316