2018 24th International Conference on Pattern Recognition (ICPR)
Beijing, China, August 20-24, 2018
Classifier Recommendation Using
Data Complexity Measures
Luı́s P. F. Garcia Ana C. Lorena
InfAI/Leipzig University Universidade Federal de São Paulo LIFO/University of Orleans
Leipzig, Germany and Instituto Tecnológico de Aeronáutica
Email: garcia@informatik São José dos Campos, SP, Brazil
.uni-leipzig.de Email: aclorena@unifesp.br
Abstract—Application of machine learning to new and unfamil-
iar domains calls for increasing automation in choosing a learning
algorithm suitable for the data arising from each domain. Meta-
learning could address this need since it has been largely used
in the last years to support the recommendation of the most
suitable algorithms for a new dataset. The use of complexity
measures could increase the systematic comprehension over the
meta-models and also allow to differentiate the performance of a
set of techniques taking into account the overlap between classes
imposed by feature values, the separability and distribution of the
data points. In this paper we compare the effectiveness of several
standard regression models in predicting the accuracies of clas-
sifiers for classification problems from the OpenML repository.
We show that the models can predict the classifiers’ accuracies
with low mean-squared-error and identify the best classifier for
a problem that results in statistically significant improvements
over a randomly chosen classifier or a fixed classifier believed to
be good on average.
I. I NTRODUCTION
The recent surge of interests in applying Machine Learning
(ML) methods to many new and diverse domains has high-
lighted a need for further automation of the analysis process.
In particular, new application domains open up the possibility
that the data available for learning are of a different nature
from those familiar to previous users of ML, like text or
image and speech signals. With unfamiliar data and class
distributions in a new context, a practitioner has little guidance
in choosing a suitable learning method, as it is well known that
different classifiers appeal to data with different geometries
and distribution characteristics [1].
A decision task like identifying a good classifier for a
dataset can be formulated as an ML problem if certain numeri-
cal or categorical features are available to describe the dataset,
and some previous examples are available for training a pre-
dictor. Meta-Learning (MtL) with data complexity measures,
where the learning task is to predict a classifier’s performance
for a dataset, and the features to use are some statistical or
geometrical characteristics of that dataset, carries a promise to
serve this purpose. The practical question, however, as in all
learning tasks, is to what extent such prediction is successful,
and whether the known measures of data complexity and
the known examples of classifier behavior are sufficient for
training such a predictor. In this paper, we demonstrate a
success in this MtL task using a large collection of ML
978-1-5386-3788-3/18/$31.00 ©2018 IEEE 874
Marcilio C. P. de Souto Tin Kam Ho
IBM Watson
Orleans, France Yorktown Heights, NY, USA
Email: marcilio.desouto@ Email: tho@us.ibm.com
univ-orleans.fr
problems from the OpenML repository [2], several standard
base classifiers and meta-predictors, and an extended set of
data complexity measures implemented in publicly released
software [3].
This paper is organized as follows: Section II presents some
related studies on the use of complexity measures in MtL.
Section III describes the complexity measures employed in
this work. Section IV presents the MtL methodology followed
in this work. Section V presents the experiments performed
along with their results. Section VI concludes the paper.
II. R ELATED S TUDIES
Complexity measures of classification problems allows one
to estimate the expected difficulty of a classification problem
by extracting descriptions of the overlap between classes
imposed by feature values, the separability and distribution
of the data points, and certain structural characteristics of the
problem based on the representation of the problem by the
training datasets available for learning [4]. Most of the mea-
sure values are highly correlated with predictive performance
of classification models, as demonstrated in previous studies
[5], [6]. Therefore, it is expected that they can play an impor-
tant role in improving the recommendation for classification
algorithms [7].
MtL can be used in ML algorithm recommendation by,
for instance, estimating the expected predictive accuracy of
classification techniques [7]. To do this, firstly, a meta-dataset
must be constructed where each meta-example is associated
with a dataset. Then, meta-features describing characteristics
of each dataset are extracted. In this paper, the complexity
measures of classification problems are used as meta-features.
The target value for the prediction is the accuracy of a
classification technique for the dataset associated with the
meta-example.
Given a meta-dataset, the next step is the induction of a
meta-model. The meta-model can be induced by ML tech-
niques and can be used in a recommendation system to select
the most suitable algorithm(s) for a new dataset. In this
paper we want to predict the accuracy of the classification
techniques, so we used a set of regression techniques and
also some standard baselines to evaluate and validate the
achieved performance [8]. The expectation is that with highly
accurate models, the MtL system can be used to select the most
suitable algorithm(s) for a new dataset based on the complexity
measures.
There are some previous work in the literature using data
complexity measures to recommend classification algorithms.
In [9] nine complexity measures from [4] are used as meta-
features in an MtL setup designed to predict the best among
five classification techniques for binary classification prob-
lems. In [6] the complexity measures are used to analyze the
expected behavior of the Nearest Neighbor (NN) classifier. The
work [10] uses the complexity measures as meta-features for
inducing multi-label recommendation algorithms for datasets
from a specific (educational) domain.
The MtL setup in the current study differs from the pre-
vious attempts in the following ways. Firstly, a larger set
of classification complexity measures is considered. This set
contains all the measures from the Data Complexity pack-
age (DCoL) [11] plus other measures from recent literature
that may complement them and were implemented in the
Extended Complexity Library package (ECoL) [3]. A larger
set of classification datasets is also considered, totalizing 141
datasets from different domains. We chose four classification
techniques with different biases to be recommended: Artifi-
cial Neural Networks (ANN) [12], Support Vector Machines
(SVM) [13], k-Nearest Neighbor (kNN) Classifiers [14] and
Decision Trees (DT) [15]. These techniques are often used
for various reasons that include simplicity, wide availability,
convenience, ease of set-up, popularity, or in some cases,
interpretability. The MtL problem is formulated as a prediction
of the expected accuracy of the four classification techniques.
To derive the meta-models, three regression techniques are
used; in addition, two baselines described in the literature are
used for evaluation [8]. We also analyzed the most important
meta-features for one of the recommender models induced.
As can be seen in our results, all meta-models yielded high
predictive accuracies and the use of the extended set of
complexity measures resulted in effective meta-models for ML
algorithm recommendation.
III. C OMPLEXITY M EASURES
Various issues may cause the difficulty of a classification
problem. Ho et al. [16] point to three main factors: (i) the
intrinsic ambiguity of the classes, or overlapping of class
distributions; (ii) the sparsity and dimensionality of the data;
and (iii) the complexity of the classification boundary. Often
there is a combination of these three factors in effect. The
measures proposed in Ho & Basu [4] focus on characterizing
the complexity of the classification boundary in different ways.
In this paper we consider an extended set of measures that
includes other measures proposed in recent related literature.
The entire set of employed measures can be divided into the
following groups:
1) feature overlapping measures: evaluate the
discriminative power of the input features.
875
2) linearity measures: try to quantify if it is possible to
separate the classes by a hyperplane.
3) neighborhood measures: analyze the neighborhoods of
individual examples and try to capture class overlap and
the shape of the decision boundary.
4) network measures: represent the dataset as a graph and
extract structural information from it.
5) dimensionality measures: give an indication of data
sparsity.
6) class balance measures: capture the differences in the
number of examples per class in a dataset.
All measures are estimated from a learning dataset T
containing n pairs of examples (xi , yi ) where i ranges from 1
to n. Each training example xi is described by m input features
and has a label yi out of nc classes. Some of the measures
are defined for features with numerical values only; in this
case the categorical features are first coded into numerical
values accordingly. Multiclass classification problems have to
be decomposed first into binary subproblems for some of the
measures. We adopt the one-versus-one (OVO) decomposition
for multiclass problems [17]. In this case, a measure for
the multiclass problem is computed as the average of the
values obtained for the pairwise sub-problems. The complexity
measures computed in this paper are presented in Table I
(category, name, acronym and original reference) and are
briefly described next.
F1 takes the largest Fisher’s discriminant ratio among those
computed from all the features, so that larger F1 values are
expected for simpler classification problems. F1v complements
F1 by searching for a vector able to separate two classes
after the training examples have been projected onto it. F2
computes the overlap of the support of the distributions of
the features values within the classes. The higher the F2
value, the greater the amount of overlap between the problem
classes and the problem’s complexity. F3 returns the maximum
ratio between the number of examples that are not in the
overlapping region of two classes and the total number of
examples, among all input features. Higher F3 values are
obtained for problems in which there are few examples in an
overlapping region for at least one of the input dimensions. F4
applies F3 successively until all examples are discriminated or
no input feature remains. Larger F4 values indicate that it is
possible to discriminate more examples using a combination
of the input features and, therefore, that the problem is simpler.
To obtain the linearity measures, a linear Support Vector
Machine (SVM) [13] is used. L1 computes the sum of the
distances of incorrectly classified examples to a linear bound-
ary used in their classification. Low values for L1 indicate that
the problem is simpler. L2 computes the error rate of the linear
SVM classifier induced from T . Higher L2 values denote more
errors and therefore a greater complexity regarding the aspect
of data linearity. L3 first creates a new dataset by randomly
interpolating pairs of examples of the same class, as in [22].
Then a linear SVM is trained on the original data and has
its error rate measured with the new data points. Higher L3
values indicate higher complexity.
 TABLE I
C OMPLEXITY MEASURES EMPLOYED .

Category Name Acronym Ref.

Maximum Fisher’s Discriminant Ratio F1 [4], [5]
Directional-vector maximum Fisher’s discriminant ratio F1v [11]
Feature overlapping Volume of the overlapping region F2 [4], [18]
Maximum individual feature efficiency F3 [4]
Collective feature efficiency F4 [11]
Sum of the error distances by linear programming L1 [4]
Linearity measures Error rate of a linear classifier L2 [4]
Non-linearity of a linear classifier L3 [4]
Fraction of borderline points N1 [4]
Ratio of intra/inter class nearest neighbor distance N2 [4]
Error rate of the nearest neighbor classifier N3 [4]
Neighborhood measures
Non-linearity of the nearest neighbor classifier N4 [4]
Fraction of hyperspheres covering data T1 [4]
Local set average cardinality LSCAvg [19]
Average density of the network Density [20]
Network measures Clustering coefficient ClsCoef [20]
Hub score Hubs [20]
Average number of points per dimension T2 [4]
Dimensionality measures Average number of points per PCA dimension T3 [18]
Ratio of the PCA dimension to the raw dimension T4 [18]
Entropy of class proportions C1 [18]
Class balance measures
Imbalance ratio C2 [21]

The neighborhood measures work over a matrix that stores
the distances between all pairs of points in T . The Gower
distance [23] is employed, which is hybrid and supports both
numerical and categorical features. N1 first builds a Minimum
Spanning Tree (MST) from data and computes the percentage
of vertices incident to edges connecting examples of opposite
classes in the MST. Higher N1 values indicate the need for
more complex boundaries to separate the classes. N2 computes
the ratio of two sums: intra-class and inter-class distances.
The former corresponds to the sum of the distances between
each example and its closest neighbor from the same class.
The later is the sum of the distances between each example
and its closest neighbor from another class (nearest enemy).
High N2 values are indicative of more complex problems.
N3 corresponds to the error rate of a 1NN (one Nearest
Neighbor) classifier, estimated using a leave-one-out procedure
in T . High N3 values indicate a more complex problem. N4
is similar to L3, but uses the 1NN classifier instead of the
linear SVM predictor. T1 builds hyperspheres centered at each
example, and grows their radii until the hypersphere reaches an
example of another class. Smaller hyperspheres contained in
larger hyperspheres are eliminated. T1 is defined as the ratio
between the number of the remaining hyperspheres and the
total number of examples in the dataset, so that lower values
are for simpler datasets. In this paper the hyperspheres are
grown until they reach another hypersphere from an opposite
class. The Local-Set (LS) of an example xi ∈ T the set of
points from T whose distance to xi is smaller than the distance
from xi to xi ’s nearest enemy [19]. LSCAvg is the average
cardinality of the LS of all examples. Lower LSCAvg values
are expected for more complex problems.
In the network measures, the dataset T is first modeled
as a graph, in which each example corresponds to a vertex,
whilst weighted edges connect pairs of examples. The -NN
method is used to build the graph, in which pairs of nodes
i and j are connected only if dist(i, j) < . As in [24]
and [20], we adopted  = 0.15d, where d is the smallest
distance between all pairs of examples. Next, as in [20] a post-
processing step is applied to the graph to prune edges between
examples of opposite classes. Density is given by the number
of edges in the graph, divided by the maximum number of
edges between the number of pairs of points. A low number of
edges will be observed for datasets of low density or for which
examples of opposite classes are near each other. Both cases
indicate a higher classification complexity. ClsCoef averages
the clustering tendency of the vertices, which, for each vertex
vi , corresponds to the ratio of the number of existent edges
between its neighbors to the total number of edges that could
possibly exist between them. ClsCoef will be larger for simpler
datasets, with dense connections among examples from the
same class. Hubs takes the average hub score of the graph’s
vertices. The hub score of a node is given by the number of
connections it has to other nodes, weighted by the number
of connections these neighbors have. Simpler datasets show
dense regions within the classes and higher hub scores.
T2 is given by the ratio between the number of examples n
and dimensionality m. Low T2 values indicate more sparsity
and therefore a higher expected complexity. T3 is similar to
T2, but uses the number of principal components needed to
represent 95% of data variability (m0 ), which can be regarded
as an estimate of the intrinsic dataset dimensionality, as the
base of data sparsity assessment. T4 divides the number of
PCA dimensions as defined for T3 (m0 ) by the original number
of dimensions (m). The larger the T4 value, the more of
the original features are needed to describe data variability,
indicating a more complex relationship of the input variables.

876
 C1 computes an entropy estimate based on the proportions
of examples per class in the dataset. It achieves maximum
value for balanced problems, which can be considered simpler
according to the class balance aspect. C2 is a multiclass
version of the imbalance ratio, in which larger values are
obtained for imbalanced problems.
IV. M ETHODOLOGY
The data complexity measures are used in an MtL setup
designed to predict the accuracy of some popular classification
techniques for a given dataset. The objective is to determine
whether the measures allow to build accurate recommendation
systems for a diverse set of classification techniques and
classification problems.
To train and evaluate a meta-learner, we build meta-datasets
using a collection of classification problems for each of which
the performance of one or more classification techniques is
known [7]. We use 141 datasets from the OpenML repository
[2] in a process as shown in Figure 1. For each dataset we
compute the complexity measures values plus the average
cross-validated predictive accuracy achieved by four ML tech-
niques: ANN based on backpropagation (also called Multilayer
Perceptron) [12]; SVM [13]; DT induced by the C4.5 learning
algorithm [15]; and kNN, a lazy learning technique [14]. As a
result, we have four meta-datasets. All of them have 22 input
features, namely the complexity measures described in Section
III, 141 examples (the datasets) and each example is labeled
with the accuracy of one of the four classifiers.
can be found at a GitHub site1 . The parameters used for the
classifiers are: ANN with learning rate of 0.3, momentum of
0.5 and one hidden layer; C4.5 with pruning; kNN with k=3;
and SVM with radial basis kernel. The parameters used in
the regression techniques are: DWNN with Gaussian kernel,
SVR with radial basis kernels and RF with 500 DTs. All
other parameter values of the classification and regression
techniques used were kept as default of the R implementation
used in the experiments.
V. M ETA - LEARNING R ESULTS
First we present an overview on the classifiers’ performance
in all the 141 datasets used in this study. Figure 2(a) shows
boxplots of the accuracies achieved by the ANN, C4.5, kNN
and SVM classifiers. We notice that all techniques achieved
a median accuracy above 0.75. Whilst the SVM classifier
achieved the highest median accuracy, the kNN classifier
showed the lowest. Figure 2(b) shows the number of times
each classifier achieved the best accuracy in these datasets.
SVM also achieved the best overall results, winning for about
43% of the datasets, whilst the kNN classifier was better only
in 13% of the datasets. Because of these results, the SVM
classifier is set as the default recommended technique. The
dominance of one base-classifier suggests that if we set up
the recommendation problem as a multi-class classification
problem, the meta-dataset is class-imbalanced. This motivated
our use of regression on the accuracies instead for the meta-
learning step.
1.00 60

0.75
Complexy Measures
Regression techniques Number of wins 40
Accuracy

Meta-datasets (DWNN, RF, SVR)

Datasets 0.50
Classification
performance ●
●
●
●
● 20
(ANN, DT, SVM, kNN) ●

0.25 ●
●
●
● ●
● ●
Classification performance
● ●
● ●
●

estimation ●

●
0
0.00
ANN C4.5 kNN SVM ANN C4.5 kNN SVM

Fig. 1. MtL experimental setup. (a) Distribution of accuracies. (b) Winning classifiers.

Each meta-dataset is submitted to a regression technique, Fig. 2. Performance of the base-classifiers.

which estimates the expected accuracy by the classifier as a
Figure 3 shows the average mean-squared-error (MSE) of
function of the 22 meta-features. The classifier with the highest
the meta-regressors for predicting the expected accuracies of
estimated accuracy for a dataset is to be recommended for the
the base-classifiers, estimated by a leave-one-out process. Each
dataset. Three regression techniques known to have different
plot represents one base-classifier and the boxplots summarize
biases were employed to make the recommendations: Random
the MSE values of each meta-regressor. We show the baselines
Forests (RF) [25] [26], Support Vector Regressors (SVR) [13]
in gray color and the 3 meta-regressors techniques (DWNN,
and Distance Weighted k-Nearest Neighbor (DWNN) [14]. To
RF and SVR) in white color. In these plots, the default baseline
train and evaluate these regressors, leave-one-out rounds are
corresponds to predicting the average accuracy in the meta-
performed.
dataset, whilst the random baseline predicts a random accuracy
All of the datasets chosen for that experiment have no among those registered in the training meta-dataset. Therefore,
missing values. They represent diverse application contexts they vary for each base-classifier, based on their corresponding
and domains, and are selected with limits of up to a maximum accuracy.
number of 10,000 examples, 500 features and 10 classes. The
code, datasets and a summary of their main characteristics 1 https://github.com/lpfgarcia/complex

877
 ANN C4.5 accuracy decrease. The accuracy increase was higher when
0.100
●
●

●
compared to the random recommendation, which is not sur-
● ●
●

●
●
●
●
●
● ●
● prising. The accuracy increase when compared to the default
●

0.075 recommendation suggests that, when choosing a classifier, if

●
● ●
●
one takes into account the data characteristics and chooses the
● ●
●
● classifier accordingly (such as using our meta-model), better
0.050 ●
●
●
●
●
●
●
●
●
●
●
●
● ●
accuracy can be achieved than if one just uses a fixed, default
●

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

●
●
choice, even if that default choice is the best on average. Again
● ●
0.025 ●
● ● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●

●
●
●
●
●
●
●
●
●
●
●
●
the best performing meta-regression technique was RF.
● ● ● ●
●
●
● ● ●
● ●
● ●
● ●
●

0.000
Random Default
MSE

kNN SVM

Percentage increase of accuracy

0.100 ●
● 150
● ●

●
● ●

●
●
0.075
●
● ●
100
●

● ● ●
0.050 ● ●
●
●

● ●
● ●
●
●
●
● ●
●
●
●
●
50
● ● ●
0.025 ●
●
●
●
●
●
●
●
●
●
● ●
● ● ● ●
●
●
●
● ●
● ● ● ●
● ● ● ● ●
● ● ● ●
● ● ● ●
●
●
● ●
●
● ● ●
● ●
●
● ● ●
●
● ●
●
●
●
●

0.000 0
DWNN RF SVR RandomDefault DWNN RF SVR RandomDefault
DWNN RF SVR DWNN RF SVR

Fig. 3. MSE of each meta-regressor.

Fig. 4. Improvement of base-classifier accuracies over baselines when the
complexity-driven recommendations are used.

We notice that the DWNN, RF and SVR meta-regressors
outperformed the baselines and are more accurate in all cases.
In general, the meta-regressors also showed a more stable
behavior when compared to the baselines, which have elon-
gated boxplots and larger standard deviation values. Among
the meta-regressors, in most of the cases RF performed better,
followed by SVR and DWNN. Using the Friedman statistical
test with the Nemenyi post-test at 95% confidence level
[27], the DWNN, RF and SVR meta-regressors presented
better predictive performance than the baselines, but performed
similarly to each other.
A dataset-dependent recommender of classifiers is useful if
the classifier it suggests for a dataset performs better than a
random or a fixed choice. Figure 4 presents the increase of
accuracy obtained by the classifiers in the datasets when they
are the ones recommended by the meta-regressors instead of
the Random and Default recommendations. The x-axis shows
the meta-regressors and the y-axis represents the increase of
accuracy (summed over all datasets) when compared to one
corresponding baseline. The random baseline here assigns
a randomly chosen classifier among the four. The default
baseline is SVM, which obtained more wins in the meta-
dataset. Since we have four meta-regressors per regression
technique (one per base-classifier), the recommendation is
obtained as follows: the accuracies predicted by each of the
four meta-models are used to rank the base-classifiers; next,
the technique for which the expected accuracy is highest is
recommended. Indeed, the meta-regressors were always able
to improve the accuracy results and we did not notice any
To learn more about the complexity measures regarding
their importance in predicting the best-to-use base-classifiers,
we show in Figure 5 the 5 top-ranked complexity measures
selected by the RF in all runs. The x-axis represents the
measures and the y-axis shows the average of their Gini index
in the RF models. The complexity measures regarded as the
most important are those based on neighborhood information
(N3, N1, N2 and T1). There is also a representative from
the structural (network) group (Density). But, since the struc-
tural measure is calculated from a proximity graph of the
examples, it can also be regarded as employing neighborhood
information. It should be noticed that these measures are very
correlated to each other and that a more sophisticated feature
selection procedure could reveal other useful combinations of
complexity measures.
It is also worth noting that some complexity measures have a
strong bias towards the kNN classifier or the DWNN regressor
(e.g., N3, N4). But this did not translate into advantages for
these techniques, which showed worst results compared to the
other classification and regression techniques employed.
One additional analysis is the trade-off between the runtime
of the complexity measures to that of evaluating all classifiers
in a cross-validation setup. Using a sequential execution for
all the datasets in a cluster node with two Intel Xeon E5-
2680v2 processors and 128 GB, the runtime of the complexity
measures was lower than that of the classifier evaluations
for 71% of the cases. For datasets with a high number of
input features, there was an overhead in the complexity mea-
sures computation. One must also notice that the complexity

878

1.5
Measure Importance
1.0
0.5
0.0
N3
N1
N2
Density
T1
Fig. 5. Top-ranked complexity measures selected by the RF meta-regressors.
measures are coded in R and are not optimized, whilst the
classification techniques are written in compiled languages.
VI. C ONCLUSION
This paper presented a study on the use of several measures
of classification complexity in the recommendation of classi-
fication algorithms. A large and updated set of complexity
measures were used to characterize a diverse set of classifica-
tion problems from different domains. We were able to build
accurate meta-models to predict the expected accuracies and
choose the best classifier among four popular classification
techniques: ANN, SVM, C4.5 and kNN.
The meta-regressors DWNN, RF and SVR used to pre-
dict the classifier with best expected performance for new
datasets were compared to two baseline recommenders. In
the experiments, the random recommendation of one of the
four classification techniques, a very naı̈ve baseline, achieved
worst results. The best meta-model was the RF and the most
important complexity measures used by this model were based
on neighborhood information and structure.
Future work shall look for the core complexity measures
best for distinguish between the performances of various
classifiers. We would also like to: (i) optimize the imple-
mentation of the complexity measures; (ii) evaluate other
MtL approaches like ranking the classifiers; (iii) investigate
parameter tuning for the classification techniques; and (iv)
study further in which cases the classifier recommendation
outperforms the default classification technique.
ACKNOWLEDGEMENTS
The second author would like to thank the financial support
of the foundations FAPESP (grant 2012/22608-8) and CNPq
(grant 308858/2014-0).
R EFERENCES
[1] D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2,
pp. 241–259, 1992.
879
[2] J. N. Van Rijn, B. Bischl, L. Torgo, B. Gao, V. Umaashankar, S. Fischer,
P. Winter, B. Wiswedel, M. R. Berthold, and J. Vanschoren, “OpenML:
A collaborative science platform,” in ECML/PKDD, 2013, pp. 645–649.
[3] L. P. F. Garcia, A. C. Lorena, and J. Lehmann, “ECoL: Com-
plexity measures for classification problems,” 2018, https://CRAN.R-
project.org/package=ECoL.
[4] T. K. Ho and M. Basu, “Complexity measures of supervised classifi-
cation problems,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 24, no. 3, pp. 289–300, 2002.
[5] R. A. Mollineda, J. S. Sánchez, and J. M. Sotoca, “A meta-learning
framework for pattern classification by means of data complexity mea-
sures,” Inteligencia artificial, vol. 10, no. 29, pp. 31–38, 2006.
[6] G. D. C. Cavalcanti, T. I. Ren, and B. A. Vale, “Data complexity
measures and nearest neighbor classifiers: a practical analysis for meta-
learning,” in 24th International Conference on Tools with Artificial
Intelligence (ICTAI), vol. 1, 2012, pp. 1065–1069.
[7] P. Brazdil, C. Giraud-Carrier, C. Soares, and R. Vilalta, Metalearning:
Applications to Data Mining. Springer, 2009.
[8] C. Soares, P. Brazdil, and P. Kuba, “A meta-learning method to select the
kernel width in support vector regression,” Machine Learning, vol. 54,
no. 3, pp. 195–209, 2004.
[9] E. Hernández-Reyes, J. A. Carrasco-Ochoa, and J. F. Martı́nez-Trinidad,
“Classifier selection based on data complexity measures,” Lecture notes
in computer science, vol. 3773, p. 586, 2005.
[10] J. L. Olmo, C. Romero, E. Gibaja, and S. Ventura, “Improving meta-
learning for algorithm selection by using multi-label classification: A
case of study with educational data sets,” International Journal of
Computational Intelligence Systems, vol. 8, no. 6, pp. 1144–1164, 2015.
[11] A. Orriols-Puig, N. Macià, and T. K. Ho, “Documentation for the data
complexity library in c++,” La Salle - Universitat Ramon Llull, Tech.
Rep., 2010.
[12] S. Haykin, Neural Networks – A Compreensive Foundation. Prentice-
Hall, 1999.
[13] N. Cristianini and J. Shawe-Taylor, An introduction to support vector
machines and other kernel-based learning methods. Cambridge uni-
versity press, 2000.
[14] T. M. Mitchell, Machine Learning, ser. McGraw Hill series in computer
science. McGraw Hill, 1997.
[15] J. R. Quinlan, “Induction of decision trees,” Machine Learning, vol. 1,
no. 1, pp. 81–106, 1986.
[16] T. K. Ho, M. Basu, and M. H. C. Law, “Measures of geometrical
complexity in classification problems,” in Data Complexity in Pattern
Recognition, 2006, pp. 1–23.
[17] A. C. Lorena, A. C. P. L. F. de Carvalho, and J. M. P. Gama, “A review on
the combination of binary classifiers in multiclass problems,” Artificial
Intelligence Review, vol. 30, no. 1, pp. 19–37, 2008.
[18] A. C. Lorena, I. G. Costa, N. Spolaôr, and M. C. P. Souto, “Analysis of
complexity indices for classification problems: Cancer gene expression
data,” Neurocomputing, vol. 75, no. 1, pp. 33–42, 2012.
[19] E. Leyva, A. González, and R. Pérez, “A set of complexity measures
designed for applying meta-learning to instance selection,” IEEE Trans-
actions on Knowledge and Data Engineering, vol. 27, no. 2, pp. 354–
367, 2014.
[20] L. P. F. Garcia, A. C. P. L. F. de Carvalho, and A. C. Lorena,
“Effect of label noise in the complexity of classification problems,”
Neurocomputing, vol. 160, pp. 108–119, 2015.
[21] A. K. Tanwani and M. Farooq, “Classification potential vs. classifica-
tion accuracy: a comprehensive study of evolutionary algorithms with
biomedical datasets,” Learning Classifier Systems, vol. 6471, pp. 127–
144, 2010.
[22] A. Hoekstra and R. P. W. Duin, “On the nonlinearity of pattern
classifiers,” in 13th ICPR, vol. 4, 1996, pp. 271–275.
[23] J. Gower, “A general coefficient of similarity and some of its properties,”
Biometrics, vol. 27, no. 4, pp. 857–871, 1971.
[24] G. Morais and R. C. Prati, “Complex network measures for data set
characterization,” in 2nd BRACIS, 2013, pp. 12–18.
[25] T. K. Ho, “The random subspace method for constructing decision
forests,” IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, vol. 20, no. 8, p. 832844, 1998.
[26] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
5–32, 2001.
[27] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,”
Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.