An Experimental Study of Learning Curves
for Statistical Pattern Classifiers
Tsutomu MATSUNAGA and Hiromi KIDA
Research and Development Headquarters
NTT DATA COMMUNICATIONS SYSTEMS CORPORATION
66-2 Horikawa-cho, Saiwai-ku, Kawasaki-shi, Kanagawa. 210 Japan
Abstract
Statistical pattern classifiers are designed by pop-
ulation panameters of pattern distributions estimated
by a set of training samples. Therefore, classification
performance depends considerably on training sam-
ple size. Learning curves exhibit asymptotic behav-
iors where a probability of misclassification decreases
as a number of training samples increases. This pa-
per presents asymptotic behaviors of eflects of training
sample size and shows that learning curves for pradi-
cal purpose can be obtained using available samples.
1 Introduction
Classifier design is an actual problem in develop-
ment of pattern recognition systems. Statistical pat-
tern classifiers are widely employed in the processing
of recognition of character and speech. The statis-
tical classifiers are designed by population parame-
ters of pattern distributions such as mean vectors and
covariance matrices [l]. The population parameters
are estimated by a set of training samples. There-
fore, classification performance depends considerably
on training sample size. However, only a finite, fre-
quently small, number of training samples are usually
available. Since collecting many samples requires time
and cost, it is useful to know the number of training
samples necessary for a desired level of classification
performance for practical implement a t ion.
Research efforts on effects of sample size have been
conducted [2, 31. But it is of little use for practical
purpose since the pattern distribution is limited. It
has been simply considered that the larger the training
sample size, the better the classification performance.
In a conventional approach, every time the number
of training samples increases, the performance of the
classifier must be re-evaluated.
1103
$4.000 1995 IEEE
0-8186-7128-9/95
Authorized licensed use limited to: UNIVERSIDAD DE ALICANTE . Downloaded on March 31,2024 at 11:03:56 UTC from IEEE Xplore. Restrictions apply.
Learning curves [4] exhibit asymptotic behaviors
where a probability of misclassification decreases as
a number of training samples increases. In this pa-
per, learning curves on statistical classifiers are eval-
uated through Monte-Carlo simulations. Experimen-
tal results of two-class problems having multi-variate
normal distributions are discussed while pattern dis-
tributions are controlled with population parameters.
Next, an experiment of a character recognition prob-
lem whose population parameters of underlying pat-
tern distributions are unknown, is examined. The aim
of this paper is to gain insight on effects of train-
ing sample size and to show that learning curves can
be obtained experimentally using available real world
samples.
2 Pattern classification
2.1 Classification problem
When there exists two classes C l and Cz and the
random variable X has a continuous density function
f i ( X ) in the first population and f 2 ( X ) in the second,
and PI and Pz are the a priori probabilities, the prob-
ability of misclassification E can be written as follows:
E = P l k 2 f l ( X ) d X + Pz Ll fZ(X)dX
= PlE1 I-P 2 E 2 (1)
where
Ei : probability that an observation X from Ci is misclassified
Classification problems having an equal a priori prob-
abilities are discussed in this paper.
2.2 Statistical pattern classifier
In this paper, three commonly used statistical clas-
sifiers: quadratic discriminant function, Fisher's linear
discriminant function and Euclidean distance Classifier
are discussed, A classifier gives a decision rule for a set
of classes to classify an observation X into one class
Ci. The types of the classifiers are defined as:
(a) Quadratic Discriminant Function (QDF):
&X) = (X- Crk)TC,'(X - Pk) + kJlCkl (2)
Assign X to class Ci if gp(X) = m;ln{gf(X)}.
(b) Fisher's Linear Discriminant Function (FLDF):
&(X) = (X- Pk)TC;l(X - c l k )
Assign X to class C, if gr(X) = mF{g:(X)}.
(c) Euclidean Distance Classifier (ED):
&(X) = ( X - I r k Y
Assign X to class Ci if gF(X) = min{gf(X)}.
k
where XT indicates the transpose of the vector X and
C-l is the inverse of the matrix C. /.tk and Ck are the
kth population mean vector and the kth population
covariance matrix respectively, and Cw is within-class
scatter matrix. Since all the population parameters
are not known in most practical situations, the sample
estimates, which are obtained from a set of training
samples, substitute for the true parameters.
3 Learning curve
Learning curves on statistical classifiers describe in-
creases in probability of misclassification due to esti-
mates of population parameters from training sam-
ples. The experimental learning curve requires
(i) a monotonous decreasing function where the
probabilities of misclassification decreases gradu-
ally up to the generalization probabilities of mis-
classification as the number of training samples
increases.
(ii) a common function that represents the wide vit
riety of asymptotic behaviors characterized by a
small number of parameters.
From the viewpoints, a function of the experimental
learning curve is chosen as:
E> = E,.,-- + ccNCe
where
E> : approximate probability of misclassification
t N + ,: generalization probability of misclassification
N : number of training samples
cc, ce : parameter ( c C l 0 , Ce50)
Equation ( 5 ) is a general formulation of learning
curves as a function of training sample size. The pa-
rameters E ~ - - , CC and ce are estimated using a least
Authorized licensed use limited to: UNIVERSIDAD DE ALICANTE . Downloaded on March 31,2024 at 11:03:56 UTC from IEEE Xplore. Restrictions apply.
mean-squared error technique between probabilities of
misclassification obtained in experiments and the cor-
responding approximate probabilities given in equa-
tion ( 5 ) . Experimental learning curves are calculated
for the combinations of the type of the classifier, the
dimensionality, and the generalization probability of
misclassification. In addition, a generalization ratio
vN is defined here as:
(3)
The generalization ratio is calculated using a learning
curve. This measures a learning effect produced by
one training sample.
(4)
4 Experiment
4.1 Artificial pattern
In the following, the probability of misclassification
related with pattern distributions is briefly described
where the populations have multi-variate normal dis-
tributions. Under the pattern distributions mentioned
above, the decision rule by quadratic discriminant
function can be optimal in the Bayes sense. If popula-
tion parameters share the same covariance matrices ',
Fisher's linear discriminant function equates t o Bayes
classification rule. Moreover, Euclidean distance clas-
sifier can perform optimally only when the covariance
matrices are identity matrices. Here, the increase in
the probability of misclassification is shown to be
proportional to and dependent on the dimensional-
ity [3].
In this experiment, mean vectors pi and covariance
matrices Ci are formalized as follows:
'The probability of misclassification only depends on Maha-
lanobis distance between classes.
2Fukunaga et al. 12) have derived theoretical approximate
equations under the assumption of equal covariance matrices.
The equation for quadric discriminant function:
1 6 - - - 164) } 6=
(5)
E ~ = d + - .1
4N
~ { P 2 + ( 1 + z ) P + ( 62
2
(7)
The equation for Fisher's linear discriminant function:
€%=E+-*
.mo((l+ --)p- 62 1)
(8)
where 2N 6
E : true probability of misclassification
6 : Mahalanobis distance between classes
4 : density function of standard normal distribution
31t is known that any two covariance matrices can be simul-
taneously diagonized and a coordinate shift can bring the mean
vector of one class to zero, without loss of generality [I].
1104

LU TP J the expected misclassifiication rate with approximate
Artificial samples are generated from each class ac-
cording to given population parameters. A classi-
fier is designed by random chosen training samples of
equal size N from each class, and then a misclassifi-
cation rate is measured on 50,000 test samples drawn
independently from each class. This procedure was
repeated 500 times independently, and an expected
probability of misclassification is obtained by averag-
ing over the misclassification rates in the 500 trials.
The training sample sizes N are 1, 2, 5, 10, 20, 50,
and 100. The dimensionality p is varied from 1 to 32
in powers of 2,except for Case 3.
4.1.1 Case 1: Cl=&
When both covariance matrices are identity matrix,
every classifier can perform optimally, yielding the
Bayes error rate 4. An experimental learning curve
of Fisher’s linear discriminant function for the Bayes
error rate of 10% is shown in Figure 1 as a function
of sample size where the dimensionality is 8. The ex-
pected misclassification rate verses sample size is also
plotted. The vertical lines on the curve show the stan-
dard deviation for each sample size. The experimental
learning curve fits quite well to the expected misclas-
sification rates. The curve shows such a similar be-
havior to the theoretical approximate equation (see
equation (8)) that the misclassification rate decreases
exponentially as the sample size increases although the
misclassification rates given in the equation are under-
estimated in the range of small sample size. The esti-
mated values of parameter ce in equation ( 5 ) does not
always equal -1 (see equation (8)), and CC increases
linearly for logarithmic scale of dimensionality as the
dimensionality increases [5].
4.1.2 Case 2: &=AX1
In Figure 2, the expected misclassification rates of
quadratic discriminant function for a covariance shift
A=4 are illustrated on a gray-scale over the sample size
and the dimensionality 5 . The figure reveals that more
training samples are required to maintain the same
degree of classification performance as the dimension-
ality increases. There exists peaks in classification ac-
curacy as dimensionality is increased, where the op-
‘The Bayes error rate can be controlled by setting the mean
vector of the first component ml to the corresponding value
and of all other to 0, and it remains unchanged for the fired my
even with different dimensionality.
5The mean vector is fixed as ml=2.56,mi=0 ( k 2 , . .,p). .
1105
Authorized licensed use limited to: UNIVERSIDAD DE ALICANTE . Downloaded on March 31,2024 at 11:03:56 UTC from IEEE Xplore. Restrictions apply.
timal dimensionality is assigned to the sample size.
The contour map in Figure 2 was drawn by replacing
misclassification rate by the estimated experimental
learning curves for the corresponding combinations of
the training sample size and the dimensionality. The
contour map mostly trace over the asymptotic behav-
ior even though the learning curves are estimated in-
dependently for the different dimensionality.
4.1.3 Case 3: Cl#&
The population parameters in equation (10) are
shown in Table 1. The estimated learning curve pa-
rameters are shown in Table 2. As expected, the esti-
mated rate, 2.08% of quadratic discriminant function
nearly agrees with the Bayes error rate, 1.9%. The
generation ratios qiooand qiooare also shown in Table
2. The ratios indicate that Euclidean distance clas-
sifier performs the least improvement in classification
performance among the three classifiers. These ratios
should be taken into consideration in classifier design,
Table 1: Population parameter
i l 2 3 4 5 6 7 8
mi 3.86 3.10 0.84 0.84 1.64 1.08 0.26 0.01
r; 8.41 12.06 0.12 0.22 1.49 1.77 0.35 2.73
Table 2: Learning curve parameter
0.0869 0.13 -1.01 1.0 X lo-’ 2.0 X
4.2 Character pattern
An experiment using handprinted Japanese Kanji
character patterns in five classes (,,”, ”%”, ”T”,
’)I$)’, and ”$ ”) is examined. Figure 3 shows an ex-
ample of the character pattern samples. The number
of samples per class used are 670 for design and 500
for testing. The training sample sizes N are 1, 2, 5,
10, 20, 50, and 60. The: misclassification rates and
standard deviations for Euclidean distance classifier
computed in 10 independent trials for each sample size
are shown in Table 3. The approximate misclassifi-
cation rates calculated by the experimental learning
6The Bayes error rate is known to be 1.9% [I].
’The PDC (Peripheral Direction Contributivity) feature [e],
known to be useful in the recognition of Kanji character, is
used as the feature for the classification. A character patterns
is represented by a 768-dimensional feature vector.
 curve are shown in Table 4. The table shows that the
approximate rates almost agree with the rates of the
corresponding sample sizes in Table 3. Finally, the
misclassification rate is 1.72% where all training sam-
ples are used (the sample size is 670). As compared
with the approximate misclassification rate (1.65%),
it is clear that the experimental learning curve gives a
good approximation.
[5] T. Matsunaga, and H. Kida, “An Experimental
Study of Learning Curve on Linear Classifiers,”
PRU94-56,pp. 67-74, 1994. (in Japanese)
[6] N. Hagita, S . Naito, and I. Masuda, “Handprinted
Kanji Characters Recognition based on Pattern
Matching Method,” Proc. ICTP’83, pp. 169-174,
1983.

-+
-
experimentalmean
exoerimental meanistandad deviation
N 1
22.6
2
13.2
5
5.74
10
3.74
20
2.53
50
1.97
60
1.98
-- exbrimentd ieaming cuwe
approximate equation by Fukunaga

U 6.55 5.55 2.76 1.08 0.454 0.457 0.342 50.0 ! \

I

Table 4: Approximate misclassification rate

5 Conclusion 1 10 100
Sample Size
Experimental results have shown that learning
curves formulated from equation ( 5 ) can be obtained Figure 1: Learning curve (FLDF) Cl=&, p = 8
experimentally and that they express the proper
asymptotic behaviors of effects of training sample size.
The main goal of this work is to establish an effec-
tive classifier design approach which makes optimal
use of a finite number of available samples. The anal-
Above
31.0 -- 355
355

ysis of the statistical nature of learning curves and .*

282 91.0

-
25.1 282-
the interval estimate of the approximate probability - 22.4 25.1
204-22.4
of misclassification will be the subject of future study. .-E7
2
--
17.8 206

E4
15.8
14.1
126
--
17.8
15.8
14.1

References
8
112 --
126

oa -
2 10.0 112
10.0
7.8- U
I<. Fukunaga, “Introduction to Statistical Pattern 1 Belor 70
Recognition,” 2nd ed., Academic Press, New-York,
1990.
K. Fukunaga and R. Hayes, “Effectsof Sample Size Figure 2: MkCh~ificatiQn
rate (QDF) &=4&
in Classifier Design,” IEEE ?Fans. PAMI-2, No.8,
pp. 873-885, 1989.
S. J . Raudys and A. I<. Jain, “Small Sample Size
Effect in Statistical Pattern Recognition: Rec-
ommend ations for Practitioners,” IEEE Trans.
PAMI-13, NO.3, pp. 252-264, 1991.
S. Amari, N. Fujita and S. Shinomoto, “Four
Types of Learning Curves,” Neural Computation, Figure 3: Example of character pattern
NO.4, pp. 605-618, 1992.

1106

Authorized licensed use limited to: UNIVERSIDAD DE ALICANTE . Downloaded on March 31,2024 at 11:03:56 UTC from IEEE Xplore. Restrictions apply.