Full Chapter Pattern Recognition Introduction Features Classifiers and Principles de Gruyter Textbook 2Nd Edition Beyerer PDF

Pattern Recognition Introduction
Features Classifiers and Principles De

Gruyter Textbook 2nd Edition Beyerer
Visit to download the full and correct content document:
https://textbookfull.com/product/pattern-recognition-introduction-features-classifiers-a
nd-principles-de-gruyter-textbook-2nd-edition-beyerer/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Organic Chemistry 25 Must Know Classes of Organic

Compounds De Gruyter Textbook 2nd Edition Elzagheid
https://textbookfull.com/product/organic-chemistry-25-must-know-
classes-of-organic-compounds-de-gruyter-textbook-2nd-edition-
elzagheid/
Carbon for Micro and Nano Devices De Gruyter Textbook

1st Edition Sharma
https://textbookfull.com/product/carbon-for-micro-and-nano-
devices-de-gruyter-textbook-1st-edition-sharma/
Thermal Engineering: Engineering Thermodynamics and

Heat Transfer (De Gruyter Textbook) 1st Edition Qiu
https://textbookfull.com/product/thermal-engineering-engineering-
thermodynamics-and-heat-transfer-de-gruyter-textbook-1st-edition-
qiu/
Differential Equations A First Course on Ode and a

Brief Introduction to Pde De Gruyter Textbook 1st
Edition Shair Ahmad
https://textbookfull.com/product/differential-equations-a-first-
course-on-ode-and-a-brief-introduction-to-pde-de-gruyter-
textbook-1st-edition-shair-ahmad/
Semiconductor Quantum Light Sources: Fundamentals,
Technologies and Devices (De Gruyter Textbook) 1st
Edition Michler
https://textbookfull.com/product/semiconductor-quantum-light-
sources-fundamentals-technologies-and-devices-de-gruyter-
textbook-1st-edition-michler/
Relaxation in Optimization Theory and Variational

Calculus De Gruyter Nonlinear Analysis and Applications
De Gruyter Series in Nonlinear Analysis and
Applications 4 2nd Edition Roubíek
https://textbookfull.com/product/relaxation-in-optimization-
theory-and-variational-calculus-de-gruyter-nonlinear-analysis-
and-applications-de-gruyter-series-in-nonlinear-analysis-and-
applications-4-2nd-edition-roubiek/
Water Chemistry Analysis and Treatment Pollutants

Microbial Contaminants Water and Wastewater Treatment
De Gruyter Textbook 1st Edition Elzagheid
https://textbookfull.com/product/water-chemistry-analysis-and-
treatment-pollutants-microbial-contaminants-water-and-wastewater-
treatment-de-gruyter-textbook-1st-edition-elzagheid/
Water Resources Management Innovative and Green

Solutions De Gruyter STEM 2nd Edition Brears
https://textbookfull.com/product/water-resources-management-
innovative-and-green-solutions-de-gruyter-stem-2nd-edition-
brears/
Introduction to Distributed Computer Systems Principles

and Features 1st Edition Ludwik Czaja (Auth.)
https://textbookfull.com/product/introduction-to-distributed-
computer-systems-principles-and-features-1st-edition-ludwik-
czaja-auth/
Jürgen Beyerer, Raphael Hagmanns, Daniel Stadler
Pattern Recognition
Also of interest
Mathematics of Deep Learning
An Introduction
Leonid Berlyand, Pierre-Emmanuel Jabin, 2023
ISBN 978-3-11-102431-8, e-ISBN (PDF) 978-3-11-102555-1
Medical Image Reconstruction

From Analytical and Iterative Methods to Machine Learning
Gengsheng Lawrence Zeng, 2023
ISBN 978-3-11-105503-9, e-ISBN (PDF) 978-3-11-105540-4
Category Theory
Invariances and Symmetries in Computer Science
Edited by Siddhartha Bhattacharyya, Vaclav Snasel, Aboul Ella Hassanien
Zoran Majkic, 2023
ISBN 978-3-11-108056-7, e-ISBN (PDF) 978-3-11-108167-0
Mathematical Logic
An Introduction
Daniel Cunningham, 2023
ISBN 978-3-11-078201-1, e-ISBN (PDF) 978-3-11-078207-3
Algorithms
Design and Analysis
Sushil C. Dimri, Preeti Malik, Mangey Ram, 2021
ISBN 978-3-11-069341-6, e-ISBN (PDF) 978-3-11-069360-7
Jürgen Beyerer, Raphael Hagmanns, Daniel Stadler
Pattern Recognition
Introduction, Features, Classifiers and Principles
2nd Edition
Authors
Prof. Dr.-Ing. habil. Jürgen Beyerer
Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB)
Fraunhoferstr. 1
76131 Karlsruhe
juergen.beyerer@iosb.fraunhofer.de
-and-
Institute for Anthropomatics and Robotics, Chair IES
Karlsruhe Institute of Technology (KIT)
Haid-und-Neu-Str. 7
76131 Karlsruhe
Raphael Hagmanns
Haid-und-Neu-Str. 7
76131 Karlsruhe
raphael.hagmanns@kit.edu
Daniel Stadler
Haid-und-Neu-Str. 7
76131 Karlsruhe
daniel.stadler@kit.edu
ISBN 978-3-11-133919-1
e-ISBN (PDF) 978-3-11-133920-7
e-ISBN (EPUB) 978-3-11-133941-2
Library of Congress Control Number: 2024931594
Bibliographic information published by the Deutsche Nationalbibliothek

The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed
bibliographic data are available on the Internet at http://dnb.dnb.de.
© 2024 Walter de Gruyter GmbH, Berlin/Boston

Printing and binding: CPI books GmbH, Leck
Cover image: Narong KHUEANKAEW / iStock / Getty Images Plus
∞ Printed on acid-free paper
Printed in Germany
www.degruyter.com
Preface
Pattern Recognition ⊂ Machine Learning ⊂ Artificial Intelligence: This
relation could give the impression that pattern recognition is only a tiny, very special-
ized topic. That, however, is misleading. Pattern recognition is a very important field
of machine learning and artificial intelligence with its own rich structure and many
interesting principles and challenges. For humans, and also for animals, their natural
abilities to recognize patterns are essential for navigating the physical world which they
perceive with their naturally given senses. Pattern recognition here performs an impor-
tant abstraction from sensory signals to categories: on the most basic level, it enables
the classification of objects into “Eatable” or “Not eatable” or, e.g., into “Friend” or “Foe”.
These categories (or, synonymously, classes) do not always have a tangible character. Ex-
amples of non-material classes are, e.g., “secure situation” or “dangerous situation”. Such
classes may even shift depending on the context, for example, when deciding whether
an action is socially acceptable or not. Therefore, everybody is very much acquainted, at
least at an intuitive level, with what pattern recognition means to our daily life. This fact
is surely one reason why pattern recognition as a technical subdiscipline is a source of so
much inspiration for scientists and engineers. In order to implement pattern recognition
capabilities in technical systems, it is necessary to formalize it in such a way, that the
designer of a pattern recognition system can systematically engineer the algorithms and
devices necessary for a technical realization. This textbook summarizes a lecture course
about pattern recognition that one of the authors (Jürgen Beyerer) has been giving for
students of technical and natural sciences at the Karlsruhe Institute of Technology (KIT)
since 2005. The aim of this book is to introduce the essential principles, concepts and chal-
lenges of pattern recognition in a comprehensive and illuminating presentation. We will
try to explain all aspects of pattern recognition in a well understandable, self-contained
fashion. Facts are explained with a mixture of a sufficiently deep mathematical treat-
ment, but without going into the very last technical details of a mathematical proof. The
given explanations will aid readers to understand the essential ideas and to compre-
hend their interrelations. Above all, readers will gain the big picture that underlies all
of pattern recognition.
The authors would like to thank their peers and colleagues for their support:
Special thanks are owed to Dr. Ioana Gheţa who was very engaged during the early
phases of the lecture “Pattern Recognition” at the KIT. She prepared most of the many
slides and accompanied the course along many lecture periods.
Thanks as well to Dr. Martin Grafmüller and to Dr. Miro Taphanel for supporting
the lecture Pattern Recognition with great dedication.
Moreover, many thanks to Prof. Michael Heizmann and Prof. Fernando Puente León
for inspiring discussions, which have positively influenced to the evolution of the lecture.
Thanks to Christian Hermann and Lars Sommer for providing additional figures
and examples of deep learning. Our gratitude also to our friends and colleagues Alexey
https://doi.org/10.1515/9783111339207-202
VI  Preface
Pak, Ankush Meshram, Chengchao Qu, Christian Hermann, Ding Luo, Julius Pfrommer,
Julius Krause, Johannes Meyer, Lars Sommer, Mahsa Mohammadikaji, Mathias Anneken,
Mathias Ziebarth, Miro Taphanel, Patrick Philipp, and Zheng Li for providing valuable
input and corrections for the preparation of this manuscript.
Lastly, we thank DeGruyter for their support and collaboration in this project.
Karlsruhe, Summer 2017

Jürgen Beyerer
Matthias Richter
Matthias Nagel
Preface of 2nd edition
In recent years, “Pattern Recognition” has emerged as one of the fastest evolving areas in
machine learning. In a landscape characterized by rapid advances, our commitment to
providing a comprehensive and contemporary resource is reflected by various content
modernizations. Several new classifiers are introduced to align with current problem
formulations. In addition, different fundamental sections have been added to provide
further insight into important topics such as Gaussian mixture models and their param-
eter estimation, hidden Markov models, or transformer models. The updates include a
thorough unification of the notation, along with fontification, aiming for a more cohesive
reading experience. All changes align with a major update of the “Pattern Recognition”
lecture at the Karlsruhe Institute of Technology (KIT).
We would like to thank Stefan Wolf for accompanying this lecture with great dedica-
tion. Special thanks go to Robert Zimmermann and Dr. Christian Frese for proofreading
various parts of the second edition. Finally, we would like to thank the team at DeGruyter
for their continuous and inexhaustible efforts in overcoming the challenges of applying
the new book template.
Karlsruhe, Spring 2024

Jürgen Beyerer
Raphael Hagmanns
Daniel Stadler
https://doi.org/10.1515/9783111339207-203
Contents
Preface  V
Preface of 2nd edition  VII
List of Tables  XIII
List of Figures  XV
Notation  XIX
Introduction  XXIII
1 Fundamentals and definitions  1

1.1 Goals of pattern recognition  1
1.2 Structure of a pattern recognition system  2
1.3 Abstract view of pattern recognition  4
1.4 Design of a pattern recognition system  5
1.5 Exercises  9
2 Features  10
2.1 Types of features and their traits  10
2.1.1 Nominal scale  10
2.1.2 Ordinal scale  12
2.1.3 Interval scale  12
2.1.4 Ratio scale and absolute scale  13
2.2 Feature space inspection  13
2.2.1 Projections  14
2.2.2 Intersections and slices  15
2.3 Transformations of the feature space  17
2.4 Measurement of distances in the feature space  17
2.4.1 Basic definitions  19
2.4.2 Elementary norms and metrics  20
2.4.3 A metric for sets  21
2.4.4 Metrics on the ordinal scale  23
2.4.5 The cosine distance  23
2.4.6 The Kullback–Leibler divergence  24
2.4.7 The t-distributed stochastic neighbor embedding  29
2.4.8 Tangential distance measure  31
X  Contents
2.5 Normalization  34
2.5.1 Alignment, elimination of physical dimension, and leveling of
proportions  35
2.5.2 Lighting adjustment of images  35
2.5.3 Distortion adjustment of images  39
2.5.4 Dynamic time warping  40
2.6 Selection and construction of features  41
2.6.1 Descriptive features  41
2.6.2 Model-driven features  45
2.6.3 Construction of invariant features  50
2.7 Dimensionality reduction of the feature space  59
2.7.1 Principal component analysis  60
2.7.2 Kernelized principal component analysis  73
2.7.3 Independent component analysis  79
2.7.4 Multiple discriminant analysis  83
2.7.5 Dimensionality reduction with t-SNE  90
2.7.6 Autoencoder  91
2.7.7 Dimensionality reduction by feature selection  92
2.7.8 Bag of words  94
2.8 Exercises  99
3 Bayesian decision theory  103

3.1 General considerations  103
3.2 The maximum a posteriori classifier  106
3.3 Bayesian classification  109
3.3.1 The Bayesian optimal classifier  109
3.3.2 Reference example: Optimal decision regions  114
3.3.3 The naive Bayes classifier  116
3.3.4 The minimax classifier  118
3.3.5 Normally distributed features  120
3.3.6 Arbitrarily distributed features  125
3.4 Gaussian mixtures  126
4 Parameter estimation  132

4.1 Maximum likelihood estimation  140
4.2 Bayesian estimation of the class-specific distributions  142
4.3 Bayesian parameter estimation  147
4.3.1 Least squared estimation error  148
4.3.2 Constant penalty for failures  148
4.4 Additional remarks on Bayesian classification  149
Contents  XI
4.5 Parameter estimation for Gaussian mixtures  150

5 Parameter free methods  161

5.1 The Parzen window method  165
5.2 The k-nearest neighbor method  169
5.3 k-nearest neighbor classification  173
6 General considerations  180

6.1 Dimensionality of the feature space  180
6.2 Overfitting  187
7 Special classifiers  191

7.1 Linear discriminants  191
7.1.1 More than two classes  191
7.1.2 Nonlinear separation  193
7.2 The perceptron  195
7.3 Linear regression  197
7.4 Artificial neural networks  198
7.5 Autoencoders  204
7.6 Deep learning  206
7.6.1 Historical difficulties and successful approaches  207
7.6.2 Unsupervised pre-training  208
7.6.3 Stochastic gradient descent  208
7.6.4 Rectified linear units  210
7.6.5 Convolutional neural networks  210
7.6.6 Residual networks  215
7.6.7 Variational autoencoders  216
7.6.8 Mixture density networks  217
7.7 Support vector machines  219
7.7.1 Linear separation with maximum margin  219
7.7.2 Dual formulation  221
7.7.3 Nonlinear mapping  223
7.7.4 The kernel trick  224
7.7.5 No linear separability  228
7.7.6 Discussion  230
7.8 Matched filters  231
7.9 Classification of sequences  234
7.9.1 Markov models  235
XII  Contents
7.9.2 Hidden Markov models  236

7.9.3 Recurrent neural networks  246
7.9.4 Transformers  249
7.9.5 Generative adversarial networks  252
8 Classification with nominal features  257

8.1 Decision trees  257
8.1.1 Decision tree learning  260
8.1.2 Influence of the features used  264
8.2 Random forests  265
8.3 String matching  269
8.4 Grammars  270
9 Classifier-independent concepts  273

9.1 Learning theory  273
9.1.1 The central problem of statistical learning  274
9.1.2 Vapnik–Chervonenkis learning theory  274
9.2 No-free-lunch theorem  277
9.3 Empirical evaluation of classifier performance  278
9.3.1 Receiver operating characteristic  281
9.3.2 Multi-class setting  282
9.3.3 Theoretical bounds with finite test sets  283
9.3.4 Dealing with small datasets  284
9.4 Boosting  286
9.5 Rejection  288
A Solutions to the exercises  293
B A primer on Lie theory  307
C Random processes  311
Bibliography  315
Glossary  319
Index  325
List of Tables
Tab. 1 Capabilities of humans and machines in relation to pattern recognition  XXV
Tab. 2.1 Taxonomy of scales of measurement  11

Tab. 2.2 Topology of the letters of the German alphabet  43
Tab. 2.3 Contour transformations and approaches for constructing invariant features  59
Tab. 7.1 Character sequences generated by Markov models of different order  237
Tab. 9.1 Common binary classification performance measures  281
https://doi.org/10.1515/9783111339207-205
List of Figures
Fig. 1 Examples of artificial and natural objects  XXIII
Fig. 2 Industrial bulk material sorting system  XXIV
Fig. 1.1 Transformation of the domain into the feature space  2

Fig. 1.2 Processing pipeline of a pattern recognition system  3
Fig. 1.3 Abstract steps in pattern recognition  5
Fig. 1.4 Design phases of a pattern recognition system  6
Fig. 1.5 Rule of thumb to partition the dataset into training, validation, and test sets  7
Fig. 2.1 Iris flower dataset  14

Fig. 2.2 Full projection and slice projection techniques  15
Fig. 2.3 Construction of two-dimensional slices  16
Fig. 2.4 Feature transformation for dimensionality reduction  18
Fig. 2.5 Unit circles for different Minkowski norms  22
Fig. 2.6 KL divergence between two Bernoulli distributions  26
Fig. 2.7 KL divergence of Gaussian distributions with equal variance  27
Fig. 2.8 KL divergence of Gaussian distributions with unequal variances  27
Fig. 2.9 Pairs of rectangle-like densities  27
Fig. 2.10 Combustion engine, microscopic image of bore texture, and texture model  28
Fig. 2.11 Systematic variations in optical character recognition  31
Fig. 2.12 Tangential distance measure  32
Fig. 2.13 Linear approximation of the variation in Figure 2.11  34
Fig. 2.14 Chromaticity normalization  36
Fig. 2.15 Normalization of lighting conditions  37
Fig. 2.16 Images of the surface of agglomerated cork  38
Fig. 2.17 Adjustment of geometric distortions  40
Fig. 2.18 Adjustment of temporal distortions  40
Fig. 2.19 Different bounding boxes around an object  42
Fig. 2.20 The convex hull around a concave object  42
Fig. 2.21 Degree of compactness  43
Fig. 2.22 Classification of faulty milling cutters  44
Fig. 2.23 Synthetic honing textures using an AR model  48
Fig. 2.24 Physical formation process and parametric model of a honing texture  49
Fig. 2.25 Synthetic honing texture using a physically motivated model  50
Fig. 2.26 Impact of object variation and variation of patterns on the features  51
Fig. 2.27 Scheme of a feature and its orbit  52
Fig. 2.28 Synthesis of a two-dimensional contour  56
Fig. 2.29 Principal component analysis, first step  61
Fig. 2.30 Principal component analysis, second step  63
Fig. 2.31 Principal component analysis, general case  64
Fig. 2.32 Variance of the dataset encoded in principal components  67
Fig. 2.33 Mean face of the YALE faces dataset  70
Fig. 2.34 First 20 eigenfaces of the YALE faces dataset  71
Fig. 2.35 First 20 eigenvalues corresponding to the eigenfaces in Figure 2.34  71
Fig. 2.36 Wireframe model of an airplane  72
Fig. 2.37 Concept of kernelized PCA  73
https://doi.org/10.1515/9783111339207-206
XVI  List of Figures
Fig. 2.38 Kernelized PCA with radial kernel function  79

Fig. 2.39 Concept of independent component analysis  80
Fig. 2.40 Effect of an independent component analysis  81
Fig. 2.41 PCA not taking class separability into account  84
Fig. 2.42 Multiple discriminant analysis  85
Fig. 2.43 First 10 Fisherfaces of the YALE faces dataset  89
Fig. 2.44 t-SNE applied to two spirals  90
Fig. 2.45 Illustration of the t-SNE algorithm  91
Fig. 2.46 Autoencoder with one hidden layer  92
Fig. 2.47 Workflow of feature selection  93
Fig. 2.48 Underlying idea of bag of visual words  95
Fig. 2.49 Example of a visual vocabulary  96
Fig. 2.50 Example of a bag of words descriptor  97
Fig. 2.51 Bag of words for bulk material sorting  98
Fig. 2.52 Structure of the bag of words approach  99
Fig. 3.1 Example of a random distribution of mixed discrete and continuous quantities  104
Fig. 3.2 The decision space  105
Fig. 3.3 Workflow of the MAP classifier  108
Fig. 3.4 3-dimensional probability simplex in barycentric coordinates  108
Fig. 3.5 Connection between the likelihood ratio and the optimal decision region  112
Fig. 3.6 Decision of a MAP classifier in relation to the a posteriori probabilities  113
Fig. 3.7 Underlying densities in the reference example for classification  114
Fig. 3.8 Optimal decision regions  115
Fig. 3.9 Naive Bayes parameter estimation  117
Fig. 3.10 Risk of the minimax classifier  119
Fig. 3.11 Decision boundary with uneven priors  122
Fig. 3.12 Decision regions of a generic Gaussian classifier  124
Fig. 3.13 Decision regions of a generic two-class Gaussian classifier  124
Fig. 3.14 Decision regions of a Gaussian classifier with the reference example  125
Fig. 3.15 Iris flowers modeled with a GMM  126
Fig. 4.1 Comparison of estimators  139

Fig. 4.2 Sequence of Bayesian a posteriori densities  145
Fig. 4.3 Illustration of the EM algorithm  153
Fig. 4.4 EM algorithm on a one-dimensional dataset  158
Fig. 5.1 The triangle of inference  161

Fig. 5.2 Comparison of Parzen window and k-nearest neighbor density estimation  164
Fig. 5.3 Decision regions of a Parzen window classifier  169
Fig. 5.4 Parzen window density estimation in one dimension  170
Fig. 5.5 Parzen window density estimation in two dimensions  171
Fig. 5.6 k-nearest neighbor density estimation  172
Fig. 5.7 Example Voronoi tessellation of a two-dimensional feature space  174
Fig. 5.8 Dependence of the nearest neighbor classifier on the metric  175
Fig. 5.9 k-nearest neighbor classifier  175
Fig. 5.10 Decision regions of a nearest neighbor classifier  176
Fig. 5.11 Decision regions of a 3-nearest neighbor classifier  176
List of Figures  XVII
Fig. 5.12 Decision regions of a 5-nearest neighbor classifier  177

Fig. 5.13 Asymptotic error bounds of the nearest neighbor classifier  178
Fig. 6.1 Increasing dimension vs. overlapping densities  182

Fig. 6.2 Dependence of error rate on the dimension of the feature space  183
Fig. 6.3 Scheme of a hypersphere with a small shell  184
Fig. 6.4 Density of a sample for feature spaces of increasing dimensionality  184
Fig. 6.5 Examples for the interrelation of the feature space dimension and the parameter space
dimension  186
Fig. 6.6 Trade-off between generalization and training error  188
Fig. 6.7 Overfitting in a regression scenario  189
Fig. 7.1 Techniques for extending linear discriminants to more than two classes  192
Fig. 7.2 Nonlinear separation by augmentation of the feature space  194
Fig. 7.3 Decision regions of a linear regression classifier  195
Fig. 7.4 Four steps of the perceptron algorithm  197
Fig. 7.5 Feed-forward neural network with one hidden layer  199
Fig. 7.6 Single layer of a feed-forward neural network  201
Fig. 7.7 Updates of weights in a feed-forward neural network  202
Fig. 7.8 Decision regions of a feed-forward neural network  205
Fig. 7.9 Neuron activation of an autoencoder with three hidden neurons  206
Fig. 7.10 Pre-training with stacked autoencoders  209
Fig. 7.11 Comparison of ReLU and sigmoid activation functions  210
Fig. 7.12 A single convolution block in a convolutional neural network  211
Fig. 7.13 High level structure of a convolutional neural network  211
Fig. 7.14 Types of features captured in convolution blocks of a convolutional neural network  213
Fig. 7.15 Detection and classification of vehicles in aerial images with CNNs  214
Fig. 7.16 Structure of an exemplary CNN used for feature extraction  215
Fig. 7.17 Building block of ResNet  216
Fig. 7.18 Scheme of a varational autoencoder  217
Fig. 7.19 Scheme of a mixture density network  217
Fig. 7.20 Classification with maximum margin  220
Fig. 7.21 Decision regions of a hard margin SVM  227
Fig. 7.22 Geometric interpretation of the slack variables  229
Fig. 7.23 Decision regions of a soft margin SVM  230
Fig. 7.24 Decision boundaries of hard margin and soft margin SVMs  231
Fig. 7.25 Toy example of a matched filter  232
Fig. 7.26 Discrete first order Markov model with three states  236
Fig. 7.27 Discrete first order hidden Markov model  238
Fig. 7.28 Illustration of a recurrent neural network  246
Fig. 7.29 Unrolling of a recurrent neural network  247
Fig. 7.30 Long short-term memory cell  248
Fig. 7.31 Transformer architecture  250
Fig. 7.32 Concept of a generative adversarial network  253
Fig. 7.33 Scheme of C-RNN-GAN  254
Fig. 8.1 Decision tree to classify fruit  259

Fig. 8.2 Binarized version of the decision tree in Figure 8.1  259
XVIII  List of Figures
Fig. 8.3 Qualitative comparison of impurity measures  261

Fig. 8.4 Decision regions of a decision tree  262
Fig. 8.5 Structure of the decision tree of Figure 8.4  262
Fig. 8.6 Impact of the features used in decision tree learning  263
Fig. 8.7 A decision tree that does not generalize well  264
Fig. 8.8 Decision regions of a random forest  268
Fig. 8.9 Strict string matching  269
Fig. 8.10 Approximate string matching  270
Fig. 8.11 String matching with wildcard symbol  270
Fig. 8.12 Bottom up and top down parsing of a sequence  271
Fig. 9.1 Relation of the world model and training and test sets  274
Fig. 9.2 Sketch of different class assignments under different model families  275
Fig. 9.3 Expected test error, empirical training error, and VC confidence vs. VC dimension  276
Fig. 9.4 Classification error probability  279
Fig. 9.5 Classification outcomes in a 2-class scenario  280
Fig. 9.6 Performance indicators for a binary classifier  280
Fig. 9.7 Examples of ROC curves  282
Fig. 9.8 Converting a multi-class confusion matrix to binary confusion matrices  283
Fig. 9.9 Five-fold cross-validation  285
Fig. 9.10 Schematic example of AdaBoost training.  287
Fig. 9.11 AdaBoost classifier obtained by training in Figure 9.10  288
Fig. 9.12 Reasons to refuse to classify an object  288
Fig. 9.13 Classifier with rejection option  289
Fig. 9.14 Rejection criteria and the corresponding rejection regions  290
Notation
General identifiers
a, . . . , z Scalar, function mapping to a scalar, or a realization of a random variable

a, . . . , z Random variable (scalar)
a, . . . , z Vector, function mapping to a vector, or realization of a vectorial random variable
a, . . . , z Random variable (vectorial)
â, . . . , ẑ Realized estimator of denoted variable
â, . . . , ẑ Estimator of denoted variable as random variable itself
A, . . . , Z Matrix
A, . . . , Z Matrix as random variable
A, . . . , Z Set
A, . . . , Z System of sets
Special identifiers
c Number of classes
d Dimension of feature space
D Set of training samples
i, j, k Indices along the dimension, i.e., i, j, k ∈ {1, . . . , d}, or along the number of samples, i.e.,
i, j, k ∈ {1, . . . , N}
I Identity matrix
j Imaginary unit, j2 = −1
J Fisher information matrix
k(⋅,⋅) Kernel function
k(⋅) Decision function
K Decision space
l Cost function l : Ω0 /∼ × Ω/∼ → ℝ
L Cost matrix ∈ ℝ(c+1)×c
m Feature vector
mi Feature vector of the i-th sample
m ij The j-th component of the i-th feature vector
M ij The component at the i-th row and j-th column of the matrix M
M Feature space
N Number of samples
o Object
ω Class of objects, i.e., ω ⊆ Ω
ω0 Rejection class
Ω Set of objects (the relevant part of the world) Ω = {o1 , . . . , o N }
Ω/∼ The domain factorized w.r.t. the classes, i.e., the set of classes Ω/∼ = {ω1 , . . . , ω c }
Ω0 /∼ The set of classes including the rejection class, Ω0 /∼ = Ω/ {ω0 }
https://doi.org/10.1515/9783111339207-207
XX  Notation
p(m) Probability density function for random variable m evaluated at m

P(ω) Probability mass function for (discrete) random variable ω evaluated at ω
Pr(e) Probability of an event e
P(A) Power set, i.e., the set of all subsets of A
S Set of all samples, S = D ⊎ T ⊎ V
T Set of test samples
V Set of validation samples
U Unit matrix, i.e., the matrix all of whose entries are 1
θ Parameter vector
Θ Parameter space
General sets
ℂ Set of complex numbers

ℍ Poincaré half plane
ℕ Set of natural numbers (without zero)
ℕ0 Set of natural numbers (including zero)
ℚ Set of rational numbers
ℚ>0 , ℚ<0 Set of positive, negative rational numbers
ℝ Set of real numbers
ℝ>0 , ℝ<0 Set of positive, negative real numbers
ℤ Set of integer numbers
Special symbols
|A| Determinant of matrix A

|A| Cardinality of set A
“Proportional to”-relation
P
→ Convergence in probability
w
→ Weak convergence
⇝ Leads to (not necessarily in a strict mathematical sense)
⊎ Disjoint union of sets, i.e., C = A ⊎ B ⇔ C = A B and A B = 0
⟨⋅,⋅⟩ Scalar product
∇, ∇e Gradient, gradient w.r.t. e
Cov{⋅} Covariance
det(⋅) Determinant
∂a
∂b Derivative of a w.r.t. b
j j j
δi Kronecker delta/symbol; δ i = 1 iff i = j, else δ i = 0
δ[⋅] Generalized Kronecker symbol, i.e., δ[Π] = 1 iff Π is true and δ[Π] = 0 otherwise
E{⋅} Expected value
Notation  XXI
N(μ, σ 2 ) Normal/Gaussian distribution with expectation μ and variance σ 2

N(m; μ, σ 2 ) Normal/Gaussian distribution with expectation μ, variance σ 2 , and explicit nomination of
the random variable
N(µ, Σ) Multivariate normal/Gaussian distribution with expectation µ and covariance matrix Σ
N(m; µ, Σ) Multivariate normal/Gaussian distribution with expectation µ, covariance matrix Σ, and ex-
plicit nomination of the random variable
tr A Trace of the matrix A
Var{⋅} Variance
Abbreviations
cf. “confer” (latin: compare)

iff if and only if
i.i.d. independent and identically distributed
n.b. “nota bene” (latin: note well, take note)
w.r.t. with respect to
Introduction
The overall goal of pattern recognition is to develop systems that can distinguish and
classify objects. The range of possible objects is vast. Objects can be physical things
existing in the real world, like banknotes, as well as non-material entities, e.g., e-mails,
or abstract concepts such as actions or situations. The objects can be of natural origin
or artificially created. Examples of objects in pattern recognition tasks are shown in
Figure 1.
On the basis of recorded patterns, the task is to classify the objects into previously
assigned classes by defining and extracting suitable features. The type as well as the num-
ber of classes is given by the classification task. For example, banknotes (see Figure 1b)
could be classified according to their monetary value or the goal could be to discriminate
between real and counterfeited banknotes. For now, we will refrain from defining what
we mean by the terms pattern, feature, and class. Instead, we will rely on an intuitive
understanding of these concepts. A precise definition will be given in the next chapter.
From this short description, the fundamental elements of a pattern recognition task
and the challenges to be encountered at each step can be identified even without a precise
definition of the concepts pattern, feature, and class:
Pattern acquisition, sensing, measuring In the first step, suitable properties of the
objects to be classified have to be gathered and put into computable representations.
Although pattern might suggest that this (necessary) step is part of the actual pattern
recognition task, it is not. However, this process has to be considered so far as to
(a) Screws (b) Banknotes (c) Handwriting
(d) Plant seeds (e) Plant leaves
Fig. 1: Examples of artificial and natural objects.
https://doi.org/10.1515/9783111339207-208
XXIV  Introduction
Computer
Camera
(line-scan)
Illumination
Bulk material
Conveyor
Ejection
stage
Background plate
(a) Schematic overview (b) Inspection and ejection stage
Fig. 2: Industrial bulk material sorting system.
provide an awareness of any possible complications it may cause in the subsequent

steps. Measurements of any kind are usually affected by random noise and other
disturbances that, depending on the application, can not be mitigated by methods
of metrology alone: for example, changes of lighting conditions in uncontrolled and
uncontrollable environments. A pattern recognition system has to be designed so
that it is capable of solving the classification task regardless of such factors.
Feature definition, feature acquisition Suitable features have to be selected based on
the available patterns and methods for extracting these features from the patterns
have to be defined. The general aim is to find the smallest set of the most informative
and discriminative features. A feature is discriminative if it varies little with objects
within a single class, but varies significantly with objects from different classes.
Design of the classifier After the features have been determined, rules to assign a class
to an object have to be established. The underlying mathematical model has to be
selected so that it is powerful enough to discern all given classes and thus solve the
classification task. On the other hand, it should not be more complicated than it needs
to be. Determining a given classifier’s parameters is a typical learning problem and
is therefore also affected by the problems pertaining to this field. These topics will
be discussed in greater detail in Chapter 1.
These lecture notes on pattern recognition are mainly concerned with the last two issues.
The complete process of designing a pattern recognition system will be covered in its
entirety and the underlying mathematical background of the required building blocks
will be given in depth.
Pattern recognition systems are generally parts of larger systems, in which pattern
recognition is used to derive decisions from the result of the classification. Industrial
sorting systems are typical of this (see Figure 2). Here, products are processed differently
depending on their class memberships.
Introduction  XXV
Tab. 1: Capabilities of humans and machines in relation to pattern recognition.
Association & cognition Combinatorics & precision

Human very good poor
Machine medium very good
Hence, as a pattern recognition system is not an end in itself, the design of such a system
has to consider the consequences of a bad decision caused by a misclassification. This
puts pattern recognition between human and machine. The main advantage of auto-
matic pattern recognition is that it can execute recurring classification tasks with great
speed and without fatigue. However, an automatic classifier can only discern the classes
that were considered in the design phase and it can only use those features that were
defined in advance. A pattern recognition system to tell apples from oranges may label
a pear as an apple and a lemon as an orange if lemons and pears were not known in
the design phase. The features used for classification might be chosen poorly and not be
discriminative enough. Different environmental conditions (e.g., lighting) in the labora-
tory and in the field that were not considered beforehand might impair the classification
performance, too. Humans, on the other hand, can use their associative and cognitive
capabilities to achieve good classification performance even in adverse conditions. In
addition, humans are capable of undertaking further actions if they are unsure about a
decision. The contrasting abilities of humans and machines in relation to pattern recog-
nition are compared in Table 1. In many cases one will choose to build a hybrid system:
easy classification tasks will be processed automatically, ambiguous cases require hu-
man intervention, which may be aided by the machine, e.g., by providing a selection of
the most probable classes.
1 Fundamentals and definitions
The aim of this chapter is to describe the general structure of a pattern recognition
system and properly define the fundamental terms and concepts that were partially
used in the Introduction already. A description of the generic process of designing a
pattern recognizer will be given and the challenges at each step will be stated more
precisely.
1.1 Goals of pattern recognition
The purpose of pattern recognition is to assign classes to objects according to some

similarity properties. Before delving deeper, we must first define what is meant by class
and object. For this, two mathematical concepts are needed: equivalence relations and
partitions.
Definition 1.1 (Equivalence relation). Let Ω be a set of elements with some relation ∼. Sup-
pose further that o, o1 , o2 , o3 ∈ Ω are arbitrary. The relation ∼ is said to be an equivalence
relation if it fulfills the following conditions:
1. Reflexivity: o ∼ o.
2. Symmetry: o1 ∼ o2 ⇔ o2 ∼ o1 .
3. Transitivity: o1 ∼ o2 and o2 ∼ o3 ⇒ o1 ∼ o3 .
Two elements o1 , o2 with o1 ∼ o2 are said to be equivalent. We further write [o]∼ ⊆ Ω to

denote the subset
↑ 󸀠
[o]∼ = {o󸀠 ∈ Ω↑ ↑o ∼ o}
↑ (1.1)
of all elements that are equivalent to o. The object o is also called a representative of the
set [o]∼ (equivalence class). In the context of pattern recognition, each o ∈ Ω denotes an
object and each [o]∼ denotes a class. A different approach to classifying every element
of a set is given by partitioning the set:
Definition 1.2 (Partition, class). Let Ω be a set and ω1 , ω2 , ω3 , . . . ⊆ Ω be a system of

subsets. This system of subsets is called a partition of Ω if the following conditions are
met:
1. ω i ω j = 0 for all i ≠ j, i.e., the subsets are pairwise disjoint and
2. ⋃i ω i = Ω, i.e., the system is exhaustive.
Every subset ω is called a class (of the partition).
It is easy to see that equivalence relations and partitions describe synonymous concepts:
every equivalence relation induces a partition and every partition induces an equiva-
lence relation.
The underlying principle of all pattern recognition is illustrated in Figure 1.1. On the
left it shows—in abstract terms—the world and a (sub)set Ω of objects that live within
https://doi.org/10.1515/9783111339207-001
2  1 Fundamentals and definitions
m2 , . . . , m c
Ω ⊆ World
ωc Rc
Sensing,
ω1 measuring,
R2
characterizing: m1
⋅⋅⋅⋅⋅⋅ o i 󳨃→ mi
ω2 Ω ...
R1
Decision
boundaries
Fig. 1.1: Transformation of the domain Ω into the feature space M.
the world. The set Ω is given by the pattern recognition task and is also called the do-
main. Only the objects in the domain are relevant to the task; this is the so called closed
world assumption. The task also partitions the domain into classes ω1 , ω2 , ω3 , . . . ⊆ Ω. A
suitable mapping associates every object o i to a feature vector mi ∈ M inside the feature
space M. The goal is now to find rules that partition M along decision boundaries so that
the classes of M match the classes of the domain. Hence, the rule for classifying an object
o is
ω̂ (o) := ω i if m (o) ∈ Ri . (1.2)
This means that the estimated class ω̂ (o) of object o is set to the class ω i if the feature
vector m (o) falls inside the region Ri . For this reason, the Ri are also called decision
regions. The concept of a classifier can now be stated more precisely:
Definition 1.3 (classifier). A classifier is a collection of rules that state how to evaluate
feature vectors in order to sort objects into classes. Equivalently, a classifier is a system
of decision boundaries in the feature space.
Readers experienced in machine learning will find these concepts very familiar. In fact,
machine learning and pattern recognition are closely intertwined: pattern recognition
is (mostly) supervised learning, as the classes are known in advance. This topic will be
picked up again later in this chapter.
1.2 Structure of a pattern recognition system
In the previous section it was already mentioned that a pattern recognition system maps
objects onto feature vectors (see Figure 1.1) and that the classification is carried out in the
feature space. This section focuses on the steps involved and defines the terms pattern
and feature.
1.2 Structure of a pattern recognition system  3
Ω
Sensing
⋅⋅⋅
Preprocessing
Segmentation
Patterns
Feature extraction
Blurry boundary Features

Fig. 1.2: Processing
Classification Classes ω i pipeline of a pattern
recognition system.
Figure 1.2 shows the processing pipeline of a pattern recognition system. In the first
steps, the relevant properties of the objects from Ω must be put into a machine readable
interpretation. These first steps (yellow boxes in Figure 1.2) are usually performed by
methods of sensor engineering, signal processing, or metrology, and are not directly part
of the pattern recognition system. The result of these operations is the pattern of the
object under inspection.
Definition 1.4 (Pattern). A pattern is the collection of the observed or measured proper-
ties of a single object.
The most prominent pattern is the image, but patterns can also be (text) documents, audio
recordings, seismograms, or indeed any other signal or data. The pattern of an object is
the input to the actual pattern recognition, which is itself composed of two major steps
(gray boxes in Figure 1.2): previously defined features are extracted from the pattern and
the resulting feature vector is passed to the classifier, which then outputs an equivalence
class according to Equation (1.2).
Definition 1.5 (feature). A feature is an obtainable, characteristic property, which will be

the basis for distinguishing between patterns and therefore also between the underlying
classes.
A feature is any quality or quantity that can be derived from the pattern, for example,
the area of a region in an image, the count of occurrences of a key word within a text, or
the position of a peak in an audio signal.
Another random document with
no related content on Scribd:
"You may guess what it is, if you like," said Forbes, "but it
would spoil all the fun to show it to you beforehand. Ask me
questions, and I'll answer yes or no."
"Well then, is it heavy?"
"Light?"
"Rather, yes."
"Can I hold it in my hand?"
"Yes."
"Is it long and straight?"
"It's quite straight—particularly so—and rather long."
"I say Forbes—it isn't—no it can't be a walking stick, of

course," said Jack growing excited, and fervently hoping it
was.
"No, but that's not a fair question, you must find out more
about it."
"Is it a useful thing?"
"Decidedly."
"Not too useful I hope," said Jack, somewhat dejectedly.
"I don't know what you mean by too useful—nothing can be

too useful."
"I mean it has nothing to do with lessons—a blotting book,

or slate, or anything of that sort."
"Well yes, it is something of that sort, but I know you
haven't got one, and really want one, I heard you ask
Geoffrey for his only the other day."
"It's a ruler!" said Jack blankly.
"Yes—but not only a ruler. Here, I'll let you feel it, old boy."
Jack felt it.
"It's one of those rulers with a pencil in it," he murmured,

then he added effusively lest Forbes should think him
ungrateful, "thanks awfully, it's jolly, it's awfully kind of
you."
Forbes felt and saw his little brother was disappointed.
"I quite forgot you didn't care for useful things, I like them
myself. But," he added, anxious to raise Jack's spirits, and
to make the best of his present which he felt was a failure,
and unappreciated, "this is a particularly nice ruler—it has a
first-rate pencil in it, and a view of the Grammar School and
Arboretum outside. That's why I got it, I thought you'd be
sure to like it."
"Forbes," said Mr. Hodson, laying a kind hand on the boy's
shoulder,
"you remind me of a verse in Proverbs, 'He that ruleth
his spirit is greater than he that taketh a city.'"
"Thanks awfully," was all that Jack could think of to say,
then after a moment's pause he asked, "What are you going
to give Geoff?"
"I'm giving him a walking stick, a regular wopper. I got it

while you and he were looking after the doll. It has a knob
at the end the size of my fist. I've asked them to send it out
to Hazelbury for me, as I was afraid Geoff would see me
carrying it."
"I say Forbes," said Jack colouring, and in a low voice, "you
wouldn't, I suppose, give Geoff the ruler and let me have
the stick?"
"No, certainly not," said Forbes angrily. "You are an

ungrateful sneaking little scamp, get away with you."
Jack burst into tears at this, and ran past Mr. Hodson and
Geoff, who had overheard Forbes' words, as he had raised
his voice in his anger.
Mr. Hodson turned round and looked at Forbes. The light of

a lamp close by shewed him the indignant light in the boy's
eyes.
"He's gone to complain to nurse now I suppose," he said,

angrily looking after Jack's little figure, as he ran crying up
the drive and into the house.
"Forbes," said Mr. Hodson, laying a kind hand on the boy's

shoulder, "you remind me of a verse in Proverbs, 'He that
ruleth his spirit is greater than he that taketh a city.' You
have the chance of being a greater man then even
Alexander—for though he conquered the world, he could not
conquer his own temper, and killed his best friend in a fit of
anger."
"Thank you Sir," said Forbes, "I'll remember."
CHAPTER IV.
TAKING A CITY.
Geoffrey remembered Mr. Hodson's words later in the

evening.
He did not turn in at the garden gate with Forbes, but

telling him he had some business to do before going in to
tea, he gave his presents into his brother's keeping, and ran
down the hill on the summit of which their house stood.
At the bottom of the hill, he came upon a tumbled-down

cottage, standing quite by itself. Old Rachel, of whom Mrs.
Green had told them, lived here.
The thought of Rachel had lain very heavily on Geoffrey's

heart the last two or three days. He could not forget that
she was a mother, and a mother neglected by her only
child, who, when she gave her anything at all, only passed
on to her what she couldn't eat herself. He was thinking of
Rachel when the apple puffs were passed round at dinner.
Now apple puffs was a favourite dish of Geoffrey's, as I
fancy it is of most boys. They looked particularly tempting
to-day, and he ate the first with a relish. It was just as he
was taking his second, that Mrs. Green's words came across
his mind, "and if ever she gives her anything, you may be
quite sure it ain't fit to eat, something they can't eat
themselves because it's turned."
Geoffrey looked at the puff as it lay so invitingly on his

plate. It was three cornered, and a little burnt at the edges,
which made it all the nicer in Geoffrey's opinion, and a nice
layer of white sugar lay on the top.
How good it looked! For a moment the boy gazed at it

undecidedly, then, when no one was looking, he put it into
his jacket pocket, and resolved to take it round to old
Rachel when they came back from Ipswich.
"For once," thought Geoff, "she shall have something that

somebody else wants."
He had had some difficulty in knowing how to stow away his

many presents so as not to crush his apple puff, but he had
managed somehow, and now as he stood outside the door
of Rachel's cottage, he took the puff out and was glad to
find it still whole. It certainly looked very tempting, and
Geoffrey was hungry after his walk. No one would see if
after all he ate it, instead of giving it to old Rachel, and no
one would consciously miss it.
For a moment the boy's resolution wavered, then he

knocked at the door.
Now an apple puff was not a very great thing to give up for
the sake of another, and perhaps some of my little readers
may think that it would not have signified very much if
Geoffrey had eaten it after all. But we must remember, that
life is made up of little things, and the great battle of life,
on which so much depends, consists often of little victories
and little losses, and this small victory that Geoffrey gained
that afternoon helped him in after years to gain a far
greater one.
When he grew up to be a man, there was something he

wanted very much, which was far more worth having than
this apple puff. He wanted it so much that he sometimes
felt he would almost give his life to possess it for ever such
a short time; but somebody else wanted it too, someone
who was weaker than he was, and who perhaps needed it
more than he did, and Geoffrey gave it right up for the sake
of that other.
I do not think he would have acted so nobly when he was a

man, if he had not begun quite early in life to deny himself.
If he had lost this little battle and had eaten the apple puff
outside old Rachel's door, in all probability he would have
lost that greater battle in after life.
"You are old Rachel, aren't you?" asked Geoffrey.
"Well, what if I be?" she answered.
"Come in," said a quavering voice as Geoffrey knocked, and
on entering, he saw a haggard looking old woman, with a
forbidding expression of face, and grey straggling hair,
crouching over a small fire.
"You are old Rachel, aren't you?" asked Geoffrey, who had
never seen her before.
"Well, what if I be?" she answered in a low gruff voice, "I

don't want no one to come interfering with me, leastways a
child. What do you want—eh?"
"I've brought you an apple puff," said Geoffrey, standing

still by the door.
"Shut the door, can't ye," said Rachel shivering, "the

draught's enough to cut one in two. An apple puff is it? That
ain't the kind of food I want, I ought to be fed on arrowroot
I tell ye, and sweet puddings and the like. But Jane never
sends me what I need, it's either somethin' that's turned
bad, or else what I can't eat."
"This is quite new and fresh," said Geoffrey, shutting the

door and coming a little nearer, while he laid the puff on the
table, "perhaps you've never tasted a puff—it's awfully good
—I wish you'd try it."
"That's a likely story, if Jane sent it," said Rachel glancing at

it, and then looking up suspiciously at Geoffrey.
"No one sent it," interposed Geoffrey. "We had them for
dinner to-day, and I thought you'd like one as they were so
good. I'm Geoffrey Fortescue, and I heard of you from Mrs.
Green."
Rachel looked back again at the fire, muttering to herself,

and Geoffrey looked round the room, and thought how bare
it was, and how lonely Rachel looked.
"Haven't you any money to get things with?" he asked.
"Money ain't for such as me: the big folk that don't need it,
they have the money. This world's comforts ain't for me."
"There's Heaven for you," said Geoffrey.
Rachel darted a quick look at the boy, and as she saw the
earnest young face looking at her so pitifully, the expression
on her own face softened, and she shook her head.
"I take it Heaven is a long way off," she said sadly.
"It doesn't seem so very far," answered Geoff, "Mother is

there, and I sometimes feel she's quite close."
"Heaven ain't meant for such as me," muttered Rachel,

cowering closer to the fire.
"I thought God loved everybody, and meant Heaven for the
whole world," said Geoff, "and," he added earnestly, "I'm
quite sure God must want you there, because you are so
lonely."
Rachel wiped away a tear or two with her apron. She had
not cried for many a long day. She had harboured too bitter
thoughts to allow of tears, but to-day, something in the
boy's simple words touched her hard old heart.
"I mustn't stop," said Geoffrey, looking out of the window at

the darkness, "or Nurse won't like it. But I'll ask Mr. Hodson
to come and see you, and I'll leave the apple puff, for it's
ever so good, if you'll only try it."
Rachel nodded her assent to the last sentence, but added:

"But don't you bring no parsons to see me. I don't want no
parsons here, unless," she added with a sob, and beginning
to rock herself backwards and forwards, "unless he can tell
me the way straight and plain to Heaven. I'd like to know
that."
Closing the door softly after him, Geoffrey ran as fast as he

could to Mr. Hodson. Although he ran the risk of a scolding
from Nurse for being late, he felt that Rachel must not be
left in her misery.
He arrived at the house nearly breathless, and told his

friend what had happened.
Mr. Hodson, who had together with the Vicar for many a
year tried in vain to overcome Rachel's objection to see a
clergyman, was glad enough of the news Geoffrey brought
him, and prepared at once to go and see her.
"Mr. Hodson," said Geoffrey anxiously, "God loves her,

doesn't He? And He won't turn her away from Heaven, if
she asks to be let in."
"If Rachel really wants to find God, He certainly will not turn
her away," answered Mr. Hodson. "The Lord Jesus Christ has
made a way there for us all, and old Rachel's way is the
same as yours and mine. Do you remember the story
Geoff," he added, as he put On his coat to start off at once,
"of the man who saved his children by making a bridge of
his own body from the window of his burning house to that
of the opposite one? The houses were very near together,
and he could reach from one window to another."
"His children one by one crossed over his body into safety,
and just as the last child was saved, the house fell in, and
the man was killed. When the Blessed Lord Jesus died on
the Cross, He made a bridge for Rachel, and for you and for
me to Heaven. You see, I have good news for your old
friend, Geoff my boy, so you run home as fast as you can or
you'll get a scolding."
And Geoff did get a scolding. Nurse met him at the door.
"Master Geoff," she cried, "I'm downright ashamed of you

for setting the children such an example. There I've been
worrying after you for the last twenty minutes, and thought
you'd come to some accident or other. I'm downright
ashamed of you."
"I had some business to do," said Geoffrey, trying to pass

her. But Nurse placed her portly figure in his way.
"Business! A chit like you talking of business! What'll you

come to next, I wonder. You're a naughty boy and ought to
be ashamed of yourself. What business have you had to do,
I should like to know, except to be a good obedient boy.
That's the business you ought to be doing I take it."
Geoffrey flushed angrily. His mother never scolded him in

this way, and he had often run messages for her as late as
this by himself.
"Let me pass, Nurse," he said angrily, trying to push past

her, "I've not been doing anything wrong."
"Nothing wrong!" exclaimed Nurse, catching hold of his

arm. "Nothing wrong to make me that anxious about you
that I didn't know what to do—nothing wrong that you've
kept us all waiting for tea, and have set a bad example to
all the children. I'm ashamed of you Master Geoff. Now I
should like to know what you've been about, and I mean to
know too."
But at the sight of the bread and water, he lost his temper
completely,
and taking up the glass, he threw it on to the ground.
Now Geoff did not wish to tell Nurse about the apple puff,
and felt exceedingly angry at being treated like a little boy,
and held by the arm in this way. So raising his hand he
struck nurse's arm as hard as he could, and then pale with
anger, he rushed into his bedroom and locked the door.
"Well I never!" ejaculated nurse. "If that isn't a wicked

temper, I don't know what is."
When she went back into the nursery a few minutes

afterwards, she informed the children that Geoff had been a
naughty boy, and was to have no tea that evening, but that
Forbes might put a glass of water and some bread outside
his door, but was not to speak a word to him.
Forbes, in utter astonishment at his elder brother being

punished in this way, obeyed wonderingly.
Now it was a great pity that Geoffrey had not at once

explained to nurse the cause of his absence. She might
have given him a slight scolding, for not asking her leave
before going, but her kind heart would have sympathized
with him, in his wish to do a kindness.
But Geoffrey's pride had stood in the way. He could not

endure being treated like a little boy, and scolded like a
naughty child, and as he paced up and down his room, his
indignation rose, and reached its climax when he heard
Forbes' footstep outside, and the sound of him quietly
laying down his tea, as he supposed, by his door without a
word. Was he to be treated then like a mere baby? And to
be held in disgrace like Jack or Dodie would have been, if
they had been naughty?
He opened his door impatiently to call after Forbes to take

away his tea, but at the sight of the bread and water, he
lost his temper completely, and taking up the glass, he
threw it on to the ground, smashing it to pieces. Then
locked his door again, and would not open it, though nurse
shook it violently.
Then it was that Mr. Hodson's words about Alexander came

into his mind, and Geoffrey stood quite still in his walk.
"'He that ruleth his spirit is greater than he that taketh a

city,'" thought Geoff.
Half an hour afterwards, to Forbes' intense astonishment,

Geoffrey appeared in the nursery and apologized to nurse.
Nurse said nothing, but going to the cupboard, she mixed

some gregory powder in a wineglass, saying, "anyone who
shows temper like that, I take it, must be ill. There my
dear," she added kindly, "you drink that, there's a good boy
—and you'll feel better to-morrow."
And Geoff drank it to the dregs, and in so doing was greater

than Alexander the Great.
CHAPTER V.
"POOR LITTLE LAD."
Geoffrey could not fail to acknowledge to himself that after

all, Nurse had been wise in putting Dodie into a frock with
high neck and long sleeves, for the winter was unusually
severe.
Snow lay for several days some inches thick in the garden,
and though the boys enjoyed the snowballing well enough,
and were able to keep themselves thoroughly warm, little
Dodie seemed to feel the cold very acutely, and often came
in from her daily walk crying from the pain of freezing
fingers.
In fact, the child did not seem herself, and Nurse began to
grow uneasy about her, particularly as in seven days' time,
Major Fortescue was expected and she was naturally
anxious that all the children should be looking their best on
his arrival.
In Geoffrey's eyes, Dodie seemed to be growing thinner and

smaller altogether, and a terrible fear seized him lest she
was going to be ill, and would be so when his Father
arrived.
As the days past, he gave up snowballing, and spent his
time in the nursery with Dodie, who was not allowed out.
"I think the cold has struck her," said Nurse, as one day she
altogether refused to eat her dinner. "I've a mind to send
for Dr. Booth, the powders I've been giving her don't seem
to be what she wants."
Geoffrey laid down his knife and fork, feeling a sudden

disinclination for the mutton and dumplings before him.
"Do you think Dodie is going to be ill?" he asked anxiously.
"I hope not, my dear, but it ain't like her to turn away from
her food, and she has a nasty little cough that don't get
better. Anyways I'll ask Dr. Booth to look in, there can't be
no harm in that. There, there my darling," she added,
taking Dodie on to her knee, "don't cry, there's a pet."
Nurse looked down at Dodie's little face which was lying on

her arm.
"I don't like the look of her," she murmured more to herself
than to anyone else, "her eyes are too bright to be natural,
and she's restless, poor little dear." Then louder she added,
"Geoff, you might run down when you've finished your
dinner and ask the doctor to be so good as to look in. You'd
catch him before he starts out on his rounds if you're
quick."
"Dr. Booth," he said,—looking up into the Doctor's face—
"will Dodie be well by the time Father comes home?"
Geoffrey, who had listened with a beating heart to all Nurse

had said, sprang up at once, and not heeding Nurse's
injunction to finish his dinner first, ran off at once for the
Doctor, and returned again in an incredibly short time.
To his excited imagination, the few minutes that elapsed

between leaving the message at the doctor's door and his
arrival seemed hours, and then at last his ring was heard,
and a minute after, he stood looking at Dodie, who still lay
in nurse's arms.
Geoff did not move his eyes from his face, till Nurse
suddenly looking up and becoming conscious of the three
little listeners who stood around, ordered them all
peremptorily out of the room. Geoffrey, however, waylaid
the Doctor as he left.
"Dr. Booth," he said, standing with his hands thrust deeply

in his pockets, and looking earnestly up into the Doctor's
face as he put on his coat in the hall, "will Dodie be well by
the time Father comes home?"
The Doctor shook his head somewhat ominously.
"That I can't tell you, my boy," he answered, as he buttoned

up his coat and smoothed his collar, "with care I hope your
little sister will get well before very long—but it will require
care—and I can't say exactly when she will be herself
again."
"Is she going to be very ill?" asked Geoff.
The doctor turned away somewhat hurriedly from the

anxious face looking up into his, and fidgeted a little
nervously with his hat before putting it on. Then clearing his
throat, he looked round again and patted the boy on the
head saying, kindly:
"Care and physic do wonderful things, my boy—for all I
know, your little sister will be having a game of snowballing
with you this day week."
"I do hope she'll be well by the time Father comes," said

Geoff with a sigh.
"Oh well—who knows!" said Dr. Booth jovially—and

ramming his hat on his head, he nodded to Geoff, and in a
minute more was driving away from the house, but not
away from the remembrance of those anxious eyes that had
been raised so beseechingly to him,—which remembrance
made him shake his head, murmuring "poor little lad."
When Geoff went up to the nursery, he saw Nurse had been

crying, but when he asked what the Doctor really thought of
Dodie, she told him she had no time to talk and that he had
better go down to the other boys in the schoolroom as
Dodie had to go straight to bed, and mustn't be disturbed
by any noise.
Geoffrey did as he was bid, but with a heavy heart, feeling

quite sure that Nurse's tears meant that Dodie was very ill.
He could not play with Forbes and Jack, or even read, but
sat by the fire, looking silently at the red coals, for an hour
or more.
It was the greatest relief when Nurse at last came down and
told him he might go upstairs and watch by Dodie's crib
while she had her tea, and that was the beginning of a
continual watching on the boy's part. Nurse finding how
gentle and tender he was, and how noiselessly he could
move about when he liked, did not object to his spending
many hours by Dodie's crib, and indeed, in her great
anxiety, she began to be thankful for the boy's presence.
For the Doctor's report of Dodie had been serious. The child

Full Chapter Pattern Recognition Introduction Features Classifiers and Principles de Gruyter Textbook 2Nd Edition Beyerer PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Full Chapter Pattern Recognition Introduction Features Classifiers and Principles de Gruyter Textbook 2Nd Edition Beyerer PDF

Uploaded by

Copyright:

Available Formats

Pattern Recognition Introduction

Features Classifiers and Principles De

Organic Chemistry 25 Must Know Classes of Organic

Carbon for Micro and Nano Devices De Gruyter Textbook

Thermal Engineering: Engineering Thermodynamics and

Differential Equations A First Course on Ode and a

Relaxation in Optimization Theory and Variational

Water Chemistry Analysis and Treatment Pollutants

Water Resources Management Innovative and Green

Introduction to Distributed Computer Systems Principles

Medical Image Reconstruction

Introduction, Features, Classifiers and Principles

Library of Congress Control Number: 2024931594

Bibliographic information published by the Deutsche Nationalbibliothek

© 2024 Walter de Gruyter GmbH, Berlin/Boston

Karlsruhe, Summer 2017

Karlsruhe, Spring 2024

Preface of 2nd edition  VII

List of Tables  XIII

1 Fundamentals and definitions  1

3 Bayesian decision theory  103

4 Parameter estimation  132

4.5 Parameter estimation for Gaussian mixtures  150

5 Parameter free methods  161

6 General considerations  180

7 Special classifiers  191

7.9.2 Hidden Markov models  236

8 Classification with nominal features  257

9 Classifier-independent concepts  273

A Solutions to the exercises  293

B A primer on Lie theory  307

C Random processes  311

Tab. 2.1 Taxonomy of scales of measurement  11

Tab. 9.1 Common binary classification performance measures  281

Fig. 1.1 Transformation of the domain into the feature space  2

Fig. 2.1 Iris flower dataset  14

Fig. 2.38 Kernelized PCA with radial kernel function  79

Fig. 4.1 Comparison of estimators  139

Fig. 5.1 The triangle of inference  161

Fig. 5.12 Decision regions of a 5-nearest neighbor classifier  177

Fig. 6.1 Increasing dimension vs. overlapping densities  182

Fig. 8.1 Decision tree to classify fruit  259

Fig. 8.3 Qualitative comparison of impurity measures  261

a, . . . , z Scalar, function mapping to a scalar, or a realization of a random variable

p(m) Probability density function for random variable m evaluated at m

ℂ Set of complex numbers

|A| Determinant of matrix A

N(μ, σ 2 ) Normal/Gaussian distribution with expectation μ and variance σ 2

cf. “confer” (latin: compare)

(a) Screws (b) Banknotes (c) Handwriting

(d) Plant seeds (e) Plant leaves

Fig. 1: Examples of artificial and natural objects.

(a) Schematic overview (b) Inspection and ejection stage

Fig. 2: Industrial bulk material sorting system.

provide an awareness of any possible complications it may cause in the subsequent

Tab. 1: Capabilities of humans and machines in relation to pattern recognition.

Association & cognition Combinatorics & precision

1.1 Goals of pattern recognition

The purpose of pattern recognition is to assign classes to objects according to some

Two elements o1 , o2 with o1 ∼ o2 are said to be equivalent. We further write [o]∼ ⊆ Ω to

Definition 1.2 (Partition, class). Let Ω be a set and ω1 , ω2 , ω3 , . . . ⊆ Ω be a system of

Fig. 1.1: Transformation of the domain Ω into the feature space M.

1.2 Structure of a pattern recognition system