10/02/2020

Introduction to
Machine Learning
B. Solaiman
Basel.solaiman@imt-atlantique.fr
Départ. Image &Traitement de l’Information
Brest, France

1. Introduction, Pattern Recognition Concepts

2. Machine Learning

3. Learning Paradigm

4. Application examples

Introduction to Machine Learning----------- B. Solaiman

1
 10/02/2020

1. Introduction:
Pattern Recognition Concepts

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 4

• Pattern Recognition :
Scientific discipline targeting to conceive systems/approaches/methods allowing to
make reasonable decisions assigning “observed patterns” from the real world to
some “categories/classes” of patterns
Assign an unknown pattern to one of several known categories (or classes)

Introduction to Machine Learning----------- B. Solaiman

2
 10/02/2020

1. Introduction : Pattern Recognition Concepts 5

• Pattern Recognition :
Scientific discipline targeting to conceive systems/approaches/methods allowing to
make reasonable decisions assigning “observed patterns” from the real world to
some “categories/classes” of patterns

Remote sensing classification

Scene classification
Medical diagnosis
Handwritten digit/letter recognition
Biometrics
Speech recognition
Defect detection in chip manufacturing
Interpreting DNA sequences
Credit Fraud Detection,
Credit Applications.
……

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 6

• Pattern Recognition :
Scientific discipline targeting to conceive systems/approaches/methods allowing to
make reasonable decisions concerning the belonging of “observed patterns” from
the real world to some “categories/classes” of patterns

What is a pattern?

What kinds of category we have?

Introduction to Machine Learning----------- B. Solaiman

3
 10/02/2020

1. Introduction : Pattern Recognition Concepts 7

Patterns Semantic Level

U : Univers of patterns
(Real World)

u1
u2 ….. uK
uk

Classes/Categories Semantic Level A pattern could be a physical object or an event

C : Univers of Classes/Categories
(Real World)
Class / Category : Set of patterns
C2 sharing some common characteristic
C1 …..
CM Subjective definition
Cm
Application related
Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 8

CLASS : A COLLECTION OF “SIMILAR” OBJECTS

Female class Male class

Introduction to Machine Learning----------- B. Solaiman

4
 10/02/2020

1. Introduction : Pattern Recognition Concepts 9

Family Simpson Employees Females Males

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 10

Patterns Semantic Level Patterns Semantic Level

U : Univers of patterns W : Univers of Features
(Real World) (Machine World)
u1 Observations – Features X1
….. uK Extraction X2 ….. XK
u2
(Patterns Knowledge
uk Representation) Xk

Classes/Categories Semantic Level

Pattern Recognition
Classes/Categories Semantic Level
algorithm
C : Univers of Classes/categories C : Univers of categories models
(Real World) (Machine World)

C2 Categories - Classes Mod2

C1 ….. Models Mod1 …..
CM (Classes Knowledge
Cm Representation) ModM
Modm

Introduction to Machine Learning----------- B. Solaiman

5
 10/02/2020

1. Introduction : Pattern Recognition Concepts 11

Problem : Sorting incoming fish on a conveyor belt.

Assumption : Two kind of fish (i.e., Two classes):sea bass, Salmon

Features

Length Average Brightness

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 12

Patterns Semantic Level

U : Univers of patterns
(Real World)

u1
u2 ….. uK
uk

Classes/Categories Semantic Level

C : Univers of Classes/categories
(Real World)

C2
C1 …..
CM
Cm

Introduction to Machine Learning----------- B. Solaiman

6
 10/02/2020

1. Introduction : Pattern Recognition Concepts 13

Pattern Recognition Cchallenges / Difficulties

Classes definition challenge
– Intra-Class Variability Huge amount of forms within the very same class

– Inter-Class Variability Similar forms from different classes

Letters/Numbers that look similar

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 14

Pattern Recognition Challenges / Difficulties

Real World object
Sensor(s) Features Machine World object :
Extraction Feature Vector
X  ( x1 , x2 , , xN )

Sensing challenge
Availability / Accecebility of the input data that can be measured/observed from
objects to be recognized !
Features extraction challenge
Intraset features : features encoding objects which are common features to all
patterns belonging to each given class
Interest features : features allowing to discriminate between pattern classes
(discrimination capacity, dimensionality reduction, …)
Introduction to Machine Learning----------- B. Solaiman

7
 10/02/2020

1. Introduction : Pattern Recognition Concepts 15

Pattern Recognition Challenges / Difficulties

Real World object
Sensor(s) Features Machine World object :
Extraction Feature Vector
X  ( x1 , x2 , , xN )

Discriminative Features

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 16

Class / Category Knowledge representation

Real World
Category
Class Knowledge
Class Model (Mod)
Representation Model

Prior as well as Sensor related knowledge Exploiatation / Projection into models

to be used by the recognition algorithm
Optimal Pattern Recognition Algorithm

Classes’ models
Machine World object :
Feature Vector Recognition
DECISION
X  ( x1 , x2 , , xN ) Algorithm

Introduction to Machine Learning----------- B. Solaiman

8
 10/02/2020

1. Introduction : Pattern Recognition Concepts 17

Knowledge representation of Classes’ Models

U
X1 W
u1 Observations –
u2 ….. uK Features Extraction X2 ….. XK
(Patterns Knowledge
uk Representation) Xk

CRUCIAL
STEP

C2 Mod2
C1 ….. Categories - Classes
Models Mod1….. ModM
CM (Classes Knowledge
Cm Modm
Representation)

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 18

Statistical, or probabilistic, pattern recognition

Knowledge is represented in terms of probability density functions

Gender Classification male

female

Introduction to Machine Learning----------- B. Solaiman

9
 10/02/2020

1. Introduction : Pattern Recognition Concepts 19

Syntactic, or structural, matching

- Objects are described as a set of units called primitives;
- A primitive is an elementary component of an object;
- An object is recognized by the sequence, called grammar, in which the primitives appear
in the object description

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 20

Syntactic, or structural, matching

- A medical sample of syntactic pattern recognition is karyotype where similar
chromosomes are intended to be grouped;
- The set of primitives describing a contour may be the following set:
{convexity(a), straight part(b), deep concavity(c), shallow concavity(d)}

Introduction to Machine Learning----------- B. Solaiman

10
 10/02/2020

1. Introduction : Pattern Recognition Concepts 21

Clustering, or similarity-based, recognition

Use of reference patterns, called centroids,
to represent class’ patterns

Case-based reasoning
(in artificial intelligence)

Introduction to Machine Learning----------- B. Solaiman

1. Introduction : Pattern Recognition Concepts 22

Expert’ Rule-based recognition

Bénigne Maligne

Ronde Ovale Lobulée Nodulaire Stellaire

If (Contour .is. Smooth) Then (Mass. is. Benin)

Introduction to Machine Learning----------- B. Solaiman

11
 10/02/2020

23

2. Machine Learning

Introduction to Machine Learning----------- B. Solaiman

2. Machine Learning 24

Pattern Recognition Machine Learning

Machine learning concerns the construction and study of systems that

can learn from data

Introduction to Machine Learning----------- B. Solaiman

12
 10/02/2020

2. Machine Learning 25

In machine learning:
- instead of injecting a massive knowledge to solve the problem (Recognition,
Estimation, Prediction…), a set of samples as well as a generic model are
injected to the “machine”
- learning refers to making the model tuned
- the resulting tuned / learned model is expected to solve the problem extremely
accurately

B Learned
MACHINE B1 Model
Generic
Test Samples
Model

Introduction to Machine Learning----------- B. Solaiman

2. Machine Learning 26

WHY machine learning ?

B
 Some tasks can only be specified by MACHINE
training data/examples Generic
 Human expertise may be very costly
 Amount of knowledge might be too large Model
for explicit encoding by humans
 Computational power is ever increasing

For doing WHAT ?

Pattern recognition Learned
Diagnosis (Fault, Medical, …) B1 Model
Prediction (Risk, ….) Test Samples
Planning
Data mining…..

Introduction to Machine Learning----------- B. Solaiman

13
 10/02/2020

2. Machine Learning 27

Training sample
dataset

Machine Learning Pattern Recognition

Prediction Expert Knowledge - based

Machine Learning approaches Pattern recognition approaches

Introduction to Machine Learning----------- B. Solaiman

2. Machine Learning 28

input Machine Learning Learned

data approach Model

The learned model is used then to:

Predict the outcome of a system
Recognize complicated patterns in the new data points
Classify input examples

Introduction to Machine Learning----------- B. Solaiman

14
 10/02/2020

2. Machine Learning 29

Validation Data
Training Data V
B ML Learned Model
Model
Generic Model Strategy Tuning 𝑓 ( . /B )
f(.)
Output Model
(program)
Expected Output
Model

Introduction to Machine Learning----------- B. Solaiman

30

3. Learning paradigm

Introduction to Machine Learning----------- B. Solaiman

15
 10/02/2020

3. Learning paradigm 31

Learning A general term denoting the way in which people and computers:
- Acquire and organize knowledge and skills (by building, modifying and organizing
internal representations of some external reality);  TRAINING
- Discover new knowledge (by creating hypotheses that explain some data or
phenomena);  MINING

Knowledge extraction from data to help humans take decisions?

Decisions automation from data?
Dynamic systems adaptation to enable better user experiences?

Learning is the procedure of estimating

Learning from Data
the model parameters so that the learned
model (i.e., algorithm) can perform a
B Training Data
specific task

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 32

Learning approaches categorization

Objective-based learning categorization

Timing-based learning categorization

Training dataset Nature learning categorization

Introduction to Machine Learning----------- B. Solaiman

16
 10/02/2020

3. Learning paradigm 33

Objective-based learning categorization

Learning Data Representation

Learning new representations

B of data that makes it easier to
extract useful information Density estimation
Quantization
dimensionality reduction
Task Learning

Learning desired models to fulfil

B Classification, Prediction, Recognition
and Regression tasks

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 34

Timing-based learning categorization

Batch Learning

Learning models using

B the entire training set

Online Learning

Model
B
New Model
Example updating

Introduction to Machine Learning----------- B. Solaiman

17
 10/02/2020

3. Learning paradigm 35

Training dataset Nature learning categorization

Individual / Situation /
Object / Product / ……

Informative Channels
(Explanatory Variables)

Data x1 x2 x3 x4 …….. xN y
N: Data dimension
Observations, measures, features..
(Qualitative, Quantitative, ….)
Target Variable
Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 36

Training dataset Nature learning categorization

B = {(Xn,Yn) n = 1,..,N}
(Xn,Yn) i.i.d. N observations of the random pair of variables (X, Y)

X RK : Feature vector

Y : Target variable, Y = f(X)
f : Unknown function associating X and Y
Inductive Learning :
The process of learning the association function Y = f(X)
(learning how to predict Y when a new and unseen X is given)

Introduction to Machine Learning----------- B. Solaiman

18
 10/02/2020

3. Learning paradigm 37

Training dataset Nature learning categorization

Real World Stored Data

Distribution
Training Data
D Real World B
(X,Y)
Data Sampling
V
Validation Data
Target variable
Example (Case/
Instances / Object / Pattern..)
Explanatory Variable
Feature Vector

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 38

Training Data
D Real World B
(X,Y)
Data Sampling
V
Validation Data
Y
No Target Variable Y  Unlabeled Training Data

Y  L= {C1, C2,…CM}  Labeled Training Data

(L: set of discrete labels)

YR  Labeled Training Data

Introduction to Machine Learning----------- B. Solaiman

19
 10/02/2020

3. Learning paradigm 39

Training dataset Nature learning categorization

Training Data
D Real World B
(X,Y)
Data Sampling
V
Validation Data
Binary Classification Y  L= {C1, C2}

Within Artificial Intelligence community :

 f ; the unknown function associating X and Y : Called Concept

 Learning f : Called Concept learning

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 40

Training dataset Nature learning categorization

D
(X,Y) Sampling
B Learning f
Example Target
variable
Learning paradigm
Given:
a set B of N examples (Xn,Yn) n = 1,2…N sampled from some a real world
distribution D, where XnRK and Yn L={C1, C2,…CM} (Classification: set of
discrete labels), or, Yn L=R (Regression)
Find:
a function f : D  L which associates ‘well’ any example X sampled from D
with its real target variable Y

Introduction to Machine Learning----------- B. Solaiman

20
 10/02/2020

3. Learning paradigm 41

Three major learning paradigms:

Supervised Learning:
Learning with a series of labelled examples (i.e. target variables Yn, n = 1,2…N are
used for learning f) with direct feedback

• Unsupervised Learning : (discovering patterns and structures in unlabeled data )

uses a series of unlabelled examples (i.e. target variables Yn, n = 1,2…N are not used
for learning f) with NO feedback

• Semi-supervised Learning : Falls between supervised and unsupervised learning.

The training dataset has a small number of labelled examples and a large number of
unlabeled examples. The goal is similar to supervised learning

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 42

• Supervised Learning
Set of Labeled
Training Data

Training Algorithm / Model /

Base Learner / Classifier / …
Inducer

Target feature : Categorical (Label /Class) Classification

: Numerical (estimation…) Regression
Instance / Example….
Pre-classified Patterns

Introduction to Machine Learning----------- B. Solaiman

21
 10/02/2020

3. Learning paradigm • Supervised Learning 43

Sensor 2

x
x
x o
x x
x
Target feature : o x
Instance o x
Categorical (Label /Class) o o
o
f( ) = “Apple” o

f( ) = “Tomato” Sensor 1
x Class 1
f( ) = “Cow” o Class 2

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm • Supervised Learning 44

Goal : Predict the value of the target feature on unseen

instances

Learned
Model

Testing pattern Predicted label

(unknown class)

Introduction to Machine Learning----------- B. Solaiman

22
 10/02/2020

3. Learning paradigm • Supervised Learning 45

Learning Step
Training
Labeled base od training
Labels
instances/patterns/samples/
examples…

Image Learned
Training
Features model

Testing/Generalization Step

Image Learned
Prediction
Features model
Test Image

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm • Supervised Learning 46

Strong Learner :
A learning algorithm that is given a labelled training set and produces a
classifier which can be close to perfect performance (close to 100%)

Weak Learner :
A learning algorithm that is given a labelled training set and produces a
classifier which is only slightly better than random classification guess
(close to 50%°

Introduction to Machine Learning----------- B. Solaiman

23
 10/02/2020

3. Learning paradigm 47

• Unsupervised Learning
Sensor 2

Set of Unlabeled
x x
Training Data x x x
x
x x
x
x
x x x
x x
x x
x
x
x
Instance / Example….

Sensor 1

• Explore the data to find some intrinsic structures in them and discover
patterns and structures in unlabeled data

Introduction to Machine Learning----------- B. Solaiman

3. Learning paradigm 48

• Semi-supervised Learning
• Labeled data is generally hard to obtain – Unlabeled data is generally cheap

Set of Semi-Supervised Learning :

Labeled Learning from a small labeled set and a
Training
Data large unlabeled set

Training Algorithm Model /

/ Base Learner / Classifier / …
Set of Inducer
unlabeled
Training
Data

Introduction to Machine Learning----------- B. Solaiman

24
 10/02/2020

49

4. Application Examples

Introduction to Machine Learning----------- B. Solaiman

50
Intelligent Transportation
The Integrated Safety
Program: PreVent

Environment Sensing /
Sensor Fusion

Introduction to Machine Learning----------- B. Solaiman

25
 10/02/2020

Intelligent Transportation 51

Introduction to Machine Learning----------- B. Solaiman

52
Intelligent Transportation

Estimated trajectory

doppler

Microwave sensor Optical sensor 2D

Complementarity

Introduction to Machine Learning----------- B. Solaiman

26
 10/02/2020

Intelligent Transportation 53

ASSISTED DRIVING

Introduction to Machine Learning----------- B. Solaiman

K – Nearest Neigbor Classifier 54

Basic Concept Standard Supervised Learning

Set of - A reasonable
Labeled Training Algorithm / Model /
distance measure
Base Learner /
Training d(X,Y) Classifier / …
Data Inducer
- K

Classifier :
 For a given instance X,
 Get the top K dataset instances that are “nearest” to X
 Inspect the category of these K instances
 Choose the category C that represent the most instances
 Conclude that X belongs to category C

Introduction to Machine Learning----------- B. Solaiman

27
 10/02/2020

K – Nearest Neigbor Classifier 55

For a test instance,

1) Calculate distances from training pts.
2) Find K-nearest neighbors (say, K = 3)
3) Assign class label based on majority

Introduction to Machine Learning----------- B. Solaiman

K – Nearest Neigbor Classifier 56

1 – Nearest Neigbor Classifier

Voronoi diagram (Decision frontiers)

Introduction to Machine Learning----------- B. Solaiman

28
 10/02/2020

K – Nearest Neigbor Classifier 57

Selecting K

Increase K:
Makes KNN less sensitive to noise

Decrease K:
Allows capturing finer structure of space

Pick K not too large, but not too small

(depends on data)

Introduction to Machine Learning----------- B. Solaiman

K – Nearest Neigbor Classifier 58

Time complexity
 Suppose there are M instances and N features in the dataset
 Nearest neighbor algorithm requires computing M distances
 Each distance computation involves scanning through each
feature value
 Running time complexity is, thus, proportional to M.N

Introduction to Machine Learning----------- B. Solaiman

29
 10/02/2020

K – Nearest Neigbor Classifier 59

Image Clusters on intensity Clusters on color

Introduction to Machine Learning----------- B. Solaiman

K – Nearest Neigbor Classifier 60

How good is it?

AnyHow other modifications?
to determine distances between values of categorical
How
attributes?
to determine value of K?
• How
Susceptible
to maketoreal-valued
noisy valuesprediction?
Alternatives:
•1. Slow because
Weighted of distance
attributes calculation
to decide final label
Alternatives:
Alternatives:
2. Alternate
Alternative: approaches:
1.1.Assign
Determine
Boolean distance to missing
distance
K experimentally. values
(1 if same, 0 ifKas
The <max>
different)
that gives minimum
• Average
3.1.K=1 Distances to representative points only neighbours
errorreturnstheclass
values
is selected. label
returned
of nearest
by K-nearest
neighbour
• Partial distance
2. Differential grading (e.g. weather – ‘drizzling’ and ‘rainy’ are
closer than ‘rainy’ and ‘sunny’ )

Introduction to Machine Learning----------- B. Solaiman

30
 10/02/2020

K – Nearest Neigbor Classifier 61

Advantages and Disadvantages of KNN

Needs distance/similarity measure and attributes that “match” target function.

For large training sets,

 Must make a pass through the entire dataset for each classification.
This can be prohibitive for large data sets.

Prediction accuracy can quickly degrade when number of attributes grows.

Introduction to Machine Learning----------- B. Solaiman

31