MACHINE

LEARNING
INTRODUCTION
LEC-1A

1 Hammad Afzal
Department of CSE
hammad.afzal@mcs.edu.pk
COURSE INTRO
 Pre-requisite
 Introductory knowledge of Probability, Statistics and Linear
Algebra

 Course meeting times

 Lectures: 3 session/week

 Course Resources
 Lectures slides, assignments (computer/written), solutions to
problems, research papers, projects, and announcements will be
uploaded on LMS page.
2
WHAT IS MACHINE LEARNING?

Make the machine Evaluate how good the

‘learn’ some thing machine has ‘learned’

3
MACHINE LEARNING

Field of study that gives computers the

ability to learn without being explicitly
programmed.

Arthur Sameul (1959)

4
MACHINE LEARNING

Machine learning is programming computers

to optimize a performance criterion using
example data or past experience.

Tom Mitchell (1998)

5
LEARNING PROBLEMS – EXAMPLES
 Learning = Improving with experience over some
task
 Improve over task T,
 With respect to performance measure P,
 Based on experience E.

 Example
T= Play checkers
 P = % of games won in a tournament
 E = opportunity to play

6
LEARNING PROBLEMS – EXAMPLES
 Handwriting recognition learning
problem
 Task T: recognizing handwritten
words within images
 Performance measure P: percent of
words correctly recognized
 Training experience E: a database of
handwritten words with given
classifications

8
LEARNING PROBLEMS – EXAMPLES
 A robot driving learning problem
 Task T: driving on public four-lane
highways using vision sensors
 Performance measure P: average
distance traveled before an error (as
judged by human overseer)
 Training experience E: a sequence of
images and steering commands
recorded while observing a human
driver

9
MACHINE LEARNING
 Nicolas learns about trucks and dumpers

10
MACHINE LEARNING
 But will he recognize others?

So learning involves ability to

generalize from labeled examples
11
MACHINE LEARNING
 There is no need to “learn” to calculate payroll

 Learning is used in:

 Data mining programs that learn to detect fraudulent credit card
transactions
 Programs that learn to filter spam email
 Programs that learn to play checkers/chess
 Autonomous vehicles that learn to drive on public highways
 Self customizing programs
 And many more…

12
MACHINE LEARNING

13
Applications

CREDIT SCORING
 Differentiating between
low-risk and high-risk
customers from their
income and savings

Discriminant: IF income > θ1 AND savings > θ2

THEN low-risk
ELSE high-risk 14
 Applications

AUTONOMOUS DRIVING
 ALVINN* – Drives 70mph on highways

*Autonomous Land Vehicle In 15
a Neural Network
OCR & HANDWRITING RECOGNITION

17
TEMPLATE MATCHING
 Problem: Recognize letters A to Z
Image is converted into 12x12 bitmap.

18
TEMPLATE MATCHING
Bitmap is represented by 12x12-matrix or by 144-vector
with 0 and 1 coordinates.
0 0 0 0 0 0 1 1 0 0 0 0
0 0 0 0 0 1 1 1 0 0 0 0
0 0 0 0 0 1 0 1 1 0 0 0
0 0 0 0 1 1 0 1 1 0 0 0
0 0 0 0 1 0 0 0 1 0 0 0
0 0 0 1 1 0 0 0 1 1 0 0
0 0 0 1 1 0 0 0 1 1 0 0
0 0 1 1 1 1 1 1 1 1 1 0
0 0 1 1 0 0 0 0 0 1 1 0
0 1 1 0 0 0 0 0 0 1 1 0
0 1 1 0 0 0 0 0 0 0 1 1
1 1 0 0 0 0 0 0 0 0 1 1
 TEMPLATE MATCHING
Training samples – templates with corresponding class:
t1  { (0,0,0,0,1,1,...,0), ' A '}
t 2  { (0,0,0,0,0,1,...,0), ' A '}
.........
t k  { (0,0,1,1,1,1,...,0), ' B'}
..........
Template of the image to be recognized:
T  { (0,0,0,0,1,1,...,0), ' A? '}

Algorithm:
1. Find ti , so that ti  T . 20
2. Assign image to the same class as ti .
FEATURES
 Features are the individual measurable properties of the
signal being observed.
 The set of features used for learning/recognition is called
feature vector.
 The number of used features is the dimensionality of the
feature vector.
 n-dimensional feature vectors can be represented as points
in n-dimensional feature space 22
FEATURES
height

 x1 
x   x
weight  2

Class 1 Class 1
Class 2 Class 2

23
FEATURE EXTRACTION
 Feature extraction aims to create discriminative features
good for learning
 Good Features
 Objects from the same class have similar feature values.
 Objects from different classes have different values.

24

“Good” features “Bad” features

FEATURES
 Use fewer features if possible
 Use features that differentiate classes well

25
 Supervised learning
 Classification

 Regression

 Unsupervised learning
 Reinforcement learning

26
CLASSIFICATION

27
SUPERVISED LEARNING -
CLASSIFICATION
 Objective
 Make Nicolas recognize what is an apple and what is an
orange

28
CLASSIFICATION

Apples Oranges

29
CLASSIFICATION
 You had some training example or
‘training data’
What is this???
 The examples were ‘labeled’

 You used those examples to make

the kid ‘learn’ the difference
between an apple and an orange

30
Its an
apple!!!
CLASSIFICATION

Apple

Pear

Tomato

Cow

Dog

Horse

Given: training images and their categories

What are the categories
of these test images? 31
CLASSIFIER: IDENTIFY THE CLASS OF
GIVEN PATTERN

Distance between Feature Vectors

 Instead of finding template exactly matching input template look
at how close feature vectors are
 Nearest neighbor classification algorithm:
1. Find template closest to the input
pattern. Class 1

2. Classify pattern to the same class Class 2

as closest template.

32
CLASSIFIER
K-Nearest Neighbor Classifier
 Use k nearest neighbors instead of 1 to classify pattern.

Class 1

Class 2

33
CLASSIFICATION
 Cancer Diagnosis – Tumor size for prediction

Malignant

Benign
Tumor Size

35
CLASSIFICATION
 Cancer Diagnosis – Generally more than one variables

Malignant
Benign

Age

Tumor Size
Why supervised – The algorithm is given a number of patients
with the RIGHT ANSWER and we want the algorithm to learn 36
to predict for new patients
CLASSIFICATION
 Cancer Diagnosis – Generally more than one variables
Predict for this
patient
Malignant
Benign

Age

Tumor Size

We want the algorithm to learn the separation line. Once a new

patient arrives with a given age and tumor size – Predict as 37

Malignant or Benign
SUPERVISED LEARNING - EXAMPLE
 Cancer diagnosis – Many more features
Patient ID # of Tumors Avg Area Avg Density Diagnosis
1 5 20 118 Malignant
2 3 15 130 Benign
3 7 10 52 Benign
4 2 30 100 Malignant

 Use this training set to learn how to classify patients

where diagnosis is not known:
Patient ID # of Tumors Avg Area Avg Density Diagnosis
101 4 16 95 ?
102 9 22 125 ?
103 1 14 80 ?

Input Data Classification

38
CONTENTS

 Supervised learning
 Classification

 Regression

 Unsupervised learning
 Reinforcement learning

39
COURSE OUTLINE
 Machine Learning: Theory and Applications
 Introduction to probability theory and Linear Algebra

 Bayesian Decision Theory

 Parametric Methods

 Dimensionality Reduction

 Frequent Pattern Analysis

 Clustering

 Decision Trees

 Artificial neural networks

 Advanced topics in Machine Learning: HMMs, Support

40
Vector Machines (SVM), …