CS364: Machine Learning

Semester 2- year 1444

Level 6 BSc CS

Concept Learning and the

General-to-Specific Ordering

Instructors:
Prof. Hachemi BinNaceur
Dr. Waad Alhoshan
Outline

• Central problem in learning

• Concept learning task
• Concept learning as search
• Find-S algorithm
• Version Spaces
• Candidate Elimination algorithm

2
Central problem in learning

• Acquiring general concepts (or categories)

from specific training examples.
– “bird”, “car”, “situations in which I should study
more in order to pass the exam” …
– “bird” a subset of animals
• A function defined over all animals, whose value is true
for birds and false for other animals.
• Concept learning
– Inferring a Boolean valued function from training
examples of its input and output.
3
Concept learning task

• Training examples for EnjoySpt

• What is the general concept?

– “days on which my friend enjoys his favorite water
sport”

4
 Concept learning task (Cont.)
• A machine learning hypothesis is a candidate model
that approximates a target function for mapping
inputs to outputs
• Representing hypotheses
• Various representations for a hypothesis h.
– e.g. h is a conjunction of constraints on attributes.

5
 Concept learning task (Cont.)
• e.g. h is a conjunction of constraints on attributes.
– Each constraint can be
• A specific value (e.g. Water = Warm)
• Don’t care, any value is acceptable (e.g. Water = ?)
• No value allowed (e.g. Water = Ø)
Sky AirTemp Humid Wind Water Forecst
<Sunny, ?, ?, Strong, ?, Same>

• The most general hypothesis (every day is a positive example)

− <?, ?, ?, ?, ?, ?>
• The most specific hypothesis (no day is a positive example)
− < Ø, Ø, Ø, Ø, Ø, Ø >
6
Concept learning task (Cont.)

• Prototypical concept learning task

• Given:
– Instances X: Possible days, each described by the attributes
Sky, AirTemp, Humid, Wind, Water, Forecst.
– Target function c: EnjoySpt: X →{0, 1}
– Hypotheses H: Conjunctions of literals.
• e.g. <Cold, High, ?, ?, ?>
– Training examples D: Positive and negative examples of
the target function
<x1, c(x1)>, … <xm, c(xm)>
• Determine:
– A hypothesis h in H such that h(x) = c(x) for all x in D

7
 Concept learning task (Cont.)
• If some instance x satisfies all the constraints of
hypothesis h, then h classifies x as a positive
example (h(x) =1)

8
The inductive learning hypothesis

• Any hypothesis found to approximate the target function

well over a sufficiently large set of training examples will
also approximate the target function well over other
unobserved examples.

9
 Concept learning as search

• Searching through a large space of hypotheses

implicitly defined by the hypothesis representation.
– Find the hypothesis that best fits the training examples.
– Assume that the attribute Sky has three values, and that
AirTemp, Humidity, Wind, Water, and Forecst each have two
possible values
• X contains:
3*2*2*2*2*2 = 96 distinct instances
• H contains:
5*4*4*4*4*4 = 5120 syntactically distinct hypotheses within H.
• Evey hypothesis containting one or more Ø classifies every instance
as negative
(4*3*3*3*3*3) +1 = 973 semantically distinct hypotheses within
H. 10
Concept learning as search (Cont.)

• Our study of learning algorithms will examine different

strategies for searching very large or infinite hypothesis
spaces, to find the hypotheses that best fit the training data.

11
Instance, hypotheses, and more-general-
than
Given hypotheses hj and hk, hj is more-general-
than-or-equal-to hk if and only if any instance
that satisfies hk also satisfies hj.

12
Instance, hypotheses, and more-general-
than (Cont.)

13
Find-S algorithm

1. Initialize h to the most specific hypothesis in H

2. For each positive training instance x
– For each attribute constraint ai in h
• If the constraint ai in h is satisfied by x
Then do nothing
Else replace ai in h by the next more general constraint
that is satisfied by x
3. Output hypothesis h

14
Hypothesis space search by Find-S

15
Unanswered questions by Find-S

• Has the learner converged to the correct target

concept?
– Cannot tell whether it has learned concept.
– Cannot tell whether it has found the only hypothesis in H
consistent with the data or whether there are many other
consistent hypothesis
• Why prefer the most specific hypothesis?
– It is unclear whether we should prefer this hypothesis over
the most general or some other hypothesis of intermediate
generality.
• Cannot tell when training data inconsistent.
– Errors or noise
– Ignores the negative examples 16
Unanswered questions by Find-S
(Cont.)

• Find-S outputs the hypothesis:

h=<Sunny, Warm, ?, Strong, ?, ?>
This is one of the six different hypotheses from
H that are consistent with these training
examples.
They constitute the Version Space relative to
this set of data.

17
Consistent vs. Satisfies

• An example x is to satisfy hypothesis h when h(x) =1,

regardless of whether x is a positive or negative
example of the target concept.

• An example x is consistent with hypothesis h depends

on the target concept; whether h(x) = c(x).

18
Version Spaces

19
Version Space: Example

20
Representing Version Spaces

21
The List-Then-Eliminate Algorithm

22
Candidate Elimination Algorithm (Cont.)

• It represents the VS by storing only its most

general members (G) and its most specific
members (S).
• Given G and S, it is possible to enumerate all
members of the VS by generating the
hypotheses that lie between G and S
• Most general hypothesis in H: G0:{<?, ?, ?, ?, ?, ?>}
• Most specific hypothesis in H: S0:{< Ø, Ø, Ø, Ø, Ø,
Ø >}
Candidate Elimination Algorithm

24
Candidate Elimination Algorithm (Cont.)

G0:
Candidate Elimination Algorithm (Cont.)

G0, G1:
Candidate Elimination Algorithm (Cont.)

G0, G1:
Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm (Cont.)

G0, G1, G2:

Candidate Elimination Algorithm: Example
(Cont.)

• Positive training examples may force the S-

boundary of the VS to become increasingly general.
• Negative training examples may force the G-
boundary of the VS to become increasingly specific
(complimentary role).

34
Candidate Elimination Algorithm: Example (Cont.)
S and G delimit the VS of candidate hypotheses

35
How Should These Be Classified?

36
 How Should These Be Classified? (Cont.)

• <Sunny, Warm, Normal, Strong, Cool, Change > +

• <Rainy, Cold, Normal, Light, Warm, Same> -
• <Sunny, Warm, Normal, Light, Warm, Same> ?

03/12/10
 References

• Gareth James, Daniela Witten, Trevor Hastie, and

Robert Tibshirani, An Introduction to Statistical
Learning with Applications in R, second edition,
Springer, New York, 2021.
• Tom M. Mitchell. Machine Learning, McGraw Hill
Inter. Editions, 1997.

38