DECISION TREES DECISION TREES

Introduction Introduction

It is a method that induces concepts from examples The target function can be Boolean or discrete valued
(inductive learning)

Most widely used & practical learning method

The learning is supervised: i.e. the classes or categories of the

data instances are known

It represents concepts as decision trees (which can be

rewritten as if-then rules)

1 2

DECISION TREES DECISION TREES

Decision Tree Representation Example

1. Each node corresponds to an attribute

2. Each branch corresponds to an attribute value

3. Each leaf node assigns a classification

3 4

DECISION TREES DECISION TREES

Example
Outlook Decision Tree Representation
Sunny Rain
Overcast
Decision trees represent a disjunction of conjunctions of
Humidity Wind constraints on the attribute values of instances

High Normal Strong Weak Each path from the tree root to a leaf corresponds to a
conjunction of attribute tests (one rule for classification)

The tree itself corresponds to a disjunction of these

conjunctions (set of rules for classification)
A Decision Tree for the concept PlayTennis
An unknown observation is classified by testing its attributes
and reaching a leaf node
5 6

1
 DECISION TREES DECISION TREES

Decision Tree Representation Basic Decision Tree Learning Algorithm

Most algorithms for growing decision trees are variants of a

basic algorithm

An example of this core algorithm is the ID3 algorithm

developed by Quinlan (1986)

It employs a top-down, greedy search through the space of

possible decision trees

7 8

DECISION TREES DECISION TREES

Basic Decision Tree Learning Algorithm Basic Decision Tree Learning Algorithm
We have
First of all we select the best attribute to be tested at the root
of the tree D12 D11
D1 - 12 observations

For making this selection each attribute is evaluated using a D2 D5

D10 D4 - 4 attributes
statistical test to determine how well it alone classifies the D6 • Outlook
training examples D3
D14 • Temperature
D8 D9 • Humidity
D7 D13 • Wind

- 2 classes (Yes, No)

9 10

DECISION TREES DECISION TREES

Basic Decision Tree Learning Algorithm Basic Decision Tree Learning Algorithm

Outlook The selection process is then repeated using the training

Sunny Rain examples associated with each descendant node to select the
Overcast best attribute to test at that point in the tree

D1 D8 D10 D6
D3
D14
D11 D4
D9 D12
D2 D7 D5
D13

11 12

2
 DECISION TREES DECISION TREES

Outlook
Sunny Rain Basic Decision Tree Learning Algorithm
Overcast
This forms a greedy search for an acceptable decision tree, in
D1 D8 D10 D6 which the algorithm never backtracks to reconsider earlier
D3
D14 choices
D11 D4
D9 D12
D2 D7
D13 D5

What is the
“best” attribute to test at this point? The possible choices are
Temperature, Wind & Humidity
13 14

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier? Which Attribute is the Best Classifier?: Definition of Entropy

The central choice in the ID3 algorithm is selecting which In order to define information gain precisely, we begin by
attribute to test at each node in the tree defining entropy

We would like to select the attribute which is most useful for Entropy is a measure commonly used in information theory,
classifying examples called.

For this we need a good quantitative measure Entropy characterizes the impurity of an arbitrary collection
of examples
For this purpose a statistical property, called information
gain is used

15 16

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Definition of Entropy Which Attribute is the Best Classifier?: Definition of Entropy

Suppose we have four independent values of a variable X: Someone tells you that there probabilities of occurrence is
not equal:
A, B, C, D
p(A) = 1/2
These values are independent and occur randomly p(B) = 1/4
p(C) = 1/8
You might transmit these values over a binary serial link by p(D) = 1/8
encoding each reading with two bits
It is now possible to invent a coding that only uses 1.75 bits
A = 00B = 01 C = 10 D = 11 on average per symbol, for the transmission, e.g.

We might see something like this: 0100001001001110110011 A= 0 B = 10 C = 110 D = 111

17 18

3
 DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Definition of Entropy Which Attribute is the Best Classifier?: Definition of Entropy

Suppose X can have m values, V1, V2, …, Vm, with
probabilities: p1, p2, …, pm
The smallest number of bits, on average, per value, needed to
transmit a stream of values of X is
If all p’s are equal for a given m, we need the highest number
of bits for transmission
If there are m possible values of an attribute, then the
entropy can be as large as log2 m

If a p = 1 and all other p’s are 0, then we need 0 bits (i.e. we

don’t need to transmit anything)

19 20

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Definition of Entropy Which Attribute is the Best Classifier?: Information Gain

This formula is called Entropy H Suppose we are trying to predict output Y (Like Film
Gladiator) & we have input X (College Major = v)
H(X) =

Major
High Entropy means that the examples have more equal Math CS
probability of occurrence (and therefore not easily
History
predictable)

Low Entropy means easy predictability

21 22

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Information gain Which Attribute is the Best Classifier?: Information Gain
We have H(X) = 1.5 H(Y) = 1.0 Conditional Entropy of Y
H(Y | X = Math) = 1.0
Conditional Entropy H(Y | X = v) H(Y | X = History) = 0
The Entropy of Y among only those records in which X = v H(Y | X = CS) = 0

Major Major
Math CS Math CS
History History

23 24

4
 DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Information Gain Which Attribute is the Best Classifier?: Information Gain
Average Conditional Entropy of Y
Information Gain is the expected reduction in entropy
caused by partitioning the examples according to an
H(Y | X) =
attribute’s value
Major
Math CS Info Gain (Y | X) = H(Y) – H(Y | X) = 1.0 – 0.5 = 0.5
History
For transmitting Y, how much bits would be saved if both
side of the line knew X

In general, we write Gain (S, A)

Where S is the collection of examples & A is an attribute

25 26

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Information Gain Which Attribute is the Best Classifier?: Information Gain

The collection of examples has 9 positive values and 5

negative ones
Let’s
investigate
the attribute
Wind Eight (6 positive and 2 negative ones) of these examples
have the attribute value Wind = Weak

Six (3 positive and 3 negative ones) of these examples have

the attribute value Wind = Strong

27 28

DECISION TREES DECISION TREES

Which Attribute is the Best Classifier?: Information Gain Which Attribute is the Best Classifier?: Information Gain

The information gain obtained by separating the We calculate the Info Gain for each attribute and select
examples according to the attribute Wind is calculated as: the attribute having the highest Info Gain

29 30

5
 DECISION TREES DECISION TREES

Select Attributes which Minimize Disorder Select Attributes which Minimize Disorder
Make decision tree by selecting tests which minimize Make decision tree by selecting tests which minimize
disorder (maximize gain) disorder (maximize gain)

31 32

DECISION TREES DECISION TREES

Select Attributes which Minimize Disorder Example

The formula can be converted from log2 to log10

logx(M) = log10M . logx10

= log10M/log10x

Hence log2(Y) = log10(Y)/log10(2)

33 34

DECISION TREES DECISION TREES

Example

Which attribute should be selected as the first test?

“Outlook” provides the most information

35 36

6
 DECISION TREES DECISION TREES

Example Example

The process of selecting a new attribute is now repeated for
each (non-terminal) descendant node, this time using only
training examples associated with that node
Attributes that have been incorporated higher in the tree are
excluded, so that any given attribute can appear at most once
along any path through the tree
This process continues for each new leaf node until either:
1. Every attribute has already been included along this path
through the tree
2. The training examples associated with a leaf node have
zero entropy

37 38

DECISION TREES DECISION TREES

Example From Decision Trees to Rules

Next Step: Make rules from the decision tree

After making the identification tree, we trace each path from

the root node to leaf node, recording the test outcomes as
antecedents and the leaf node classification as the consequent

For our example we have:

If the Outlook is Sunny and the Humidity is High then No

If the Outlook is Sunny and the Humidity is Normal then Yes
...
39 40

DECISION TREES DECISION TREES

Hypothesis Space Search Hypothesis Space Search

ID3 can be characterized as It begins with an empty tree,

searching a space of then considers more and
hypotheses for one that fits more elaborate hypothesis
the training examples in search of a decision tree
that correctly classifies the
The space searched is the set training data
of possible decision trees
The evaluation function that
ID3 performs a simple-to- guides this hill-climbing
complex, hill-climbing search is the information
search through this gain measure
hypothesis space
41 42

7
 DECISION TREES
Hypothesis Space Search
Some points to note:
• The hypothesis space of all decision trees is a complete
space. Hence the target function is guaranteed to be
present in it.
43
DECISION TREES
Hypothesis Space Search
• ID3 performs no backtracking, therefore it is
susceptible to converging to locally optimal solutions
• ID3 uses all training examples at each step to refine its
current hypothesis. This makes it less sensitive to
errors in individual training examples.
However, this requires that all the training examples
are present right from the beginning and the learning
cannot be done incrementally with time
45
DECISION TREES
Learning Bias during Induction
• We cannot describe precisely the bias, but we can say
approximately that:
- Its selection prefers shorter trees over longer ones
- Trees that place high info. gain attributes close to the
root are preferred over those that do not
47
DECISION TREES
Hypothesis Space Search
• ID3 maintains only a single current hypothesis as it
searches through the space of decision trees.
By determining only a single hypothesis, ID3 loses the
capabilities that follow from explicitly representing all
consistent hypotheses.
For example, it does not have the ability to determine
how many alternative decision trees are consistent with
the training data, or to pose new instance queries that
optimally resolve among these competing hypotheses
44
DECISION TREES
Learning Bias during Induction
• Given a collection of training examples, there are
typically many decision trees consistent with these
examples
• Describing the inductive bias of ID3 means describing the
basis by which it chooses one of these consistent
hypotheses over the others
46
DECISION TREES
Learning Bias during Induction
• We can say “it absolutely prefers shorter trees over
longer ones” if there is an algorithm such that:
- It begins with an empty tree and searches breadth
first through progressively more complex trees,
first considering “all” trees of depth 1, then “all”
trees of depth 2, etc.
- Once it finds a decision tree consistent with the
training data, it returns the smallest consistent
tree
48
8
 DECISION TREES DECISION TREES

Learning Bias during Induction Learning Bias during Induction

• Is this bias for shorter trees a sound basis for • Example:
generalization beyond the training data? Let there be a small set of 20 training examples

• William of Occam, in the year 1320, thought of the We might expect to be able to find many 500 node
following bias (called Occam’s razor): decision trees consistent with these examples, than 5 node
Prefer the simplest hypothesis that fits the data decision trees

• One argument in its favor is that because there are fewer We might therefore believe that a 5-node tree is less likely
short hypotheses than long ones, it is less likely that a to be a statistical coincidence and prefer this hypothesis
short hypothesis coincidentally fit the training data over the 500 node hypothesis

49 50

DECISION TREES DECISION TREES

Decision Trees: Issues in Learning Avoiding Over-fitting the Data

Practical issues in learning decision trees include:
• How deeply to grow the decision tree
• Handling continuous attributes
• Choosing an appropriate attribute selection measure
The ID3 algorithm grows each branch of the tree just deeply
enough to perfectly classify the training examples
While this is sometimes a reasonable strategy, in fact it can
lead to difficulties when there is noise in the data, or when
the number of training examples is too small to produce a
representative sample of the true target function

• Handling training data with missing attribute values In either of these cases, ID3 can produce trees that over-fit
the training examples
• Handling attributes with differing costs
51 52

DECISION TREES DECISION TREES

Avoiding Over-fitting the Data Avoiding Over-fitting the Data

A hypothesis over-fits the training examples if some other

hypothesis that fits the training examples less well actually
performs better over the entire distribution of instances (i.e.
including instances beyond the training set)

Figure illustrates the impact of over-fitting in a typical

application of decision tree learning

53 54

9
 DECISION TREES DECISION TREES

Avoiding Over-fitting the Data Avoiding Over-fitting the Data

Example:
Over-fitting can occur when the training examples contain
random errors or noise
Sunny, Hot, Normal Humidity, Strong Wind = No Tennis
Example:
If we have the following incorrect training observation

Sunny, Hot, Normal Humidity, Strong Wind = No Tennis

The decision tree will become more complex to accommodate

this training observation

55 56

DECISION TREES DECISION TREES

Avoiding Over-fitting the Data Avoiding Over-fitting the Data

Over-fitting can also occur when small number of training
examples are associated with leaf nodes There are several approaches to avoid over-fitting.

In this case, it is possible for coincidental regularities to One popular approach is to prune over-fit trees
occur, in which some attribute happens to partition the
examples very well, despite being unrelated to the actual A key question is: what criterion is to be used to determine
target function the correct final tree size

Whenever such coincidental regularities exist, there is a risk A commonly used practice is to use a separate set of
of overfitting examples, distinct from training examples, (called validation
set) for post-pruning nodes
Example: If Days is an attribute, and we have only one or
two observations for each day
57 58

DECISION TREES DECISION TREES

Avoiding Over-fitting the Data Avoiding Over-fitting the Data

In this approach, the available observations are separated
into sets:
- A training set: which is used to learn the decision
tree
- A validation set: which is used to prune the tree
The motivation: Even though the learner may be misled by
random errors and coincidental regularities within
the training set, the validation set is unlikely to exhibit
the same random fluctuations
Therefore, the validation set can be expected to provide a
safety check against over-fitting the spurious
characteristics of the training set
Of course, it is important that the validation set be large
enough to itself provide a statistically significant
sample of the instances
One common heuristic is to withhold one-third of the
available examples for the validation set, using the
other two-thirds for training

59 60

10
 DECISION TREES
Avoiding Over-fitting the Data: Reduced Error Pruning
One approach is called “reduced error pruning”
It is a form of backtracking in the hill climbing search
of decision tree hypotheses space
It considers each of the decision nodes in the tree to be
candidates for pruning
Pruning a decision node consists of
- removing the sub-tree rooted at that node, making it
a leaf node, and
- assigning it the most common classification of the
training examples affiliated with that node
61
DECISION TREES
Avoiding Over-fitting the Data: Reduced Error Pruning
Pruning of the node continues until further pruning is
harmful (i.e. decreases accuracy of the tree over the
validation set)
Here, the available data has been split into three sub-sets:
- the training examples
- the validation examples for pruning
- the test examples used to provide an unbiased estimate
of accuracy of the pruned tree
63
DECISION TREES
Avoiding Over-fitting the Data: Reduced Error Pruning
The major drawback of this approach is that when data is
limited, withholding part of it for the validation set
reduces even further the number of examples
available for training
65
DECISION TREES
Avoiding Over-fitting the Data: Reduced Error Pruning
Nodes are removed only if the resulting pruned tree
performs no worse than the original over the
validation set
This has the effect that any leaf node added due to
coincidental regularities in the training set is likely to
be pruned because these same coincidences are
unlikely to occur in the validation set
Nodes are pruned iteratively, always choosing the node
whose removal most increases the decision tree
accuracy over the validation set
62
DECISION TREES
Avoiding Over-fitting the Data: Reduced Error Pruning
64
DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
Rule post-pruning involves the following steps:
1. Infer the decision tree from the training set (allowing
over-fitting to occur)
2. Convert the learned tree into an equivalent set of rules by
creating one rule for each path from the root node to a
leaf node
3. Prune (generalize) each rule by pruning any
preconditions that result in improving its estimated
accuracy
4. Sort the pruned rules by their estimated accuracy, and
consider them in this sequence when classifying
subsequent instances
66
11
 DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
Example: If (Outlook = sunny) and (Humidity = high)
then Play Tennis = no
Rule post-pruning would consider removing
preconditions one by one
It would select whichever of these removals produced the
greatest improvement in estimated rule accuracy, then
consider pruning the second precondition as a further
pruning step
No pruning is done if it reduces the estimated rule accuracy
DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
Rule post-pruning involves the following steps:
1. Infer the decision tree from the training set (allowing
over-fitting to occur)
2. Convert the learned tree into an equivalent set of rules by
creating one rule for each path from the root node to a
leaf node
3. Prune (generalize) each rule by pruning any
preconditions that result in improving its estimated
accuracy
4. Sort the pruned rules by their estimated accuracy, and
consider them in this sequence when classifying
subsequent instances
DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
The main advantage of this approach:
Each distinct path through the decision tree produces a
distinct rule
Hence removing a precondition in a rule does not
mean that it has to be removed from other rules as
well
In contrast, in the previous approach, the only two
choices would be to remove the decision node
completely, or to retain it in its original form
DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
The main advantage of this approach:
Each distinct path through the decision tree produces a
the distinct rule
Hence removing a precondition in a rule does not
mean that it has to be removed from other rules as
well
In contrast, in the previous approach, the only two
choices would be to remove the decision node
completely, or to retain it in its original form
67 68
DECISION TREES
Avoiding Over-fitting the Data: Rule Post-Pruning
Example: If (Outlook = sunny) and (Humidity = high)
then Play Tennis = no
Rule post-pruning would consider removing the
preconditions one by one
It would select whichever of these removals produced the
greatest improvement in estimated rule accuracy, then
consider pruning the second precondition as a further
pruning step
No pruning is done if it reduces the estimated rule accuracy
69 70
DECISION TREES
Decision Trees: Issues in Learning
Practical issues in learning decision trees include:
• How deeply to grow the decision tree
• Handling continuous attributes
• Choosing an appropriate attribute selection measure
• Handling training data with missing attribute values
• Handling attributes with differing costs
71 72
12
 DECISION TREES
Continuous Valued Attributes Continuous Valued Attributes
• If an attribute has continuous values, we can dynamically • Example:
define new discrete-valued attributes that partition
the continuous attribute value into a discrete set of
intervals
• In particular, for an attribute A that is continuous
valued, the algorithm can dynamically create a new
Boolean attribute Ac that is true if A < c and false
otherwise
• The only question is how to select the best value for the
threshold c
73
DECISION TREES
Continuous Valued Attributes Continuous Valued Attributes
• We sort the examples according to the continuous
attribute A
• Then identify adjacent examples that differ in their
target classification
• We generate a set of candidate thresholds midway
between the corresponding values of A
• These candidate thresholds can then be evaluated by
computing the information gain associated with each
75
DECISION TREES
Continuous Valued Attributes Training Examples with Missing Attribute Values
• This dynamically created Boolean attribute can then
compete with other discrete valued candidate
attributes available for growing the decision tree
• An extension to this approach is to split the continuous
attribute into multiple intervals rather than just two
intervals (i.e. the attribute become multi-valued,
instead of Boolean)
77
DECISION TREES
Let the training examples associated with a particular
node have the following values for the continuous
valued attribute Temperature and the target
attribute Play Tennis
Temperature: 40 48 60 72 80 90
Play Tennis: No No Yes Yes Yes No
74
DECISION TREES
• In the current example, there are two candidate
thresholds, corresponding to the values of
Temperature at which the value of Play Tennis
changes: (48 + 60)/2 and (80 + 90)/2
• The information gain is computed for each of these
attributes, Temperature > 54 and Temperature > 85,
and the best is selected (Temperature > 54)
76
DECISION TREES
• In certain cases, the available data may have some
examples with missing values for some attributes
• In such cases the missing attribute value can be
estimated based on other examples for which this
attribute has a known value
• Suppose Gain(S,A) is to be calculated at node n in the
decision tree to evaluate whether the attribute A is the
best attribute to test at this decision node
• Suppose that <x, c(x)> is one of the training examples
with the value A(x) unknown
78
13
 DECISION TREES
Training Examples with Missing Attribute Values
• One strategy for filling in the missing value: Assign it
the value most common for the attribute A among
training examples at node n
• Alternatively, we might assign it the most common
value among examples at node n that have the
classification c(x)
• The training example using the estimated value can
then be used directly by the decision tree learning
algorithm
79
DECISION TREES
Training Examples with Missing Attribute Values
• A fractional 0.6 of instance x is distributed down the
branch for A = 1, and a fractional 0.4 of x down the
other tree branch
• These fractional examples, along with other “integer”
examples are used for the purpose of computing
information Gain
• This method for handling missing attribute values is
used in C4.5
81
DECISION TREES
Handling Attributes with Differing Costs
• In some learning tasks, the attributes may have
associated costs
• For example, we may attributes such as Temperature,
Biopsy Result, Pulse, Blood Test Result, etc.
• These attributes vary significantly in their costs
(monetary costs, patient comfort, time involved)
• In such tasks, we would prefer decision trees that use
low-cost attributes where possible, relying on high
cost attributes only when needed to provide reliable
classifications
83
DECISION TREES
Training Examples with Missing Attribute Values
• Another procedure is to assign a probability to each of
the possible values of A (rather than assigning only
the highest probability value)
• These probabilities can be estimated by observing the
frequencies of the various values of A among the
examples at node n
• For example, given a Boolean attribute A, if node n
contains six known examples with A = 1 and four with
A = 0, then we would say the probability that A(x) = 1
is 0.6 and the probability that A(x) = 0 is 0.4
80
DECISION TREES
Classification of Instances with Missing Attribute Values
• The fractioning of examples can also be applied to
classify new instances whose attribute values are
unknown
• In this case, the classification of the new instance is
simply the most probable classification, computed by
summing the weights of the instance fragments
classified in different ways at the leaf nodes of the tree
82
DECISION TREES
Handling Attributes with Differing Costs
• In ID3, attribute costs can be taken into account by
introducing a cost term into the attribute selection
measure
• For example, we might divide the Gain by the cost of the
attribute, so that lower-cost attributes would be
preferred
• Such cost-sensitive measures do not guarantee
finding an optimal cost-sensitive decision tree
• However, they do bias the search in favor of low cost
attributes
84
14
 DECISION TREES DECISION TREES

Handling Attributes with Differing Costs
• Another example of selection measure is:
Gain2 (S,A) / Cost(A)
where S = collection of examples & A = attribute
• Yet another selection measure can be
2Gain (S,A) – 1 / {Cost(A) + 1}w
Alternate Measures for Selecting Attributes
• There is a problem in the information gain measure. It
favors attributes with many values over those with few
values
• Example: An attribute “Date” would have the highest
information gain (as it would alone perfectly fit the
training data)
• To cushion this problem the Info. Gain is divided by a
term called “Split Info”

where w  [0, 1] is a constant that determines the

relative importance of cost versus information gain
85 86

DECISION TREES DECISION TREES

Alternate Measures for Selecting Attributes Alternate Measures for Selecting Attributes

• Example:
Let there be 100 training examples at a node A1, with
where Si is the subset of S for which A has value vi 100 branches (one sliding down each branch)
Note that the attribute A can take on c different values,
Split Info (S, A1) = - 100 * 1/100 * log2 (0.01)
e.g. if A = Outlook,
= log2(100) = 6.64
then v1 = Sunny, v2 = Rain, v3 = Overcast

Let there 100 training examples at a node A2, with 2

When divided by Split Information the measure is called
branches (50 sliding down each branch)
Gain Ratio
Split Info (S, A2) = - 2 * 50/100 * log2 (0.5) = 1

87 88

DECISION TREES DECISION TREES

Alternate Measures for Selecting Attributes Decision Boundaries

• Problem with this Solution!!!

The denominator can be zero or very small when
Si  S
for one of the Si

To avoid selecting attributes purely on this basis, we

can adopt some heuristic such as first calculating the
Gain of each attribute, then applying the Gain Ratio
test only those considering those attributes with above
average Gain

89 90

15
 DECISION TREES DECISION TREES

Decision Boundaries Advantages

• Easy Interpretation: They reveal relationships between

the rules, which can be derived from the tree. Because
of this it is easy to see the structure of the data.

• We can occasionally get clear interpretations of the

categories (classes) themselves from the disjunction of
rules produced, e.g. Apple = (green AND medium) OR
(red AND medium)

91 92

DECISION TREES DECISION TREES

Advantages Disadvantages

• Classification is rapid & computationally inexpensive
• Trees provide a natural way to incorporate prior
knowledge from human experts
• They may generate very complex (long) rules, which
are very hard to prune
• They generate large number of rules. Their number
can become excessively large unless some pruning
techniques are used to make them more
comprehensible.

• They require big amounts of memory to store the

entire tree for deriving the rules.

93 94

DECISION TREES DECISION TREES

Disadvantages Appropriate Problems for Decision Tree Learning

• They do not easily support incremental learning.
Although ID3 would still work if examples are
supplied one at a time, but it would grow a new
decision tree from scratch every time a new example
is given
• There may be portions of concept space which are not
labeled
e.g. If low income and bad credit history then high
risk
but what about low income and good credit history?
95
• Instances are represented by discrete attribute-value pairs
(though the basic algorithm was extended to real-valued
attributes as well)
• The target function has discrete output values
• Disjunctive hypothesis descriptions may be required
• The training data may contain errors
• The training data may contain missing attribute values
96

16
 DECISION TREES

Reference

Sections 3.1 – 3.7.1 of T. Mitchell

97

17