DECISION LISTS AND

TREES
Intelligent Engineering Algorithms
COE 544

John Geagea
Rami Challita
 2

OUTLINE
1. Introduction
2. Background
3. Decision Lists
4. Decision Tree Representation
5. Learning Algorithms
1. ID3 Algorithm
2. ID3 Algorithm Example
6. Overfitting
7. Avoiding Overfitting
8. Pruning
9. Decision Tree Simulation Example
10.Conclusion
 3

INTRODUCTION

• Decision Trees are among the most widely used methods for inductive inference.

• It is a method for approximating discrete-valued functions.

• Robust to noisy data and can learn disjunctive expressions.

• Composed of a series of If-Then rules.

• Decision Trees are a method of supervised learning.

 4

BACKGROUND

• Invented by John Ross Quinlan in 1986.

• Main fields of use:

• Medical diagnosis
• Data mining
• Risk analysis
• Finance
• Machine learning
• Project management
 5

DECISION LISTS

• A decision list (DL) is an ordered list of conjunctive rules.

• Rules can overlap, so the order is important.

• A k-DL: the length of every rule is at most k.

• DLs can be represented as Decision Trees

 6

PROS AND CONS

• Pros:
• Easy to interpret and intuitive.
• Automatic prioritization of attributes.
• Versatile and can be used in various situations.
• Easily implemented in code.

• Cons:
• Can become complex very fast.
• All data used in building the tree should be precise, since a small change can
cause large changes in the tree.
• Data used to build the tree should be up-to-date.
• Hard to build without the aid of a computer.
 7

DECISION TREE(DT)
REPRESENTATION (1)

• Classifies instances by sorting them down the tree from the root to a leaf
node.

• Each node specifies a test of some attribute of the instance.

• Each branch descending from a node corresponds to one of the possible

values for the attribute.
 8

DECISION TREE(DT)
REPRESENTATION (2)
• k-DT: The depth of a DT is at most k.

• A DT defines a boolean formula: it looks at the paths where the leaf node is
1.
 9

DECISION TREE(DT)
REPRESENTATION (3)
 10

DECISION TREE(DT)
REPRESENTATION (4)
(Outlook = Sunny) ᴧ (Humidity = Normal)
ᴠ
Outlook = Overcast
ᴠ
(Outlook = Rain) ᴧ (Wind = Weak)
 11

LEARNING ALGORITHMS

• Many learning algorithms exist.

• All are variations of the ID3 algorithm (Iterative Dichotomiser 3).

• A successor of ID3 is C4.5 invented in 1993.

 12

ID3 ALGORITHM

• Employs a top-down greedy search.

• Searches for the best attribute to act as the root node.

• Repeats this process recursively for each child.

• Stops when:
• All the instances have the same target attribute value.
• There are no more attributes.
• There are no more instances.
 13

Consider the following ID3 EXAMPLE(1)

Dataset
 14

ID3 EXAMPLE(2)
 15

ID3 EXAMPLE(3)

• Same calculations for Outlook, Humidity, Temperature

 Gain(S, Wind) = 0.048

 Gain(S, Outlook) = 0.246
 Gain(S, Humidity) = 0.151
 Gain(S, Temperature) = 0.029
 16

ID3 EXAMPLE(4)
The Outlook attribute is the root Element having
the largest gain
 17

ID3 EXAMPLE(5)
Computing the gains of the attribute Sunny relative to:
• Humidity
• Temperature
• Wind

Gain(Ssunny , Humidity) = 0.97 – (3/5)*0 – (2/5)*0 = 0.97

Gain(Ssunny , Temperature) =0.97 – (2/5)*0 – (2/5)*1 –(1/5)*0=

0.570

Gain(Ssunny , Wind) = 0.97 – (2/5)*1 - (3/5)*0.918 = 0.019

Gain(Ssunny , Humidity) is the largest gain amongst them

CONTINUOUS-VALUED ATTRIBUTES
- Create a new Boolean value that is true when
humidity<=c and false otherwise.
- C: threshold
- We select, for example, the (Humidity: sunny) fields

Adjacent elements
with distinctive
choices
 19

OVERFITTING

Is a result of irrelevant data (noise).

Overfitting occurs in decision trees that are more complex than necessary.

A learning algorithm is said to overfit when it is:

• More accurate in fitting known data.
• Less accurate in predicting new data.
 20

OVERFITTING
 21

AVOIDING OVERFITTING

In order to avoid overfitting:

• Stop growing the tree.

• Grow the tree to the maximum then apply pruning.

 22

PRUNING
Means reducing size of larger and deeper trees
Needed to avoid overfiting

Main types of pruning:

• Reduced-Error Pruning

• Post Pruning
 23

REDUCED-ERROR PRUNING

• Consider each node for pruning.

• A node is removed if the resulting tree does not harm the prediction outcome.

• Nodes are removed iteratively.

• Pruning continues until further pruning is harmful.

• More effective when large amount of data is available.

 24

REDUCED-ERROR PRUNING
 25

RULE-POST PRUNING

• Grow the tree until overfitting occurs.

• Convert tree to equivalent set of rules.

• Prune each rule by removing any preconditions in order to increase its

estimated accuracy.

• Sort the pruned rules by their estimated accuracy and consider them in this
sequence when classifying.
 26

RULE-POST PRUNING EXAMPLE

• An IF THEN rule is generated for
each leaf node.
• IF (Outlook = Sunny) ∧ (Humidity =
High)
THEN PlayTennis = No

• Prune by removing unnecessary

conditions (Outlook = Sunny) or
(Humidity = High).

• Remove the condition where its

removal results in improved
estimated rule accuracy.

• Estimate rule accuracy by using a

validation set of examples disjoint
from the training set.
 27

CONCLUSION

• Decision tree is a method for inductive inference.

• The ID3 family of algorithms infers decision trees by growing them from top to
bottom.

• Overfitting leads to less accuracy in predicting new data.

• Pruning is the process of removing unnecessary tree nodes to improve

prediction accuracy.
 28

DEMO
• http://webdocs.cs.ualberta.ca/~aixplore/learning/DecisionTrees/Applet/De
cisionTreeApplet.html
 29

REFERENCES
• T. M. Mitchell, “Decision Tree Learning” in Machine Learning, 1st ed. USA,
1997, ch. 3, pp. 52-78.

• https://people.csail.mit.edu/rivest/Rivest-DecisionLists.pdf

• https://cs.uwaterloo.ca/~mli/dl.ps