ISOM3360 Data Mining for Business Analytics

Decision Tree Learning

Instructor: Yi Yang
Department of ISOM
Spring 2023
q Last lecture
q Data preparation

q This Lecture
q Decision tree

2
Data Mining Process

3
Supervised Learning

q Classification is used to predict which class

(discrete value) a data point belongs to

q Fraud detection, customer churn prediction

q Regression is used to predict continuous value.

q Stock price prediction, housing price prediction

4
Training vs. Testing

1. Learn the model on all data, evaluate on parts of data

2. Split data into two parts, learn the model on one part, and
evaluate on the other part.

Recall that ML model is used to make predictions on

unseen data.
5
Training vs. Testing

1. Learn the model on all data, evaluate on parts of data

2. Split data into two parts, learn the model on one part, and
evaluate on the other part.

Supervised learning rule of thumb:

Never ever use testing data to learn your model 6

Supervised Learning data split

q training set—a subset to train a model.

q test set—a subset to test the trained model.

q Large enough to yield statistically meaningful results

q Is representative of the data set as a whole. In other

words, don't pick a test set with different
characteristics than the training set.

7
Decision Tree

q Decision trees are one of the most popular data mining

tools.

q Decision Trees are easy to understand, implement and

use, and computationally cheap.

q “It is probably the machine learning workhorse most

widely used in practice to date.”

q The model comprehensibility is important for

communication to non-DM-savvy stakeholders.

8
Decision Tree

categorical continuous
Employed Balance Age Default
Yes 123,000 50 No
No 51,100 40 Yes
No 68,000 55 No
Yes 34,000 46 Yes
Yes 50,000 44 No
No 100,000 50 Yes

Yes 70,000 35 ?????

Predicting customers who will default on loan payments

9
Decision tree
Tree
v A upside down if-else tree, start Root Node
Employed
with all training data
Yes No
Node Class =
v Each node has an if-else condition Not
Balance Node
about one feature. (Which feature?) Default

v Dataset is split into subset based >=50K <50K

on the condition Class =
v Root node contains all training Not
Default Age
examples
v Leaf node contains a subset of >=45 <45
training examples Class =
Class =
v (Optional) numerical features are Not
Default
discretized Default
Leaf
Node

10
The essence of Decision Tree

q The essence of supervised learning (prediction) is to find

features that are informative and have high
predictability.

q Decision tree methods iteratively select a feature so that,

after splitting, the sub dataset becomes more
pure/homogenous. In other words, the selected feature
has high predictability.

q Information gain is one way to measure informativeness.

11
Entropy

q Entropy H(S) is a measure of the amount of

uncertainty/impurity in the dataset S (i.e. entropy
characterizes the dataset S). It measures chaos.

p(x) is the proportion of class x

in the data S

q Eg. A dataset is composed of 16 cases of class

“Positive” and 14 cases of class “Negative”

Entropy (dataset) =

12
Entropy Exercise
𝑡𝑖𝑝: 0 log ! 0 = 0

13
Let’s play a game. I have someone in my mind,
and your job is to guess this person. You can only
ask yes/no question.

This person is a HK celebrity.

Go!

14
Information Gain
q The information gain is based on the
decrease in entropy after a dataset is split
on a feature.

A: has credit card??

Weighted average of subset entropy
• H(S) – Entropy of set S

• T – The subsets created from splitting set S by feature A

• p(t) – The proportion of subset t to set S

Yes No
• H(t) – Entropy of subset t

15
Information Gain Example
A1: has credit card?? A2: is student??

Entropy=

before

Yes No Yes No after

Entropy=0.837 Entropy = 0.722

IG= 0.997 - 0.615 = 0.382 IG= 0.997 - 0.779 = 0.218 16

Information Gain Exercise
A1 A2

before

Yes No Yes No after

Without calculation, which split (left or right) gives the highest

information gain? Which feature (A1 or A2) do we prefer?
17
Decision Tree: ID3
ID3: Only for classification, only handles
categorical features.
q Step 1: Calculate the information gain of every feature
q Step 2: Split the set S into subsets using the feature for
which the information gain is maximum
q Step 3: Make a decision tree node containing that feature,
divide the dataset by its branches and repeat the same
process on every branch.
q Step 4a: A branch with entropy of 0 is a leaf node.
q Step 4b: A branch with entropy more than 0 needs further
splitting.

18
Outlook: Sunny,
Overcast, Rain

Temp: Hot, Mild,

Cool

Humidity: High,
Normal

Wind: Weak,
Strong

Decision:
Yes (9), No (5)

19
q On board demonstration

20
Recap: The essence of Decision Tree

q The essence of supervised learning (prediction) is to find

features that are informative and have high
predictability.

q Decision tree methods select a feature so that, after

splitting, the sub dataset becomes more homogenous. In
other words, the selected feature has high predictability.

q Information gain is one way to measure informativeness.

21