10

1 2 3 4 5 6 Salary (feature_
 Confusion Matrix
Bias Variance &
Ensemble Methods
Bagging, Boosting, Random Forests
Confusion Matrix
ACTUAL
Cat 1 spam Cat2inbox Cat3promo Cat4social
PREDICTED Cat1 15 T P 3 FP 1 FP 12 FP
Cat2 3 F N 14 TN 9 TN 6 TN
Cat3 6 F N 6 TN 12 TN 7 TN

Cat4 2 F N 2 TN 10 TN 10 TN

 
Confusion Matrix
ACTUAL
Cat 1 Cat2 Cat3 Cat4
PREDICTED Cat1 15 T P 3 FP 1 FP 12 FP
Cat2 3 F N 14 TN 9 TN 6 TN
Cat3 6 F N 6 TN 12 TN 7 TN

Cat4 2 F N 2 TN 10 TN 10 TN

 
Confusion Matrix
ACTUAL
Cat 1 Cat2 Cat3 Cat4
PREDICTED Cat1 15 T P 3 FP 1 FP 12 FP
Cat2 3 F N 14 TN 9 TN 6 TN
Cat3 6 F N 6 TN 12 TN 7 TN

Cat4 2 F N 2 TN 10 TN 10 TN

1. Calculate Recall for Cat 1

2. Calculate Recall for Cat 2

3. Calculate overall Recall

 Micro and Macro average
• 

( /)i
Receiver Operating Characteristics ROC
Area Under the Curve AUC

= sensitivity

U
Confusion Matrix
ACTUAL
Cat 1 Cat2 Cat3 Cat4
PREDICTED Cat1 15 T P 3 FP 1 FP 12 FP
Cat2 3 F N 14 TN 9 TN 6 TN
Cat3 6 F N 6 TN 12 TN 7 TN

Cat4 2 F N 2 TN 10 TN 10 TN

5. Also important when quality of output product is imp

Recap confusion Matrix
Actual class
P N

Predict P TP FP
ed
class N FN TN

 
Confusion Matrix for Regression

Ground Truth
True Null False Null
Correct Conclusion Type I error
Don’t reject / (True negative (False Negative
Null
Prediction probability )= 1- Probability) =
Type II Error Correct Conclusion
Reject Null (False Positive (True Positive
Probability)= Probability) = 1-

1. Type 1 Error : Incorrectly accepted a False Null Hypothesis

2. Type II Error: Incorrectly rejected a True Null Hypothesis
Alpha
Generalization error
• 
Bias/Variance Tradeoff
• 

Expected Error SSE

 

Bias/Variance for all (x,y) is expectation over P(x,y).

Can also incorporate measurement noise, data entry errorb. Variance Bias
Bias/Variance Tradeoff
• 

Expected Error SSE

 

Bias/Variance for all (x,y) is expectation over P(x,y).

Can also incorporate measurement noise, data entry error. Variance Bias
 Too much specific
Error components OVERFITTED MODEL

Too much general

OR
UNDERFITTED MODEL R
E ER
C

ERROR
N
RIA
VA

BIAS
ERRO
R

NOISE

MODEL COMPLEXITY