Confusion Matrix, Scoring(Accuracy, Recall, Precision)
Introduction to Confusion Matrix
A confusion matrix is a table that illustrates the performance of a classification model. It compares
the actual outcomes with the model's predictions. For instance, consider a Covid prediction model:
The model can predict two categories: whether a patient has the disease (Predicted: YES) or
doesn't (Predicted: NO).
Among 165 tested patients, the classifier predicted that 55 patients don't have the disease,
and 110 do.
In reality, 60 patients don't have the disease, and 105 do.
Categories of Predictions
Correct Predictions:
True Positive (TP): Cases where the model correctly predicted the patient has the
disease.
True Negative (TN): Cases where the model correctly predicted the patient doesn't
have the disease.
Incorrect Predictions:
False Positive (FP) or Type 1 Error: Cases where the model incorrectly predicted the
patient has the disease.
False Negative (FN) or Type 2 Error: Cases where the model incorrectly predicted the
patient doesn't have the disease.
Categories of Predictions:
True Negative (TN): 50 patients correctly predicted as not having the disease.
True Positive (TP): 100 patients correctly predicted as having the disease.
False Positive (FP): 10 patients incorrectly predicted as having the disease.
False Negative (FN): 5 patients incorrectly predicted as not having the disease.
Metrics Derived from the Confusion Matrix:
1. Prevalence:
How often the actual condition appears in our sample.
Actual YES / n = 105 / 165 = 64%
2. Accuracy:
How often the classifier makes correct predictions.
(TP + TN) / n = (100 + 50) / 165 = 91%
3. Precision:
When the model predicts YES, how often is the actual outcome YES?
TP / (TP + FP) = 100 / (100 + 10) = 91%
4. Recall (Sensitivity, True Positive Rate):
Of all patients who actually have the disease (YES), how many were correctly
predicted by the model?
TP / (TP + FN) = 100 / (100 + 5) = 95%
5. Specificity (Selectivity, True Negative Rate):
Of all patients who actually don't have the disease (NO), how many were correctly
predicted by the model?
TN / (TN + FP) = 50 / (50 + 10) = 83%
6. False Positive Rate (Fall-out):
Among the patients who actually don't have the disease (NO), how often did the
model predict they have the disease (YES)?
FP / (FP + TN) = 10 / (10 + 50) = 17%
7. False Negative Rate (Miss Rate):
Among the patients who actually have the disease (YES), how often did the model
predict they don't have the disease (NO)?
FN / (FN + TP) = 5 / (5 + 100) = 5%
Figure sources: https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/#:~:text=A
%20confusion%20matrix%20is%20a,the%20true%20values%20are%20known
Churn example in Orange
Dataset is here: https://www.kaggle.com/datasets/blastchar/telco-customer-churn
Results are derived from a decision tree.
Total observations: n=22,490.
Categories of Predictions:
1. True Negative (TN): 14,967 observations - the model correctly predicted the customer would
stay.
2. True Positive (TP): 2,971 observations - the model correctly predicted the customer would
churn.
3. False Positive (FP): 1,759 observations - the model incorrectly predicted the customer would
churn.
4. False Negative (FN): 2,793 observations - the model incorrectly predicted the customer
would stay.
Scoring:
Accuracy: Reflects the proportion of correct predictions made by the classifier.
(TP + TN) / n = (14,967 + 2,971) / 22,490 = 79.8%
Precision: When the model predicts a customer will churn (YES), how often is this prediction correct?
TP / (TP + FP) = 2,971 / (2,971 + 1,759) = 62.8%
Now switch in the top right corner of confusion matrix to the „proportion of actual“.
Recall (Sensitivity, True Positive Rate): Of all customers who actually churned (YES), how many were
correctly predicted by the model?
TP / (TP + FN) = 2,971 / (2,971 + 2,793) = 51.5%
Specificity (Selectivity, True Negative Rate): Of all customers who actually stayed (NO), how many
were correctly predicted by the model?
TN / (TN + FP) = 14,967 / (14,967 + 1,759) = 89.5%
False Positive Rate (Fall-out):
Among the customers who actually stayed (NO), how often did the model predict they would
churn (YES)?
FP / (FP + TN) = 1,759 / (1,759 + 14,967) = 10.5%. In 10.5% of cases where a customer
stays, the model predicts the customer will churn.
False Negative Rate (Miss Rate):
Among the customers who actually churned (YES), how often did the model predict they
would stay (NO)?
FN / (FN + TP) = 2,793 / (2,793 + 2,971) = 48.5%. In 48.5% of cases where a customer
churns, the model predicts the customer will stay.
Observations:
The "Proportion of Actual" mode displays percentages as a proportion of the actual cases.
This means:
The percentage (Predicted_No, Actual_No) of the total Actual_No is Specificity.
The percentage (Predicted_Yes, Actual_No) of the total Actual_No is the False
Positive Rate.
The percentage (Predicted_No, Actual_Yes) of the total Actual_Yes is the False
Negative Rate.
The percentage (Predicted_Yes, Actual_Yes) of the total Actual_Yes is Recall.
The "Test & Score" component verifies these results. Ensure "Target Class: Yes" is selected.
You will see that the Accuracy, Precision, and Recall match the manually calculated values
Differences between Recall and Precision:
Recall: Deteriorates with an increasing number of false negatives. A false negative
means the customer churned, but the model predicted they would stay.
Precision: Deteriorates with an increasing number of false positives. A false positive
means the customer stays, but the model predicted they would churn.
From the "Test & Score" analysis, we observe that Precision is higher than Recall. This
suggests our model suffers more from the count of false negatives than false positives. This
is confirmed by the earlier table showing a false negative rate of 48.5% and a false positive
rate of only 10.5%. Hence, Precision is higher.
