CONFUSION MATRIX:

A confusion matrix, also called a contingency table or error

matrix, is used to visualize the performance of a classifier.
The columns of the matrix represent the instances of the
predicted classes and the rows represent the instances of the
actual class. (Note: It can be the other way around as well.)
The confusion matrix shows the ways in which your classification
model is confused when it makes predictions.

Let’s take an example:

In this confusion matrix, the "correct" cells are:

 TN: the number of true negatives, i.e., patients who
did not have cancer whom we correctly diagnosed
as not having cancer.
 TP: the number of true positives, i.e., patients who did
have cancer whom we correctly diagnosed as having
cancer

And the "error" cells are:

 FN: the number of false negatives, i.e., patients who did
have cancer whom we incorrectly diagnosed
as not having cancer
 FP: the number of false positives, i.e., patients who
did not have cancer whom we incorrectly diagnosed as
having cancer

Precision is
(Correctly predicted Positive) / (total predicted Positive) =
(TP) / (TP+FP)

Which tells us what proportion of patients we diagnosed as

having cancer actually had cancer.

Recall is
(Correctly predicted Positive) / (total correct Positive
observation)= (TP) / (TP+FN)

Which tells us what proportion of patients that actually had

cancer were diagnosed by us as having cancer.

Like so:
Accuracy: Overall, how often is the classifier correct?
Accuracy = (TP+TN) / (TP +FP+TN+FN)

Note:
Recall tells you how much of the +ve's you can find.
Precision tells you how much junk there is in your
predicted +ve's.

Another Example:

Confusion Predicted classes

Matrix
male female

Actual male 42 8
classes
female 18 32
This means that the classifier correctly predicted a male
person in 42 cases and it wrongly predicted 8 male
instances as female.

It correctly predicted 32 instances as female. 18 cases

had been wrongly predicted as male instead of female.

Four Outcomes of Binary Classification

 True positives: data points labeled as positive that are

actually positive
 False positives: data points labeled as positive that are
actually negative.
 True negatives: data points labeled as negative that are
actually negative
 False negatives: data points labeled as negative that are
actually positive

Recall and Precision Metrics

 Recall: ability of a classification model to identify all relevant

instances, means: Out of all the positive classes, how much we
predicted correctly. It should be high as possible.

Recall = True Positive / (True Positive + False Negative)

 Precision: ability of a classification model to return only

relevant instances, means: Out of all the positive classes we
have predicted correctly, how many are actually positive.

Precision: True Positive / (True Positive + False Positive)

  F1 score: It’s a measure of a test's accuracy. It considers
both the precision and the recall of the test to compute the
score using the harmonic mean.

2*(Precision*recall / precision + recall)

More formulas:

 TRUE POSITIVE RATE(Sensitivity)

TPR =TP/TP+FN

 FALSE POSITIVE RATE (Specificity)

FPR = (FP/FP+TN

 POSITIVE = TP+FN
 NEGATIVE = TP+FP

Example. (Why Recall is important) Suppose we have a weather

forecasting device; to predict rainy days. If this device has a high
precision, it means when it says “it is going to rain”, we can trust it. But
this doesn’t give us any information about all the times it says “it is not
going to rain”! If the false claims is going to be costly for our business,
then we might want to ask about the number of times this device
misses a rainy day.
Why precision is important then? Imagine our device is so stupid that it
ALWAYS claims that “tomorrow is going to rain”! Then, surprisingly, it is
not going to mis-predict one single rainy day! That means: Recall =
100%! Should I conclude that this is a perfect device? No, I should ask
for precision now.
Let me explain this with a real life example.
*You are chatting with your friend and your mom
calls*
Mom - Bring some items from the shop. Guests are
coming in the evening.
You - Okay mom, tell me what to bring.
*10 minutes later *
*On shop*
As you were busy in “chatting” you couldn’t
remember the items your mom asked you to bring.
So you try to recall it in your memory (trying to
remember). And finally, you remembered everything,
you bought the items from the shop and went to
your way home thinking “today i didn’t make any
mistakes. I brought everything my mom asked
properly.”
*10 minutes later*
*At home*
Your mom asked you to bring 4 items. You
brought 6 items (4 items your mom asked you to
bring + 2 extra items).
I mean yeah. You brought everything your mom
asked you to bring. So there is no problem right??
But no, you wasted extra money on the things she
didn’t need. (Irrelevant things)
In this example recall is 100% because you brought
all the 4 out of 4 items your mom asked to bring.
What is the precision here?? Isn’t it the same thing? I
brought everything my mom asked me to
right?? Precision should be 100%, right??
No, precision here basically means what are things
you brought that was actually needed. (Relevant
result)
Yeah only 4 of the 6 items were actually needed or
were relevant to mom.
Your precision is 66.66 % (4/6 * 100).