EC 6430 Pattern Recognition and Analysis

National Institute of Technology Calicut

Monsoon 2011
Lecture Notes - 9

ROC - Reciver Operating Characteristic

Consider a two-class prediction problem (binary classication), in which the

outcomes are labeled either as positive (p ) or negative (n ) class. There are four
possible outcomes from a binary classier.

1. If the outcome from a prediction is p and the actual value is also p, then
it is called a true positive (TP);

2. however if the actual value is n then it is said to be a false positive (FP).

3. Conversely, a true negative (TN) has occurred when both the prediction
outcome and the actual value are n,

4. and false negative (FN) is when the prediction outcome is n while the
actual value is p.

A receiver operating characteristic (ROC), or simply ROC curve, is a graphical

plot of the true positive rate vs. false positive rate. Where,

F P (negatives incorrectly classif ied)

f prate = (1)
N (T otal negatives)
T P (P ositives correctly classif ied)
tprate = (2)
P (T otal positives)
TP + TN
accuracy = (3)
P +N

Multiple Classes

Consider a 3-class (A, B, C) problem. The FP value can be obtained by looking

at the number of (class B, C classied as A + class A, C classied as B + class
A, B classied as C). The TP value can be obtained by, number of classes A,
B, C classied as A, B, C respectively.

NA = total B + total C (4)

NB = total A + total C (5)

1
 NC = total A + total B (6)

F PA F PB F PC
f prate = + + (7)
NA NB NC
and similarly we obtain the true positive rate as,

T PA T PB T PC
tprate = + + (8)
PA PB PC

OK, but how does this look?

Figure 1: ROC example from wiki

But, how did I get all these multiple values?

From our experience till now, we know that a classier would have a xed
accuracy value. So then how did we generate all these values for the ROC?

2
Note that ROC has only a small relation with the accuracy, we are purely
looking at True Positives and False Positives. We can set up a threshold for
classier boundary between classes and vary it between +∞ and −∞.
Now if we were to look at the FP and TP values for each of these threshold
values, we would obtain multiple points.

Reading this

Informally, one point in ROC space is better than another if it is to the north-
west (tp rate is higher, fp rate is lower, or both) of the rst. Classiers appearing
on the left hand-side of an ROC graph, near the X axis, may be thought of as
 conservative : they make positive classications only with strong evidence so
they make few false positive errors, but they often have low true positive rates as
well. Classiers on the upper right-hand side of an ROC graph may be thought
of as  liberal : they make positive classications with weak evidence so they
classify nearly all positives correctly, but they often have high false positive
rates [1].

References

[1] ROC Graphs: Notes and Practical Considerations for Researchers

Tom Fawcett (tom.fawcett@hp.com) HP Laboratories, MS 1143,
1501 Page Mill Road, Palo Alto, CA 94304