Overview

◮ Exercise 12
Exercises in Machine Learning ◮ Exercise 13
Dr. Thomas Gabel

Albert-Ludwigs-Universität Freiburg
Lehrstuhl für Maschinelles Lernen und natürlichsprachliche Systeme

Exercises in Machine Learning — page 1/29 Exercises in Machine Learning — page 2/29

Exercise 12: Probabilities Exercise 12: Probabilities

We consider a medical diagnosis task. We have knowledge that over the entire (a) Suppose we observe a patient for whom the laboratory test returns a positive
population of people 0.8% have cancer. There exists a (binary) laboratory test result. Calculate the a posteriori probability that this patient truly suffers from
that represents an imperfect indicator of this disease. That test returns a correct cancer.
positive result in 98% of the cases in which the disease is present, and a correct
negative results in 97% of the cases where the disease is not present. ◮ We know P (cancer) = 0.008 and P (¬cancer) = 0.992.
◮ For the laboratory test, we know that
(a) Suppose we observe a patient for whom the laboratory test returns a positive P (⊕|cancer) = 0.98 ⇒ P (⊖|cancer) = 0.02
result. Calculate the a posteriori probability that this patient truly suffers from P (⊕|¬cancer) = 0.03 ⇒ P (⊖|¬cancer) = 0.97
cancer.
◮ We are looking for the maximum a posteriori (MAP) hypothesis after the
(b) Knowing that the lab test is an imperfect one, a second test (which is laboratory test.
assumed to be independent of the former one) is conducted. Calculate the a
posteriori probabilities for cancer and ¬cancer given that the second test
◮ We obtain
has returned a positive result as well.
P (⊕|cancer) · P (cancer) = 0.98 · 0.008 = 0.0078
P (⊕|¬cancer) · P (¬cancer) = 0.03 · 0.992 = 0.0298

Exercises in Machine Learning — page 3/29 Exercises in Machine Learning — page 4/29
Exercise 12: Probabilities Exercise 12: Probabilities

(a) Suppose we observe a patient for whom the laboratory test returns a positive (b) Knowing that the lab test is an imperfect one, a second test (which is
result. Calculate the a posteriori probability that this patient truly suffers from assumed to be independent of the former one) is conducted. Calculate the a
cancer. posteriori probabilities for cancer and ¬cancer given that the second test
has returned a positive result as well.
◮ Thus, hM AP = arg maxh∈H P (D|h) · P (h) = ¬cancer .
◮ In particular, we get (after normalization) ◮ We know P (cancer) = 0.008 and P (¬cancer) = 0.992.
0.0078
P (cancer|⊕) = 0.0078+0.0298 = 0.21 and P (¬cancer|⊕) = 0.79 ◮ For the laboratory test, we know that
P (⊕|cancer) = 0.98 ⇒ P (⊖|cancer) = 0.02
P (⊕|¬cancer) = 0.03 ⇒ P (⊖|¬cancer) = 0.97
◮ We are looking for the maximum a posteriori (MAP) hypothesis after the
second laboratory test which is assumed to be independent of the former
one.

Exercises in Machine Learning — page 5/29 Exercises in Machine Learning — page 6/29

Exercise 12: Probabilities Exercise 12: Probabilities

(b) Knowing that the lab test is an imperfect one, a second test (which is We turn to politics. For the upcoming mayor election, 1000000 people are allowed
assumed to be independent of the former one) is conducted. Calculate the a to vote either for candidate A or candidate B. There were 1000 registered voters
posteriori probabilities for cancer and ¬cancer given that the second test who have already voted by postal voting.
has returned a positive result as well. (c) Assume that all postal voters have voted for candidate A. Moreover, we
assume that all remaining voters decide by flipping a (non-manipulated) coin.
◮ Therefore, we have
What is the probability that candidate A wins the election?
P (⊕ ⊕ |cancer) · P (cancer) = P (⊕|cancer) · P (⊕|cancer) ·
P (cancer) = 0.98 · 0.98 · 0.008 = 0.007644 ◮ It’s higher than most people would guess. ;-)
P (⊕ ⊕ |¬cancer) · P (¬cancer) = P (⊕|¬cancer) · P (⊕|¬cancer) · → Blackboard
P (¬cancer) = 0.03 · 0.03 · 0.992 = 0.000894
◮ Thus, hM AP = arg maxh∈H P (D|h) · P (h) = cancer .
◮ In particular, we get (after normalization)
0.007644
P (cancer| ⊕ ⊕) = 0.007644+0.000894 = 0.895 and
P (¬cancer| ⊕ ⊕) = 0.105.

Exercises in Machine Learning — page 7/29 Exercises in Machine Learning — page 8/29
Overview Exercise 13: Naive Bayes Classifier

◮ Exercise 12 In the following, we consider the data set introduced in Assignment 1 where the
◮ Exercise 13 task is to describe whether a person is ill. We use a representation based on four
features per subject to describe an individual person. These features are “running
nose”, “coughing”, “reddened skin”, and “fever”, each of which can take the value
true (‘+’) or false (‘–’).

Training N C R F Classification

Example (running nose) (coughing) (reddened skin) (fever)

d1 + + + – positive (ill)

d2 + + – – positive (ill)

d3 – – + + positive (ill)

d4 + – – – negative (healthy)

d5 – – – – negative (healthy)

d6 – + + – negative (healthy)

Exercises in Machine Learning — page 9/29 Exercises in Machine Learning — page 10/29

Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(a) Given the data set in the table on the previous slide, determine all ◮ P (a1 , . . . , an |vj ) is hard to assess which is why the naive Bayes classifier
probabilities required to apply the naive Bayes classifier for predicting whether assumes that the attributes are independent of one another, i.e.
a new person is ill or not. Use the m-estimate of probability with an equivalent
sample size m = 4 and a uniform prior p. P (a1 , . . . , an |vj ) = Πni=1 p(ai |vj )

◮ We describe instances as tuples di = ha1 , . . . , an i. ◮ Thus, in contrast to the maximuma a-posteriori estimate, the naive Bayes
◮ In our case it holds n = 4 and V = {ill, healthy}. estimate is given by

◮ In the end, the naive Bayes classifier searches for that vj ∈ V such that vN B = arg max p(vj ) · Πni=1 P (ai |vj )
vj ∈V
vM AP = arg max P (vj |a1 , . . . , an )
vj ∈V ◮ With the data given and V = {ill, healthy} we have
P (ill) = P (healthy) = 36 .
◮ Using the Bayes rule this gives rise to

P (a1 , . . . , an |vj ) · P (vj )

vM AP = arg max
vj ∈V P (a1 , . . . , an )

Exercises in Machine Learning — page 11/29 Exercises in Machine Learning — page 12/29
Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

◮ Accordingly, for the conditional probabilities for the presence of the four ◮ We obtain:
different symptoms we obtain the following. ◮ running Nose (N): P (N |ill) = 2+m·0.5 = 47 ,
3+m
2 1
◮ running Nose (N): P (N |ill) = 3
, P (N |healthy) = 3 P (N |healthy) = 1+m·0.5 = 37
3+m
◮ Coughing (C): P (C|ill) = 23 , P (C|healthy) = 1
3 ◮ Coughing (C): P (C|ill) = 47 , P (C|healthy) = 3
7
◮ Reddened skin (R): P (R|ill) = 23 , P (R|healthy) = 1
3 ◮ Reddened skin (R): P (R|ill) = 47 , P (R|healthy) = 3
7
◮ Fever (F): P (F |ill) = 13 , P (F |healthy) = 0
3 ◮ Fever (F): P (F |ill) = 37 , P (F |healthy) = 2
7
◮ Next, we make use of the m-estimate of probability using an equivalent
sample size m = 4 and a uniform prior p.
◮ This, of course, changes all probabilities recently computed.

Exercises in Machine Learning — page 13/29 Exercises in Machine Learning — page 14/29

Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(b) Verify whether the naive Bayes classifier classifies all training examples (b) Verify whether the naive Bayes classifier classifies all training examples
(d1 , . . . , d6 ) correctly. (d1 , . . . , d6 ) correctly.

◮ Consider d1 = hN, C, R, F i. ◮ Consider d2 = hN, C, R, F i.

• P (ill) = P (healthy) = 12 • P (ill) = P (healthy) = 12
4
• P (N, C, R, F |ill) = 7
· 47 · 74 ·
4
7
= 256
74
• P (N, C, R, F |ill) = 4
7
· 47 · 73 ·
4
7
= 16·12
74
3 3 3 5
• P (N, C, R, F |healthy) = · · · = 135
7 7 7 7 74
• P (N, C, R, F |healthy) = 3 3 4 5
· · · = 15·12
7 7 7 7 74
n n
• vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill • vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill
⇒ correctly classified ⇒ correctly classified

Exercises in Machine Learning — page 15/29 Exercises in Machine Learning — page 16/29
Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(b) Verify whether the naive Bayes classifier classifies all training examples (b) Verify whether the naive Bayes classifier classifies all training examples
(d1 , . . . , d6 ) correctly. (d1 , . . . , d6 ) correctly.

◮ Consider d3 = hN , C, R, F i. ◮ Consider d4 = hN, C, R, F i.

• P (ill) = P (healthy) = 12 • P (ill) = P (healthy) = 12
3
• P (N , C, R, F |ill) = 7
· 37 · 47 ·
3
7
= 108
74
• P (N, C, R, F |ill) = 4
7
· 37 · 37 ·
4
7
= 12·12
74
4 4 3 2
• P (N , C, R, F |healthy) = · · · = 96
7 7 7 7 74
• P (N, C, R, F |healthy) = 3 4 4 5
· · · = 12·20
7 7 7 7 74
n n
• vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill • vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = healthy
⇒ correctly classified ⇒ correctly classified

Exercises in Machine Learning — page 17/29 Exercises in Machine Learning — page 18/29

Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(b) Verify whether the naive Bayes classifier classifies all training examples (b) Verify whether the naive Bayes classifier classifies all training examples
(d1 , . . . , d6 ) correctly. (d1 , . . . , d6 ) correctly.

◮ Consider d5 = hN , C, R, F i. ◮ Consider d6 = hN , C, R, F i.
• P (ill) = P (healthy) = 12 • P (ill) = P (healthy) = 12
3
• P (N , C, R, F |ill) = 7
· 37 · 37 ·
4
7
= 9·12
74
• P (N , C, R, F |ill) = 3
7
· 47 · 47 ·
4
7
= 12·16
74
4 4 4 5
• P (N , C, R, F |healthy) = · · · = 16·20
7 7 7 7 74
• P (N , C, R, F |healthy) = 4 3 3 5
· · · = 12·15
7 7 7 7 74
n n
• vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = healthy • vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill
⇒ correctly classified ⇒ wrongly classified

Exercises in Machine Learning — page 19/29 Exercises in Machine Learning — page 20/29
Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(c) Apply your naive Bayes classifier to the test patterns corresponding to the (c) Apply your naive Bayes classifier to the test patterns corresponding to the
following subjects: a person who is coughing and has fever, a person whose following subjects: a person who is coughing and has fever, a person whose
nose is running and who suffers from fever, and a person with a running nose nose is running and who suffers from fever, and a person with a running nose
and reddened skin (d7 = (N , C, R, F ), d8 = (N, C, R, F ), and and reddened skin (d7 = (N , C, R, F ), d8 = (N, C, R, F ), and
d9 = (N, C, R, F )). d9 = (N, C, R, F )).

◮ Consider d7 = hN , C, R, F i. ◮ Consider d8 = hN, C, R, F i.

• P (ill) = P (healthy) = 12 • P (ill) = P (healthy) = 12
3
• P (N , C, R, F |ill) = 7
· 47 · 37 ·
3
7
= 12·9
74
• P (N, C, R, F |ill) = 4
7
· 37 · 37 ·
3
7
= 12·9
74
4 3 4 2
• P (N , C, R, F |healthy) = · · · = 12·8
7 7 7 7 74
• P (N, C, R, F |healthy) = 3 4 4 2
· · · = 12·8
7 7 7 7 74
n n
• vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill • vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill
⇒ A person that is coughing and has fever is classified to be ill. ⇒ A person that has a running nose as well as fever is classified to be ill.

Exercises in Machine Learning — page 21/29 Exercises in Machine Learning — page 22/29

Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

(c) Apply your naive Bayes classifier to the test patterns corresponding to the (d) Now, we no longer distinguish between positive and negative training
following subjects: a person who is coughing and has fever, a person whose examples, but each instance is assigned one out of k classes. The
nose is running and who suffers from fever, and a person with a running nose corresponding training data is provided in the table below. Calculate all
and reddened skin (d7 = (N , C, R, F ), d8 = (N, C, R, F ), and probabilities required for the application of a naive Bayes classifier, i.e. P (v)
d9 = (N, C, R, F )). and P ((aF , aV , aD , aSh )|v) for v ∈ {H, I, S, B} and
aF ∈ {no, average, high} and aV , aD , aSh ∈ {yes, no}. Again, use the
◮ Consider d9 = hN, C, R, F i. m-estimate of probability method with m = 6 and p uniform.
• P (ill) = P (healthy) = 12
4 Training Fever Vomiting Diarrhea Shivering Classification
• P (N, C, R, F |ill) = 7
· 37 · 47 ·
4
7
= 12·16
74 d1 no no no no healty (H)
3 4 3 5
• P (N, C, R, F |healthy) = · · · = 12·15
7 7 7 7 74 d2 average no no no influenza (I)
n
• vN B = arg maxvj ∈V P (vj ) · Πi=1 P (ai |vj ) = ill d3 high no no yes influenza (I)

⇒ A person that has a running nose as well as reddened skin is d4 high yes yes no salmonella poisoning (S)

d5 average no yes no salmonella poisoning (S)

classified to be ill.
d6 no yes yes no bowel inflammation (B)

d7 average yes yes no bowel inflammation (B)

Exercises in Machine Learning — page 23/29 Exercises in Machine Learning — page 24/29
Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

◮ Conditional probabilities for the Vomiting attribute using the m-estimate of

Training Fever Vomiting Diarrhea Shivering Classification
probability (continued):
d1 no no no no healty (H) 0+3
P (V |H) = 1+6
= 37 , P (V |H) = 1+31+6
= 47 ,
d2 average no no no influenza (I)

d3 high no no yes influenza (I) P (V |I) = 0+3

2+6
= 38 , P (V |I) = 2+3
2+6
= 58 ,
d4 high yes yes no salmonella poisoning (S) P (V |S) = 1+3
2+6
= 48 , P (V |S) = 1+3
2+6
= 48 ,
d5 average no yes no salmonella poisoning (S)
P (V |B) = 2+3
2+6
= 58 , P (V |B) = 0+3
2+6
= 38
d6 no yes yes no bowel inflammation (B)

d7 average yes yes no bowel inflammation (B) ◮ Conditional probabilities for the Diarrhea attribute using the m-estimate of
probability:
0+3
◮ Obviously, it holds P (H) = 17 , P (I) = 27 , P (S) = 27 , and P (B) = 27 . P (D|H) = 1+6
= 37 , P (D|H) = 1+31+6
= 47 ,
◮ Conditional probabilities for the Vomiting attribute using the m-estimate of P (D|I) = 0+3
2+6
= 38 , P (D|I) = 2+3
2+6
= 58 ,
probability: P (D|S) = 2+3
2+6
= 58 , P (D|S) = 2+6
0+3
= 38 ,
0+3
P (V |H) = 1+6
= 37 , P (V |H) = 1+3
1+6
= 47 , P (D|B) = 2+3
2+6
= 58 , P (D|B) = 0+3
2+6
= 38
0+3
P (V |I) = 2+6
= 83 , P (V |I) = 2+3
2+6
= 58 ,

Exercises in Machine Learning — page 25/29 Exercises in Machine Learning — page 26/29

Exercise 13: Naive Bayes Classifier Exercise 13: Naive Bayes Classifier

◮ Conditional probabilities for the Shivering attribute using the m-estimate of (e) Apply your recently constructed naive Bayes classifier to a person with high
probability: fever, i.e. to (high, no, no, no), as well as to a person who suffers from
0+3
P (Sh|H) = 1+6
= 37 , P (Sh|H) = 1+31+6
= 47 , vomitting and shivering, i.e. to (no, yes, no, yes).
P (Sh|I) = 1+3 = 48 , P (Sh|I) = 1+3 = 48 ,
2+6 2+6 ◮ Consider d = (high, no, no, no), i.e. hFhigh , V , D, Shi.
P (Sh|S) = 0+3 = 38 , P (Sh|S) = 2+3 = 58 ,
2+6 2+6 ◮ Calculate P (v|x) = P (v) · Π4i=1 P (ai |v) for all v ∈ {H, I, S, B}.
P (Sh|B) = 0+3
2+6
= 38 , P (Sh|B) = 2+3
2+6
= 58 1
◮ P (H) · Π4i=1 P (ai |H) = 7
· 72 · 47 · 47 · 4
7
= 1
8·16
74
7
·
≈ 0.0076
◮ Conditional probabilities for the Fever attribute using the m-estimate of
probability: ◮ P (I) · Π4i=1 P (ai |I) = 27 · 38 · 58 · 58 · 48 = 27 · 15·20
84
≈ 0.0209
1+2
P (Fno |H) = = 37 , P (Favg |H) = 0+2 = 27 , P (Fhigh |H) = 0+2 = 27 ,
1+6 1+6 1+6 ◮ P (S) · Π4i=1 P (ai |S) = 27 · 38 · 48 · 38 · 58 = 27 · 9·20
84
≈ 0.0126
P (Fno |I) = 0+2
2+6
= 2
8
, P (F avg |I) = 1+2
2+6
= 3
8
, P (F high |I) = 1+2
2+6
= 3
8
,
2 3 3 3 5 2 9·15
0+2 1+2 1+2
P (Fno |S) = 2+6 = 8 , P (Favg |S) = 2+6 = 8 , P (Fhigh |I) = 2+6 = 38 ,
2 3
◮ P (B) · Π4i=1 P (ai |B) = 7 · 8 · 8 · 8 · 8 = 7 · 84 ≈ 0.0094
P (Fno |B) = 1+2 = 38 , P (Favg |B) = 0+2 = 28 , P (Fhigh |I) = 1+2 = 38
2+6 2+6 2+6
◮ Thus, after normalization, the naive Bayes classifier concludes that person d
is healthy with a probability of 15.0%, suffers from influenza with 41.4%,
from salmonella poisoning with 25.0%, and from bowel inflammation with
18.6%.
Exercises in Machine Learning — page 27/29 Exercises in Machine Learning — page 28/29
Exercise 13: Naive Bayes Classifier

(e) Apply your recently constructed naive Bayes classifier to a person with high
fever, i.e. to (high, no, no, no), as well as to a person who suffers from
vomitting and shivering, i.e. to (no, yes, no, yes).

◮ Consider d = (no, yes, no, yes), i.e. hFno , V, D, Shi.

◮ Calculate P (v|x) = P (v) · Π4i=1 P (ai |v) for all v ∈ {H, I, S, B}.
1
◮ P (H) · Π4i=1 P (ai |H) = 7
· 37 · 37 · 47 · 3
7
= 1
9·12
74
7
·
≈ 0.0064
◮ P (I) · Π4i=1 P (ai |I) = 27 · 28 · 38 · 58 · 48 = 27 · 12·10
84
≈ 0.0084
◮ P (S) · Π4i=1 P (ai |S) = 27 · 82 · 48 · 38 · 38 = 27 · 12·6
84
≈ 0.0050
◮ P (B) · Π4i=1 P (ai |B) = 27 · 38 · 58 · 38 · 38 = 27 · 9·15
84
≈ 0.0094
◮ Thus, after normalization, the naive Bayes classifier concludes that person d
is healthy with a probability of 21.9%, suffers from influenza with 28.8%,
from salmonella poisoning with 17.1%, and from bowel inflammation with
32.2%.
Exercises in Machine Learning — page 29/29