EEF 271E

PROBABILITY AND STATISTICS

FALL 2022

Lecture 03
Conditional Probability, Bayes' Theorem

Dr. Ramazan Çağlar

Lecture Notes 03 1
Conditional Probability, Bayes' Theorem
 Lesson Overview

Conditional Probability

• Conditional Probability
• Total Probability
• Bayes’ Theorem

2
Lecture Notes 03 Conditional Probability, Bayes' Theorem
 Lesson Overview

Conditional Probability

Let A, B  F.
Then the
‘‘conditional probability of A given B’’
denoted 𝑃 𝐴 𝐵 is defined as

𝑃 𝐴∩𝐵
𝑃 𝐴𝐵 = if 𝑃 𝐵 > 0
𝑃 𝐵
(It is undefined if 𝑃 𝐵 = 0)

3
Lecture Notes 03 Conditional Probability, Bayes' Theorem
 𝑃 𝐴𝐵 has the interpretation of the likelihood
of event A occurring, given that B has occurred.
Why?

Intuitive explanation 1:
If B occurs, the sample space
 is ‘‘reduced’’ to . Then A
AB
A B also occurs if and only if AB
occurs. We then renormalize
P(AB) by 1/P(B)
4
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Intuitive explanation 2:
(Using relative frequency approach)

Say an experiment is performed N times, and B

occurs NB times. Of this NB trials, A also occurs
NAB times. Then

𝑁𝐴∩𝐵
P(A occurs given that B occurs) ≅
𝑁𝐵

𝑁𝐴∩𝐵 𝑁𝐴∩𝐵 /𝑁 𝑁𝐴∩𝐵 /𝑁 𝑃(𝐴 ∩ 𝐵)

= → lim =
𝑁𝐵 𝑁𝐵 /𝑁 𝑁→∞ 𝑁𝐵 /𝑁 𝑃(𝐵)
5
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Intuitive explanation 3:

The evet AB can be thought of as

P(A ∩ 𝐵) = 𝑃(𝐵) ∙ 𝑃 𝐴 𝐵

‘’B occurs and ‘’ B occurs ’’ ‘’ A occurs given

A occurs ’’ that B occurs ’’

P(A ∩ 𝐵) = 𝑃(𝐵) ∙ 𝑃 𝐴 𝐵

Lecture Notes 03 Conditional Probability, Bayes' Theorem 6

 Special cases for A and B

(1) 𝑨 ∩ 𝑩 = ∅ (i.e., A and B are ‘’disjoint’’

or ‘’mutually exclusive’’)

Then 𝑷 𝑨 𝑩 = 𝟎 ( If P(B) > 0 )

(2) 𝑩 ⊂ 𝑨 If B occurs, A also occurs.

𝑷(𝑨 ∩ 𝑩) 𝑷(𝑩)
𝑷 𝑨𝑩 = = =𝟏
𝑷(𝑩) 𝑷(𝑩)
(3) 𝑨 ⊂ 𝑩 Then
𝑷(𝑨 ∩ 𝑩) 𝑷(𝑨)
𝑷 𝑨𝑩 = = ≥ 𝑷(𝑨)
𝑷(𝑩) 𝑷(𝑩)

Lecture Notes 03 Conditional Probability, Bayes' Theorem 7

 Check that conditional Probability satisfies
axioms of Probability

For any event 𝑩 ∈ 𝓕 with 𝑷(𝑩) > 𝟎,

(1) 𝑷 𝑨 𝑩 ≥ 𝟎 ? (For any A∈ 𝓕 ? ) (YES)

(2) 𝑷 𝛀 𝑩 = 𝟏 ? (YES, since 𝑩 ⊂ 𝜴)

(special case 2 on previous page)
(3) 𝑷 𝑨 ∪ 𝑪 |𝑩 = 𝑷 𝑨 𝑩 + 𝑷 𝑪 𝑩 If 𝑨 ∩ 𝑪 = ∅

Conditional Probability, Bayes' Theorem 8

Lecture Notes 03
 Check that conditional Probability satisfies
axioms of Probability

(3) 𝑷 𝑨 ∪ 𝑪 𝑩 = 𝑷 𝑨 𝑩 + 𝑷 𝑪 𝑩 If 𝑨 ∩ 𝑪 = ∅ ?
YES, 𝑨 ∪ 𝑪 ∩ 𝑩 = 𝑨 ∩ 𝑩 ∪ 𝑪 ∩ 𝑩
because 𝑨 ∩ 𝑩 and 𝑪 ∩ 𝑩 disjoint events

𝑷 (𝑨 ∪ 𝑪) ∩ 𝑩 𝑷 (𝑨 ∩ 𝑩) ∪ 𝑷(𝑪 ∩ 𝑩)
𝑷 𝑨∪𝑪 𝑩 = =
𝑷(𝑩) 𝑷(𝑩)
𝑷(𝑨 ∩ 𝑩) 𝑷(𝑪 ∩ 𝑩)
= + =𝑷 𝑨 𝑩 +𝑷 𝑪 𝑩
𝑷(𝑩) 𝑷(𝑩)

Lecture Notes 03 Conditional Probability, Bayes' Theorem 9

 Example 1:
Expt. Roll a fair die
Sample space: Ω = {1, 2, 3, 4, 5, 6}
Define A = 1, 2 , B = 2, 4, 6 ,
#𝐴 2 1
𝑃 𝐴 = = =
#Ω 6 3
(from Classical arguments)
#𝐵 3 1
𝑃 𝐵 = = =
#Ω 6 2

Lecture Notes 03 Conditional Probability, Bayes' Theorem 10

Example 1 (cont.): Expt. Roll a fair die

Sample space: Ω = {1, 2, 3, 4, 5, 6}

Define A = 1, 2 , B = 2, 4, 6 ,
1
A∩𝐵 = 2 , so P A ∩ 𝐵 =
6
𝑃(𝐴 ∩ 𝐵) 1/6 1
P A|𝐵 = = =
𝑃(𝐵) 1/2 3

i.e. probability of rolling 1 or 2, given that an even

number was gotten is 1/3.

Lecture Notes 03 11
Conditional Probability, Bayes' Theorem
Example 1 (cont.): Exp. Roll a fair die
Sample space: Ω = {1, 2, 3, 4, 5, 6}

Define A = 1, 2 , B = 2, 4, 6 ,
1
A∩𝐵 = 2 , so P A∩𝐵 =
6
Next
P(A ∩ B) 1/6 1
P BA = = =
P(A) 1/3 2
i.e. if the outcome was 1 or 2, the probability that
the outcome was given is 1/2.

Lecture Notes 03 12
Conditional Probability, Bayes' Theorem
MULTIPLICATION AND TOTAL PROBABILITY RULES

Multiplication Rule:
The definition of conditional probability
𝑃(𝐴 ∩ 𝐵)
𝑃 𝐵𝐴 = , 𝑓𝑜𝑟𝑃 𝐴 > 0
𝑃(𝐴)
can be rewritten to provide a general expression for the
probability of the intersection of two events.
P A ∩ 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴 = 𝑃 𝐴 𝐵 𝑃(𝐵)
This formula is referred to as a multiplication rule for
probabilities.

Lecture Notes 03 13
Conditional Probability, Bayes' Theorem
Total Probability Rule

The multiplication rule is useful for determining

the probability of an event that depends on
other events.

Lecture Notes 03 14
Conditional Probability, Bayes' Theorem
 Probability Level of Probability
Example 2: of Failure Contamination of Level
0.1 High 0.2
0.005 Not High 0.8

Suppose that in semiconductor manufacturing the probability is 0.10

that a chip that is subjected to high levels of contamination
during manufacturing causes a product failure.

The probability is 0.005 that a chip that is not subjected to high

contamination levels during manufacturing causes a product failure.

In a particular production run, 20% of the chips are subject to

high levels of contamination. What is the probability that a
product using one of these chips fails? (Find the probability of
failure.)
15
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Clearly, the requested probability depends on
whether or not
the chip was exposed to high Levels of contamination.

We can solve this problem by the following reasoning.

For any event
B, we can write B as the union of the part of B in A and
the part of B in A’ .

Lecture Notes 03 16
Conditional Probability, Bayes' Theorem
This result is shown in the Venn diagram Because A and A’ are
mutually exclusive, AB and A’B are mutually exclusive.
Therefore, from the probability of the union of mutually
exclusive events and the Multiplication Rule, the
following total probability rule is obtained.

Total Probability Rule (two events)

For any events A and B,
𝑷 𝑩 = 𝑷 𝑩 ∩ 𝑨 + 𝑷 𝑩 ∩ 𝑨′ = 𝑷 𝑩|𝑨 𝑷 𝑨 + 𝑷 𝑩|𝑨′ 𝑷(𝑨′ )

Lecture Notes 03 17
Conditional Probability, Bayes' Theorem
Solution of the Example 2 Probability Level of Probability
of Failure Contamination of Level
0.1 High 0.2
0.005 Not High 0.8

Let F denote the event that the product fails.

Let H denote the event that the chip is exposed to high

contamination during manufacture. Then

P(F|H)=0.100 and P(H)=0.2, so P(F∩H) = P(F|H).P(H) = 0.020

P(F|Hc)=0.005 and P(Hc)=0.8, so P(F∩Hc) = P(F|Hc).P(Hc) = 0.004

P(F) = P(F∩ H) + P(F∩ Hc) (Using Total Probability rule)

= 0.020 + 0.004 = 0.024

Lecture Notes 03 18
Conditional Probability, Bayes' Theorem
 Conditional probability and total probability

Suppose 𝐵1 , 𝐵2 , … , 𝐵𝑛 are mutually exclusive events

such that
𝐁𝟏 ∪ 𝐁𝟐 ∪ ⋯ ∪ 𝐁𝐧 = 𝛀
Then
𝐀 = 𝐀 ∩ 𝛀 = 𝐀 ∩ 𝐁 𝟏 ∪ 𝐁𝟐 ∪ ⋯ ∪ 𝐁𝐧

= 𝐀 ∩ 𝐁 𝟏 ∪ 𝐀 ∩ 𝐁 𝟐 ∪ ⋯ ∪ 𝐀 ∩ 𝐁𝐧
(disjoint union)

Lecture Notes 03 Conditional Probability, Bayes' Theorem 19

 Conditional probability and total probability

𝐀 = 𝐀 ∩ 𝛀 = 𝐀 ∩ 𝐁𝟏 ∪ 𝐁𝟐 ∪ ⋯ ∪ 𝐁𝐧

So

𝐏(𝐀) = 𝐏 𝐀 ∩ 𝐁𝟏 + 𝐏 𝐀 ∩ 𝐁𝟐 + ⋯ + 𝐏 𝐀 ∩ 𝐁𝐧

= 𝐏 𝐀 𝐁𝟏 𝐏 𝐁𝟏 + 𝐏 𝐀 𝐁𝟐 𝐏 𝐁𝟐 + ⋯ + 𝐏 𝐀 𝐁𝒏 𝐏 𝐁𝐧
𝒏

𝐏(𝐀) = ෍ 𝐏 𝐀 𝐁𝒊 𝐏 𝐁𝒊
𝒊=𝟏
‘’weighted sum of
‘’total probability’’
conditional probabilities’’

20
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Example 3:
Bin numbers
Ohms
1 2 3 Total
10 Ω 500 0 200 700
100 Ω 300 400 600 1300
1000 Ω 200 600 200 1000
Total Ω 1000 1000 1000 3000

Experiment:
A bin selected at random,
and a resistor from that bin is selected at random.
Let A = { Resistor has resistance 10 },
Find P(A)
Lecture Notes 03 21
Conditional Probability, Bayes' Theorem
Solution of Example 3:
Let 𝐁𝒊 be the event ‘’ ith bin is chosen’’
Then
1
P(𝐵𝑖 ) = , 𝑖 = 1, 2, 3
3
Next
500
P 𝐴 𝐵1 = = 0.5
1000
0
P 𝐴 𝐵2 = = 0.0
1000
200
P 𝐴 𝐵3 = = 0.2
1000
Lecture Notes 03 22
Conditional Probability, Bayes' Theorem
Solution of the Example 3 (continued):

Thus
P(A) = P A B1 P B1 + P A B2 P B2 + P A B3 P B3
1 1 1
= 0.5 + 0.0 + 0.2
3 3 3
7
=
30

Lecture Notes 03 Conditional Probability, Bayes' Theorem 23

 Solution of the example 3 (a shown in single page):
Let 𝐁𝒊 be the event ‘’ ith bin is chosen’’. Then
1
P B𝑖 = , 𝑖 = 1, 2, 3
3
Next
500
P A B1 = = 0.5
1000
0
P A B2 = = 0.0
1000
200
P A B3 = = 0.2
1000

Thus P(A) = P A B1 P B1 + P A B2 P B2 + P A B3 P B3
1 1 1 7
P A = 0.5 + 0.0 + 0.2 =
3 3 3 30

24
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Example 4:
Let us ask another question.
Suppose we are given that
the resistor drawn has
resistance 10 .
What is the Probability that it came from bin 1
(or bin 2, or bin 3).
In other word we want
P(B1|A) , P(B2|A) , P(B3|A)
These probabilities are also called ‘’a posteriori’’
probabilities (after the fact)
25
Lecture Notes 03 Conditional Probability, Bayes' Theorem
Recall that
𝑃 𝐵1 𝐴 𝑃(𝐴) = 𝑃(𝐵1 ∩ 𝐴)
(or in general 𝑃 𝐵𝑖 𝐴 𝑃(𝐴) = 𝑃(𝐵𝑖 ∩ 𝐴) ) so
𝑃(𝐵1 ∩ 𝐴)
𝑃 𝐵1 𝐴 = (𝑝𝑟𝑜𝑣𝑖𝑑𝑒𝑑 𝑃 𝐴 > 0)
𝑃(𝐴)
or in general
𝑃(𝐴|𝐵𝑖 )
𝑃 𝐵𝑖 𝐴 =
𝑃 𝐴 𝐵1 𝑃 𝐵1 + ⋯ + 𝑃 𝐴 𝐵𝑛 𝑃 𝐵𝑛

This is called BAYES’ THEOREM

Lecture Notes 03 26
Conditional Probability, Bayes' Theorem
Example 4 and its solution (cont.):
Suppose we are given that the
resistor drawn has resistance 10 .
What is the Probability that it came
from bin 1 (or bin 2, or bin 3).

Applying Bayes’ theorem to this example,

𝑃 𝐴 𝐵1 𝑃(𝐵1 ) (0.5) 1/3 5
𝑃 𝐵1 𝐴 = = =
𝑃(𝐴) (0.7) 1/3 7
𝑃 𝐴 𝐵2 𝑃(𝐵2 ) (0.0) 1/3
𝑃 𝐵2 𝐴 = = = 0.0
𝑃(𝐴) (0.7) 1/3
𝑃 𝐴 𝐵3 𝑃(𝐵3 ) (0.2) 1/3 2
𝑃 𝐵3 𝐴 = = =
𝑃(𝐴) (0.7) 1/3 7
Note that 𝑃 𝐵1 𝐴 + 𝑃 𝐵2 𝐴 + 𝑃 𝐵3 𝐴 = 1.0 (why?)

Lecture Notes 03 Conditional Probability, Bayes' Theorem 27

Example 5:
Suppose that an Integrated Supplier
Failure probability Number
Circuits (IC) is used in a P(F|supplier) ordered
device. The producer orders A 0,10 1000
B 0,05 2000
ICs from 3 different
C 0,08 3000
suppliers, named A, B and C
respectively.
The number of ordered IC’s and the conditional failure
probability are given in a table above.
Expt: ICs mixed together, one picked at random.
a) Find P(failure)
b) Given that the device fails, find P ({supplier was A}).

Lecture Notes 03 28
Conditional Probability, Bayes' Theorem
Solution of the Example 5:

Total Probability formula:

P(F) = P F A P A + P F B P B + P F C P C
#𝑁𝐴𝑖
From classical definition: P Ai = 𝑛
σ𝑖=1 #𝑁𝐴
𝑖
1000 2000 3000
P A = , P B = , P C =
6000 6000 6000
So
1 2 3
P F = 0.1 + 0.05 + 0.08 = 0.0733
6 6 6

Lecture Notes 03 29
Conditional Probability, Bayes' Theorem
b) Given that the device fails, find P ({supplier was A}).

So we want P A F

𝑃 𝐹 𝐴 𝑃(𝐴)
𝑃 𝐴𝐹 = (𝐵𝐴𝑌𝐸𝑆 ′ 𝑅𝑈𝐿𝐸)
𝑃(𝐹)
1
(0.1) 6
= = 0.2273
0.0733

Lecture Notes 03 30
Conditional Probability, Bayes' Theorem
Example 6:
Suppose the longevity of an Integrated Circuits
(IC) is described by:

𝑃 IC still function after t second of usage

= 𝑒 −𝛼𝑡 , 𝑡 > 0
(𝛼 > 0 is a parameter that is usually measured
empirically. This is called an “exponential law” )

Lecture Notes 03 31
Conditional Probability, Bayes' Theorem
Example 6 (cont.):

Suppose that an IC manufacturing process

produces both “good” and “bad” ICs with
probabilities (1-p) and p respectively.
(1-p) + p = 1.0
Let the lifetime of these IC follows the
exponential law with parameters
𝛼1 = 𝛼 and 𝛼2 = 1000𝛼 respectively.
(𝑒 −𝛼1𝑡 , and 𝑒 −𝛼2𝑡 )
Experiment: A chip is selected at random.

Lecture Notes 03 32
Conditional Probability, Bayes' Theorem
Example 6 (cont.):

a) Find the probability that IC still

function after t second usage.
Event A: IC chip still function after t seconds.
Event G: IC is a good chip
Event B: IC is a bad chip

𝑃 𝐴 =𝑃 𝐴 𝐺 𝑃 𝐺 +𝑃 𝐴 𝐵 𝑃 𝐵
= 𝑒 −𝛼𝑡 1 − 𝑝 + 𝑒 −(1000𝛼)𝑡 𝑝

Lecture Notes 03 33
Conditional Probability, Bayes' Theorem
Example 6 (cont.):

b) Suppose a chip is found to be still working

after t second; what is the probability that it is
a “good” chip?
𝑃 𝐺∩𝐴 𝑃 𝐴𝐺 𝑃 𝐺
𝑃 𝐺𝐴 = =
𝑃 𝐴 𝑃 𝐴
𝑒 −𝛼𝑡 1 − 𝑝
=
𝑒 −𝛼𝑡 1 − 𝑝 + 𝑒 −1000𝛼𝑡 𝑝
(Interpret this result for very small t, and very
large t... )

Lecture Notes 03 34
Conditional Probability, Bayes' Theorem
Example 7:
A producer wants to maximize its sale of special product
which influenced directly weather conditions. In this sense, it
must predict temperature and wind. Let's denote the event
"warm day" as event A and the event "windy day" as event B.
The weather forecast indicates that the probability of a
windy day is 0.80, the probability of a warm day is 0.30, and
the probability of a warm and windy day is 0.28
a) What is the probability of the event "windy day will be
warm"?
b) What is the probability of the event "warm day will be
windy"?
c) What is the probability of the event "windy and not
warm"?

Lecture Notes 03 35
Conditional Probability, Bayes' Theorem
Solution of the Example 7:
P A∩B 0.28
a) P AB = = = 0.35
P B 0.80
𝑃 𝐴∩𝐵 0.28
b) P BA = = = 0.93
𝑃 𝐴 0.30

c) Denote the event "not warm" as ഥA.

Then 𝑃 𝐵 = 𝑃 𝐴ҧ ∩ 𝐵 + 𝑃 𝐴 ∩ 𝐵 = 0.80
since A and 𝐴ҧ are complements. Therefore,
ഥ ∩ B = P B − P A ∩ B = 0.80 − 0.28 = 0.52
P A

Lecture Notes 03 36
Conditional Probability, Bayes' Theorem
Example 8:
A manufacturing firm puts into service three machines (A, B, C) to
produce the Integrated Circuits (ICs).
• 40% of the firm’s daily ICs production comes from machine A,
• 35% of the firm’s daily ICs production comes from machine B and,
• 25% of the firm’s daily ICs production comes from machine C.
• 1% of ICs produced on machine A are defective, while the defective
rates for machines B and C are 2% and 3% respectively.
• If one IC is selected at random from a day’s production and is found to
contain a defect,
a) What is the probability that the selected IC is found to contain a
defect?
b) What is the probability that it was produced on machine B?

Lecture Notes 03 37
Conditional Probability, Bayes' Theorem
Solution of the Example 8:
Let use the symbol D for the defective IC.

𝑃 𝐴 = 0.40, 𝑃 𝐵 = 0.35, 𝑃 𝐶 = 0.25

𝑃 𝑫 𝐴 = 0.01, 𝑃 𝑫 𝐵 = 0.02, 𝑃 𝑫 𝐶 = 0.03

a) 𝑃 𝑫 = 𝑃 𝐴 . 𝑃 𝑫 𝐴 + 𝑃 𝐵 . 𝑃 𝑫 𝐵 + 𝑃 𝐶 . 𝑃(𝑫|𝐶)
𝑃 𝑫 = 0.40 0.01 + 0.35 0.02 + 0.25 (0.03)
𝑃 𝑫 = 0.004 + 0.007 + 0.0075 = 0.0185

𝑃(𝐵).𝑃 𝑫𝐵 0.35 (0.02)

b) 𝑃 𝐵|𝑫 = = = 0.378
𝑃(𝑫) 0.0185

Lecture Notes 03 38
Conditional Probability, Bayes' Theorem
Example 9:

Consider the system below, and assume that each

component functions independently of the others with
probability p = 0.8.
What is the probability that the entire system works?

2 3
1 5
4

Lecture Notes 03 39
Conditional Probability, Bayes' Theorem
Answer for the example 9:

Since each component functions independently of the

others, the probability of the system success that is the
system functions to achieve their duty is given by following
equations.
C2 C3
C1 C5
C4

P(S) =P (the system functions) = P(C1).P[(C2C3)C4)].P(C5)

Let’s rename (C2C3) as D, and (DC4) as E then,

D
C1 =
C5 C1 E C5
C4

Lecture Notes 03 40
Conditional Probability, Bayes' Theorem
Answer for example 9 (continued):
• P(S) = P(C1)P(E)P(C5)
• P(E) = P(DC4)
• P(S) = P(C1)P(DC4)P(C5)
•P D = P C2 ∩ C3 = P C2 P C3 = 0.8 × 0.8 = 0.64
• P(E) = P(DC4) = P(D) + P(C4) – P(D ∩ C4)
•P E = P D + P C4 – P D P C4
= 0.64 + 0.8 – 0.64 × 0.8 = 1.44 – 0.512 = 0.928

• P(S) = P(C1)P(E)P(C5) = 0.8 × 0.928 × 0.8 = 0.512

Lecture Notes 03 41
Conditional Probability, Bayes' Theorem
Example 9:
Two persons have a date at a given time, and each will
arrive at the meeting place with a delay between 0 and 1
hour. The first to arrive will wait for 15 minutes and will
leave if the other has not yet arrived.
What is the Probability that they will meet?
Solution:
Let the persons be Romeo & Juliet.
Assume that they will meet at 0 o’clock.
Romeo delays for x minutes
Juliet delays for y minutes
If |x-y|< 15 minutes they will meet.

Lecture Notes 03 42
Conditional Probability, Bayes' Theorem
Solution:
Romeo delays for x minutes
Juliet delays for y minutes
If |x-y|< 15 minutes they will meet.
y
15 minutes 1/4 hours
1
M event

M = { (x, y) | |x-y| 1/4 , 0 x < 1, 0 y<1}

= [(0,1)x(0,1)] : square area ( Sample Space)

1/4

x
1/4 1

Lecture Notes 03 43
Conditional Probability, Bayes' Theorem
Solution:
Romeo delays for x minutes, Juliet delays for y minutes
If |x-y|< 15 minutes they will meet.
y
15 minutes 1/4 hours
1
M event

M = { (x, y) | |x-y| 1/4 , 0 x < 1, 0 y<1}

= [(0,1)x(0,1)] : square area ( Sample Space)

1/4

x
1/4 1

𝐴𝑟𝑒𝑎 𝑜𝑓 𝑀 7/16 7
P M = = =
𝐴𝑟𝑒𝑎 𝑜𝑠 𝑆 1 16

Lecture Notes 03 44
Conditional Probability, Bayes' Theorem