INDIAN INSTITUTE OF TECHNOLOGY ROORKEE

CSN-373: Probability Theory for Computer Engineers

Lecture 5: Bayes’ Theorem, Reliability of a System

Dr. Sudip Roy (a.k.a., SR)

Department of Computer Science & Engineering
Outline of Module 1:

● Concept of probability
● Random variables
● Distribution functions: discrete and continuous
● Moments and moment generating functions

2
Rules of Probability Recap:

3
Joint Probability:

• For events A and B, joint probability P(AB) stands for the

probability that both events happen.
• Example: A={HH}, B={HT, TH}, what is the joint probability P(AB)?

4
Bayes’ Theorem:

The Rev Thomas Bayes

The multiplication rule gives (1702-1761)

Bayes’ Theorem

𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝐵 𝑃 𝐵 = 𝑃 𝐵 𝐴 𝑃(𝐴)

Note: as in the example, the Total Probability rule is often used to evaluate P(B):

𝑃 𝐵𝐴 𝑃 𝐴
𝑃 𝐴 𝐵) =
∑ 𝑃 𝐵|𝐴 𝑃(𝐴 )
𝑃 𝐴 and 𝐵
=
𝑃 𝐴 and 𝐵 𝑃 𝐴 and 𝐵 𝑃 𝐴 and 𝐵 ⋯

If you have a model that tells you how likely B is given A, Bayes’ theorem allows you
to calculate the probability of A if you observe B. This is the key to learning about
your model from statistical data.
5
Baye’s Theorem:

● Let A1, A2,...,Ak be events such that P(Ai) > 0, i = 1, 2,...,k. Assume
further that A1, A2,...,Ak form a partition of the sample space C. Let B be
any event. Then

● Proof: Based on the definition of conditional probability, we have

● The result then follows by the law of total probability.

● This theorem is the well-known Bayes’ Theorem.
● This permits us to calculate the conditional probability of Aj , given B,
from the probabilities of A1, A2,...,Ak and the conditional probabilities of
B, given Ai, i = 1, 2,...,k.

6
Bayes’ Theorem:
• Example: Evidence in court
The cars in a city are 90% black and 10% grey.
A witness to a bank robbery briefly sees the escape car, and says it is grey. Testing the
witness under similar conditions shows the witness correctly identifies the colour 80% of the
time (in either direction).
What is the probability that the escape car was actually grey?

• Answer:

Let G = car is grey, B=car is black, W = Witness says car is grey.

Know 𝑃 𝑊 𝐺 want 𝑃(𝐺|𝑊)
𝑃 𝑊∩𝐺 𝑃 𝑊𝐺 𝑃 𝐺
Bayes’ Theorem 𝑃 𝐺𝑊 = = .
𝑃 𝑊 𝑃 𝑊
Use total probability rule to write

𝑃 𝑊 =𝑃 𝑊𝐺 𝑃 𝐺 𝑃 𝑊𝐵 𝑃 𝐵 = 0.8 0.1 0.2 0.9 = 0.26

𝑃 𝑊𝐺 𝑃 𝐺 0.8 0.1
Hence: 𝑃 𝐺 𝑊 = = 0.31
𝑃 𝑊 0.26
7
Bayes’ Theorem:

• Example: Failing a drugs test

A drugs test for athletes is 99% reliable: applied to a drug taker it gives a positive
result 99% of the time, given to a non-taker it gives a negative result 99% of the
time. It is estimated that 1% of athletes take drugs.
A random athlete is tested and gives a positive result. What is the probability the
athlete takes drugs?
• Answer:
Let F=“fails test” (a drug taker gets +ve result)
Let D=“takes drugs”
Question tells us
𝑃 𝐷 = 0.01, 𝑃(𝐹|𝐷) = 0.99, 𝑃 𝐹 𝐷 = 0.01
𝑃 𝐹𝐷 𝑃 𝐷
Bayes’ Theorem gives 𝑃 𝐷𝐹 =
𝑃 𝐹

We need 𝑃 𝐹 = 𝑃 𝐹 𝐷 𝑃 𝐷 𝑃 𝐹𝐷 𝑃 𝐷 = 0.99 0.01 0.01 0.99

= 0.0198

𝑃 𝐹𝐷 𝑃 𝐷 0.99 0.01 0.0099 1

Hence: 𝑃 𝐷𝐹 = = = =
𝑃 𝐹 0.0198 0.0198 2
8
Bayes’ Rule:

• Given two events A and B and suppose that P(A) > 0. Then
P( 𝐴𝐵) P( 𝐴|𝐵) P( 𝐵)
P( 𝐵|𝐴) = =
P( 𝐴) P( 𝐴)

• Example:

P(R) = 0.8
R: It is a rainy day
P(W|R) R ¬R
W: The grass is wet
W 0.7 0.4 P(R|W) = ?
¬W 0.3 0.6

9
Bayes’ Rule:

R ¬R
R: It rains
W 0.7 0.4
W: The grass is wet
¬W 0.3 0.6

Information
P(W|R)
R W

Inference
P(R|W)

10
Bayes’ Rule:

R ¬R
R: It rains
W 0.7 0.4
W: The grass is wet
¬W 0.3 0.6

Information: P(E|H)
Hypothesis H Evidence E
Posterior Likelihood
Inference: Pr(H|E)
Prior

11
Bayes’ Rule: More Complicated

Suppose that B1, B2, … Bk form a partition of S:

Bi  B j = ∅; i Bi = S
Suppose that P(Bi) > 0 and P(A) > 0. Then

P( 𝐴|𝐵 ) P( 𝐵 )
P( 𝐵 |𝐴) =
P( 𝐴)
P( 𝐴|𝐵 ) P( 𝐵 )
=
∑ P( 𝐴𝐵 )
P( 𝐴|𝐵 ) P( 𝐵 )
=
∑ P( 𝐵 ) P( 𝐴|𝐵 )

12
A More Complicated Example:

R It rains
R
W The grass is wet

U People bring umbrella

W U
P(UW|R)=P(U|R)P(W|R)
P(R) = 0.8 P(UW| ¬R)=P(U| ¬R)P(W| ¬R)

P(W|R) R ¬R P(U|R) R ¬R
W 0.7 0.4 U 0.9 0.2

¬W 0.3 0.6 ¬U 0.1 0.8

P(U|W) = ?
13
Reliability of a System:

• General approach: bottom-up analysis. Need to break down the system into
subsystems just containing elements in series or just containing elements in
parallel.
• Find the reliability of each of these subsystems and then repeat the process at
the next level up.

14
Reliability of a System:
Series subsystem: in the diagram 𝑝 = probability that element i fails, so 1 𝑝=
probability that it does not fail.

p p p p
1 2 3 n

The system only works if all n elements work. Failures of different elements are
assumed to be independent (so the probability of Element 1 failing does alter after
connection to the system).

𝑃 𝑠𝑦𝑠𝑡𝑒𝑚 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑓𝑎𝑖𝑙 = 𝑃(1 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑓𝑎𝑖𝑙 𝐴𝑁𝐷 2 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑓𝑎𝑖𝑙 𝐴𝑁𝐷 … 𝑛 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑓𝑎𝑖𝑙)

= 1 𝑝 1 𝑝 … 1 𝑝 = (1 𝑝)

Hence 𝑃 𝑠𝑦𝑠𝑡𝑒𝑚 𝑑𝑜𝑒𝑠 𝑓𝑎𝑖𝑙 = 1 𝑃(𝑠𝑦𝑠𝑡𝑒𝑚 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑓𝑎𝑖𝑙)

=1 (1 𝑝)
15
Reliability of a System:

Parallel subsystem: the subsystem only fails if all the elements fail.

p
1

p
2

p
n

𝑃 𝑠𝑦𝑠𝑡𝑒𝑚 𝑓𝑎𝑖𝑙𝑠 = 𝑃(1 𝑓𝑎𝑖𝑙𝑠 𝐴𝑁𝐷 2 𝑓𝑎𝑖𝑙𝑠 𝐴𝑁𝐷 … 𝑛 𝑓𝑎𝑖𝑙𝑠)

[Special multiplication rule
= 𝑃 1 𝑓𝑎𝑖𝑙𝑠 𝑃 2 𝑓𝑎𝑖𝑙𝑠 … 𝑃(𝑛 𝑓𝑎𝑖𝑙𝑠) assuming failures independent]

= 𝑝 𝑝 …𝑝 = 𝑝

16
Example:

Subsystem 3:
P(Subsystem 3 fails) =
0.1 x 0.1 = 0.01

Subsystem 1: Subsystem 2: (two units of subsystem 1)

P(Subsystem 1 doesn't fail) 0.0785 P(Subsystem 2 fails) =

= 1 0.05 1 0.03 = 0.9215 0.0785 x 0.0785 =
0.0785 0.006162
Hence P(Subsystem 1 fails)= 0.0785

Answer:
0.02 0.006162 0.01 P(System doesn't fail) =
(1 - 0.02)(1 - 0.006162)(1 - 0.01) =
0.964
Answer to (b)

Let B = event that the system does not fail

Let C = event that component * does fail

We need to find P(B and C).

Use 𝑃 𝐵 ∩ 𝐶 = 𝑃(𝐵|𝐶)𝑃 𝐶 . We know P(C) = 0.1.

P(B | C) = P(system does not fail given component * has failed)

If * failed replace with

Final diagram is then 0.02 0.006162 0.1

P(B | C) = (1 - 0.02)(1 – 0.006162)(1 - 0.1) = 0.8766

Hence since P(C) = 0.1

P(B and C) = P(B | C) P(C) = 0.8766 x 0.1 = 0.08766

Triple Redundancy:

1
What is probability that this system does not fail,
3
given the failure probabilities of the components?
1
3

1
2

P(failing) = P(1 fails)P(2 fails)P(3 fails)= =

Hence: P(not failing) = 1 – P(failing) = 1 =

20
Next Class…

21