Basics of Probability

Theory
Background
 Games of chance and gambling in the 17 th
century sparked interests in probability theory

 A short history of probability:

http://homepages.wmich.edu/~mackey/Teaching/145/probHist.html
Probability Theory
 “Probability theory is nothing but
common sense reduced to calculation”.

Pierre-Simon Laplace in 1812

Probability Theory
 Classical/Frequentist View
 Bayesian View
Classical/Frequentist View
 Probability is estimated in terms of the
frequencies of random, repeatable events

 E.g., coin toss, dice roll etc.

Definition of Probability
 Experiment: toss a coin twice
 Sample space: possible outcomes of an experiment
 S = {HH, HT, TH, TT}
 Event: a subset of possible outcomes
 A={HH}, B={HT, TH}
 Probability of an event : a number assigned to an event, i.e.,
P(A)
 Axiom 1: 0 ≤ P(A) ≤ 1

 Axiom 2: P(S) = 1
Definition of Probability
 Experiment: toss a coin twice
 Sample space: possible outcomes of an experiment
 S = {HH, HT, TH, TT}
 Event: a subset of possible outcomes
 A={HH}, B={HT, TH}
 Probability of an event : a number assigned to an event, i.e.,
P(A)
 Axiom 3:
If an experiment can result in any one of ‘N’ different equally
likely outcomes, and if exactly ‘n’ of these outcomes correspond to
event A, then the probability of event A is:
P(A) = n / N
Example
 A coin is tossed twice. What is the probability that at least
1 head occurs?

 S = {HH, HT, TH, TT}

 A = {HH, HT, TH}

 P (A) = n / N = 3/4 (Axiom 3)

Additive Rule
 If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap

P(A U B) = P(A) + P(B) – P(A ∩ B)

Additive Rule
 If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap
P(A U B) = P(A) + P(B) – P(A ∩ B)
Additive Rule
 If A and B are two events, then the probability that A or B
will occur is the sum of the probability of each event minus
the overlap
P(A U B) = P(A) + P(B) – P(A ∩ B)

 If A and B are mutually exclusive, then

P(A U B) = P(A) + P(B)
Conditioning
 If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A) 
Pr( A)
Conditioning
 If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A) 
Pr( A)
 Example: Drug test
A = {Patient is a Woman}
Women Men B = {Drug fails}
Success 200 1800 Pr(B|A) = ?
Failure 1800 200 Pr(A|B) = ?
Conditioning
 If A and B are events with Pr(A) > 0, the conditional
probability of B given A is
Pr( AB)
Pr( B | A) 
Pr( A)
 Example: Drug test
A = {Patient is a Woman}
Women Men B = {Drug fails}
Success 200 1800 Pr(B|A) = 1800/2000 = 9/10
Failure 1800 200 Pr(A|B) = 1800/2000 = 9/10
Independence
 Two events A and B are independent if and only if

P(B|A) = P(B) or P(A|B) = P(A)

 Example:
 A = {You catch Covid-19}
 B = {Joe Biden catches Covid-19}

 Otherwise, A and B are dependent

The Product/Multiplicative Rule
 Enables us to calculate the joint probability
P(AB) that two events A and B will both
occur
P(A∩B) = P(A) P(B|A)
P(B∩A) = P(B) P(A|B)
P(A∩B) = P(B∩A)
The Product/Multiplicative Rule
 P(A∩B) = P(A) P(B|A)
 Example:
 Suppose that we have a fuse box containing 20
fuses, of which 5 are defective. If 2 fuses are
selected at random and removed from the box in
succession without replacing the first, what is the
probability that both fuses are defective?
The Product/Multiplicative Rule
 Solution:
 A = {First fuse is defective}
 B = {Second fuse is defective}
 A∩B = {A occurs and then B occurs after A has
occurred}
 P(A) = 5/20 = 1/4
 P(B|A) = 4/19
 P(A∩B) = P(A) P(B|A) = 1/4 * 4/19 = 1/19
Independence
 P(A∩B) = P(A) P(B|A)

 Two events A and B are independent if and only if

P(A ∩ B) = P(A) P(B)
 Therefore, to obtain the probability that two
independent events will both occur, we simply find
the product of their individual probabilities

 A set of events {Ai} is independent in case

Pr(i Ai )   i Pr( Ai )
Probability Theory
 Classical/Frequentist View
 Bayesian View
Bayesian View
 More general view based on the Bayes’ Rule
 Probabilities provide a quantification of
uncertainty in real-life events or phenomena
 Examples
 What is the probability that it will rain tomorrow?
 How likely is it that this stock will go up by the
end of the month?
 What is the likelihood that two companies will be
merged together?
Bayes’ Rule
 Given two events A and B and suppose that Pr(A) > 0. Then
Pr( B  A)
Pr( B | A) 
Pr( A)

Pr( B  A)  Pr( B ) Pr( A | B)

Pr( B  A) Pr( B ) Pr( A | B )

Pr( B | A)  
Pr( A) Pr( A)
Bayes’ Rule
 Given two events A and B and suppose
that Pr(A) > 0. Then
Thomas Bayes
Pr( B ) Pr( A | B ) (1701 - 1761)
Pr( B | A)  England
Pr( A)
Bayes’ Rule
 Given two events A and B and suppose that Pr(A) > 0. Then
Pr( B ) Pr( A | B )
Pr( B | A) 
Pr( A)
 Example:
Pr(R) = 0.8
R: It is a rainy day
Pr(W|R) R R
W: The grass is wet
W 0.7 0.3
Pr(R|W) = ?
W 0.3 0.7
Bayes’ Rule
 Given two events A and B and suppose that Pr(A) > 0. Then

Pr( B ) Pr( A | B )
Pr( B | A) 
Pr( A)
 Example:
Pr(R) = 0.8
Pr(W|R) R R R: It is a rainy day

W 0.7 0.3 W: The grass is wet

Pr(R|W) = 0.56
W 0.3 0.7

Pr( R  W ) Pr( R) Pr(W | R) 0.8 * 0.7

Pr( R | W )     0.56
Pr(W ) Pr(W ) 1.0
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7

Information
Pr(W|R)
R W

Inference
Pr(R|W)
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7

Information: Pr(E|H)
Hypothesis H Evidence E
Posterior Likelihood
Inference: Pr(H|E) Prior

Pr( E | H ) Pr( H )
Pr( H | E ) 
Pr( E )
Bayes’ Rule
R R
R: It rains
W 0.7 0.3
W: The grass is wet
W 0.3 0.7

Information: Pr(E|H)
Hypothesis H Evidence E
Posterior Likelihood
Inference: Pr(H|E) Prior

Pr( E | H ) Pr( H )
Pr( H | E ) 
Pr( E )
Reference
 Chapter 2: Probability & Statistics for
Engineers & Scientists, 9th edition, Walpole et
al, Prentice Hall (2012).