WIA2003/WIB2003 Probability and Statistics: Bayes' Theorem

WIA2003/WIB2003
PROBABILITY AND STATISTICS

Bayes’ Theorem
1
WIA2003/WIB2003 Probability and Statistics
Revisit:
Probability and its notations
• Fundamentals of Probability
– Deterministic and random processes
– Events and sample space
– Permutation and combination
– Probability
• Intersection (mutual exclusive), union, conditional probability
(independence), complement
• Rule of subtraction, multiplication and addition
– Random Variables
• Discrete and continuous
2
Probability and its notations WIA2003/WIB2003 Probability and Statistics
How to calculate sample space and events?
• When the order is taken into account: Permutations

• When the order is not important: Combinations
• A permutation is an ordered combinations
– Permutation with repetition: 𝑛!

"!
– Permutation without repetition:
"$! !
"! "
– Combination without repetition: OR !
!! "$! !
!%"$& !
– Combination with repetition:
!! "$& !
where 𝑛 is the number of things to choose from, and 𝑟 of them are
chosen.
3
Permutation with repetition
• Given a travel lock that have 3 numbers to set from

choices from 0-9. What are the total possible orders
to set?
• 𝑛 = 10, 𝑟 = 3;
• Total possible orders à 𝑛! = 10" = 1000.
4
Permutation without repetition
• In a pool game, there will be a total of 16 different

pool balls. What are the total possible orders for the
first 3 pool balls be in?
• 𝑛 = 16, 𝑟 = 3.
• Total possible orders à
#! &'!
#%! !
= (&'%&")!
= 16 × 15 × 14 = 3360.
5
Combination without repetition

pool balls. What are the total possible outcome for
the first 3 pool balls be in, regardless of its order?
• 𝑛 = 16, 𝑟 = 3.
• Total possible outcome à
#! &'! &' × &+ ×&,
= = = 560.
!! #%! ! "!(&'%")! "×-×&
6
Combination with repetition?

the first 3 pool balls be in, regardless of its order, if
the pool ball will be put back once it get in?
• 𝑛 = 16, 𝑟 = 3.
(#.!%&)! (&'."%&)! &/! &/ × &0 ×&'
!! #%& !
= "!(&'%&)!
= "!&+! " × - × &
= 816.
7
Combination with repetition?

the first 3 pool balls be in, regardless of its order, if
the pool ball will be put back once it get in?
• 𝑛 = 16, 𝑟 = 3.
(#.!%&)! (&'."%&)! &/! &/ × &0 ×&'
!! #%& !
= "!(&'%&)!
= "!&+! " × - × &
= 816.
8
Combination with repetition
• Given that there are 5 choices of ice cream flavor.

How many combinations are they if we can have 3
scoops?
• 𝑛 = 5, 𝑟 = 3.
(#.!%&)! (+."%&)! 0!
!! #%& !
= "!(+%&)!
= "!,!
= 35.
9
Revisit: Probability and its notations WIA2003/WIB2003 Probability and Statistics
Try this!
1. What is the probability for the number of the first

dice roll to be higher than the second dice roll?
15
𝑃 𝑋>𝑌 =
36
10
Revisit: Probability and its notations WIA2003/WIB2003 Probability and Statistics
Try this!
2. Given that they are 50 students in a class, where 30

of them are males, and 20 of them are wearing
spectacles. What is the potential range of
probability for male students that are wearing
spectacles?
• There are two extreme cases, where the intersection between "males"
and "wearing spectacles" 𝑃 𝑀 ∩ 𝑊 is maximum or minimum.
• When 𝑃 𝑀 ∩ 𝑊 is maximum, means 20 of the males is wearing
spectacles. The probability is 20/30 = 0.67.
• When 𝑃 𝑀 ∩ 𝑊 is minimum, means none of the males is wearing
spectacles, which is possible because 20 of the students that wear
spectacles can be all female. The probability is 0/30 = 0.
11
• Therefore, the range now is 0 - 0.67.
Bayes’ Theorem WIA2003/WIB2003 Probability and Statistics
Example:
• Consider a fictitious computer company. We might
ask the following questions:
– What is the probability that consumers will actually buy a new

computer?
– What is the probability that consumers are planning to buy a new
computer?
– What is the probability that consumers are planning to buy and
actually will buy a new computer?
– Given that a consumer is planning to buy, what is the probability of a
purchase?
12
Probability Review
• If we have an event 𝐴 there will be a complement to
𝐴 called 𝐴′ or 𝐵.
• Marginal probabilities
– Event 𝐴 consists of two outcomes, 𝑎& and 𝑎' :
𝐴 = {𝑎& , 𝑎' }
– The complement 𝐵 consists also of two outcomes, 𝑏& ahd 𝑏' :
𝐵 = {𝑏& , 𝑏' }
– Two events are mutually exclusive/disjoint if both events cannot
occur together
– A set of events is collectively exhaustive if one of the events must
occur
13
Disjoint (mutual exclusive)

• If A and B are disjoint (mutually exclusive) events,
then
𝑃 𝐴 or 𝐵 = 𝑃 𝐴 + 𝑃(𝐵)
Example:
Let A: respondent lives in KL
Let B: respondents lives in Penang
If 𝑃 𝐴 = 0.087
𝑃 𝐵 = 0.106
Then 𝑃 𝐴 or 𝐵 = 0.193 14
Probability of Combined Events

• More generally, if 𝐴 and 𝐵 are not mutually
exclusive,
𝑃 𝐴 or 𝐵 = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃(𝐴 and 𝐵)
Example:
Let 𝐴 = event that respondent sleep less than 6 hours
per night.
Let 𝐵 = event that respondent reports trouble sleep
most or all of the time.
𝑃 𝐴 = 0.139; 𝑃 𝐵 = 0.152; 𝑃 𝐴 and 𝐵 = 0.061
Thus 𝑃 𝐴 or 𝐵 = 0.139 + 0.152 − 0.061 = 0.230
Exhaustive Events
• Two or more events are said to be exhaustive (or
collective exhaustive) if at least one of them must
occur.
• 𝐴∪𝐵 =𝑆
• For example, if 𝐴 is the event that the respondent

sleeps less than 6 hours per night and 𝐵 is the event
that the respondent sleeps at least 6 hours per night,
then 𝐴 and 𝐵 are exhaustive.
16
Exhaustive VS mutual exclusive

• For a dice roll:
• 1,2,3,4,5 and 6 (all possible values): exhaustive and mutual exclusive
• 2 and 6: mutual exclusive but not exhaustive
• “even” and “not 6”: not mutual exclusive but exhaustive
• For a coin toss?
17
Independence
• Two events are independent if the occurrence of one
in no way affects the probability of the other.
• If events 𝐴 and 𝐵 are independent, then

𝑝 𝐴 and 𝐵 = 𝑝 𝐴 𝑝 𝐵
• If events 𝐴 and 𝐵 are not independent, then

𝑝 𝐴 and 𝐵 = 𝑝 𝐴 𝑝(𝐵|𝐴)
18
Example
• The sample space (of 1000 customers) looks like this:
Actually purchase
Yes (𝑏! ) No (𝑏" ) Total
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000
19
Example
• If 𝐴 = planned to purchase and 𝐵 = actually
purchased:
𝑃(planned to buy)= 𝑃(planned & did) + 𝑃(planned & did not)
= 𝑃 𝑎! and 𝑏! + 𝑃(𝑎! and 𝑏" )
Actually purchase
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000
"## $# "$#
= + = = 0.25
!### !### !###
20
Probability Review
• Example, there are 40 female students in a class of
100. 10 of them are from some foreign countries. 20
male students are also foreign students.
– 𝐴: student from a foreign country

– 𝐵: a female student
21
Venn Diagrams
30+10 = 40 20+10 = 30
Foreign
Female (10) Student
(30)
(20)
Male non-foreign student Female foreign student (10)

(40)
22
Probability Review
• Complement: 𝑝 𝐴3 = 1 − 𝑝 𝐴 ; 𝑝 𝐵3 = 1 − 𝑝(𝐵)
Non Female
Female
Non Foreign
Foreign
Student
student
23
Conditional probability
n The conditional probability of 𝐴 given 𝐵 is the joint
probability of 𝐴 and 𝐵, divided by the marginal
probability of 𝐵:
𝑃 𝐴, 𝐵
𝑃 𝐴𝐵 =
𝑃 𝐵
n Thus if 𝐴 and 𝐵 are statistically independent,
𝑃 𝐴, 𝐵 𝑃 𝐴 𝑃 𝐵
𝑃 𝐴𝐵 = = = 𝑃(𝐴)
𝑃 𝐵 𝑃 𝐵
n However, if 𝐴 and 𝐵 are statistically dependent, then
𝑃 𝐴 𝐵 ≠ 𝑃(𝐴)
24
Conditional Probabilities
• Conditional probabilities can be written as
𝑃 𝑋&𝑌 Joint probability of 𝑋 & 𝑌
= = 𝑃(𝑋|𝑌)
𝑃 𝑋 Marginal probability 𝑜𝑓 𝑋
– This is read as “The probability of X given Y”

– For example: The probability that someone earns between $200 and
$300, given that he/she has completed 10 years of education
25
Marketing Example
Actually purchase
Plan to Yes (𝑎! ) 200 50 250
purchase No (𝑎" ) 100 650 750
Total 300 700 1000
• Let’s look at the probability you purchased a computer given

that you planned to purchase:
𝑃 actually purchased planned to purchase
= 𝑃(𝑏&|𝑎&)
4 5! ,7!
= 4 7
-88
=
-+8
= 0.8 26
Useful Properties of Conditional

Probabilities
• Property 1. The Conditional Probability for
Independent Events
– If A and B are independent events, then:
𝑃 𝐴∩𝐵 𝑃 𝐴 𝑃 𝐵
𝑃 𝐴𝐵 = = = 𝑃(𝐴)
𝑃 𝐵 𝑃 𝐵
• Property 2. The Multiplication Rule for Conditional

Probabilities
– In an experiment involving two non-independent events 𝐴 and 𝐵, the
probability that both 𝐴 and 𝐵 occurs can be found in the following two
ways:
𝑃 𝐴∩𝐵 =𝑃 𝐵 𝑃 𝐴 𝐵
or
27
𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃(𝐵|𝐴)
Bayes’ Theorem/Rule
n Someone is shooting at a target. If it is windy, he has
a 40% chance hitting the target. If there is no wind,
his chance is 70%.
n A. If there is a 12% chance of being windy, what is his

chance of hitting the target?
n B. If he hits the target, what is the chance of it being

windy? à Bayes Theorem
28
Bayes’ Theorem
• Introduced by Reverand Thomas Bayes (1702-1761)
• Bayes Theorem for probability events A and B
𝑃 𝐵𝐴 𝑃 𝐴
𝑃 𝐴𝐵 =
𝑃 𝐵
Or for a set of mutually exclusive and exhaustive
events (i.e. 𝑝 ⋃M 𝐴M = ∑M 𝑝 𝐴M = 1), then
𝑃 𝐵 𝐴 𝑃 𝐴(
𝑃 𝐴( 𝐵 =
∑) 𝑃 𝐵 𝐴) 𝑃(𝐴) )
29
Bayes’ Theorem
• Think of the Bayes Theorem as probability in reverse
– You can update your probabilities in light of new information
• Suppose you have a product with a known
probability of success
𝑃 success = 𝑃 𝑆 = 0.4
𝑃 failure = 𝑃 𝑆 * = 0.6
• We also know that a consumer group will write
either a favorable or unfavorable report on the
product
𝑃 𝐹 𝑆 = 0.8; 𝑃 𝐹 𝑆 * = 0.3
30
Bayes’ Theorem
• We want to find the probability that the product will
be successful given a favorable report
𝑃(𝑆|𝐹)
• In this case, Bayes says
𝑃 𝐹𝑆 𝑝 𝑆
𝑃 𝑆𝐹 =
𝑃 𝐹 𝑆 𝑃 𝑆 + 𝑃 𝐹 𝑆 * 𝑃(𝑆 * )
• We can plug values into the above equation to find
0.8 0.4 0.32
𝑃 𝑆𝐹 = = = 0.64 = 64%
0.8 0.4 + (0.3)(0.6) 0.50
• We can use the theorem to update the probability of
a successful product given that the product gets a
favorable report
31
Bayes’ Theorem/Rule
• Bayes theorem deals with sequential events,
whereby new additional information is obtained for a
subsequent event, and that new information is used
to revise the probability of the initial event.
– Prior probability- is an initial probability value originally

obtained before any additional information is obtained.
– Posterior probability- is a probability value that has been

revised by using additional information that is later
obtained.
32
Bayes’ Theorem/Rule - example

The Gallup organization randomly selects an adult
American for a survey about credit card usage. Use
subjective probabilities to estimate the following.
a) What is the probability that the selected subject is a male?

b) After selecting a subject, it is later learned that this person
was smoking a cigar during the interview. What is the
probability that the selected subject is a male?
c) Which of the preceding two results is a prior probability?
Which is a posterior probability?
33

a) Roughly half of all Americans are males, so we estimate the
probability of selecting a male subject to be 0.5. Denoting a
male by 𝑀, we can express this probability as follows:
𝑃 𝑀 = 0.5.
b) Although some women smoke cigars, the vast majority of

cigar smokers are males. A reasonable guess is that 85% of
cigar smokers are males. Based on this additional subsequent
information that the survey respondent was smoking a cigar,
we estimate the probability of this person being a male as
0.85. Denoting a male by 𝑀 and denoting a cigar smoker by
𝐶, we can express this result as follows: 𝑃 𝑀 𝐶 = 0.85.
34

c) In part a), the value of 0.5 is the initial probability, so we
refer to it as the prior probability. Because the probability of
0.85 in part b) is a revised probability based on the additional
information that the survey subject was smoking a cigar, this
value of 0.85 is referred to a posterior probability.
35
Bayes’ Theorem
• In Orange County, 51% of the adults are males. One
adult is randomly selected for a survey involving
credit card usage.
a) Find the prior probability that the selected person is a male.

b) It is later learned that the selected survey subject was
smoking a cigar. Also, 9.5% of males smoke cigars, whereas
1.7% of females smoke cigars.
c) Use this additional information to find the probability that
the selected subject is a male.
36
Bayes’ Theorem
• Let's use the following notation:
𝑀 = male 𝑀′ = female (or not male)
𝐶 = cigar smoker C′ = not a cigar smoker.
a. Before using the information given in part b, we know only

that 51% of the adults in Orange County are males, so the
probability of randomly selecting an adult and getting a male
is given by 𝑝 𝑀 = 0.51.
37
Bayes’ Theorem
b. Based on the additional given information, we have the
following:
– 𝑃(𝑀) = 0.51 because 51% of the adults are males
– 𝑃(𝑀’) = 0.49 because 49% of the adults are females (not
males)
– 𝑃(𝐶|𝑀) = 0.095 because 9.5% of the males smoke cigars
(The probability of getting someone who smokes cigars,
given that the person is a male, is 0.095.)
– 𝑃(𝐶|𝑀’) = 0.017. because 1.7% of the females smoke
cigars (The probability of getting someone who smokes
cigars, given that the person is a female, is 0.017.)
38
Bayes’ Theorem
c. Initially we knew that the survey subject smoked a cigar,
there is a 0.51 probability that the survey subject is male,
however, after learning that the subject smoked a cigar, we
revised the probability to 0.853. This makes sense, because
the likelihood of a male increases dramatically with the
additional information that the subject smokes cigars
(because so many more males smoke cigars than females).
4 9 ⋅4(;|9)
𝑃 𝑀𝐶 =
4 9 ⋅4(;|9).4 9" ⋅4(;|9" )
8.+&⋅8.8>+
=
8.+&⋅8.8>+ .[8.,>⋅8.8&0]
= 0.85329341
= 0.8533 (rounded)
39
More Examples
• Given:
– A doctor knows that meningitis (𝑀) causes stiff neck (𝑆) 50%
of the time
– Prior probability of any patient having meningitis is 1/50,000
– Prior probability of any patient having stiff neck is 1/20
• If a patient has stiff neck, what’s the probability he/she

has meningitis?
1 1
𝑃 𝑆𝑀 𝑃 𝑀 0.5× 0.5×
𝑃 𝑀𝑆 = = 50000 = 50000 = 0.0002
𝑃(𝑆) 𝑃(𝑆) 1
20
40
An example using Bayes’ theorem

• Suppose the probability (for anyone) to have AIDS is:
𝑃(AIDS)=0.001; 𝑃(no AIDS)=0.999 ← prior probabilities, i.e.,
before any test carried out
Consider an AIDS test: result is + or –
𝑃 + 𝐴𝐼𝐷𝑆 = 0.98; 𝑃 − 𝐴𝐼𝐷𝑆 = 0.02; ← probabilities to (in)correctly

identify an infected person
𝑃 + 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.03; 𝑃 − 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.97 ← probabilities

to (in)correctly
identify an
uninfected
person
Suppose your result is +. How worried should you be?

41
Bayes’ theorem example (cont.)

• The probability to have AIDS given a + result is
• Posterior probability:
A(.|BCDE)A BCDE
𝑃 𝐴𝐼𝐷𝑆 + = A(.|BCDE)A BCDE .A(.|#F BCDE)A(#F BCDE)
8.>/×8.88&
= 8.>/×8.88&.8.8"×8.>>>
= 0.032
• Your viewpoint: my degree of belief that I have AIDS

is 3.2%
• Your doctor’s viewpoint: 3.2% of people like this will
have AIDS 42
The End
43

WIA2003/WIB2003 Probability and Statistics: Bayes' Theorem

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

WIA2003/WIB2003 Probability and Statistics: Bayes' Theorem

Uploaded by

Copyright:

Available Formats

WIA2003/WIB2003

PROBABILITY AND STATISTICS

How to calculate sample space and events?

• When the order is taken into account: Permutations

– Permutation with repetition: 𝑛!

Permutation with repetition

• Given a travel lock that have 3 numbers to set from

Permutation without repetition

• In a pool game, there will be a total of 16 different

Combination without repetition

• In a pool game, there will be a total of 16 different

Combination with repetition?

• In a pool game, there will be a total of 16 different

Combination with repetition?

• In a pool game, there will be a total of 16 different

Combination with repetition

• Given that there are 5 choices of ice cream flavor.

1. What is the probability for the number of the first

2. Given that they are 50 students in a class, where 30

– What is the probability that consumers will actually buy a new

Disjoint (mutual exclusive)

Probability of Combined Events

• For example, if 𝐴 is the event that the respondent

Exhaustive VS mutual exclusive

• For a coin toss?

• If events 𝐴 and 𝐵 are independent, then

• If events 𝐴 and 𝐵 are not independent, then

– 𝐴: student from a foreign country

Male non-foreign student Female foreign student (10)

– This is read as “The probability of X given Y”

• Let’s look at the probability you purchased a computer given

Useful Properties of Conditional

• Property 2. The Multiplication Rule for Conditional

n A. If there is a 12% chance of being windy, what is his

n B. If he hits the target, what is the chance of it being

– Prior probability- is an initial probability value originally

– Posterior probability- is a probability value that has been

Bayes’ Theorem/Rule - example

a) What is the probability that the selected subject is a male?

Bayes’ Theorem/Rule - example

b) Although some women smoke cigars, the vast majority of

Bayes’ Theorem/Rule - example

a) Find the prior probability that the selected person is a male.

a. Before using the information given in part b, we know only

• If a patient has stiff neck, what’s the probability he/she

An example using Bayes’ theorem

𝑃 + 𝐴𝐼𝐷𝑆 = 0.98; 𝑃 − 𝐴𝐼𝐷𝑆 = 0.02; ← probabilities to (in)correctly

𝑃 + 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.03; 𝑃 − 𝑛𝑜 𝐴𝐼𝐷𝑆 = 0.97 ← probabilities

Suppose your result is +. How worried should you be?

Bayes’ theorem example (cont.)

• Your viewpoint: my degree of belief that I have AIDS

You might also like