ProbStat2019 02 Prob

Basic terminology Probability types and laws Independent events Bayes’ theorem
Statistics
Introduction to Basic Probability
Ling-Chieh Kung
Department of Information Management

National Taiwan University
Basic Probability 1 / 52 Ling-Chieh Kung (NTU IM)

An example of inferential Statistics
I Quality control: For all LED lamps of brand IM, we are interested in
µ, the average number of hours of luminance.
I Let’s select a random sample of 40 lamps. A test shows that the
sample average is x̄ = 28000 hours.
I What’s the probability that µ = x̄?
I What’s the probability that µ ∈ [27000, 29000]?
I What’s the probability that µ ∈ [26000, 30000]?
I To assess these probabilities, we need to study Probability.

Probability
I Chapters 2 and 3: Descriptive Statistics.

I Chapters 8, 9, and above: Inferential Statistics.
I Now: Probability. The tool we need for inferential Statistics.

Road map
I Basic terminology.
I Probability types and laws.
I Independent events.
I Bayes’ theorem.

Experiments and events
I An experiment is a process that produces (random) outcomes.

I Tossing a coin. Outcomes: Head or Tail.
I Testing a new drug on a patient: Outcomes: Effective, not effective,
getting worse.
I Interviewing 20 consumers regarding how many will buy a new product.
Outcomes: 10, 15, 0, etc.
I Sampling every 200th bottle of ketchup for its weight. Outcome?
I An event is an outcome of an experiments.
I Each event has its probability to occur.

Elementary events
I An elementary event is an event that cannot be decomposed into

smaller events.
I Consider the experiment of rolling a dice.
I Getting 3 is an elementary event.
I How about getting a number larger than 3?
I The event of getting larger than 3 can be decomposed into three
elementary events: getting 4, 5, and 6.
I How about getting an even number?
I For asking 20 consumers regarding how many will buy a new product,
is “ten are willing to buy” an elementary event?

Sample spaces
I The sample space of an experiment is the collection of all elementary

events.
I A sample space contains “everything that may happen”.
I For an experiment, the sample space is unique.
I Nothing outside the sample space can occur.
I What is the sample space of:
I rolling a dice?
I rolling two dices?
I asking 20 consumers?
I testing a new drug?
I A sample space is a set. Elementary elements are elements of the set.
Events are subsets of the set.

Unions and intersections
I Let X, Y , and S be three sets. Let X ⊂ S and Y ⊂ S.

I Treat X and Y as events and S as the sample space.
I The union of X and Y , denoted by X ∪ Y , contains elements that are
in X or Y .
I X ∪ Y = {x|x ∈ X or x ∈ Y }.
I E.g., {2, 3, 5} ∪ {1, 5, 6} = {1, 2, 3, 5, 6}.
I The intersection of X and Y , denoted by X ∩ Y , contains elements
that are in X and Y .
I X ∩ Y = {x|x ∈ X and x ∈ Y }.
I E.g., {2, 3, 5} ∩ {1, 5, 6} = {5}.

Unions and intersections
I Venn diagrams for unions and intersections:

Two special cases
I Sets are mutually exclusive if there is no intersection.

I X ∩ Y = ∅.
I Events are mutually exclusive if all their elementary events are different.
I E.g., rolling an even number and rolling 5.
I Events are collectively exhaustive if they together cover the whole
sample space.
I S =X ∪Y.
I Events are collectively exhaustive if one of them must occur.
I E.g., rolling an even number and rolling smaller than six.
I Note that two collectively exhaustive sets are not necessarily mutually
exclusive!

Complements
I The complement of X, denoted by X 0 , contains all elements not in

X.
I X 0 = {x|x ∈/ X}.
I E.g., rolling one and rolling greater than one.
I For any set X, X and its complement X 0 are mutually exclusive and
collectively exhaustive.

Counting the possibilities
I There are three ways to count possibilities (not probabilities).

I The mn counting rule.
I Sampling with replacement.
I Sampling without replacement (combinations).

The mn counting rule
I The rule:
I Suppose experiment 1 has m outcomes and experiment 2 has n outcomes.
I Suppose the realization of an outcome in experiment 1 does not affect
the realization of outcomes of experiment 2.
I Then the total number of outcomes of the two experiments, in order, is
mn.
I Do the mn counting rule applies to the following examples?
I Rolling a dice twice.
I First asking one’s sex and then her/his marital status.
I First asking one’s marital status and then her/his years of marriage.

Sampling with or without replacement

I Sampling n items from a population of size N with replacement
provides
Nn
possibilities.
I Asking people (blindly) pass through a place for ten days.
I Sampling n items from a population of size N
without replacement provides

N N!
=
n n!(N − n)!
possibilities.
I Lottery: sampling six numbers from 42 numbers.

Sampling with or without replacement
I Note that
N! n
→ N n as → 0.
n!(N − n)! N
What does this imply?
I When n N , sampling without replacement can be approximated
by sampling with replacement.
I When we randomly sample 1000 consumers from the whole Taiwan,
which formula should we use?

Road map
I Bayes’ theorem.

An Example: dry cleaning business
I Among all dry-cleaning customers:

I 51% say discounts are important.
I 57% would try another cleaner if a discount were offered.
I Among those who say discounts are important, 90% would try another
cleaner if a discount were offered.
I Let’s try to organize these numbers into a joint probability table.

Joint probability tables
I In a joint probability table, we list relevant probabilities for two

experiments.
I For the dry cleaning business:
Switch Don’t switch Total

Important ? ? ?
Unimportant ? ? ?
Total ? ? 1
I (Some) events of experiment 1 are listed in the first column.

I (Some) events of experiment 2 are listed in the first row.
I Leave a column and a row at the end for totals.

Marginal probabilities
I Those probabilities that are with respect to the whole population

enters the “Total” column and “Total” row.
I Among all dry-cleaning customers:
I 51% say discounts are important.
I 57% would try another cleaner if a discount were offered.

Important ? ? 0.51
Unimportant ? ? 0.49
Total 0.57 0.43 1
I These are marginal probabilities. Why “marginal”?

Joint probabilities
I Mathematically:
I Use I for the event “Important” and U for “unimportant.”
I Use S for the event “Switch” and N for “Don’t switch.”
I Marginal probabilities: Pr(I) = 0.51, Pr(S) = 0.57, etc.

Joint probabilities
I A cell at the intersection of two events contains the joint probability

of that joint event.
I Joint probabilities: Pr(I ∩ S) or Pr(IS) or Pr(I and S).

Important Pr(I ∩ S) ? 0.51
Total 0.57 0.43 1
I The probability that one will think a discount is important and will
switch when there is a discount?
I How to find the values for joint probabilities?

Conditional probabilities
I To fill in the blanks, we need to use the last information:

I Those probabilities that are with respect to an outcome are
conditional probabilities.
I One would try another cleaner if a discount were offered with probability
90%, given that she thinks a discount is critical.
I Pr(S|I) = 0.9 denotes this conditional probability.
I What is the relationship among marginal probabilities, joint
probabilities, and conditional probabilities?

Conditional probability
I Conditional probabilities are defined based on joint and marginal

probabilities.
Definition 1 (Conditional probability)

Let X and Y be two events. The conditional probability of X
given Y is
Pr(X ∩ Y )
Pr(X|Y ) ≡ .
Pr(Y )

Conditional probability
I Y : Discounts are important. X: Would switch for a discount.
I Among Y , 90% are X ∩ Y : Pr(X ∩ Y ) = 0.9 Pr(Y ).
I This 90% is defined as the conditional probability Pr(X|Y ).

Law of multiplication
I The definition leads us to the following law:
Proposition 1 (Law of multiplication)

For any two events X and Y , we have
Pr(X ∩ Y ) = Pr(X|Y ) Pr(Y ) = Pr(Y |X) Pr(X).
Proof. By definition, we have Pr(X|Y ) ≡ Pr(X∩Y )

Pr(Y ) . Rearranging these
terms results in the first equality. The second equality is established
similarly.

Law of conditional probability
I Now let’s go back to our joint probability table:

Important Pr(I ∩ S) ? 0.51
Total 0.57 0.43 1
I Among those we say discounts are important, 90% will switch. So
Pr(I)Pr(S|I) = 0.51 × 0.9 = 0.459 = Pr(S ∩ I).

Finding the joint probabilities
I We now have one more number in the table:

Important 0.459 ? 0.51
Total 0.57 0.43 1
I How to complete the table?

Important 0.459 0.051 0.51
Unimportant 0.111 0.379 0.49
Total 0.57 0.43 1

Finding hidden conditional probabilities
I We now have all the four joint probabilities.

I Pr(S ∪ I) = 0.459, Pr(S ∪ N ) = 0.051, etc.

Important 0.459 0.051 0.51
Unimportant 0.111 0.379 0.49
Total 0.57 0.43 1
I We may also find all “hidden” conditional probabilities.

Pr(S ∩ U ) 0.111
I Pr(S|U ) = = ≈ 0.227.
Pr(U ) 0.49
I Pr(I|S)? What does this mean?

Union probabilities
I How about the probability of the union of two events?

I Pr(S ∪ I)?
I By looking at the table, we know
Pr(S ∪ I) = 0.459 + 0.111 + 0.051 = 0.621.

Important 0.459 0.051 0.51
Unimportant 0.111 0.379 0.49
Total 0.57 0.43 1
I Is there a general formula?

Law of addition
I We have the following law:
Proposition 2 (Law of addition)

For any two events X and Y , we have
Pr(X ∪ Y ) = Pr(X) + Pr(Y ) − Pr(X ∩ Y ).
Proof. First, note that

Pr(X ∪ Y ) = Pr(X ∩ Y ) + Pr(X 0 ∩ Y ) + Pr(X ∩ Y 0 ). Then
Pr(X ∩ Y ) + Pr(X 0 ∩ Y ) = Pr(X) and
Pr(X ∩ Y ) + Pr(X ∩ Y 0 ) = Pr(Y ) complete the proof.

Law of addition
I Graphical representation of the proof:
I In short, Pr(X ∩ Y ) is double counted.

Special law of addition
I The law of addition can be specialize as follows:
Proposition 3 (Special law of addition)

For any two mutually exclusive events X and Y , we have
Pr(X ∪ Y ) = Pr(X) + Pr(Y ).
Proof. We have Pr(X ∩ Y ) = Pr(X) + Pr(Y ) − Pr(X ∩ Y ) according to

the law of addition. Because X and Y are mutually exclusive, we have
Pr(X ∩ Y ) = 0. The proof is thus completed.

Summary
I Four types of probabilities:

I Marginal.
I Joint.
I Union.
I Conditional.
I Which two can be read from a joint probability table? Which two
require calculation?
I Two laws for calculating probabilities:
Law of Is for calculating Based on
Addition Union Marginal and joint
Multiplication Joint Marginal and conditional

Road map
I Bayes’ theorem.

Independent events
I Two events are independent if whether one occurs does not affect
the probability for the other one to occur.
I Two events are dependent if they are not independent.
I Two experiments are independent if they produce independent events.
I A set of events are independent if all pairs of events are independent.

Independent events
I Are the following pairs of events independent?

I Rolling two today and rolling three tomorrow with a fair dice.
I A customer is a male and he likes watching baseball.
I One’s phone number contains “7” and she was born on July.
I Two outcomes with sampling with replacement.
I Two outcomes with sampling without replacement.
I A laptop is defective and it has a 14-inch screen.

Independent events
I The definition of independent events is as follows:
Definition 2 (Independent events)

For any two events X and Y , they are independent if
Pr(X|Y ) = Pr(X) and Pr(Y |X) = Pr(Y ).
I In other words, knowing that one has occurred does not help me
modify my belief on whether the other will occur.

Special law of multiplication
I The law of multiplication can be specialized as follows:
Proposition 4 (Special law of multiplication)

For any two independent events X and Y , we have
Pr(X ∩ Y ) = Pr(X) Pr(Y ).
Proof. We have Pr(X ∩ Y ) = Pr(X|Y ) Pr(Y ) according to the law of

multiplication. Because X and Y are independent, we have
Pr(X|Y ) = Pr(X). This completes the proof.

Joint probability tables revisited
I If two experiments are independent, how may we find the value of a

cell in a joint probability table?
I Do we still need conditional probabilities?
I Are the following two experiments independent?
B1 B2 B3 Total
A1 0.21 0.35 0.14 0.7
A2 0.09 0.15 0.06 0.3
Total 0.3 0.5 0.2 1

Joint probability tables revisited
I How about these two?

B1 B2 B3 Total
A1 0.24 0.08 0.28 0.6
A2 0.16 0.02 0.22 0.4
Total 0.4 0.1 0.5 1
I A1 and B1 are independent. Knowing that B1 has occurred tells us

nothing about whether A1 occurs and vice versa.
I But whether B3 occurs matters:
I If we do not observe B3 , we believe Pr(A1 ) = 0.6.
I If we observe B3 , we’ll update out belief: Pr(A1 |B3 ) = 0.56.

Belief updating
I In general, people keep updating their believes upon receiving new

information.
I If there have been ten consecutive days that a product is sold out, how
may we update our belief on consumers’ valuation?
I If a treatment makes one patient feel uncomfortable, how may we update
the belief on the side effect?
I If a girl does not MSN me for a week, how may we update my belief on
whether she likes me?
I A systematic way of belief updating relies on Bayes’ theorem.

Road map
I Bayes’ theorem.

Law of total probability
I The following law is a component of Bayes’ rule:
Proposition 5 (Law of total probability)

Let events Y1 , Y2 , ..., and Yk be mutually exclusive and completely
exhaustive and X be another event, then
k
X
Pr(X) = Pr(Yi ) Pr(X|Yi ).
i=1
Proof. Homework!

Law of total probability
I Graphical representation of the law:

Belief updating
I For some unknowns, we have some original estimates.

I We form a prior belief or assign a prior probability to the
occurrence of an event.
1
I Before I toss a coin, my belief of getting a head is 2
.
I If our estimation is accurate, the relative frequency of the
occurrence of the event should be close to my prior belief.
48
I In 100 trials, probably I will see 48 heads. 100
≈ 21 .
I What if I see 60 heads? What if 90?
I In general, we expect observations to follow our prior belief.
I If this is not the case, we probably should update our prior belief into a
posterior belief.

Example: Popularity of a product
I Suppose we have a product to sell.

I We do not know how consumers like it.
I Two possibilities (events): popular (P ) and unpopular (U ).
I Our prior belief on P is 0.7.
I We believe, with confidence 70%, that the product is popular.
I When one comes, she may buy it (B) or go away (G).
I If popular, the buying probability is 0.6.
I If unpopular, the buying probability is 0.2.
I Suppose event G occurs once, what’s our posterior belief?


I We have the marginal probabilities Pr(P ) and Pr(U ):
B G Total
P ? ? 0.7
U ? ? 0.3
Total ? ? 1
I We have the conditional probabilities Pr(B|P ), Pr(U |P ), Pr(B|U ), and

Pr(G|U ):
B G Total
P 0.42 0.28 0.7
U 0.06 0.24 0.3
Total ? ? 1

I We then have the marginal probabilities Pr(B) and Pr(G):

B G Total
P 0.42 0.28 0.7
U 0.06 0.24 0.3
Total 0.48 0.52 1
I Now, we observe one consumer goes away (event G). Given this, what
is the posterior belief that the product is indeed popular (event P )?
I This is the conditional probability Pr(P |G).
Pr(P ∩ G) 0.28
I Pr(P |G) = = ≈ 0.54.
Pr(G) 0.52

I Note that we update our belief on P from 0.7 to 0.54.

I The fact that one goes away makes us less confident.
I If another consumer goes away, the updated belief on P becomes 0.37.
I Use Pr(P |G) as Pr(P ) and Pr(U |G) as Pr(U ) and repeat.
I After five consumers go away in a row, our confidence on that the
product is popular becomes 0.07.
I We tend to believe the product is unpopular!

Bayes’ theorem
Pr(P ∩ G)
I Recall that we have done this: Pr(P |G) = .
Pr(G)
I Combining this with the law of total probability, we establish
Bayes’ theorem:
Proposition 6 (Bayes’ theorem)

Let events Y1 , Y2 , ..., and Yk be mutually exclusive and completely
exhaustive and X be another event, then
Pr(Yj ) Pr(X|Yj )
Pr(Yj |X) = Pk ∀j = 1, 2, ..., k.
i=1 Pr(Yi ) Pr(X|Yi )

Bayes’ theorem
I Sometimes we have events {Yi }i=1,...,k and X:

I It is clear how Yi s affect X but not the other way.
I Bayes’ theorem is applied to use X to infer {Yi }i=1,...,k .
I In our product popularity example:
I P and U are mutually exclusive and completely exhaustive.
I They naturally affect G but not the other way.
I So we apply Bayes’ theorem to use G to infer P and U :
Pr(P ) Pr(G|P )
Pr(P |G) =
Pr(P ) Pr(G|P ) + Pr(U ) Pr(G|U )
0.7 × 0.4
= = 0.54.
0.7 × 0.4 + 0.3 × 0.8

Some remarks
I Bayes’ theorem is powerful and used widely.

I In particular, Engineering and Computer Science.
I Using Bayes’ theorem for belief updating is called
Bayesian updating or Bayesian inference.
I There is a formula for Bayesian updating after multiple observations.
I Unfortunately (fortunately?), we do not have time to go deeper in this
topic.

ProbStat2019 02 Prob

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ProbStat2019 02 Prob

Uploaded by

Copyright:

Available Formats

Basic terminology Probability types and laws Independent events Bayes’ theorem

Department of Information Management

Basic Probability 1 / 52 Ling-Chieh Kung (NTU IM)

An example of inferential Statistics

Basic Probability 2 / 52 Ling-Chieh Kung (NTU IM)

I Chapters 2 and 3: Descriptive Statistics.

Basic Probability 3 / 52 Ling-Chieh Kung (NTU IM)

Basic Probability 4 / 52 Ling-Chieh Kung (NTU IM)

Experiments and events

I An experiment is a process that produces (random) outcomes.

Basic Probability 5 / 52 Ling-Chieh Kung (NTU IM)

I An elementary event is an event that cannot be decomposed into

Basic Probability 6 / 52 Ling-Chieh Kung (NTU IM)

I The sample space of an experiment is the collection of all elementary

Basic Probability 7 / 52 Ling-Chieh Kung (NTU IM)

Unions and intersections

I Let X, Y , and S be three sets. Let X ⊂ S and Y ⊂ S.

Basic Probability 8 / 52 Ling-Chieh Kung (NTU IM)

Unions and intersections

I Venn diagrams for unions and intersections:

Basic Probability 9 / 52 Ling-Chieh Kung (NTU IM)

Two special cases

I Sets are mutually exclusive if there is no intersection.

Basic Probability 10 / 52 Ling-Chieh Kung (NTU IM)

I The complement of X, denoted by X 0 , contains all elements not in

Basic Probability 11 / 52 Ling-Chieh Kung (NTU IM)

Counting the possibilities

I There are three ways to count possibilities (not probabilities).

Basic Probability 12 / 52 Ling-Chieh Kung (NTU IM)

The mn counting rule

Basic Probability 13 / 52 Ling-Chieh Kung (NTU IM)

Sampling with or without replacement

Basic Probability 14 / 52 Ling-Chieh Kung (NTU IM)

Sampling with or without replacement

Basic Probability 15 / 52 Ling-Chieh Kung (NTU IM)

Basic Probability 16 / 52 Ling-Chieh Kung (NTU IM)

An Example: dry cleaning business

I Among all dry-cleaning customers:

Basic Probability 17 / 52 Ling-Chieh Kung (NTU IM)

Joint probability tables

I In a joint probability table, we list relevant probabilities for two

Switch Don’t switch Total

I (Some) events of experiment 1 are listed in the first column.

Basic Probability 18 / 52 Ling-Chieh Kung (NTU IM)

I Those probabilities that are with respect to the whole population

Switch Don’t switch Total

Basic Probability 19 / 52 Ling-Chieh Kung (NTU IM)

Basic Probability 20 / 52 Ling-Chieh Kung (NTU IM)

I A cell at the intersection of two events contains the joint probability

Switch Don’t switch Total

Basic Probability 21 / 52 Ling-Chieh Kung (NTU IM)

I To fill in the blanks, we need to use the last information:

Basic Probability 22 / 52 Ling-Chieh Kung (NTU IM)

I Conditional probabilities are defined based on joint and marginal

Definition 1 (Conditional probability)

Basic Probability 23 / 52 Ling-Chieh Kung (NTU IM)

Basic Probability 24 / 52 Ling-Chieh Kung (NTU IM)

I The definition leads us to the following law:

Proposition 1 (Law of multiplication)

Pr(X ∩ Y ) = Pr(X|Y ) Pr(Y ) = Pr(Y |X) Pr(X).

Proof. By definition, we have Pr(X|Y ) ≡ Pr(X∩Y )

Basic Probability 25 / 52 Ling-Chieh Kung (NTU IM)

Law of conditional probability

I Now let’s go back to our joint probability table: