Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability

Conditional
Probability
and the
General
Probability Judgment
”And” Rule

Total
Probability
and the
General ”Or”
Rule Manoogian College of Business and Economics, AUA
Bayes’s Rule

Bayesian
Updating

Discussion

1 / 20
 Decision-Making under Uncertainty

Introduction

Fundamentals
of Probability 1 Introduction
Theory

Unconditional
Probability 2 Fundamentals of Probability Theory
Conditional
Probability 3 Unconditional Probability
and the
General
”And” Rule 4 Conditional Probability and the General ”And” Rule
Total
Probability
and the 5 Total Probability and the General ”Or” Rule
General ”Or”
Rule

Bayes’s Rule
6 Bayes’s Rule
Bayesian
Updating 7 Bayesian Updating
Discussion
8 Discussion

2 / 20
 Introduction

Introduction

Fundamentals
of Probability
Theory

Unconditional
• Although the theory of choice under certainty is helpful for
Probability a range of purposes, most real-life decisions are not
Conditional
Probability
choices under certainty.
and the
General • In this module, we explore the theory of probability that
”And” Rule
many consider as a correct normative theory of
Total
Probability probabilistic judgment.
and the
General ”Or” • Like the theory of rational choice under certainty,
Rule

Bayes’s Rule
probability theory is axiomatic, so we start learning about
Bayesian decision-making under uncertainty from a set of axioms in
Updating
probability.
Discussion

3 / 20
 Important Definitions

Introduction

Fundamentals Definition of ”outcome space”

of Probability
Theory The outcome space is the set of all possible individual
Unconditional
Probability
outcomes.
Conditional
Probability
• We represent outcome spaces using curly brackets and
and the
General
commas.
”And” Rule

Total
Definition of ”outcome”
Probability
and the
An outcome is a subset of the outcome space.
General ”Or”
Rule

Bayes’s Rule Definition of ”probability”

Bayesian
Updating
The probability function is a function P r(·) that assigns a real
Discussion
number in [0, 1] to each outcome. The probability of an
outcome A is the number P r(A) assigned to A by the
probability function P r(·).
4 / 20
 Important Definitions, cont.

Introduction

Fundamentals
of Probability
Definition of ”equiprobable”
Theory
Outcomes are equiprobable if they occur with the same
Unconditional
Probability probability.
Conditional
Probability
and the Definition of ”mutual exclusivity”
General
”And” Rule Outcomes are mutually exclusive if no more than one of them
Total
Probability
can occur at any one time.
and the
General ”Or”
Rule
Definition of ”independence”
Bayes’s Rule

Bayesian
Outcomes are independent if the occurrence of the one does
Updating not affect the probability of the other one occurring.
Discussion

Note that when two outcomes are mutually exclusive they are
not independent.
5 / 20
 Rules of (Unconditional) Probability
Introduction

Fundamentals
of Probability
Theory

Unconditional The Equiprobable Rule

Probability

Conditional
If all individual outcomes in an outcome space
Probability
and the
{A1 , A2 , · · · , An } are equiprobable, then the probability of any
General
”And” Rule
one individual outcome Ai is 1/n, that is: P r(Ai ) = 1/n.
Total
Probability
and the The ”Or” Rule
General ”Or”
Rule If outcomes A and B are mutually exclusive, then the
Bayes’s Rule probability of A or B equals the probability of A plus the
Bayesian
Updating
probability of B, that is: P r(A ∨ B) = P r(A) + P r(B).
Discussion

6 / 20
 Examples

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability

Conditional
• A roulette wheel has spaces numbered from 1–36
Probability
and the
alternating red and black plus two additional spaces, 0 and
General 00, both colored green. Thus, there are 18 red spaces, 18
”And” Rule
black spaces, and 2 green spaces. What is:
Total
Probability – The probability of getting black?
and the
General ”Or” – The probability of getting black or green?
Rule – The probability of spinning an odd number, 0, or 00?
Bayes’s Rule

Bayesian
Updating

Discussion

7 / 20
 Rules of (Unconditional) Probability, cont.
Introduction

Fundamentals
of Probability
Theory
The ”Everything” Rule
Unconditional The probability of the entire outcome space equals 1.
Probability

Conditional
Probability The ”Not” Rule
and the
General
”And” Rule
The probability that some outcome A will not occur equals 1
Total
minus the probability that it does, that is:
Probability
and the
P r(¬A) = 1 − P r(A).
General ”Or”
Rule

Bayes’s Rule The ”And” Rule

Bayesian If outcomes A and B are independent, the probability of A and
Updating

Discussion
B equals the probability of A times the probability of B, that
is:P r(A&B) = P r(A) × P r(B).

8 / 20
 More Examples

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability

Conditional
Probability • Consider the roulette wheel again. What is:
and the
General – The probability of not getting a number between 1 and 10?
”And” Rule
– The probability of getting red, black, or green?
Total
Probability – The probability of getting black on the first spin and green
and the
General ”Or”
on the second spin?
Rule

Bayes’s Rule

Bayesian
Updating

Discussion

9 / 20
 Conditional Probability

Introduction

Fundamentals
of Probability
Theory

Unconditional Definition of Conditional Probability

Probability

Conditional The probability that something happens given that something

Probability
and the else has happened is called a conditional probability.
General
”And” Rule

Total • We denote probability of A given C, that is probability of

Probability
and the A conditional on C, as P r(A|C)
General ”Or”
Rule • Notice that P r(A|C) is not the same thing as P r(C|A).
Bayes’s Rule
Though these two probabilities may be identical, they need
Bayesian
Updating not be.
Discussion

10 / 20
 Conditional Probability, cont.

Introduction

Fundamentals
• Example: Suppose that you draw one card from a well-shuffled deck, and
of Probability that you are interested in the probability of drawing the ace of spades given
Theory
that you draw an ace.
Unconditional
Probability • Answer: Given that you just drew an ace, there are four possibilities: the
Conditional ace of spades, ace of clubs, ace of hearts, or ace of diamonds. Because only
Probability
and the one of the four is the ace of spades, and because all four outcomes are
General
”And” Rule equally likely, this probability is 1/4.
Total • You can get the same answer by dividing the probability that you draw the
Probability
and the ace of spades by the probability that you draw an ace: 1/52 divided by
General ”Or” 4/52, which is 1/4.
Rule

Bayes’s Rule

Bayesian
Formal Definition of Conditional Probability
Updating
If A and B are two outcomes, the probability of A
Discussion
conditional of B equals the probability of A and B divided
by the probability of B, that is: P r(A|B) = P Pr(A&B)
r(B)
.
11 / 20
 The General ”And” Rule and Independence
Conditions
Introduction

Fundamentals The General ”And” Rule

of Probability
Theory The probability of A and B equals the probability of A
Unconditional conditional on B times the probability of B, that is:
Probability
P r(A&B) = P r(A|B) × P r(B).
Conditional
Probability
and the
General • This rule is a generalization of the previously stated ”and”
”And” Rule

Total
rule because it doesn’t require A and B to be independent.
Probability
and the
• The following equivalence holds:
General ”Or”
Rule
P r(A|B) × P r(B) = P r(B|A) × P r(A)
Bayes’s Rule

Bayesian
• Also, the following independence conditions are logically
Updating
equivalent: A and B are independent if
Discussion
i. P r(A|B) = P r(A)
ii. P r(B|A) = P r(B)
iii. P r(A&B) = P r(A) × P r(B)
12 / 20
 Total Probability

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability

Conditional
Probability
and the
The Rule of Total Probability
General
”And” Rule P r(D) = P r(D&B) + P r(D&¬B)
Total or
Probability
and the P r(D) = P r(D|B) × P (B) + P r(D|¬B) × P r(¬B)
General ”Or”
Rule

Bayes’s Rule

Bayesian
Updating

Discussion

13 / 20
 The General ”Or” Rule

Introduction

Fundamentals The General ”Or” Rule

of Probability
Theory The probability of A or B equals the probability of A plus the
Unconditional probability of B minus the probability of A and B, that is:
Probability

Conditional
P r(A ∨ B) = P r(A) + P r(B) − P r(A&B).
Probability
and the
General
”And” Rule
• This rule is a generalization of the previously stated ”or”
Total rule because it doesn’t require A and B to be mutually
Probability
and the
exclusive.
General ”Or”
Rule • Example: Suppose you roll two dice. What is the
Bayes’s Rule probability that the first roll results in 2 or the second roll
Bayesian results in 2 (That is 2 comes up at least once)?
Updating

Discussion
• Property: P r(A ∨ B) = 1 − P r(¬A & ¬B)
• Example contd: Answer the question in the above
example using the property above.
14 / 20
 Bayes’s Rule

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability Bayes’s Rule
Conditional P r(D&B)
Probability P r(B|D) = P r(D)
and the
General
”And” Rule or
Total P r(D|B)×P r(B)
Probability P r(B|D) = P r(D)
and the
General ”Or”
Rule
or
P r(D|B)×P r(B)
Bayes’s Rule P r(B|D) = P r(D|B)×P (B)+P r(D|¬B)×P r(¬B)
Bayesian
Updating

Discussion

15 / 20
 Example

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability

Conditional
Probability • Ten percent of a company’s employees are smokers. For
and the
General each nonsmoker, the chance of taking a sick day in the
”And” Rule
next year is 0.01. For each smoker, the chance of taking a
Total
Probability sick day is 0.05. Given that a worker is sick, what is the
and the
General ”Or” chance he was a smoker?
Rule

Bayes’s Rule

Bayesian
Updating

Discussion

16 / 20
 Bayesian Updating

Introduction

Fundamentals
• We update beliefs in light of new evidence all the time.
of Probability
Theory
• Bayes’s rule is often interpreted as describing how we
Unconditional should update our beliefs in light of new evidence.
Probability
• Suppose you have a belief or hypothesis. The question is
Conditional
Probability how you should change your belief – that is, the probability
and the
General that you assign to the possibility that the hypothesis is
”And” Rule
true – in light of the fact that the evidence obtains.
Total
Probability • Let H stand for the hypothesis and E for the evidence.
and the
General ”Or” • The probability of H,P r(H), is called the prior
Rule
probability: it is the probability that H is true before you
Bayes’s Rule
learn whether E is true.
Bayesian
Updating • The probability of H given E, P r(H|E), is called the
Discussion posterior probability: it is the probability that H obtains
given that the evidence E is true. The question is what
the posterior probability should be.
17 / 20
 Bayesian Updating

Introduction

Fundamentals
of Probability
Theory
• The last question on the previous slide is answered by a
Unconditional
Probability simple application of the Bayes’s rule:
Conditional
Probability
and the P r(E|H) × P r(H)
General P (H|E) =
”And” Rule P r(E|H) × P r(H) + P r(E|¬H) × P r(¬H)
Total
Probability • Thus Bayes’s rule tells you that the probability you assign
and the
General ”Or” to H being true should go from P r(H), prior probability
Rule

Bayes’s Rule
to P r(H|E), posterior probability.
Bayesian • Bayesian updating is the process of updating beliefs in
Updating
accordance with Bayes’s Rule.
Discussion

18 / 20
 Bayesian Updating, an Example

Introduction

Fundamentals
of Probability
Theory

Unconditional
Probability
• Wes is trying to determine whether or not a coin has two
Conditional
Probability
and the
heads. He cannot see the coin, he only knows the outcome
General
”And” Rule
after it is flipped. He believes with 20% confidence that it
Total
has two heads. The first flip is heads. What is his updated
Probability
and the
probability? The second flip is also head. What is his
General ”Or”
Rule
updated probability? The third flip is tails. What is
Bayes’s Rule
updated probability?
Bayesian
Updating

Discussion

19 / 20
 Discussion

Introduction
• In this chapter we have explored the theory of probability.
Fundamentals
of Probability • Probability theory is critical to a wide range of applications, among other
Theory
things as the foundations of statistical inference.
Unconditional • It is relevant here because it can be interpreted as a theory of judgment,
Probability

Conditional
that is, as a theory of how to revise beliefs in light of evidence.
Probability • How plausible is this theory?
and the
General • Again, we must separate the descriptive question from the normative
”And” Rule
question. Do people in fact update their beliefs in accordance with Bayes’s
Total
Probability
rule? Should they?
and the • The axioms might seem weak and uncontroversial, both from a descriptive
General ”Or”
Rule and from a normative standpoint.
Bayes’s Rule • Yet the resulting theory is anything but weak. As in the case of the theory
Bayesian of choice under certainty, we have built a remarkably powerful theory on
Updating
the basis of a fairly modest number of seemingly weak axioms.
Discussion • The theory is not intended to describe the actual cognitive processes you go
through when updating your beliefs.
• But it does specify exactly how your posterior probability must relate to
your prior, given various conditional and unconditional probabilities. 20 / 20