Probability & Random Variables

Probability

Probability
Probability theory is the branch of math that deals with random
events, processes, and variables

• What does randomness mean to you?

• How would you define probability in your own words?

01:830:400 Spring 2019 2

 Probability

Probability
Probability theory is the branch of math that deals with random
events, processes, and variables

• Probability is used to describe how likely a particular outcome

is in a random event
– the probability of obtaining “heads” in a single coin flip
– the probability of getting into a car accident today

• Probabilities are always between 0 and 1

– A probability near 1 indicates that the event is very likely
– A probability near 0 indicates that the event is very unlikely

01:830:400 Spring 2019 3

 Probability

Interpretations of Probability
• Classical or Analytic view
– Definition of probability in terms of an analysis of possible (and equally likely)
outcomes for a set of events. I.e., p=(favorable cases/all possible cases)
– Examples:
• Proportion of ways in which two 6-sided dice can yield 10 total dots
• Proportion of 5-card combinations that can form a royal flush

• Frequentist view
– Definition of probability in terms of past performance
• p=(f/N)
– Example:
• If we roll a die 1000 times and it comes up 6 250 times, then we estimate the probability
of rolling a 6 as 0.25 (Note: such a die is probably loaded)

• Subjective or Bayesian view

– Probability as “degree of belief”
– Examples:
• Probability that a particular candidate will win the next election
• Probability that the die in the example above is fair
• Probability that a sound I hear in the middle of the night is caused by a burglar

01:830:400 Spring 2019 4

 Probability

Basic Terminology
• Trial: one of a number of repetitions of an experiment
– E.g, a coin flip, die roll, or measurement procedure

• Event (Variable): the outcome (value) of a trial

– E.g., heads or tails, the showing face on a die, the value of the measurement

• Disjoint (mutually exclusive) events: events related such that the

occurrence of one precludes the occurrence of the other
– E.g., heads vs. tails for a single coin flip, color of M&M for a single draw from a bag,
the suit of a card on a single draw

• Sample space (exhaustive set): the set of all possible outcomes

– E.g., heads & tails, all possible M&M colors, all possible card suits

• Complementary events: a set of two disjoint events that form an

exhaustive set
– E.g., the outcomes “1” and “not 1” in a single dice roll

01:830:400 Spring 2019 5

 Probability

Axioms of Probability (Kolmogorov)

A function P() on a sample space Ω is a probability measure if:

1. The measure of the function on the entire sample space is 1

P   1

2. The measure of any subset of the sample space is greater than or

equal to 0
P  A   0, A  

3. The measure of the union of any set of mutually exclusive events in

the sample space is equal to the sum of the measures of the individual
events
P  B  C   P( B )  P (C ), B, C   s.t. B 
 C

01:830:400 Spring 2019 6

 Probability

Mutually Exclusive Events Intersecting Events

All Events All Events

Set A Set B Set A A∩B Set B

All Events
Set C

Note: Here, Set C

Mutually Exclusive & is meant to indicate
Set A Set B items that are
Exhaustive Set neither in A nor in B

01:830:400 Spring 2019 7

 Probability

Standard Deck of Cards

52 cards: 4 suits x 13 ranks variables: suit, color, rank,

rank type (face vs. number)

01:830:400 Spring 2019 8

 Probability

Basic Terminology
• Marginal probability: the probability of one event, ignoring
the occurrence or nonoccurrence of other (simultaneous)
events
– Denoted: P(event1)
– Examples: P(height>68”), P(suit=clubs)

All Events

P(A)

01:830:400 Spring 2019 9

 Probability

Marginal Probability: P(diamond)

P(diamond) = 13/52 = 0.25

01:830:400 Spring 2019 10

 Probability

Basic Terminology
• Joint probability: the probability of (simultaneous)
occurrence of two or more events
– Denoted: P(event1,event2) or P(event1 ∩ event 2)
– Examples: P(height>68”, gender=female), P(suit = clubs, rank = king)

Mutually Exclusive Events Intersecting Events

All Events All Events

Set A Set B Set A P(A,B) Set B P(A,B)

P(A,B) = 0 P(A,B) > 0

01:830:400 Spring 2019 11

 Probability

Joint Probability: P(diamond, face card)

P(diamond) = 13/52 = 0.25

P(face card) = 12/52 = 0.231
P(diamond,face card) = 3/52 = 0.058
01:830:400 Spring 2019 12
 Probability

Basic Terminology
• Conditional probability: the probability that one event will
occur given the occurrence of some other event
– Denoted: P(event1 | event2)
– Examples: P(height>68” | gender=female), P(suit=clubs | color=black)

All Events
P(A|B) = P(A,B)/P(B)

P(A,B)

Set A P(A,B) Set B

P(B)

01:830:400 Spring 2019 13

 Probability

Conditional Probability: P(diamond | face card)

P(diamond | face card) = 3/12 = 0.25

01:830:400 Spring 2019 14

 Probability

Conditional Probability: P(face card | diamond)

P(face card | diamond) = 3/13 =

0.231

01:830:400 Spring 2019 15

 Probability

Basic Terminology
• Independent events (variables): events (variables) related
such that the occurrence of one has no effect on the
probability of occurrence of the other
– E.g., successive coin flips, successive M&M draws from bag sampled
with replacement
– A special category of independent events is IID or independent and
identically distributed events
• Many of the techniques in this course depend on IID assumptions

• Dependent events (variables): events (variables) related

such that the occurrence of one affects the probability of
occurrence of the other
– E.g., single die outcome in pair & sum of die outcomes, successive
M&M draws from bag sampled without replacement

01:830:400 Spring 2019 16

 Probability

A Note on “Replacement”
• Most of the tools that we use in statistics assume very large
populations

• We normally sample from populations without replacement

• However, as population size (N) grows, the effects of

withdrawing individual scores disappears, allowing us to treat
individual scores as IID
– Statistics for IID samples are much easier to compute

01:830:400 Spring 2019 17

 Probability

Independent Events: (diamond, face card)

P(diamond) = 13/52 = 0.25

P(diamond | face card) = 3/12 = 0.25

01:830:400 Spring 2019 18

 Probability

Dependent Events: (diamond,red card)

P(diamond) = 13/52 = 0.25

P(diamond | red card) = 13/26 = 0.5

01:830:400 Spring 2019 19

 Probability

Basic Laws of Probability

• Additive law: Given a set of mutually exclusive events (e.g.,
{A,B,C}), the combined probability of occurrence of any of the
events in the set (e.g., P(A or B or C)) is equal to the sum of
their separate probabilities.
– Examples:
• For a fair flipped coin, the probability of heads is 0.5 and the probability of
tails is 0.5. Since these outcomes are mutually exclusive, the probability of
obtaining either heads or tails is 0.5+0.5 = 1
• For a fair 6-sided die, the probability of occurrence of each face is 1/6. Since
the die can only land on one face at a time (the face outcomes are mutually
exclusive), the probability of obtaining a 4 or a 5 or a 6 is (3*1/6) = ½.

– Note: the combined probability of occurrence of any of the events in a

set is always greater than (or equal to) the probability of occurrence of
any individual event within the set

01:830:400 Spring 2019 20

 Probability

Additive Law: P(diamond or spade)

P(diamond) = 13/52 = 0.25

P(spade) = 13/52 = 0.25
P(diamond or spade) = P(diamond) +
01:830:400 Spring 2019 P(spade) = 0.5 21
 Probability

Basic Laws of Probability

• Multiplicative law: Given a set of independent events (e.g.,
{D,E,F}), the combined probability of occurrence of all the
events in the set (P(D,E,F))is the product of their individual
probabilities.
– Example:
• Successive coin flips: the probability of a particular sequence (e.g., H,T) is
equal to the product of the individual probabilities (e.g., P(H,T) = P(H)P(T) =
0.5 x 0.5 = 0.25)

– Note: because probabilities are [0,1], the combined probability of

occurrence all of the events in a set of independent events is always
smaller than (or equal to) the probability of any individual event in the
set

01:830:400 Spring 2019 22

 Probability

Multiplicative Law: P(diamond, face card)

P(diamond) = 13/52 = 0.25

P(face card) = 12/52 = 0.231
P(diamond,face card) = P(diamond) P(face
01:830:400 Spring 2019 card) = 0.058 23
 Probability

Generalizing the Additive & Multiplicative Laws

• The book simplifies things by giving you rules only for
mutually exclusive or independent events.
• However, both laws can be written more generally using
conditional and joint probabilities

• Generalized Additive Law: P(A or B) = P(A)+P(B) – P(A,B)

– Note: if A and B are not mutually exclusive, then we have to subtract off
their intersection (A∩B) to keep from counting it twice

• Generalized Multiplicative Law: P(A,B) = P(A|B)×P(B)

– Note: if A and B are independent, then P(A|B) = P(A), which is how we
get the law for the independent case.

01:830:400 Spring 2019 24

 Probability

Generalized Additive Law (number or red card)

P(red card) = 26/52

P(number card) = 40/52
P(number,red card) = 20/52
01:830:400 Spring 2019 25
 Probability

Relating Joint and Conditional Probabilities

Given a set of two simultaneous outcomes (e.g., gender and
smoking) the joint probability of both outcomes is equal to the
product of the conditional probability of one event (conditioned
on the other) multiplied by the marginal probability of the other.

Example:

P  smoke, female   P  smoke | female  P  female  Equivalently,

or P  smoke, female 
P  female | smoke  
P  smoke, female   P  female | smoke  P  smoke  P  smoke 

Bayes’ Rule

01:830:400 Spring 2019 26

 Probability

Bayes’ Rule (A Practical Example)

Lupus is an (overdiagnosed) autoimmune disorder whose prevalence in the
population is estimated about 2%

The most common test for lupus correctly identifies the disease in 98% of
people who have it and correctly rejects the disease in 74% of people who do
not have it.

• A patient tests positive for lupus. What is the probability that the
patient has the disease?

Remind me to say something here about the

distinction between parameters and variables

01:830:400 Spring 2019 27

 Probability

Bayes’ Rule (A Practical Example)

P  lupus   0.02 P  lupus, test 
P  lupus | test  
P  test|lupus   0.98 P  test 
P  test|lupus   0.74
P  lupus  P  test|lupus 

P  lupus, test   P  lupus, test 

P  lupus   1  P  lupus   0.98 P  lupus  P  test|lupus 

P  test|lupus   1  P  test|lupus   0.26

P  lupus  P  test|lupus   P  lupus  P  test|lupus 

0.02(0.98)
P  lupus | test    0.07
0.02(0.98)  0.98(0.26)

“It’s never lupus.”

–House, MD

01:830:400 Spring 2019 28