MA-2203: Introduction to Probability and Statistics

Lectures notes
by
Dr. Suchandan Kayal

Department of Mathematics
National Institute of Technology Rourkela
Rourkela - 769008, Odisha, India
Autumn, 2020
Outline

Motivation and preliminary notions

Methods of assigning probabilities
Classical method
Relative frequency method
Axiomatic approach to probability
Motivation

Probability has its origin in the study of gambling and

insurance in seventeenth century.
The theory of probability is now an indispensible tool of
both the social and natural sciences.
A gambler’s dispute in 1654 led to the creation of the
probability theory by two French mathematicians Fermat
and Pascal.
Their motivation came from a problem related to gambling
proposed by the nobleman Chevalier De Mere. There was
an apparent contradiction concerning a popular dice game.
The game consisted in throwing a pair of dice 24 times.
The problem was to decide whether or not to bet even
money on the occurence of at least one ‘double six’ during
the 24 throws.
Motivation (contd...)

A seemingly well-established gambling rule led De Mere to

believe that betting on a ‘double six’ in 24 throws would be
profitable. But his own calculations indicate just the
opposite.
This problem and others posed by de Mere led to an
exchange of letters between Pascal and Fermat in which
the fundamental principles of probability theory were
formulated for the first time.
Because of the inherent appeal of games of chance,
probability theory soon became popular and the subject
developed rapidly during the 18th century.
Motivation (contd...)

One of the fundamental features of probability is that the

phenomena we are interested are random in nature.
Tossing of a coin
Birth of a offspring
The long term behaviour of an event is known as statistical
regularity. This encourages to study the subject
probability.
Basic notions

Experiment
An experiment is observing something happen or conducting
something under certain conditions which result in some
outcomes.

Example
Rainfall: It is a consequence of several things such as cloud
formation, elnino occurrence, humidity, atmospheric pressure
etc. Finally, we observe that there is rainfall. Thus, observing
weather is an experiment.

Types of experiment
Deterministic experiment: It results known outcomes under
certain conditions.
Random experiment: Under fixed conditions, the outcomes
are not known.
Basic notions

Random experiment
An experiment is said to be a random experiment if the
following conditions are satisfied.
The set of all possible outcomes of the experiment is known
in advance.
The outcomes of a particular performance (trial) of the
experiment can not be predicted in advance.
The experiemnt can be repeated under identical conditions.

Sample space
The collection of all possible outcomes of a random experiment
is called the sample space. It is denoted by Ω.
Methods of assigning probabilities

A. Classsical approach
Assumptions:
A random experiment results in a finite number of equally
likely outcomes.
Let Ω = {ω1 , · · · , ωn } be a finite sample space with n ∈ N
possible outcomes, N denotes the set of natural numbers.
For a subset E of Ω, |E| denotes the number of elements in
E.
Result:
The probability of occurrence of an event E is given by

# of outcomes favourable to E |E| |E|

P (E) = = = .
Total # of outcomes in Ω |Ω| n
Methods of assigning probabilities/Classsical approach
(contd...)

Observations
For any event E, P (E) ≥ 0
For mutually exclusive events E1 , · · · , En ,
∑n
| ∪ni=1 Ei | ∑
n ∑
n
i=1 |Ei | |Ei |
P (∪ni=1 Ei ) = = = = P (Ei )
n n i=1
n i=1

|Ω|
P (Ω) = |Ω| = 1.
Methods of assigning probabilities/Classsical approach
(contd...)
Example-1
Suppose that in your section, we have 150 students born in
the same year. Assume that a year has 365 days. Find the
probability that all the students of your section are born on
different days of the year.

Solution
Denote the event that all the students are born on different
days of the year by E. Here,

|Ω| = 365140 and |E| = 365 × 364 × · · · × 266 = 365P 140 .

Thus,
|E| 365
P 140
P (E) = = .
|Ω| 365140
Methods of assigning probabilities/Classsical approach
(contd...)
Example-2
Find the probability of getting exactly two heads in three
tosses of a fair coin.

Solution
Denote the event that getting exactly two heads in three
tosses of a fair coin by E. Here,

Ω = {HHH, HHT, HT H, T HH, T HT, T T H, HT T, T T T }

and
E = {HHT, HT H, T HH}.
Thus,
|E| 3
P (E) = = .
|Ω| 8
Methods of assigning probabilities/Classsical approach
(contd...)

Drawbacks
The random experiment must produce equally likely
outcomes.
The total number of outcomes of the random experiment
must be finite.
Methods of assigning probabilities

A. Relative frequency approach

Assumptions:
Suppose that a random experiment can be repeated
independently (the outcome of one trial is not affected by
the outcome of another trial) under identical conditions.
Let an denote the number of times (frequency) an event E
occurs in n trials of a random experiment.
Result:
Using weak law of large numbers, under mild conditions, it
can be shown that the relative frequencies an /n stabilize in
certain sense as n gets large.
an
P (E) = lim , provided the limit exists.
n→∞ n
Methods of assigning probabilities/Relative frequency
approach (contd...)

Observations
For any event E, P (E) ≥ 0
For mutually exclusive events E1 , · · · , En ,
( n )
∪ ∑
n
P Ei = P (Ei )
i=1 i=1

P (Ω) = 1.
Methods of assigning probabilities/Relative frequency
Example-3
After tossing a fair coin, we have the following outputs:

HHT HHT HHT HHT · · ·

Using relative frequency approach, find P (H).

Solution
Note that 


2k−1
k = 1, 2, · · ·
an 1 2 2 3 4 4  3k−2 ,
= , , , , , , · · · = 3k−2
2k
, k = 1, 2, · · ·
n 1 2 3 4 5 6 

 2k , k = 1, 2, · · ·
3k

an 2
Thus, lim = = P (H).
n→∞ n 3
Methods of assigning probabilities/Relative frequency
approach (contd...)

Drawbacks
The probability has been calculated based on an
approximation.
The random experiment has to be conducted a large
number of times. This is not always possible since some
experiments are costly (launching satellite).
√
n
lim = 0 ⇒ P (E) = 0 (not correct !).
n→∞ n
√
n− n
lim = 1 ⇒ P (E) = 1 (not correct !).
n→∞ n
Axiomatic approach to probability

Basic concepts
A set whose elements are themselves set is called a class of
sets. For example, A = {{2}, {2, 3}}.
A set function is a real-valued function whose domain is a
class of sets.
A sigma-field of subsets of Ω is a class F of subsets of Ω
satisfying the following properties:
(i) Ω ∈ F
(ii) E ∈ F ⇒ E c = Ω − E ∈ F (closed under complement)
(iii) Ei ∈ F , i = 1, 2, · · · ⇒ ∪∞
i=1 Ei ∈ F (closed under countably
infinite unions)
F = {ϕ, Ω} is a sigma (trivial) field.
Suppose A ⊂ Ω. Then, F = {ϕ, Ω, A, Ac } is a sigma field of
subsets of Ω.
Axiomatic approach to probability (contd...)

Definition
Let Ω be a sample space of a random experiment. Let F be the
event space or a sigma field of subsets of Ω. Then, a probability
function or a probability measure is a set function P , defined on
F, satisfying the following three axioms:
For any event E ∈ F , P (E) ≥ 0 (nonnegativity)
For a countably infinite collection of mutually exclusive
events E1 , E2 , · · · , we have
∞
∪ ∞
∑
P( Ei ) = P (Ei )
i=1 i=1

(countably infinite additive)

P (Ω) = 1 (Probability of the sample space is one)
Axiomatic approach to probability (contd...)

Consequences of the axiomatic definition

Let (Ω, F, P ) be a probability space. Then,
(i) P (ϕ) = 0
(ii) for all E ∈ F , 0 ≤ P (E) ≤ 1 and P (E c ) = 1 − P (E)
(iii) for n mutually exclusive events Ei , i = 1, · · · , n,

∪
n ∑
n
P( Ei ) = P (Ei )
i=1 i=1

(iv) Let E1 .E2 ∈ F and E1 ⊂ E2 . Then,

P (E2 − E1 ) = P (E2 ) − P (E1 ) and P (E1 ) ≤ P (E2 )
(v) For E1 , E2 ∈ F , P (E1 ∪ E2 ) = P (E1 ) + P (E2 ) − P (E1 ∩ E2 )

Proof
See it during lecture.
Inequalities

Boole’s inequality (union bound proposed by George Boole)

Let (Ω, F, P ) be a probability space and let E1 , · · · , En ∈ F ,
where n ∈ N. Then,
∪
n ∑
n
P( Ei ) ≤ P (Ei ).
i=1 i=1

Proof
See it during the lecture.

Note
To prove Boole’s inequality for the countable set of events, we
can use ∪ni=1 Ei → ∪∞
i=1 Ei for n → ∞ along with the continuity
of the probability measure P.
Inequalities (contd...)

Bonferroni’s inequality
Let (Ω, F, P ) be a probability space and let E1 , · · · , En ∈ F ,
where n ∈ N. Then,
( n )
∩ ∑
n
P Ei ≥ P (Ei ) − (n − 1).
i=1 i=1

Proof
See it during the lecture.

Note
The Bonferroni’s inequality holds only for the probability of
finite intersection of events!
Conditional probability

Example
Let us toss two fair coins. Let A denote that both coins show
same face and B denote at least one coin shows head. Obtain
the probability of happening of A given that B has already
occured.

Solution
Listen to my lecture.

Definition
Let (Ω, F, P ) be a probability space and B ∈ F be a fixed event
such that P (B) > 0. Then, the conditional probability of event
A given that B has already occured is defined as

P (A ∩ B)
P (A|B) = .
P (B)
Conditional probability (contd...)

Example
Six cards are dealt at random (without replacement) from a
deck of 52 cards. Find the probability of getting all cards of
heart in a hand (event A) given that there are at least 5 cards
of heart in the hand (event B).

Solution
Clearly,
(13) (13)(39) (13)
+
P (A ∩ B) = P (A) = 6
(52) and P (B) = 5 1
(52) 6
.
6 6

Thus, (13)
P (A|B) = (13)(39)6 (13) .
5 1 + 6
Conditional probability (contd...)

Note
For events E1 , E2 · · · , En ∈ F , n ≥ 2, we have
P (E1 ∩ E2 ) = P (E1 )P (E2 |E1 ) if P (E1 ) > 0
P (E1 ∩ E2 ∩ E3 ) = P (E1 )P (E2 |E1 )P (E3 |E1 ∩ E2 ) if
P (E1 ∩ E2 ) > 0. This condition also gurantees that
P (E1 ) > 0, since E1 ∩ E2 ⊂ E1
P (∩ni=1 Ei ) =
P (E1 )P (E2 |E1 )P (E3 |E1 ∩E2 ) · · · P (En |E1 ∩E2 ∩· · ·∩En−1 ),
provided P (E1 ∩ E2 ∩ · · · ∩ En−1 ) > 0, which also
guarantees that P (E1 ∩ E2 ∩ · · · ∩ Ei ) > 0, for
i = 1, 2, · · · , n − 1.
Conditional probability (contd...)

Example
An urn contains four red and six black balls. Two balls are
drawn successively, at random and without replacement, from
the urn. Find the probability that the first draw resulted in a
red ball and the second draw resulted in a black ball.

Solution
Let A denote the event that the first draw results in a red ball
and B that the second ball results in a black ball. Then,
4 6 12
P (A ∩ B) = P (A)P (A|B) = × = .
10 9 45
Total probability

Theorem of total probability

Let (Ω, F, P ) be a probability space and let {Ei ; i ∈ A} be a
countable collection of mutually exclusive and exhaustive events
(that is, Ei ∩ Ej = ϕ for i ̸= j and P (∪i∈A Ei ) = P (Ω) = 1) such
that P (Ei ) > 0 for all i ∈ A. Then, for any event E ∈ A,
∑ ∑
P (E) = P (E ∩ Ei ) = P (E|Ei )P (Ei ).
i∈A i∈A

Proof
Let B = ∪i∈A Ei . Then, P (B) = P (Ω) = 1 and
P (B c ) = 1 − P (B) = 0.
Thank You