DSC6132: Probability and Statistical Modelling

DSC6132: Probability and Statistical Modelling
Lecture 1: Probability Calculus
Dr. Joseph Nzabanita and Dr. Annie Uwimana
University of Rwanda
ACE-DS
Semester 1, 2020-2021
Dr. Joseph Nzabanita and Dr. Annie Uwimana DSC6132 Part 1-Probability, MSc. in DS
Outline 1
Sample space and events

Probability space
Finite probability space
Conditional probability
Independent events
Law of total probability
Bayes Theorem
Introduction 2
Probability theory is the branch of mathematics concerned with

the analysis of random phenomena.
The interpretation of the word probability involves synonyms

such as chance, odds, uncertainty, prevalence, risk, expectancy,
etc.
We use probability when we want to make an affirmation, but

we are not quite sure.
Introduction (cont.) 3
There are many applications of probability theory.
Probability theory is especially fundamental to Mathematical

Statistics.
Mathematical Statistics is concerned with the development of

methods and their applications for collecting, analysing and
interpreting quantitative data in such a way that the reliability
of a conclusion based on data may be evaluated objectively by
means of probability statements.
Probability theory is used to evaluate the reliability of

conclusions and inferences based on data.
Introduction (cont.) 4
The mathematical theory of probability has its roots in

attempts to analyze games of chance by Gerolamo Cardano in
the sixteenth century, and by Pierre de Fermat and Blaise
Pascal in the seventeenth century.
Initially, probability theory mainly considered discrete events,

and its methods were mainly combinatorial. Eventually,
analytical considerations compelled the incorporation of
continuous variables into the theory.
This culminated in modern probability theory, on foundations

laid by Andrey Nikolaevich Kolmogorov.
Random experiment 5
A random (statistical) experiment is an experiment in which:

All possible outcomes of the experiment are known in
advance.
Any performance of the experiment results in an outcome
that is not known in advance.
The experiment can be repeated under the same conditions.
Sample space 6
An outcome is the result of random experiment.
The set of all possible outcomes of a random experiment is

called a sample space and is usually denoted by S.
A sample space may be discrete (if it has a finite or countably

infinite number of outcomes) or continuous (if it is
uncountably infinite).
Event 7
An event is any subset of the sample space.
Any event which consists of a single outcome in the sample

space is called an elementary or simple event.
Events which consist of more than one outcome are called

compound events.
Examples 8
Experiment: rolling a die once.
S = {1, 2, 3, 4, 5, 6}.
Experiment: tossing a coin three times.

Sample space: Denote head by H and tail by T .
S = {HHH, HHT, HT H, T HH, HT T, T HT, T T H, T T T }.
Events could be
A :“The head occurred twice”:
A = {HHT, HT H, T HH}.
B=“The number of heads is greater than the number of
tails”:
B = {HHH, HHT, HT H, T HH}.
Examples (cont.) 9
C :“The tail occurred twice”:

C = {HT T, T HT, T T H}.
Experiment: measuring the height (cm) of a girl on her
first day at school.
Sample space:
S = the set of all possible real numbers.
In the two first examples the sample space is discrete whereas in

the last one it is continuous.
Operations on events 10
Set theory is used to represent relationships among events.

∅ (the empty set): “An event that never occur in a random
experiment” is called an impossible event.
In the previous example the event:

“The head occurred twice and the tail occurred twice” is
impossible.
S (the sample space): “ an event that is certain to occur”.
Operations on events (cont.) 11
In general, if A and B are two events in the sample space S,

then
A ∪ B (A union B) = “either A or B occurs or both occur”.
Example In the previous example the event

A ∪ C : “The head occurred twice or the tail occurred
twice”:
A ∪ C = {HHT, HT H, T HH, HT T, T HT, T T H}.
A ∩ B (A intersection B) = “both A and B occur”.
Example
In the previous example the event A ∩ C : “The head
occurred twice and the tail occurred twice”:
A ∩ C = ∅.
Two events A and B are incompatible or mutually

exclusive if A ∩ B = ∅.
A ⊂ B (A is a subset of B) = “if A occurs, so does B”.

Example
In the previous example the event A implies the event B:
A ⊂ B.
Ā or Ac (The complement of A in S) = “The event that
occurs if and only if A does not”.
Example
In the previous example the event B̄:“The number of heads
is less than the number of tails”:
B̄ = {HT T, T HT, T T H, T T T }.
Note that B ∪ B̄ = S.
A\B or A − B (the difference) = “The event which occurs

whenever A occurs and B does not”.
Example
In the previous example the event
B\A = {HHH}.
We have
A\B = A ∩ B̄
and
A∆B = (A\B) ∪ (B\A) = (A ∩ B̄) ∪ (B ∩ Ā).
Figure 1: Venn diagrams for union and intersection
Figure 2: Venn diagrams for complement and difference
Figure 3: Venn diagram illustrations of A, B disjoint, and A ⊂ B
Properties 17
If A, B, C ∈ S and ∅ -impossible event, then

P1 : S̄ = ∅ and ¯∅ = S;
P : Ā¯ = A;
2
P3 : A ⊂ A ∪ B and B ⊂ A ∪ B;
P4 : A ∩ B ⊂ A and A ∩ B ⊂ B;
P5 : A ∩ B ⊆ A ∪ B;
P6 : A ∩ A = A and A ∪ A = A;
P7 : A ∩ B = B ∩ A and A ∪ B = B ∪ A; (commutative laws)
P8 : (A ∩ B) ∩ C = A ∩ (B ∩ C) and (A ∪ B) ∪ C = A ∪ (B ∪ C);
(associative laws)
Properties (cont.) 18
P9 : A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C) and
A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C); (distributive laws)
P10 : A ∩ ∅ = ∅ and A ∪ ∅ = A;
P11 : A ∩ S = A and A ∪ S = S;
P12 : A ∩ Ā = ∅ and A ∪ Ā = S;
P13 : If B ⊂ A, then A ∩ B = B and A ∪ B = A;
P14 : A ∪ B = Ā ∩ B̄ and A ∩ B = Ā ∪ B̄; (De Demorgan’s Laws)
Algebra of events 19
Let S be a sample space. A non-empty set A of events of S is

an algebra of S if:
1 ∀A ∈ A, Ā ∈ A;
2 ∀A, B ∈ A, A ∪ B ∈ A.
As a consequence, if A is an algebra of S, then
∅ ∈ A and S ∈ A;
∀A1 , A2 , . . . , An ∈ A, ∪ni=1 Ai ∈ A and ∩ni=1 Ai ∈ A.
σ−Algebra of events 20
When the sample space is infinite, the operations can be carried

over a countable infinite number of events.
Let A be an algebra of S.
A is a σ−Algebra or σ−Field of S if for any countable infinite
sequence of events A1 , A2 , . . . , An , . . . of A, then
∪∞
i=1 Ai ∈ A.
As a consequence
∩∞
i=1 Ai ∈ A.
In this case, the couple (S, A) is called a measurable space.
KOLMOGOROV’s axioms of probability 21
Let S be a sample space and A a σ−Algebra of S. A

probability measure (or simply a probability) is a function
P : A → R which assigns a real number to every event of A and
satisfies the following three axioms:
Axiom 1 P (A) ≥ 0 for any event A ∈ A;
Axiom 2 P (S) = 1;
P∞
Axiom 3 P(∪∞i=1 Ai ) = i=1 P (Ai ) if A1 , A2 , . . . , Ai , . . . are mutually
exclusive events of S.
Probability space 22
The triplet (S, A, P ) is called a probability space.
Any set function which satisfies the above three axioms is a

probability measure on S.
For a given sample space S, there may be more than one

probability measure.
The probability of an event A is the value of the probability

measure at A, i.e., P (A).
Properties of a probability 23
P1 P (∅) = 0.
P2 Let {A1 , A2 , . . . , An } be a finite collection of n events such
that Ai ∩ Aj = ∅ for i 6= j. Then
n
X
P (∪ni=1 Ai ) = P (Ai ).
i=1
P3 P (Ā) = 1 − P (A).
P4 If A ⊆ B ⊆ S, then P (A) ≤ P (B).
P5 If A is any event in S, then 0 ≤ P (A) ≤ 1.
P6 If A and B are any two events, then
P (A ∪ B) = P (A) + P (B) − P (A ∩ B).
P7 If A1 and A2 are two events such that A1 ⊆ A2 , then
P (A2 \A1 ) = P (A2 ) − P (A1 )
Finite probability space 24
Let S be a finite sample space, say S = {s1 , s2 , . . . , sn };

A = PS , the set of all sub-sets of S. Then we know that the
cardinal of PS is 2n , we write
#PS = 2n .
The function
#A
P : PS −→ R, A 7→ P (A) = ,
#S
allows us to construct a probability on (S, PS ) and the triplet

(S, PS , P ) is called a finite probability space.
Exercise: Show that this way of defining a probability verifies

the axioms of Kolmogorov.
Finite probability space (cont.) 25
An assignment of probability is said to be equally likely (or

uniform) if each elementary event in S is assigned the same
probability. Thus, if S contains n elementary events
si , P ({si }) = 1/n, i = 1, ..., n.
With this assignment we get the classical model of probabilities

number of simple events in A
P (A) = .
number of simple events in S
Self reading: Section 1.4 in Rohatgi and Saleh (2001).
Conditional probability 26
Let (S, A, P ) be a probability space and B ∈ A, such that

P (B) > 0. Then, the set function
P (A ∩ B)
PB : A −→ R, A 7→ PB (A) = ,
P (B)
is a probability called conditional probability.
Next, PB (A) will be denoted P (A/B) and we read “ probability

of A given B” but (B/A) is not an event.
P (A/B) is the probability that A occurs given that B has

occurred.
Exercise: Check that the conditional probability verifies the

axioms of Kolmogorov.
Properties 27
P1 . ∀A, B ∈ A, with P (A) > 0 and P (B) > 0,
P (A ∩ B) = P (A)P (B/A) = P (B)P (A/B).

P2 . ∀A, B ∈ A, such that B ⊂ A and P (B) > 0,
P (A/B) = 1.
Indeed,
P (A ∩ B) P (B)
P (A/B) = = = 1.
P (B) P (B)
Example 28
An urn contains two white balls and three black balls.
We draw two balls from the urn at random.
Find the probability that both balls are white.
Example 29
Answer:
Denote the events

A1 : “The 1st ball is white”; A2 : “The 2nd is white”.
Then
A1 ∩ A2 : “The two balls are white”.
We have
P (A1 ∩ A2 ) = P (A2 /A1 )P (A1 ) = 0.1,
where P (A1 ) = 2/5 and P (A2 /A1 ) = 1/4.
Independent events 30
Two events A and B are independent if P (A/B) = P (A) (or

P (B/A) = P (B)). In other words the occurring of A does not
influence that of B and vice versa.
In the case of independence, we have the theorem of

multiplication of probabilities.
Two events A and B of a sample space S are independent if and

only if
P (A ∩ B) = P (A)P (B).
Independent events (cont.) 31
In general, if A1 , A2 , ..., An is any sequence of events, then
P (A1 ∩ A2 ∩ · · · ∩ An ) = P (A1 )P (A2 /A1 )P (A3 /(A1 ∩ A2 )) · · ·

P (An / ∩n−1
i=1 Ai ).
In the case where events A1 , A2 , ..., An are independent,
P (A1 ∩ A2 ∩ · · · ∩ An ) = P (A1 )P (A2 ) · · · P (An ).
Remark
If A and B are two incompatible events, then they cannot be
independent, unless P (A)P (B) = 0.
Some properties of the independence of events 32
P1 If A and B are independent events, then
P (A ∪ B) = P (A) + P (B) − P (A)P (B).
P2 If A and B are events, then the following assertions are

equivalent
(i) A and B are independent.
(ii) A and B̄ are independent.
(iii) Ā and B are independent.
(iv) Ā and B̄ are independent.
Exercise: Prove these properties.
Example 33
The probability that the machine ”a” fails during any day is
0.025; similarly for the machine ”b” that probability is 0.015.
Given that the machines operate independently, find
(a) The probability that both machines fail simultaneously.
(b) The probability that no machine fails.
(c) The probability that one or the other has a fault.
Example 34
Answers
Let A :“The machine ”a” fails” and B :“The machine “b” fails”.
Then, P (A) = 0.025, P (B) = 0.015.
(a) P (A ∩ B) = P (A)P (B) = 0.0004
(b) P (Ā ∩ B̄) = (1 − P (A))(1 − P (B)) = 0.96
(c) P (A ∪ B) = P (A) + P (B) − P (A)P (B) = 0.0396
Law of Total Probability 35
Let {Bi }ni=1 be a collection of mutually exclusive and exhaustive

events of S. That is,
Bi ∩ Bj = ∅ if i 6= j and ∪ni=1 Bi = S.
We say that the events Bi constitute a partition of the sample

space S.
Now, let A be an arbitrary event and assume

P (Bi ) > 0, i = 1, ..., n. Then,
n
X n
X
P (A) = P (A ∩ Bi ) = P (A/Bi )P (Bi ).
i=1 i=1
Law of Total Probability-Example 36
Consider three identical baskets.
The 1st contains two white balls and one black, the 2nd
contains three white and one black, the 3rd contains two white
and 2 black.
We choose a basket at random and draw one ball. Find the

probability that this ball is white.
Law of Total Probability-Example (cont.) 37
Answer:
Define the events:
B1 : “the 1st basket is chosen”, B2 : “the 2nd basket is chosen”,
B3 : “the 3rd basket is chosen”, A: “the drawn ball is white.
We have that B1 ∪ B2 ∪ B3 = S, A ⊂ S and
P (B1 ) = P (B2 ) = P (B3 ) = 31 .
Using the law of total probability
P (A) = P (A/B1 )P (B1 ) + P (A/B2 )P (B2 ) + P (A/B3 )P (B3 )

2 1 3 1 2 1
= × + × + × ≈ 0.64
3 3 4 3 4 3
Bayes Theorem 38
With the same conditions in the law of total probability,

suppose that we wish to calculate P (Bi /A), for i = 1, ..., n.
Then, we have
P (A/Bi )P (Bi )
P (Bi /A) = Pn .
k=1 P (A/Bk )P (Bk )
This formula is called Bayes’ formula.
We have also, Bayes’ rule:
P (A/B)P (B)
P (B/A) = ,
P (A)
if P (A)P (B) > 0.
Bayes Theorem-Example 39
A device can be made up with high quality parts or parts of

ordinary quality.
In the 1st case reliability, i.e., the probability of operation

without failure is 0.95. In the second case this probability is 0.7.
Approximately 40% of devices are manufactured with high

quality parts.
A device has been tested and proved good. Find the probability
that it is made up with high quality parts.
Bayes Theorem-Example (cont.) 40
Answer:
Define the following events
B1 : “A device is made up with parts of high quality”,
B2 : “A device is made up with parts of ordinary quality”,
A: “The drawn device is good”.
We are asked to determine P (B1 /A).
From the of Bayes formula, we get
P (A/B1 )P (B1 )
P (B1 /A) = = 0.475,
P (A/B1 )P (B1 ) + P (A/B2 )P (B2 )
in which we used, P (B1 ) = 0.4, P (B2 ) = 0.6, P (A/B1 ) = 0.95

and P (A/B2 ) = 0.7.
Exercises 41
Separate sheet!

DSC6132: Probability and Statistical Modelling

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DSC6132: Probability and Statistical Modelling

Uploaded by

Copyright:

Available Formats

DSC6132: Probability and Statistical Modelling

Lecture 1: Probability Calculus

Dr. Joseph Nzabanita and Dr. Annie Uwimana

Sample space and events

Probability theory is the branch of mathematics concerned with

The interpretation of the word probability involves synonyms

We use probability when we want to make an affirmation, but

There are many applications of probability theory.

Probability theory is especially fundamental to Mathematical

Mathematical Statistics is concerned with the development of

Probability theory is used to evaluate the reliability of

The mathematical theory of probability has its roots in

Initially, probability theory mainly considered discrete events,

This culminated in modern probability theory, on foundations

A random (statistical) experiment is an experiment in which:

An outcome is the result of random experiment.

The set of all possible outcomes of a random experiment is

A sample space may be discrete (if it has a finite or countably

An event is any subset of the sample space.

Any event which consists of a single outcome in the sample

Events which consist of more than one outcome are called

Experiment: rolling a die once.

Experiment: tossing a coin three times.

S = {HHH, HHT, HT H, T HH, HT T, T HT, T T H, T T T }.

C :“The tail occurred twice”:

In the two first examples the sample space is discrete whereas in

Set theory is used to represent relationships among events.

In the previous example the event:

In general, if A and B are two events in the sample space S,

Example In the previous example the event

Two events A and B are incompatible or mutually

A ⊂ B (A is a subset of B) = “if A occurs, so does B”.

A\B or A − B (the difference) = “The event which occurs

A∆B = (A\B) ∪ (B\A) = (A ∩ B̄) ∪ (B ∩ Ā).

Figure 1: Venn diagrams for union and intersection

Figure 2: Venn diagrams for complement and difference

Figure 3: Venn diagram illustrations of A, B disjoint, and A ⊂ B

If A, B, C ∈ S and ∅ -impossible event, then

Let S be a sample space. A non-empty set A of events of S is

When the sample space is infinite, the operations can be carried

In this case, the couple (S, A) is called a measurable space.

Let S be a sample space and A a σ−Algebra of S. A

The triplet (S, A, P ) is called a probability space.

Any set function which satisfies the above three axioms is a

For a given sample space S, there may be more than one

The probability of an event A is the value of the probability

P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

P7 If A1 and A2 are two events such that A1 ⊆ A2 , then

P (A2 \A1 ) = P (A2 ) − P (A1 )

Let S be a finite sample space, say S = {s1 , s2 , . . . , sn };

allows us to construct a probability on (S, PS ) and the triplet

Exercise: Show that this way of defining a probability verifies

An assignment of probability is said to be equally likely (or

With this assignment we get the classical model of probabilities

Self reading: Section 1.4 in Rohatgi and Saleh (2001).

Let (S, A, P ) be a probability space and B ∈ A, such that

is a probability called conditional probability.

Next, PB (A) will be denoted P (A/B) and we read “ probability

P (A/B) is the probability that A occurs given that B has

Exercise: Check that the conditional probability verifies the

P1 . ∀A, B ∈ A, with P (A) > 0 and P (B) > 0,

P (A ∩ B) = P (A)P (B/A) = P (B)P (A/B).

An urn contains two white balls and three black balls.