Professional Documents
Culture Documents
!
ENCS%6161/4%*%Probability%and%Stochastic%Processes%–%Fall%2015%
!
Instructor:!Dr.!Hassan!Rivaz,!EV5.154,!hrivaz@ece.concordia.ca!
http://users.encs.concordia.ca/~hrivaz/!!
Lectures:!Hall!540,!Mondays!5:45PM!to!8:15PM!
Office%hours:!Fridays!9:30AM!to!12noon.!
Textbook:!Alberto!LeonLGarcia,!Probability,!Statistics,!and!Random!Processes!For!Electrical!
Engineering,!3rd.!Ed.!Prentice!Hall,!2008.!
Syllabus:!
1. Introduction:!Motivation!and!Overview.!Axioms!of!Probability.!Independent!Events.!Total!
Probability.!Conditional!Probability.!Bay’s!Theorem.!Bernoulli!Trials!(Chapters!1!and!2).!
2. Concepts!of!Discrete!and!Continuous!Random!Variables!(RV).!Probability!Mass!Functions!(PMF)!
and!Probability!Density!Functions!(PDF),!Cumulative!Distribution!Functions!(CDF).!Conditional!
Distribution!and!Probability!Density!Functions!of!Random!Variables!(Chapters!3!and!4).!
3. Multiple!random!variables,!joint!PDF,!conditional!probability!and!conditional!expectation,!
correlation!and!covariance,!joint!Gaussian!random!variables!(Chapters!5!and!6).!
4. Vector!Random!Variables.!Sums!of!Random!Variables.!LongLTerm!Averages.!Central!Limit!
Theorem,!Confidence!Intervals!(Chapters!7,8)!
5. Random!Processes,!Mean,!AutoL!correlation!and!CrossLcorrelation,!Autocovariance,!Ergodicity,!
Stationarity!(Chapter!9).!
6. Power!Spectral!Density,!Response!of!linear!systems!to!random!signals!(Chapter!10).!
7. Markov!Chains!(Chapter!11)!
Midterm:!Monday!Oct!26,!in!class.!No!makeLup!tests!offered.!
Grading%Schemes:%
1. Assignments!6%,!Midterm!30%,!Final!64%!
2. Assignments!6%,!Midterm!0%,!Final!94%!
!The!better!mark!will!be!used!to!calculate!the!final!grade.!
Assignments:!
Assignment!1:!posted!on!Mon!Sept!28.!Due!on!Mon!Oct!5!at!the!beginning!of!the!lecture.!!
Assignment!2:!posted!on!Mon!Nov!16.!Due!on!Mon!Nov!23!at!the!beginning!of!the!lecture.!!
Violation!of!the!Academic!Code!of!Conduct!in!any!form!will!be!severely!dealt!with!(see!
http://www.concordia.ca/students/academicLintegrity.html).!You!must!sign!the!following!
form!and!hand!it!in!with!assignment!1:!
http://www.concordia.ca/content/dam/encs/docs/ExpectationsLofLOriginalityLFeb14L2012.pdf!!
Most%slides%are%graciously%
provided%by%Dr.%Dongyu%Qiu%
Importance of probability
• Artificial intelligence (Google, Facebook,
self-driving cars, face recognition)
ultrasound elastography
file:///Users/hrivaz/Dropbox/Teaching_CV/01_courses/new_medical_imag/application_slot_course/index.html 1/2
Chapter 2
Basic Concepts of Probability
Theory
ENCS6161 - Probability and Stochastic
Processes
Concordia University
Specifying Random Experiments
Examples of random experiments: tossing a coin,
rolling a dice, the lifetime of a harddisk.
Sample space: the set of all possible outcomes of a
random experiment.
Sample point: an element of the sample space S
Examples: S = {H, T }
S = {1, 2, 3, 4, 5, 6}
S = {t|1 < t < 10}
Event: a subset of a sample space A ⊆ S
A = {H}
A = {2, 4, 6}
A=∅
ENCS6161 – p.1/16
The Axioms of Probability
A probability measure is a set function P (·) that
satisfies the following axioms.
1. P (A) ≥ 0
2. P (S) = 1
3. If A ∩ B = ∅, then P (A ∪ B) = P (A) + P (B)
4. If A1 , A2 , · · · are events s.t Ai ∩ Aj = ∅ for all i ̸= j
! ∞ " #∞
then P ∪k=1 Ak = k=1 P (Ak )
Corollary 1: P (Ac ) = 1 − P (A)
Corollary 2: P (A) ≤ 1 ∀A
Corollary 3: P (∅) = 0
ENCS6161 – p.2/16
The Axioms of Probability
Corollary 4: If A1 · · · An are mutually exclusive, i.e.
Ai ∩ Aj = ∅, ∀i ̸= j then P (∪nk=1 Ak ) = nk=1 P (Ak )
#
A B
A∩B
ENCS6161 – p.3/16
Mutually Exclusive Events
Pictures from
http://www.mathsisfun.com/data/probability-events-mutually-exclusive.html
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Counting Techniques
• There are three special rules, or counting
techniques, used to determine the number of
outcomes in events.
• They are :
1. Multiplication rule
2. Permutation rule
3. Combination rule
• Each has its special purpose that must be
applied properly – the right tool for the right
job.
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Counting – Multiplication Rule
• Multiplication rule:
– Let an operation consist of k steps and there
are
• n1 ways of completing step 1,
• n2 ways of completing step 2, … and
• nk ways of completing step k.
– Then, the total number of ways to perform k
steps is:
• n1 · n2 · … · nk
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Counting Sample Points
Permutation: n distinct objects, how many ways can
we arrange them?
n · (n − 1) · (n − 2) · · · 2 · 1 = n!
Selection with order: select k objects from n ≥ k
objects
n!
n · (n − 1) · · · (n − k + 1) = = n Pk
(n − k)!
Combination (selection without order): select k
objects from n ≥ k objects $ %
n n! n
Ck = =
(n − k)!k! k
ENCS6161 – p.4/16
Counting Sample Points
Example: choose a president and a treasurer from 50
students
1. no restrictions
2. A will serve only if he is president
3. B & C will serve together or not at all
4. D & E will not serve together
ENCS6161 – p.5/16
Example 2-7 : Hospital Schedule
• In a hospital, a operating room needs to schedule
three knee surgeries and two hip surgeries in a
day. The knee surgery is denoted as k and the hip
as h.
– How many sequences are there?
Since there are 2 identical hip surgeries and 3
identical knee surgeries, then
5! 5 ⋅ 4 ⋅ 3!
= = 10
2!⋅3! 2 ⋅1⋅ 3!
– What is the set of sequences?
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Example 2-7 : Hospital Schedule
• In a hospital, a operating room needs to schedule
three knee surgeries and two hip surgeries in a
day. The knee surgery is denoted as k and the hip
as h.
– How many sequences are there?
Since there are 2 identical hip surgeries and 3
identical knee surgeries, then
5! 5 ⋅ 4 ⋅ 3!
= = 10
2!⋅3! 2 ⋅1⋅ 3!
– What is the set of sequences?
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Example 2-8: Sampling w/o Replacement-1
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Example 2-8: Sampling w/o Replacement-3
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Conditional Probability
The conditional prob. of A given B has occurred is
defined by :
P (A ∩ B)
P (A|B) = , P (B) > 0
P (B)
Similarly,
P (A ∩ B)
P (B|A) = , P (A) > 0
P (A)
So
ENCS6161 – p.6/16
Bayes’ Rule
If B1 · · · Bn are mutually exclusive and ∪Bi = S, we
call these sets a partition of S.
Theorem of total probability
For any event A, if B1 , · · · , Bn is a partition of S,
P (A) = P (A ∩ B1 ) + P (A ∩ B2 ) + · · · + P (A ∩ Bn )
= P (A|B1 )P (B1 ) + P (A|B2 )P (B2 ) + · · · + P (A|Bn )P (Bn )
Bayes’ Rule
Let B1 , · · · , Bn be a partition of S , and P (A) ≥ 0, then
P (A ∩ Bj ) P (A|Bj )P (Bj )
P (Bj |A) = = n
#
P (A) k=1 P (A|Bk )P (Bk )
ENCS6161 – p.7/16
Bayes’ Rule
Example: 2% of people have on type of blood
disease. If a person has the disease and take a
blood test, with 96% probability, the result is positive
and with 4% probability, the result is negtive. If a
person without the disease takes a blood test, then
94% negative and 6% positive. Find
P (have disease|positive)
ENCS6161 – p.11/16
Bayes’ Rule
Solution: Let
D: a person has the disease D′ : no disease
B : blood test positive B ′ : blood test negative
Then P (D) = 2% and
P (B|D) = 0.96 P (B ′ |D) = 0.04
P (B|D′ ) = 0.06 P (B ′ |D′ ) = 0.94
Apply Bayes Rule:
ENCS6161 – p.12/16
Bayes’ Rule
We can also use tree diagram.
D’ ′ ′
0.94 B’ 0.98 ∗ 0.94 = 0.9212 (D ∩ B )
P (D ∩ B) 0.0192
P (D|B) = = = 0.246
P (B) 0.078
ENCS6161 – p.13/16
Independence of Events
A and B are independent iff
P (A ∩ B) = P (A)P (B)
P (A ∩ B)
P (A|B) =
P (B)
P (A)P (B)
=
P (B)
= P (A)
Similarly
P (B|A) = P (B)
ENCS6161 – p.15/16
Independence of Events
Events A1 , A2 , · · · , An are independent if
ENCS6161 – p.16/16
Conditional Probability
• P(B | A) is the probability of event B
occurring, given that event A has already
occurred.
• Example:
S={1,2...6} in throwing a die
B={2}. P(B) = 1/6
A={1}. P(B| A) = ?
( ) (
P D ' | F = P D '∩ F ) P (F ) = 30
400
40
400
= 30
40