EE891 STOCHASTIC

SYSTEMS

Lecture 2
 Outline
◦ This lecture is an introduction to elementary
probability theory

◦ Probability Model
◦ Categorize the observations
◦ Assign probabilities to the categories
◦ Axioms of Probability
◦ Corollaries
◦ Joint Probability
◦ Conditional Probability
 Definition of Probability
◦ Probability is the measure of the likeliness that
an event will occur
◦ There is a 20% chance of rain
◦ Rain is an event
◦ 20% is a “measure of the likeliness” that rain will
occur (probability)

◦ Probability is quantified as a number between

0 and 1
◦ 0 indicates impossibility
◦ 1 indicates certainty
Probability Model
 Probability Model
◦ Components of a Real world experiment:
Experiment: Procedure + Observation

◦ Probability model: A simplified math model to

study/analyze the experiment

◦ Components of a probability model:

1. Categorize the observations
2. Assign probabilities to the categories
Probability Model
◦ Categorizing the Observations

◦ Outcome (𝝃) : is one of the possible observations

◦ An outcome cannot be further decomposed into
other outcomes

◦ Sample space (Ω or 𝑆): set of all possible outcomes

◦ Event (𝐴): Set of outcomes that are of interest to us

◦
Probability Model
◦ Consider a multiuser uplink communication scenario

◦ Assume 6 mobile phone users need to send data to the BS

◦ If only one channel is available for transmission, then we
have to select only one user for transmission
◦ User Selection scheme
◦ User is selected at random
◦ Assume selection scheme is fair
 Probability Model
Experiment
Outcome
Procedure (selected user
(Select a user at random) [u1,u2,u3,u4,u5,u6] )

◦ Probability model:
◦ Outcome (𝝃) : 𝑢1 , 𝑢2 … 𝑢6
◦ An outcome cannot be further decomposed into other outcomes
◦ Sample space (𝑆): 𝑆 𝑢1, 𝑢2, 𝑢3, 𝑢4, 𝑢5, 𝑢6
◦ The sample space is the set of all possible outcomes, called S
◦ Events (𝐴,𝐵): Set of outcomes that are of interest to us
◦ A = {𝜉: such that user id is an even number}
◦ 𝐵 {𝜉: such that user id is greater than 4}
 Probability Model
◦ In terms of a Venn diagram our experiment will look like

u5

u6
u2 u4

u3
u1
S
◦ So far we have just seen how to analyze the observations
from an experiment
◦ We have not assigned any probabilities yet
 Probability Model
◦ Assigning probabilities
◦ Probability is assigned to outcomes and
events

◦ Probability is “a function of an event that

produces a numerical quantity that measures
the likelihood of that event”

◦ Multiple ways to define such a function

◦ The function has to follow some rules!
◦ Axioms of Probability
 Axioms of Probability
1. Probability of any event is non-negative

2.
◦ The sample space includes all possible
outcomes. Probabilities are normalized so that
the maximum value is unity.

3. The probability of the union of mutually

exclusive events is equal to the sum of their
probabilities
Axioms of Probability
◦

A u5
u1

u6
u2 u4
B

S u3

◦ Generalization: If are mutually exclusive

events then
Corollaries of Axioms of
Probability
Corollaries of Axioms of Probability
1. All probabilities are between 0 and 1

◦ Proof follows from Corollary 1

Corollaries of Axioms of Probability

3.
Corollaries of Axioms of Probability
4. Discarding the condition of exclusivity i.e.
◦ If then
+ -

u5

u6
u2 u4

u1 u3

S
Corollaries of Axioms of Probability
Corollaries of Axioms of Probability
5. Partition Property: For any event , and disjoint
partition set we can write
Corollaries of Axioms of Probability
Corollaries of Axioms of Probability
6. If then
Assigning Probabilities
 Assigning Probabilities

◦ Any assignment that satisfies the given axioms is

acceptable

◦ Probability assignment actually represents the likelihood

of occurrence of that event (should be intuitive)
Assigning Probabilities
◦ Two techniques are typically used

◦ Classical approach:
◦ atomic outcomes: decomposed events into simplest
events
◦ assigning equal probabilities to all atomic outcomes
in an experiment

◦ Relative Frequency Approach

Probability Model
◦ Consider a multiuser uplink communication scenario

◦ Assume 6 mobile phone users need to send data to the BS

◦ If only one channel is available for transmission, then we
have to select only one user for transmission
◦ User Selection scheme
◦ User is selected at random
◦ Assume selection scheme is fair
 Assigning Probabilities -
Classical approach
◦ Example: User Selection
◦ Six users are equally likely to be selected

◦ Probability assignment is
1
Pr 𝑢1 Pr 𝑢2 Pr 𝑢6
6

◦ From this assignment we can determine the probability

of more complicated events
Assigning Probabilities -
Classical approach
◦ Probability that the user selected has an even id

Pr 𝑈𝑠𝑒𝑟 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 ℎ𝑎𝑠 𝑒𝑣𝑒𝑛 𝑖𝑑

Pr 𝑢2 𝑖𝑠 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 𝑂𝑅 𝑢4 𝑖𝑠 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑 𝑂𝑅 𝑢6 𝑖𝑠 𝑠𝑒𝑙𝑒𝑐𝑡𝑒𝑑

Pr 𝑢2 ∪ 𝑢4 ∪ 𝑢6

Pr 𝑢2 Pr 𝑢4 Pr 𝑢6 𝑏𝑦 𝐴𝑥𝑖𝑜𝑚 3

1 1 1
𝑏𝑦 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑎𝑠𝑠𝑖𝑔𝑛𝑚𝑒𝑛𝑡
6 6 6

1/2.
 Assigning Probabilities -
Classical approach
◦ Drawbacks:
◦ Not suitable for unequal probability assignment

◦ Experiment: Measure Height of Students in a Room

◦ Very tall or very short students are less probable than those
of medium height

◦ Overcome by using the relative frequency approach.

 Assigning Probabilities:
Relative frequency approach
◦ Requires that the experiment is repeatable
◦ Repeat experiment number of times, say 𝑛 times
◦ Count how many times the event 𝐴 actually occurs, say
𝑛 times
◦ Probability of the event A can be assigned as
Joint Probability
Joint Probability
◦ If 𝐴 ∩ 𝐵 𝜙 i.e. Events 𝐴 and 𝐵 have no common
outcome then the Probability of Event 𝐴 or 𝐵 is
Pr 𝐴 ∪ 𝐵 Pr 𝐴 Pr 𝐵 A u1 u5

u6
u4
B u2
S u3

◦ If 𝐴 ∩ 𝐵 𝜙 i.e. Events 𝐴 and 𝐵 have common outcome(s)

then the Probability of Event 𝐴 or 𝐵 is
u5
Pr 𝐴 ∪ 𝐵 Pr 𝐴 +Pr 𝐵 -Pr 𝐴 ∩ 𝐵 u6
u2 u4

u1 u3
S

◦ Probability of Events 𝐴 and 𝐵 occurring?

◦ Notation: Pr 𝐴 ∩ 𝐵 Pr 𝐴, 𝐵
◦ This probability is termed as joint probability
Joint Probability: Probability of
Events and occurring
◦ If , i.e. events and are
disjoint/mutually exclusive
A u1 u5

u6
u4
B u2
S u3

◦ Then probability of events and happening at

the same time is impossible. Thus,
 Joint Probability: Probability of
Events and occurring
◦ If , then the probability of this event can
be calculated using either u5

u6
u2 u4

u1 u3
S

◦ Relative Frequency Approach: Let 𝑛 , be the number of

times 𝐴 and 𝐵 simultaneously occurs in 𝑛 trials, then

𝑛 ,
Pr 𝐴, 𝐵 lim
→ 𝑛

◦ Classical Approach: Assign probabilities to atomic

outcomes and calculate joint probability based on that
 Joint Probability u5

u6
◦ Classical Approach u2 u4

◦ Consider user selection example u3

u1
S
◦ Assigned probabilities to atomic outcomes were
1
Pr 𝑢1 Pr 𝑢2 Pr 𝑢6
6
◦ Based on these probabilities the joint probability of
events 𝐴 and 𝐴 can be calculated as
1
Pr 𝐴 , 𝐴 Pr 𝐴 ∩ 𝐴 Pr 𝑢6
6
Conditional Probability
 Conditional probability in
Daily life
◦ Chance/probability of raining given overcast conditions

◦ Chance/Probability of a person getting affected by X

disease given Y symptoms

◦ Chance/Probability of Pakistan cricket team winning

given it plays India in Worldcup

◦ Chance/Probability of a surprise quiz given no quiz for

last 5 lectures
 Conditional Probability
◦ Given that event B has already occurred, what is the
probability that event A will occur?
◦ Given that event B has already occurred, reduces the
sample space of A

u5 Event B has u5
already
occurred
u6 u6
u4 => u2, u4, u3, u1 u2 u4
u2
cannot occur
u3 u3
u1 u1
S S
 Conditional Probability
u5

u6
u2 u4

u3
S u1

◦ Given that event 𝐵 has already occurred, the new

conditional sample space only contains 𝐵’s outcomes
𝑆 𝑢5, 𝑢6
◦ The new event space for 𝐴 is the intersection of 𝐴 and 𝐵:
Event space → 𝐸 | 𝐴∩𝐵 𝑢6
◦ How does this effect the probability?
 Conditional Probability
◦ Experiment is performed 𝑛 times
◦ Event 𝐵 occurs 𝑛 times
◦ Events A and B occurs 𝑛 ∩ times
◦ Number of times that event 𝐴 occurs when B has occurred
is 𝑛 ∩
◦ Probability of event 𝐴 given event 𝐵 has occurred is
𝑛 ∩
using relative frequency
𝑛
𝑛 ∩ /𝑛 𝑃 𝐴 ∩ 𝐵
→ 𝑃 𝐴𝐵
𝑛 /𝑛 𝑃 𝐵
 Conditional Probability
◦ For two events A and B:
◦ The probability of event 𝐴 conditioned on knowing that
event 𝐵 has occurred is

◦ Rearranging yields

◦ In some cases, the conditional probabilities are easier to

compute than the corresponding joint probabilities
 Conditional Probability
◦ For any events 𝐴 and 𝐵 such that Pr 𝐴 0
◦ The probability of event 𝐵 conditioned on knowing that
event 𝐴 has occurred is
Pr 𝐴 ∩ 𝐵
Pr 𝐵 𝐴
Pr 𝐴
Rearranging yields
(1)
From prev slide
(2)
 Conditional Probability

◦ Equating equations (1) and (2) yields

(3)

◦ Rearranging yields
(4)

(5)
 Conditional Probability
◦ Useful for calculating certain conditional
probabilities

◦ In many problems, it may be quite difficult to

compute directly, whereas calculating
may be straightforward
 Conditional Probability -
Properties
◦ Conditional probability satisfies the axioms

◦ Probability of an event conditioned on its self is one

◦ If event is a subset of event , Probability of event

A given event B has occurred is

◦ For mutually exclusive events

Implying
 Example
◦ Assume that two bits are transmitted from the Tx to the Rx
◦ Bit is rcvd correctly (c), Bit is rcvd in error (e).
◦ 4 possible Outcomes: 𝑆 𝑒𝑒, 𝑒𝑐, 𝑐𝑒, 𝑐𝑐
Pr 𝑒𝑒 0.01, Pr 𝑒𝑐 0.01, Pr 𝑐𝑒 0.01, Pr 𝑐𝑐 0.97
◦ Find the probability of events
◦ A = “second bit is in error”
◦ B = “first bit is in error”
◦ conditional probability that the second bit is in error given that the first bit
is in error
◦ Solution
◦ Let 𝐴 denote the event that the second bit is in error
◦ 𝐴 𝑒𝑒, 𝑐𝑒
◦ Let B denote the event that the first bit is in error
◦ 𝐵 𝑒𝑒, 𝑒𝑐
 Solution
◦ A “second bit is in error”
◦𝐴 𝑒𝑒, 𝑐𝑒
◦ 𝐴 is the union of two disjoint events (outcomes) 𝑒𝑒 and
𝑐𝑒
Pr 𝐴 Pr 𝑒𝑒 Pr 𝑐𝑒 0.02

◦ B “first bit is in error”

◦𝐵 𝑒𝑒, 𝑒𝑐
◦ 𝐵 is the union of two disjoint events (outcomes) 𝑒𝑐 and
𝑒𝑒
Pr 𝐵 Pr 𝑒𝑒 Pr 𝑒𝑐 0.02
 Solution
◦ A “second bit is in error”
◦ B “first bit is in error”

◦ Conditional probability that the second bit is in error given that

the first bit is in error
Pr 𝐴|𝐵
◦ Using definition
Pr 𝐴 ∩ 𝐵
Pr 𝐴 𝐵
Pr 𝐵
◦ Event 𝐴 ∩ 𝐵 𝑒𝑒 ; implies Pr 𝐴 ∩ 𝐵 Pr 𝑒𝑒 0.01
∩ .
Pr 𝐴 𝐵 0.5
.
 Example

◦ Shuffle a deck of cards and select a card

◦ What is the conditional probability the selected card is

the ace of clubs given that the selected card is a black
card?
 Solution
◦ No. of cards in a Deck = 52
◦ Each card has equal probability of getting selected, Pr 𝐶𝑎𝑟𝑑

◦ Probability that the card is ace of club,Pr 𝐴𝑐𝑒 𝐵𝑙𝑎𝑐𝑘 Pr 𝐶𝑎𝑟𝑑

◦ Probability that a card is black is Pr 𝐵𝑙𝑎𝑐𝑘

◦ Probability that that selected card is Ace of Black given that the
selected card is black is,
∩
◦ Pr 𝐴𝑐𝑒 𝑜𝑓 𝐵𝑙𝑎𝑐𝑘|𝐵𝑙𝑎𝑐𝑘 𝐶𝑎𝑟𝑑 𝑆𝑒𝑙𝑒𝑐𝑡𝑒𝑑
1
Pr 𝐴𝑐𝑒 𝑜𝑓 𝐵𝑙𝑎𝑐𝑘 52 1
Pr 𝐵𝑙𝑎𝑐𝑘 𝐶𝑎𝑟𝑑 𝑆𝑒𝑙𝑒𝑐𝑡𝑒𝑑 26 26
52
 Generalization of Conditional
Probability
◦ The idea can be generalized to 3 events

◦ In general, for 𝑀 events