Lecture1 PDF

Introduction to
Probability
INSTRUCTOR: SHAHROKH FARAHMAND
Basic definitions
Probability space:
 Consists of sample space, sigma-field of events, probability function

 Sample space: All possible outcomes of a particular experiment
 Basic Event: Each member of which can be an outcome of the
experiment
 Event: A collection of basis events and a subset of sample space, after an
experiment is carried out an event occurs if the experiment yields one of the
basic events that are present in that particular event
 Example: finite sample space
 Number of distinct E=
 Sure event E=
 Empty event=
SLIDES
SLIDES BY:BY SHAHROKH
SHAHROKH FARAHMAND
FARAHMAND 2
Examples
Ex1. One coin toss
Ex2. Two coin tosses
Ex3. Number of hairs on a person’s head
Ex4. Age of married couples
Ex5. Power across a resistor
SLIDES BY: SHAHROKH FARAHMAND

SLIDES BY: SHAHROKH FARAHMAND 3
Various scenarios for sample space
Sample space is a finite set
 No need to define sigma-fields concept
 Use all possible subsets of as the sigma field of events
 Probability can be assigned in a consistent manner to all events (clarified
later on)
Sample space is an infinite countable set
 Need sigma-field concept
 Probability can be assigned in a consistent manner to al events
Sample space is infinite uncountable
 Need sigma-field concept
 Probability can not be assigned in a consistent manner to all sets (Cantor set)
 Sets of importance to us are consistent
SLIDES BY:BY:
SLIDES SHAHROKH FARAHMAND
SHAHROKH FARAHMAND 4
Set Algebra
Consider E,F to be two sets
Well-known properties
DeMorgan Laws:
Generalization:

Sigma-field for a sample space
Consider a sample space and a collection of its subsets as
is a field if
1)
2)
3)
Note that set of all subsets for a finite sample space is a field
Furthermore, is a sigma-field if:

Kolmogorov probability axioms
Remember
P[.] is a function that assigns to every event a number P[E]
called the probability of E such that

Some properties resulting from axioms

Examples for probability spaces
Ex1. One coin toss
Ex2. Tossing of a fair die

Examples (continued)
Ex3. Picking at random a ball from balls numbered 1-12 from the urn
Ex4. Toss of two fair dies

Motivation for union bound
Question. Relationship between independent sets and disjoint sets?
Exact solution is complicated and of limited use. Apply union bound instead
Union bound
Bound is tight when sets are disjoint

Note that RHS should be less than one for non-triviality
Wide applications in performance analysis

Union bound application
Ex. Digital communications: 4-QAM (4-PSK) through AWGN
Complex model but I,Q components are real

Suppose is transmitted

Application (continued)
Union bound leads to a simple PoE expression

When number of constellation points increases
 It becomes considerably more complex to apply the exact formula
 However, union bound remains almost as simple as 4-PSK

Joint probabilities
Frequency definition of probability
 Repeat independent experiments for a large number of times say n
 Count the number of times an event A has occurred
 We will have (approximately)
 Note that RHS is random but LHS is deterministic

 This observation will become rigorous later on via law of large numbers (LLN)
Joint probability
 Repeat the experiment n times but observe the number of times that both
events A and B have occurred

Joint probability example
Ex. Want to compute the probability for following events
 A: temperature equals or exceeds 10 C
 B: precipitation equals or exceeds 5 mm
 C: both A and B happen
We observed the weather for n=1000 days
Now, we can compute

Conditional probability
We want to know that given A has occurred what is the probability
that B occurs?
Use the same frequency interpretation: run n experiments, count
the number of occurrences of A or , count the number of
occurrences of both A and B
Hence, we define

Examples
Ex1. Weather forcast.
Ex2. Binary Symmetric Channel

(BSC)
 For example a B-PAM with hard
Thresholding
 Best & worst values instead of 0.9?

Examples: BSC

Body building contest
 N contestants with different scores
 Contestants pass in front of jury sequentially and randomly
Judges must decide on the spot if the current contestant is best
If they let him pass, they can not elect them as the winner later
Question. What is the best strategy to make sure best contestant is
selected? What is the corresponding probability of success?
Formulate a similar problem. N unknown numbers (contestant score is
summarized into one overall number) are placed in a bag and are drawn
out randomly
First, we limit ourselves to the following strategy:
 We let first contestants pass but record the largest score =
 Then we select the first contestant that appears with a score >

Body building contest 2
Define X to be the draw with the largest number coming out

To make the correct decision, one should have

Body building contest 3

Law of total probability
Consider mutually exclusive
events meaning
Then

Proof

Independence
Two events A and B are called independent if and only if (iff)
This means that occurrence of B does not have an impact on probability of

event A and vice versa
Ex. Tossing of two fair die
Question1. What about tossing of a fair and unfair die?
Question2. Relation between being disjoint and independent?
Independence also means

Independence 2

Example 1
Urn containing black/white, heavy/light, even/odd balls
A: picking a black ball

B: picking a light ball
C: picking an even numbered ball

Example 1 (continued)
Are these events independent?

Example 2: BSC
Are two events indep?
Question. Do you think this is good or bad?

Example 2: BSC
Instead of (0.9,0.1)
Consider (0.5,0.5)
Are two events indep?
Question. Do you think this is good or bad?

Bayes theorem
Proof
Then apply law of total probability

Example. BSC
What is the following probability?
Aposteriori vs. apriori probabilities

Example. Cancer test
A= the test states that tested person has cancer
B= person really has cancer
Need to evaluate E[B|A]:

Example. Guess the door
Bayes theorem offers the rigorous answer

Example. Guess the door 2
Lets say award is positioned behind a door at random
Similarly
Note that it is not guaranteed to win the award. Remember the frequency
interpretation of probability SLIDES BY: SHAHROKH FARAHMAND 35
Review of combinatorics
Ordered sample sets of size r (ex. CEO, VCEO, Secretary)
Suppose there are n numbered balls inside an urn and we take out r of
them at random.
How many cases are possible?
 Sampling with replacement
1 2 r
 Sampling without replacement
1 2 r

Review of combinatorics 2
Non-ordered samples of size r (Ex. Three board members)
Suppose there are n numbered balls inside an urn and we take out r of
them at random without replacement. How many cases are possible?
 Consider an unordered set of size r, how many ordered sets can be made
from that?
 Answer: r!
 Hence, number of unordered sets equals the number of ordered sets
divided by r!

Review of combinatorics 3
How many cases?
Ex. Tossing of 12 dies (E:= each face appears twice)

Occupancy problems
Problem1. Distributing randomly a set of r numbered balls into n
boxes. How many different cases?
Ex. r=3, n=2
Problem2. Distributing randomly a set of r similar balls into n boxes.

How many different cases?
Ex. r=3, n=2

Occupancy problems 2
Let us consider n boxes with n+1 lines:
Each ball is a *
Problem becomes number of possible orderings of r *’s and n-1 |’s
With fixed 2 lines at the ends, we want to select r members out of

n+r-1 elements

Example
For the case of r > n, determine the number of distinguishable cases
such that no box is empty?
To ensure n boxes are not empty, n *’s must be placed between each
two lines. Then, we need to select the position of r-n remaining balls
from (r-n)+n-1=r-1 positions, yielding

Bernoulli trials
Each experiment has two outcomes:
Success (s) with probability p
Failure (f) with probability q=1-p
Ex1. Flip of a coin

Ex2. Toss of a die with
Example. successive independent Bernoulli trials
3 Tosses of a biased coin: 8 possible outcomes

Example (continued)
What is the probability of two heads?
One of these three disjoint equiprobable events should happen
How to determine the number of events? Choose 2 out of 3
Sample space

Binomial law
N independent repetitions of a Bernoulli trial
Remember
Hence
Intuition: We have 0 s, or 1 s, or 2 s,…, or n s

Example1. Binary pulses

Example 2. Selecting an umpire

Example 3. Umpire selection time

Example 4. Missile strike

Poisson law
Proof:

Evaluation of RHS limit

Example. Computer failure

Example. Random points on a line

Properties of Poisson law

Example. Switchboard saturation
Note the relation between queuing theory and this example

Example. Repetition coding
Consider the BSC with P0=P1=0.5
Instead of a 0 Tx 000
Instead of a 1 Tx 111
Majority voting at the Rx
Note that X only has two states but
Y has 8 states

Example. Repetition coding 2
We arrive at this equivalent
channel
Note that the channel quality

improves threefold:
Remember
Question. What is the cost paid?
Note that BSC passes bits independently. But, if transmitted bits have a
correlation, then they should be decoded jointly
Example. Erasure channel
Consider the BSC with P0=P1=0.5
We Tx two independent bits but
add a third odd parity bit
If the received bits have odd parity, then they are decoded. If it has even parity, an
erasure is declared meaning that there is certainly an error in the received bits.
Two options for erasure: 1) Ask for a retransmission
2) Use a longer outer code to correct the error
Example. Erasure channel 2

Example. Erasure channel 3
From Bayes theorem
By symmetry, we get the same result for other probabilities of correct decoding
Note that this solution is a compromise between no coding and repetition coding:
1) We get better performance than no coding but worse than repetition coding
2) We get a better rate than repetition coding (1/3) but worse than no coding (1)
Thank You!


Lecture1 PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture1 PDF

Uploaded by

Copyright:

Available Formats

Introduction to

 Consists of sample space, sigma-field of events, probability function

Ex3. Number of hairs on a person’s head

Ex4. Age of married couples

Ex5. Power across a resistor

SLIDES BY: SHAHROKH FARAHMAND

SLIDES BY: SHAHROKH FARAHMAND 5

SLIDES BY: SHAHROKH FARAHMAND 6

SLIDES BY: SHAHROKH FARAHMAND 7

SLIDES BY: SHAHROKH FARAHMAND 8

Ex2. Tossing of a fair die

SLIDES BY: SHAHROKH FARAHMAND 9

Ex4. Toss of two fair dies

SLIDES BY: SHAHROKH FARAHMAND 10

Question. Relationship between independent sets and disjoint sets?

Bound is tight when sets are disjoint

SLIDES BY: SHAHROKH FARAHMAND 12

Complex model but I,Q components are real

SLIDES BY: SHAHROKH FARAHMAND 13

Union bound leads to a simple PoE expression

SLIDES BY: SHAHROKH FARAHMAND 14

 Note that RHS is random but LHS is deterministic

SLIDES BY: SHAHROKH FARAHMAND 15

Now, we can compute

SLIDES BY: SHAHROKH FARAHMAND 16

SLIDES BY: SHAHROKH FARAHMAND 17

Ex2. Binary Symmetric Channel

SLIDES BY: SHAHROKH FARAHMAND 18

SLIDES BY: SHAHROKH FARAHMAND 19

SLIDES BY: SHAHROKH FARAHMAND 20

Define X to be the draw with the largest number coming out

SLIDES BY: SHAHROKH FARAHMAND 21

SLIDES BY: SHAHROKH FARAHMAND 22

SLIDES BY: SHAHROKH FARAHMAND 23

SLIDES BY: SHAHROKH FARAHMAND 24

This means that occurrence of B does not have an impact on probability of

SLIDES BY: SHAHROKH FARAHMAND 25

SLIDES BY: SHAHROKH FARAHMAND 26

A: picking a black ball

SLIDES BY: SHAHROKH FARAHMAND 27

SLIDES BY: SHAHROKH FARAHMAND 28

Question. Do you think this is good or bad?

Question. Do you think this is good or bad?

Then apply law of total probability

SLIDES BY: SHAHROKH FARAHMAND 31

Aposteriori vs. apriori probabilities

SLIDES BY: SHAHROKH FARAHMAND 32

Need to evaluate E[B|A]:

SLIDES BY: SHAHROKH FARAHMAND 33

Bayes theorem offers the rigorous answer

SLIDES BY: SHAHROKH FARAHMAND 34

SLIDES BY: SHAHROKH FARAHMAND 36

SLIDES BY: SHAHROKH FARAHMAND 37

How many cases?

Ex. Tossing of 12 dies (E:= each face appears twice)

SLIDES BY: SHAHROKH FARAHMAND 38

Ex. r=3, n=2

Problem2. Distributing randomly a set of r similar balls into n boxes.

SLIDES BY: SHAHROKH FARAHMAND 39

With fixed 2 lines at the ends, we want to select r members out of

SLIDES BY: SHAHROKH FARAHMAND 40

SLIDES BY: SHAHROKH FARAHMAND 41