Probability, Random Variables

Chapter 1: Probability
Prof. Sung Jin Yoo

School of Electrical and Electronics Engineering

Chung-Ang University

Probability, random variables Chapter 1 1

1. Probability
 What is a random signal ?
- A time waveform characterized only in some probability manner
<e.g.>
a) Broad casting radio receiver 
voice = desired waveform(information) + Undesired waveform(noise)
 v (t )  n (t )
b) Television system  Picture interference called “snow”
: white noise
c) Wind-powered generator  Output voltage (random signal) due
to wind speed (random signal)

Note) we use the following notations:

,  : implication, ,  : necessary & sufficient
 : for all , : there exists, s.t.: such that
i.e.: id est (That is ),  : therefore,  : Because
Probability, random variables Chapter 1 2
1. Probability – set definitions
Def.) Set: a collection of objects <e.g.> A, B,.....
Elements: objects in a set <e.g.> a  A, e  A where A  {a, b, c, d }
Class: a collection of sets <e.g.> { A, B, C}

 Two methods in specifying a set

1) Tabular method : explicitly enumerated elements <e.g> A  {6, 7,8,9}
2) Rule method : implicitly describe a set by some rules
<e.g.> B  {5  b  10, b  integers}
Def.) Countable set: its elements 1 to 1 correspondence with natural
numbers (opp. uncountable set)
Empty set: a set without any elements (symbol:  ~ null set)
Finite set: either empty set or countable set with a finite
number of elements (opp. infinite set)
<e.g.> a countable infinite set : the set of the natural numbers
N  {1, 2,3,......}
Probability, random variables Chapter 1 3
1. Probability – set definitions
Def.) Subset A of B : A is contained in B ( A  B )
Proper subset A :  at least one element  A ( A  B )

Note) Null set is clearly a subset of all other sets. (  any sets )

Def.) Mutually Exclusive (or disjoint) sets: no common elements

(A  B  )

Probability, random variables Chapter 1 4

1. Probability – set definitions
 Ex 1.1-1) A  {1,3,5, 7}
B  {1, 2,3,......}
C  {0.5  c  8.5}
D  {0}
E  {2, 4, 6,8,10,12,14}
F  {5.0  f  12.0}
 A  B, C , F
C, D  F
EB
A, D, E ~ Mutually exclusive

Def.) Universal set S : the largest or all-encompassing set of

elements

Probability, random variables Chapter 1 5

1. Probability – set definitions
 Ex 1.1-2) A problem of rolling a die.

Probability, random variables Chapter 1 6

1. Probability – set operations
 Venn diagram
A, B , C  S
C is disjoint with both A and B.
B A

 Equality and Difference

Def.) equality : A  B  A  B and B  A
difference : A  B
 e.g .  A  {0.6  a  1.6}
B  {1.0  b  2.5}
 A  B  {0.6  c  1.0}
B  A  {1.6  d  2.5}
 A B  B  A

Probability, random variables Chapter 1 7

1. Probability – set operations
 Union and Intersection
Def.) Union of two sets A and B
C  A  B (sum of two sets)
N
C  A1  A2    AN   An
n 1

Def.) Intersection of two sets A and B

D  A  B (product of two sets)
N
D  A1  A2    AN   An
n 1
 Ex.
 Complement
Def.) Complement of a set A : A  S  A
ex)   S , S   , A  A  S , A  A  

Probability, random variables Chapter 1 8

1. Probability – Prob. introduced through sets and relative
frequency
 Algebra of sets
Def.) Commutative law : A  B  B  A
A B  B  A
Distributive law : A  ( B  C )  ( A  B)  ( A  C )
A  ( B  C )  ( A  B)  ( A  C )
Associative law : ( A  B)  C  A  ( B  C )  A  B  C
( A  B)  C  A  ( B  C )  A  B  C
De Morgan’s law : A  B  A  B

A B  A B

Probability, random variables Chapter 1 9

1. Probability – Prob. introduced through sets and relative
frequency
 Ex 1.2-2) Prove de Morgan’s law: A  B  A  B

 Duality principle
~ The identity is preserved by changing S into  ,  into , and vice versa.
ex) A  ( B  C )  ( A  B)  ( A  C )
 A  ( B  C )  ( A  B)  ( A  C )

Probability, random variables Chapter 1 10

1. Probability – Prob. introduced through sets and relative
frequency

trial (physical) outcome

experiment

 Experiments and sample spaces

Def.) Experiments : an activity that you try out to see what happens
ex) rolling a single die and observing the number
Sample space : all possible events for an experiment
( universal set S )
 Discrete and Continuous sample space
ex) {1, 2,3}: Discrete and finite
{1, 2,3,}: Discrete and infinite
{0  s  12}: Continuous
 Events
Def.) Event : a subset of the sample space

Probability, random variables Chapter 1 11

1. Probability – Prob. introduced through sets and relative
frequency
 Probability definitions and axioms
Def.) Probability : a function of events ( P( A): the prob. of event A)
Axi. 1) P( A)  0
Axi. 2) P( S )  1
N N
Axi. 3) P( Ai )   P( Ai ) if Ai  Aj   (M.E. events), i  j
i 1 i 1

with 1  N   (note: N   is possible)

 Mathematical model of experiments

* Procedure defining of real experiment
1) assignment of a sample space
2) definition of events of interest
3) making prob. assignment to the events

Probability, random variables Chapter 1 12

 Ex 1.3-2) Observing the sum of the numbers showing up when two
dice are thrown.

Probability, random variables Chapter 1 13

1. Probability – Joint and conditional probability
 Joint Probability
Def.) Joint prob. :
P ( A  B ) if, for two events A and B ,  common elements in A  B.
1) P ( A  B )  P ( A)  P ( B )  P ( A  B )
2) P ( A  B )  P ( A)  P ( B )  P ( A  B )  P ( A)  P ( B )
with equality holds if A  B   (M.E. events)
(i.e., P ( A  B )  P ( A)  P ( B ) )

 Conditional Probability
Def.) Conditional Probability
Given a nonzero prob. P ( B )  0,
Cond. prob. of an event A, given B, by
P( A  B) figure
P( A | B) 
P( B)
 if A and B are M.E., then A  B   . P( A | B)  0.
if A and C are M.E., then P ( A  C | B )  P ( A | B )  P (C | B )
Probability, random variables Chapter 1 14
 Ex 1.4-1

Probability, random variables Chapter 1 15

1. Probability – Joint and conditional probability
 Total Probability
T he prob. P ( A ) of any event A can be expressed in terms of N
conditional probabilities of M .E. events Bi whose union equals
the sample space S .
i) M .E. events Bi  B j   i  j
N
ii) B
i 1
i S

Def.) The total prob. of events A is

N
P ( A)   P ( A | Bi ) P ( Bi )
i 1
N N
proof) A  A  S  A  ( Bi )   ( A Bi )
i 1 i 1

N  N
P ( A)  P  ( A Bi )    P (A  Bi )  A  Bi 's are M.E.
 i 1  i 1
N
P ( A  Bi ) N
P ( A)   P ( Bi )   P ( A | Bi )P ( Bi )
i 1 P ( Bi ) i 1

Probability, random variables Chapter 1 16

1. Probability – Joint and conditional probability
 Bayes’ Theorem
* Problem statement : How to find “inverse prob. of P( A | Bi ) ” ?
P( Bi | A)  P( A | Bi ) : inverse relationship
Problem: How do we calculate P ( Bi | A) using P( A | Bi ) ?
P ( Bi  A)
P( Bi | A)  if P( A)  0
P ( A)
P ( A  Bi )
P( A | Bi )  if P( Bi )  0
P( Bi )
P ( A | Bi ) P( Bi ) P ( A | Bi ) P ( Bi )
 P( Bi | A)  
P( A) P( A | B1 ) P( B1 )  P( A | BN ) P( BN )

Probability, random variables Chapter 1 17

 Ex 1.4-2

Probability, random variables Chapter 1 18

1. Probability – Independent events
 Two events
Def.) Statistically independent (S. I.)
if P ( A)  0 and P ( B )  0, and
if the prob. of one event A is not affected by the other event B
and vice versa.
(i.e., P( A | B)  P( A) and P ( B | A)  P ( B ) ), then
 P( A | B) P( B)
 P( A  B)    P( A) P( B)
 P ( B | A) P ( A)

Note) M.E. events P( A  B)  0  A  B  

This means P ( A) P ( B )  0 for M.E. events.
It is not possible in S. I. because P( A)  0 and P( B)  0.
 It cannot be both M.E. and S.I..

Probability, random variables Chapter 1 19

 Ex 1.5-1

Probability, random variables Chapter 1 20

1. Probability – Independent events
 Multiple events : # of conditions for N events= 2 N  N  1
P( Ai  Aj )  P( Ai ) P( Aj ) i  j
P( Ai  Aj  Ak )  P( Ai ) P( Aj ) P( Ak ) i  j  k

P( A1  A2    AN )  P( A1 ) P( A2 ) P( AN )
 Permutations
~ multiple trials in which outcomes are elements of a finite
sample space and they are not replaced after each trial
~ the number of possible sequence of the outcomes is important.
Def.) For n total elements,
ordering of r elements taken from n elements
 n(n  1)(n  2) (n  r  1)
n!
  Prn , r  1, 2, , n
(n  r )!

Probability, random variables Chapter 1 21

1. Probability – Combined experiments
 Combinations
~ The number of sequences where order is not important
Def.) notation  n   Pr  n!
n

 
r  Prr (n  r )!r !

n
Note) The numbers   are called binomial coefficients
r 
because it is coefficient of (x  y ) n
n
 n  r nr
(x  y )     x y
n

r 0  r 

Note) 0!  1,  n   1,  n   1
0 n

Probability, random variables Chapter 1 22

1. Probability – Bernoulli trials
Def.) Repeated independent experiments = Bernoulli trials
ex) flipping a coin, passing or failing an exam,
winning or losing a game,
receiving 0 or 1 in a computer bit-stream
A
Let be an elementary event of repeated independent trials
P ( A)  p (prob. of success)
P ( A)  1  p (prob. of failure)
N repeated trials, k success trials
Prob. of one sequence : P ( A) P ( A) P ( A) P ( A) P ( A)  p k (1  p ) N  k
     
k terms N  k terms

The number of ways of taking k objects at a time from N objects:

N N!
  
 k  k !( N  k )!
N k
 P{ A occurs exactly k time}=   p (1  p ) N  k
k 
Probability, random variables Chapter 1 23
 Ex 1.7-1

Probability, random variables Chapter 1 24

1. Probability – Combined sample space
 Combined sample space
Let S1 and S 2 be the sample space of two subexperiments.
Elements: s1 and s2
 Combined sample space: S  S1  S 2  {s1 , s2 }
ex)

Probability, random variables Chapter 1 25

1. Probability – Events on combined space
 Events on combined space
A : Event defined on S1
B : Event defined on S 2
C  A  B :Event defined on S  S1  S 2  {s1 , s2 } ( s1  A, s2  B )
C  A  B  ( A  S 2 )  ( S1  B )
ex)

Probability, random variables Chapter 1 26