Conditional Probability Density Functions

Conditional Probability Density Function

P( A  B )
P( A | B) = , P ( B )  0.
P( B)

FX ( x ) = P X ( )  x ,

P ( X ( )  x )  B 
FX ( x | B ) = P X ( )  x | B  = .
P( B)

P ( X ( )  + )  B  P( B)
FX ( + | B ) = = = 1,
P( B ) P( B)
P ( X ( )  − )  B  P ( )
FX ( − | B ) = = = 0.
P( B ) P( B)
Conditional Probability Density Function

▪ Further, P ( x1  X ( )  x2 )  B 
P ( x1  X ( )  x2 | B ) =
P( B )
= FX ( x2 | B ) − FX ( x1 | B ),

▪ Since for x2  x1 ,
( X ( )  x2 ) = ( X ( )  x1 )  ( x1  X ( )  x2 ) .

dFX ( x | B )
f X ( x | B) = ,
dx
x
FX ( x | B ) =  −
f X (u | B )du.

P ( x1  X ( )  x2 | B ) =
x2
 x1
f X ( x | B ) dx.
 Example

▪ Toss a coin and X(T)=0, X(H)=1.

▪ Suppose B = {H }.
▪ Determine FX ( x | B ).

Solution

▪ FX (x ) has the following form.

▪ We need FX ( x | B ) for all x.
▪ For x  0, X ( )  x =  , so that  ( X ( )  x )  B =  ,
▪ and FX ( x | B ) = 0.

FX (x ) FX ( x | B )

1 1
q
x x
1 1
(a) (b)
 Example - continued

▪ For 0  x  1, X ( )  x =  T , so that

 ( X ( )  x )  B =  T   H  =  and FX ( x | B ) = 0.

▪ For x  1, X ( )  x = , and
 ( X ( )  x )  B =    B = {B} and FX ( x | B ) =
P( B)
=1
P( B)
FX ( x | B )

1
x
1
 Example

▪ Given FX (x ), suppose B = X ( )  a. Find f X ( x | B ).

Solution

▪ We will first determine FX ( x | B ).

P ( X  x )  ( X  a ) 
FX ( x | B ) = .
P( X  a )

▪ For x  a , (X  x )  ( X  a ) = ( X  x ) so that
P(X  x ) F ( x)
FX ( x | B ) = = X .
P(X  a ) FX ( a )

▪ For x  a , (X  x)  (X  a ) = ( X  a) so that FX ( x | B ) = 1.
 Example - continued

▪ Thus
 FX ( x )
 , x  a,
FX ( x | B ) =  FX ( a )

 1, x  a,
▪ and hence

 f X ( x)
d  , x  a,
f X ( x | B) = FX ( x | B ) =  FX ( a )
dx 
 0, otherwise.

FX ( x | B )
f X ( x | B)
1
f X (x )
FX (x )

a x a x
(a) (b)
 Example

▪ Let B represent the event  a  X ( )  b with b  a.

▪ For a given FX (x ), determine FX ( x | B ) and f X ( x | B ).

Solution

FX ( x | B ) = P X ( )  x | B 
P ( X ( )  x )  (a  X ( )  b ) 
=
P (a  X ( )  b )
P( X ( )  x )  (a  X ( )  b )
= .
FX (b) − FX ( a )
 Example - continued

▪ For x  a, we have  X ( )  x  a  X ( )  b =  ,

and hence FX ( x | B ) = 0.

▪ For a  x  b, we have  X ( )  x  a  X ( )  b = {a  X ( )  x}

and hence
P (a  X ( )  x ) FX ( x ) − FX ( a )
FX ( x | B ) = = .
FX ( b ) − FX ( a ) FX ( b) − FX ( a )

▪ For x  b, we have  X ( )  x  a  X ( )  b =  a  X ( )  b

so that FX ( x | B ) = 1.

f X ( x | B)
▪ Thus,  f X ( x)
 , a  x  b,
f X ( x | B ) =  FX (b) − FX ( a ) f X ( x)

 0, otherwise.
a x
b
Conditional p.d.f & Bayes’ Theorem
▪ First, we extend the conditional probability results to random variables:
▪ We know that If U = [ A1 , A2 ,  , An )] is a partition of S and B is an arbitrary event,
then:
P ( B ) = P ( B | A1 ) P ( A1 ) +  + P ( B | An ) P ( An )
▪ By setting B = { X  x} we obtain:

P ( X  x) = P ( X  x | A1 ) P ( A1 ) +  + P ( X  x | An ) P ( An )

Hence

F ( x) = F ( x | A1 ) P ( A1 ) +  + F ( x | An ) P ( An )

f ( x) = f ( x | A1 ) P ( A1 ) +  + f ( x | An ) P ( An )

A1 ,  An form a partition on S
Conditional p.d.f & Bayes’ Theorem

▪ Using: P ( B | A) P ( A)
P( A | B) = .
P( B)
▪ We obtain: P (X  x | A) F (x | A )
PA | X  x ) = P ( A) =
P (X  x ) F (x ) P ( A)

▪ For B = x1  X ( )  x2  ,

P ( x1  X  x 2 | A)
PA | x1  X  x 2  = P ( A)
P ( x1  X  x 2 )
x2

F ( x | A) − FX ( x1 | A)
= X 2 P ( A) =
 x1
f X ( x | A) dx
P ( A).
FX ( x 2 ) − FX ( x1 ) x2
 x1
f X ( x ) dx
Conditional p.d.f & Bayes’ Theorem

▪ Let x1 = x , x2 = x +  ,   0, so that in the limit as  → 0,

f X ( x | A)
lim PA | ( x  X  x +  ) = P ( A | X = x ) = P ( A).
 →0 f X ( x)
▪ or
P( A | X = x ) f X ( x)
f X | A ( x | A) = .
P ( A)

▪ we also get

+ +
P ( A)  f X ( x | A)dx =  P ( A | X = x ) f X ( x )dx ,
− −
  
1

+
▪ or P ( A) =  −
P ( A | X = x ) f X ( x )dx (Total Probability Theorem)
Bayes’ Theorem (continuous version)

▪ using total probability theorem in

P( A | X = x) f X ( x)
f X | A ( x | A) = ,
P ( A)

▪ We get the desired result

P( A | X = x ) f X ( x )
f X | A ( x | A) = +
.
−
P ( A | X = x ) f X ( x )dx
Excersise: Coin Tossing Problem

▪ p = P (H ) probability of obtaining a head in a toss.

▪ For a given coin, a-priori p can possess any value in (0,1).
▪ f P ( p ) : A uniform in the absence of any additional information
▪ After tossing the coin n times, k heads are observed.
▪ How can we update f P ( p ) this is new information?

Solution

▪ Let A= “k heads in n specific tosses”. f P ( p)

▪ Since these tosses result in a specific sequence,
P ( A | P = p ) = p k q n −k , p
▪ and using Total Probability Theorem we get 0 1
1 1 ( n − k )! k!
P ( A) =  P ( A | P = p ) f P ( p )dp =  p k (1 − p ) n −k dp = .
0 0 ( n + 1)!
 Example - continued

▪ The a-posteriori p.d.f f P| A ( p | A) represents the updated information given the

event A,
f P| A ( p | A)
P( A | X = x) f X ( x)
▪ Using f X | A ( x | A) = ,
P ( A)
p
0 1
P ( A | P = p )f P ( p )
f P |A ( p | A ) =
P (A )
(n + 1)!
= p k q n −k , 0  p  1  (n , k ).
(n − k )!k !

▪ This is a beta distribution.

▪ We can use this a-posteriori p.d.f to make further predictions.
▪ For example, in the light of the above experiment, what can we say about the
probability of a head occurring in the next (n+1)th toss?
 Example - continued

▪ Let B= “head occurring in the (n+1)th toss, given that k heads have occurred in n
previous tosses”.
▪ Clearly P ( B | P = p ) = p, 1
▪ From Total Probability Theorem, P ( B ) =  P ( B | P = p ) f P ( p | A)dp.
0

1 ( n + 1)! k +1
▪ Using (1) in (2), we get: P (B ) =  p . k n −k
p q dp =
0 ( n − k )!k ! n +2
▪ Thus, if n =10, and k = 6, then
7
P( B) = = 0.58,
12

▪ which is more realistic compare to p = 0.5.

Bayesian Updating: Application Of Bayes’
Theorem (recall chapter 1)

▪ Suppose that A and B are dependent events and A has apriori

probability of P(A ) .
▪ How does Knowing that B has occurred affect the probability of A?
▪ The new probability can be computed based on Bayes’ Theorem.

▪ Bayes’ Theorm shows how to incorporate the knowledege about B’s

occuring to calculate the new probability of A.

17