Professional Documents
Culture Documents
Probability and Random Variables
Probability and Random Variables
CHAPTER 2
U
2.1 Introduction
At the start of Sec. 1.1.2, we had indicated that one of the possible ways
of classifying the signals is: deterministic or random. By random we mean
unpredictable; that is, in the case of a random signal, we cannot with certainty
predict its future value, even if the entire past history of the signal is known. If the
signal is of the deterministic type, no such uncertainty exists.
Consider the signal x ( t ) = A cos ( 2 f1 t + ) . If A , and f1 are known,
then (we are assuming them to be constants) we know the value of x ( t ) for all t .
( A , and f1 can be calculated by observing the signal over a short period of
time).
Now, assume that x ( t ) is the output of an oscillator with very poor
frequency stability and calibration. Though, it was set to produce a sinusoid of
frequency f = f1 , frequency actually put out maybe f1' where f1' ( f1 f1 ) .
Even this value may not remain constant and could vary with time. Then,
observing the output of such a source over a long period of time would not be of
much use in predicting the future values. We say that the source output varies in
a random manner.
2.1
Indian Institute of Technology Madras
Principles of Communication
(radio) signal is from a highly stable source, the voltage at the terminals of a
receiving antenna varies in an unpredictable fashion. This is because the
conditions of propagation of the radio waves are not under our control.
2.2
Indian Institute of Technology Madras
Principles of Communication
Statistical
regularity
of
averages
is
an
experimentally
verifiable
2.2.1. Terminology
Def. 2.1: Outcome
2.3
Principles of Communication
S.
and is
(A
This concept can be generalized to the union of more than two events.
The intersection of two events, A and B , is the set of all outcomes which
belong to A as well as B . The intersection of A and B is denoted by ( A B )
or simply ( A B ) . The intersection of A and B is also referred to as a joint event
2.4
Indian Institute of Technology Madras
Principles of Communication
but not in A .
n
P ( A ) = lim A
n n
(2.1)
nA
n represents the fraction of occurrence of A in n trials.
n
For small values of n , it is likely that A will fluctuate quite badly. But
n
n
as n becomes larger and larger, we expect, A to tend to a definite limiting
n
value. For example, let the experiment be that of tossing a coin and A the event
n
'outcome of a toss is Head'. If n is the order of 100, A may not deviate from
n
2.5
Indian Institute of Technology Madras
Principles of Communication
1
by more than, say ten percent and as n becomes larger and larger, we
2
1
n
expect A to converge to .
2
n
Def. 2.12: The classical definition:
The relative frequency definition given above has empirical flavor. In the
classical approach, the probability of the event
is found without
NA
N
(2.2)
(2.3a)
P2) P ( S ) = 1
(2.3b)
P3) ( AB ) = , then P ( A + B ) = P ( A ) + P ( B )
(2.3c)
(Note that in Eq. 2.3(c), the symbol + is used to mean two different things;
namely, to denote the union of A and B and to denote the addition of two real
numbers). Using Eq. 2.3, it is possible for us to derive some additional
relationships:
i) If A B , then P ( A + B ) = P ( A ) + P ( B ) P ( A B )
(2.4)
(2.5a)
2.6
Indian Institute of Technology Madras
Principles of Communication
b) A1 + A2 + ...... + An =
S.
(2.5b)
Then, P ( A ) = P ( A A1 ) + P ( A A2 ) + ...... + P ( A An )
(2.6)
(2.7)
that A has occurred. In a real world random experiment, it is quite likely that the
occurrence of the event B is very much influenced by the occurrence of the
event A . To give a simple example, let a bowl contain 3 resistors and 1
capacitor. The occurrence of the event 'the capacitor on the second draw' is very
much dependent on what has been drawn at the first instant. Such dependencies
between the events is brought out using the notion of conditional probability .
The conditional probability P ( B | A ) can be written in terms of the joint
probability P ( A B ) and the probability of the event P ( A ) . This relation can be
arrived at by using either the relative frequency definition of probability or the
classical definition. Using the former, we have
nAB
P ( AB ) = lim
n
n
n
P ( A ) = lim A
n n
2.7
Indian Institute of Technology Madras
Principles of Communication
nAB
P ( B | A ) = lim
nA
nAB
= lim n
n nA
n
P ( AB )
, P ( A) 0
=
P ( A)
(2.8a)
P ( A B ) = P (B | A) P ( A)
or
P ( A | B) =
P ( AB )
P (B )
, P (B ) 0
(2.8b)
(2.9)
P (B | A) =
P (B ) P ( A | B )
, P ( A) 0
(2.10a)
P ( A) P (B | A)
, P (B ) 0
P (B )
(2.10b)
P ( A)
Similarly
P ( A | B) =
2.8
Indian Institute of Technology Madras
Principles of Communication
P ( A BC ) = P ( A B ) P (C | AB )
= P ( A ) P ( B | A ) P (C | AB )
(2.11)
Exercise 2.1
P Aj | B =
) ( )
P (B | Aj ) P ( Aj )
P B | Aj P Aj
(2.12)
i = 1
(2.13)
(2.14)
Either Eq. 2.13 or 2.14 can be used to define the statistical independence
of
two
events.
Note
that
if
and
are
independent,
then
2.9
Indian Institute of Technology Madras
Principles of Communication
P ( A B ) = P ( A) P (B )
P ( AC ) = P ( A ) P (C )
P ( BC ) = P ( B ) P (C )
and
P ( A BC ) = P ( A ) P ( B ) P (C )
We shall illustrate some of the concepts introduced above with the help of
two examples.
Example 2.1
Priya (P1) and Prasanna (P2), after seeing each other for some time (and
after a few tiffs) decide to get married, much against the wishes of the parents on
both the sides. They agree to meet at the office of registrar of marriages at 11:30
a.m. on the ensuing Friday (looks like they are not aware of Rahu Kalam or they
dont care about it).
However, both are somewhat lacking in punctuality and their arrival times
are equally likely to be anywhere in the interval 11 to 12 hrs on that day. Also
arrival of one person is independent of the other. Unfortunately, both are also
very short tempered and will wait only 10 min. before leaving in a huff never to
meet again.
a)
b)
Let the event A stand for P1 and P2 meet. Mark this event on the sample
space.
c)
Find the probability that the lovebirds will get married and (hopefully) will
live happily ever after.
a)
2.10
Indian Institute of Technology Madras
Principles of Communication
Fig. 2.1(a):
b)
The diagonal OP
of Example 2.1
2.11
Indian Institute of Technology Madras
Principles of Communication
c)
Probability of marriage =
Shaded area
11
=
Total area
36
Example 2.2:
Let two honest coins, marked 1 and 2, be tossed together. The four
possible outcomes are T1 T2 , T1 H2 , H1 T2 , H1 H2 . ( T1 indicates toss of coin 1
resulting in tails; similarly T2 etc.) We shall treat that all these outcomes are
equally likely; that is the probability of occurrence of any of these four outcomes
is
1
. (Treating each of these outcomes as an event, we find that these events
4
are mutually exclusive and exhaustive). Let the event A be 'not H1 H2 ' and B be
the event 'match'. (Match comprises the two outcomes T1 T2 , H1 H2 ). Find
P ( B | A ) . Are A and B independent?
We know that P ( B | A ) =
P ( AB )
P ( A)
A B is the event 'not H1 H2 ' and 'match'; i.e., it represents the outcome T1 T2 .
Hence P ( A B ) =
1
. The event A comprises of the outcomes T1 T2 , T1 H2 and
4
H1 T2 ; therefore,
P ( A) =
3
4
1
1
P (B | A) = 4 =
3
3
4
Intuitively, the result P ( B | A ) =
1
is satisfying because, given 'not H1 H2 the
3
toss would have resulted in anyone of the three other outcomes which can be
2.12
Indian Institute of Technology Madras
Principles of Communication
1
. This implies that the outcome T1 T2 given
3
1
.
3
1
1
and P ( B | A ) = , A and B are dependent events.
2
3
si
from
[a1 , a2 ] .
2.3.1 Distribution function:
Taking a specific case, let the random experiment be that of throwing a
die. The six faces of the die can be treated as the six sample points in
2.13
Indian Institute of Technology Madras
S ; that is,
Principles of Communication
then the events on the sample space will become transformed into appropriate
segments of the real line. Then we can enquire into the probabilities such as
P {s : X ( s ) < a1}
P {s : b1 < X ( s ) b2 }
or
P {s : X ( s ) = c}
These and other probabilities can be arrived at, provided we know the
Distribution Function of X, denoted by FX (
which is given by
FX ( x ) = P {s : X ( s ) x}
(2.15)
That is, FX ( x ) is the probability of the event, comprising all those sample points
which are transformed by X into real numbers less than or equal to x . (Note
that, for convenience, we use x as the argument of FX (
is a function
whose domain is the real line and whose range is the interval [0, 1] ).
1
,
4
1
1
1
, P ( s3 ) =
and P ( s4 ) = . If X ( si ) = i 1.5, i = 1, 2, 3, 4 , then
8
8
2
2.14
Indian Institute of Technology Madras
Principles of Communication
i)
FX ( x ) 0, < x <
ii)
FX ( ) = 0
iii)
FX ( ) = 1
iv)
v)
If a > b , then FX ( a ) FX ( b )
represents the
{s : X ( s )
b} {s : b < X ( s ) a} = {s : X ( s ) a}
Referring to the Fig. 2.3, note that FX ( x ) = 0 for x < 0.5 whereas
FX ( 0.5 ) =
1
1
at the point
. In other words, there is a discontinuity of
4
4
P {s : X ( s ) = a} = Pa
(2.16)
2.15
Indian Institute of Technology Madras
Principles of Communication
PT
Functions such as X (
the real line need not be a random variable. For example, let
into
consist of six
sample points, s1 to s6 . The only events that have been identified on the sample
space are: A = {s1 , s2 }, B = {s3 , s4 , s5 } and C = {s6 } and their probabilities
are P ( A ) =
2
1
1
, P ( B ) = and P (C ) = . We see that the probabilities for the
6
2
6
space.
1
TP
PT
We intuitively feel that as 0 , the limit of the set s : a < X ( s ) a + is the null set and
can be proved to be so. Hence,
F
That is, F
( a+ ) FX ( a )
= 0 , where a
lim
(a + )
2.16
Indian Institute of Technology Madras
Principles of Communication
Exercise 2.2
Let
identified
on
are
the
same
as
above,
Let Y (
namely,
A = {s1 , s2 } ,
1
1
1
, P ( B ) = and P (C ) = .
3
2
6
be the transformation,
Y ( si )
1, i = 1, 2
= 2 , i = 3, 4, 5
3 , i = 6
Show that Y (
fX ( x ) =
d FX ( x )
(2.17a)
dx
or
FX ( x ) =
f () d
(2.17b)
i)
ii)
2.17
Indian Institute of Technology Madras
Principles of Communication
example, for the CDF shown in Fig. 2.3, the PDF will be,
fX ( x ) =
1
1 1
1 1
3 1
5
x + + x + x + x
4
2 8
2 8
2 2
2
1
8
at
(2.18)
x =
1
2
as
1
1
lim
1
2
1
x
ga ) because,
f X ( x ) dx = lim
1
2
1
1
1
x dx
8
2
16
1
2
2.18
Indian Institute of Technology Madras
Principles of Communication
1
2
However, lim
f X ( x ) dx =
1
2
1
. This ensures,
8
1
1
2
8 , 2 x < 2
FX ( x ) =
3 , 1 x < 3
8
2
2
Such an impulse is referred to as the left-sided delta function.
As FX is non-decreasing and FX ( ) = 1, we have
fX ( x ) 0
i)
(2.19a)
ii)
f ( x ) dx
X
= 1
(2.19b)
PT
PT
simply by X .
2.19
Indian Institute of Technology Madras
(2.20)
Principles of Communication
That is, FX ,Y ( x , y ) is the probability associated with the set of all those
sample points such that under X , their transformed values will be less than or
equal to x and at the same time, under Y , the transformed values will be less
than or equal to y . In other words, FX ,Y ( x1 , y1 ) is the probability associated with
the set of all sample points whose transformation does not fall outside the
shaded region in the two dimensional (Euclidean) space shown in Fig. 2.5.
S,
sample points s
FX ,Y ( x , y ) 0 ,
ii)
FX ,Y ( , y ) = FX ,Y ( x , ) = 0
iii)
FX ,Y ( , ) = 1
iv)
FX ,Y ( , y ) = FY ( y )
v)
FX ,Y ( x , ) = FX ( x )
2.20
Indian Institute of Technology Madras
Principles of Communication
vi)
f X ,Y ( x , y ) =
or
FX ,Y ( x , y ) =
(2.21a)
f ( , ) d d
X ,Y
(2.21b)
The notion of joint CDF and joint PDF can be extended to the case of k random
variables, where k 3 .
That is, FX ( x1 ) =
f ( , ) d d
X ,Y
f X ( x1 ) =
d 1
f X ,Y ( , ) d d
d x1
(2.22)
f X ( x1 ) =
or
fX ( x ) =
d FX ( x )
dx
=
x = x1
f ( x , ) d
X ,Y
f ( x , y ) dy
(2.23a)
X ,Y
Similarly, fY ( y ) =
f ( x , y ) dx
(2.23b)
X ,Y
(In the study of several random variables, the statistics of any individual variable
is called the marginal. Hence it is common to refer FX ( x ) as the marginal
2.21
Indian Institute of Technology Madras
Principles of Communication
f X |Y ( x | y1 ) =
f X ,Y ( x , y1 )
(2.24)
fY ( y1 )
f X |Y ( x | y ) =
f X ,Y ( x , y )
(2.25a)
fY ( y )
f X ,Y ( x , y )
and
fY | X ( y | x ) =
or
f X ,Y ( x , y ) = f X |Y ( x | y ) fY ( y )
(2.25c)
= fY | X ( y | x ) f X ( x )
(2.25d)
(2.25b)
fX ( x )
(2.26a)
and
f ( x | y ) dx
X |Y
= 1
(2.26b)
2.22
Indian Institute of Technology Madras
Principles of Communication
f X i ( xi )
(2.27)
i = 1
(2.28)
(2.29)
or
f X |Y ( x | y ) = f X ( x )
(2.30a)
Similarly, fY | X ( y | x ) = fY ( y )
(2.30b)
Eq. 2.30(a) and 2.30(b) are alternate expressions for the statistical independence
between X and Y . We shall now give a few examples to illustrate the concepts
discussed in sec. 2.3.
Example 2.3
0 , x < 0
FX ( x ) = K x 2 , 0 x 10
100 K , x > 10
i)
ii)
2.23
Indian Institute of Technology Madras
Principles of Communication
iii)
What is f X ( x ) ?
i)
FX ( ) = 100 K = 1 K =
ii)
1
P ( x 5 ) = FX ( 5 ) =
25 = 0.25
100
1
.
100
P ( 5 < X 7 ) = FX ( 7 ) FX ( 5 ) = 0.24
d FX ( x )
fX ( x ) =
dx
,
0
= 0.02 x ,
0
,
x < 0
0 x 10
x > 10
Example 2.4
bx
i)
ii)
iii)
Find P [1 < X 2] .
i)
That is, a e b x dx =
0
ii)
1
; hence b = 2 a .
2
a e b x ,
fX ( x ) = b x
a e ,
x < 0
x 0.
2.24
Indian Institute of Technology Madras
ae
bx
dx = 2 a e b x dx = 1 .
0
Principles of Communication
b b
1
e d = eb x .
2
2
f (x) d x
X
= 1 fX ( x ) d x
2
f (x) d x
X
1 bx
e
, x > 0
2
FX ( x ) =
1
iii)
FX ( 2 ) FX (1) =
1 bx
e
,
2
1 bx
e
,
2
1 b
e e 2 b
2
x < 0
x 0
Example 2.5
1
, 0 x y, 0 y 2
Let f X ,Y ( x , y ) = 2
0 , otherwise
Find
(a)
(a)
i) fY | X ( y | 1)
(b)
fY | X ( y | x ) =
and
ii) fY | X ( y | 1.5 )
f X ,Y ( x , y )
fX ( x )
2.25
Indian Institute of Technology Madras
Principles of Communication
2 dy
x
= 1
x
, 0 x 2
2
Hence,
b)
1
2 = 1 , 1 y 2
1
0 , otherwise
2
(i)
fY | X ( y | 1) =
(ii)
1
2 , 1.5 y 2
fY | X ( y | 1.5 ) = 2 =
1
0 , otherwise
4
2.26
Indian Institute of Technology Madras
Principles of Communication
Exercise 2.3
For the two random variables X and Y , the following density functions
have been given. (Fig. 2.6)
f X ,Y ( x , y )
b)
Show that
y
, 0 y 10
100
fY ( y ) =
1 y , 10 < y 20
5 100
2.27
Indian Institute of Technology Madras
Principles of Communication
Our interest is to obtain fY ( y ) . This can be obtained with the help of the following
theorem (Thm. 2.1).
Theorem 2.1
f X ( x1 )
g ' ( x1 )
+ ........... +
f X ( xn )
g ' ( xn )
+ .........
(2.31)
Proof: Consider the transformation shown in Fig. 2.7. We see that the equation
y 1 = g ( x ) has three roots namely, x1 , x2 and x3 .
2.28
Indian Institute of Technology Madras
Principles of Communication
+ P [ x3 < X x 3 + d x3 ]
This probability is the shaded area in Fig. 2.7.
P [ x1 < X x1 + d x1 ] = f X ( x1 ) d x1 , d x1 =
dy
g ' ( x1 )
P [ x2 + d x2 < x x2 ] = f X ( x2 ) d x2 , d x2 =
dy
g ' ( x2 )
P [ x3 < X x3 + d x 3 ] = f X ( x 3 ) d x3 , d x 3 =
dy
g ' ( x3 )
2.29
Indian Institute of Technology Madras
Principles of Communication
We conclude that
fY ( y ) d y =
f X ( x1 )
g ' ( x1 )
dy +
f X ( x2 )
g ' ( x2 )
dy +
f X ( x3 )
g ' ( x3 )
dy
(2.32)
Example 2.6
Y = g ( X ) = X + a , where a is a constant. Let us find fY ( y ) .
fY ( y ) d y =
fX ( x )
g '(x)
fY ( y ) = f X ( y a )
1 x , x 1
Let f X ( x ) =
0 , elsewhere
and a = 1
Then, fY ( y ) = 1 y + 1
As y = x 1, and x ranges from the interval ( 1, 1) , we have the range of y
as ( 2, 0 ) .
2.30
Indian Institute of Technology Madras
Principles of Communication
1 y + 1 , 2 y 0
Hence fY ( y ) =
0
, elsewhere
Example 2.7
x . As g ' ( x ) = b , we have fY ( y ) =
1
Y . Again for a given y , there is a unique
b
1
y
fX .
b
b
1 , 0 x 2
Let f X ( x ) =
2
0 , otherwise
and b = 2 , then
2.31
Indian Institute of Technology Madras
Principles of Communication
fY ( y ) = 2
1 + 4 , 4 y 0
0
, otherwise
Exercise 2.4
1 y b
fX
.
a
a
Example 2.8
Y = a X 2 , a > 0 . Let us find fY ( y ) .
2.32
Indian Institute of Technology Madras
Principles of Communication
y
, and Eq. 2.31 yields
a
1 y
f X
+ f X
= 2a y
a
fY ( y )
Let a = 1 , and f X ( x ) =
y
, y 0
a
, otherwise
x2
exp
, < x <
2
2
1
2 y
y
y
exp 2 + exp 2
2
1
y
exp , y 0
= 2 y
2
0
, otherwise
Example 2.9
0 , X 0
Y =
X , X > 0
a)
b)
1
3
1
, < x <
Let f X ( x ) = 2
2
2
0 , otherwise
We shall compute and sketch fY ( y ) .
2.33
Indian Institute of Technology Madras
y
and
a
Principles of Communication
a)
f ( y ) w ( y ) + FX ( 0 ) ( y )
fY ( y ) = X
0 , otherwise
1 , y 0
where w ( y ) =
0 , otherwise
b)
1
3
1
, <x<
Specifically, let f X ( x ) = 2
2
2
0 , elsewhere
Then, fY ( y )
1
4 (y ) , y = 0
3
1
=
, 0 < y
2
2
, otherwise
0
2.34
Indian Institute of Technology Madras
Principles of Communication
Example 2.10
1, X < 0
Let Y =
+ 1, X 0
a)
b)
a)
In this case, Y assumes only two values, namely 1. Hence the PDF of Y
has only two impulses. Let us write fY ( y ) as
fY ( y ) = P1 ( y 1) + P1 ( y + 1) where
P 1 = P [ X < 0 ] and P1 [ X 0 ]
b)
3
1
and P 1 = . Fig. 2.11
4
4
Note that this transformation has converted a continuous random variable X into
a discrete random variable Y .
2.35
Principles of Communication
Exercise 2.5
Let a random variable X with the PDF shown in Fig. 2.12(a) be the
input to a device with the input-output characteristic shown in Fig. 2.12(b).
Compute and sketch fY ( y ) .
Fig. 2.12: (a) Input PDF for the transformation of exercise 2.5
(b) Input-output transformation
Exercise 2.6
2.36
Indian Institute of Technology Madras
Principles of Communication
Example 2.11
Let Y = X 2 .
a)
b)
a)
1
If f X ( x ) =
6
If f X ( x ) =
1
6
( x i ) , find fY ( y ) .
i = 1
3
i = 2
( x i ) , find fY ( y ) .
b)
1
, then Y takes the
6
1
1
. That is, fY ( y ) =
6
6
(x i ) .
6
i = 1
1
, then
6
1 1 1 1
, , ,
respectively.
6 3 3 6
That is,
fY ( y ) =
1
1
( y ) + ( y 9 ) + ( y 1) + ( y 4 )
6
3
2.37
Indian Institute of Technology Madras
Principles of Communication
z, w
J
where
x, y
z
x
z, w
J
=
w
x, y
x
=
z
y
w
y
z w z w
x y y x
That is, the Jacobian is the determinant of the appropriate partial derivatives. We
shall now state the theorem which relates fZ ,W ( z , w ) and f X ,Y ( x , y ) .
(2.33a)
h ( x , y ) = w1 ,
(2.33b)
h ( x , y ) = w1 ,
fZ ,W ( z , w ) =
f X ,Y ( x1 , y1 )
(2.34)
z, w
J
x1 , y1
Proof of this theorem is given in appendix A2.1. For a more general version of
this theorem, refer [1].
2.38
Indian Institute of Technology Madras
Principles of Communication
Example 2.12
1
, x 1
2
fY ( y ) =
1
, y 1
2
and (b) fZ ( z ) .
a)
x =
From
y =
the
given
transformations,
we
obtain
1
(z + w )
2
and
1
( z w ) . We see that the mapping is one-to-one. Fig. 2.14(a)
2
A on which f X ,Y ( x , y ) is non-zero.
2.39
Indian Institute of Technology Madras
Principles of Communication
A space
B space
The line x = 1
The line x = 1
The line y = 1
The line y = 1
The space
1
(z + w ) = 1 w = z + 2
2
1
( z + w ) = 1 w = ( z + 2)
2
1
(z w ) = 1 w = z 2
2
1
(z w ) = 1 w = z + 2
2
is
1 1
z, w
J
= 2 and J (
=
1 1
x, y
Hence fZ ,W
b)
fZ ( z ) =
= 2.
1
1
, z, w B
4
= 8
(z, w ) =
2
0 , otherwise
f (z, w ) d w
Z ,W
From Fig. 2.14(b), we can see that, for a given z ( z 0 ), w can take
values only in the z to z . Hence
fZ ( z ) =
+ z
8 dw
1
z, 0 z 2
4
8 dw
z
1
z, 2 z 0
4
2.40
Indian Institute of Technology Madras
Principles of Communication
Hence fZ ( z ) =
1
z , z 2
4
Example 2.13
X 2 + Y 2 and
Y
= arc tan where we assume R 0 and < < . It is given that
X
x 2 + y 2
1
f X ,Y ( x , y ) =
exp
,
2
2
< x, y < .
Let us find fR , ( r , ) .
J =
r
x
r
y
cos
= sin
r
y
sin
1
cos =
r
r
r
r2
exp , 0 r , < <
Hence, fR , ( r , ) = 2
2
, otherwise
0
It is left as an exercise to find fR ( r ) , f ( ) and to show that R and are
independent variables.
obtained.
2.41
Indian Institute of Technology Madras
Principles of Communication
As J = 1 ,
fZ ,W ( z , w ) = f X ,Y ( z w , w )
and
fZ ( z ) =
f (z w , w ) d w
(2.35)
X ,Y
fZ ( z ) =
f ( z w ) f (w ) d w
X
(2.36a)
That is, fZ = f X fY
(2.36b)
Example 2.14
2.42
Indian Institute of Technology Madras
Principles of Communication
1
.
12
Example 2.15
X
; let us find an expression for fZ ( z ) .
Y
Let Z =
As J =
1
, we have
w
fZ ( z ) =
w f X ,Y ( zw , w ) dw
1 + x y
, x 1, y 1
Let f X ,Y ( x , y ) = 4
0
, elsewhere
Then fZ ( z ) =
1 + zw 2
dw
4
2.43
Indian Institute of Technology Madras
Principles of Communication
f X ,Y ( x , y ) is non-zero if
(x, y ) A .
where
B . Under
ii)
z
1
1+
4
2
1 + zw 2
1 1
1
0 4 w dw = 4 z2 + 2 z3
fZ ( z ) = 2
iii)
fZ ( z ) = 2
1
z
1 + zw 2
1 1
1
w dw =
2 +
4
4 z
2 z3
2.44
Indian Institute of Technology Madras
Principles of Communication
1
4
Hence fZ ( z ) =
1
4
1 + 2
, z 1
1
1
2 +
, z > 1
2 z3
z
Exercise 2.7
Let Z = X Y and W = Y .
a)
Show that fZ ( z ) =
b)
1
z
f X ,Y , w d w
w
w
and
1
1 + x2
y
y
e
fY ( y ) =
0
Show that fZ ( z ) =
, x 1
1
2
, y 0
, otherwise
2
z
, < z < .
Example 2.16
P [ X = 0] = P [ X = 1] =
1
. The output of the channel is given by Z = X + Y
2
2.45
Indian Institute of Technology Madras
Principles of Communication
1
2
y2
2
fZ ( z ) .
Let us first compute the distribution function of Z from which the density
function can be derived.
P ( Z z ) = P [ Z z | X = 0] P [ X = 0 ] + P [ Z z | X = 1] P [ X = 1]
As Z = X + Y , we have
P [ Z z | X = 0 ] = FY ( z )
Similarly P [ Z z | X = 1] = P Y ( z 1) = FY ( z 1)
Hence FZ ( z ) =
1
1
d
FZ ( z ) , we have
FY ( z ) + FY ( z 1) . As fZ ( z ) =
2
2
dz
z2
1 1
+
exp
fZ ( z ) =
2
2 2
z 1 2
(
)
exp
2
2
d FY ( y )
dy
. Similarly, for
Example 2.17
2.46
Indian Institute of Technology Madras
Principles of Communication
P [Z z ] = P [ X + Y z ]
= P [Y z x ]
This probability is the probability of ( X , Y ) lying in the shaded area shown in Fig.
2.17.
That is,
FZ ( z ) =
z x
d
x
fX ,Y ( x , y ) d y
z x
fZ ( z ) =
d x f X ,Y ( x , y ) d y
z
d x z
z x
f X ,Y ( x , y ) d y
f (x, z x) d x
(2.37a)
X ,Y
2.47
Indian Institute of Technology Madras
Principles of Communication
fZ ( z ) =
f (z y , y ) d y
(2.37b)
X ,Y
x fX ( x ) d x
(2.38)
g (x) f (x) d x
X
(2.39)
Remarks: The terminology expected value or expectation has its origin in games
of chance. This can be illustrated as follows: Three small similar discs, numbered
1,2 and 2 respectively are placed in bowl and are mixed. A player is to be
2.48
Indian Institute of Technology Madras
Principles of Communication
blindfolded and is to draw a disc from the bowl. If he draws the disc numbered 1,
he will receive nine dollars; if he draws either disc numbered 2, he will receive 3
dollars. It seems reasonable to assume that the player has a '1/3 claim' on the 9
dollars and '2/3 claim' on three dollars. His total claim is 9(1/3) + 3(2/3), or five
dollars. If we take X to be (discrete) random variable with the PDF
fX ( x ) =
1
2
( x 1) + ( x 2 ) and g ( X ) = 15 6 X , then
3
3
E g ( X ) =
(15 6 x ) f ( x ) d x
X
= 5
x n fX ( x ) d x
(2.40)
The most widely used moments are the first moment ( n = 1, which results in the
mean value of Eq. 2.38) and the second moment ( n = 2 , resulting in the mean
square value of X ).
E X 2 = X 2 =
x 2 fX ( x ) d x
(2.41)
n
E ( X m X ) =
(x m )
X
fX ( x ) d x
(2.42)
2.49
Indian Institute of Technology Madras
Principles of Communication
E g ( X , Y ) =
g (x, y ) f (x, y ) d x d y
(2.43)
X ,Y
E [Z ] =
( x + y ) f ( x , y ) d x dy
X ,Y
( x ) fX ,Y ( x , y ) d x dy +
( y ) f ( x , y ) d x dy
X ,Y
Integrating out the variable y in the first term and the variable x in the second
term, we have
E [Z ] =
x f (x) d x + y f (y ) d y
X
= X + Y
2.5.1 Variance
Coming back to the central moments, we have the first central moment
being always zero because,
E ( X m X ) =
(x m ) f (x) d x
X
= mX mX = 0
= E X 2 2 m X2 + m X2
= X 2 m X2 = X 2 X
2.50
Indian Institute of Technology Madras
Principles of Communication
The second central moment of a random variable is called the variance and its
(positive) square root is called the standard deviation. The symbol 2 is
generally used to denote the variance. (If necessary, we use a subscript on 2 )
P g ( X ) c
E g ( X )
(2.44)
g (x) f (x) d x
X
E g ( X )
g (x) f (x) d x
X
If x A , then g ( x ) c , hence
E g ( X ) c f X ( x ) d x
A
But f X ( x ) d x = P [ x A] = P g ( X ) c
A
Note: The kind of manipulations used in proving the theorem 2.3 is useful in
establishing similar inequalities involving random variables.
2.51
Indian Institute of Technology Madras
Principles of Communication
To see how innocuous (or weak, perhaps) the inequality 2.44 is, let g ( X )
represent the height of a randomly chosen human being with E g ( X ) = 1.6m .
Then Eq. 2.44 states that the probability of choosing a person over 16 m tall is at
most
1
! (In a population of 1 billion, at most 100 million would be as tall as a
10
i)
1
P X m X k X 2 , k > 0
k
(2.45a)
ii)
1
P X m X < k X > 1 2
k
(2.45b)
k
In other words,
1
P X m X k X 2
k
which is the desired result. Naturally, we would take the positive number k to be
greater than one to have a meaningful result. Chebyshev inequality can be
interpreted as: the probability of observing any RV outside k standard
deviations off its mean value is no larger than
1
. With k = 2 for example, the
k2
2.52
Indian Institute of Technology Madras
Principles of Communication
Note that it is not necessary that variance exists for every PDF. For example, if
2.5.2 Covariance
An important joint expectation is the quantity called covariance which is
obtained by letting g ( X , Y ) = ( X m X ) (Y mY ) in Eq. 2.43. We use the
symbol to denote the covariance. That is,
X Y = cov [ X ,Y ] = E ( X m X ) (Y mY )
(2.46a)
(2.46b)
X Y =
E [ X Y ] mX mY
(2.47)
X Y
x y f X ,Y ( x , y ) d x d y
x y f X ( x ) fY ( y ) d x d y
x fX ( x ) d x
y f (y ) d y
Y
2.53
Indian Institute of Technology Madras
= m X mY
Principles of Communication
sum
,
number of trials
(y
( x mX ) .
Then we
X and Y be so conditioned
(y
and Y
are said to be
random variables are independent, they are uncorrelated. However, the fact that
they are uncorrelated does not ensure that they are independent. As an example,
let
2.54
Indian Institute of Technology Madras
Principles of Communication
1
< x <
,
fX ( x ) =
2
2,
0 , otherwise
and Y = X 2 . Then,
XY = X3
1
1
= x3 d x =
x3 d x = 0
E [Y ] = k1 m1 + k 2 m2
E Y 2 = E k1 X12 + k 2 X 22 + 2 k1 k 2 X1 X 2
(2.48a)
Y2 = k1 12 + k2 22
(2.48b)
2.55
Indian Institute of Technology Madras
Principles of Communication
Y =
i = 1
Y2 =
X i , then
i = 1
2
i
2i + 2 k i k j i j i j
i
(2.49)
i < j
where the meaning of various symbols on the RHS of Eq. 2.49 is quite obvious.
We shall now give a few examples based on the theory covered so far in
this section.
Example 2.18
8
FX ( x ) = 2
x
16
We shall find
a)
x < 0
, 0 x 2
,
2 x 4
4 x
and
a) X
b) 2X
8
fX ( x ) =
1 x
8
0
x < 0
, 0 x 2
,
2 x 4
4 x
Therefore,
E[X] =
1
1 2
0 8 x d x + 2 8 x d x
2.56
Indian Institute of Technology Madras
Principles of Communication
1 7
31
+
=
4 3
12
b)
2X = E X 2 E [ X ]
2
1 2
1 3
0 8 x d x + 2 8 x d x
E X 2 =
1 15
47
+
=
3
2
6
2
X
47 31
167
=
=
6
144
12
Example 2.19
1, < x <
fX ( x ) =
2
2
0, otherwise
Let us find E [Y ] and Y2 .
E [Y ] =
cos ( x ) d x =
1
2
E Y =
2
Hence Y2 =
1
2
2
= 0.0636
cos2 ( x ) d x =
1
2
1
= 0.5
2
1
4
2 = 0.96
2
Example 2.20
Principles of Communication
Let us find
a)
a) E X Y 2 and
b) X Y
x y 2 f X ,Y ( x , y ) dx dy
1 1
x y ( x + y ) dx dy
0 0
b)
17
72
E [ X ] = E [Y ] =
Hence X Y =
1
11
g ( x ) f ( x | y ) dx
X |Y
(2.50)
x f X |Y ( x | y ) dx
(2.51)
Similarly, we can define the conditional variance etc. We shall illustrate the
calculation of conditional mean with the help of an example.
Example 2.21
2.58
Indian Institute of Technology Madras
Principles of Communication
1
, 0 < x < 1, 0 < y < x
f X ,Y ( x , y ) x
.
0 , outside
Let us compute E [ X | Y ] .
x dx
f X ,Y ( x , y )
fY ( y )
= ln y , 0 < y < 1
1
1
x
f X |Y ( x | y ) =
=
,
ln y
x ln y
Hence E ( X | Y ) =
y < x < 1
x x ln y d x
y
y 1
ln y
Consider a random experiment in which our interest is in the occurrence or nonoccurrence of an event A . That is, we are interested in the event A or its
complement, say, B . Let the experiment be repeated n times and let p be the
2.59
Indian Institute of Technology Madras
Principles of Communication
probability of occurrence of A on each trial and the trials are independent. Let X
denote random variable, number of occurrences of A in n trials'. X can be
equal to 0, 1, 2, ........., n . If we can compute P [ X = k ], k = 0, 1, ........., n , then
we can write f X ( x ) .
Taking a special case, let n = 5 and k = 3 . The sample space
(representing the outcomes of these five repetitions) has 32 sample points, say,
s1 , ..........., s32 . The sample point s1 could represent the sequence ABBBB . The
sample points such as ABAAB , A A A B B etc. will map into real number 3 as
shown in the Fig. 2.18. (Each sample point is actually an element of the five
dimensional Cartesian product space).
5
There are = 10 sample points for which X ( s ) = 3 .
3
In other words, for n = 5 , k = 3 ,
5
2
P [ X = 3] = p3 (1 p )
3
Generalizing this to arbitrary n and k , we have the binomial density, given by
2.60
Indian Institute of Technology Madras
Principles of Communication
fX ( x ) =
P (x i )
i = 0
(2.52)
n
ni
where Pi = p i (1 p )
i
As can be seen, f X ( x ) 0 and
fX ( x ) d x =
i = 0
Pi =
n i
n i
p (1 p )
0
i =
= (1 p ) + p
= 1
Example 2.22
ii)
Find the probability that the number of errors per block is greater than or
equal to 3.
Let X be the random variable representing the number of errors per block.
E [ X ] = n p = 16 0.01 = 0.16;
2.61
Indian Institute of Technology Madras
Principles of Communication
ii)
P ( X 3 ) = 1 P [ X 2]
= 1
i ( 0.1) (1 p )
16
i =0
16 i
= 0.002
Exercise 2.8
fX ( x ) =
m = 0
( x m)
m e
m!
(2.53)
Evidently f X ( x ) 0 and
f X ( x ) d x = 1 because
Since,
m
e =
,
m = 0 m!
( )
d e
d
= e =
E[X] =
we have
m m 1
1
=
m!
m = 0
m = 1
m m e
= e e =
m!
m = 1
2.62
Indian Institute of Technology Madras
m
m!
m
= e .
m = 0 m!
Principles of Communication
E X 2 = 2 + . Hence 2X = .
1
, a x b
fX ( x ) = b a
0
, elsewhere
(2.54)
(b a)
a+b
E[X] =
and 2X =
2
12
Note that the variance of the uniform PDF depends only on the width of the
interval ( b a ) . Therefore, whether X is uniform in ( 1, 1) or ( 2, 4 ) , it has the
same variance, namely
1
.
3
ii) Rayleigh:
2.63
Indian Institute of Technology Madras
Principles of Communication
x
x2
exp
, x 0
fX ( x ) = b
2 b
, elsewhere
0
(2.55)
2.64
Indian Institute of Technology Madras
Principles of Communication
Exercise 2.9
a)
dz
.
2
Show that if X
b
2
and
E X 2 = 2 b
iii) Gaussian
By far the most widely used PDF, in the context of communication theory
is the Gaussian (also called normal) density, specified by
fX ( x ) =
2 X
( x m X )2
, < x <
exp
2 2X
(2.56)
where m X is the mean value and 2X the variance. That is, the Gaussian PDF is
completely specified by the two parameters, m X and 2X . We use the symbol
N ( m X , 2X ) to denote the Gaussian density1. In appendix A2.3, we show that
PT
As can be seen from the Fig. 2.21, The Gaussian PDF is symmetrical with
respect to m X .
1
TP
PT
In this notation, N ( 0, 1) denotes the Gaussian PDF with zero mean and unit variance. Note that
if X is N m X , X , then Y =
2
X mX
is N ( 0, 1) .
X
2.65
Principles of Communication
Hence FX ( m X ) =
mX
f (x) d x
X
= 0.5
Consider P [ X a ] . We have,
P [ X a] =
2 X
( x m X )2
exp
dx
2 2X
x mX
z =
, we have
X
P [ X a] =
a mX
X
1
2
z2
2
dz
a mX
= Q
X
where Q ( y ) =
x2
exp
dx
2
2
1
(2.57)
Principles of Communication
Example 2.23
( ln y )2
1
, y 0
exp
fY ( y ) = 2 y
2 2
0
, otherwise
b)
Find E (Y )
c)
a)
2.67
Indian Institute of Technology Madras
Principles of Communication
dx
1
1
=
J =
dy
y
y
Also as y 0 , x and as y , x
( x )2
,
exp
2 2
2
Hence f X ( x ) =
< x <
Note that X is N ( , 2 )
( x )
1
x
2 2
e
e
Y = E e X =
b)
dx
x ( + 2 )
2 2
e
d x
= e
2
+
2
2
2
P [Y m ] = P [ X ln m ]
c)
1 ( x m X )2 ( y mY )2
( x mX ) ( y mY )
exp
fX , Y ( x , y ) =
+
2
2
2
k1
X
Y
X Y
k2
where,
k1 = 2 X Y
1 2
2.68
Indian Institute of Technology Madras
(2.58)
Principles of Communication
k 2 = 2 (1 2 )
= Correlation coefficient between X and Y
P2)
PT
f X Y ( x , y ) = f X ( x ) fY ( y ) iff = 0
That is, if the Gaussian variables are uncorrelated, then they are
independent. That is not true, in general, with respect to non-Gaussian
variables (we have already seen an example of this in Sec. 2.5.2).
P3) If Z = X + Y where and are constants and X and Y are jointly
1
TP
PT
Note that the converse is not necessarily true. Let fX and fY be obtained from fX , Y and let fX
and fY be Gaussian. This does not imply fX , Y is jointly Gaussian, unless X and Y
are
independent. We can construct examples of a joint PDF fX , Y , which is not Gaussian but results in
fX and fY that are Gaussian.
2.69
Indian Institute of Technology Madras
Principles of Communication
< 0.
Example 2.24
1
. Let us find the probability of
2
2.70
Indian Institute of Technology Madras
(X,Y)
Principles of Communication
Let
D of example 2.24
A P ( x , y ) B
1
2
1
x + 2 . Hence the required probability is,
2
2.71
Indian Institute of Technology Madras
Principles of Communication
X
X
P Y +
1 P Y +
2
2
2
Let Z = Y +
X
2
1
1
X =
2
2
1 2
1
X + Y2 + 2 . X Y
4
2
1
1 1
3
+ 1 2. . =
4
2 2
4
1 3
That is, Z is N , . Then W =
2 4
P [ Z 1] = P W
Z+
3
4
1
2 is N ( 0, 1)
P [ Z 2] = P W
( 3 ) Q
2.72
Indian Institute of Technology Madras
5
( 0.04 0.001) = 0.039
3
Principles of Communication
Exercise 2.10
A shown in
Fig. 2.25.
A of Exercise 2.10
(Z , W )
(X,Y)
into
2.73
Indian Institute of Technology Madras
B.
Principles of Communication
Exercise 2.11
X = ( 2 loge U1 ) 2 cos ( 2 U2 )
(2.59a)
Y = ( 2 loge U1 ) 2 sin ( 2 U2 )
(2.59b)
X1
Y = Y1 sin , where = 2 U2 .
Note: The transformation given by Eq. 2.59 is called the Box-Muller
transformation and can be used to generate two Gaussian random number
sequences from two independent uniformly distributed (in the range 0 to 1)
sequences.
2.74
Indian Institute of Technology Madras
Principles of Communication
Appendix A2.1
Proof of Eq. 2.34
The proof of Eq. 2.34 depends on establishing a relationship between the
differential area d z d w in the z w plane and the differential area d x d y in
the x y plane. We know that
fZ ,W ( z , w ) d z d w = P [ z < Z z + d z , w < W w + d w ]
Infinitesimal rectangle
ABC D
in the z w
parallelogram in the x y plane. (We may assume that the vertex A transforms
to A' , B to B' etc.) We shall now find the relation between the differential area of
the rectangle and the differential area of the parallelogram.
2.75
Indian Institute of Technology Madras
Principles of Communication
g 1
h 1
P2 = x +
dz, y +
d z
z
z
x
y
dz, y +
d z
= x +
z
z
x
y
P3 = x +
dw , y +
dw
w
w
That is,
V1 =
x
y
dz i +
dz j
z
z
V2 =
x
y
dw i +
dw j
w
w
where i and j are the unit vectors in the appropriate directions. Then, the area
2.76
Indian Institute of Technology Madras
Principles of Communication
V1 V2 =
x y y x
d z dw
z w z w
x, y
A = J
d z dw
z, w
That is,
x, y
fZ,W ( z , w ) = f X ,Y ( x , y ) J
z, w
=
fX , Y ( x , y )
z, w
J
x, y
2.77
Indian Institute of Technology Madras
Principles of Communication
Appendix A2.2
Q(
Function Table
Q () =
1
2
2
x
dx
Q (y )
Q (y )
Q (y )
0.0179
0.0139
0.0107
0.0082
0.0062
0.0047
0.0035
0.0026
0.0019
0.0013
0.0010
2.78
Indian Institute of Technology Madras
Q (y )
10 3
10 3
2
10 4
10 4
2
10 5
10 6
3.10
3.28
3.70
3.90
4.27
4.78
Principles of Communication
given by
erfc ( ) = 1 erf ( ) =
and the error function, erf ( ) =
Hence Q ( ) =
1
erfc
.
2
2
2.79
Indian Institute of Technology Madras
Principles of Communication
Appendix A2.3
f (x) d x
establishing
2
v
Let I =
dv =
Then, I
2
y
dy.
v2
y2
2
dv e 2 d y
= e
v2 + y2
2
dv d y
y
Let v = r cos and y = r sin . Then, r = v 2 + y 2 and = tan1 ,
v
e
0
r2
2
r dr d
= 2 or I =
That is,
Let v =
1
2
v2
2
dv = 1
(A2.3.1)
x mX
x
(A2.3.2)
dx
x
(A2.3.3)
Then, d v =
Using Eq. A2.3.2 and Eq. A2.3.3 in Eq. A2.3.1, we have the required result.
2.80
Indian Institute of Technology Madras
Principles of Communication
References
1)
2)
3)
2.81
Indian Institute of Technology Madras