You are on page 1of 12

Chapter 6

Expectation and Conditional Expectation

Lectures 24 - 30 In this chapter, we introduce expected value or the mean of

Lectures 24 - 30

In this chapter, we introduce expected value or the mean of a random variable. First we define expectation for discrete random variables and then for general random variable. Finally we introduce the notion of conditional expectations using conditional probabilities.

Definition 6.1. Two random variables

defined on a probability spaceprobabilities. Definition 6.1. Two random variables equal almost surely (in short a.s.) if are said to

equal almost surely (in short

a.s.) ifdefined on a probability space equal almost surely (in short are said to be Now we

a probability space equal almost surely (in short a.s.) if are said to be Now we
a probability space equal almost surely (in short a.s.) if are said to be Now we

are said to be

Now we give a useful characterization of discrete random variables.

be a discrete random variable defined on a probability space of and
be a
discrete random variable defined on a probability space
of
and

Letrandom variable defined on a probability space of and Theorem 6.0.23 Let there exists a partition

random variable defined on a probability space of and Let Theorem 6.0.23 Let there exists a

Theorem 6.0.23 Let

there exists a partition

. Then

such that

may be .
may be
.

may beLet there exists a partition . Then such that may be . where Proof. Let .

where

Proof. Let

. Then such that may be . may be where Proof. Let . Here be the

. Here

be the distribution function of

be where Proof. Let . Here be the distribution function of . Since be the set

. Since

Proof. Let . Here be the distribution function of . Since be the set of all

be the set of all discontinuities of

is discrete, we have

be the set of all discontinuities of is discrete, we have Set Then is pairwise disjoint

Set

Then

is pairwise disjoint andof . Since be the set of all discontinuities of is discrete, we have Set Then

set of all discontinuities of is discrete, we have Set Then is pairwise disjoint and Now
set of all discontinuities of is discrete, we have Set Then is pairwise disjoint and Now

Now define

set of all discontinuities of is discrete, we have Set Then is pairwise disjoint and Now

Then

is a partition ofof . Since be the set of all discontinuities of is discrete, we have Set Then

andthe set of all discontinuities of is discrete, we have Set Then is pairwise disjoint and

set of all discontinuities of is discrete, we have Set Then is pairwise disjoint and Now

Remark 6.0.6 If

is a discrete random variable on a probability spaceRemark 6.0.6 If , then the 'effective' range of is at the most countable. Here 'effective'

If is a discrete random variable on a probability space , then the 'effective' range of

, then the 'effective'

on a probability space , then the 'effective' range of is at the most countable. Here
on a probability space , then the 'effective' range of is at the most countable. Here

range of is at the most countable. Here 'effective' range means those values taken by which has positive probability. This leads to the name 'discrete' random variable.

Remark 6.0.7 If

that

is a discrete random variable, then one can assume without the loss of generality'discrete' random variable. Remark 6.0.7 If that Since if , then set and for . Theorem

Since if

, then setthen one can assume without the loss of generality Since if and for . Theorem 6.0.24

assume without the loss of generality Since if , then set and for . Theorem 6.0.24

and

for
for

.

Theorem 6.0.24 Let

Then if

Since if , then set and for . Theorem 6.0.24 Let Then if be such that

be such that

then set and for . Theorem 6.0.24 Let Then if be such that then Proof. For

then

set and for . Theorem 6.0.24 Let Then if be such that then Proof. For each
set and for . Theorem 6.0.24 Let Then if be such that then Proof. For each

Proof. For each

6.0.24 Let Then if be such that then Proof. For each , set Then clearly Also

, set

6.0.24 Let Then if be such that then Proof. For each , set Then clearly Also

Then clearly

Also if

such that then Proof. For each , set Then clearly Also if then . Therefore This

then

. Therefore
. Therefore

This completes the proof.

is a countable partition of

.
.

Definition 6.2. Let

proof. is a countable partition of . Definition 6.2. Let be a discrete random variable represented

be a discrete random variable represented by

expectation of

denoted bypartition of . Definition 6.2. Let be a discrete random variable represented by expectation of is

of . Definition 6.2. Let be a discrete random variable represented by expectation of denoted by

is defined as

of . Definition 6.2. Let be a discrete random variable represented by expectation of denoted by
of . Definition 6.2. Let be a discrete random variable represented by expectation of denoted by

. Then

provided the right hand side series converges absolutely.

Remark 6.0.8

In view of Remark 6.0.5., if

has range
has range

, then

Example 6.0.37 Let

Example 6.0.37 Let

Hence

Example 6.0.38 Let

Example 6.0.38 Let

Hence

be a Bernoulli(

) random variable. Then
) random variable. Then
6.0.38 Let Hence be a Bernoulli( ) random variable. Then be a Binomial random variable. Then
be a Binomial random variable. Then
be a Binomial
random variable. Then
) random variable. Then be a Binomial random variable. Then Here we used the identity Example

Here we used the identity

Example 6.0.39

Let

Hence

Example 6.0.40

Let

Hence

6.0.39 Let Hence Example 6.0.40 Let Hence be a Poisson ( ) random variable. Then be

be a Poisson (6.0.39 Let Hence Example 6.0.40 Let Hence ) random variable. Then be a Geometric ( )

Let Hence Example 6.0.40 Let Hence be a Poisson ( ) random variable. Then be a

) random variable. Then

be a Geometric ( ) random variable. Then
be a Geometric (
) random variable. Then
Theorem 6.0.25 (Properties of expectation) means. Then (i) If Let and be discrete random variables

Theorem 6.0.25 (Properties of expectation) means. Then

(i) If

Let

6.0.25 (Properties of expectation) means. Then (i) If Let and be discrete random variables with finite

and

(Properties of expectation) means. Then (i) If Let and be discrete random variables with finite .

be discrete random variables with finite

Then (i) If Let and be discrete random variables with finite . , then (ii) For
.
.

, then

(ii) For

be a representation of . Then
be a representation of
. Then

Proof. (i) Let

then (ii) For be a representation of . Then Proof. (i) Let . Hence implies for

. Hence

implies

a representation of . Then Proof. (i) Let . Hence implies for all . Now by

for all

. Now by setting
. Now by setting

(ii) Let

(i) Let . Hence implies for all . Now by setting (ii) Let has a representation

has a representation

one can use same partition for

(ii) Let has a representation one can use same partition for and . Therefore Hence Definition

and

Let has a representation one can use same partition for and . Therefore Hence Definition 6.3.

. Therefore

one can use same partition for and . Therefore Hence Definition 6.3. (Simple random variable) A

Hence

one can use same partition for and . Therefore Hence Definition 6.3. (Simple random variable) A

Definition 6.3. (Simple random variable) A random variable is said to be simple if it is discrete and the distribution function has only finitely many discontinuities.

function has only finitely many discontinuities. Theorem 6.0.26 Let be random variable in satisfying . of
function has only finitely many discontinuities. Theorem 6.0.26 Let be random variable in satisfying . of

Theorem 6.0.26Let

be random variable in

satisfying .
satisfying
.

of simple random variables

(i) For each

,
,

such that

, then there exists a sequenceTheorem 6.0.26 Let be random variable in satisfying . of simple random variables (i) For each

. (ii) For each Proof. For as , define simple random variable as follows: 's
.
.

(ii) For each

Proof. For

as

, define simple random variable as follows:
, define simple random variable
as follows:

's satisfies the following:

random variable as follows: 's satisfies the following: be a non negative random variable and Then
be a non negative random variable and
be a non negative random variable and
the following: be a non negative random variable and Then Lemma 6.0.3 Let be a sequence

Then

Lemma 6.0.3 Let

be a sequence of simple random

variables satisfying (i) and (ii) of Theorem 6.0.25. Then

Proof. Since

(i) and (ii) of Theorem 6.0.25. Then Proof. Since , we have exists and is given

, we have

exists and is given by

6.0.25. Then Proof. Since , we have exists and is given by (see exercise). Hence exists.
6.0.25. Then Proof. Since , we have exists and is given by (see exercise). Hence exists.

(see exercise). Hence

exists. Also since 's are simple, clearly, Therefore
exists. Also since
's are simple, clearly,
Therefore
simple and ,
simple and
,
since 's are simple, clearly, Therefore simple and , Hence to complete the proof, it suffices

Hence to complete the proof, it suffices to show that for

Let

where

Since

Also

the proof, it suffices to show that for Let where Since Also . Fix is a
the proof, it suffices to show that for Let where Since Also . Fix is a

. Fix

it suffices to show that for Let where Since Also . Fix is a partition of
it suffices to show that for Let where Since Also . Fix is a partition of
it suffices to show that for Let where Since Also . Fix is a partition of
it suffices to show that for Let where Since Also . Fix is a partition of

is a partition of

, set for

and

.
.

, we have for each

,
,
it suffices to show that for Let where Since Also . Fix is a partition of

Hence

From the definition of we have Hence     (6.0.1) Using continuity property of probability,

From the definition of

From the definition of we have Hence     (6.0.1) Using continuity property of probability, we

we have

From the definition of we have Hence     (6.0.1) Using continuity property of probability, we

Hence

 
    (6.0.1)
 

(6.0.1)

Using continuity property of probability, we have

 
 
   
 

Now let,

in (6.0.1), we get

in (6.0.1), we get

 
 

Since,

is arbitrary, we get

is arbitrary, we get

 
we get   Since, is arbitrary, we get   This completes the proof. Definition 6.4. The

This completes the proof.

Definition 6.4. The expectation of a non negative random variable

is defined as

is defined as

 
 
    (6.0.2)
 

(6.0.2)

where

is a sequence of simple random variables as in Theorem 6.0.25.

is a sequence of simple random variables as in Theorem 6.0.25.

 

Remark 6.0.9

One can define expectation of

, non negative random variable, as

, non negative random variable, as

 
 

But we use Definition 6.4., since it is more handy.

 

Theorem 6.0.27 Let

Theorem 6.0.27 Let be a continuous non negative random variable with pdf

be a continuous non negative random variable with pdf

Theorem 6.0.27 Let be a continuous non negative random variable with pdf . Then

. Then

 
   
 

Proof. By using the simple functions given in the proof of Theorem 6.0.25, we get

 
 

(6.0.3)

where is the point given by the mean value theorem.
where
is the point given by the mean value theorem.

Hence

Definition 6.5. Let

exists if either

where

theorem. Hence Definition 6.5. Let exists if either where be a random variable on or is
theorem. Hence Definition 6.5. Let exists if either where be a random variable on or is

be a random variable ontheorem. Hence Definition 6.5. Let exists if either where or is finite. In this case .

or

6.5. Let exists if either where be a random variable on or is finite. In this

is finite. In this case

where be a random variable on or is finite. In this case . The mean or
where be a random variable on or is finite. In this case . The mean or

. The mean or expectation of

is defined as

In this case . The mean or expectation of is defined as is said to Note
In this case . The mean or expectation of is defined as is said to Note

is said to

case . The mean or expectation of is defined as is said to Note that is

Note that

mean or expectation of is defined as is said to Note that is the positive part

is the positive part and

is defined as is said to Note that is the positive part and is the negetive

is the negetive part of

Note that is the positive part and is the negetive part of Theorem 6.0.28 Proof. Set

Theorem 6.0.28

Proof. Set

Let

be a continuous random variable with finite mean and pdf
be a continuous random variable with finite mean and pdf

. Then

the negetive part of Theorem 6.0.28 Proof. Set Let be a continuous random variable with finite

Hence

Then is a sequence of simple random variables such that Similarly, set Hence Then Now

Then

Then is a sequence of simple random variables such that Similarly, set Hence Then Now and

is a sequence of simple random variables such that

Then is a sequence of simple random variables such that Similarly, set Hence Then Now and

Similarly, set

sequence of simple random variables such that Similarly, set Hence Then Now and (6.0.4) (6.0.5) The

Hence

of simple random variables such that Similarly, set Hence Then Now and (6.0.4) (6.0.5) The last

Then

Now

random variables such that Similarly, set Hence Then Now and (6.0.4) (6.0.5) The last equality follows
random variables such that Similarly, set Hence Then Now and (6.0.4) (6.0.5) The last equality follows

and

random variables such that Similarly, set Hence Then Now and (6.0.4) (6.0.5) The last equality follows

(6.0.4)

(6.0.5)

The last equality follows by the arguments from the proof of Theorem 6.0.26. Combining (6.0.4) and (6.0.4), we get

of Theorem 6.0.26. Combining (6.0.4) and (6.0.4), we get Now as in the proof of Theorem

Now as in the proof of Theorem 6.0.26, we complete the proof.

We state the following useful properties of expectation. The proof follows by approximation argument using the corresponding properties of simple random variables

Theorem 6.0.29 Let

properties of simple random variables Theorem 6.0.29 Let be random variables with finite mean. Then (i)

be random variables with finite mean. Then

,
,

(ii) For

(iii) Let

, (ii) For (iii) Let . Then be a random variable such that has finite mean
, (ii) For (iii) Let . Then be a random variable such that has finite mean
. Then
. Then

be a random variable such that

has finite mean and

.
.

In the context of Riemann integration, one can recall the following convergence theorem.

`` If

is a sequence of continuous functions defined on theone can recall the following convergence theorem. `` If , then such that uniformly in i.e.,

`` If is a sequence of continuous functions defined on the , then such that uniformly

, then

is a sequence of continuous functions defined on the , then such that uniformly in i.e.,

such that

of continuous functions defined on the , then such that uniformly in i.e., to take limit

uniformly in

functions defined on the , then such that uniformly in i.e., to take limit inside the

i.e., to take limit inside the integral, one need uniform convergence of functions. In many situations in it is highly unlikely to get uniform convergence. In fact uniform convergence is not required to take limit inside an integral. This is illustrated in the following couple of theorem. The proof of them are beyond the scope of this course.

Theorem 6.0.30 (Monotone convergence theorem)

random variables such that

be an increasing sequence of nonnegative(Monotone convergence theorem) random variables such that Let . Then means .] [Here Theorem 6.0.31 (Dominated

Let

. Then means
. Then
means

.]

an increasing sequence of nonnegative Let . Then means .] [Here Theorem 6.0.31 (Dominated Convergence Theorem)

[Here

Theorem 6.0.31 (Dominated Convergence Theorem)

(i)

.] [Here Theorem 6.0.31 (Dominated Convergence Theorem) (i) has finite mean. Let (ii) (iii) Then be

has finite mean.

Let

(Dominated Convergence Theorem) (i) has finite mean. Let (ii) (iii) Then be random variables such that

(ii)

(iii)

Convergence Theorem) (i) has finite mean. Let (ii) (iii) Then be random variables such that Definition

Then

Theorem) (i) has finite mean. Let (ii) (iii) Then be random variables such that Definition 6.6.

be random variables such that

Definition 6.6. (Higher Order Moments) Let

moment of

that Definition 6.6. (Higher Order Moments) Let moment of be a random variable. Then th central

be a random variable. Then

th central moment of

moment of be a random variable. Then th central moment of th . The second central

th

. The second central moment is

Then th central moment of th . The second central moment is is called the and

is called the

moment of th . The second central moment is is called the and is called the
moment of th . The second central moment is is called the and is called the
moment of th . The second central moment is is called the and is called the

and

is called the

called the variance. Now we state the following theorem whose proof is beyond the scope of this course.

Theorem 6.0.32 Let be a continuous random variable with pdf function such that the integral
Theorem 6.0.32 Let
be a continuous random variable with pdf
function such that the integral
is finite. Then

and

pdf function such that the integral is finite. Then and be a continuous The above theorem

be a continuous

The above theorem is generally referred as the ``Law of unconscious statistician'' since often users treat the

above as a definition itself. Now we define conditional expectation denoted by E[Y|X] of the random variable Y given the information about the random variable X. If Y is a Bernoulli (p) random variable and X any discrete random variable, then we expect E[Y|X = x] to be P{Y = 1|X = x}, since we know that EY = p = P{Y = 1}. i.e.,

Where is the
Where
is the

conditional pmf of Y given X. Now since we expect conditional expectation to be

liner and any discrete random variable can be written as a liner combination of Bernoulli random variable we get the following definition.

Definition 6.7. Let

we get the following definition. Definition 6.7. Let are discrete random variable with conditional pmf

are discrete random variable with conditional pmf

conditional expectation of

variable with conditional pmf conditional expectation of given is defined as . Then Example 6.0.41 .

given

with conditional pmf conditional expectation of given is defined as . Then Example 6.0.41 . Calculate

is defined as

.
.

Then

pmf conditional expectation of given is defined as . Then Example 6.0.41 . Calculate Let be

Example 6.0.41

expectation of given is defined as . Then Example 6.0.41 . Calculate Let be independent random

. Calculate

Let
Let

be independent random variables with geometric distribution of parameter

, where

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

Set

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

For

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore
Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

Now

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

Therefore

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

i.e.,

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

Now

Let be independent random variables with geometric distribution of parameter , where Set For Now Therefore

When X and Y are discrete random variable. E[Y|X] is defined using conditional pmf of Y given X. Hence we define E[Y|X] when X and Y are continuous random variable with joint pdf f in a similar way as follows.

Definition 6.8. Let

pdf f in a similar way as follows. Definition 6.8. Let be continuous random variable with

be continuous random variable with conditional pdf

expectation of

random variable with conditional pdf expectation of given is defined as . Then conditional Remark 6.0.10

given

random variable with conditional pdf expectation of given is defined as . Then conditional Remark 6.0.10

is defined as

with conditional pdf expectation of given is defined as . Then conditional Remark 6.0.10 One can

. Then conditional

pdf expectation of given is defined as . Then conditional Remark 6.0.10 One can extend the

Remark 6.0.10 One can extend the defination of E[Y|X] when X is any random variable (discrete, continuous or mixed) and Y is a any random variable with finite mean. But it is beyound the scope of this course.

, marginal pmfs
, marginal pmfs

Theorem 6.0.33 (i)

scope of this course. , marginal pmfs Theorem 6.0.33 (i) respectively. Then if Let be discrete

respectively. Then if

, marginal pmfs Theorem 6.0.33 (i) respectively. Then if Let be discrete random variables with joint

Let

marginal pmfs Theorem 6.0.33 (i) respectively. Then if Let be discrete random variables with joint pmf

be discrete random variables with joint pmf

and

has finite mean, then

random variables with joint pmf and has finite mean, then (ii) Let Then if be continuous
random variables with joint pmf and has finite mean, then (ii) Let Then if be continuous
random variables with joint pmf and has finite mean, then (ii) Let Then if be continuous
random variables with joint pmf and has finite mean, then (ii) Let Then if be continuous

(ii) Let

Then if

be continuous random variables with joint pdf

, marginal pdfs

and

respectively.

has finite mean, thenvariables with joint pdf , marginal pdfs and respectively. Proof. We only prove (ii). Example 6.0.42

pdf , marginal pdfs and respectively. has finite mean, then Proof. We only prove (ii). Example

Proof. We only prove (ii).

has finite mean, then Proof. We only prove (ii). Example 6.0.42 Let . Note that be

Example 6.0.42 Let

mean, then Proof. We only prove (ii). Example 6.0.42 Let . Note that be continuous random
mean, then Proof. We only prove (ii). Example 6.0.42 Let . Note that be continuous random

. Note that

be continuous random variables with joint pdf given by

Let . Note that be continuous random variables with joint pdf given by Find and hence

Find

and hence calculate

Let . Note that be continuous random variables with joint pdf given by Find and hence

and

Let . Note that be continuous random variables with joint pdf given by Find and hence

elsewhere. Hence for

,
,
Also elsewhere. Therefore

Also

Also elsewhere. Therefore

elsewhere. Therefore

Also elsewhere. Therefore