154 Statistical Computing with R

6.2 Monte Carlo Methods for Estimation

Suppose X1 , . . . , Xn is a random sample from the distribution of X. An
estimator θ̂ for a parameter θ is an n variate function
θ̂ = θ̂(X1 , . . . , Xn )
of the sample. Functions of the estimator θ̂ are therefore n-variate func-
tions of the data, also. For simplicity, let x = (x1 , . . . , xn )T ∈ Rn , and let
x(1) , x(2) , . . . denote a sequence of independent random samples generated
from the distribution of X. Random variates from the sampling distribution
of θ̂ can be generated by repeatedly drawing independent random samples
(j) (j)
x(j) and computing θ̂(j) = θ̂(x1 , . . . , xn ) for each sample.

6.2.1 Monte Carlo estimation and standard error

Example 6.1 (Basic Monte Carlo estimation)

Suppose that X1 , X2 are iid from a standard normal distribution. Estimate
the mean difference E|X1 − X2 |.
To obtain a Monte Carlo estimate of θ = E[g(X1 , X2 )] = E|X1 − X2 | based
(j) (j)
on m replicates, generate random samples x(j) = (x1 , x2 ) of size 2 from
the standard normal distribution, j = 1, . . . , m. Then compute the replicates
(j) (j)
θ̂(j) = gj (x1 , x2 ) = |x1 − x2 |, j = 1, . . . , m, and the mean of the replicates
1  (j) 1  (j)
m m
(j)
θ̂ = θ̂ = g(X1 , X2 ) = |x − x2 |.
m i=1 m i=1 1
This is easy to implement, as shown below.
m <- 1000
g <- numeric(m)
for (i in 1:m) {
x <- rnorm(2)
g[i] <- abs(x[1] - x[2])
}
est <- mean(g)
One run produces the following estimate.
> est
[1] 1.128402
√ .
One can derive by integration that E|X1 − X2 | = 2/ π = 1.128379 and
 1 − X2 |) = .2 − 4/π. In this example the standard error of the estimate
V ar(|X
is (2 − 4/π)/m = 0.02695850. 
 Monte Carlo Methods in Inference 155

Estimating the standard error of the mean


The standard error of a mean X of a sample size n is V ar(X)/n. When the
distribution of X is unknown we can substitute for F the empirical distribution
Fn of the sample x1 , . . . , xn . The “plug-in” estimate of the variance of X is

1
n
V$
ar(x) = (xi − x̄)2 .
n i=1

Note that V$ ar(x) is the population variance of the finite pseudo population
{x1 , . . . , xn } with cdf Fn . The corresponding estimate of the standard error
of x̄ is
n 1/2 n 1/2
1 1 1 


se(x̄) =√ (xi − x̄)2 = (xi − x̄)2 .
n n i=1 n i=1

Using the unbiased estimator of V ar(X) we have

1/2
1 
n
1


se(x̄) = √ (xi − x̄)2 .
n n − 1 i=1

In a Monte Carlo experiment, the sample size is large and the two estimates
of standard error are approximately equal.
In Example 6.1 the sample size is m (the number of replicates of θ̂), and
the estimate of standard error of θ̂ is

> sqrt(sum((g - mean(g))^2)) / m

[1] 0.02708121
 .
In Example 6.1 we have the exact value se(θ̂) = (2 − 4/π)/m = 0.02695850
for comparison.

6.2.2 Estimation of MSE

Monte Carlo methods can be applied to estimate the MSE of an estima-
tor. Recall that the MSE of an estimator θ̂ for a parameter θ is defined by
MSE(θ̂) = E[(θ̂ − θ)2 ]. If m (pseudo) random samples x(1) , . . . , x(m) are gen-
erated from the distribution of X, then a Monte Carlo estimate of the MSE
of θ̂ = θ̂(x1 , . . . , xn ) is

1  (j)
m

M SE = (θ̂ − θ)2 ,
m j=1

(j) (j)
where θ̂(j) = θ̂(x(j) ) = θ̂(x1 , . . . , xn ).