Monte Carlo Methods and Statistical Computing: My Personal Experience

Monte Carlo Methods and Statistical Computing:
My Personal Experience
Debasis Kundu
Department of Mathematics & Statistics

Indian Institute of Technology Kanpur
November 29, 2014

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E
Preface
A Brief History
Application
Major Ingradients
What we can do?
Statistical Computation
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
6 Statistical Computation
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Limitations:
1. I must admit that the topics I am going to

cover are definitely not exhaustive.
2. Topics are purely of my own interest which
have developed over the last 30 years.
3. I am not going to describe any statistical
package.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Advantages:
1. Packages have their own problems.

2. Different packages can give different answers
even on a relatively simple problem.
3. We should know the limitations of the
packages.
4. I will try to provide a general approach

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Journals:
1. Journal of Computational and Graphical

Statistics.
2. Computational Statistics and Data Analysis.
3. Journal of Statistical Computation and
Simulation.
4. Statistical Computing
5. Communications in Statistics - Simulation
and Computation
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Books:
1. Simulation by Sheldon Ross

2. Nonuniform Random Deviate Generator, L.
Devroye
3. Simulation modelling and analysis, Law and
Kelton Journal of Statistical Computation and
Simulation.
4. Statistical Computing: J.F. Keneddy and R.
Gentle
5. Statistical Computing, D. Kundu and A. Basu
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Monte Carlo Method: Definition
1. A broad class of numerical alogrithm depends

on repeated random sampling.
2. If it is not possible to obtain the exact
analytical solution often Monta Carlo method
can be used to provide a very good approximate
solution

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Monte Carlo Method: A brief history
1. It was invented by Stanislaw Ulam, a famous

Polish Mathematician, in the late 1940.
2. John von Neumann first wrote the computer
code to perform Monte Carlo simulations
3. Metropolis gave this name

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Where it can be used?
1. Calculating the area below a curve.

2. Calculating multidimensional integration.
3. Optimization.
4. Analyzing any complicated stochastic system
(model).

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Examples
Suppose we want to compute

Z b
2
e −x dx.
a
Or suppose we want to compute
Z b1 Z bk
... f (x1 , . . . , xk )dx1 . . . dxk .
a1 ak

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Examples:Contd.
Suppose we want to find the maximum or minimum
of the following function
f (x1 , . . . , xk ),
where a1 ≤ x1 ≤ b1 , . . . , ak ≤ xk ≤ bk .
Or suppose we want to analyze the following
non-linear model
y (x1 , . . . , xk ) = f (x1 , . . . , xk , θ) + e.
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Back Ground
1. Knowledge of Basic Probability.

2. Discrete and Continuous random variables.
3. Stochastic models.
4. Generation of random numbers.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Knowlwedge of Basic Probability
1. Idea of a random experiment.

2. Basic idea of convergence of random
variables.
3. Weak and strong law of large numbers.
4. Central limit theorem.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Discrete Random Variables
1. Uniform.
2. Binomial.
3. Geometric.
4. Poisson.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Continuous Random Variables
1. Uniform.
2. Exponential.
3. Normal.
4. Gamma.
5. Log-concave probability density function

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Generation of Random Numbers
First we need to know how to generate Uniform

random numbers. This is the most basic problem.
In this respect we use group theory results and
machine powers.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Generation of Non-Uniform Random Numbers
The most popular method is the inverse

transformation. The following result can be used. If
X is a random variable with the distribution
function F (x), then F (X ) follows uniform
distribution. Therefore
X = F −1 (U)

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Generation of Discrete Random Numbers
All the discrete distributions can be generated using

inverse transformation method. Suppose
P(X = ai ) = pi , for i = 1, 2, . . .. Without loss of
generality we can assume a1 < a2 < . . .. Draw a
uniform
Pk−1 randomP number say u, if
k
i=1 pi < u < i=1 pi , then X takes the value ak .

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Generation of Continuous Random Numbers
Many continuous random variables can be

generated using inverse transformation method, for
example exponential, Weibull, generalized
exponential distributions etc. On the other hand
several well known distribution cannot be obtained
using inverse transformation method. For example
normal, gamma etc.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Generation of Continuous Random Numbers
If a continuous distribution cannot be generated

using inverse transformation method, one of the
most useful method is the acceptance rejection
method. The idea is as follows. If we want to
generate from f (x), try to find g (x), from which
generation is simple so that it satisfies the following
f (x) ≤ cg (x).

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Acceptance Rejection Method: Algorithm
1.Generate Y from g (x).

2. Generate a uniform random vaiable U.
3. If U ≤ f (Y )/cg (Y ), set X = Y , otherwise
return to 1.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Acceptance Rejection Method: Theorem
Theorem:
1. The random variable generated by this
method has density function f (x)
2. The number of iterations of the algorithm
that are needed is a geometric random variable
with mean c,

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Acceptance Rejection Method: Example
Example 1:
Suppose we want to generate from
f (x) = 20x(1 − x)3 ; 0 < x < 1.
Take
g (x) = 1, 0 < x < 1.
c = 135/64.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold

Example 2:
2
f (x) = √ x 1/2 e −x ; 0 < x < ∞.
π
Take
2
g (x) = e −2x/3 0 < x < ∞.
3
and
33/2
c= .
(2πe)1/2
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example 3:
2 2
f (x) = √ e −x /2 ; 0 < x < ∞.
2π
Take
g (x) = e −x ; 0 < x < ∞.
and p
c= 2e/π.
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Very Simple Example
Consider the following simple linear regression model
Y = Xb + e
We know the LSE’s can be obtained as

b = (XT X)−1 XT Y.
b
We have a complete very nice theory when all the

components of the errors are i.i.d. normal random
variables.
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Very Simple Example: Contd.
Consider some slightly different conditions of the

same model.
1. What will happen if the errors are not
normal?
2. What will happen if the errors are heavy tail?
3. What will happen if there are outliers?
4. What will happen if the errors are correlated?

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Very Simple Example: Contd.
In all these cases Monte Carlo Method can be used

to asses the performances of the estmimators. It is
very simple also.
1. Generate e
2. Generate Y.
3. b
Calculate b.
4. Repeat step 1 to step 4, several times.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example
Consider the following simple linear regression model
Y = Xb + e
Suppose we want to estimate b by minimizing the

least absolute errors i.e.
b = argmin|Y − Xb|.
b
Theories are quite complicated. All the results are

asymptotic in nature.
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example
Consider the following non-linear regression model
Y = f (X, θ) + e
Here f is a known function the vector X is also
known, the paramete vector θ is unknown. The
problem is to estimate the parameter vector θ,
based on a sample of size n.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example
Natural estimators will be

n
X
θb = argmin| Yi − f (Xi , θ)|2 .
i=1
or n
X
θb = argmin| Yi − f (Xi , θ)|.
i=1
Theories are quite complicated. All the results are
asymptotic in nature.
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example
Monte Carlo method can be used to asses the

perofrmance of the estimators.
Based on the Monte Carlo method the biases
and the mean squared errors can be calculated.
Based on Bootstrap method confidence intervals
also can be obtained.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Example: Importance Sampling
In Bayesian analysis often we need to compute the

posterior mean as follows:
Z
θ = E (h(X)) = h(x)f (x)dx.
Here f (x) is the PDF of X, and x can be a very

high dimensional. In Bayesian analysis f (x) is the
posterior density function.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Monte Carlo simulation technique can be used to

approximate the value of θ as follows:
N
b 1X
θ= h(Xi ),
N
i=1
here X1 , . . . , XN is a random sample of size N from

f (X).

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Often it is observed that it is not very easy to

generate samples from f (x).
Z Z
h(x)f (x)
θ = h(x)f (x)dx = g (x)d(x).
g (x)
N
X
1 h(xi )f (xi )
θb = ,
N g (xi )
i=1
here X1 , . . . , XN is a random sample of size N from
g (X).
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Important Issues
1. Finding Maximum likelihood estimators in a

general problem.
2. Finding least squares estimators of linear
regresssion model when the design matrix is
close to a singular matrix
3. Non-linear regression model if the number of
parameters are very high

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
MLE
It basically invloves maximizing a function of the

form:
f (θ1 , . . . , θp )
Standard method is to use Newton-Raphson
method:

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Newton-Raphson Method
Assuming sufficiently smooth f (θ), we want to solve
∂f (θ)
=0
∂θ
Standard method is to use Newton-Raphson
method. Using Taylor series expansion, it can be
easily obtained:
2 (k) −1
(k+1) (k) ∂ f (θ ) ∂f (θ(k) )
θ =θ −
∂θ∂θT ∂θ
Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Profile Likelihood Method
1. For fixed θ1 , . . . , θk , try to maximize with

respect to θk+1 , . . . , θp
2. Maximize with respect to θ1 , . . . , θk .

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
EM Algorithm
Suppose the data are coming from a mixture model,

and we compute the MLEs of the unknown
parameters
k
X
f (x) = πj fj (x; θj ),
j=1
Pk
πj ≥ 0, j=1 πj = 1.

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Mixture Model: MLE
Based on a random sample x1 , . . . , xn , we want to

compute the MLEs of the unknown parameters
n X
Y k
L(π, θ) = πj fj (xi ; θj ).
i=1 j=1

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Missing Value Problem
We treat this as a missing value problem

1. Assume the data are of the form (x, δ)
2. Compute E (δ|Data)
3. Continue the process

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Copula Method
Any multivarite distribution can be written uniquely

as follows:
F (x1 , . . . , xp ; θ) = C (F1 (x1 ; θ1 ), . . . , Fp (xp : θp ); γ)

First estimate the marginal parameters, and then
estimate the copula parameters

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Non-linear regression
Consider the following model
p
X
y (t) = [Ak cos(ωk t) + Bk sin(ωk t)] + e(t)
k=1
Estimate the unknown parameters

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Outline
1 Preface
2 A Brief History
3 Application
4 Major Ingradients
5 What we can do?
7 Stories Untold

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Very important areas
Bayesian comutation: mainly MCMC

Classification problem
Small n large p problem
Non-parametric regression
Functional data analysis

Preface
A Brief History
Application
Major Ingradients
What we can do?
Stories Untold
Thank You

Monte Carlo Methods and Statistical Computing: My Personal Experience

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Monte Carlo Methods and Statistical Computing: My Personal Experience

Uploaded by

Copyright:

Available Formats

Monte Carlo Methods and Statistical Computing:

Department of Mathematics & Statistics

November 29, 2014

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

1. I must admit that the topics I am going to

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

1. Packages have their own problems.

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

1. Journal of Computational and Graphical

1. Simulation by Sheldon Ross

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Monte Carlo Method: Definition

1. A broad class of numerical alogrithm depends

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Monte Carlo Method: A brief history

1. It was invented by Stanislaw Ulam, a famous

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Where it can be used?

1. Calculating the area below a curve.

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Suppose we want to compute

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

1. Knowledge of Basic Probability.

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Knowlwedge of Basic Probability

1. Idea of a random experiment.

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Discrete Random Variables

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Continuous Random Variables

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Generation of Random Numbers

First we need to know how to generate Uniform

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Generation of Non-Uniform Random Numbers

The most popular method is the inverse

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Generation of Discrete Random Numbers

All the discrete distributions can be generated using

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Generation of Continuous Random Numbers

Many continuous random variables can be

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Generation of Continuous Random Numbers

If a continuous distribution cannot be generated

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Acceptance Rejection Method: Algorithm

1.Generate Y from g (x).

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Acceptance Rejection Method: Theorem

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Acceptance Rejection Method: Example

f (x) = 20x(1 − x)3 ; 0 < x < 1.

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Acceptance Rejection Method: Example

Acceptance Rejection Method: Example

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal E

Very Simple Example

Consider the following simple linear regression model