Geostatistics for

Large Data Sets

Whitney Huang
Geostatistical Modeling for Large Data Sets
Motivation

Methods
Covariance
Whitney Huang tapering
Low–rank
approximation
Likelihood
Department of Statistics approximation
Purdue University Gaussian Markov
random field
approximation

October 28, 2014

 Geostatistics for
Outline Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
Motivation Low–rank
approximation
Likelihood
approximation
Gaussian Markov
random field
approximation
Methods
Covariance tapering
Low–rank approximation
Likelihood approximation
Gaussian Markov random field approximation
 Geostatistics for
Gaussian process (GP) geostatistics Large Data Sets

Whitney Huang
Model:
Motivation
Y (s) = µ(s) + η(s) + ε(s), s ∈ S ⊂ Rd Methods
Covariance
tapering
where Low–rank
approximation
Likelihood
approximation
µ(s) = XT (s)β, {η(s)}s∈S ∼ GP (0, C (·, ·)) Gaussian Markov
random field
approximation

C (s, s0 ) = σ 2 ρθ (ks − s0 k), and ε(s) ∼ N(0, τ 2 ) ∀s ∈ S

Log-likelihood:
Given data Y = (Y (s1 ), · · · , Y (sn ))T

1
ln (β, θ, σ 2 , τ 2 ) ∝ − log Σ(θ, σ 2 ) + τ 2 In 
 
2
1 −1
− (Y − XT β)T Σ(θ, σ 2 ) + τ 2 In

(Y − Xβ)
2
where Σ(θ, σ 2 )i,j = σ 2 ρθ (ksi − sj k), i, j = 1, · · · , n
 Geostatistics for
“Big n Problem” in geostatistics Large Data Sets

Whitney Huang

Motivation

I Modern environmental instrument has produced a Methods

Covariance
wealth of space–time data ⇒ n is big tapering
Low–rank
approximation
I Evaluation of the likelihood function involves factorizing Likelihood
approximation
large covariance matrices that generally requires Gaussian Markov
random field
3 approximation
I O(n ) operations
I O(n2 ) memory
I Modeling strategies are needed to deal with large spatial
data set.
I parameter estimation ⇒ MLE, Bayesian
I spatial interpolation ⇒ Kriging
I multivariate spatial data, spatio-temporal data
 Geostatistics for
“Big n Problem” in geostatistics Large Data Sets

Whitney Huang

Motivation

I Modern environmental instrument has produced a Methods

Covariance
wealth of space–time data ⇒ n is big tapering
Low–rank
approximation
I Evaluation of the likelihood function involves factorizing Likelihood
approximation
large covariance matrices that generally requires Gaussian Markov
random field
3 approximation
I O(n ) operations
I O(n2 ) memory
I Modeling strategies are needed to deal with large spatial
data set.
I parameter estimation ⇒ MLE, Bayesian
I spatial interpolation ⇒ Kriging
I multivariate spatial data, spatio-temporal data
 Geostatistics for
“Big n Problem” in geostatistics Large Data Sets

Whitney Huang

Motivation

I Modern environmental instrument has produced a Methods

Covariance
wealth of space–time data ⇒ n is big tapering
Low–rank
approximation
I Evaluation of the likelihood function involves factorizing Likelihood
approximation
large covariance matrices that generally requires Gaussian Markov
random field
3 approximation
I O(n ) operations
I O(n2 ) memory
I Modeling strategies are needed to deal with large spatial
data set.
I parameter estimation ⇒ MLE, Bayesian
I spatial interpolation ⇒ Kriging
I multivariate spatial data, spatio-temporal data
 Geostatistics for
Modeling strategies in the literature Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
I Covariance tapering (Furrer et al. 06, Kaufman et al. Low–rank
approximation
08, Du et al. 09) Likelihood
approximation
Gaussian Markov
I Low–rank approximation (Cressie & Johannesson 08, random field
approximation
Banerjee et al. 08)
I Likelihood approximation (Vecchia 88, Stein 04)
I Gaussian Markov random field approximation (Rue &
Tjelmeland 02, Rue & Held 05, Lindgren et al. 11)
 Geostatistics for
Modeling strategies in the literature Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
I Covariance tapering (Furrer et al. 06, Kaufman et al. Low–rank
approximation
08, Du et al. 09) Likelihood
approximation
Gaussian Markov
I Low–rank approximation (Cressie & Johannesson 08, random field
approximation
Banerjee et al. 08)
I Likelihood approximation (Vecchia 88, Stein 04)
I Gaussian Markov random field approximation (Rue &
Tjelmeland 02, Rue & Held 05, Lindgren et al. 11)
 Geostatistics for
Modeling strategies in the literature Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
I Covariance tapering (Furrer et al. 06, Kaufman et al. Low–rank
approximation
08, Du et al. 09) Likelihood
approximation
Gaussian Markov
I Low–rank approximation (Cressie & Johannesson 08, random field
approximation
Banerjee et al. 08)
I Likelihood approximation (Vecchia 88, Stein 04)
I Gaussian Markov random field approximation (Rue &
Tjelmeland 02, Rue & Held 05, Lindgren et al. 11)
 Geostatistics for
Modeling strategies in the literature Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
I Covariance tapering (Furrer et al. 06, Kaufman et al. Low–rank
approximation
08, Du et al. 09) Likelihood
approximation
Gaussian Markov
I Low–rank approximation (Cressie & Johannesson 08, random field
approximation
Banerjee et al. 08)
I Likelihood approximation (Vecchia 88, Stein 04)
I Gaussian Markov random field approximation (Rue &
Tjelmeland 02, Rue & Held 05, Lindgren et al. 11)
 Geostatistics for
Outline Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
Motivation Low–rank
approximation
Likelihood
approximation
Gaussian Markov
random field
approximation
Methods
Covariance tapering
Low–rank approximation
Likelihood approximation
Gaussian Markov random field approximation
 Geostatistics for
Covariance tapering (Furrer et al. 06) Large Data Sets

We replace the C (khk) by Whitney Huang

Ctap (h; γ) = ρtap (h; γ) ◦ C (h) Motivation

Methods
Covariance
tapering
Low–rank
where ρtap (h; γ) is an isotropic correlation function with approximation
Likelihood
compact support (ρtap (h) = 0 if h ≥ γ) and ◦ denotes the approximation
Gaussian Markov
random field
Schur product approximation
 Geostatistics for
Covariance tapering cont’d Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
Low–rank
approximation
Likelihood
approximation
Gaussian Markov
random field
approximation

I Ctap (h) is a valid covariance function

I Sparse matrix algorithm can be used
 Geostatistics for
Low–rank approximation Large Data Sets

Whitney Huang

Motivation

Hierarchical Representation (assume zero mean spatial Methods

Covariance
process) tapering
Low–rank
approximation
Likelihood
approximation
Y = η + ε, ε ∼ MVN(0, Σε ) Gaussian Markov
random field
approximation

η = Hα + ξ, ξ ∼ MVN(0, Σξ )
α ∼ MVN(0, Σα )
where α = (α1 , · · · , αp )T such that p  n and H is
mapping from the latent process, α, to the true spatial
process of interest, η. Σε and Σξ and diagonal.
 Geostatistics for
Low–rank approximation cont’d Large Data Sets

Whitney Huang

To carry out the spatial interpolation (i.e. kriging) of Motivation

η(s0 )|{Y (si )}ni=1 one need to compute Methods

Covariance
tapering
 −1 Low–rank

HΣα H T + V approximation
Likelihood
approximation
Gaussian Markov
random field
where V = Σε + Σξ . approximation

Sherman–Morrison–Woodbury formula

−1
(A + BCD)−1 = A−1 − A−1 B C −1 + DA−1 B DA−1
In the case of low–rank model, we have
 −1  −1
HΣα H T + V = V−1 −V−1 H Σ−1 α + H T −1
V H H T V−1
 Geostatistics for
Fixed Rank Kriging (Cressie & Johannesson 08) Large Data Sets

Whitney Huang

Motivation

Methods
Y = Xβ + ZW∗ + ε Covariance
tapering
Low–rank
approximation
Likelihood
approximation

Let W∗ = {w (s∗i )pi=1 } be be latent variables at p  n known Gaussian Markov

random field
approximation
knots {si∗ }pi=1 and Z (·) be a known basis function

The fixed rank kriging is equivalent to the following low rank

model
p
X
Y (s) = X(s)β + Z (s − s∗j )Wj + ε(s)
j=1
 Geostatistics for
Gaussian Predictive Process (Banerjee et al. 08) Large Data Sets

Whitney Huang

Motivation

Methods
Use a model Covariance
tapering
Low–rank

Y (s) = X(s)T β + Hα(s) + ε(s)

approximation
Likelihood
approximation
Gaussian Markov
random field
to approximate the original spatial process approximation

Y (s) = X(s)T β + η(s) + ε(s)

Knots: {s∗1 , · · · , s∗p } where p  n

T
⇒ α = {α(s∗i )}pi=1 , H(θ) = Cov(si , s∗j ; θ) [Σα ]−1

 Geostatistics for
Likelihood approximation (Vecchia 88) Large Data Sets

Whitney Huang

Motivation

Methods
Partition the observation vector Y into sub–vector Covariance
tapering
Y1 , · · · , Yb and let Y(j) = (YT T T
1 , · · · , Yj ) Low–rank
approximation
Likelihood
The exact likelihood approximation
Gaussian Markov
random field
approximation
b
Y
p(Y; β, θ) = p(Y1 ; β, θ) p(Yj |Y(j−1) ; β, θ)
j=2

Approximate the exact likelihood by replacing Y(j−1) by a

sub–vector S(j−1) of Y(j−1)
 Geostatistics for
Markov Random Fields Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
tapering
Low–rank
approximation
Likelihood
approximation
Gaussian Markov
random field
approximation
 Geostatistics for
Gaussian Markov Random Fields (GMRF) Large Data Sets

Whitney Huang

Motivation

Methods
Covariance
Definition tapering
Low–rank
approximation
Let the neighbors to a point i be the points Ni that are Likelihood
approximation
“close" to i. A Gaussian random field X ∼ N(µ, Σ = Q −1 ) Gaussian Markov
random field
that satisfies approximation

p(Xi |Xj , j 6= i) = p(Xi |Xj : j ∈ Nj )

is a Gaussian Markov random field (GMRF) with Qij = 0 iff

Xi ⊥ Xj |X−ij
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
Remarks: GP vs. GMRF in geostatistical modeling Large Data Sets

Whitney Huang

Motivation
I +: GP model is widely used in modeling continuously
Methods
indexed spatial data in which the covariance function Covariance
tapering
characterizes the process properties Low–rank
approximation
Likelihood
I –: Inference involves factorizing covariance matrices approximation
Gaussian Markov
random field
I +: GMRF model is computationally efficient due to the approximation

sparse precision matrix

I –: Only for discretely indexed spatial data
Main idea of GMRF approach:
SPDE
GP ⇐⇒ GMRF
|{z} | {z }
inference computation
 Geostatistics for
GP/Stochastic Partial Differential Equation Large Data Sets

Whitney Huang
(SPDE) connection (Whittle 1954, 1963)
Motivation

Methods
Covariance
Gaussian process Y (s) with Matern covariance function is a tapering
Low–rank
stationary solution to the linear fractional stochastic partial approximation
Likelihood
differential equation: approximation
Gaussian Markov
random field
approximation

κ2 d
α2 − ∆ Y (s) = W(s), κ=ν+ ,ν > 0
2
where
I W(s) is a spatial Gaussian white noise
P ∂2
∆ = i ∂s 2 is the Laplacian operator
I
i
I d is the dimension of the spatial domain
 Geostatistics for
An explicit link between GP and GMRF via SPDE Large Data Sets

Whitney Huang
(Lindgren et al. 11)
Motivation

Methods
Covariance
tapering
Low–rank
approximation
I Establish the link between GP with Matérn covariance Likelihood
approximation
function (with ν + d2 are integers) and GMRF Gaussian Markov
random field
approximation
I (Bayesian) inference can be done by using Integrated
nested Laplace approximation (INLA) approach
I The extensions to nonstationary models, models on
manifolds, multivariate models, spatio-temporal models
are relatively easy
 Geostatistics for
An explicit link between GP and GMRF via SPDE Large Data Sets

Whitney Huang
(Lindgren et al. 11)
Motivation

Methods
Covariance
tapering
Low–rank
approximation
I Establish the link between GP with Matérn covariance Likelihood
approximation
function (with ν + d2 are integers) and GMRF Gaussian Markov
random field
approximation
I (Bayesian) inference can be done by using Integrated
nested Laplace approximation (INLA) approach
I The extensions to nonstationary models, models on
manifolds, multivariate models, spatio-temporal models
are relatively easy
 Geostatistics for
An explicit link between GP and GMRF via SPDE Large Data Sets

Whitney Huang
(Lindgren et al. 11)
Motivation

Methods
Covariance
tapering
Low–rank
approximation
I Establish the link between GP with Matérn covariance Likelihood
approximation
function (with ν + d2 are integers) and GMRF Gaussian Markov
random field
approximation
I (Bayesian) inference can be done by using Integrated
nested Laplace approximation (INLA) approach
I The extensions to nonstationary models, models on
manifolds, multivariate models, spatio-temporal models
are relatively easy
 Geostatistics for
Extensions Large Data Sets

Whitney Huang
I non-stationary model on a sphere
Motivation

κ
α2 (s) + ∆ 2 τ (s)Y (s) = W(s), s ∈ S2 Methods
Covariance
tapering
I non-separable anisotropic space-time model Low–rank
approximation
Likelihood
 κ approximation
∂ 2
2 Gaussian Markov
+ (α + m · ∇ − ∇ · H∇ Y (s, t) = W(s, t) random field
approximation
∂t

where (s, t) ∈ S2 × R
 Geostatistics for
For Further Reading I Large Data Sets

Whitney Huang

Appendix

H. Rue, and L. Held For Further

Reading

Gaussian Markov Random Fields: Theory and

Applications.
Chapman & Hall/CRC, 2005.
S. Banerjee, A. E. Gelfand, A. O. Finley, and H. Sang
Gaussian Predictive Process Models for Large Spatial
Data Sets
JRSSB, 70:825–848, 2008.
N. A. C. Cressie, and G. Johannesson
Fixed Rank Kriging for Very Large Spatial Data Sets
JRSSB, 70:209–226, 2008.
 Geostatistics for
For Further Reading II Large Data Sets

Whitney Huang

J. Du, H. Zhang, and V. S. Mandrekar

Appendix
Fixed–Domain Asymptotic Properties of Tapered For Further
Reading
Maximum Likelihood Estimators
The Annals of Statistics, 37:3330–3361, 2009.
R. Furrer, M. G. Genton, and D. W. Nychka
Covariance Tapering for Interpolation of Large Spatial
Datasets
Journal of Computational and Graphical Statistics,
15:502–523, 2006.
C. G. Kaufman, M. J. Schervish, and D. W. Nychka
Covariance Tapering for Likelihood–Based Estimation in
Large Spatial Data Sets
Journal of the American Statistical Association,
103:1545–1555, 2008.
 Geostatistics for
For Further Reading III Large Data Sets

Whitney Huang

Appendix
Lindgren, F., Rue, H., & Lindström, J. For Further
Reading
An explicit link between Gaussian fields and Gaussian
Markov random fields: the stochastic partial differential
equation approach.
JRSSB, 73:423–498
H. Rue, and H. Tjelmeland
Fitting Gaussian Markov Random Fields to Gaussian
Field.
Scandinavian Journal of Statistics, 29:31–49
M. L. Stein, Z. Chi, and L. J. Welty
Approximating Likelihoods for Large Spatial Data Sets
JRSSB, 66:275–296, 2004.
 Geostatistics for
For Further Reading IV Large Data Sets

Whitney Huang

Appendix
For Further
Reading

A. V. Vecchia
Estimation and Model Identification for Continuous
Spatial Processes
JRSSB, 50:297–312, 1988.