Ufuk Turuncoglu

(1,2)

(1) Informatics Institute,

Computational Science and Engineering,
ITU, Turkey
(2) ESP Section,
ICTP, Italy

Developer School for HPC applications in Earth Sciences,

Overview of Model Coupling in

Earth System Sciences and
Recent Developments

Outline
Earth System Modeling and Evolution
Model Coupling
Techniques and Tools

Basic Definitions and Design Strategies

Interpolation
Land-sea Mask Issues
Automatization via WF

RegESM

Components
Design
Run Sequence
Concurrent vs. Sequential
Project Home and Documentation

Earth System

http://paos.colo
rado.edu

Combination of different components and

each one of them interact with both positive
and negative feedback mechanisms

MetOffice

Earth System
Time Scales

Complex
interactio
n and
feedback
mechanis
ms
among
componen

IPCC Report

Time
scales and
response
times are
different !

Climate
Modeling
NWF by
Richardson

Timeline

Automatized NWF
by Von Neumann

Rapid development
after 80s
The complexity of
the models are
getting increase
by Steve Easterbrook

First ESM

http://prezi.com/pakaaiek3nol/timeline-of-

Coordinated Experiments
CMIPs

High Performance Computing

Timeline
There is a close
relationship
between
development in
HPC and climate
modeling
The development
curve (or slope) is
more steep after
80s!

Complexity of the Climate Models

Growth of Climate Modeling

by

main breaking point

Model
complexity is
increasing: in
terms of
Physics
Components

Along with the

development in HPCs
Increased Horizontal
and Vertical
Resolution

Climate Assessments

More processes
(i.e. chemistry, interactive
vegetation etc.) are
represented by
models
In higher temporal and
spatial scale

IPCC

Evolution of Models

Type of Models
Simple energy balance models very sophisticates
ESMs

Atmosphere-Ocean General Circulation Models (AOGCMs)

Earth System Models (ESMs)

AOGCM + Biogeochemical Cycles (Carbon, Sulphur,
Ozone)
Earth System Models in Intermediate Complexity (EMICs)
Idealized or lower resolution.
Applied to answer specific questions.
Needs less computational power

Regional Climate Models (RCMs)

Limited area models to downscale results of the lower
resolution models. It might also have multiple
components!

http://www.climatechange2013.org/images/report/WG1AR

Climate Assessments

Climate Assessments

No decrease in uncertainty!
but models are more accurate in 20C

http://www.scientificamerican.com/article/climateassessments-25-years-of-the-ipcc/

Evolution of IPCC Reports

Evolution of Published Papers

Keyword: Coupled climate model (in Topic)

Coupled Modeling

Source: Web of Science

Keyword: Earth System Model (in Topic)

What is Model Coupling?

It is a sophisticated way of simulate climate
system and
complex interactions between its
components.
The individual components are not perfect !
Our theoretical understanding of climate is
still incomplete, and certain simplifying
assumptions are unavoidable when building
these models (Reichler and Kim, 2008) !
Coupled model does not always gives best
results but help to understand interactions
and processes between components !

Example from Atmosphere + Ocean Coupling

1. Initial state of the ocean is not precisely known

Lack of continuous and high resolution observations.
Remote sensing might help but covers only ocean
surface and
recent decades. The quality also depends on many
issues (atmospheric correction and used algorithm,
coastal regions etc.)
2. An imbalance in the surface flux (heat, momentum
and freshwater) much smaller than the observational
accuracy is enough to cause a drifting of coupled GCM
simulations into unrealistic states
It is not easy tightly couple models that are written by
different groups
High accuracy of conservation is needed!

http://www.ipcc.ch/publications_and_data/ar4/wg1/e
n/ch1s1-5-3.html

Difficulties in Model Coupling?

Model Coupling
But still, common trend is to create multicomponent and
multi-scale model applications and
simulations
Land Surface +
Hydrology
It needs close collaboration between
Chemistry
different
Earth
Oceanography
System disciplines and
IS-ENES2,
research groups
Science
Meteorology
METAPHOR+CIM
CESS: Computational
Computer
Earth
Curator
Science
CMIPs
(CMIP3/5)
System
Science
Science &
Engineerin
g

ESGF

Designing efficient and scalable models

Handling high volume of data (data
compression, parallel I/O)
Automatization of model handling,
reproducibility and metadata/provenance

Types of Model Coupling

There are two types of model coupling
1. Offline
Models run sequentially and the interaction
between them is in only one way. The
simplest example is dynamical downscaling
(GCM to RCM) or one-way nesting (RCM to
RCM)
2. Online
In this case, models interact in both way and
feedback mechanisms among components
represented (two way nesting, fully coupled
atmosphere-ocean models or ESMs.)

The ESM components might be coupled using

different programming approaches
1. Merge individual model codes
host / parent
program atm

! read initial and

boundary data
call read()
! run model
call run()
! write results to file

end program atm

program atm

! read initial and

boundary data
call read()
! run model
call run()
! run ocn model
call ocn(exchange
vars)
! write results to file

end program atm

ATM

program ocn

! read initial and

boundary data
call read()
! run model
call run()
! write results to file

end program ocn

subroutine ocn
(exchange vars)

! read initial and

boundary data
call read()
! run model
call run()
! write results to file

end subroutine ocn

OCN

easy to
code,
notportable
flexible
and
generic

single
executable!

http://www.cerfacs.fr/~coquart/pagecerfacs/seminars/201405_OASIS3

Techniques for Model Coupling

more information in Valcke, 2009 EGU

Techniques for Model Coupling

The ESM components might be coupled using
different programming approaches
2. Use existing communication protocols: MPI,
InterComm

host / parent

existing
code and
concurrent
components
not easy to

program atm

! read initial and

boundary data
call read()
! run model
call run()
! write results to file

end program atm

program ocn

! read initial and

boundary data
call read()
! run model
call run()
! write results to file

end program ocn

program atm

! receive exchange
field
call recv()
! run model
call run()
! send exchange field
call send()
! write results to file

end program atm

program ocn

! receive exchange
field
executable!
call recv()
! run model
ATM
OCN
call run()
! send exchange field
call send()
! write results to file

more information in Valcke, 2009 EGU

end program ocn

ATM

OCN

implement,
not flexible
and
generic
multiple

Techniques for Model Coupling

The ESM components might be coupled using
different programming approaches
3. Use coupling framework & libraries: FMS, ESMF,
MCT
. Code divided into units and calling interface
redesigned
(i.e. initialization, run and finalize)
. Hierarchical model
are supported
structures
flexible, efficient,
portable,
support to use generic
utilities for interpolation, time
management and unit
conversions), support both
sequential and concurrent
Existing codes must be
components
restructured

more information in Valcke, 2009 EGU

Techniques for Model Coupling

The ESM components might be coupled using
different programming approaches
4. Use a coupler: OASIS, OASIS-MCT, MPCCI, PALM

.
.

Coupler link the model components and handle

coupling processes such as interpolation,
synchronization etc.
Components models might
be separate
single
flexible,
efficient, or
portable,
support toexecutable
use generic
utilities for interpolation, time
management and unit
conversions),
support
multi-executable:
concurrent
components
more difficult
to debug;

harder to manage for the

OS and possible waste of
more information
Valcke, 2009 EGU
resource
if seq. in
execution

What is driver / coupler ?

The driver can be defined as a glue
It merges the different components of the modeling
system
In general, the modeling system might be designed
to have
Single executable
Multiple executable (each component has an
executable)

In the multiple executable case, the usage of the

modeling system might be difficult
The driver is generally responsible for
Interaction among the components:
exchange boundary data between
the components, regridding etc.
Synchronization:
to coordinate the time evolution
of the physical models (coupling time step)

Coupled Model Design

http://climatesight.files.wordpress.com/20

Global Models (GCMs and ESMs)

There is no single design and ultimate
solution!

Kaitlin Alexander and Steve Easterbrook

Coupled Model Design

Examples for Regional Models

http://www.smhi.se/polopoly_fs/1.28917!/RO_47.
pdf

MORCE, IPSL

RCA4-NEMO,
SMHI
REMO-RCM,
Max Planck
http://www.remo-rcm.de/Atmosphere-Ocean-

Data exchange and interpolation is

one of the most important
component of the model design

Common Problems in ES Modeling

The models are developed by different groups /
organizations
Various types of grids (curvilinear, regular, triangular etc.)
Written in various programming languages (Fortran, C, C+
+)
Different I/O strategies (ASCII, binary and netCDF)
Different configuration mechanisms (simple ASCII files or
well structured XML files - CESM)
Each component (ATM, OCN, etc.) has its own learning curve
Generally, does not follow common programming standards
(CF conventions, GridSpec etc.)

Requirements:
Need to know the way of real programming
Need close collaboration between scientists and
programmers
The standardization is very crucial. We need to think more
generic solutions: standardized codes = better

interoperability

Common Problems in Coupled

Model Design

Need more attention

Data exchange and interpolation between model

grids
Support for different types of grids
Conservation of flux fields (high order conservative
regridding?)
Extrapolation might be needed for unmatched land-sea
masks.
Exchange grid might help!

Self describing models

Repeatability = Reproducibility
Need to document a simulation or run
It is important for reproducibility
Both metadata and provenance Re-run Re-run Confirm
information needed for trackingthe code and get results
and get
using a
Is it possible to collect automatically?
same
different
different
Synchronization of multi-component
results
results
method

models

Tools like OASIS and ESMF help!

adapted from Steve Easterbrook

Interpolation / Regridding
The problem arises when numerical grid of
the model components are not same (i.e.
atmosphere and ocean) ?
There are two main types:
Conservative
Non-conservative

https://wiki.cc.gatech.edu/CW2013/images/8/89/201302_OASIS3MCT_CW2013.pdf

Interpolation / Regridding
Conservative type regridding is used to
preserve flux fields (i.e. moisture and heat
fluxes)
It can be implemented to conserve field in
Local: integral of the local patches are equal
Global: integral of the source and destination
grid is equal

The current tools uses 1st order

conservative regridding
The resource field might have artifacts
if the ration between source and
destination grid resolution
is high
N
1 in 50 km
i
i
(i.e. atmosphere Q
model
but

Q
w
o
an n
ocean model 10 km)
A

o n1

http://www.earthsystemmodeling.org/presentations/pres_1002_siam_ryan.pdf

Interpolation / Regridding
In general, coupled models follows SCRIP
convention
Steps involved regridding

The weight generation might be

online (via subroutine calls): ESMF
offline (external applications such as,
ESMF_RegridWeightGen
and SCRIP): OASIS, MCT, ESMF

http://oceans11.lanl.gov/trac/SCRIP

Generate interpolation weights

Apply weights to interpolate field using a sparse
matrix multiply

Interpolation / Regridding
SCRIP vs. ESMF_RegridWeightGen
SCRIP does calculations in (lat,lon) space but
ESMF does it in (x,y,z) space
Different area calculation for conservative
remapping
SCRIP is serial but ESMF provide also parallel
ESMF Interpolation
execution
is currently used by
NCL and ESMPy

Peggy Li, 2010, ESMFvsSCRIP-v2.doc

http://pantar.cerfacs.fr/globc/publication/technicalreport/2012/SCRIP_ESMF

Unmatched Land-sea Masks

Different model components with different land-sea
mask
OCN
ATM Mismatch
Play with land-sea mask of each
components to match them
It is hard and very time consuming
Must be repeated for each
individual applications. It is not
a generic solution

Use exchange grid (needs conservative type

regridding)
orphan grid cell
assigned to land
(arbitrarily)
clipped

Two step interpolation Interpolation + Extrapolation

mapped

Two step interpolation

(i.e. interpolation over ocean)
1. Interpolate from ATM to
OCN
using bilinear
Use
2. interpolation.
Use result of previous
only
step,sea grid points
interpolate data from OCN
to OCN
from mapped grid points
unmapped
using
3. to
Merge
results ones
of 1 and
nearest-neighbor
type
2 to
regridding
create filled field

Still has problem in

some applications
(sharp gradient in
some cases) but used

STEP 2

STEP 3

s to ESMF Group (especially to Bob Oehmke) for their support and help

Unmatched Land-sea STEP

Masks
1

unmappe
d

Automatization
The models are getting more complex (lots
of configuration file and parameters, huge
output) and it is not easy to run and keep
track of simulations. It is very time
consuming.
Get
Proxy Info

Set workflow
parameters

WORKFLOW - I

Stop

NO

Globus Job (Fork)

create - modify build

Job
OK

Globus Job (Fork)

unzip source

Copy files
GridFTP
Upload

Job
OK

YES

Globus Job
(Batch)
run

NO

YES
Transfer
OK

YES

Stop

NO

Stop

Create
Job Description
RSL*

Create
CCSM
commands

Create
Job Description
RSL*

Create
CCSM
run command

Create
Job Description
RSL*

Get
Proxy Info

Stop

Trigger
other
workflow
via
standard
e-mail

Send case directory information

WORKFLOW - II
Visualization
NCL

Process
Output, CDO

Stop

XML
NO
Merge
Provenance
Info
* RSL, Resource Specification Language

YES

Job
OK

YES
Copy files
GridFTP
Download

NO
Job
Finish

Conceptual
CCSM / CESM
Workflow

 Scientific workflow tools might help to

simplify the regular tasks!
Teragrid:
build model

collect
provenance
and metadata
info

creates WS-GRAM XML

job description files

run model

runcoglu, CG: 2011, EMS: 2013 and ESM Book (Vol. 5): 2012

Automatization

 Scientific workflow tools might help to

simplify the regular tasks!
Modified workflow for conventional
build model
run
model
cluster

collect
provenance and
metadata info
workflow wide global
parameters

runcoglu, CG: 2011, EMS: 2013 and ESM Book (Vol. 5): 2012

Automatization

 RegCM
Regional Climate Model (hydrostatic)
Sophisticated sub-grid parameterization
Coupled with Atm. Chemistry, 1D Lake and Slab
Ocean Models

AGCM (Speedy) Intermediate Complexity

Model
Simplified GCM (Molteni and Kucharski)
Spectral Model (hydrostatic)
Slab models for land and sea-ice

ICTP-NEMO Coupled Model

Speedy + NEMO + OASIS3
T30/L8 and 2 deg (+ equatorial refinement 1/3
deg)
Supports one and two way coupling

http://indico.ictp.it/event/a10154/session/6/contribution/4/material/0/
0.pdf

Modeling Experiences in ICTP

New regional modeling system

(RegESM)?

To design easy to use and extend modeling system

ENEAs PROTHEUS system uses
MITgcm+RegCM3+OASIS
It is not easy to use, extend and upgrade (It uses
multiple executable, no online interpolation support,
hard to add new component such as wave)

To support Med-CORDEX
Set of coordinated experiments to better understand
Mediterranean climate: standalone and coupled RCMs
It would be good to have another regional coupled
modeling system that uses different model
components

To gain experience about design and use of coupled

modeling systems
The applications of coupled modeling system getting
increased in last decade along with the rapid
development in HPC

Evolution of RegESM
The first prototype version is created in 2012

There were no driver

RegCM was host the whole coupled modeling system
Single ocean component was supported (ROMS)
Poor energy conservation (for exchange fields)
Applied to Caspian Sea (Turuncoglu et al., 2013, GMD)

Then, more generic version is designed and released

in 2013
Centralized driver using ESMFs NUOPC layer (via
connectors)
All the components plugged into driver (less
dependence to the model component code)
Support for multiple ocean model (ROMS and MITgcm)
Energy conservation is improved (global conservation)
Support to extrapolation (land-sea mask mismatch)
Applications: Mediterranean, Black Sea and Caspian Sea

RegESM Design & Components

Currently, RegESM supports three different
component:
ATM:
RegCM
Two different
land surface
model

atmospheric
chemistry

OCN:
ROMS /
MITgcm
RTM:
HD (special
thanks to
Prof. Stefan
Hagemann)
DRIVER:

Driver Design in ESMF/NUOPC

The driver defines;

The model components (i.e. RegCM, ROMS) grid,
fields, states etc.
Coupler component (connector)
Global clock and calendar

and synchronizes the sequence of actions

https://www.earthsystemcog.org/projects/nuopc/

The current driver uses only connector

component (NUOPC)

Driver Designs in ESMF/NUOPC

Simple Coupling
with Connectors
ATM+OCN

Coupling through a
Mediator

Connectors

The mediator approach is much more efficient

when the number of components increase

https://www.earthsystemcog.org/projects/nuopc/

Different designs are possible with NUOPC

layer

Run Sequences in ESMF/NUOPC

The NUOPC Layer is capable of
implementing many different coupling
schemes
Explicit:
Exchange data in same time. Two
connector (i.e. backward and
forward)
Explicit with slow and fast
clock support:
Different interaction time
step among the components
Semi-implicit (leap-frog style
interaction)
Fully-implicit

Run Sequence in RegESM

The RegESM might use both explicit and
semi-implicit coupling schemes along with
the support of fast and slow time steps.

Fast interaction between ATM and OCN

Slow interaction between ATM and RTM, RTMOCN
Example for
3 component and explicit coupling:
Here, RTM is optional component

Sequential vs. Concurrent Execution

The modeling system supports two different
approach to run the model components.

In sequential mode: model components are

run in order
In concurrent mode: all models are active at
same time (it does not allow overlapping of
the used PETs cores / CPUs)

Repository and Project Home

The RegESM project is hosted by GitHub
https://github.com/uturuncoglu/RegESM

The project home page

Implemented by using CoG1

http://cess.be.itu.edu.tr/projects/rege

I) https://www.earthsystemcog.org/projects/cog/

Documentation
Initial version of user guide is ready
Basic model design
Installation of libraries, components
and RegESM model
Definition of configuration files
(exchange fields and model)
Known bugs and limitations

Send any comments by e-mail

Experience about installation in
different computing system
Bugs
Unfortunately, there is no support
ticket handling system yet!

Results of your applications

(tested domains, detail about configuration etc.)

Questions!
Contact:
u.utku.turuncoglu@be.itu.edu.tr

Dont miss Lab-Session related with

ESMF at 16:30 !!!