Archiving of UERRA data in

MARS and related data services

Richard Mládek, Manuel Fuentes,

Sebastien Villaume, Shahram Najm, Matthew
Manoussakis

UERRA GA Tarragona 28-30.11.2017

 Highlights since the previous GA

 Final list of 9 UERRA datasets

 MESAN dataset split to 2 versions
New type=oi implemented in MARS & ECCODES(former GRIB-API)
 COSMO
Deterministic dataset out of scope
Might be still archived during 2018
 5 of 9 UERRA datasets fully archived
 Grand total: 762,937,000,463,558 (693.887 Tbytes)
 Number of fields: 3,637,248,971
 3 datasets still being archived
 UM datasets - completion expected by January 2018

UERRA GA Tarragona 28-30.11.2017

 UERRA datasets - Progress status

https://software.ecmwf.int/wiki/display/UER/Progress+status
 The former MESAN(SMHI) dataset renamed & split to 2 datasets (V1/V2)
 3 temporary datasets (short period; expver=test in MARS) not needed any more
 MESCAN-SURFEX already archived (less parameters; fc only +6H; 2008)
 UM & UM/En (full data but not the final model version; 2008)

UERRA GA Tarragona 28-30.11.2017

 UERRA datasets (MARS set up)
MARS keys (class=ur)

origin stream type number remarks

Production datasets (expver=prod)
1 COSMO edzw oper an/fc - Out of scope

2 COSMO/En edzw enda an/fc 1..20 Archived

3 HARMONIE/V1 eswi oper an/fc - Archived
4 HARMONIE/V2 eswi enda an/fc 1 Archived; not real eps
5 MESAN /V1 eswi oper oi - Archived; only tcc
6 MESAN /V2 eswi enda oi 1 Xmas 2017; only tcc; not real eps
7 MESCAN-SURFEX lfpw oper an/fc - Xmas 2017
8 MESCAN-SURFEX/En lfpw enda an/fc 1..8 Archived
9 UM/4DVAR egrr oper an/fc - Jan 2018
10 UM/En4DVAR egrr enda an/fc 1..20 Jan 2018

 New type=oi created for MESAN data in MARS (using external non-ECMWF
GRIB exploring tools it will look like type=an)

UERRA GA Tarragona 28-30.11.2017

 MARS achievements

 MARS & ECCODES design for UERRA

 Fully WMO compliant GRIB2 definitions for all UERRA parameters
 Any UERRA data could be archived (some peculiarities because of late
information)
 New level type type=hl (height levels)
 New soil level type type=sol
Allowing varying grid dependant soil level depths
– Not crucial for UERRA at the end but is ready for future demands
 Parameters “since previous post-processing” (varying validity periods)
Surface air maximum temperature since previous post-processing
Surface air minimum temperature since previous post-processing
10 metre wind gust since previous post-processing

UERRA GA Tarragona 28-30.11.2017

 MARS achievements

 New parameters defined for UERRA & approved by WMO

 Percolation
 Downward short-wave radiation flux, clear sky
 Upward short-wave radiation flux, clear sky
 Downward long-wave radiation flux, clear sky
 Soil heat flux (updated version)

 Data services
 Dedicated data portal
Data discovery and small sample retrievals
 Web-API for large retrievals
UERRA data usage statistics will be available

UERRA GA Tarragona 28-30.11.2017

 UERRA web site at ECMWF
 Static pages
 Data portal (not fully updated for datasets with ongoing archiving)
 Issues with data (identified problems)
 UERRA retrieval efficiency (important)
 Support
Contacts
– providers’ contacts will be needed for post UERRA as well
– Prefereably dedicated mailing list like uerra.support@...
FAQ, Forum..
 Description of project, parameters, datasets..
The latest final models’ description required
 Resources (info from development phase; info about data
interpolation/visualization etc)

UERRA GA Tarragona 28-30.11.2017

 UERRA web site at ECMWF
 Tracking actual state
 Progress status (with links to the model specific log pages)
 Parameter availability (with links to MARS Catalogue for each dataset)
 Data production and archiving schedule (overview of completed and still
expected periods)

UERRA GA Tarragona 28-30.11.2017

 UERRA web pages at ECMWF

UERRA GA Tarragona 28-30.11.2017

 UERRA data portals

1. Public access
 Self registration still required before accessing the dedicated UERRA
data portal

2. Users with ECMWF account

 Full access including MARS catalogue

UERRA GA Tarragona 28-30.11.2017

 UERRA data portals

 Public access

UERRA GA Tarragona 28-30.11.2017

 UERRA data portals

 Users with ECMWF account

 Full access to MARS catalogue
 Direct links to MARS catalogue for each UERRA dataset from the page
Parameters availability

UERRA GA Tarragona 28-30.11.2017

 Model specific features

 Model levels
 COSMO: 1..40
 HARMONIE: 1..65
 UM: 1..63
 Soil levels
 COSMO: 8 soil layers
level borders at 0, 0.01, 0.03, 0.09, 0.27, 0.81, 2.43, 7.29, 21.87 m
 HARMONIE: 3 soil levels/layers (sot on levels, vsw on layers)
level depths are grid dependent
 MESCAN-SURFEX: 14 soil layers (only 6 layers for preliminary shorter re-
analysis runs)
level borders at 0, 0.01, 0.04, 0.1, 0.2, 0.4, 0.6 m 
 UM: 4 soil layers
level borders at  0, 0.1, 0.35, 1 and 3 m

UERRA GA Tarragona 28-30.11.2017

 Model specific features
​Surface level parameters output frequencies
https://software.ecmwf.int/wiki/display/UER/Description
Analysis:
 UM, HARMONIE, MESCAN-SURFEX
 analysis output in six hourly intervals (at 00/06/12/18UTC)
 MESCAN-SURFEX
 precipitation analysis (24 h accumulation from 06 UTC to 06 UTC of next day) output at 06TC
 COSMO, MESAN
 hourly analysis outputs for 1..23UTC runs
Forecast:
 UM, COSMO
 forecast output at T+1,2,3,4,5,6,9,12,15,18,21,24,27,30 started at 00 UTC and 12 UTC
 forecast output at T+1,2,3,4,5,6 started at 06 UTC and 18 UTC
 HARMONIE (started at 00/06/12/18UTC):
 sd, sr, al, rsn, slhf, sshf parameters (derived from ALADIN model driving SURFEX) at T+1,2,3,4,5,6 only
 all other parameters at T+1,2,3,4,5,6,9,12,15,18,21,24,27,30
 MESCAN-SURFEX (started at 00/06/12/18UTC):
 parameters sp,tp,2r,2t,10si,10wdir, tidirswrf from forecast output at T+6 only (fields from ALADIN
model driving SURFEX)
 all other parameters from forecast output at T+1,2,3,4,5,6
UERRA GA Tarragona 28-30.11.2017
 Model specific features

 COSMO
 Model levels above 100 hPa not available
 Step 0 not available in forecasts ("fc") because of nudging
 MESCAN-SURFEX
 Contains analysis of total precipitation accumulated between 6H of the
previous day and 6H of the day encoded in the GRIB2
 7 parameters sp, tp, 2r, 2t, 10si, 10wdir, tidirswrf from forecast output
archived at T+6 only (fields from ALADIN model driving SURFEX)
 HARMONIE
 Steps only up to +6H available for parameters derived from ALADIN model
driving SURFEX
 UM
 Model level increases with height MO (opposite to other models)

UERRA GA Tarragona 28-30.11.2017

 HIRLAM datasets

 HARMONIE/V1
 Full 55 years archived (1961-2015)
 HARMONIE/V2
 Full 5 years period 2006-2010 archived
 Archived under stream=enda and number=1 but it is not ensemble
 MESAN/V1
 Full 6 year period 2006-2010 archived
 New type=oi used in MARS due to technical reasons
 MESAN/V2
 Expected by Xmas 2017; period 1993-2006; the work in progress (new
satellite data etc)
 Will be archived under stream=enda and number=1 but it is not ensemble

UERRA GA Tarragona 28-30.11.2017

 HIRLAM datasets

 2 identified issues
 Wrong 0% values in relative humidity, cloud cover and albedo fields
There are some isolated spots of one or a few grid points with wrong
zero values in the fields of relative humidity (both at 2m and on
pressure levels). It happened because of a wrong algorithm used for
re-computation of the values from 0-1 interval to 0-100% (the
supersaturation was set to 100% instead of 0% value).
The same issue can be probably found in cloud cover and albedo fields
as well but its extent there is not known.
 Missing values in surface sensible heat flux and surface latent heat flux
fields
The sensible heat flux and surface latent heat flux fields appear to
have missing values over the land and because of that should not be
used.

UERRA GA Tarragona 28-30.11.2017

 COSMO datasets

 COSMO/En
 Full 5 years period 2006-2010 archived

 COSMO
 Out of scope
 Still might be archived during 2018

UERRA GA Tarragona 28-30.11.2017

 Met Office datasets

 UM/4DVAR and UM/En4DVAR datasets

 Simultaneous archiving (2 streams)
 Completed periods:
1979-1989
2000-2011
 In progress:
1990-1999
2012-2016

UERRA GA Tarragona 28-30.11.2017

 Météo-France datasets

 MESCAN-SURFEX/En
 Full 5 years period 2006-2010 archived
Re-archived from test to prod mode in MARS
– Done quickly thanks to new MARS consolidation suite (@Sebastien
Villaume) otherwise it would take many weeks(months)

 MESCAN-SURFEX/En
 40 years period 1961-2000 archived
 2001-2015 in progress (expected finish before Xmas 2017)

UERRA GA Tarragona 28-30.11.2017

 ECMWF links for UERRA

 UERRA at ECMWF (high level page): http://www.ecmwf.int/en/research/projects/uerra

 UERRA at ECMWF: https://software.ecmwf.int/wiki/display/UER
 UERRA data portal: http://apps.ecmwf.int/datasets/data/uerra
 UERRA retrieval efficiency :
https://software.ecmwf.int/wiki/display/UER/UERRA+retrieval+efficiency
 Issues with data : https://software.ecmwf.int/wiki/display/UER/Issues+with+data
 Parameter list: https://software.ecmwf.int/wiki/display/UER/Parameters
 Parameter availability:
https://software.ecmwf.int/wiki/display/UER/Parameter+availability
 Expected archiving schedule:
https://software.ecmwf.int/wiki/display/UER/Data+production+and+archiving+schedule
 Data interpolation and visualization:
https://software.ecmwf.int/wiki/display/UER/Data+interpolation+and+visualization

UERRA GA Tarragona 28-30.11.2017