9.

Practical use cases of SDMX:

Census Hub

Nadezhda Vlahova
Eurostat
Unit B3: “IT for statistical production”

SDMX Basic course, 2017

 The European Census Hub: key issues
 Dissemination of the data from the 2011 & 2021 population
and housing censuses in the European Union

 Data that are methodologically comparable and structured

according to “hypercubes” agreed with Member States
(Census Regulation)

 Providing users with an easy access to detailed census data

and metadata (advanced functionalities)

 Management of massive amounts of data produced and

controlled by Member States
 Maximum flexibility to cross-tabulate data from different
sources
 What we need?
 Single point to access and retrieve detailed census
data, numerical and textual metadata
 Environment for dissemination and comparison of
massive amounts of harmonised data which is easy to
use and reuse
 Countries Census data and metadata should be made
available until 2035

Do not need
 No validation and aggregation on the fly are required,
neither supported
 The Hub concept
 The Hub is based on the concept of data sharing, where a
group of partners agree on providing access to their data
according to standard processes, formats and
technologies.

 The SDMX Hub approach offers several advantages:

• decoupling of NSIs' systems from the central hub via
standard formats and techniques for the exchange;
• Limited investment, re-usability (with the advantage
of using recognized international standards).
• Software (SDMX-RI) is supplied by Eurostat.
 Hub approach – PULL data for collection and
dissemination
register SDMX Registry

NSI query

Hub Dissemination
P
Received
Eurostat Pull
U Loader Eurobase
Requestor data in
L
SDMX-ML
L Dissemination

Verification /
eDAMIS
P Conversion XSL for
U To SDMX SDMX-ML
S Warehouse
Intermediate
H storage
Data Input storage
 What is the Census Hub ?
 Interoperability/Initiative (32 countries participating in the
project)
 Business driven project

Delivered:

 Information system to query and display SDMX formatted

data (e.g. Census 2011/2021 data) retrieved from
different data providers – Census Hub
 Universal framework for exposure and translation in
SDMX-ML format of data stored in a legacy dissemination
database – SDMX-Reference Infrastructure (SDMX-RI)
Data preparation

Macrodata
database Aggregated
dataset n

Non-SDMX local
dissemination data

Aggregated
dataset 5

Aggregated dataset 1
Aggregated dataset 2 Aggregated
Aggregated dataset 3 dataset 4
Example: DSD for Table 6 (Marital Status)
ID
Dimensions
CONCEPT CODELIST ID
Attributes
ATTACHMENT CODELIST
LEVEL
TIME Time period or range CL_TIME
OBS_STATUS Observation CL_OBS_STATUS
GEO Geographical area CL_GEO
OBS_LEVEL Observation CL_OBS_LEVEL
SEX Sex CL_SEX
OBS_NOTE Observation
FST Family status CL_FST
HC_NOTE Series
LMS Legal marital status CL_LMS

CAS Current activity status CL_CAS

POB Country/place of birth CL_POB

Measures
COC Country of citizenship CL_COC
ID NAME
AGE Age CL_AGE
OBS_VALUE Observation value
FREQ Frequency CL_FREQ
Census datasets (hypercubes)

Hypercube HC06
GEO.L. SEX. FST.H. LMS. CAS.L. POB.M. COC.M. AGE.M.

(57) (3) (17) (13) (6) (15) (13) (28)

Theoretical
number = 57 * 3 * 17 * 13 * 6 * 15 * 13 * 28 = …
of cells
= 1,238,033,160 cells
(NB: for one country and one table…)
Census datasets (hypercubes)
Hypercube HC04
GEO.L. SEX. HST.H. LOC. CAS.L. POB.L. COC.L. AGE.M.

Using the naming conventions of Eurobase

its name would be:

Population in NUTS2 regions by sex, five-years age

groups, household status, size of the locality and broad
groups of current activity status, place of birth and country
of citizenship
 we need to change the way in which we give
access to data
SDMX-RI architecture overview
SDMX-RI – User Interfaces
Data providing Data collecting
organisation organisation

Web/Test Client
Mapping
Assistant
Internal
network
SDMX-RI – “Under the hood”
Non-SDMX SDMX-formatted
local data data
 SDMX exchange and supporting tools
NSI
Process workflow
Dissemina
SDMX Extract Transform
SDMX file tion/Trans
codes files file mission

SDMX Converter

NSI NSI SDMX

EDAMIS
software software Converter Processing
for sending

Non-SDMX SDMX-RI
local data EDAMIS
Mapping Test client Processing
for sending
Assistant
NSI client
NSI Web service HUB

NSI development
EDAMIS
NSI software
Eurostat tools NSI developed software
 National Statistical Institute

How the Hub works

National Statistical Institute

Eurostat Census
Hub
https://ec.europa.eu/CensusHub2/
 Census 2021 Project Roadmap
Input Census 2021
Legal base Initiation 2017 Planning 2018- Execution 2019- March 2021
2019 2021

•Define the •Elaborate the •Implementation •Final version of

Statistical WG scope scope of the project the software
•Collect •Create work plan ready for
requirements packages •Produce the dissemination
•Define the •Design project •Data ready to
Social statistics WG deliverables be
outcome solutions
disseminated

Census IT TF
Monitor & Control

End product: Principles:

 Better use and presentation of social  Reuse solutions, experiences and set up
statistics data;  Utilise the investment from 2011 Census
 Modular and reusable;  Reduce burden for data providers
 Compare Census runs data;  Reduce cost
 Design post 2021?
15
 Census 2021 IT Task Force
 Composed by Census 2021 data providers
 Focused on technical matters (IT experts)
 Support the implementation of the Project Roadmap
 Has own programme and deliverables

2017 2018 2019 2020 2021 2024

Initiation Planning & Execution March 2021 2024

Preparation

•Define the •Prepare •Software •National set up •Data from all

scope and work guidelines installation prepared data providers
programme •Elaborate •Training •Countries can are available
•Elaborate on requirements •Testing provide data
requirements •Share
•Define use experience
cases
•Analyse the
changes of the 16
regulation
 Data Hub is:
 System of DSDs built in SDMX 2.0 and in use for 32
countries of the ESS
 Data dissemination portal based on SDMX data model
 communicating with data providers via SDMX Web
Service
 No data processing (no editing or aggregation)
 Additional reusable modules for
 Configuration management
 Tool for handling SDMX structural metadata
 Innovative user interface -> extract data starting from a
statistical concept
 MSs status: 32 up and running
 A data user can…
 Browse the Hub to define a dataset of interest, navigating
via structural metadata:
 Search by topic (filters) and select data (level of
detail, breakdowns)
 Select layout (axes)
 View a table

 Export a file (CSV, Excel, SDMX-ML)

 User registration
 Registered users can:
 Save, retrieve, modify or delete stored queries
 Receive an e-mail notification when offline queries are
executed
 Reusability
 Installed within and outside ESS
 Used for multiple domains
 Used for dissemination and reporting
 Supports data sharing, push and pull modes
 Generic and SDMX based (2.0 and 2.1)
 Extensible and modular approach
 Free, open source solution, maintained by Eurostat
 Support different platforms and DB vendors
For more information

ESTAT-CENSUSHUB@ec.europa.eu

https://webgate.ec.europa.eu/fpfis/mwikis/sdmx

https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/Census_
Hub
Thank you for your attention

21
DEMO
DEMO
 EU Census: Implementing measures
 Regulation (EC) 763/2008 on population and housing
censuses authorises the European Commission to adopt
implementing measures on:

 technical specifications of the topics and their

breakdown (Regulation (EC) 1201/2009)

 programme of the statistical data and metadata to

be transmitted to Eurostat (Regulation (EU)
519/2010)

 quality reporting and technical format of data

transmission (Regulation (EU) 1151/2010)