HYMEP II Activity #3

Proposal for the National

Hydrological Data Provision System
Takeshi Matsumura
HYMEP2 Project JICA Expert
International Meteorological Consultant Inc.
Workflow using the Data Provisioning
System
Users

6. Download the 1. Select the data from the catalog

generated data and request

Manager /
Data Provisioning 2. Approve the request Sensitive Data Release
System Committee

(Automation of steps 3 to 5)
5. Upload the
generated data 3. Receive the request

4. Produce the requested data Hydro-met Database

GWRD Staff
(AQUARIUS TS)
Collaboration with the CSIRO Project
 1. Browse and search the data catalog

User Experience
• ECMWF Copernicus CDS

3. Check the request status and download the data 2. Specify the data options and submit the request
System Overview Shared with ANWIS

DPS Data Processing Workstation

(on AWS Cloud Service) (in NWARA Office)

User

1. Browse Data Catalog Data Catalog

Web browser
(GeoNode) 0. Provision Data Catalog

Web browser 4. Approve Data

2. Request Data
Request
3. Notify the Data Request
Data Request Manager
8. Notify Request Completion
7. Update Data Request Status Staff

Web browser / 5. Manual Data

Aquarius Data Generation
Extractor
9. View & Download the data Data Storage / 6. Upload Data
Web browser /
Visualization
GIS Software
(GeoNode)
5. Automatic Data
Generation
Deployment Data Provisioning
System (on Cloud)
Internet

Fiber

WGDR NWARA
ONU

Workstation #1 Workstation #2 Workstation #5 Router / Firewall

(Dell Optiplex) (Dell Optiplex) (Dell Optiplex)

Switch (Cisco SF100D-16) Switch

AQUARIUS TS Data Processing

(Dell PE R710) NWARA
Workstation
LAN
Sub-systems
Data Catalog
• Admin users can
• Create data catalog via Web UI.
• Update the metadata of data catalog via Web UI.
• Publish the created data catalog via Web UI.
• With approval. TBD How to approve it.

• Users can
• Browse the data catalog via Web UI.
• Choose a data set in the data catalog and request the data set via Web UI.
• TBD: Auth related functions
Data Storage & Visualization
• It corresponds to the following functions in the ANWIS architecture:
• Data uploading, downloading, subsetting, mapping, visualization
• Admin users can upload geo and non-geo data via Web UI.
• Aquarius Data Extractor can upload geo and non-geo data via Rest API.
• Users can
• Download the uploaded data via Web UI.
• Use the visualization functions of the Web UI.
• Download and visualize the data by using GIS software.

• TBD: Auth related functions

Data Request Manager
• Web UI for users to
• Specify parameters of the data set in the request (e.g. Period, Station Name)
• View the Data Requests created by the user
• Subscribe for the user's Data Request to get notified upon status changes
• Web UI for Admin Users to
• View the Data Requests created by all users
• Update and Delete a Data Request
• Subscribe for arrival of a new Data Request
• Subscribe for a Data Request to get notified upon status changes.
• REST API for the Aquarius Data Extractor to
• Update a Data Request
• Subscribe for arrival of a new Data Request
• TBD: Auth related functions
Interworking
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Aquarius Data Aquarius
(Browser) Visualization Manager Authorization Extractor Database

UC: Admin User Auth

UC: Update Data Catalog

UC: Machine Auth

UC Overview

UC: User Auth

UC: Browse Data Catalog

UC: Request Data

UC: Process Data Request

UC: Visualize and Download Data

 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization Database

UC: Admin Auth

UC: Update Data Catalog (Manually)

Browse the data catalog

List of data sets

Create a Data Set (e.g. Layer) to the Catalog

Private Data Set is

created

Update meta-data of the Data Set

Publish the Data Set

The Data Set is

visible to anyone
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization Database

Browse the data catalog

List of data sets

UC: Browse Data Catalog

User selects a data

set in the list.

Get the data set details

Data set details

User reads the data set

details to decide if she
requests the data set.
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization Database

UC: Browse Data Catalog

Click "Download
Request" Button
in the Data Set
Details Page

Get the Data Request Form for the selected Data Set
UC: Request Data

Data Request Form

Fill in the Data Request

Form and Click the
Submit button

Submit the Data Request Form

Store the request

in a DB

Display "Accepted"
request status
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization (Browser) Database

UC: Request Data

In this flow the data source can be
anything. It doesn't necessarily be
UC: Process Data Request (Manual)

the Aquarius

Notify new Data Request

Manually generate the requested data

Generated Data
Upload the generated data (GeoNode Web UI)

Page for the generated data

Update Data Request Status (w/ Fetched GeoNode URL via a "Share" button.)

Update the request

status in the DB

Notify the Task Completion (w/ GeoNode URL) In this flow the data type can be
anything (e.g. MS Office Document)

Display "Completed"
request status
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization Database

UC: Request Data

UC: Process Data Request (Automatic)

Request Data
Notify new Data Request Generation (Aquarius
REST API)

Generated Data
Upload the generated data (GeoNode REST API)

URL to access the page for the uploaded data (i.e. Layer)

Update Data Request Status (Returned GeoNode URL)

Update the
request in a DB
Notify the Task Completion (w/ GeoNode URL)

Display "Completed"
request status This UC will be implemented in the
2nd phase.
 GeoNode

User Data Catalog Data Storage & Data Request Authentication & Admin User Custom Script Aquarius
(Browser) Visualization Manager Authorization Database

UC: Handle Data Request

Access the Geo Node URL for the generated data

UC: Visualize and Download Data

Render the page for

visualizing the data

User changes a
parameter
GeoNode does not visualize time
series data (e.g. daily precipitation).
To be studied.
Fetch the filtered data according to the set parameter

Render the new data

Request to download the data

GeoNode's function
Data Request Manager Details
Data Request Manager Model
• id: Generated unique id assigned to each data request.
• create_time: Date time when the data request was created.
• requester_id: Requester user id.
• status: New / Approved / Rejected
• approval_time: Date time when the data request was approved or rejected.
• approver_id: Approver user id.
• approval_reason: Text to explain why the request was approved or rejected.
• data_id: ID of the data for which the request was issued. (i.e. Layer id or Document id)
• data_parameters: Specific parameters for each data. (e.g. start time, station id)
Data Request Manager REST API
• POST /drm/api/data_requests: Create a Data Request
• Authenticated User
• GET /drm/api/data_requests/{id}: Get a Data Request
• id: Data Request ID
• Authenticated User
• PUT /drm/api/data_requests/{id}: Update a Data Request
• id: Data Request ID
• Admin User
• DELETE /drm/api/data_requests/{id}: Delete a Data Request
• id: Data Request ID
• Admin User
Data Request Manager REST API (contd.)
• GET /drm/api/data_requests: Get Data Requests
• Authenticated User: Return the list of Data Requests for the user
• Admin User: Return the list of Data Requests from all the users (or from a user specified by "user=")
• Query Parameters:
• ordering (string) – Which field to use when ordering the results.
• reverse (string) – True if decending order, False if ascending order. Default: False.
• page (integer) – A page number within the paginated result set.
• page_size (integer) – Number of results to return per page.

• TBD: Notification API

Software / Library Selection
GeoNode for Data Catalog + Data Storage &
Visualization
• GeoNode has the following features suitable for these 2 sub-systems.
• Control the target to publish the data to the world, a group, or individual. So, it can be used as the Data
Catalog as well as the Data Storage to each individual.
• Support simple approval process at publishing data, which can be a little tweaked and applied for our
case.
• To be studied how to make the tweak.
• Take ordinary documents (office documents, images, pictures etc.) in addition to Geo-data.
• Geo-data format conversion (thanks to the Geoserver in the backend.)
• Various visualization tools for geo-data.

• The programmer in the team has an experience in the Django based app
development. GeoNode is also based on Django framework and thus we can
leverage his skill in this project.
Why not other CMS
• Wordpress and Drupal
• Visualization of geo-data is possible by using extensions. However, not as
good as GeoNode does.
• CKAN
• Seems not possible to deliver data only to a single user. CKAN is designed for
open data.
Software Candidates to implement the Data
Request Manager
• OpenCart: Shopping cart program but can be used for this purpose?
• AWS AppSync: GraphQL with DynamoDB backend. Web UI must be developed.
• Any good one?
Open Questions
Option 1 Option 2

Django Geonode Container Django DPS Django GeoNode

Container Container

Data Request Data Request

GeoNode GeoNode
Manager Manager

django:8000/drm django:8000/ drm:9000/ django:8000/

Nginx Nginx
Container Container

Option 2: Can the user session be shared between GeoNode and Data Request Manager, and how?