Professional Documents
Culture Documents
Abstract
The Remote Sensing Centre (RSC), within the Queensland Department of
Environment and Resource Management (DERM) is one of the largest remote
sensing groups within Australia. The scope of the RSC operation is
considerable, both in terms of data managed and projects undertaken. The
centre undertakes extensive processing of large archives of remotely sensed
imagery, along with manual image interpretation and significant field work to
produce a variety of spatial products. These operations rely heavily on the
centre’s core processing and data management systems, which have been
primarily developed around open source tools such as Linux, PostGIS, QGIS,
GDAL/OGR, Python and R. While commercial software packages, such as
ERDAS Imagine, ESRI ArcGIS and ENVI, continue to play an important role,
the centre has largely moved away from the use of proprietary software for its
automated processing. This paper discusses the operational application of
open source systems in place at RSC and our experience with their
implementation, use and resource considerations.
Introduction
The Remote Sensing Centre is engaged in numerous remote sensing projects
across Queensland, including: monitoring of vegetation clearing and vegetation
structure; ground cover monitoring; land use mapping; gully mapping; fire scar
mapping and monitoring of selected weeds. To facilitate these projects the
centre obtains, manages and processes large quantities of data from various
sources.
A current priority area for both the Federal and State governments is the Great
Barrier Reef and catchments draining into it. RSC is undertaking a range of
remote sensing projects and programs in these catchments. An overview of
these monitoring activities is provided by Tindall and Witte (2010).
Landsat imagery has traditionally been the primary source of imagery for our
remote sensing projects. Satellite imagery requires significant pre-processing
prior to use in automated image analysis. As a result, in addition to operational
projects, the centre undertakes research into various corrections, such as
radiometric, geometric and topographic correction as well as automated cloud,
cloud shadow and water masking.
1
Apart from Landsat, RSC also obtains various other satellite and remotely
sensed imagery including SPOT4 and SPOT5, MODIS, ALOS PALSAR and
higher resolution data such as Quickbird, Ikonos and LiDAR for priority areas in
the state. In addition to remotely sensed data, there is an extensive field
program dating back 15 years.
The volume of these data is steadily increasing and this has necessitated an
automated approach to data storage, management and user access. To
achieve this, an operational software system has been built around high
performance computing and mass storage facilities (hardware). While
commercial software continues to play an important role in various desktop
operations at RSC (for example, see Grounds and Tindall 2010), the core
automated system at the centre is built entirely on open source tools. Open
source products were initially selected due to their portability across platforms
and low cost, but have numerous other benefits.
2
Remote Sensing Centre Open Source Systems
The culture within the department has primarily supported the adoption of
proprietary software, due to the guaranteed support provided. As such,
implementing open source solutions has historically been challenging. Despite
this, the practicalities of managing an operation of the scale of the Remote
Sensing Centre and the inherent limitations in licensing associated with
proprietary software have led to open source solutions gradually dominating the
operational processes at the centre.
For ease of discussion within this paper, specific details of our operation have
been divided into three key areas: data storage and automated processing
core; spatial database interface; and projects and research.
3
the GDAL library. GDAL and Python are both open source and well supported
in the geospatial community, as such, there is much existing code available for
reuse. As PyModeller was built using Python, any of this code can be accessed
from within it, including the scientific modules SciPy and NumPy. PyModeller
can also be easily incorporated into Python programs to perform image
processing, due to its easy to use scripting interface. Since it is built on top of
open source software, PyModeller will run on almost any platform, including
Linux and Cygwin. All automated raster processing scripts are now operating
with PyModeller.
Figure 1. PyModeller screen shot. Model building screen for PyModeller illustrating a
theoretical model incorporating a number of rasters and a colour table.
GDAL and Python have also made it possible to easily incorporate metadata
into the imagery. Metadata is now inserted into the header file of each
processing stage, documenting how the given image was processed and what
parent images were used. The Python script ‘HistoryView’ allows this
information to be easily viewed.
For a long period, data searching and retrieval on the filestore was managed
solely via command line tools and specialised scripts. While well designed file
naming conventions made it possible to customise scripts allowing for the
search and retrieval of desired imagery, the limitations of these methods and
the growing size of the RSC operation eventually necessitated the introduction
of a spatial database to facilitate data management and querying.
4
Spatial Database Interface
A spatial database is a database which has been spatially enabled to manage
geographic data. Spatial databases are not necessarily relational, but most are,
allowing for querying of data via the SQL query language.
The centre introduced the PostGIS spatial database in September 2006.
PostGIS was selected as the spatial database because it is a powerful, spatially
enabled relational database and is compliant with SQL standards and
compatible with the SGI IRIX and Linux.
The database was initially introduced to manage records of Landsat imagery
and ground control points, but has since expanded to incorporate other
imagery, field data and various other relevant data sources. Currently the
database contains references to over 19,000 Landsat images; over 11,000
MODIS scenes; over 1,200 SPOT images; approximately 25,000
georeferencing points; approximately 1,400,000 field observations; thumbnails
of over 50,000 field images; along with numerous other miscellaneous data
tables relevant to the centre’s operation (Figure 2).
Geometries
Tables
Image Thumbnails
5
The introduction of the database has greatly improved the management and
access to the extensive record of field data captured over the last 15 years.
Until recently, however, these data were stored in text files of varying formats,
and, as such, were difficult to access effectively due to lack of querying
capabilities.
PostGIS acts as the gateway for staff to available data at the centre. PostGIS
has Application Programming Interfaces (API’s) to various other open source
tools in use at the centre including: Python, Quantum GIS (QGIS) and R
making it possible to integrate queries from the database seamlessly into
scripts and, in the case of QGIS, allows for viewing of available data.
QGIS is an open source GIS system with viewing and editing capabilities. It has
developed in parallel with PostGIS resulting in a strong integration of the two
software packages. QGIS is easily customisable using Python, allowing for the
development of specialised ‘plugins’. Within the centre a number of plugins
have been created to assist with the querying, viewing and downloading of
various data (Figure 3). In addition, there are many plugins freely distributed
within the open source community.
Figure 3. Quantum GIS screen. Displayed are a number of plugins, some developed
on-site, used at the centre.
6
easy to learn. As such, Python is suitable for use by operational staff with
limited programming experience.
As discussed previously, GDAL is a raster translation library and is used as the
interface for all raster formats within the centre’s scripts. GDAL is primarily a
translation library, however it also provides a number of command line tools
which easily enable common raster processing functions such as mosaicing,
resampling, masking and reprojecting. GDAL allows for the possibility of time
series analysis through virtual rasters. Virtual rasters effectively stack a series
of rasters upon each other allowing for individual pixel values to be investigated
across time.
OGR is the vector library component of GDAL. The group is also gradually
moving into automating various vector processes using OGR. The OGR library
has recently been used to automate accuracy assessment processes for land
use mapping.
R is the major package used for statistical analysis and modelling within the
centre. R is also open source, and is well supported within the geospatial open
source framework. For example, GDAL supports the conversion of data directly
into a compatible R format and PostGIS has an API for R allowing for direct
querying within R scripts.
QGIS is being adopted by operational staff for some GIS operations; however,
it has not replaced major proprietary software for most desktop GIS processes.
For example, the centre has a major requirement for interactive raster editing
for which no adequate open source solution exists.
7
high levels of skill and experience in programming available in-house.
Interestingly, the transparent and extensible nature of open source does appear
to have increased the overall level of scripting skills at the centre. As a result,
increased levels of automation are being implemented across all aspects of the
centres activities. Despite this, anyone desiring to implement a system around
open source tools will need to consider their support requirements and how
they will meet these, as a certain level of expertise will be required.
Acknowledgements
The authors would like to thank Neil Flood for his efforts in implementing open
source systems at the centre, ongoing support of these systems and
commitment to training staff in the operation of these systems.
References
Open Source Initiative, 2010, The open source definition, viewed 6 August
2010, http://opensource.org/docs/osd
Grounds, S. and Tindall, D., 2010, Mapping in a dynamic state: Operational
mapping for a rather large area. In 15th Australasian Remote Sensing and
Photogrammetry Conference. Alice Springs.
Ramsey, P., 2009, Beyond nerds bearing gifts: the future of the open source
economy. Keynote address (video) at: FOSS4G: Free and Open Source
Software for Geospatial, 20-23 October 2009, Sydney, Australia,
http://www.youtube.com/watch?v=zB_a28vBtBk
Schmidt, M, and Gillingham, S., 2008, Raster data analysis made simple with a
scriptable open source framework: PyModeller. Poster session at: 14th
Australasian Remote Sensing and Photogrammetry Conference. 30 September
- 2 October 2008, Darwin, Australia.
Shorter, C., 2010, Overview of geospatial open source software which is robust,
feature rich and standards compliant. In FIG Congress: Facing the Challenges
– Building the Capacity. 11-16 April 2010, Sydney, Australia.
Tindall, D and Witte, C., 2010, Legislation, policies and research: Queensland
Remote Sensing Centre supporting great barrier Reef conservation and
management initiatives In 15th Australasian Remote Sensing and
Photogrammetry Conference. Alice Springs.