Organizing earth observation data inside a spatial data infrastructure

In the last half decade the use of spatial information has grown in importance mainly in public sectors as well as in different research fields, such as Earth Observation (EO), environmental monitoring and computer science. This area is one of the fastest growing today (Litwin and Rossa 2011) and the global revenue in this sector is 10 % and is growing every year (Litwin and Rossa 2011). The popularization of geographical content on the world wide web has a significant impact, mainly driven by the online services provided by Internet pioneers such as Google, Yahoo, and Microsoft, in which the geolocation takes an important role. Also in the Remote Sensing research field spatial data has a significant importance. Research institutes, such as NASA or DLR, that daily receive from the receiving station a huge amount of satellite data in heterogeneous formats, need to manage this information efficiently, in order to avoid having space problems, inconsistency because of data redundancy and other kind of problems provoked by the quantity and diversity of data.

The very first objective of the institute for Applied Remote Sensing is to focus on the integrated environmental monitoring in mountain regions. Therefore the scientists are dealing with different kind of vector and raster data acquired from different sources (satellite, ground sensor, web, partners, third-party providers). This information is stored in different repositories (file-servers, databases, data-tapes) and the big challenge consists in organizing this large amount of heterogeneous data in an efficient, reliable, robust and intuitive way.

A Spatial Data Infrastructure (SDI) that provides services for storing, discovering, querying and exchanging geographic information, simplifies the management for large and heterogeneous data. For retrieving information, spatial data can be managed in a similar way as books are organized in a library, using a catalog. A spatial catalog is a system that allows to search for spatial data based on metadata and it is a component of the SDI. The term metadata refers to “data about data”. At the basic level, metadata should characterize an entity by answering at least the following questions (Litwin and Rossa 2011): What — what it is and what it refers to; Why — to what purpose it was created; When — when it was produced, published or updated; Who — who created or developed it; How — how it was produced, is it reliable, how to get access on it; Where — what area it refers to. Wilson (2008) defines three different levels in the usage of metadata: (a) discovery metadata: discover the semantics of a data element in data sets (metadata scanning); (b) exploration metadata: comprises detailed information about quality, accuracy and origin of the data; (c) exploitation metadata: expresses how to read, transfer and integrate this data in applications.

The benefit in using metadata is a quick access to the desired information, analog like the book search in a library using a catalog. A spatial catalog offers to search for any kind of spatial data independent on the location where it is stored. Without a catalog it would not be possible to perform a search on two datasets coming from different sources (e.g. filestore vs. database). For a spatial catalog metadata provide information about the purpose, quality, actuality and accuracy of a spatial entity set and consequentially provide the relevant input parameter for an accurate search. Moreover a catalog allows to granularize the search including a keyword based search, temporal and spatial filters and to manage user policies.

Most catalogs use a standardized format for structuring their metadata. The standard technical committee established within ISO (ISO/TC 211) in cooperation with other organizations developed the series of International Standards and Technical Specifications in the range starting at 19101 (Ostensen and Danko 2005). For the organization of spatial metadata the following ISO standards have a significant importance: ISO 19115 provides an abstract and logical model for geospatial metadata. Because of its international acceptance, this standard has become a part of the “OpenGIS Specification” as the abstract model for the management of metadata. ISO 19115 defines the metadata to record information about the identification of geospatial data sets, possible approaches for distributing the data, details about the quality, geographic facets about the coordinate reference system and the coverage, temporal information about the date of acquisition, the owner of the data.

However, this standard can not fulfill a big part of the requirements for imagery and gridded data that are relevant for EO-data. In order to cover these missing information in a standardized format, there was published the extension for gridded and imagery data (ISO 19115-2) in the year 2009. That new standard details, how a dataset was acquired, what kind of instrument has been used to produce the data, what kind of algorithm was used to process the data and remarks about the quality of a dataset.

Since XML became as one of the most frequently used exchange format, there has been developed a corresponding XML encoding for these two abstract models. The standards ISO 19139 being the implementation of ISO 19115 and ISO 19139-2 being the one of ISO 19115-2 consist of a set of XML schema files that define the grammar of the structure.

There exist some online catalog products, such as the ESRI Geoportal Server or Geonetwork Opensource, that use some of these ISO standards for managing metadata. Both products are Open Source and offer predefined metadata schema templates fulfilling ISO 19139. But none of these products support to organize metadata of ISO 19139-2.

This paper describes how large and heterogeneous amount of EO-data coming from different sources can be efficiently organized in a centralized SDI using a spatial catalog. Due to the vast data volume an automatic data deployment approach is necessary to register datasets in the SDI, to extract metadata from the raw data, to populate it in the catalog and to archive data in a store.

The contribution of the actual article includes:

Nowadays, with this enhancement it is possible to manage all the institute’s relevant satellite products within the SDI.

The paper is organized as following. Section Related work delineates related work about SDI implementations, search catalogs and metadata generation methods. Section Processing of earth observation data and metadata generation describes the architecture for managing EO-data including near-real-time processing, the metadata extraction and creation process. In Section Extending and implementing ISO schema for EO data are presented the developed metadata models suitable for EO-data and how its integration in the spatial catalog. Section Conclusion and outlook summaries the paper and points out ongoing and future activities.

A geoportal is a type of web portal used to discover, view and access spatial information using geographic services (display, editing, analysis, etc.) via the Internet. Geoportals are important for an effective use of geographic information systems (GIS) and a key element of Spatial Data Infrastructure (SDI). Geographic information providers, including government agencies and commercial sources, use geoportals to publish geospatial metadata. Geographic information consumers, professional or casual, use geoportals to search and access the information they need. As a consequence geoportals serve an increasingly important role in the sharing of geographic information, and they can assist in avoiding duplicated efforts, inconsistencies, delays, confusion, and wasted resources.

The most common online geoportals are the ESRI Geoportal Server and Geonetwork Opensource. Both of them are Open Source products, offering predefined metadata schema templates for ISO-19115 and ISO-19139, as well as the possibility to define and create customized extensions. Furthermore, Geonetwork offers a large amount of documentation, examples and templates. Geonetwork offers a predefined set of schema plugins based on ISO standards for describing different kind of data. The standard technical committee established within ISO (ISO/TC 211) in cooperation with other organizations developed the series of International Standards and Technical Specifications in the range starting at 19101 (Ostensen and Danko 2005).

Previously there have been elaborated general metadata standards. In particular for the geographic area the Spatial Data Transfer Standards (SDTS), the Vector Product Format (VPF) and the Digital Exchange Standards (DIGEST) were developed to allow the encoding of digital spatial data sets to be exchanged in spatial data software. These standards include the support of metadata, but have not been considered until recently to standardize their encoding for the export of the data. On the contrary, ISO 19115 defines metadata elements and schema in order to document geographical data in standardized and comprehensive way. The XML Schema implementation derived from ISO 19115 could be encoded according to ISO 19139.

Considering the fact that the institute mainly works with the Earth Observation data, the ISO 19115 could not fulfill all the requirements for metadata description. For this reason the extension ISO 19115-2 and its XML schema implementation ISO 19139 were taken into account. Thus ISO 19115-2 extends the existing geographic metadata standard by defining the schema required for describing imagery and gridded data (National Coastal Data Development Center et al. 2012b). According to the this extension, the acquisition information could be described including platform and instrument characteristics, information about environmental condition, plan of the measurements and other details about the acquisition process. Despite the standard covers a great deal of the information, the diversity of geographic data causes difficulties in satisfying all the requirements. Hence the ISO metadata standard provides a standardized way for users to extend their metadata and still ensure interoperability allowing other users to comprehend and exploit this extended metadata (Ostensen and Danko 2005). Under the analysis of the Earth Observation data it was revealed that imagery information obtained from different satellites could not be fully defined within the framework of ISO 19115 and its extension ISO 19115-2. Thus it was decided to work on extension for ISO 19115-2 which can fully cover any scientific interesting satellite mission, optical and SAR, such as Landsat, MODIS, S-NPP, RapidEye, Sentinel-1 and Sentinel-2.

Currently, the main concern is that most geospatial data comes with metadata in different formats or sometimes even without metadata, the latter usually happens to vector data. Spatial metadata can be created and updated manually or with semi-automatic and automatic approaches (Olfat et al. 2010). The first two approaches are considered as monotonous, time consuming, and a labor-intensive processes by organizations and they are commonly viewed as an overhead and extra cost (Olfat et al. 2010). Much research has been focused in the field of automatic metadata extraction in particular in the research field of digital libraries where interoperability is essential for exchanging articles between different institutions (e.g. IEEE). There have been used several methods for automatic metadata extraction: regular expressions, machine learning, and rule-based parsers are the most popular (Giles et al. 2003). Regular expressions techniques filters out metadata from the data using specific patterns. Machine learning methods are approaches that include training data and machine self-correction based on errors in machine performance against the training set (Greenberg et al. 2005). Machine learning methods are robust and adaptable, and theoretically can be used in any document set (Olfat et al. 2010). For information extraction they include symbolic learning, inductive logic programming, grammar induction, Support Vector Machines, Hidden Markov models, and statistical methods. Rule-based methods, rule discovery or rule extraction from data are the data mining techniques aimed at understanding data structures that providing comprehensible description instead of only black-box prediction (Dubitzky et al. 2011). In metadata extraction or creation this means a set of rules based on a particular standard are predefined which governs the metadata parsing and creation process. For highly structured tasks rule-based methods are easier to implement (Olfat et al. 2010). Rule-based parsers are straightforward to implement and they depend on a specific application domain and need experts to set up the rules of generation, but compared to machine learning methods they are easier and faster to be implemented. Having in-house experts and due to lack of time the rule-based parser approach has been chosen as metadata extraction method. Basically this method uses XML technologies (XML, XSLT, XPath) for the extraction and generation of geospatial metadata.

Considering the different kind of data as well as the large amount of daily received data (receiving station, ground sensor, third-party satellite data providers) it was necessary to provide a solution for the issue of automatic handling of data. A concept of data ingestion and data management in general has been implemented through a near real time processing chain as illustrated in Fig. 1.

Following the ideal workflow of one specific dataset, the Data Feeder collects new data, triggers the processing, extracts and generates the metadata. Then it transforms the generated metadata to a standardized format (e.g. ISO 19139-2), stores data and their metadata into the SDI. There are some key requirements which lead to the proposed solution, namely allowing flexibility in generating ad-hoc metadata information for particular purposes, and possibly have interoperability versus third party or custom metadata formats. Further, the concept has been developed considering also the necessity to generate on demand (offline) metadata information for existing, third party heterogeneous datasets. In order to meet these requirements we introduced the concept of an “Intermediate Metadata Format” which consists of a dynamically generated XML file containing all information which can be extracted directly from the data (e.g. pixel reference information, dimensions, etc.) and a statically coded XML file that holds any other needed metadata fields (e.g. INSPIRE metadata). This intermediate metadata format is transformed to different target formats. The technologies used for the implementation of the overall architecture and data processing chain are Java, XML and XSLT.

An important component in the processing chain is the Data Exchange Server (DES). The DES is an application that enables to transfer data among different systems. It was developed to fit the requirements of the heterogeneous data handling, like ingestion, processing and transferring from one system to another. The DES also allows to deliver data to external users by implementing a concurrent, multi-threaded mechanism to transfer different kind of data in PUSH and GET mode using standard protocols such as SFTP, FTP, SSH, WEBDAV. In general the DES might be considered as a multi-tasking application that can perform any kind of configurable tasks or jobs by executing dedicated plugins.

The task and jobs directly involved into the metadata generation will be described in the following sections. Section The near real time metadata generation will focus on the NRT processing chain metadata generation used for data acquired directly by the receiving station or downloaded from the ESA Sci-Hub. Section Offline metadata generation describes the offline metadata generation used for third-party data.

As already mentioned in the introduction Section Introduction there exists the standard XML encoding ISO 19139 for GIS metadata and its extension ISO 19139-2 for imagery and spatial data used in particular for EO metadata. Nevertheless these metadata standards can not fulfill all requirements of EO scientists.

For instance, for a refined search the following queries could not be answered:

Moreover any optical sensor based on satellite mission, like for example MODIS, S-NPP VIIRS LANDSAT, RapidEye, Sentinel-2A, have specific characteristics about the single bands of the sensor, that are not mentioned in the existing schema, but they are relevant for researchers. Thus it leads to the necessity of extending the metadata standard 19139-2 with new classes. The following table gives a review about the existing standards, how they are implemented as an XML encoding and how they are supported in the Open Source catalog Geonetwork (Table 1).

The metadata standard for geographic data in the abstract model ISO 19115 has the XML encoding ISO 19139 that is fully implemented in XML and completely integrated as a schema plugin in Geonetwork (tested with version >2.6). Its extension ISO 19115-2, the metadata standard for imagery and gridded data, with the XML encoding ISO 19139-2 is not fully implemented in XML (missing packages and inconsistency according to the the abstract model) and there is not an available schema plugin in Geonetwork. For this reason EO-data can only be stored in the catalog with the basic information from the abstract model ISO 19115. Naturally, for the new introduced extension ISO-19115-2e until now there is not an available abstract model, XML encoding and a schema plugin for Geonetwork. How these missing gaps are filled is described in the next two sections.

Metadata is one of the most important information associated with spatial data, because it provides vital information about the identification of a dataset. In this paper there was presented an automatic method based on the rule-based methodology that extracts metadata from a heterogeneous set of remote sensor data. The used metadata formats are based on ISO standards following the XML encoding from ISO 19139 and ISO 19139-2. The metadata standard ISO 19139-2 for describing imagery and gridded information was extended with additional information to cover all relevant metadata information of different satellite data formats on the most scientific interesting missions, like LANDSAT, MODIS, S-NPP VIIRS, RapidEye and ESA Sentinel-2A and Sentinel-1A. ISO 19139-2 and its extension were incorporated in the spatial catalog Geonetwork Opensource by implementing new schema plugins. The schema plugin developed for ISO 19139-2 was released for the Geonetwork community and can be downloaded at the Geonetwork github repository (https://​github.​com/​geonetwork/​schema-plugins). The implementation of this work provokes three important benefits. First, the automatic metadata extraction that replaces the manual metadata creation saves a lot of time for the creation of metadata. Second, the management of the most relevant remote sensor data via metadata inside a spatial catalog, has improved the organization of the data inside the institute. The extended search functionality was improved to obtain the dataset, someone is looking for, easily and quickly. As a consequence a drastic reduction of data duplicates has occurred. Nowadays, thanks to the used catalog, researchers can interactively browse through spatial data without the need in additional GIS software. Last but not least, the produced results are beneficial for the entire Geonetwork community. The integration of the standard ISO 19139-2 as a new schema plugin in this widely used catalog and providing this plugin publicly available expand the application area of the catalog to cover the EO application field.

For the future development it is planned to expand the metadata format and the schema plugin for other type of sensor data. Inspired by the mission of Sentinel-2 launched in March 2014, in the near future a new extension of ISO 19115-2 will be elaborated in order to satisfy all requirements of EO metadata. Moreover, on the basis of new projects starting in field of environmental monitoring, a new metadata schema dedicated for ground sensor data will be finalized.