7. Evaluation Support System for Open Challenges on Earth Observation Topics

This chapter presents a model-driven evaluation support system (ESS) for open competitions within Earth Observation (EO) topics, which was part of a project financed by ESA (European Space Agency). One objective of the project was to deliver a collaborative platform that, through Data Challenges (i.e. open competitions), could improve the adoption and outreach of new applications and methods to process EO data. A second objective was to develop a common environment (web portal) where the challenges could be developed, deployed and executed; and thirdly, to easily publish results in a visualisation platform providing a user friendly interface for validation, evaluation and transparent peer comparisons. In general, the developed tool is an evaluation support system with four main requirements: support initiators/evaluators to define a general challenge in the EO domain; enable easy interaction with the candidates/contestants to submit their entries; provide criteria ratings and candidate evaluation score; and transparent peer comparison. Further, the ESS should be adaptable to several different topics within the EO domain and robust enough to deal with qualitative and quantitative data. In this work the focus is on the conceptual evaluation model of ESS, which is based on multi-criteria concepts and a data fusion algorithm capable of standardizing all criteria and providing a sophisticated aggregation process to rank the candidates. The proposed ESS model includes a tree representation of the criteria and their relationships, and then a bottom-up hierarchical process to aggregate the results from the leaf nodes to the top final score. To better explain the evaluation model and the bottom-up synthetizing process we used a step-by-step example and then an experimental case.