Partial order investigation of multiple indicator systems using variance-based sensitivity analysis

https://doi.org/10.1016/j.envsoft.2011.01.008Get rights and content

Abstract

Partial order tools can be used in multiple criteria analysis to prioritize and rank a set of objects. In this setting the starting point is generally a matrix Mnxk of k observed indicators on n objects. Given that indicators are measured at least at the ordinal level, from matrix M its corresponding partially ordered set - poset - is set up to form the basis of multi-criteria ranking. The partial order may be very complex even when the number of objects to be compared is relatively small. The reason of such complexity is often due to the intrinsic nature of partial order which is exclusively based on the ordinal properties of the data matrix. Incomparabilities between objects can be due even to very small differences in the observed values of indicators thus causing ‘irrelevant’ incomparabilities. Also, objects may have been characterized with a redundant set of variables so that the change in the values of some indicators should not seriously affect the structure of the poset. These two opposite conditions are directly linked to the indicator level of influence and call for an indicator value related sensitivity analysis for testing the robustness of posets. In this work we propose a method to carry out a sensitivity analysis for posets by using variance-based sensitivity indices to detect main effects and interactions between indicators. To this aim, we characterize the poset structure with scalar measures and compute variance-based sensitivity indices according to the most recent practice for a fully exploratory sensitivity analysis. These indices allow for detecting least and most influencing indicators.

Introduction

Our setting is the ranking of objects on a multi-criteria basis along some complex, non-observable dimension such as e.g. competitiveness, innovation or sustainability. This calls for a set of indicators (attributes) which serve as proxy for the underlying unobservable phenomenon. The measures need then to be combined to obtain a composite indicator on which the ranking is based. A rich literature has shown that partial order is a useful step to understand the role of indicators within a ranking system (examples can be found in Voigt et al., 2004, Brüggemann and Carlsen, 2006). Saltelli (2007) characterizes composite indicators as sitting between advocacy and analysis. To the extent that a partial order analysis can be considered as an alternative or a complement in composite indicators building,1 one can say that it sits on the analytical side of the scale, having attracted considerable attention as a tool for the study of the property of indicators in an aggregation system.

Any multi-indicator system, applied on a set of objects, is an instance for a partial order. A survey is given in Bruggemann and Patil, in press, Brüggemann and Patil, 2010. A key to ranking and prioritization is the process of comparison. The comparison between two objects, say a and b, may be formalized by a ≤ b which expresses that b is to be preferred over a (or b is ‘greater’ than a, or b is ‘better’ than a, etc.). Here we study the comparison of objects based on the information stored in the data matrix which contains the values of the indicators for each object. The preference relation between two objects a ≤ b is defined as b scoring not less than a for all the indicators with at least a strict order relation. In this sense the preference relation is an ‘and’ operator, as used for instance in the terminology of the ordered weighted averaging operators (Yager, 1988). The mathematical discipline which provides the background is the theory of partial order in its special variant of Hasse diagram technique. An overview of this technique is given in Section 2.1.

Sensitivity analysis looks at the robustness of model based inference (in the case of the present work the inference is about the order relations of a set of objects) with respect to uncertainty in the model specification and model input assumptions (Leamer, 1990). Here uncertainties are in the value of the indicators or their inclusion/exclusion in the model. We use in particular global, variance-based sensitivity measures, which have established themselves as a good practice in the discipline (Sobol’, 1993, Saltelli et al., 2000, Saltelli and Tarantola, 2002, Saisana et al., 2005).

Section snippets

Preliminaries

The starting point of the analysis is the data matrix M of dimension n × k with objects under investigations in rows and indicators in columns (herein the term ‘indicator’ is used as synonymous of the term ‘factor’ or ‘attribute’). Given that indicators are measured at least at the ordinal level, matrix M can be can be associated to its corresponding poset as described next.

Proposed method: VB-SA for posets

We tackle here the computation of variance-based sensitivity indices, both main Si and total order STi, to assess the level of influence of the indicators which define a certain Hasse diagram for a set of elements (IVRS setting, Section 2.2). A simple example will be clarifying the approach. The value of three indicators (columns) are observed on a set of 10 objects (rows). The data matrix is matrix M10x3, Table 1.

The partial order associated to M is a very peculiar case where no comparability

Numerical results for the toy model

The recommended approach in the estimation of VB-SA is to use quasi-random sequences - QR (Saltelli et al., 2010), that is deterministically uniformly distributed sequences that are designed to place sample points as uniformly as possible in the multi-dimensional space of the Xi. Several QR sequences may be used, for a review of possible sequences see Bratley and Fox (1988). The Sobol’ quasi-random sequence is used in this work Bratley et al., 1992, Sobol’, 1998.

Results for the toy case

An environmental chemistry case

Assessing and managing the risk of chemicals is of high concern to both regulators and society at large. Activities like REACH - Registration, Evaluation, Authorization of CHemicals and EUSES, the model following the Technical Guidance Document guidelines of the EU for chemicals’ risk assessment, may serve as examples. Of particular interest is which chemicals should be analyzed in more detail and ranking systems were used for this purpose (Hansen et al., 1999).

Our case study consists of 12

Conclusions

The use of variance-based sensitivity analysis for the special case of partially ordered sets has been explored and proved to be particularly powerful. Being independent from weights and aggregation strategies, posets are a rather robust ordering strategy, which may appeal to stake-holders who dislike weights as ‘too normative’. Coupling partial order with global sensitivity analysis one can check for the influence on the poset structure of the set of indicators. Given that indicators’

References (29)

  • R. Brüggemann et al.

    Applying the concept of partially ordered sets on the ranking of near -shore sediments by a battery of tests

    Journal of Chemical Information and Computer Sciences

    (2001)
  • Bruggemann, R., Patil, G. Ranking and Prioritization of Multi-Indicator Systems-Introduction to Partial Order...
  • R. Brüggemann et al.

    Multicriteria prioritization and partial order in environmental sciences

    Environmental and Ecological Statistics

    (2010)
  • R. Caflisch et al.

    Valuation of mortgage backed securities using Brownian bridges to reduce effective dimension

    Journal of Computational Finance

    (1997)
  • Cited by (48)

    • A new dependence measure for importance analysis: Application to an environmental model

      2019, Applied Mathematical Modelling
      Citation Excerpt :

      This link allows variance-based indices to identify the structure of the input-output model function in addition to distinguishing important and unimportant inputs for the output variance. Due to this reason, many numerical methods for implementing variance-based IA have been developed to facilitate the application of this IA technique in many research fields [1,2,18–21]. Different from variance-based IA technique, moment-independent IA focuses on analyzing the effect of the input on the entire distribution of the model output without referring to any specific moment, by averaging the distance between the unconditional and the conditional (on that input) distributions of the output.

    • A comprehensive comparison of two variable importance analysis techniques in high dimensions: Application to an environmental multi-indicators system

      2015, Environmental Modelling and Software
      Citation Excerpt :

      All the input variables in these three examples are assumed to be independently and uniformly distributed between 0 and 1. Various VIA techniques have been adopted by several researchers for reducing the uncertainty of the object rankings of multi-indicators systems (see for example Annoni et al., 2011; Annoni et al., 2012; Rocco and Tarantola, 2014). In this paper, the PVIM and Morris' screening design are applied to an environmental multi-indicators system which concerns the environmental hazard of 34 chemicals.

    View all citing articles on Scopus
    View full text