Partial order investigation of multiple indicator systems using variance-based sensitivity analysis
Introduction
Our setting is the ranking of objects on a multi-criteria basis along some complex, non-observable dimension such as e.g. competitiveness, innovation or sustainability. This calls for a set of indicators (attributes) which serve as proxy for the underlying unobservable phenomenon. The measures need then to be combined to obtain a composite indicator on which the ranking is based. A rich literature has shown that partial order is a useful step to understand the role of indicators within a ranking system (examples can be found in Voigt et al., 2004, Brüggemann and Carlsen, 2006). Saltelli (2007) characterizes composite indicators as sitting between advocacy and analysis. To the extent that a partial order analysis can be considered as an alternative or a complement in composite indicators building,1 one can say that it sits on the analytical side of the scale, having attracted considerable attention as a tool for the study of the property of indicators in an aggregation system.
Any multi-indicator system, applied on a set of objects, is an instance for a partial order. A survey is given in Bruggemann and Patil, in press, Brüggemann and Patil, 2010. A key to ranking and prioritization is the process of comparison. The comparison between two objects, say a and b, may be formalized by a ≤ b which expresses that b is to be preferred over a (or b is ‘greater’ than a, or b is ‘better’ than a, etc.). Here we study the comparison of objects based on the information stored in the data matrix which contains the values of the indicators for each object. The preference relation between two objects a ≤ b is defined as b scoring not less than a for all the indicators with at least a strict order relation. In this sense the preference relation is an ‘and’ operator, as used for instance in the terminology of the ordered weighted averaging operators (Yager, 1988). The mathematical discipline which provides the background is the theory of partial order in its special variant of Hasse diagram technique. An overview of this technique is given in Section 2.1.
Sensitivity analysis looks at the robustness of model based inference (in the case of the present work the inference is about the order relations of a set of objects) with respect to uncertainty in the model specification and model input assumptions (Leamer, 1990). Here uncertainties are in the value of the indicators or their inclusion/exclusion in the model. We use in particular global, variance-based sensitivity measures, which have established themselves as a good practice in the discipline (Sobol’, 1993, Saltelli et al., 2000, Saltelli and Tarantola, 2002, Saisana et al., 2005).
Section snippets
Preliminaries
The starting point of the analysis is the data matrix M of dimension n × k with objects under investigations in rows and indicators in columns (herein the term ‘indicator’ is used as synonymous of the term ‘factor’ or ‘attribute’). Given that indicators are measured at least at the ordinal level, matrix M can be can be associated to its corresponding poset as described next.
Proposed method: VB-SA for posets
We tackle here the computation of variance-based sensitivity indices, both main Si and total order , to assess the level of influence of the indicators which define a certain Hasse diagram for a set of elements (IVRS setting, Section 2.2). A simple example will be clarifying the approach. The value of three indicators (columns) are observed on a set of 10 objects (rows). The data matrix is matrix M10x3, Table 1.
The partial order associated to M is a very peculiar case where no comparability
Numerical results for the toy model
The recommended approach in the estimation of VB-SA is to use quasi-random sequences - QR (Saltelli et al., 2010), that is deterministically uniformly distributed sequences that are designed to place sample points as uniformly as possible in the multi-dimensional space of the Xi. Several QR sequences may be used, for a review of possible sequences see Bratley and Fox (1988). The Sobol’ quasi-random sequence is used in this work Bratley et al., 1992, Sobol’, 1998.
Results for the toy case
An environmental chemistry case
Assessing and managing the risk of chemicals is of high concern to both regulators and society at large. Activities like REACH - Registration, Evaluation, Authorization of CHemicals and EUSES, the model following the Technical Guidance Document guidelines of the EU for chemicals’ risk assessment, may serve as examples. Of particular interest is which chemicals should be analyzed in more detail and ranking systems were used for this purpose (Hansen et al., 1999).
Our case study consists of 12
Conclusions
The use of variance-based sensitivity analysis for the special case of partially ordered sets has been explored and proved to be particularly powerful. Being independent from weights and aggregation strategies, posets are a rather robust ordering strategy, which may appeal to stake-holders who dislike weights as ‘too normative’. Coupling partial order with global sensitivity analysis one can check for the influence on the poset structure of the set of indicators. Given that indicators’
References (29)
- et al.
The concept of stability fields and hot spots in ranking of environmental chemicals
Environmental Modelling and Software
(2008) - et al.
Importance measures in global sensitivity analysis of model output
Reliability Engineering and System Safety
(1996) - et al.
Monte carlo evaluation of derivative-based global sensitivity measures
Reliability Engineering and System Safety
(2009) - et al.
How to avoid a perfunctory sensitivity analysis
Environmental Modelling and Software
(2010) - et al.
Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index
Computer Physics Communications
(2010) On Quasi-Monte Carlo integrations
Mathematics and Computers in Simulations
(1998)- P. Annoni, R. Brüggemann, A. Saltelli. Random and quasi-random designs for high dimensional problems. Reliability...
- et al.
Implementation and test of low discrepancy sequences
ACM Transactions on Modeling and Computer Simulation
(1992) - et al.
Algorithm 659 implementing sobol’s quasirandom sequence generator
ACM Transactions on Mathematical Software
(1988) - et al.
Partial Order in Environmental Sciences and Chemistry
(2006)
Applying the concept of partially ordered sets on the ranking of near -shore sediments by a battery of tests
Journal of Chemical Information and Computer Sciences
Multicriteria prioritization and partial order in environmental sciences
Environmental and Ecological Statistics
Valuation of mortgage backed securities using Brownian bridges to reduce effective dimension
Journal of Computational Finance
Cited by (48)
Deriving a minimum set of indicators to assess network component importance
2022, Decision Analytics JournalAssessing manufacturing flow lines under uncertainties in processing time: An application based on max-plus equations, multicriteria decisions, and global sensitivity analysis
2021, International Journal of Production EconomicsA new dependence measure for importance analysis: Application to an environmental model
2019, Applied Mathematical ModellingCitation Excerpt :This link allows variance-based indices to identify the structure of the input-output model function in addition to distinguishing important and unimportant inputs for the output variance. Due to this reason, many numerical methods for implementing variance-based IA have been developed to facilitate the application of this IA technique in many research fields [1,2,18–21]. Different from variance-based IA technique, moment-independent IA focuses on analyzing the effect of the input on the entire distribution of the model output without referring to any specific moment, by averaging the distance between the unconditional and the conditional (on that input) distributions of the output.
Robust global sensitivity analysis under deep uncertainty via scenario analysis
2016, Environmental Modelling and SoftwareA comprehensive comparison of two variable importance analysis techniques in high dimensions: Application to an environmental multi-indicators system
2015, Environmental Modelling and SoftwareCitation Excerpt :All the input variables in these three examples are assumed to be independently and uniformly distributed between 0 and 1. Various VIA techniques have been adopted by several researchers for reducing the uncertainty of the object rankings of multi-indicators systems (see for example Annoni et al., 2011; Annoni et al., 2012; Rocco and Tarantola, 2014). In this paper, the PVIM and Morris' screening design are applied to an environmental multi-indicators system which concerns the environmental hazard of 34 chemicals.