Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions

Highlights

• The literature on eliciting dependence for probabilistic modelling in risk analysis and uncertainty modelling is reviewed.

• A general modelling context which shows how modelling and eliciting dependence are related is presented.

• Elicitation for quantifying various dependence models is discussed.

• Dependence parameters that are commonly elicited are reviewed together with the implications for an expert’s assessment burden.

Abstract

Many applications in decision making under uncertainty and probabilistic risk assessment require the assessment of multiple, dependent uncertain quantities, so that in addition to marginal distributions, interdependence needs to be modelled in order to properly understand the overall risk. Nevertheless, relevant historical data on dependence information are often not available or simply too costly to obtain. In this case, the only sensible option is to elicit this uncertainty through the use of expert judgements. In expert judgement studies, a structured approach to eliciting variables of interest is desirable so that their assessment is methodologically robust. One of the key decisions during the elicitation process is the form in which the uncertainties are elicited. This choice is subject to various, potentially conflicting, desiderata related to e.g. modelling convenience, coherence between elicitation parameters and the model, combining judgements, and the assessment burden for the experts. While extensive and systematic guidance to address these considerations exists for single variable uncertainty elicitation, for higher dimensions very little such guidance is available. Therefore, this paper offers a systematic review of the current literature on eliciting dependence. The literature on the elicitation of dependence parameters such as correlations is presented alongside commonly used dependence models and experience from case studies. From this, guidance about the strategy for dependence assessment is given and gaps in the existing research are identified to determine future directions for structured methods to elicit dependence.

Introduction

In decision making under uncertainty it is vital that dependencies between uncertain variables are appropriately modelled, as otherwise the model may not be fit for purpose. Dependent uncertainty may arise either directly because variables in the model are correlated, or indirectly when an uncertainty analysis of model parameters is carried out to explore model robustness. Both cases exhibit complex interrelations and dependencies which need to be considered if assumptions such as independence are not justifiable.

However, it is often not straightforward to either model or quantify dependence. In particular whenever no relevant historical data are available, the only sensible way to achieve uncertainty quantification is through eliciting expert judgements. When performed rigorously, the elicited quantities, often aggregated from multiple experts, offer reliable information for model quantification. Nevertheless, there are several different broad approaches and many choices to be made by the analyst, all of which can affect the elicitation burden for experts and ultimately also the reliability of the outcome.

While research and reviews that offer guidance exist for methods addressing the elicitation of univariate quantities (Cooke, 1991; European Food and Safety Authority (EFSA), 2014; French, 2011, Jenkinson, 2005, O’Hagan, Buck, Daneshkhah, Eiser, Garthwaite, Jenkinson, Oakley, Rakow, 2006, Ouchi, 2004), and while dependence modelling is an active research area (Kurowicka & Cooke, 2006), little guidance exists about the elicitation of dependencies. The exceptions are Bayesian (Belief) nets (BNs), though also for these modelling and elicitation challenges remain, as shown later. In fact, developing defensible elicitation processes for multivariate quantities is still much under development despite its fundamental importance for decision as well as risk analysis (Moskowitz, Bunn, 1987, Smith, Von Winterfeldt, 2004). Some of the first studies that elicit dependence are Cooke and Kraan (1996), Keeney and von Winterfeldt (1991), Kunda and Nisbett (1986), Gokhale and Press (1982) and Kadane, Dickey, Winkler, Smith, and Peters (1980). Since then more ways for quantifying multivariate distributions and models through experts have been investigated, yet on the actual elicitation only little discussion and guidance is available. References that introduce some aspects are Daneshkhah and Oakley (2010), Kurowicka and Cooke (2006), O’Hagan et al. (2006) and Garthwaite, Kadane, and O’Hagan (2005). However, a complete and systematic way of comparing different dependence parameters as elicited quantities, and reflecting their use in dependence models in the form of a literature review has been non-existent so far. Therefore, research and applications of several dependence measures in models and their elicitation methods are presented. With a practical focus, case studies are discussed whenever available. This paper addresses elicitation processes for dependence information and aims at providing understanding of their use in applications. It offers guidance on making robust choices about which summary of expert knowledge on multivariate distributions should be elicited, and how they might be used within a dependence modelling context, as these are key decisions within the overall elicitation process. This is achieved by outlining how much is understood about the complexity of approaches to dependence modelling and the cognitive assessment burden for experts.

Throughout this paper we use the word “dependence” in a general sense (in contrast to specific association measures) to refer to situations where there are multiple uncertain quantities and gaining information about one would change uncertainty assessments for some others. More formally, two unknown quantities X and Y, are independent (for me) if I do not change my beliefs about X when given information about Y. For higher dimensions I regard all quantities independent of one another if knowledge of one group of variables does not change my belief about other variables. Dependence is simply the absence of independence. It is a property of an expert’s belief about the quantities. This definition relates to Lad (1996) who reminds us that in a subjective probability context one expert’s (in-) dependence assessment might not be shared with another expert possessing a different state of knowledge.

The definition of dependence as we use it here relates directly to the scope of this review. A first comment on the scope is that the word “dependence” is used in many ways within Operational Research (OR) and related fields, and it is worth clarifying how its use here differs from its meaning in other OR contexts. The underlying framework adopted is that of subjective probability (as aforementioned), which plays a key role within expected utility maximisation for decision making. Dependence then, refers to the way we model and assess the probability dependence structure required for such decision support processes. We do not consider non-probabilistic representations of uncertainty, nor do we consider approaches to represent dependence between criteria used to model the preferences of the decision maker as discussed widely in the multi-criteria decision analysis (MCDA) literature.

The foundations of subjective probability are drawn from a wide literature, in which Savage (1954) provides one of the most sophisticated accounts. In this account, probabilities can be assessed through preferences over lotteries, and there are implied consistency rules for preferences which can be empirically validated. It is well known that there is a distinction between normative and empirical validation, so the degree to which researchers choose to be led by normative or empirical consistency has led to many different approaches. For instance, Dubois, Prade, and Sabbadin (2001) provide a theoretical framework which attempts to tie these strands together in the context of possibility theory, and the implications of this are discussed in detail by Cooke (2004).

The modelling of dependence between attributes in MCDA is the subject of a wide literature, and as discussed above, is outside the scope of this review. Facilitative approaches within multi-attribute utility theory provide a variety of models, for which (whenever possible) problem structuring is used to ensure preference independence (Von Winterfeldt, Fasolo, 2009, Wallenius, Dyer, Fishburn, Steuer, Zionts, Deb, 2008), while other approaches have been inspired by issues such as assessing the range of preferences within a stakeholder group (Flari, Chaudhry, Neslo, Cooke, 2011, Neslo, Cooke, 2011), or trying to model preferences based on a limited number of attributes or limited resolution of attribute measurement. For the latter, in particular interaction among criteria in complex systems and dependence of attributes is modelled. This is done for instance to assess the aggregated importance of correlated criteria or further investigate dependent attributes for predictive modelling. Common methods in the OR literature are: non-additive aggregation models such as Choquet and Sugeno integrals (Angilella, Greco, Lamantia, Matarazzo, 2004, Grabisch, 1996, Marichal, 2004), Robust Ordinal Regression (Figueira, Greco, Słowiński, 2009, Greco, Mousseau, Słowiński, 2014) and (Dominance-Based) Rough Set Approaches which use decision rules in the form of if [condition] then [consequent] (Błaszczyński, Greco, Słowiński, 2007, Greco, Matarazzo, Słowiński, 2001, Greco, Matarazzo, Słowiński, 2004).Another interesting approach in this regard is Abbas (2009) who constructs a multi-attribute utility function through a copula, a dependence model that is introduced later for modelling probabilistic dependence. A frequently considered empirical area for MCDA-based approaches is financial portfolio optimisation (Ehrgott, Klamroth, & Schwehm, 2004).

A last comment on the scope is that while we discuss the cognitive complexity of assessing dependence in various ways, such as already considered by Kruskal (1958), and while insights from psychological studies are mentioned, corresponding research streams for causal and association judgements are not reviewed exhaustively. Normative and descriptive models for causal reasoning or mental conceptualisation of correlations, which origin is often attributed to Smedslund (1963), are found for instance in Mitchell, De Houwer, and Lovibond (2009), Gredebäck, Winman, and Juslin (2000), Beyth-Marom (1982) and Allan (1980). An overview and introduction to these areas is given in Hastie (2016) and Shanks (2004).

The paper is organised as follows. Section 2 discusses the extent to which findings from eliciting univariate quantities apply to the elicitation of multivariate ones in order to provide the reader with an indication for the scope of the overall topic. Section 3 introduces the modelling context which shows how modelling and eliciting dependence are related. This offers an overall structure to the research problem. Then, Section 4 discusses how elicitation is approached for quantifying various dependence models. Section 5 presents dependence parameters that are commonly elicited together with its implications for experts’ assessment burden before Section 6 briefly reviews findings on mathematical aggregation of dependence assessments. Section 7 provides an overview of the empirical contributions in the literature based on which Section 8 formulates directions for future research and concludes the paper. We refer to Appendix B (Supplementary material) whenever a technical term needs a more detailed explanation, however the original references should be considered for an extended introduction.