Skip to main content

Über dieses Buch

This book is about elicitation: the facilitation of the quantitative expression of subjective judgement about matters of fact, interacting with subject experts, or about matters of value, interacting with decision makers or stakeholders. It offers an integrated presentation of procedures and processes that allow analysts and experts to think clearly about numbers, particularly the inputs for decision support systems and models. This presentation encompasses research originating in the communities of structured probability elicitation/calibration and multi-criteria decision analysis, often unaware of each other’s developments.

Chapters 2 through 9 focus on processes to elicit uncertainty from experts, including the Classical Method for aggregating judgements from multiple experts concerning probability distributions; the issue of validation in the Classical Method; the Sheffield elicitation framework; the IDEA protocol; approaches following the Bayesian perspective; the main elements of structured expert processes for dependence elicitation; and how mathematical methods can incorporate correlations between experts.

Chapters 10 through 14 focus on processes to elicit preferences from stakeholders or decision makers, including two chapters on problems under uncertainty (utility functions), and three chapters that address elicitation of preferences independently of, or in absence of, any uncertainty elicitation (value functions and ELECTRE). Two chapters then focus on cross-cutting issues for elicitation of uncertainties and elicitation of preferences: biases and selection of experts.

Finally, the last group of chapters illustrates how some of the presented approaches are applied in practice, including a food security case in the UK; expert elicitation in health care decision making; an expert judgement based method to elicit nuclear threat risks in US ports; risk assessment in a pulp and paper manufacturer in the Nordic countries; and elicitation of preferences for crop planning in a Greek region.



Chapter 1. Elicitation: State of the Art and Science

This book is about elicitation, which may be defined as the facilitation of the quantitative expression of subjective judgement, whether about matters of fact or matters of value. To motivate, we review case studies from human health (swine flu); provision of public services (airport location); natural hazards (assessment of the risk of earthquakes) and environmental protection (in the case of radioactive waste) where elicitation was or could have been profitably used to inform decisions. It is often argued that uncertainties are too deep or human values are too profound for quantitative thinking to be applicable: we argue on the contrary (drawing again on cases) that it is impossible to think about important problems without dealing with problems of “how big” and “how much”. We provide an overview of chapters in the book, which, we argue, shows that there is a huge body of knowledge and expertise about how to elicit both probabilities and preferences in important social problems, and conclude with future trends that make the subject of this book (in our view) particularly timely.
Luis C. Dias, Alec Morton, John Quigley

Chapter 2. Elicitation in the Classical Model

The Classical Model (CM) is a performance-based approach for mathematically aggregating judgements from multiple experts, when reasoning about target questions under uncertainty. Individual expert performance is assessed against a set of seed questions, items from their field, for which the analyst knows or will know the true values, but the experts do not; the experts are, however, expected to provide accurate and informative distributional judgements that capture these values reliably. Performance is measured according to metrics for each expert’s statistical accuracy and informativeness, and the two metrics are convolved to determine a weight for each expert, with which to modulate their contribution when pooling them together for a final combined assessment of the desired target values. This chapter provides mathematical and practical details of the CM, including describing the method for measuring expert performance and discussing approaches for devising good seed questions.
John Quigley, Abigail Colson, Willy Aspinall, Roger M. Cooke

Chapter 3. Validation in the Classical Model

Validation is the hallmark of science. For expert judgment to contribute to science-based uncertainty quantification, it must become amenable to empirical validation. Using data in which experts quantify uncertainty on variables from their fields whose true values are known post hoc, this chapter explains how validation is performed in the Classical Model for structured expert judgment and reviews results for different combination methods.
Roger M. Cooke

Chapter 4. SHELF: The Sheffield Elicitation Framework

The Sheffield elicitation framework is an expert knowledge elicitation framework that has been devised over a number of years and many substantial expert knowledge elicitation exercises to give a transparent and reliable way of collecting expert opinions. The framework is based on the principles of behavioural aggregation where a facilitator-guided group interact and share information to arrive at a consensus. It was originally designed for helping to elicit judgements about single uncertain variables, but, in recent years, the framework and the associated software implementations have been extended to accommodate judgements about more complex multidimensional variables and geographically-dispersed experts. In this chapter, we discuss the aims and foundations of the framework, its extensions and its notable applications.
John Paul Gosling

Chapter 5. IDEA for Uncertainty Quantification

It is generally agreed that an elicitation protocol for quantifying uncertainty will always benefit from the involvement of more than one domain expert. The two key mechanisms by which judgements may be pooled across experts are through striving for consensus, via behavioural aggregation, where experts share and discuss information, and via mathematical methods, where judgements are combined using a mechanistic rule. Mixed approaches combine elements of both deliberative (behavioural) and mechanical (mathematical) styles of aggregation.
This chapter outlines a mixed-aggregation protocol called IDEA. It synthesises specific elements from several of the classical structured expert judgement approaches. IDEA encourages experts to Investigate, Discuss, and Estimate, and concludes with a mathematical Aggregation of judgements.
Anca M. Hanea, Mark Burgman, Victoria Hemming

Chapter 6. Elicitation and Calibration: A Bayesian Perspective

There are relatively few published perspectives on processes and procedures for organising the elicitation, aggregation and documentation of expert judgement studies. The few that exist emphasise different aggregation models, but none build a full Bayesian model to combine the judgements of multiple experts into the posterior distribution for a decision maker. Historically, Bayesian concepts have identified issues with current modelling approaches to aggregation, but have led to models that are difficult to implement. Recently Bayesian models have started to become more tractable, so it is timely to reflect on elicitation processes that enable the model to be applied. That is our purpose in this Chapter. In particular, the European Food Safety Authority have provided the most detailed and thorough prescription of the procedures and processes needed to conduct an expert judgement study. We critically review this from a Bayesian perspective, asking how it might need modifying if Bayesian models are included to analyse and aggregate the expert judgements.
David Hartley, Simon French

Chapter 7. A Methodology for Constructing Subjective Probability Distributions with Data

Our methodology is based on the premise that expertise does not reside in the stochastic characterisation of the unknown quantity of interest, but rather upon other features of the problem to which an expert can relate her experience. By mapping the quantity of interest to an expert’s experience we can use available empirical data about associated events to support the quantification of uncertainty. Our rationale contrasts with other approaches to elicit subjective probability which ask an expert to map, according to her belief, the outcome of an unknown quantity of interest to the outcome of a lottery for which the randomness is understood and quantifiable. Typically, such a mapping represents the indifference of an expert on making a bet between the quantity of interest and the outcome of the lottery. Instead, we propose to construct a prior distribution with empirical data that is consistent with the subjective judgement of an expert. We develop a general methodology, grounded in the theory of empirical Bayes inference. We motivate the need for such an approach and illustrate its application through industry examples. We articulate our general steps and show how these translate to selected practical contexts. We examine the benefits, as well as the limitations, of our proposed methodology to indicate when it might, or might not be, appropriate.
John Quigley, Lesley Walls

Chapter 8. Eliciting Multivariate Uncertainty from Experts: Considerations and Approaches Along the Expert Judgement Process

In decision and risk analysis problems, modelling uncertainty probabilistically provides key insights and information for decision makers. A common challenge is that uncertainties are typically not isolated but interlinked which introduces complex (and often unexpected) effects on the model output. Therefore, dependence needs to be taken into account and modelled appropriately if simplifying assumptions, such as independence, are not sensible. Similar to the case of univariate uncertainty, which is described elsewhere in this book, relevant historical data to quantify a (dependence) model are often lacking or too costly to obtain. This may be true even when data on a model’s univariate quantities, such as marginal probabilities, are available. Then, specifying dependence between the uncertain variables through expert judgement is the only sensible option. A structured and formal process to the elicitation is essential for ensuring methodological robustness. This chapter addresses the main elements of structured expert judgement processes for dependence elicitation. We introduce the processes’ common elements, typically used for eliciting univariate quantities, and present the differences that need to be considered at each of the process’ steps for multivariate uncertainty. Further, we review findings from the behavioural judgement and decision making literature on potential cognitive fallacies that can occur when assessing dependence as mitigating biases is a main objective of formal expert judgement processes. Given a practical focus, we reflect on case studies in addition to theoretical findings. Thus, this chapter serves as guidance for facilitators and analysts using expert judgement.
Christoph Werner, Anca M. Hanea, Oswaldo Morales-Nápoles

Chapter 9. Combining Judgements from Correlated Experts

When combining the judgements of experts, there are potential correlations between the judgements. This could be as a result of individual experts being subject to the same biases consistently, different experts being subject to the same biases or experts sharing backgrounds and experience. In this chapter we consider the implications of these correlations for both mathematical and behavioural approaches to expert judgement aggregation. We introduce the ideas of mathematical and behavioural aggregation and identify the possible dependencies which may exist in expert judgement elicitation. We describe a number of mathematical methods for expert judgement aggregation, which fall into two broad categories; opinion pooling and Bayesian methods. We qualitatively evaluate which of these methods can incorporate correlations between experts. We also consider behavioural approaches to expert judgement aggregation and the potential effects of correlated experts in this context. We discuss the results of an investigation which evaluated the correlation present in 45 expert judgement studies and the effect of correlations on the resulting aggregated judgements from a subset of the mathematical methods. We see that, in general, Bayesian methods which incorporate correlations outperform mathematical methods which do not.
Kevin J. Wilson, Malcolm Farrow

Chapter 10. Utility Elicitation

This chapter introduces key concepts in modelling preferences under uncertainty, focusing on utility elicitation, both in single and multiple attribute problems. We also discuss issues in relation with adversarial preference assessment. We illustrate all concepts with a case combining aspects of energy and homeland security.
Jorge González-Ortega, Vesela Radovic, David Ríos Insua

Chapter 11. Elicitation in Target-Oriented Utility

Target-oriented utility theory interprets the utility of a consequence as the probability of the consequence exceeding some benchmark random variable. This shifts the focus of utility assessment to the identification of the benchmark and the sources of uncertainty in that benchmark. Identification of the benchmark is often easy when the benchmark is based on a status quo outcome, a preferred outcome or an undesirable outcome. Benchmarks are generally easy to communicate and easy to track. Once identified, data and models can then be used to describe the uncertainty in the benchmark. This approach can be useful in those applications where the utility function needs to be justified to others.
Robert F. Bordley

Chapter 12. Multiattribute Value Elicitation

Multiattribute Value Theory (MAVT) methods are perhaps the most intuitive multicriteria methods, and have the most theoretically well-understood basis. They are employ a divide-and-conquer modelling strategy in which the value of an option is conceptualised as a function (typically the sum) of the scores associated with the performance of the option on different attributes. This chapter outlines the concept of preferential independence, which has a critical underpinning role of elicitation within the MAVT paradigm. I also present MAVT elicitation in the context of the overall Decision Analysis process, comprising three broad stages: Designing and Planning; Structuring the Model; and Analysing the Model. I outline some of the main practical methods for arriving at the partial values and weighting them to arrive at an overall value score, including both traditional methods relying on cardinal assessment, and the MACBETH approach which uses qualitative difference judgements. A running example of a house choice problem is used to illustrate the different elicitation approaches.
Alec Morton

Chapter 13. Disaggregation Approach to Value Elicitation

The philosophy of preference disaggregation in multicriteria decision analysis encapsulates the assessment/inference of preference models, from given preferential structures, and the implementation of decision aid activities through consistent and robust operational models. This chapter presents a new outlook on the well-known UTA method, which is devoted to the elicitation of values through the inference of multiple additive value models. On top of that, it incorporates the latest theoretical developments, related to the robustness control of both the decision model and the surfacing decision aiding conclusions. An application example on job evaluation is elaborated as an educative example, while other potential areas for future use applications of the methodological framework are listed. The chapter concludes with several promising directions for future research.
Nikolaos F. Matsatsinis, Evangelos Grigoroudis, Eleftherios Siskos

Chapter 14. Eliciting Multi-Criteria Preferences: ELECTRE Models

Outranking methods are a specific type of Multi-Criteria Decision Aiding methods. They are based on the construction of binary relations validating or invalidating, for any pair of alternatives (a, b), the assertion “a outranks b”. This comparison is grounded on the evaluation vectors of both alternatives, and on additional information concerning the decision maker’s preferences, typically accounting for two conditions: concordance and non-discordance. In decision processes using these methods, the analyst should interact with the decision maker in order to elicit values for the parameters that define a preference model. This can be done either directly or through a disaggregation procedure that infers parameter values from holistic judgements provided by the decision maker. In this chapter we discuss the elicitation of an outranking-based preference model, focusing on the valued outranking relation used in the ELECTRE III and ELECTRE TRI methods.
Luis C. Dias, Vincent Mousseau

Chapter 15. Individual and Group Biases in Value and Uncertainty Judgments

Behavioral decision research has demonstrated that value and uncertainty judgments of decision makers and experts are subject to numerous biases. Individual biases can be either cognitive, such as overconfidence, or motivational, such as wishful thinking. In addition, when making judgements in groups, decision makers and experts might be affected by group-level biases. These biases can create serious challenges to decision analysts, who need judgments as inputs to a decision or risk analysis model, because they can degrade the quality of the analysis. This chapter identifies individual and group biases relevant for decision and risk analysis and suggests tools for debiasing judgements for each type of bias.
Gilberto Montibeller, Detlof von Winterfeldt

Chapter 16. The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation

Several different EKE protocols are reviewed in this volume, each with their pros and cons, but any is only as good as the quality of the experts and their judgments. In this chapter a structured approach to the selection of experts for EKE is presented that is grounded in psychological research.
In Part I various definitions of expertise are considered, and indicators and measures that can be used for the selection of experts are identified. Next, some ways of making judgements of uncertain quantities are discussed, as are factors influencing judgment quality.
In Part II expert selection is considered within an overall policy-making process. Following the analysis of Part I, two new instruments are presented that can help guide the selection process: expert profiles provide structure to the initial search, while a questionnaire permits matching of experts to the profiles, and assessment of training needs. Issues of expert retention and documentation are also discussed.
It is concluded that although the analysis offered in this chapter constitutes a starting point there are many questions still to be answered to maximize EKE’s contribution. A promising direction is research that focusses on the interaction between experts and the tasks they perform.
Fergus Bolger

Chapter 17. Eliciting Probabilistic Judgements for Integrating Decision Support Systems

When facing extremely large and interconnected systems, decision-makers must often combine evidence obtained from multiple expert domains, each informed by a distinct panel of experts. To guide this combination so that it takes place in a coherent manner, we need an integrating decision support system (IDSS). This enables the user to calculate the subjective expected utility scores of candidate policies as well as providing a framework for incorporating measures of uncertainty into the system. Throughout this chapter we justify and describe the use of IDSS models and how this procedure is being implemented to inform decision-making for policies impacting food poverty within the UK. In particular, we provide specific details of this elicitation process when the overarching framework of the IDSS is a dynamic Bayesian network (DBN).
Martine J. Barons, Sophia K. Wright, Jim Q. Smith

Chapter 18. Expert Elicitation to Inform Health Technology Assessment

In the face of constrained budgets, unavoidable decisions about the use of health care interventions have to be made. Decision makers seeking to maximise health for their given budget should use the best available information on effectiveness and cost-effectiveness, and for this purpose they may use a process of gathering and combining existing evidence in this context called Health Technology Assessment (HTA). In informing decisions, utilising HTA, expert elicitation can provide valuable information, particularly where evidence is missing, where it may not be as well developed (e.g. diagnostics, medical devices, early access to medicines scheme or public health) or limited (insufficient, not very relevant, contradictory and/or flawed). Here, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision making process, in addition to the important role in reducing bias and the use of heuristics. There have been a limited number of applications of expert elicitation in health care decision making, and in part this may be due to a number of methodological uncertainties regarding the applicability and transferability of techniques from other disciples, such as Bayesian statistics and engineering, to health care. This chapter discusses the distinguishing features of health care decision making and the use of expert elicitation to inform this, drawing on applied examples in the area illustrating some of the complexities and uncertainties.
Marta O. Soares, Laura Bojke

Chapter 19. Expert Judgment Based Nuclear Threat Assessment for Vessels Arriving in the US

We demonstrate the use of extended pairwise comparisons for estimating the relative likelihood that a vessel approaching US waters contains a nuclear threat. We demonstrate an expert judgment based method consisting of a designed set of extended pairwise comparisons and parameter estimation for a predictive probability model using log-linear regression. Results are based on a proof-of-concept questionnaire completed by eight experts in port security. The model and parameter estimates obtained are used to demonstrate the type of predictions that can be obtained.
Jason R. W. Merrick, Laura A. Albert

Chapter 20. Risk Assessment Using Group Elicitation: Case Study on Start-up of a New Logistics System

This chapter presents a risk assessment for the start-up of a new logistics system within the pulp and paper manufacturer Stora Enso. The risk assessment was realised as a structured expert elicitation workshop using a computerised group support system. Experts representing different parts of the logistics system were invited to a one-day workshop to assess risks concerning the system start-up. The main topics of the workshop were hazard identification, risk estimation and risk control. Each identified risk scenario was assessed with regard to its likelihood and three consequence types related to logistics: timeliness, product quality and information quality. The top priority risks were identified and risk controls were outlined.
The computerised group support system made the workshop more efficient due to the possibility of simultaneous inputs from all participants to a shared environment, versatile processing possibilities of the inputs, voting features with instant results and automated documentation. An essential factor for the success of the workshop was thorough preparation in cooperation between the analysts and the problem owner. Each step of the workshop process was specified and special attention was given to ensure the elicitation questions were clear and unambiguous.
The risk assessment resulted in a prioritised list of realistic risk scenarios for the start-up of the logistics system and control ideas for the most important risks. The results helped the company structure their work to ensure a problem free start-up. In addition, the workshop participants found it valuable to meet representatives from other parts of the logistics chain.
Markus Porthin, Tony Rosqvist, Susanna Kunttu

Chapter 21. Group Decision Support for Crop Planning: A Case Study to Guide the Process of Preferences Elicitation

The land of Paggaio, Kavala, Greece although very rich, has been cultivated in ways that affected both local environment and economies disadvantageously giving rise to the crucial problem of strategic crop planning. However, because of the many actors involved, and of their conflicting interests, reaching a consensus about what the objectives of such a planning should be, is a complex and challenging task. So as a first, preparatory step for strategic crop planning, the interested parties should acquire a clear view about what are the differences in the preferences of the involved actors. In this chapter, we present the steps that we followed in order to execute an end-to-end process for a client that needed to unveil what are the criteria that guide the preferences of the actors and which actors converge (or diverge) the most, with respect to the evaluation on these criteria. Following a prescriptive approach (that assumes that a preference model exists), we sketched the relevant problem situation and problem formulation, constructed an evaluation model based on a multiple criteria technique, and eventually reached some recommendations. The case study we present in this work could help analysts to structure their own decision aid processes based on an established roadmap, as well as to become aware of the process pitfalls. Regarding the referenced case study, it showed that actors have strongly diverging preferences, so that it was not possible to discover a robust collective model. However, we were able to identify the points of major conflict in two criteria (environmental friendliness and economical performance) and amongst certain stakeholders.
Pavlos Delias, Evangelos Grigoroudis, Nikolaos F. Matsatsinis
Weitere Informationen

Premium Partner