Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top

2021 | Book

Introduction and Overview of Structured Expert Judgement Read chapter Read first chapter

Expert Judgement in Risk and Decision Analysis

Editors: Anca M. Hanea, Gabriela F. Nane, Tim Bedford, Simon French

Publisher: Springer International Publishing

Book Series : International Series in Operations Research & Management Science

Part of: Springer Professional "Wirtschaft+Technik" , Springer Professional "Wirtschaft"

Login to get access
insite
SEARCH

About this book

This book pulls together many perspectives on the theory, methods and practice of drawing judgments from panels of experts in assessing risks and making decisions in complex circumstances. The book is divided into four parts: Structured Expert Judgment (SEJ) current research fronts; the contributions of Roger Cooke and the Classical Model he developed; process, procedures and education; and applications.
After an Introduction by the Editors, the first part presents chapters on expert elicitation of parameters of multinomial models; the advantages of using performance weighting by advancing the “random expert” hypothesis; expert elicitation for specific graphical models; modelling dependencies between experts’ assessments within a Bayesian framework; preventive maintenance optimization in a Bayesian framework; eliciting life time distributions to parametrize a Dirichlet process; and on an adversarial risk analysis approach for structured expert judgment studies.
The second part includes Roger Cooke’s oration from 1995 on taking up his chair at Delft University of Technology; one of the editors reflections on the early decade of the Classical Model development and use; a current overview of the theory of the Classical Model, providing a deep and comprehensive perspective on its foundations and its application; and an interview with Roger Cooke.
The third part starts with an interview with Professor Dame Anne Glover, who served as the Chief Scientific Advisor to the President of the European Commission. It then presents chapters on the characteristics of good elicitations by reviewing those advocated and applied; the design and development of a training course for SEJ; and on specific experiences with SEJ protocols with the intention of presenting the challenges and insights collected during these journeys.
Finally, the fourth (and largest) part begins with some reflections from Willy Aspinall on his many experiences in applying the Classical Model in several application domains; it continues with related reflections on imperfect elicitations; and then it presents chapters with applications on medicines policy and management, supply chain cyber risk management, geo-political risks, terrorism and the risks facing businesses looking to internationalise.

Table of Contents

Frontmatter
Chapter 1. Introduction and Overview of Structured Expert Judgement
Abstract
This chapter sets the background for when, and discusses the contexts in which, eliciting expert judgements is paramount. The way judgements are elicited and aggregated plays an essential part in distinguishing structured/formal elicitation protocols from informal ones. We emphasise the importance of properly reporting the steps and decision taken during an elicitation, and draw a parallel to the reporting of experimental designs underpinning the data collection. Directions for future research are proposed, and the chapter ends with an outline of the book.
Simon French, Anca M. Hanea, Tim Bedford, Gabriela F. Nane

Current Research

Frontmatter
Chapter 2. Recent Advances in the Elicitation of Uncertainty Distributions from Experts for Multinomial Probabilities
Abstract
In this chapter, we consider the problem of the elicitation and specification of an uncertainty distribution based on expert judgements, which may be a subjective prior distribution in a Bayesian analysis, for a set of probabilities which are constrained to sum to one. A typical context for this is as a prior distribution for the probabilities in a multinomial model. The Dirichlet distribution has long been advocated as a natural way to represent the uncertainty distribution over the probabilities in this context. The relatively small number of parameters allows for specification based on relatively few elicited quantities but at the expense of a very restrictive structure. We detail recent advances in elicitation for the Dirichlet distribution and recently proposed alternative approaches, which offer greater flexibility at the expense of added complexity. In order of increasing flexibility, they are the generalised Dirichlet distribution, multivariate copulas and vines. An extension of multinomial models containing covariates is discussed.
Kevin J. Wilson, Fadlalla G. Elfadaly, Paul H. Garthwaite, Jeremy E. Oakley
Chapter 3. Are Performance Weights Beneficial? Investigating the Random Expert Hypothesis
Abstract
Expert elicitation plays a prominent role in fields where the data are scarce. As consulting multiple experts is critical in expert elicitation practices, combining various expert opinions is an important topic. In the Classical Model, uncertainty distributions for the variables of interest are based on an aggregation of elicited expert percentiles. Aggregation of these expert distributions is accomplished using linear opinion pooling relying on performance-based weights that are assigned to each expert. According to the Classical Model, each expert receives a weight that is a combination of the expert’s statistical accuracy and informativeness for a set of questions, the values of which are unknown at the time the elicitation was conducted. The former measures “correspondence with reality,” a measure of discrepancy between the observed relative frequencies of seed variables’ values falling within the elicited percentile values and the expected probability based on the percentiles specified in the elicitation. The later gauges an expert’s ability to concentrate high probability mass in small interquartile intervals. Some critics argue that this performance-based model fails to outperform the models that assign experts equal weights. Their argument implies that any observed difference in expert performance is just due to random fluctuations and is not a persistent property of an expert. Experts should therefore be treated equally and equally weighted. However, if differences in experts’ performances are due to random fluctuations, then hypothetical experts created by randomly recombining the experts’ assessments should perform statistically as well as the actual experts. This hypothesis is called the random expert hypothesis. This hypothesis is investigated using 44 post-2006 professional expert elicitation studies obtained through the TU Delft database. For each study, 1000 hypothetical expert panels are simulated whose elicitations are a random mix of all expert elicitations within that study. Results indicate that actual expert statistical accuracy performance is significantly better than that of randomly created experts. The study does not consider experts’ informativeness but still provides strong support for performance-based weighting as in the Classical Model.
Deniz Marti, Thomas A. Mazzuchi, Roger M. Cooke
Chapter 4. Customized Structural Elicitation
Abstract
Expert elicitation is a powerful tool when modelling complex problems, especially in the common scenario when current probabilities are unknown and data is unavailable for certain regions of the probability space. Such methods are now widely developed, well understood, and have been used to model systems in a variety of domains including climate change, food insecurity, and nuclear risk assessment Barons et al. (2018), Rougier and Crucifix (2018), Hanea et al. (2006). However, eliciting expert probabilities faithfully has proved to be a sensitive task, particularly in multivariate settings. We argue that first eliciting structure is critical to the accuracy of the model, particularly as conducting a probability elicitation is time and resource-intensive.
Rachel L. Wilkerson, Jim Q. Smith
Chapter 5. Bayesian Modelling of Dependence Between Experts: Some Comparisons with Cooke’s Classical Model
Abstract
A Bayesian model for analysing and aggregating structured expert judgement (sej) data of the form used by Cooke’s classical model has been developed. The model has been built to create predictions over a common dataset, thereby allowing direct comparison between approaches. It deals with correlations between experts through clustering and also seeks to recalibrate judgements using the seed variables, in order to form an unbiased aggregated distribution over the target variables. Using the Delft database of sej studies, compiled by Roger Cooke, performance comparisons with the classical model demonstrate that this Bayesian approach provides similar median estimates but broader uncertainty bounds on the variables of interest. Cross-validation shows that these dynamics lead to the Bayesian model exhibiting higher statistical accuracy but lower information scores than the classical model. Comparisons of the combination scoring rule add further evidence to the robustness of the classical approach yet demonstrate outperformance of the Bayesian model in select cases.
David Hartley, Simon French
Chapter 6. Three-Point Lifetime Distribution Elicitation for Maintenance Optimization in a Bayesian Context
Abstract
A general three-point elicitation model is proposed for eliciting distributions from experts. Specifically, lower and upper quantile estimates and a most likely estimate in between these quantile estimates are to be elicited, which uniquely determine a member in a flexible family of distributions that is consistent with these estimates. Multiple expert elicited lifetime distributions in this manner are next used to arrive at the prior parameters of a Dirichlet Process (DP) describing uncertainty in a lifetime distribution. That lifetime distribution is needed in a preventive maintenance context to establish an optimal maintenance interval or a range thereof. In practical settings with an effective preventive maintenance policy, the statistical estimation of such a lifetime distribution is complicated due to a lack of failure time data despite a potential abundance of right-censored data, i.e., survival data up to the time the component was preventively maintained. Since the Bayesian paradigm is well suited to deal with scarcity of data, the formulated prior DP above is updated using all available failure time and right-censored maintenance data in a Bayesian fashion. Multiple posterior lifetime distribution estimates can be obtained from this DP update, including, e.g., its posterior expectation and median. A plausible range for the optimal time-based maintenance interval can be established graphically by plotting the long-term average cost per unit time of a block replacement model for multiple posterior lifetime distribution estimates as a function of the preventive maintenance frequency. An illustrative example is utilized throughout the paper to exemplify the proposed approach.
J. René van Dorp, Thomas A. Mazzuchi
Chapter 7. Adversarial Risk Analysis as a Decomposition Method for Structured Expert Judgement Modelling
Abstract
We argue that adversarial risk analysis may be incorporated into the structured expert judgement modelling toolkit for cases in which we need to forecast the actions of competitors based on expert knowledge. This is relevant in areas such as cybersecurity, security, defence and business competition. As a consequence, we present a structured approach to facilitate the elicitation of probabilities over the actions of other intelligent agents by decomposing them into multiple, but simpler, assessments later combined together using a rationality model of the adversary to produce a final probabilistic forecast. We then illustrate key concepts and modelling strategies of this approach to support its implementation.
David Ríos Insua, David Banks, Jesús Ríos, Jorge González-Ortega

Cooke and the Classical Model

Frontmatter
Chapter 8. A Number of Things
Abstract
This is the oration delivered in accepting the position of Professor in Applications of Decision Theory at the Faculty of Technical Mathematics and Informatics at the Delft University of Technology, Delft on November 8, 1995 by Dr. R. M. Cooke (Translated from Dutch by the author).
Roger M. Cooke
Chapter 9. The Classical Model: The Early Years
Abstract
Roger Cooke and his colleagues at the Delft University of Technology laid the foundations of the Classical Model for aggregating expert judgement in the 1980s. During 1985–1989, a research project funded by the Dutch Government saw the Classical Model developed and embedded in expert judgement procedures along with a Bayesian and a paired comparison method. That project and a subsequent working group report from the European Safety and Reliability Research and Development Association were instrumental in moving structured expert judgement procedures into the toolbox of risk analysts, particularly within Europe. As the number of applications grew, the Classical Model and its associated procedures came to dominate in applications. This chapter reflects on this early work and notes that almost all the principles and practices that underpin today’s applications were established in those early years.
Simon French
Chapter 10. An In-Depth Perspective on the Classical Model
Abstract
The Classical Model (CM) or Cooke’s method for performing Structured Expert Judgement (SEJ) is the best-known method that promotes expert performance evaluation when aggregating experts’ assessments of uncertain quantities. Assessing experts’ performance in quantifying uncertainty involves two scores in CM, the calibration score (or statistical accuracy) and the information score. The two scores combine into overall scores, which, in turn, yield weights for a performance-based aggregation of experts’ opinions. The method is fairly demanding, and therefore carrying out a SEJ elicitation with CM requires careful consideration. This chapter aims to address the methodological and practical aspects of CM into a comprehensive overview of the CM elicitation process. It complements the chapter “Elicitation in the Classical Model” in the book Elicitation (Quigley et al. 2018). Nonetheless, we regard this chapter as a stand-alone material, hence some concepts and definitions will be repeated, for the sake of completeness.
Anca M. Hanea, Gabriela F. Nane
Chapter 11. Building on Foundations: An Interview with Roger Cooke
Abstract
Prof. Roger Cooke is the Chauncey Starr Senior Fellow at Resources for Future in Washington and an emeritus professor at the Technical University of Delft in The Netherlands. This chapter presents an interview with Roger Cooke in which he reflects on the Classical Model and the processes of SEJ in conversation with Gabriela F.(Tina) Nane and Anca M. Hanea.
Gabriela F. Nane, Anca M. Hanea

Process, Procedures and Education

Frontmatter
Chapter 12. Scientific Advice: A Personal Perspective in Dealing with Uncertainty. An Interview with Prof Dame Anne Glover, in Conversation with Tim Bedford
Abstract
Prof Dame Anne Glover is former Chief Scientific Advisor to the President of the European Commission and prior to that to the Scottish Government. In this article she discusses uncertainty and scientific advice to policy makers in conversation with Prof Tim Bedford.
Anne Glover, Tim Bedford
Chapter 13. Characteristics of a Process for Subjective Probability Elicitation
Abstract
The elicitation of subjective probabilities from experts can be critical in determining a course of action when making decisions under uncertainty. A sound process to elicit probabilistic judgement is necessary to ensure that good quality data are used to inform the decision-making, as well as to provide protection to those accountable for the consequences of the determined actions. We synthesise the characteristics of a good elicitation process by critically reviewing those advocated and applied. We compare the processes inherent in the guidance produced by two professional bodies to exemplify the manner in which the characteristics manifest themselves in practice. We examine whether standardisation is meaningful given the maturity of processes for the elicitation of subjective probability.
John Quigley, Lesley Walls
Chapter 14. Developing a Training Course in Structured Expert Judgement
Abstract
The chapter discusses the design and development of a training course in structured expert judgement (SEJ). We begin by setting the course in the context of previous experiences in training SEJ to postgraduates, early career researchers and consultants. We motivate our content, discussing the theoretical framework that guides the design of such a course. We describe our experiences in presenting the course on two occasions. Detailed analysis of the different course components—the learners/participants, the content, the context and the method, was carried out through surveys given to participants. This helped identify the successful course characteristics, which were then summarised in a customised design template that can be used to guide its conceptual, structural and navigation design.
Philip Bonanno, Abigail Colson, Simon French
Chapter 15. Expert Judgement for Geological Hazards in New Zealand
Abstract
Expert judgement is important for the short- and long-term assessments of natural hazards in New Zealand, contributing to their risk analyses and informing decision-making. The problems are complex and usually require input from experts from different sub-disciplines. Expert judgement, like all human cognitive processes, is prone to biases. Therefore, we aim to use methods that are robust, transparent, reproducible and help reduce biases. The Classical Model treats expert opinion as scientific data and its performance-based weighting of experts allows us to measure the uncertainty of a quantifiable problem. We have developed a protocol for risk assessment, including structured expert judgement, which is centred around workshop-style interactions between experts to share knowledge. The protocol borrows heavily from the framework for the risk management process of the International Organization for Standardization. We outline seven recent applications of structured judgement, mostly in seismology and volcanology. Most of them use the Classical Model to aggregate the expert judgement. We discuss challenges and insights, concluding that developing an optimal protocol for expert judgement is a continuing journey.
Annemarie Christophersen, Matthew C. Gerstenberger
Chapter 16. Using the Classical Model for Source Attribution of Pathogen-Caused Illnesses
Lessons from Conducting an Ample Structured Expert Judgment Study
Abstract
A recent ample Structured Expert Judgment (SEJ) study quantified the source attribution of 33 distinct pathogens in the United States. The source attribution for five transmission pathways: food, water, animal contact, person-to-person, and environment has been considered. This chapter will detail how SEJ has been applied to answer questions of interest by discussing the process used, strengths identified, and lessons learned from designing a large SEJ study. The focus will be on the undertaken steps that have prepared the expert elicitation.
Elizabeth Beshearse, Gabriela F. Nane, Arie H. Havelaar

Applications

Frontmatter
Chapter 17. Reminiscences of a Classical Model Expert Elicitation Facilitator
Abstract
In this chapter, I trace my introduction to the Classical Model and to the thoughts and philosophy of Roger Cooke, and then go on to recount some experiences of acting as a facilitator in many real world and sometimes crucial expert elicitations. The essence of my own history is that it took me a very long time to start to understand, and appreciate, the elegance of Roger’s Classical Model (Cooke 1991), its mathematical probity and how it is best deployed in application. I am sure I still haven’t fully mastered the probability calculus entirely, and don’t doubt others might quarrel with my preferred way of conducting elicitations that rely on the Classical Model, using a plenary workshop approach. This said, I can’t find, devise or even imagine, a better alternative to the Classical Model. And, on top of this extraordinary intellectual achievement, Roger Cooke has been a beneficent collaborator nonpareil, always willing to help me, and anyone else, avoid self-inflicted elicitation and probabilistic infelicities.
Willy Aspinall
Chapter 18. Dealing with Imperfect Elicitation Results
Abstract
The trial-and-roulette method is a popular method to extract experts’ beliefs about a statistical parameter. However, most studies examining the validity of this method only use ‘perfect’ elicitation results. In practice, it is sometimes hard to obtain such neat elicitation results. In our project about predicting fraud and questionable research practices among Ph.D. candidates, we ran into issues with imperfect elicitation results. The goal of the current chapter is to provide an overview of the solutions we used for dealing with these imperfect results, so that others can benefit from our experience. We present information about the nature of our project, the reasons for the imperfect results and how we resolved these supported by annotated R-syntax.
Rens van de Schoot, Elian Griffioen, Sonja D. Winter
Chapter 19. Structured Expert Judgement for Decisions on Medicines Policy and Management
Abstract
Many decisions related to the marketing authorisation of medicinal products as well as decisions for processes such as Health Technology Assessment (HTA), reimbursement and pricing of medicines, and the setting of clinical guidelines, are taken in the face of significant uncertainties. Moreover, decision-making can be impacted by biases resulting from psychological heuristics. In other domains where decisions have to be taken with imperfect or incomplete evidence, Structured Expert Judgement (SEJ) has been found to be useful in making the best use of available evidence, and synthesising it with professional expertise, stakeholders’ values and concerns. To date, formal SEJ has only been used to a limited extent in healthcare. Aspects affecting decisions for marketing authorisation and health technology assessment, reimbursement and pricing of medicines are described and the main risks and uncertainties are identified. Some considerations and recommendations for the use of SEJ to strengthen these decisions are made.
Patricia Vella Bonanno, Alec Morton, Brian Godman
Chapter 20. Structured Expert Judgement Issues in a Supply Chain Cyber Risk Management System
Abstract
The escalation of cyberthreats is a major problem for supply chain managers with potentially enormous impacts affecting service availability and reputation, among other performance indicators. We sketch a framework and system to support supply chain cyber risk management. As data regarding impacts of cyberattacks are scarce and difficult to obtain, we describe how we acquire the required operational parameters through structured expert judgement techniques. We then describe how the whole framework is set up and implemented.
Alberto Torres-Barrán, Alberto Redondo, David Rios Insua, Jordi Domingo, Fabrizio Ruggeri
Chapter 21. Structured Expert Judgement in Adversarial Risk Assessment: An Application of the Classical Model for Assessing Geo-Political Risk in the Insurance Underwriting Industry
Abstract
For many decision and risk analysis problems, probabilistic modelling of uncertainties provides key information for decision-makers. A common challenge is lacking relevant historical data to quantify the models used in decision and risk analyses. Therefore, experts are often sought to assess uncertainties in cases of incomplete or non-existing historical data. As experts might be prone to cognitive fallacies, a structured approach to expert judgement elicitation is encouraged with the aim to mitigate such fallacies. Further, it enhances the assessment’s transparency. An area, in which the assessment and modelling of uncertainties are particularly challenging due to incomplete or non-existing historical data is adversarial risk analysis (ARA). In contrast to more traditional application areas of decision and risk modelling, in ARA intelligent adversaries add more complexity to assessing uncertainties given that their behaviour and motivations can be versatile so that they adapt and react to decision-makers’ actions, including actions based on traditional risk assessments. This often inhibits the availability of historical data. This additional complexity is also shown by the challenges that machine learning methods face when informing adversarial risk assessments. As such, using expert judgements for assessing adversarial risk (at least supplementary) often provides a more robust decision. In this chapter, we discuss the importance of structured expert judgement for ARA and present an application of the Classical Model as a structured way for eliciting uncertainty from experts on geo-political adversarial risks. We elicit the frequency of terrorist attacks and strikes, riots and civil commotions (SR & CCs), including insurgencies and civil wars, in various global regions of interest. Assessing such uncertainties is of particular interest for insurance underwriting.
Christoph Werner, Raveem Ismail
Chapter 22. Expert Judgement in Terrorism Risk Assessment
Abstract
Since 9/11, the probabilistic risk assessment of losses from terrorism has formed a quantitative basis for informed terrorism risk management. An irreducible element is the elicitation of expert judgement. In any application domain, the reliance on expert judgement can be minimized through the establishment of core conceptual principles, such as economic game theory and adversarial risk analysis, which govern the risk phenomena under consideration. For non-state threat actors, such as the Jihadi groups, Al Qaeda and ISIS, their limited logistical resources compared with western counter-terrorism intelligence and law enforcement capacity, greatly constrain the spectrum of their operations, which can be modelled quite reliably in a probabilistic manner. However, state-sponsored terrorism poses a much more severe challenge, especially in connection with the use of weapons of mass destruction, such as nuclear and chemical weapons. In this paper, the fundamental principles of terrorism risk assessment are reviewed, and the use of expert judgement is illustrated in relation to state-sponsored nuclear and chemical weapon deployment.
Gordon Woo
Chapter 23. Decision-Making in Early Internationalization: A Structured Expert Judgement Approach
Abstract
The aim of this chapter is to show how a structured approach to elicit expert judgement (SEJ) can guide the practice of early internationalization. We applied SEJ to forecast some critical issues upon which an innovative start-up wished to base their decision of whether to expand their initial operations in Poland and Czech Republic to Brazil. Sixteen participants of an Executive MBA program acted as experts and underwent the procedure for eliciting their judgements. The performance of experts was quantified in terms of statistical accuracy and informativeness, which were combined to provide a performance-based weight for each expert according to Classical Model. The combination of weighted expert judgements led to improved statistical accuracy and informativeness of the forecast. The procedure demonstrates how entrepreneurs can take advantage of expert knowledge in deciding about risky endeavours when lacking their own experiences and reliable data that can guide their choices.
Michał Zdziarski, Gabriela F. Nane, Grzegorz Król, Katarzyna Kowalczyk, Anna O. Kuźmińska
Metadata
Title
Expert Judgement in Risk and Decision Analysis
Editors
Anca M. Hanea
Gabriela F. Nane
Tim Bedford
Simon French
Copyright Year
2021
Electronic ISBN
978-3-030-46474-5
Print ISBN
978-3-030-46473-8
DOI
https://doi.org/10.1007/978-3-030-46474-5