16. The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation

This is a manifestation of the ‘paradox of expertise’ (e.g. Dror 2011), which is that experts become worse in some respects as they become more expert e.g. less flexible, creative and responsive; more biased etc. This is because knowledge and reasoning become ‘fossilized’: less amenable to inspection, change and communication.

In the Delphi procedure, ‘groups’ of experts—who never meet or interact directly, and are anonymous to each other (all to reduce sources of social bias)—are polled for their opinions. These opinions are usually point estimates or forecasts of event occurrence (see e.g. Chap. 5 for a discussion of the elicitation and evaluation of such judgments) but can also be judgments of uncertain quantities expressed as probability intervals or distributions (see Bolger et al. 2014): reasons for judgments are also often elicited. Once experts have individually expressed their opinions they are collated by a facilitator and fed back to the expert panel (most normally quantitative estimates are averaged in some manner, and qualitative responses summarized, although individual responses may also be fed back if the group is not too large). The experts are then invited to revise their opinions in the light of the feedback and resubmit them to the facilitator. The process continues through a number of iterations usually until opinion change ceases. Normally the aggregated judgements from the final round are the output although partially aggregated or disaggregated judgements can be submitted if the process fails to lead to consensus.

Although potentially non-probabilistic modes of expressing uncertainty, such as natural-language terms, could be used these have not been found to be easily converted to the probabilities usually required for policy and decision-making (see e.g. Dhami and Wallsten 2005; Wallsten and Budescu 1995).

For instance, stock price movements have been characterized as random (e.g. Fama 1965). Although more recent research suggests that stock markets are, in fact, predictable in the long term (e.g. De Bondt and Thaler 1989) it is still agreed that it is not in the short-term, contrary to the beliefs of ‘day-traders’. It may often be the case that ‘experts’ believe there to be predictability where there is not, or it is rather low. In such situations, there can, of course, be little or no expertise (see e.g. forecasting of GDP growth, Budescu and Chen 2015) nor variation in performance. Further, perceived ability where there is none is another name for ‘overconfidence’ (more generally, insensitivity to task difficulty will lead to miscalibration).

Bolger and Rowe (2015a) identify a number of problems with this approach, including finding a sufficient number of suitable seeds—ones that draw on the same expert knowledge as the target. They also comment that this ‘Classical Method’? is atheoretical, in that it is not founded on any particular conceptualization of expert knowledge, and propose a cue-based approach that would provide a reasoned basis for the selection of similar seeds (i.e. those that are related by the cues used to judge them). This cue-based approach is outlined in Sections “Cue-Based Judgements” and 16.2.4.2 below.

The other two traits being extroversion and neuroticism.

It is possible that the advantages of some personality traits are protocol-dependent. For example, conscientiousness might be good for a remotely administered elicitation such as is often the case in Delphi, while agreeableness might be particularly helpful in protocols that require face-to-face interaction. Openness to experience is probably a useful characteristic in both protocols as it should assist opinion change towards the true value.

Indeed, the ‘Theory of Errors’ (Dalkey 1975; Parenté and Anderson-Parenté 1987), which is the leading account for why the Delphi technique works, assumes those who stick are on average closer to the truth than those who shift—Bolger and Wright (2011) propose that in order to achieve ‘virtuous opinion change’—i.e. opinion change towards the truth—rationales for opinions should be fed back between Delphi rounds rather than confidence as the former will be better indicators that the expert is knowledgeable about the topic than the latter.

There is some empirical support for the suggestion that relative frequency is a more natural way of representing uncertainty than probability (Gigerenzer and Hoffrage 1995) thus posing questions as relative frequencies rather than probabilities might be an alternative to training.

In principle, many of these things could be done remotely but I am not aware of any existing protocols or software to support this.

However, Quigley and his colleagues use maps when eliciting priors with engineers assessing the reliability of new systems (e.g., Hodge et al. 2001; Walls et al. 2006—with the aerospace industry).

Support Theory also is well-developed with regard to uncertainty assessment but its primary focus is the assessment of the likelihood of categorical assessments (e.g. of event occurrences, correctness, or truth) rather than values of continuous variables.

Although as already indicated, there is still an ongoing debate as to the source and degree (and even existence) of overconfidence in such tasks—see Olsson 2014, for a recent review.

Bazerman and Moore (2008) suggest that experts also need to have coherent models (mental or formal) in order to make good quality (probability) judgements. So models may be added to learnability and ecological validity of tasks as a criterion for well-calibrated experts.

For example, ‘bootstrap’ linear models—models derived from regressing expert judgments onto the cues that they are presumed to use—make better predictions of a criterion than do the original unaided experts (e.g. Goldberg 1970) because they apply the judgment model more consistently.

In contrast to the ‘crowdsourcing’ approach mentioned above whereby a large number of people with little or no domain expertise are polled (see, e.g. Budescu and Chen 2015).

These experts will be referred to later as ‘super-experts’ since they have an overview of the problem as a whole and are responsible for recruitment, selection and management of any other experts used in the process.

This questionnaire was an adaptation of the one first developed by Wright et al. (2004), based on earlier work by Bolger and Wright (1994) and Rowe and Wright (2001): the example questions presented in Table 16.2 have been further edited by the current author since the 2014 version.

However, in some other important respects, expert selection is not like job selection i.e. you want to find people with the right skills rather than reject people with the wrong skills… so it is more akin to head hunting.

Potentially, scores for weighting could be derived from this questionnaire but this has not been attempted yet, let alone any validation of weights thus derived, therefore this is something for future research.