EditorialGroup-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda
Introduction
Group-based forecasting can be achieved in many ways. The simplest is to average individual opinions and take the achieved figure as the forecast. Alternatively, individuals can meet to discuss the issue in groups, either with or without some formal structure to the process. Unstructured group processes have historically been typical, and are seen as a benchmark for judging the performance of structured methods. Research has demonstrated the way in which various social factors can undermine good forecasting (and decision making in general) in unstructured groups, and hence identified the need to somehow control human interactions in order to pre-empt or ameliorate these factors. Within one such structured-interaction technique — the Nominal Group Technique (NGT) — first, individuals make personal forecasts, then the group members meet to discuss the forecast problem, and finally the individuals are given the opportunity to revise their earlier forecasts (Van de Ven & Delbecq, 1974). The average of these revised opinions can be taken as the group forecast. The structure of the Delphi technique is similar to that of the NGT, with the exception that the group members exchange their initial forecasts with other group members anonymously, and are then given the opportunity to revise their individual forecasts over several Delphi rounds, with the final round average being taken as the group forecast (Rowe, Wright, & Bolger, 1991). Other group-based methods include role-playing or “simulated interaction”, where individuals simulate the interactions of groups in conflict situations in order to understand/predict how the conflict is likely to be resolved (Green, 2002, Green, 2005). Recently, prediction markets have also been studied to see whether the evolving monetary-derived predictions — produced by self-interested participants — are accurate (Berg, Nelson, & Rietz, 2008).
The structured methods vary in terms of the amount and type of information that is exchanged between group members, and also the process by which the information exchange is managed. The interaction of these aspects ultimately impacts upon the degree to which the “advice” of others is integrated with the individual group members’ own opinions. Another important feature affecting the quality of the forecasts produced by the different methods is the extent to which the group membership is relatively homogeneous or heterogeneous. Finally, some group-based methods are more acceptable to group members than others — a practical aspect that is often overlooked when choosing which forecasting method to use in any particular situation.
In this article, we integrate the contributions of the authors in this special issue of the International Journal of Forecasting toward addressing these key themes in group-based forecasting research, before summarising some general issues for future research to consider. Finally, we contextualise the current forecasting research with respect to other relevant research areas. We first consider the issue of aggregating individual forecasts.
Section snippets
Aggregating individual forecasts
In the first article, Kerr and Tindale (2011) focus on how to aggregate individual opinions in order to achieve an accurate group-based judgment. They distinguish between judgment in intellective tasks, in which the deduction of the (already-existing) truth is the focus of attention, and judgmental forecasting, in which the forecasters can only explain and defend their judgments, since the outcome has not yet occurred. This review suggests that pre-existing majority opinions generally determine
The advantages of heterogeneity in group membership
Yaniv (2011) reports an empirical study on the susceptibility to framing effects as a measure of judgment quality. Yaniv labelled groups made up of individuals who had all been assigned to the same framing manipulation within a version of Kahneman and Tversky’s classic Asian Disease problem (Kahneman & Tversky, 1984) as homogeneous. By contrast, the heterogeneous groups were made up of individuals with a mix of prior frames for a formally identical decision problem. Yaniv demonstrated that
The impact of others’ opinions
Soll and Mannes (2011) note the well-documented finding that judges often overweight their own opinions relative to advice from another (when the single advisor offers simple numerical advice) — the so called egocentric discounting. In such instances, the averaging of one’s own opinion and that of the advisor would often have led to a greater accuracy. However, as Soll and Mannes note, when the experimental task instead focuses on combining the opinions of others, the relative weights applied
The importance of perceptions of trust
In many real-world settings, there are additional complexities to conducting research, and answering the fundamental question of how to obtain a ‘good’ forecast or decision. Two studies in this special issue are noteworthy in their reporting of large-scale real-world interventions in organizations with a future-oriented focus, and these exemplify some of the research difficulties of identifying a ‘good’ forecast, as well as the definitional difficulty. That is, when one cannot easily control
Naturalistic decision making in Academia
Benda and Engels (2011) take an unusual — yet actually highly salient — perspective on group-based forecasting. They focus on the operation of the peer review process in the selection of both academic manuscripts for journal publication and grant applications for research funding, arguing that in each case, what editors/reviewers are doing is attempting to forecast the success of a paper or project (as indicated by a paper’s future citations, for example). They argue that inter-referee
Comparison of group-based forecasting methods
Graefe and Armstrong (2011) compare the accuracy of unstructured face-to-face groups with that of three structured methods: (i) nominal groups, (ii) Delphi, and (iii) prediction markets. Their task was the quantitative estimation of ten almanac quantities, such as the percentage of the US population aged over 65 years in 2000. They found few differences between the four methods overall, but all three of the structured group interaction methods improved on the group members’ individual prior
Summary of key findings in the special issue papers
From the papers discussed in the preceding paragraphs, we suggest that an individual’s opinion change following group deliberation is most likely to be appropriate where:
- 1.
The group membership is heterogeneous. Artificial heterogeneity can and should be achieved by role-playing rather than role-thinking.
- 2.
The minority opinion is protected from a majority pressure to conform — which might be achieved best through the anonymity of the participants’ judgments.
- 3.
Information exchange between the group
Group decision making
Schweiger, Sandberg, and Ragan (1986) discuss approaches for engendering debate and the evaluation of decisions in management teams. They differentiate between dialectical inquiry and devil’s advocacy. Both methods systematically introduce conflict and debate by using sub-groups that role-play. In dialectical inquiry, the subgroups develop opposing alternatives and then come together to debate their assumptions and recommendations. In devil’s advocacy, one subgroup offers a proposal, while the
Acceptance of advice
Expert systems capture the reasoning of experts within computer systems, and can then act to replicate the expert’s decision making. Such systems are often used by less-expert decision makers as an aid to decision making. Arnold, Clark, Collier, Leech, and Sutton (2006) found that novice users of expert systems tend to accept the systems’ recommendations, while more-expert users have a stronger interest in examining the explanations that the systems generate for particular recommendations. As
Extending past study of the reasons underpinning proffered advice
We have had a longstanding concern regarding the nature of the information exchange between participants in nominal groups performing forecasting tasks (see Rowe & Wright, 1996), and in particular, the need to understand the processes leading to judgment/forecast change in individuals within such groups (and interacting groups more widely) (Rowe et al., 1991). That is, what factors are responsible for leading participants to either accept others’ judgments/forecasts or amend their own to some
Scenario planning and stakeholder perspectives
The scenario method explores the complex relationships between social, economic, technological, environmental and political factors from multiple perspectives, enables us to make sense of their interactions, and provides a vehicle for the development of plausible futures that may impact on the focal organization.
The approach entails some consideration of stakeholder values and actions in order to add realism to already-constructed scenarios. In practice, stakeholder analysis is an optional
Scenario planning and heterogeneity of participants
In most scenario planning exercises, the scenarios are developed by participants from a single organization. It therefore follows that these participants are likely to have a homogeneous frame on the nature of the future. In practice, one way this potential bias is countered is to employ outsiders — so-called ‘remarkable people’ — who hold minority viewpoints about the future. Such deliberately-invoked diversity is likely to reduce frame blindness in the context of a facilitated process
Public engagement processes
A contemporary domain in which many of the topics in this special issue are currently being played out is that of public engagement in agenda setting and policy making. (A caveat is needed here, as many alternative terms have also been used to describe this general domain — for example, replacing the prefix ‘public’ with ‘stakeholder’ or ‘citizen’ and the suffix ‘engagement’ with ‘communication’, ‘consultation’ or participation’; for example, see Rowe & Frewer, 2005, for a discussion of
Conclusion
The papers in this issue reveal that group-based forecasting is a complex, multi-faceted issue. One of the distinctions drawn in several of the papers is between intellective and judgmental (forecasting) tasks, with some evidence being put forward that different factors may be more or less relevant in determining the outputs of groups considering each task type. In group-based forecasting practice, there are likely to be aspects of both of these types of tasks, with experts both bringing
References (40)
- et al.
The predictive validity of peer review: a selective review of the judgmental forecasting qualities of peers, and implications for innovation in science
International Journal of Forecasting
(2011) - et al.
Prediction market accuracy in the long run
International Journal of Forecasting
(2008) - et al.
Risk management and governance: a post-normal science approach
Futures
(1999) - et al.
Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task
International Journal of Forecasting
(2011) Game theory, simulated interaction, and unaided judgement for forecasting decisions in conflicts: further evidence
International Journal of Forecasting
(2005)- et al.
Role thinking: standing in other people’s shoes to forecast decisions in conflicts
International Journal of Forecasting
(2011) - et al.
The limits of forecasting in anticipating rare events
Technological Forecasting and Social Change
(2010) - et al.
Forecasting clients’ reactions: how does perception of strategic behaviour influence acceptance of advice?
International Journal of Forecasting
(2011) - et al.
Group-based forecasting?: a social psychological analysis
International Journal of Forecasting
(2011) - et al.
A virtual and anonymous, deliberative and analytic participation process for planning and evaluation: the concept mapping policy Delphi
International Journal of Forecasting
(2011)
People consultation to construct the future: a Delphi application
International Journal of Forecasting
Influence of differentiated roles on group forecasting accuracy
International Journal of Forecasting
The impact of task characteristics on the performance of structured group forecasting techniques
International Journal of Forecasting
The Delphi technique: a re-evaluation of research and theory
Technological Forecasting and Social Change
Groups under uncertainty: an examination of confidence in group decision making
Organizational Behavior and Human Decision Processes
Judgmental aggregation strategies depend on whether the self is involved
International Journal of Forecasting
Forecasting another’s enjoyment versus giving the right answer: trust, shared values, task effects, and confidence in improving the acceptance of advice
International Journal of Forecasting
Game theory, game theorists, university students, role-playing and forecasting ability
International Journal of Forecasting
Decision making and planning under low levels of predictability: enhancing the scenario method
International Journal of Forecasting
Group diversity and decision quality: amplification and attenuation of the framing effect
International Journal of Forecasting
Cited by (28)
Using scenarios to forecast outcomes of a refugee crisis
2022, International Journal of ForecastingWhen experts disagree: Using the Policy Delphi method to analyse divergent expert expectations and preferences on UK energy futures
2020, Technological Forecasting and Social ChangeCitation Excerpt :For both topics, only a minority of the participants changed their views between rounds. While this may in part be due to participant groups lacking trust in the context of anonymised surveys (Wright and Rowe, 2011), the comments reveal a range of explanations, including diverging parametric uncertainty prioritisation, and diverging responses to structural features such as the appropriate level of agency – differences which are underpinned by value differences. Finally, while there were some instances of ‘epistemic challenge’ between participants which could provoke debate and encourage learning (Rip, 2003; Kattirtzi, 2016), these were rare, and overall the findings suggest limits to the potential for expert elicitation methods such as Policy Delphi to resolve policy disputes on contested topics.
Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process
2019, Technological Forecasting and Social ChangeCoordinating judgmental forecasting: Coping with intentional biases
2019, Omega (United Kingdom)Citation Excerpt :Documented examples of intentional biases include forecasters who inflate forecasts to ensure that suppliers give them priority [79] or to increase the publicity of the forecast [5]. Wright and Rowe [88, p. 12] conclude that “[w]e must always remember that forecasts are rarely, in themselves, disinterested and innocent products of the group process in which they are produced and this reality should cause us to reconsider the way in which we evaluate forecasts.” This particularly holds for the S&OP process – a collaborative, essentially cross-functional decision-making process, in which representatives from various departments in the organization, ranging from sales to operations, and finance, jointly generate a corporate forecast and a joint planning in a physical meeting [44,48,61,62].
The human factor in supply chain forecasting: A systematic review
2019, European Journal of Operational ResearchCitation Excerpt :Group-based forecasting can occur in many ways with numerous techniques widely available and utlized in industry. However, the simplest and often most practical technique is to average the individual forecasts of group members and use this figure as the final forecast (Wright & Rowe, 2011). A number of studies laud the Delphi technique as being a very versatile group forecasting approach which tends to yield higher forecasting accuracies compared to other group forecasting techniques such as dictator, consensus, and dialectic techniques (Bolger & Wright, 2011; Chang & Wang, 2006; Lin & Song, 2015; Rowe & Wright, 1996, 1999).
Forecasts or fortune-telling: When are expert judgements of safety risk valid?
2017, Safety ScienceCitation Excerpt :Some approaches use both mechanisms in sequential combination, with individual forecasts followed by group discussion, group discussion followed by individual forecasts, or multiple rounds of forecast and discussion. For group-based decision making to improve forecast accuracy, it is necessary that some members will, after interacting with the rest of the group, update their original forecast in the right direction (Wright and Rowe, 2011). There are certain types of problems, known as “intellective” or “Eureka” problems, where the right answer is obvious once it is known (Laughlin and Ellis, 1986).