Editorial
Group-based judgmental forecasting: An integration of extant knowledge and the development of priorities for a new research agenda

https://doi.org/10.1016/j.ijforecast.2010.05.012Get rights and content

Abstract

We review and integrate the extant knowledge on group-based forecasting, paying particular attention to the papers included in this special issue of the International Journal of Forecasting. We focus on the relative merits of different methods of aggregating individual forecasts, the advantages of heterogeneity in group memberships, the impact of others’ opinions on group members, and the importance of perceptions of trust. We conclude that a change of opinion following group-based deliberation is most likely to be appropriate where the group membership is heterogeneous, the minority opinion is protected from pressure to conform, information exchange between group members has been facilitated, and the recipient of the advice is able — by reasoning processes — to evaluate the reasoning justifying the proffered advice. Proffered advice is least likely to be accepted where the advisor is not trusted — an evaluation which is based on the advisor having different perceived values to the recipient and being thought to be self-interested. In contrast, the outcome of a group-based deliberation is most likely to be accepted when there is perceived to be procedural fairness and the participants in the process are perceived to be trustworthy. Finally, we broaden our discussion of group-based forecasting to include a consideration of other group-based methodologies which are aimed at enhancing judgment and decision making. In particular, we discuss the relevance of research on small-group decision making, the nature and quality of the advice, group-based scenario planning, and public engagement processes. From this analysis, we conclude that, for medium- to long-term judgemental forecasting, a variety of non-outcome criteria need to be considered in the evaluation of alternative group-based methods.

Introduction

Group-based forecasting can be achieved in many ways. The simplest is to average individual opinions and take the achieved figure as the forecast. Alternatively, individuals can meet to discuss the issue in groups, either with or without some formal structure to the process. Unstructured group processes have historically been typical, and are seen as a benchmark for judging the performance of structured methods. Research has demonstrated the way in which various social factors can undermine good forecasting (and decision making in general) in unstructured groups, and hence identified the need to somehow control human interactions in order to pre-empt or ameliorate these factors. Within one such structured-interaction technique — the Nominal Group Technique (NGT) — first, individuals make personal forecasts, then the group members meet to discuss the forecast problem, and finally the individuals are given the opportunity to revise their earlier forecasts (Van de Ven & Delbecq, 1974). The average of these revised opinions can be taken as the group forecast. The structure of the Delphi technique is similar to that of the NGT, with the exception that the group members exchange their initial forecasts with other group members anonymously, and are then given the opportunity to revise their individual forecasts over several Delphi rounds, with the final round average being taken as the group forecast (Rowe, Wright, & Bolger, 1991). Other group-based methods include role-playing or “simulated interaction”, where individuals simulate the interactions of groups in conflict situations in order to understand/predict how the conflict is likely to be resolved (Green, 2002, Green, 2005). Recently, prediction markets have also been studied to see whether the evolving monetary-derived predictions — produced by self-interested participants — are accurate (Berg, Nelson, & Rietz, 2008).

The structured methods vary in terms of the amount and type of information that is exchanged between group members, and also the process by which the information exchange is managed. The interaction of these aspects ultimately impacts upon the degree to which the “advice” of others is integrated with the individual group members’ own opinions. Another important feature affecting the quality of the forecasts produced by the different methods is the extent to which the group membership is relatively homogeneous or heterogeneous. Finally, some group-based methods are more acceptable to group members than others — a practical aspect that is often overlooked when choosing which forecasting method to use in any particular situation.

In this article, we integrate the contributions of the authors in this special issue of the International Journal of Forecasting toward addressing these key themes in group-based forecasting research, before summarising some general issues for future research to consider. Finally, we contextualise the current forecasting research with respect to other relevant research areas. We first consider the issue of aggregating individual forecasts.

Section snippets

Aggregating individual forecasts

In the first article, Kerr and Tindale (2011) focus on how to aggregate individual opinions in order to achieve an accurate group-based judgment. They distinguish between judgment in intellective tasks, in which the deduction of the (already-existing) truth is the focus of attention, and judgmental forecasting, in which the forecasters can only explain and defend their judgments, since the outcome has not yet occurred. This review suggests that pre-existing majority opinions generally determine

The advantages of heterogeneity in group membership

Yaniv (2011) reports an empirical study on the susceptibility to framing effects as a measure of judgment quality. Yaniv labelled groups made up of individuals who had all been assigned to the same framing manipulation within a version of Kahneman and Tversky’s classic Asian Disease problem (Kahneman & Tversky, 1984) as homogeneous. By contrast, the heterogeneous groups were made up of individuals with a mix of prior frames for a formally identical decision problem. Yaniv demonstrated that

The impact of others’ opinions

Soll and Mannes (2011) note the well-documented finding that judges often overweight their own opinions relative to advice from another (when the single advisor offers simple numerical advice) — the so called egocentric discounting. In such instances, the averaging of one’s own opinion and that of the advisor would often have led to a greater accuracy. However, as Soll and Mannes note, when the experimental task instead focuses on combining the opinions of others, the relative weights applied

The importance of perceptions of trust

In many real-world settings, there are additional complexities to conducting research, and answering the fundamental question of how to obtain a ‘good’ forecast or decision. Two studies in this special issue are noteworthy in their reporting of large-scale real-world interventions in organizations with a future-oriented focus, and these exemplify some of the research difficulties of identifying a ‘good’ forecast, as well as the definitional difficulty. That is, when one cannot easily control

Naturalistic decision making in Academia

Benda and Engels (2011) take an unusual — yet actually highly salient — perspective on group-based forecasting. They focus on the operation of the peer review process in the selection of both academic manuscripts for journal publication and grant applications for research funding, arguing that in each case, what editors/reviewers are doing is attempting to forecast the success of a paper or project (as indicated by a paper’s future citations, for example). They argue that inter-referee

Comparison of group-based forecasting methods

Graefe and Armstrong (2011) compare the accuracy of unstructured face-to-face groups with that of three structured methods: (i) nominal groups, (ii) Delphi, and (iii) prediction markets. Their task was the quantitative estimation of ten almanac quantities, such as the percentage of the US population aged over 65 years in 2000. They found few differences between the four methods overall, but all three of the structured group interaction methods improved on the group members’ individual prior

Summary of key findings in the special issue papers

From the papers discussed in the preceding paragraphs, we suggest that an individual’s opinion change following group deliberation is most likely to be appropriate where:

  • 1.

    The group membership is heterogeneous. Artificial heterogeneity can and should be achieved by role-playing rather than role-thinking.

  • 2.

    The minority opinion is protected from a majority pressure to conform — which might be achieved best through the anonymity of the participants’ judgments.

  • 3.

    Information exchange between the group

Group decision making

Schweiger, Sandberg, and Ragan (1986) discuss approaches for engendering debate and the evaluation of decisions in management teams. They differentiate between dialectical inquiry and devil’s advocacy. Both methods systematically introduce conflict and debate by using sub-groups that role-play. In dialectical inquiry, the subgroups develop opposing alternatives and then come together to debate their assumptions and recommendations. In devil’s advocacy, one subgroup offers a proposal, while the

Acceptance of advice

Expert systems capture the reasoning of experts within computer systems, and can then act to replicate the expert’s decision making. Such systems are often used by less-expert decision makers as an aid to decision making. Arnold, Clark, Collier, Leech, and Sutton (2006) found that novice users of expert systems tend to accept the systems’ recommendations, while more-expert users have a stronger interest in examining the explanations that the systems generate for particular recommendations. As

Extending past study of the reasons underpinning proffered advice

We have had a longstanding concern regarding the nature of the information exchange between participants in nominal groups performing forecasting tasks (see Rowe & Wright, 1996), and in particular, the need to understand the processes leading to judgment/forecast change in individuals within such groups (and interacting groups more widely) (Rowe et al., 1991). That is, what factors are responsible for leading participants to either accept others’ judgments/forecasts or amend their own to some

Scenario planning and stakeholder perspectives

The scenario method explores the complex relationships between social, economic, technological, environmental and political factors from multiple perspectives, enables us to make sense of their interactions, and provides a vehicle for the development of plausible futures that may impact on the focal organization.

The approach entails some consideration of stakeholder values and actions in order to add realism to already-constructed scenarios. In practice, stakeholder analysis is an optional

Scenario planning and heterogeneity of participants

In most scenario planning exercises, the scenarios are developed by participants from a single organization. It therefore follows that these participants are likely to have a homogeneous frame on the nature of the future. In practice, one way this potential bias is countered is to employ outsiders — so-called ‘remarkable people’ — who hold minority viewpoints about the future. Such deliberately-invoked diversity is likely to reduce frame blindness in the context of a facilitated process

Public engagement processes

A contemporary domain in which many of the topics in this special issue are currently being played out is that of public engagement in agenda setting and policy making. (A caveat is needed here, as many alternative terms have also been used to describe this general domain — for example, replacing the prefix ‘public’ with ‘stakeholder’ or ‘citizen’ and the suffix ‘engagement’ with ‘communication’, ‘consultation’ or participation’; for example, see Rowe & Frewer, 2005, for a discussion of

Conclusion

The papers in this issue reveal that group-based forecasting is a complex, multi-faceted issue. One of the distinctions drawn in several of the papers is between intellective and judgmental (forecasting) tasks, with some evidence being put forward that different factors may be more or less relevant in determining the outputs of groups considering each task type. In group-based forecasting practice, there are likely to be aspects of both of these types of tasks, with experts both bringing

References (40)

  • J. Landeta et al.

    People consultation to construct the future: a Delphi application

    International Journal of Forecasting

    (2011)
  • D. Onkal et al.

    Influence of differentiated roles on group forecasting accuracy

    International Journal of Forecasting

    (2011)
  • G. Rowe et al.

    The impact of task characteristics on the performance of structured group forecasting techniques

    International Journal of Forecasting

    (1996)
  • G. Rowe et al.

    The Delphi technique: a re-evaluation of research and theory

    Technological Forecasting and Social Change

    (1991)
  • J.A. Sniezek

    Groups under uncertainty: an examination of confidence in group decision making

    Organizational Behavior and Human Decision Processes

    (1992)
  • J.B. Soll et al.

    Judgmental aggregation strategies depend on whether the self is involved

    International Journal of Forecasting

    (2011)
  • L. Van Swol

    Forecasting another’s enjoyment versus giving the right answer: trust, shared values, task effects, and confidence in improving the acceptance of advice

    International Journal of Forecasting

    (2011)
  • G. Wright

    Game theory, game theorists, university students, role-playing and forecasting ability

    International Journal of Forecasting

    (2002)
  • G. Wright et al.

    Decision making and planning under low levels of predictability: enhancing the scenario method

    International Journal of Forecasting

    (2009)
  • I. Yaniv

    Group diversity and decision quality: amplification and attenuation of the framing effect

    International Journal of Forecasting

    (2011)
  • Cited by (28)

    • Using scenarios to forecast outcomes of a refugee crisis

      2022, International Journal of Forecasting
    • When experts disagree: Using the Policy Delphi method to analyse divergent expert expectations and preferences on UK energy futures

      2020, Technological Forecasting and Social Change
      Citation Excerpt :

      For both topics, only a minority of the participants changed their views between rounds. While this may in part be due to participant groups lacking trust in the context of anonymised surveys (Wright and Rowe, 2011), the comments reveal a range of explanations, including diverging parametric uncertainty prioritisation, and diverging responses to structural features such as the appropriate level of agency – differences which are underpinned by value differences. Finally, while there were some instances of ‘epistemic challenge’ between participants which could provoke debate and encourage learning (Rip, 2003; Kattirtzi, 2016), these were rare, and overall the findings suggest limits to the potential for expert elicitation methods such as Policy Delphi to resolve policy disputes on contested topics.

    • Coordinating judgmental forecasting: Coping with intentional biases

      2019, Omega (United Kingdom)
      Citation Excerpt :

      Documented examples of intentional biases include forecasters who inflate forecasts to ensure that suppliers give them priority [79] or to increase the publicity of the forecast [5]. Wright and Rowe [88, p. 12] conclude that “[w]e must always remember that forecasts are rarely, in themselves, disinterested and innocent products of the group process in which they are produced and this reality should cause us to reconsider the way in which we evaluate forecasts.” This particularly holds for the S&OP process – a collaborative, essentially cross-functional decision-making process, in which representatives from various departments in the organization, ranging from sales to operations, and finance, jointly generate a corporate forecast and a joint planning in a physical meeting [44,48,61,62].

    • The human factor in supply chain forecasting: A systematic review

      2019, European Journal of Operational Research
      Citation Excerpt :

      Group-based forecasting can occur in many ways with numerous techniques widely available and utlized in industry. However, the simplest and often most practical technique is to average the individual forecasts of group members and use this figure as the final forecast (Wright & Rowe, 2011). A number of studies laud the Delphi technique as being a very versatile group forecasting approach which tends to yield higher forecasting accuracies compared to other group forecasting techniques such as dictator, consensus, and dialectic techniques (Bolger & Wright, 2011; Chang & Wang, 2006; Lin & Song, 2015; Rowe & Wright, 1996, 1999).

    • Forecasts or fortune-telling: When are expert judgements of safety risk valid?

      2017, Safety Science
      Citation Excerpt :

      Some approaches use both mechanisms in sequential combination, with individual forecasts followed by group discussion, group discussion followed by individual forecasts, or multiple rounds of forecast and discussion. For group-based decision making to improve forecast accuracy, it is necessary that some members will, after interacting with the rest of the group, update their original forecast in the right direction (Wright and Rowe, 2011). There are certain types of problems, known as “intellective” or “Eureka” problems, where the right answer is obvious once it is known (Laughlin and Ellis, 1986).

    View all citing articles on Scopus
    View full text