Judgment in Predictive Analytics

We describe the development of a machine learning-based object detection system for an augmented reality application. Our development is a proof of concept to successfully integrate the idea of augmented intelligence into human-operated industrial inspection settings that cannot be fully automated. While we offer some details on the design thinking approach and technical implementation of the development, our main focus will be on the processes that led to certain decisions during the development. Our analysis draws on concepts from decision-making, especially heuristics, and clusters our decisions using the concept of decision pyramids. In doing so, we analyse our decisions and decision paths and how they influenced each other. We discuss decision types we encountered, their importance and how they influenced the overall development outcome. Our discussion aims to make a contribution on the subjective nature of many decisions underpinning the development of intelligent augmentation.

In this paper, we explored how judgment can be used to improve the selection of a forecasting model. We compared the performance of judgmental model selection against a standard algorithm based on information criteria. We also examined the efficacy of a judgmental model-build approach, in which experts were asked to decide on the existence of the structural components (trend and seasonality) of the time series instead of directly selecting a model from a choice set. Our behavioral study used data from almost 700 participants, including forecasting practitioners. The results from our experiment suggest that selecting models judgmentally results in performance that is on par, if not better, to that of algorithmic selection. Further, judgmental model selection helps to avoid the worst models more frequently compared to algorithmic selection. Finally, a simple combination of the statistical and judgmental selections and judgmental aggregation significantly outperform both statistical and judgmental selections.

We study the conditions that influence judgmental forecasting effectiveness when predicting demand in the context of fashion products. Human judgment is of practical importance in this setting. Our goal is to investigate what type of decision support, in particular historical and/or contextual predictors, should be provided to human forecasters to improve their ability to detect and exploit linear and nonlinear cue-criterion relationships in the task environment. Using a field experiment on new product forecasts in the music industry, our analysis reveals that when forecasters are concerned with predictive accuracy and only managerial judgments are employed, providing both types of decision support data is beneficial. However, if judgmental forecasts are combined with a statistical forecast, restricting the decision support provided to human judges to contextual anchors is beneficial. We identify two novel interactions demonstrating that the exploitation of nonlinearities is easiest for human judgment if contextual data are present but historical data are absent. Thus, if the role of human judgment is to detect these nonlinearities (and the linearities are taken care of by some statistical model with which judgments are combined), then a restriction of the decision support provided would make sense. Implications for the theory and practice of building decision support models are discussed.

Previous empirical studies have shown that a simple combination of the formal (statistical) forecast and its judgmentally revised counterpart (expert forecast) can lead to a more accurate final forecast. However, it is argued that a further adjustment of the expert adjusted forecast would ultimately lead to a long-term change of forecasters’ behavior. Expecting that forecasters will not act hyper-rational to reach an equilibrium, the degree of their behavior change is not easy to be estimated. In this study, we try to assess the degree of this behavior change through a laboratory experiment. The adjustments of experts, with and without a 50–50% combination of system-expert forecast being occurred, are recorded and analyzed. We observe that the experts’ adjustments increase in size once they are informed that a subsequent adjustment takes place; one that essentially halves the expert adjustment. In other words, participants in our experiment try to mitigate for that further adjustment and retain the ownership of the final forecasts.

In most forecasting contexts, each target event has a resolution time point at which the “ground truth” is revealed or determined. It is reasonable to expect that as time passes, and information relevant to the event resolution accrues, the accuracy of individual forecasts will improve. For example, we expect forecasts about stock prices on a given date to be more accurate as that date approaches, or forecasts about sport tournament winners to become more accurate as the tournament progresses. This time dependence presents several issues for extracting the wisdom of crowds, and for optimizing differential weights when members of the crowd forecast the same event at different times. In this chapter, we discuss the challenges associated with this time dependence and survey the quality of the various solutions in terms of collective accuracy. To illustrate, we use data from the Hybrid Forecasting competition, where volunteer non-professional forecasters predicted multiple geopolitical events with time horizons of several weeks or months, as well as data from the European Central Bank’s Survey of Professional Forecasters which includes only a few select macroeconomic indices, but much longer time horizons (in some cases, several years). We address the problem of forecaster assessment, by showing how model-based methods may be used as an alternative to proper scoring rules for evaluating the accuracy of individual forecasters; how information aggregation can weigh concerns of forecast recency as well as sufficient crowd size; and explore the relationship between crowd size, forecast timing and aggregate accuracy. We also provide recommendations both for managers seeking to select the best analysts from the crowd, as well as aggregators looking to make the most of the overall crowd wisdom.

Despite advances in predictive analytics there is much evidence that algorithm-based forecasts are often subject to judgmental adjustments or overrides. This chapter explores the role of scenarios in supporting the role of judgment when algorithmic (or model-based) forecasts are available. Scenarios provide powerful narratives in envisioning alternative futures and play an important role in both planning for uncertainties and challenging managerial thinking. Through offering structured storylines of plausible futures, scenarios may also enhance forecasting agility and offer collaborative pathways for information sharing. Even though the potential value of using scenarios to complement judgmental forecasts has been recognized, the empirical work remains scarce. A review of the relevant research suggests the merit of supplying scenarios to judgmental forecasters is mixed and can result in an underestimation of the extent of uncertainty associated with forecasts, but a greater acceptance of model-based point predictions. These findings are generally supported by the results of a behavioral experiment that we report. This study was used to examine the effects of scenario tone and extremity on individual and group-based judgmental predictions when a model-based forecast was available. The implications of our findings are discussed with respect to (i) eliciting judgmental forecasts using different predictive formats, (ii) sharing scenarios with varying levels of optimism and pessimism, and (iii) incorporating scenario approaches to address forecast uncertainty.

Forecasting time series perturbed by external events is a difficult challenge both for statistical models and for forecasters using their judgment. External events can disturb the historical timeline significantly and add complexity. But not all external events are the same. Here, we first provide a taxonomy of external events in the context of forecasting from time series by classifying both the properties of the events themselves and the characteristics of their impacts. We then discuss research into the various ways in which judgment is used in making forecasts from time series disrupted by external events. The evidence suggests that it is generally flawed and susceptible to inconsistencies and various biases. However, there may be ways in which these problems can be minimized. We go on to discuss developments in the world of modelling and statistical forecasting in this area. There are now analytical techniques that enable disturbances caused by external events to be incorporated into time series forecasting. Some of these models are transparent: the features of the data that they extract and the way in which they are processed is made explicit. Other models, particularly those using machine learning techniques, are not transparent: the manner in which they process the data is hidden within a ‘black box’. As yet, it is not clear which of these two approaches produces more accurate forecasts. However, we suggest that transparent approaches are likely to be more acceptable to both forecasters and users of forecasts.

Forecasting researchers acknowledge that improving our understanding of forecasting’s organizational aspects could shed light on challenges such as prediction accuracy, forecasting techniques implementation, and forecast alignment between firms’ functions. However, despite the potential of an organizational research program, the literature has often maintained its emphasis on technical aspects or has approached organizational complexity from a functionalistic lens; assuming a concrete reality “out there” that is predictable and exists independently of the participants’ beliefs. Consequently, subjectivity and nuanced organizational dynamics are often disregarded as problematic behavior that needs to be extricated from the forecasting process. Within this context, this article proposes a paradigmatic shift toward a functionalist-interpretive “transition zone” where the inherent subjectivity of human organizations can be incorporated into the forecasting process to describe it more accurately and crucially, refine prescriptions. To bridge the functionalist-interpretive world views, this article brings forward the mindful organizing program, a framework that introduces a nuanced template of groups’ real-life interactions focused on collective interpretive work, the quality of organizational attention, and a particular sensitivity to analyze errors and near misses (Weick, Sensemaking in organizations. Sage, 1995; Weick et al., Research in organizational behavior, Elsevier Science/JAI Press, 1999). The incorporation of these concepts can contribute to the forecasting field from three angles: (a) substantiates the inherent subjectivity of the forecast process where actors can influence prediction outcomes, (b) offers a representation of collective judgment debiasing mechanisms and (c) emphasizes the process of collective learning via error deliberation. Under this approach, achieving forecast accuracy is less critical than unveiling collective learning mechanisms, which will eventually yield higher forecast adaptation levels in the long run.