Setting accuracy targets for short-term judgemental sales forecasting
Introduction
This research was motivated by an organisation that wished to implement a quality initiative throughout its production and inventory management. Like many companies, this organisation sought to encourage a quality culture in their operations by the setting of targets for a number of key measurable activities (Juran & Gryna, 1993). However, one important activity that presents special problems for such quality target-setting is short-term sales forecasting. This paper addresses this problem, using the company as a case study.
Benchmarking against industry leaders, and top performing companies in similar functional areas in other industries, is worthwhile for target-setting in many instances of total quality management (Hradesky, 1995). However, cross-company comparisons have not generally been relevant, nor feasible, in the area of setting forecasting quality goals. Company specific and company sensitive market issues often preclude this.
Furthermore, when we look at the research literature on forecasting, it is evident that the focus is more upon models than processes, and that the quality of a forecasting model tends to be judged by how it compares, in terms of accuracy, to a reasonable alternative statistical model. However, the value of such a comparison clearly depends on the quality of the benchmark model. Moreover, as research has shown that the fit of a model to historical data is not always a good guide to the post-sample accuracy of the model (Makridakis, 1986, Pant & Starbuck, 1990), forecasters have been advised to judge accuracy based on post-sample prediction error. Thus, we have seen many published studies deriving the post-sample forecast errors from a variety of statistical models (such as the M-Competition, Makridakis et al., 1982). However, to the extent that the process of most business forecasting in practice involves considerable well-informed judgemental adjustments to simple time series methods, or may indeed be mostly judgemental, this research is therefore quite limited for the task of quality target-setting.
Indeed, it is clear that in circumstances where judgemental inputs are of proven value in forecasting, the usefulness of statistical model benchmarking is, at best, to provide lower quality bounds on performance. This, therefore, still leaves open the issue of assessing upper bounds which are theoretically feasible, but strongly challenging and can thereby provide a viable motivation for managerial forecasters.
To address this, we present a methodological framework which considers the forecast error associated with a prediction to consist of two components: the irreducible error due to the intrinsic unpredictable uncertainty in the variable, and the error due to less than perfect modelling and estimation. The intrinsic uncertainty clearly presents a bound on the accuracy of the forecasting process. Hence, our derivation of an upper bound is based on the estimation of this irreducible component of uncertainty in the data. The analogy is with the study of physical systems, where observation noise can be seen as an upper limit to the accuracy of systematic measurements. This concept has been extended in forecasting research. For example, Bunn and Seigal (1983) found that there was an upper bound on the accuracy of minute-by-minute electricity load forecasts due to load measurement problems and used this as a basis for assessing the performance of various short-term predictors. Compared to a measure of ex post accuracy, which evaluates the forecast against the actual out-turn, the proposed quality target is clearly more reasonable, but is still an idealised upper bound on performance.
Measures of actual ex post accuracy are, of course, essential for monitoring and, as we have observed, simple model based comparisons can provide a reasonable lower bound on performance. It would seem reasonable, therefore, to evaluate the usefulness of quality target bounds where the upper ones are based upon estimates of irreducible uncertainty, and the lower ones are derived from a simple time series model (e.g. random walk).
We applied this approach to the quality initiatives of our collaborating company. This company operates worldwide in the fast changing, high-technology sector, selling a range of personal computers directly to consumers, either by telephone or internet. They hold no inventories of finished goods, just component parts, and assemble to order. The products have quite short life cycles, with sales very dependent upon pricing and advertising. Their forecasts are mostly judgemental estimates, using sales force knowledge plus market information on product innovations and promotions, against a background of daily monitoring of underlying sales trends per product line. In this respect the case study described here is typical of a more general class of consumer forecasting problems where there is high frequency data (e.g. EPOS, internet or phone), and a necessarily substantial judgemental component.
The hypothesis of this study is that, in the spirit of TQM, a forecast quality target, implemented with regular monthly feedback, will motivate and monitor improved forecasting throughout the company and that such a target could consist of two bounds. The upper bound could be an estimator which forecasts with error due only to intrinsic uncertainty, whilst the lower bound could be a naı̈ve statistical method based upon a random walk. It is important to understand that this paper is not concerned with the problem of estimating prediction intervals. A prediction interval conveys the interval within which an actual out-turn is likely to fall with a given probability, such as 95%. A quality target is a measure that one must try to attain. Since forecast quality is assessed by accuracy measures, such as MSE, this paper aims to provide a methodology for deriving a value for the measure which would serve as a quality target.
In Section 2, we consider the literature on forecast accuracy measures, in order to provide an appropriate metric for these bounds. We present the company’s own metric and then, in Section 3, we present a framework for deriving bounds on forecast accuracy. Section 4 discusses the limitations of an analytical approach to assessing the accuracy bounds, whilst Section 5 presents an alternative approach, which uses Monte Carlo simulation. Section 6 reports the application of the simulation procedure to the company’s data and the final section offers some concluding comments.
Section snippets
Forecast accuracy measures
In a survey of practitioners and academicians, Carbone and Armstrong (1982) found that the most preferred measure of forecast accuracy is the Mean Square Error (MSE). Chatfield (1992) writes that, for a single series, it is perfectly reasonable to fit a model by least squares and evaluate forecasts from different models by the MSE. However, the MSE has been broadly criticised for use in comparing forecasting methods across series, as it can be disastrous to average MSE from different series
The components of forecast error
Having chosen an accuracy metric as a measure of forecast quality, we now address the problem of deriving target values for the metric. The way that we approach the problem is to estimate the limits on the accuracy of the company’s sales forecasts. Our methodological framework considers the forecast error associated with a prediction as consisting of two components: the irreducible error, et, due to the intrinsic unpredictable uncertainty in the variable, and the error, εt, due to less than
Analytical approach to assessing accuracy bounds for SP
We can work towards a bound on accuracy by considering the limit on the forecasting performance of an ideal predictor. The forecasts, pt, of an ideal predictor have εt=0, so that:If we assume that et is normally distributed, we can say that with 5% probability, et will fall outside the interval [−1.96σt, 1.96σt]. Using this, and recalling the definition of the similarity percentage (SP) given in expression (1), we can make the following probability statements for the forecast of the
Simulation approach to assessing accuracy targets for SP and WSP
Upper bounds may be derived for the accuracy measure by simulating the actual sales, xit, for product i in month t. We proceed by considering the observed actual as being a random variable consisting of a non-stochastic expectation component, E(xit), plus an intrinsic error term, eit. Having estimated the standard deviation, σit, of eit, we are then in a position to simulate values of xit as:where eit is a value derived by Monte Carlo sampling from a normal distribution with zero
Case study results
We had data for 11 months. We used 1000 iterations in the Monte Carlo simulations. In other words, we produced a thousand simulated actuals from which we calculated the SP and WSP for the ideal predictor and for the random walk. The 5th percentiles of the resultant distributions were then used as upper and lower bounds, respectively, on accuracy. The upper bound can serve as a target for forecast quality.
The company groups all of its products into one of three different categories. The weighted
Concluding comments
Based upon the assumptions of intrinsic randomness, we have presented a simulation framework for deriving an upper and lower bound for the weighted accuracy measure. In the actual case study, the approach assumed that there is unforecastable week-by-week variation within each month, but that the average, from which these weeks are statistical outcomes, is predictable.
The upper bound was computed at the 95% confidence level, in the sense that, with ideal forecasting of the monthly means, the
Acknowledgements
We would like to acknowledge the helpful comments of an associate editor and two anonymous referees.
Biographies: Derek W. BUNN is Professor and Chairman of Decision Sciences at London Business School. He is also editor of the Journal of Forecasting and, in addition to his interest in judgemental aspects of forecasting, he is actively involved in applications to the energy sector.
References (15)
- et al.
Error measures for generalising about forecasting methods: empirical comparisons
International Journal of Forecasting
(1992) A commentary on error measures
International Journal of Forecasting
(1992)- et al.
Stock market volatility and the informational content of stock index options
Journal of Econometrics
(1992) The evaluation of extrapolative forecasting methods
International Journal of Forecasting
(1992)- et al.
On the asymmetry of the symmetric MAPE
International Journal of Forecasting
(1999) The art and science of forecasting: an assessment and future directions
International Journal of Forecasting
(1986)Accuracy measures: theoretical and practical concerns
International Journal of Forecasting
(1993)
Cited by (11)
On revenue management and the use of occupancy forecasting error measures
2014, International Journal of Hospitality ManagementShort-term sales forecasting with change-point evaluation and pattern matching algorithms
2012, Expert Systems with ApplicationsCitation Excerpt :He studies both (a) historical forecasting methods (simple averages and weighted averages) and (b) pickup-based forecasting methods (classical and advanced pickup methods) and finds that in general, pickup-based forecasting methods provide the most accurate forecasts of the two groups he has studied. Bunn and Taylor (2001) propose an approach to consider forecast error as consisting of irreducible error due to intrinsic unpredictable uncertainty, and error due to less than perfect modelling, estimation and forecasting. The case-study proposed was taken from the short-term sales forecasting process of a major, international high-technology manufacturer.
The design features of forecasting support systems and their effectiveness
2006, Decision Support SystemsJudgmental forecasting in the presence of loss functions
2005, International Journal of ForecastingCitation Excerpt :The above examples suggest that the loss function encountered by a user of a forecast is likely to affect the process of forecasting. While there has been considerable debate on appropriate metrics to be used in measuring accuracy in forecasting (Armstrong & Collopy, 1992; Bunn & Taylor, 2001), little debate has occurred as to the shape of common loss functions and the effect of this shape on the process of forecasting. In a perfect world the minimization of error (i.e. a zero error) will also minimize cost and maximize benefit.
The Use of Forecast Accuracy Indicators to Improve Planning Quality: Insights from a Case Study
2020, European Accounting Review
Biographies: Derek W. BUNN is Professor and Chairman of Decision Sciences at London Business School. He is also editor of the Journal of Forecasting and, in addition to his interest in judgemental aspects of forecasting, he is actively involved in applications to the energy sector.
James W. TAYLOR is a lecturer at the Saı̈d Business School, University of Oxford. His research interests include prediction intervals, quantile regression, combining forecasts, exponential smoothing, volatility forecasting and electricity demand forecasting.