Evaluating Econometric Forecasts of Economic and Financial Variables

By a forecast will be meant any statement about ‘the future’, where the future is relative to the analyst’s viewpoint. So as well as the common sense notion of a forecast of what will happen tomorrow, or next Saturday, the term will equally apply to the outcome of the 1997 General Election made now but based on what was known at the end of 1996, for example. Forecasts are often constructed ex post as a way of evaluating a particular forecasting model or forecasting device, presumably with the hope that the past forecast performance of the model will serve as a useful guide to how well it might forecast in the future. In any event, forecasting the past as the ‘relative future’ means that forecasts can be evaluated as they are made, without having to wait to see what actually happens tomorrow, or on the coming Saturday, and a large sample of forecasts can be generated (with associated outcomes available), which might allow a statistical analysis of the forecast performance of the model. My forecast of rain might turn out to be wrong, but that might just be bad luck. Suppose my forecasting model is that I forecast rain in the afternoon if at 11 a.m. in the morning the cows in a certain field are lying down. Given daily observations on afternoon rainfall and the morning stance of cows over the last year, one could devise a statistical test of whether my forecasting model was a good predictor of meteorological conditions.

Sections 2.1 and 2.2 consider the evaluation of sequences of point forecasts in terms of the first- and second-moment properties of the forecast errors. Section 2.3 allows that there is at least one rival set of forecasts of the variable of interest, and asks which of the two is better, as well as whether even the less good of the two provides some useful information. In Section 2.4 we explicitly allow that the forecasts have been generated by models. At this point, the question becomes not which of the sets of forecasts is best, but which of the models generates more accurate forecasts, as judged by out-of-sample tests of predictive ability. Section 2.5 considers a number of issues that arise in the evaluation of forecasts from non-linear models.

Forecasting the conditional variance of a process is primarily of interest if the conditional variance is changing over time.¹ For a large number of financial time-series, as well as some macroeconomic time-series (such as inflation), time-varying conditional variances are an important feature. The autoregressive conditional heteroskedasticity (ARCH) model of Engle (1982), and its generalizations,² have become almost indispensable in the modelling of financial series. ARCH models are capable of capturing variances that change (giving rise to clusterings of large (small) changes in the series), as well as other features typical of many financial series, such as thick-tailed unconditional distributions. As an example, Figure 3.1 plots monthly observations on three-month US Treasury Bill interest rates and ten-year Treasury bond interest rates (taken from the Federal Reserve of St Louis database, http://​www.​stls.​frb.​org/fred) and the first differences of these series. The clustering of large and small changes is clearly evident.

In recent years there has been considerable interest in density forecasts. This has been fuelled by the rapidly expanding field of financial risk management, as well as by the literature on inflation forecasting.¹ For example, if the goal is to achieve an inflation rate in a certain range or target band, a point forecast of inflation is of limited value. A histogram (or density forecast) that assigns probabilities to inflation falling in certain intervals will be more informative about the likelihood of the target being met. That said, the forecast histogram will only be of value to the extent that the forecast probabilities accurately capture the true probabilities. As in previous chapters, our focus will be on evaluation, where the forecasts are now densities, or histograms, or probability distributions.

Forecasts are generally made for a purpose. If we suppose an environment whereby agents make decisions (equivalently, select actions) based on a particular forecast, then we can evaluate that forecast in terms of its expected economic value (equivalently, expected loss), where the expectation is calculated using the actual probabilities of the states of nature. Typically, we might expect users to have different economic value (or loss) functions, so that the actions and expected losses induced by two rival sets of forecasts need not be such that each user’s expected economic value is maximized by the same set of forecasts. In Section 6.2 we show following Diebold et al. (1998)¹ that only when a density forecast coincides with the true conditional density will it be optimal (in the sense of maximizing economic value) for all users regardless of their loss functions. This is a compelling reason to assess how well the forecast distribution matches the actual distribution, as in Section 5.2 — a forecast density that provides a close match to the true density can be used by all with equanimity, no matter what their individual loss functions. For decision-based evaluation in general we require the whole forecast density.

In recent years, forecasts that give a more complete description of the likely future values of economic and financial variables than the expected mean have become increasingly prominent, in academia as well as in government and financial regulation. The focus of this book has been on the evaluation of these forecasts. A number of the issues relevant to the evaluation of point forecasts are equally germane to the evaluation of interval and density forecasts. There are also new problems to be overcome, such as the fact that volatility is unobserved, when forecasts of conditional variance are evaluated.

Some sample code is given and described. The data sets and programs can be downloaded from the Palgrave Macmillan web page http://​www.​palgrave.​com/economics/Clements/index.asp. The Gauss code given below is for illustrative purposes. It is not meant to illustrate good programming technique, or to be especially general. It is hoped that the sample code might encourage the reader to experiment, and some suggestions are given which may be taken up as simple exercises.