Decision-based Evaluation

Forecasts are generally made for a purpose. If we suppose an environment whereby agents make decisions (equivalently, select actions) based on a particular forecast, then we can evaluate that forecast in terms of its expected economic value (equivalently, expected loss), where the expectation is calculated using the actual probabilities of the states of nature. Typically, we might expect users to have different economic value (or loss) functions, so that the actions and expected losses induced by two rival sets of forecasts need not be such that each user’s expected economic value is maximized by the same set of forecasts. In Section 6.2 we show following Diebold et al. (1998)1 that only when a density forecast coincides with the true conditional density will it be optimal (in the sense of maximizing economic value) for all users regardless of their loss functions. This is a compelling reason to assess how well the forecast distribution matches the actual distribution, as in Section 5.2 — a forecast density that provides a close match to the true density can be used by all with equanimity, no matter what their individual loss functions. For decision-based evaluation in general we require the whole forecast density.