Elicitability and identifiability of set-valued measures of systemic risk

In the financial mathematics literature, there is a great interest in various types of risk and in particular its quantitative measurement. The axiomatic approach to the quantitative assessment of risk was pioneered by Artzner et al. [6] and has since then been discussed from various angles in many further works; see Föllmer and Schied [33, Chap. 4] for a comprehensive overview.

The financial crisis of 2007 – 2009 and its aftermaths in the last decade have starkly underpinned the need to quantitatively assess the risk of an entire financial system rather than merely its individual entities. One of the first academic works on systemic risk is the seminal paper by Eisenberg and Noe [20]. The focus of this work, however, lies on modelling the financial system rather than measuring its systemic risk. Since then, financial mathematicians have developed a rich strand of literature, encompassing different approaches and emphasising various aspects of systemic risk. The model of [20] has been generalised in different ways, for instance by considering illiquidity (Rogers and Veraart [57]) or central clearing (Amini et al. [4]). One strand of literature defines systemic risk measures by applying a scalar risk measure to the distribution of the total profits and losses of all firms in the system (Acharya et al. [2], Adrian and Brunnermeier [3]). Recognising the drawbacks of treating the economy as a portfolio, Chen et al. [13] introduce an axiomatic approach to measuring systemic risk, further extended by Kromer et al. [48] and Hoffmann et al. [40]. The axiomatic approach of [13] is widely used and amounts to systemic risk measures of the form \(\rho (\Lambda (Y))\), where \(Y\) is a \(d\)-dimensional random vector representing the financial system, \(\rho \) is a scalar risk measure and \(\Lambda \colon \mathbb{R}^{d}\to \mathbb{R}\) an increasing aggregation function. However, this approach has the drawback that it results in the measurement of bailout costs rather than capital requirements that prevent a financial crisis. These types of risk measures are also called insensitive as they do not take into account the impact of capital allocations on the system.

As an alternative, so-called sensitive systemic risk measures have been introduced by Feinstein et al. [23]; see also Biagini et al. [11] and Armenti et al. [5] for related approaches. Here, one first adds capital to the \(d\) financial institutions and then applies an aggregation function, resulting in systemic risk measures of the form Thus one takes into account how the regulation changes the system itself as it aggregates the system \(Y+k\) after regulation. In this paper, we mainly focus on this type of systemic risk measures; see Sect. 2.1 for more details. Above, \(R(Y)\) specifies the set of all capital allocations \(k\in \mathbb{R}^{d}\) such that the new system \(Y+k\) is deemed acceptable with respect to \(\rho \) after being aggregated via \(\Lambda \). As such, \(R\) takes an ex ante perspective prescribing the injections to (and withdrawals from) each financial firm adequate to prevent the system \(Y\) from a crisis, whereas \(\rho (\Lambda (Y))\), as described above, can be interpreted as the bailout costs of the system after a systemic event has occurred.

The field of quantitative risk management has seen a lively debate about which scalar risk measure is most appropriate in practice; see Embrechts et al. [21], Emmer et al. [22] for detailed academic discussions and [8] for a regulatory perspective in banking. Besides differences in axiomatic properties such as coherence [6] and convexity [32] of risk measures, the debate has also considered more statistical aspects of risk measures. The two most widely discussed statistical desiderata are robustness in the sense of Hampel [39] (see also Cont et al. [14] and Krätschmer et al. [47]) and elicitability.

The term elicitability is due to Osband [55, Chap. 2] and Lambert et al. [50]. Using the terminology of mathematical statistics, a law-invariant risk measure \(\rho \) mapping to \(\mathbb{R}^{*}:= (-\infty ,\infty ]\) is elicitable on some class ℳ of distributions if it admits an \(M\)-estimator in the sense of Huber and Ronchetti [42, Chap. 3], i.e., if there is a loss or scoring function \(S\colon \mathbb{R}^{*}\times \mathbb{R}\to \mathbb{R}^{*}\) such that for all \(F\in \mathcal{M}\) and all \(x\in \mathbb{R}^{*}\), \(x\neq \rho (F)\). Any scoring function \(S\) satisfying (1.2) is called strictly ℳ-consistent for \(\rho \colon \mathcal{M}\to \mathbb{R}^{*}\). Besides their usage in \(M\)-estimation and regression [46, 45, 53], the fact that strict consistency encourages truthful forecasting opens the way to meaningful forecast comparison, see Gneiting [35], which is closely related to comparative backtests in finance; see Fissler et al. [31] or Nolde and Ziegel [54]. In a nutshell, let \((Y_{t})_{t=1, \ldots , N}\) be the profits and losses at the time points \(t=1, \ldots , N\) and let \(X_{t}\) be a one-step ahead point forecast for \(Y_{t}\), based on the information \(\mathfrak{F}_{t-1}\) available to the forecaster at time point \(t-1\). That is, \(X_{t}\) is an \(\mathfrak{F}_{t-1}\)-measurable random variable. The aim of \(X_{t}\) is to correctly specify the conditional risk measure \(\rho \) of \(Y_{t}\) given \(\mathfrak{F}_{t-1}\), that is, the minimal capital requirement one needs to add to \(Y_{t}\) to make \(Y_{t} + X_{t}\) acceptable under \(\rho \), given the information \(\mathfrak{F}_{t-1}\). The simplest example is when \(\rho \) is the negative expectation. Then the ideal forecast can be written as \(X_{t}^{*} = - \mathbb{E}[ Y_{t}\,|\,\mathfrak{F}_{t-1} ]\). For a general law-invariant risk measure \(\rho \), the ideal forecast for \(Y_{t}\) can be expressed as \(X_{t}^{*} = \rho ( F_{Y_{t}\,|\, \mathfrak{F}_{t-1}} ) \), where \(F_{Y_{t}\,|\, \mathfrak{F}_{t-1}}\) is the conditional distribution of \(Y_{t}\) given \(\mathfrak{F}_{t-1}\). Note that \(X_{t}\) might be misspecified in the sense that \(X_{t} \neq X_{t}^{*} = \rho ( F_{Y_{t}\,|\, \mathfrak{F}_{t-1}} ) \), e.g. due to possible estimation errors when coming up with an estimate of \(F_{Y_{t}\,|\, \mathfrak{F}_{t-1}}\), due to a misuse of the information \(\mathfrak{F}_{t-1}\), or due to calculation errors when applying the risk measure to the conditional distribution. We refer to Sect. 5 for some specific examples. If there is another forecaster with his/her own information sets \(\mathfrak{G}_{t-1}\) for time \(t- 1\) who issues alternative \(\mathfrak{G}_{t-1}\)-measurable point forecasts \(Z_{t}\), \(t=1, \ldots , N\), for \(Y_{t}\), then the predictive performance of \((X_{t})_{t=1, \ldots , N}\) is deemed better than the performance of \((Z_{t})_{t=1, \ldots , N}\) with respect to the scoring function \(S\) if \(\frac{1}{N} \sum _{t=1}^{N} S(X_{t},Y_{t}) < \frac{1}{N} \sum _{t=1}^{N} S(Z_{t},Y_{t})\). In the rest of this article, we conveniently call a point forecast for \(Y_{t}\) aiming at correctly specifying \(\rho (F_{Y_{t}|\mathfrak{F}_{t-1}})\) for some \(\sigma \)-algebra \(\mathfrak{F}_{t-1}\) a forecast for \(\rho \).

Ziegel [63] showed that expectiles are basically the only elicitable and coherent risk measures. In line with this, the prominent risk measure value at risk at level \(\alpha \in (0,1)\) (\(\operatorname{VaR}_{\alpha }\)), which corresponds to the negative of the lower \(\alpha \)-quantile (see equation (4.1)), turns out to be elicitable, subject to mild conditions, but not coherent. On the other hand, expected shortfall at level \(\alpha \in (0,1)\) (\(\operatorname{ES}_{\alpha }\)), a tail expectation, is coherent, but fails to be elicitable; see Gneiting [35] and Weber [62]. Interestingly, Fissler and Ziegel [27] showed that the pair \((\operatorname{VaR}_{\alpha }, \operatorname{ES}_{\alpha })\) is elicitable despite ES’s failure to have a strictly consistent scoring function on its own; see also Acerbi and Szekely [1] for a slightly weaker result, and Fissler and Ziegel [30] for a similar result for range value at risk.

Closely related to the notion of elicitability is the concept of identifiability. While the former is useful for forecast comparison or model selection, the latter aims at model and forecast validation or checks for calibration. A law-invariant risk measure \(\rho \colon \mathcal{M}\to \mathbb{R}^{*}\) is identifiable on ℳ if it admits a \(Z\)-estimator, i.e., if there is some function \(V\colon \mathbb{R}\times \mathbb{R}\to \mathbb{R}\) such that for all \(F\in \mathcal{M}\) and all \(x\in \mathbb{R}\). Any function \(V\) satisfying (1.3) is called a strict ℳ-identification function for \(\rho \). Here we generalise the definition of identifiability in the literature to risk-measures possibly assuming the value \(\infty \). Clearly, if \(\rho (F)=\infty \), (1.3) means that there is no real number \(x\) such that \(\int V(x,y) \,\mathrm{d}F(y) = 0\). Steinwart et al. [58] showed that under appropriate regularity conditions, the identifiability of a real-valued risk measure is equivalent to its elicitability. Coherently, \(\operatorname{VaR}_{\alpha }\) is identifiable under mild regularity conditions, using a simple coverage check, whereas \(\operatorname{ES}_{\alpha }\) fails to have a strict identification function. For a discussion of identifiability and calibration in the context of evaluating risk measures, we refer the reader to Davis [15] and Nolde and Ziegel [54].

The aim of this paper is to establish elicitability and identifiability results for systemic risk measures of the form (1.1) and for derived quantities thereof. This facilitates backtests of these risk measures and renders regression frameworks possible. Since these risk measures are set-valued, we use the terminological distinction between selective and exhaustive reports introduced in Fissler et al. [25] along with the corresponding notions of elicitability and identifiability. In a nutshell and translated to the setting of systemic risk measures of the form (1.1), a selective forecast specifies a single capital allocation that makes the system acceptable. On the other hand, exhaustive forecasts are more ambitious, aiming at reporting all adequate capital allocations simultaneously in the form of a set. Consequently, exhaustive scoring or identification functions take sets as their first argument, whereas their selective counterparts work with points as inputs. The corresponding definitions along with basic properties and assumptions on systemic risk measures defined in (1.1) and derived quantities such as efficient cash-invariant allocation rules (EARs), see Feinstein et al. [23], are gathered in Sect. 2.

Section 3 contains our main original contributions, most notably Theorem 3.1 asserting the existence of oriented selective identification functions for the functional \(R_{0}(Y) = \{k\in \mathbb{R}^{d}\colon \rho (\Lambda (Y+k))=0\}\), and Theorem 3.10, which uses these identification functions to construct strictly consistent exhaustive scoring functions for \(R\) from (1.1). Interestingly, these scoring functions arise as an integral construction of elementary scores, exploiting the orientation of the identification function. This can be considered a higher-dimensional analogue to the mixture representation of scoring functions for one-dimensional forecasts established in the seminal paper by Ehm et al. [19]. Similarly, this gives rise to the diagnostic tool of Murphy diagrams facilitating the assessment of forecast dominance; see Sect. 3.2.5. Thanks again to the orientation of the identification functions, we derive order-sensitivity results of these consistent scoring functions (Proposition 3.13). Concerning EARs as mentioned above, Proposition 3.6 establishes strict selective identification functions for EARs, interestingly mapping to a function space. On top of these main original contributions, we exploit the mutual exclusivity result of Fissler et al. [25, Theorem 3.7] to conclude that systemic risk measures of the form (1.1) are generally not selectively elicitable (Theorem 3.16).

The elicitability results on \(R\) rely on the identifiability of the underlying scalar risk measure \(\rho \). This spells doom for the elicitability of systemic risk measures induced by ES as a scalar risk measure. Section 4 outlines this issue and establishes a solution to this challenge at the cost of a higher forecast complexity. Similarly to the scalar case, considering a pair of \(R\) based on ES together with a VaR-related quantity leads to selective identifiability and exhaustive elicitability results (Proposition 4.1 and Theorem 4.3).

The practical applicability of our results is demonstrated in terms of a simulation study, which is the content of Sect. 5. Employing Diebold–Mariano tests, we examine how well the strictly consistent scores are able to distinguish different forecast performances. We also graphically illustrate the diagnostic tool of Murphy diagrams in a simulation example, utilising a traffic-light approach suggested in Fissler et al. [31]. We close the paper with a brief discussion.

In an online preprint version [26], we gather results on positively homogeneous and translation-invariant scoring functions, additional simulation results as well as results concerning risk measures insensitive with respect to capital allocations.

Let \((\Omega , \mathfrak{F}, \mathbb{P})\) be an atomless probability space, and for some integer \(d\ge 1\), let \(\mathcal{Y}^{d} \subseteq L^{0}( \Omega ;\mathbb{R}^{d})\) be a collection of \(d\)-dimensional random vectors which is closed under translation, meaning that \(Y\in \mathcal{Y}^{d}\) and \(k\in \mathbb{R}^{d}\) implies that \(Y+k\in \mathcal{Y}^{d}\). From a risk management perspective, the random vector \(Y = (Y_{1}, \ldots , Y_{d})^{\top }\in \mathcal{Y}^{d}\) represents the respective gains and losses of a system of \(d\) financial firms. That is, positive values of the component \(Y_{i}\) represent gains of firm \(i\) and negative values correspond to losses. Let \(\mathcal{M}^{d}\) be the class of probability distributions of elements of \(\mathcal{Y}^{d}\). Let \(\Lambda \colon \mathbb{R}^{d}\to \mathbb{R}\) be a measurable aggregation function, meaning that it is non-constant and increasing with respect to the componentwise order. An aggregation function is typically, but not necessarily, assumed to be continuous or even concave. Let \(\mathcal{Y}\subseteq L^{0}(\Omega ;\mathbb{R})\) be a collection of random variables which is closed under translation and contains \(\{\Lambda (Y)\colon Y\in \mathcal{Y}^{d}\}\). Similarly to \(\mathcal{M}^{d}\), let ℳ be the class of probability distributions of elements of \(\mathcal{Y}\).

We consider a scalar monetary law-invariant risk measure \(\rho \colon \mathcal{Y}\cup \mathbb{R}\to \mathbb{R}^{*}\); see Artzner et al. [6]. That is, for all \(X,Z\in \mathcal{Y}\cup \mathbb{R}\) and \(m\in \mathbb{R}\), it holds that \({\rho (X+m) = \rho (X) - m}\) (cash-invariance) and \(\rho (X)\le \rho (Z)\) if \(X\ge Z\) ℙ-a.s. (monotonicity). Moreover, we assume that \(\rho (0)\in \mathbb{R}\) so that cash-invariance induces a unique mapping \(\rho \colon \mathbb{R}\to \mathbb{R}\). Exploiting law-invariance, we identify \(\rho \colon \mathcal{Y}\cup \mathbb{R}\to \mathbb{R}^{*}\) with its induced risk functional \(\hat{\rho }\colon \mathcal{M}\cup \{\delta _{x}:x\in \mathbb{R}\}\to \mathbb{R}^{*}\), where \(\hat{\rho }(F) = \rho (X)\) for some \(X\sim F\), and simply write \(\rho \) in both cases.

We present the two most natural law-invariant set-valued measures of systemic risk that are based on \(\rho \) and \(\Lambda \), namely In (2.2) and later, we use the shorthand \(\bar{k} := \sum _{i=1}^{d} k_{i}\) for \(k = (k_{1}, \ldots , k_{d})^{\top }\in \mathbb{R}^{d}\). Note the difference between \(R\) and \(R^{\text{ins}}\). The risk measure \(R\) takes an ex ante perspective in the sense that it specifies all capital allocations \(k\in \mathbb{R}^{d}\) needed to be added to the system \(Y\) to make the aggregated system \(\Lambda (Y+k)\) acceptable under \(\rho \). On the other hand, \(R^{\text{ins}}\) takes an ex post perspective on quantifying the risk of the system \(Y\): It first considers the current aggregated system \(\Lambda (Y)\) and then specifies the total capital requirement \(\bar{k}\) one needs to add to make the aggregated system acceptable, which amounts to specifying the bail-out costs of the aggregated system \(\Lambda (Y)\) under \(\rho \). In particular, the risk measure \(R^{\text{ins}}\) is insensitive to the capital allocation to each financial firm, disregarding possible transaction costs or other dependence structures between the financial firms and ignoring how the addition of capital changes the system itself. This justifies the mnemonic terminology. Both risk measures \(R\) and \(R^{\text{ins}}\) can be of interest in applications, taking into regard the different perspectives on systemic risk. However, the mathematical treatment and complexity differ considerably: Due to the cash-invariance of \(\rho \), \(R^{\text{ins}}\) takes the equivalent form \(R^{\text{ins}}(Y) = \{k\in \mathbb{R}^{d}\colon \rho (\Lambda (Y)) \le \bar{k}\} \). This means that \(R^{\text{ins}}\) is actually a bijection of the scalar risk measure \(\rho \circ \Lambda \colon \mathcal{Y}^{d}\to \mathbb{R}^{*}\) considered in Chen et al. [13]. Therefore, one has to evaluate the risk measure \(\rho \) only once to determine \(R^{\text{ins}}\). In contrast, such an appealing equivalent formulation is generally not available for \(R\) unless \(\Lambda \) is additive, or is even the sum in which case \(R\) and \(R^{\text{ins}}\) coincide. Consequently, in general, one is bound to evaluate \(\rho \) infinitely often to compute \(R\); see also the discussion in Feinstein et al. [23]. The main focus of this paper are elicitability and identifiability results for systemic risk measures of the form (2.1) and (2.2). However, since one can exploit the one-to-one relation between \(R^{\text{ins}}\) and \(\rho \circ \Lambda \) and make use of the revelation principle, see Fissler [24, Sect. 2.3], Gneiting [35] and Osband [55, Sect. 2.1], to establish (exhaustive) elicitability and identifiability results, we do not present results about \(R^{\text{ins}}\) in this paper, but rather defer them to Fissler et al. [26, Supplementary Material].

For the sake of completeness, we recall the most important properties of \(R\) presented in Feinstein et al. [23]. Because \(\rho \) is cash-invariant and \(\Lambda \) is increasing, the values of both \(R\) and \(R^{\text{ins}}\) defined in (2.1) and (2.2) are upper sets, i.e., \({R(Y) = R(Y) +\mathbb{R}_{+}^{d}} \) for any \(Y\in \mathcal{Y}^{d}\), where \(\mathbb{R}_{+}^{d}\) denotes the collection of vectors in \(\mathbb{R}^{d}\) with only nonnegative elements and for any two sets \(A, B\subseteq \mathbb{R}^{d}\), \(A+B:= \{a+b\colon a\in A, \ b\in B\}\) is the usual Minkowski sum. Recall that we have \(A+\emptyset = \emptyset + A = \emptyset \). Following the notation of [23], we denote the collection of upper sets in \(\mathbb{R}^{d}\) with ordering cone \(\mathbb{R}_{+}^{d}\) by \(\mathcal{P}(\mathbb{R}^{d}; \mathbb{R}^{d}_{+}) := \{B\subseteq \mathbb{R}^{d}\colon B = B+\mathbb{R}^{d}_{+}\}\). Both \(\mathbb{R}^{d}\) and \(\emptyset \) are elements of \(\mathcal{P}(\mathbb{R}^{d}; \mathbb{R}^{d}_{+}) \). Moreover, \(R\) defined in (2.1) can attain these values even if the underlying scalar risk measure \(\rho \) maps to ℝ only, e.g. when \(\Lambda \) is bounded. While \(R(Y) = \emptyset \) corresponds to the case that a scalar risk measure of the financial position \(Y\) is \(+\infty \), meaning that the system \(Y\) is deemed risky no matter how much capital is injected, the case \(R(Y) = \mathbb{R}^{d}\) corresponds to \(-\infty \) in the scalar case. The latter situation of “cash cows” with the possibility to withdraw any finite amount of money without rendering the position risky is usually deemed unrealistic and is excluded. Therefore, we usually only discuss the case \(R(Y)=\emptyset \), but remark that a treatment of the case \(R(Y)=\mathbb{R}^{d}\) would also be possible for most results. Monotonicity and cash-invariance carry over to \(R\) in that \(R(Y)\supseteq R(Z)\) for all \(Y,Z\in \mathcal{Y}^{d}\) with \(Y\ge Z\) ℙ-a.s. componentwise, and \(R(Y+k) = R(Y) - k\) for all \(k\in \mathbb{R}^{d}\). Monotonicity also carries over to \(R^{\text{ins}}\); note, however, that \(R^{\text{ins}}\) is in general not cash-invariant. We introduce further subclasses of \(\mathcal{P}(\mathbb{R}^{d}; \mathbb{R}^{d}_{+})\), where \(\mathcal{B}(\mathbb{R}^{d})\) denotes the Borel-\(\sigma \)-algebra on \(\mathbb{R}^{d}\).

For any set \(A\subseteq \mathbb{R}^{d}\), we denote its topological boundary by \(\partial A\). We introduce the law-invariant map Occasionally and when explicitly stated, we impose one of the following assumptions.

A sufficient condition for \(R(Y)\in \mathcal{F}(\mathbb{R}^{d};\mathbb{R}^{d}_{+})\) is that for any convergent sequence \((k_{n})_{n\in \mathbb{N}}\subseteq \mathbb{R}^{d}\) with limit \(k\), we have The inequality in (2.4) holds e.g. if \(\Lambda \) is continuous and if \(\rho (X)\le \liminf _{n\to \infty } \rho (X_{n})\) for all sequences \((X_{n})\) in \(\mathcal{Y}\) converging almost surely to \(X\in \mathcal{Y}\). In particular, if \(\mathcal{Y}= L^{\infty }(\Omega ;\mathbb{R})\), \(\rho \) is convex, \(\Lambda \) continuous, and either (a) \(\Lambda \) is bounded (invoking the law-invariance of \(\rho \) and the results from Jouini et al. [44] and Svindland [60]) or (b) \(\Lambda \) is uniformly continuous, then (2.4) holds. For instance in the network model considered in Sect. 5.1, \(\Lambda \) is bounded. Otherwise in the literature, \(\Lambda \) is often a concave function and thus not bounded (unless it is constant). Hence, for concave \(\Lambda \), one could check whether it is uniformly continuous. This is clearly the case for instance for the most straightforward choice—the sum—as well as for instance for the aggregation function suggested by Amini et al. [4], also considered in Sect. 5.1. Moreover, note that we provide sufficient conditions only and not necessary ones, so that one may also check \(R(Y)\in \mathcal{F}(\mathbb{R}^{d};\mathbb{R}^{d}_{+})\) on a case-by-case basis.

If \(\emptyset \neq R(Y) \in \mathcal{F}(\mathbb{R}^{d};\mathbb{R})\), then \(R_{0}(Y)\neq \emptyset \). Since \(\Lambda \) is increasing and \(\rho \) is cash-invariant, one obtains \(R(Y) = R_{0}(Y) + \mathbb{R}_{+}^{d} \). Hence the values of \(R_{0}\) determine \(R\) completely. Moreover, if \(\Lambda \) is strictly increasing, we have \(R_{0}(Y) = \partial R(Y)\), meaning that \(R_{0}(Y)\) contains the efficient capital allocations that make \(Y\) acceptable under \(R\). Therefore, under Assumption 2.3, \(R\) and \(R_{0}\) are connected via a one-to-one relation. Again invoking the revelation principle [35, Theorem 4], this means that exhaustive elicitability results for \(R\) (Theorem 3.10 (iii)) carry over to \(R_{0}\). In a nutshell, the revelation principle asserts that if there is a bijection, say \(g\), such that \(R_{0} = g(R)\), then \(R_{0}\) is (exhaustively) elicitable if and only if \(R\) is elicitable. Moreover, \(S(A,y)\) is strictly consistent for \(R\) if and only if \(S(g^{-1}(A),y)\) is strictly consistent for \(R_{0}\).

Finally, we recall the definition of an important scalarisation of the systemic risk measure \(R\), called efficient cash-invariant allocation rule (EAR), as introduced in Feinstein et al. [23]. Roughly speaking, for \(Y\in \mathcal{Y}^{d}\), \(\text{{EAR}}(Y)\) specifies the capital allocations with minimal weighted cost among allocations in \(R(Y)\). For simplicity, we confine our attention to the situation when \(R(Y)\) is closed and to EARs with a fixed price or weight vector \(w\in \mathbb{R}^{d}_{++}:= \{x\in \mathbb{R}^{d}\colon x_{1}, \ldots , x_{d}>0\}\).

For \(Y\in \mathcal{Y}^{d}\), if there is a supporting hyperplane of \(R(Y)\) orthogonal to \(w\), then \(\text{{EAR}}_{w}(Y)\) is the intersection of \(\partial R(Y)\) and this hyperplane. Hence \(\text{{EAR}}_{w}(Y)\) is not necessarily a singleton. If there is no supporting hyperplane of \(R(Y)\) orthogonal to \(w\), the function \(R(Y)\ni k \mapsto w^{\top }k\) is unbounded from below and we set \(\text{{EAR}}_{w}(Y)=\emptyset \).

Since \(\rho \) is law-invariant, so are the derived quantities \(R\), \(R^{\text{ins}}\), \(R_{0}\) and \(\text{{EAR}}_{w}\). Therefore, in analogy to our treatment of \(\rho \), we identify \(R\) with the risk functional \(\hat{R}\), where \(\hat{R}(F) = R(Y)\) for \(Y\sim F\in \mathcal{M}^{d}\), and simply write \(R\) for either; we use analogous conventions for \(R_{0}\) and \(\text{{EAR}}_{w}\).

We have already mentioned the definitions of elicitability and identifiability for scalar risk measures \(\rho \colon \mathcal{M}\to \mathbb{R}^{*}\) in (1.2) and (1.3), where we slightly extend the common definitions to account for \(\rho \) possibly attaining \(\infty \). All other risk measures considered here, \(R\), \(R_{0}\) and \(\text{{EAR}}\), are set-valued, attaining subsets of \(\mathbb{R}^{d}\). Hence we make use of the theoretical framework on forecast evaluation of set-valued functionals introduced in Fissler et al. [25]. The main idea is to have a thorough distinction concerning the form of the forecasts between a selective notion where forecasts are single points, and an exhaustive mode where forecasts are set-valued. Moreover, corresponding notions of identifiability and elicitability are introduced and discussed in a very general setting, with the main result being that—subject to mild conditions—a set-valued functional is elicitable either in the selective, or the exhaustive sense, or not elicitable at all [25, Theorem 2.14]. We confine ourselves to introducing only the notions we discuss in this paper and do so directly in terms of \(R\) and \(R_{0}\); the case of \(\text{{EAR}}\)s is considered separately later. In the sequel, let \(\mathcal{A}\subseteq 2^{\mathbb{R}^{d}}\). Moreover, for scoring functions \(S\colon \mathcal{A}\times \mathbb{R}^{d}\to \mathbb{R}^{*}\) or identification functions \(V\colon \mathbb{R}^{d}\times \mathbb{R}^{d}\to \mathbb{R}\), we use the shorthands \(\bar{S}(A,F) := \int S(A,y)\,\mathrm{d}F(y)\) and \(\bar{V}(x,F) := \int V(x,y)\,\mathrm{d}F(y)\) for \(A\in \mathcal{A}\), \(x\in \mathbb{R}^{d}\), and tacitly assume that these integrals exist for all \(F\in \mathcal{M}^{d}\), where we say that the integral \(\int g(y)\,\mathrm{d}F(y)\) of a function \(g \colon \mathbb{R}\to [-\infty , \infty ]\) exists if \(g\) is measurable and \(\int g(y)^{+}\,\mathrm{d}F(y)<\infty \) or \(\int g(y)^{-}\,\mathrm{d}F(y)<\infty \). In that case, we set

Note that the strict consistency of an exhaustive scoring function \(S\) for \(R\) implies that \(\bar{S}(R(F),F) \in \mathbb{R}\) for all \(F\in \mathcal{M}^{d}\).

Interestingly, if \(V_{R_{0}}\colon \mathbb{R}^{d}\times \mathbb{R}^{d}\to \mathbb{R}\) is an oriented strict selective \(\mathcal{M}^{d}\)-identification function for \(R_{0}\) and \(\bar{V}_{R_{0}}(\cdot ,F)\), \(F\in \mathcal{M}^{d}\), is continuous, then \(R(F)\) is closed.

In a similar spirit as Remark 3.3, one might also wonder whether the (oriented) strict selective identification functions constructed in Theorem 3.1 are the only (oriented) strict selective identification functions for \(R_{0}\). This is definitely not the case since due to the linearity of the expectation, any function \(V'_{R_{0}}\colon \mathbb{R}^{d}\times \mathbb{R}^{d}\to \mathbb{R}\) with where \(h\colon \mathbb{R}^{d}\to \mathbb{R}\) is non-vanishing, is again a strict selective \(\mathcal{M}^{d}\)-identification function for \(R_{0}\). Moreover, if \(V_{R_{0}}\) is oriented, then \(V'_{R_{0}}\) defined in (3.5) is oriented if and only if \(h>0\). In particular, the constant \(g(0)\) appearing in Remark 3.3 can be incorporated into the function \(h\) so that we see that it does not matter which (oriented) strict exhaustive identification function \(\widetilde{V}_{\rho }\) we choose to end up with the form in (3.5). The next result establishes that basically all selective \(\mathcal{M}^{d}\)-identification functions for \(R_{0}\) are of the form (3.5).

Finally, we turn our attention to EARs as introduced in Definition 2.4. For any price or weight vector \(w\in \mathbb{R}^{d}\), we use the notation \(w^{\perp }:= \{x\in \mathbb{R}^{d}\colon w^{\top }x =0\}\) for the orthogonal complement of the subspace spanned by \(w\). With \(\mathbb{R}^{w^{\perp }}\), we denote as usual the space of all functions from \(w^{\perp }\) to ℝ.

If the underlying risk measure \(R\) is known to assume convex sets only (e.g. if \(\rho \) is convex and \(\Lambda \) concave, see Feinstein et al. [23]), it is even sufficient to evaluate \(\bar{V}_{\mathrm{{EAR}}_{w}}(k,F)(x)\), or its empirical counterpart, for \(x\in w^{\perp }\) in a neighbourhood of 0, which can also be seen nicely in Fig. 1.

In Fissler et al. [25, Sect. 3.1], different versions of the convex level sets (CxLS) property are introduced and their necessity for identifiability and elicitability for set-valued functionals is discussed. Our next result establishes the so-called selective CxLS* property of \(\mathrm{{EAR}}_{w}\). That is, for all \(F_{0},\ F_{1}\in \mathcal{M}^{d}\) and all \(\lambda \in (0,1)\) such that \(F_{\lambda }:=(1-\lambda )F_{0}+\lambda F_{1}\in \mathcal{M}^{d}\), it holds that

If ℳ is such that \(\mathrm{{EAR}}_{w}\) is ‘truly set-valued’ on ℳ, and in particular if it satisfies the proper subset property of Fissler et al. [25, Definition 3.4], i.e., there exist \(F,G\in \mathcal{M}\) such that \(\emptyset \neq \mathrm{{EAR}}_{w}(G) \subsetneq \mathrm{{EAR}}_{w}(F)\) and ℳ is convex, then [25, Theorem 3.5] asserts that \(\mathrm{{EAR}}_{w}\) is not exhaustively elicitable on ℳ under the conditions of Proposition 3.7.

We end this section by noting that the identifiability of \(\rho \) and the selective identifiability of \(R_{0}\) are even equivalent if \(\Lambda \colon \mathbb{R}^{d}\to \mathbb{R}\) possesses a measurable right inverse.

In the seminal paper Ehm et al. [19], it is shown that subject to regularity conditions, any nonnegative scoring function \(S\colon \mathbb{R}\times \mathbb{R}\to [0,\infty ]\) consistent for the \(\alpha \)-quantile (the \(\tau \)-expectile) can be written as a mixture or Choquet representation where \(H\) is a nonnegative measure on \(\mathcal{B}(\mathbb{R})\) and \(S_{\theta }\), \(\theta \in \mathbb{R}\), are nonnegative elementary scoring functions for the \(\alpha \)-quantile (the \(\tau \)-expectile). In particular, \(S_{\theta }\) takes the form where \(V\) is an oriented identification function for the \(\alpha \)-quantile (the \(\tau \)-expectile). The score in (3.9) is strictly consistent if and only if the measure \(H\) is strictly positive, that is, puts positive mass on any open nonempty set. Ziegel [64] and Dawid [16] argued that besides expectiles and quantiles, this construction also works for more general one-dimensional functionals which admit an oriented identification function; see Jordan et al. [43]. Steinwart et al. [58] showed that for one-dimensional functionals satisfying certain regularity conditions, the existence of an oriented identification function is equivalent to the elicitability of the functional. While the orientation of the identification function immediately gives rise to the consistency of the elementary scores and thus of the mixtures in (3.9), an answer to the question as to whether all nonnegative scoring functions for a certain functional are necessarily of the form in (3.9) can typically only be answered by invoking Osband’s principle (see Fissler and Ziegel [27] and Osband [55, Theorem 2.1]), hence assuming smoothness and regularity conditions.

Our construction of strictly consistent exhaustive scoring functions for the systemic risk measures \(R\) also exploits the key result in Theorem 3.1 about the existence of oriented strict selective identification functions for \(R_{0}\) and is similar in nature to the approach described above. For any \(y\in \mathbb{R}^{d}\), we use the notation \(R(y) := R(\delta _{y})\).

For the proof of Theorem 3.10, we need the following lemma.

Note that condition (3.11) or the similar condition on \(\bar{V}_{R_{0}}\) in part (iii) of Theorem 3.10 does not imply that \(V_{R_{0}}\) is an identification function for \(R_{0}\). This relaxation is particularly beneficial if \(\rho \) is value at risk and ℳ contains discontinuous distributions, so that value at risk possibly fails to be identifiable on ℳ.

Let \(\mathcal{M}_{\mathrm{inc, cont}} \subseteq \mathcal{M}\) be the subclass of continuous and strictly increasing distribution functions in ℳ. Let \(\mathcal{M}_{\mathrm{inc, cont}}^{d} \subseteq \mathcal{M}^{d}\) be the subclass of distributions such that for any \(Y \sim F \in \mathcal{M}_{\text{inc,cont}}^{d}\), the distribution of \(\Lambda ( Y+k)\) is in \(\mathcal{M}_{\mathrm{inc, cont}}\) for all \(k\in \mathbb{R}^{d}\). A strict \(\mathcal{M}_{\mathrm{inc, cont}}\)-identification function \(V\colon \mathbb{R}^{2}\times \mathbb{R}\to \mathbb{R}^{2}\) for the pair \((\operatorname{VaR}_{\alpha },\operatorname{ES}_{\alpha }): \mathcal{M}\to \mathbb{R}\times \mathbb{R}^{*}\) is given, for \((v,e)\in \mathbb{R}^{2}\) and \(y\in \mathbb{R}\), by which can be verified by a straightforward calculation. This function induces a (non-strict) selective \(\mathcal{M}^{d}\)-identification function \(U\colon \mathbb{R}^{\mathbb{R}^{d}}\times \mathbb{R}^{d}\times \mathbb{R}^{d}\to \mathbb{R}^{2}\) for the pair \((T^{ \operatorname{VaR}_{\alpha }}, R^{\operatorname{ES}_{\alpha }}_{0}) \colon \mathcal{M}^{d}\to \mathbb{R}^{\mathbb{R}^{d}}\times 2^{ \mathbb{R}^{d}}\). For \(v\colon \mathbb{R}^{d}\to \mathbb{R}\), \(k\in \mathbb{R}^{d}\) and \(y\in \mathbb{R}^{d}\), \(U(v,k,y)\) is defined as

We introduce the following regularity assumption on \(T^{\operatorname{VaR}_{\alpha }}\) defined in (4.2).

Clearly, Assumption 4.2 is satisfied if \(\Lambda \) is continuous. For any increasing function \(g\colon \mathbb{R}\to \mathbb{R}\), introduce . Recall from Gneiting [36, Theorem 2.6] that \(S_{\alpha ,g}\) is a nonnegative consistent selective scoring function for the \(\alpha \)-quantile. Moreover, if \(g\) is strictly increasing, \(S_{\alpha ,g}\) is a strictly consistent selective scoring function for the \(\alpha \)-quantile relative to any class ℳ of distributions such that \(g\) is ℳ-integrable; see Gneiting [36].

Theorem 4.3 (ii) suggests that there is again the possibility to consider Murphy diagrams to assess the quality of forecasts for \((T^{\operatorname{VaR}_{\alpha }}, R^{\operatorname{ES}_{\alpha }})\) simultaneously over all scoring functions given in (4.6). However, a direct implementation would amount to defining them on the \(2d\)-dimensional Euclidean space. If one further decomposes the functions \(g_{k}\) in the spirit of Ehm et al. [19], one would even end up with a map defined on \(\mathbb{R}\times \mathbb{R}^{d}\times \mathbb{R}^{d}\). However, arguing along the lines of Ziegel et al. [65], the measure \(\pi _{1}\) only accounts for forecast accuracy in the VaR component. Therefore, if interest focuses on the ES component, it makes sense to set \(\pi _{1}=0\) to facilitate the analysis. This implies that one can consider the Murphy diagram \(\mathbb{R}^{d}\ni k \mapsto \mathbb{E}[S_{k}(v,A,Y)] \) with the elementary scores \(S_{k}\) given in (4.5). The empirical formulation in the spirit of (3.15) is straightforward.

In this section, we demonstrate via a simulation study the discrimination ability of the consistent exhaustive scoring functions constructed in Theorem 3.10. We do so in the context of the prediction space setting introduced in Gneiting and Ranjan [38]. This means that we explicitly model the information sets of each forecaster. For the sake of simplicity and following Gneiting et al. [37] and Fissler and Ziegel [30], we choose to consider only one-step-ahead forecasts and prediction–observation sequences that are independent and identically distributed over time. Despite this simplification, there are still a variety of parameters to consider in the simulation study: We confine ourselves to the following choices of these parameters.

(i)–(iii) We work with two different combinations of \(Y_{t}\) and \(\Lambda \). In both cases, we work with a financial system of \(d=5\) participants.

(a) The vector \(Y_{t}\) models the gains and losses of the participants in the system. At any time point \(t\), \(Y_{t}=\mu _{t}+\epsilon _{t}\), where the risk factor \(\mu _{t}\) follows a 5-dimensional normal distribution with mean 0, correlations 0.5 and variances 1, and \(\epsilon _{t}\) follows a 5-dimensional standard normal distribution. Moreover, \(\mu _{t}\) and \(\epsilon _{t}\) are independent for all \(t\). Thus conditionally on \(\mu _{t}\), \(Y_{t}\) has distribution \(\mathcal{N}_{5}(\mu _{t},I_{5})\), whereas unconditionally, \(Y_{t}\sim \mathcal{N}_{5}(0,\Sigma )\) with \(\Sigma _{ij}=0.5\) for \(i, j=1,\ldots ,5\), \(i\neq j\) and 2 otherwise. Following the suggestion of Amini et al. [4], the aggregation function \(\Lambda _{1}\) is of the form \(\Lambda _{1}(Y_{t})=(1-\beta )\sum _{i=1}^{d}Y_{i,t}^{+}-\beta \sum _{i=1}^{d}Y_{i,t}^{-}\), and we set \(\beta =0.75\). In this way, both gains and losses influence the value of the aggregation function, but the losses have a higher weight.

(b) We consider an extended model of Eisenberg and Noe [20]; see Feinstein et al. [23]. The participants have liabilities towards each other, and \(L_{ij,t}\) represents the nominal liability of participant \(i\) towards participant \(j\) at time point \(t\), for \(i,j=1, \ldots , 5\). Moreover, each participant \(i\) owes an amount \(L_{is,t}\) to society at time point \(t\). To simplify the simulations and shorten the computing time, we assume that the liabilities matrix is deterministic and constant in time so that we can write \(L_{is}\) and \(L_{ij}\) instead of \(L_{is,t}\) and \(L_{ij,t}\). Moreover, we denote by \(\bar{L}_{s}=\sum _{i=1}^{d} L_{is}\) the sum of all payments promised to society. The vector \(Y_{t}\) represents the endowments of the participants at time point \(t\). As suggested in [20], if some of the endowments are negative, we introduce a so-called sink node and interpret the negative endowments as liabilities towards this node. The value of the aggregation function \(\Lambda _{2}\) corresponds to the sum of all payments society obtains in the clearing process as described in [20], lowered by 90% of the amount promised to society to ensure that the aggregation function can attain both positive and negative values. This ensures that the system is acceptable if \(\rho \) applied to the sum of all payments society obtains does not exceed \(-0.9\bar{L}_{s}\); see also [23]. To simulate the endowments \(Y_{t}\) of the participants, we assume that \(Y_{it}=(\mu _{it}+\epsilon _{it})^{2}\) for \(i=1,\ldots ,5\) with \(\mu _{t}\) and \(\epsilon _{t}\) specified in (a). We construct the system in the following way: The probability of a participant owing to another participant is 0.8. If there is a liability from \(i\) to \(j\), its nominal value is 2. In addition, each participant owes 2 to society.

(iv) For \(\rho \), we consider the scalar risk measures \(\operatorname{VaR}_{\alpha }\) and \(\operatorname{ES}_{\alpha }\), \(\alpha \in (0,1)\), as well as the expectile-based version of \(\operatorname{VaR}\) defined as \(\operatorname{EVaR}_{\tau }(X)=-e_{\tau }(X)\), \(\tau \in (0,1)\), where the expectile \(e_{\tau }\) is the unique solution to the equation for \(X\in L^{1}(\Omega , \mathbb{R})\); see Newey and Powell [53]. For the interpretation of expectile-based risk measures in finance, we refer to Bellini and Di Bernardino [10]; for a novel economic angle, see Ehm et al. [19]. Using the standard identification functions for \(\operatorname{VaR}_{\alpha }\) and \(\operatorname{EVaR}_{\tau }\) from Gneiting [35] and the explicit construction in (3.1), the selective identification function for \(R_{0}\) is given by for \(\rho = \operatorname{VaR}_{\alpha }\), and by \(V_{R_{0}}(k,y)=\tau (\Lambda (k+y))^{+}-(1-\tau )(\Lambda (k+y))^{-}\) for \(\rho =\operatorname{EVaR}_{\tau }\). As mentioned in Sect. 4, if the focus is on \(\operatorname{ES}_{\alpha }\), we can and do set the measure \(\pi _{1}\) to 0 in (4.6), meaning that no particular choice of a scoring or identification function need be made for this risk measure.

(v) We consider two ideal forecasters with different information sets. Anne has access to the risk factor \(\mu _{t}\) and uses the correct conditional distribution of \(Y_{t}\) given \(\mu _{t}\) for her predictions. That is, for \(t=1, \ldots , N\), she issues forecasts Here, \(\mathcal{N}_{d}(m,\Sigma )^{2}\) denotes the distribution of a random vector \((X_{1}^{2}, \ldots , X_{d}^{2})^{\top }\) with \((X_{1}, \ldots , X_{d})^{\top }\sim \mathcal{N}_{d}(m,\Sigma )\). On the other hand, Bob does not use information about the risk factor \(\mu _{t}\). Therefore he uses the correct unconditional distribution of \(Y_{t}\) for his forecasts and predicts, for \(t=1, \ldots , N\),

(vi) We choose \(\pi \) to be a 5-dimensional Gaussian measure with mean \(m\in \mathbb{R}^{5}\) and covariance \(I_{5}\). We also set \(\pi _{2}=\pi \) in (4.6). To enhance the discrimination ability of the score \(S_{R,\pi }\), we choose \(m\) close to the boundary of \(R(Y_{t})\). Here we work with \(m=2\times \mathbf{1}\) as this value appears to be fairly close to Bob’s unconditional forecasts in all cases. This choice of \(\pi \) turns out to be beneficial with respect to integrability considerations and renders our scores finite. Indeed, since \(V_{R_{0}}\) for \(\rho =\operatorname{VaR}_{\alpha }\) is bounded, it is \(\pi \otimes F\)-integrable for any finite measure \(\pi \). In the case of \(\rho =\operatorname{EVaR}_{\tau }\), more considerations are necessary. From the construction of \(\Lambda _{2}\), it is clear that it is a bounded function; in particular, the values lie in the interval \([-0.9 \bar{L}_{s},\sum _{i=1}^{d} L_{is}-0.9\bar{L}_{s}]\). This in turn implies that the identification function \(V_{R_{0}}\) is bounded. Therefore \(V_{R_{0}}\) is \(\pi \otimes F\)-integrable for any finite measure \(\pi \). Finally, since \(\Lambda _{1}\) only grows linearly and both \(\pi \) and \(Y_{t}\) are Gaussian, the integrability is also guaranteed in this case. Similar considerations confirm that the sufficient integrability conditions are also satisfied for \(\rho =\operatorname{ES}_{\alpha }\).

(vii) We work with sample sizes \(N=250\), a good proxy for the number of working (and trading) days in a year. In a financial context, the two forecasters Anne and Bob could be both portfolio managers or regulators. While \(\Lambda _{1}\) would be an appropriate aggregation function if Anne and Bob are portfolio managers, \(\Lambda _{2}\) would be more appropriate for regulators. The choices of including or omitting information contained in the risk factor \(\mu _{t}\) might stem from different data access, or deliberate different choices of risk factors. From a different angle, Bob might deliberately choose to use the unconditional profit and loss distribution to come up with a more prudent risk measurement approach than Anne who uses the conditional distribution; see McNeil et al. [51, Sect. 9.2.1].

To compare Anne’s with Bob’s forecast performance, we employ the classical Diebold–Mariano test [17] based on the scoring functions \(S_{R,\pi }\) of the form (3.13) and (4.6). We repeat the experiment 1000 times for setting (a) and 100 times for setting (b) since due to the presence of clearing, the computation time tends to be quite lengthy in setting (b). We approximate \(\pi \) with a Monte Carlo draw of size 100 000. The computations are performed with the statistics software R, and in particular its Rcpp package to also integrate parts of C++ code to enhance the computational speed.

We consider tests with two different one-sided null hypotheses. The null hypothesis \(H_{0}\colon \mathbb{E}[S_{R,\pi }(A_{1},Y_{1})]\geq \mathbb{E}[S_{R, \pi }(B_{1},Y_{1})]\), or in short \(H_{0}\colon A\succeq B\), means that Bob has a better forecast performance than Anne on average, evaluated in terms of \(S_{R,\pi }\). On the other hand, the null hypothesis \(H_{0}\colon \mathbb{E}[S_{R,\pi }(A_{1},Y_{1})]\leq \mathbb{E}[S_{R, \pi }(B_{1},Y_{1})]\) or \(H_{0}\colon A\preceq B\) stands for asserting that Anne’s forecasts are superior to Bob’s on average, in terms of \(S_{R,\pi }\). In Table 1, we report the relative frequencies of rejections for the respective null hypotheses. Invoking the sensitivity of consistent scoring functions with respect to increasing information sets established in Holzmann and Eulert [41], we expect that Anne’s forecasts are deemed superior to Bob’s predictions. And in fact, the null \(A\preceq B\) is never rejected for either scenario, while \(A\succeq B\) is rejected in between 67% and 100% of all experiments over the various scenarios. In particular, with rejection rates for \(H_{0}\colon A\succeq B\) between 0.94 and 1, we observe that the discrimination ability between Bob and Anne is considerably higher for model (a) than for (b), where we obtain rejection rates ranging from 0.67 to 0.90. This might be due to the fact that \(\Lambda _{1}\) is unbounded whereas \(\Lambda _{2}\) only takes values between 0 and \(\bar{L}_{s}\), which might translate into a smaller influence of the predictive distributions upon which the forecasts are based. Moreover, for fixed probability level \(\alpha =\tau \), both in case (a) and (b), the number of instances when Anne’s forecasts are preferred to Bob’s is the highest for \(\rho =\operatorname{EVaR}_{\alpha }\) and the lowest for \(\rho =\operatorname{ES}_{\alpha }\), with the exception of \(\operatorname{ES}_{0.05}\) in case (a). This ordering might be in line with the observation that a normally distributed \(X\) is deemed differently risky with respect to the three risk measures considered when evaluated at the same probability level: \(\operatorname{EVaR}_{\alpha }(X)\le \operatorname{VaR}_{\alpha }(X) \le \operatorname{ES}_{\alpha }(X)\) for \(\alpha \in (0,0.5)\), see Nolde and Ziegel [54, Table 1].

We illustrate the use of Murphy diagrams, following Corollary 3.15. We work within case (a) of Sect. 5.1 with \(d=2\). In particular, we have \(Y_{t}=\mu _{t}+\epsilon _{t}\), where \(\mu _{t}\) follows a 2-dimensional normal distribution with mean 0, variances 1 and correlations 0.5, and \(\epsilon _{t}\) follows a 2-dimensional standard normal distribution. As the scalar risk measure \(\rho \), we only consider \(\operatorname{VaR}_{0.05}\), and we use the aggregation function \(\Lambda _{1}\colon \mathbb{R}^{2}\to \mathbb{R}\), i.e., \(\Lambda (x)=0.25(x_{1}^{+}+x_{2}^{+})-0.75(x_{1}^{-}+x_{2}^{-})\). We should like to combine this with an illustration of the Murphy diagrams in the context of comparative backtests as introduced in Fissler et al. [31] and further developed in Nolde and Ziegel [54]. To this end, suppose Bob’s ideal unconditional risk measure forecasts \(B_{t} = R(\mathcal{N}_{2}(0_{2}, \Sigma ))\) play the role of the regulator’s standardised procedure. Then we have two internal risk measurement procedures making use of the additional risk factor given by \(\mu _{t}\). The first are Anne’s ideal forecasts \(A_{t} = R(\mathcal{N}_{2}(\mu _{t},I_{2})) = R(\mathcal{N}_{2}(0,I_{2}))- \mu _{t}\). The second are Celia’s forecasts. Confused about different sign conventions in the literature, she issues sign-reversed forecasts \(C_{t}\) assuming that \(Y_{t}\sim \mathcal{N}_{2}(-\mu _{t},I_{2})\), resulting in the forecasts \(C_{t}= R(\mathcal{N}_{2}(-\mu _{t},I_{2})) = R(\mathcal{N}_{2}(0,I_{2}))+ \mu _{t}\). Again, we consider a time horizon of \(N=250\).

In the left panel of Fig. 3, we illustrate the differences of empirical Murphy diagrams, where \(f_{t} \in \{ A_{t} , C_{t} \}\) stands for either of the two internal procedures produced by Anne or Celia. For the comparison of Bob vs. Anne, we can see that the Murphy diagram \(\hat{s}_{250,B}(k) - \hat{s}_{250,A}(k)\) is mostly nonnegative, indicating the superiority of Anne’s more informed forecast over Bob’s prudential unconditional one. On the other hand, comparing Bob vs. Celia, the Murphy diagram \(\hat{s}_{250,B}(k) - \hat{s}_{250,C}(k)\) is nonpositive everywhere. This means that all elementary scores indicate the inferiority of Celia’s sign-reversed forecasts. The regions with the highest score differences (in absolute values) seem to correspond to a blurred version of the boundary of the considered risk measure forecasts. Quite intuitively, the magnitude of the score difference with a maximum of approximately 0.05 is smaller when comparing the two ideal forecasts issued by Bob and Anne in comparison to the situation involving the sign-reversed Celia, where the maximal difference between the Murphy diagrams is in magnitude larger than 0.15. We have performed this experiment several times and observed that the stylised facts are qualitatively stable, though there is still recognisable sample variation present. For transparency reasons, we have depicted the first experiment performed, reporting some more experiments in [26, Supplementary Material].

In the right panel of Fig. 3, we depict the results of pointwise comparative backtests using the traffic-light illustration suggested in [31], which is the analogue for comparative backtests to the three-zone approach for traditional backtests in [7, Appendix B, Sect. III]. In other words, we perform two Diebold–Mariano tests using the elementary score \(S_{R,k}\) for each \(k\) in a grid of \([-5,5]^{2}\) at a significance level of 0.05. The grid cell is coloured in green if the conservative null hypothesis \(H_{0}^{+}\colon B \preceq f\) is rejected. This means that the internal procedure \(f\) performs significantly better than Bob’s standardised procedure in terms of \(S_{R,k}\). In contrast, the grid cell is depicted in red if the null \(H_{0}^{-} \colon B \succeq f\) is rejected, which indicates that the standardised procedure is significantly superior to the internal risk measurement procedure. For all \(k\) in the yellow region, none of the two nulls is rejected, meaning that the procedure is indecisive at significance level 0.05. Finally, the grey area corresponds to those points where the score difference is constantly zero for all \(t=1, \ldots , N\). Due to the vanishing variance, a Diebold–Mariano test is not possible there. But clearly, this still means that the two forecasts are just equally good in that region.

The specific results nicely correspond to the situations obtained in the left panel of Fig. 3. For both pairwise comparisons and for \(k\) close to the four corners of the area \([-5,5]^{2}\), the score differences vanish identically, resulting in a grey colouration. Again, in both cases, there is a ‘continuous’ behaviour in that the grey region adjoins a yellow stripe before turning into a fairly broad green or red stripe. For the comparison involving Celia, it is reassuring that a substantial region is coloured in red. In this region, the procedure is decisive, deeming Celia significantly inferior to Bob. Moreover, for this particular simulation, there is no green region. The situation comparing the two ideal forecasters Anne and Bob is somewhat more involved. Most points \(k\) are coloured in green or yellow/grey, so that either Anne’s more informed forecasts are deemed significantly superior to Bob’s standardised forecasts or the procedure is indecisive between the two methods. However, there is a small red stripe close to the upper right corner. For \(k\) in that region and for this particular simulation, this means that Bob’s forecasts outperform Anne’s. While this observation is somewhat unexpected, it reflects the finite sample nature of the simulation, rendering such outcomes possible. Having a look at some more experiments [26, Supplementary Material] shows that this red region is not stable over different simulations (which would clearly violate the sensitivity of consistent scoring functions with respect to increasing information sets established in Holzmann and Eulert [41]), but moves and occasionally also vanishes (on the region \([-5,5]^{2}\) considered). Interestingly, in all events with a red region present, this red region was located roughly in a similar area.