3.1 The Usual Stochastic Order ≤_st

Choosing a best action among {a₁, a₂}, we ought to compare the random variables X, Y in some meaningful way. Without any further restrictions on the support or distribution, we may take the usual stochastic order [22] ≤_st for that purpose, which calls X ≤_st Y if and only if (2) Condition (2) can be stated equivalently by demanding E(ϕ(X)) ≤ E(ϕ(Y)) for all increasing functions ϕ for which the expectations exist (so-called test-functions). In the latter formulation, it is easy to see that, for example, the ≤_st-ordering in particular entails E(X) ≤ E(Y), so that a comparison based on Eq (1) comes out the same under ≤_st. Moreover, in restricting X and Y to take on only positive values, as our above definition of implies, X ≤_st Y implies that all moments are in pairwise ≤-order, since the respective functions ϕ(x) = x^k delivering them are all increasing on . Under this restriction, comparisons based on the second and third moment [20] are also covered under ≤_st.

3.2 Generalizing ≤_st: The ⪯-Ordering

In cases where it is sufficient to lower risk under an acceptance threshold, rather than truly minimizing them, we may indeed relax the ≤_st-ordering in several ways: we can require Eq (2) only for large damages in (x₀, ∞) for a threshold x₀ that may be different for various application domains, or we may not use all increasing functions, but only a few selected ones (our construction will use the latter and entail the former relaxation). Given that moments are being used to analyze risks and are related to risk attitudes [20], let us take the functions ϕ(x) = x^k for , which are all increasing on . To assure the existence of all moments E(ϕ(X)) < ∞ and the monotony of all members in our restricted set of test-functions, we impose the following assumptions on a general random variable R, which we hereafter use to quantitatively model “risk”:

Definition 1. Let be the set of all random variables R, who satisfy the following conditions:

• R has a known distribution F with compact support (note that this implies that R is upper-bounded).
• R ≥ 1 (w.l.o.g., since as R is bounded, we can shift it into the region [1, ∞)).
• The probability measure induced by F is either discrete or continuous and has a density function f. For continuous random variables, the density function is assumed to be continuous.

Requirement 1 assures that all moments exist. Requirements 2 and 3 serve technical reasons that will be made clear in Lemma 2. In brief, these two assure that the ordering obtained will be total, and simplifies proofs by defining the order as equal to the natural ordering of hyperreal numbers. This will be made rigorous in Theorem 1 below. The permission to restrict our attention to moments rather than the whole random variable is given by the following well known fact:

Lemma 1. Let two random variables X, Y have their moment generating functions μ_X(s), μ_Y(s) exist within a neighborhood U_ε(0). Assume that m_X(k) := E(X^k) = E(Y^k) =: m_Y(k) for all . Then X and Y have the same distribution.

Proof (Sketch). The proof is a simple matter of combining well-known facts about power-series and moment-generating functions (see [15] for a description).

In the following, let us write m_X(k) to mean the k-th moment of a random variable X. Our next lemma establishes a total relation (so far not an ordering) between two random variables from , on which our ordering will be based:

Lemma 2. For any two probability distributions F₁, F₂ and associated random variables R₁ ∼ F₁, R₂ ∼ F₂ according to Definition 1, there is a so that either [∀k ≥ K : m_R1(k) ≤ m_R2(k)] or [∀k ≥ K : m_R1(k) ≥ m_R2(k)].

The proof of lemma 2 is given as supporting information S1 Proofs. The important fact stated here is that between any two random variables R₁, R₂, either a ≤ or a ≥ ordering holds asymptotically on the moment sequence. Hence, we can take Lemma 2 to justify the following relaxation of the usual stochastic order:

Definition 2 (⪯-Preference Relation over Probability Distributions). Let R₁, R₂ ∈ be two random variables with distribution functions F₁, F₂. We prefer R₁ over R₂, respectively the distribution F₁ over F₂, written as (3) Strict preference is denoted and defined as

For this definition to be a meaningful ordering, we need to show that ⪯ behaves like other orderings, say ≤ on the real numbers. We get all useful properties almost for free, by establishing an isomorphism between ⪯ and another well known ordering, namely the natural ≤ order on the hyperreal space :

Theorem 1. Let be according to definition 1. Assume every element X ∈ to be represented by hyperreal number , where is any free ultrafilter. Let X, Y ∈ be arbitrary. Then, X ⪯ Y if x ≤ y in , irrespectively of .

Proof. (cf. [14]) Let F₁, F₂ be two probability distributions, and let R₁ ∼ F₁, R₂ ∼ F₂. Lemma 2 assures the existence of some so that F₁ ⪯ F₂ iff m_R1(k) ≤ m_R2(k) whenever k ≥ K. Let L be the set of indices where m_R1(k) ≤ m_R2(k), then complement set is finite (it has at most K − 1 elements). Let be an arbitrary free ultrafilter. Since is finite, it cannot be contained in as is free. And since is an ultrafilter, it must contain the complement a set, unless it contains the set itself. Hence, , which implies the claim.

Theorem 1 has quite some useful implications: first, the asserted independence of the ultrafilter spares us the need to explicitly construct (note that the general question of whether or not non-isomorphic hyperreal fields would arise from different choices of ultrafilters is still unanswered by the time of writing this article). Second, the ⪯-ordering on inherits all properties (e.g., transitivity) of the natural ordering ≤ on , which by the transfer principle [16], hold in the same way as for ≤ on . More interestingly for further applications, topological properties of the hyperreals can also be transferred to . This allows the definition of a whole game theory on top of ⪯, as was started in [17]. It must be noted, however, that the ⪯-ordering still behaves different to ≤ on , since, for example, the equivalence-relation induced by ⪯ does not entail an identity between distributions (since a finite number of moments is allowed to mismatch in any case).

Interestingly, although not demanded in first place, the use of moments to compare a distribution entails a similar fact as inequality Eq (2) upon which the usual stochastic order was defined:

Theorem 2. Let X, Y ∈ have the distributions F₁, F₂. If X ⪯ Y, then there exists a threshold x₀ ∈ supp(F₁) ∪ supp(F₂) so that for every x ≥ x₀, we have Pr(X > x) ≤ Pr(Y > x).

The proof of this appears in the supporting information S1 Proofs. Intuitively, Theorem 2 can be rephrased into saying that:

If F₁ ⪯ F₂, then “extreme events” are less likely to occur under F₁ than under F₂.

Summarizing the results obtained, we can say that the ⪯-ordering somewhat resembles the initial definition of the usual stochastic order ≤_st, up to the change of restricting the range from (−∞, ∞) to a subset of [1, ∞) and in allowing a finite number of moments to behave arbitrarily. Although this allows for an explicit disregard of the first few moments, the overall effect of choosing a ⪯-minimal distribution is shifting all the probability mass towards regions of lower damages, which is a consequence of Theorem 2. As such, this result could by itself be taken as a justification to define this ordering in first place. However, in the way developed here, the construction roots in moments and their recognized relation to risk attitudes [13, 20, 21], and in the end aligns itself to both, the intuition behind ≤_st and the focus of risk management on extreme events, without ever having stated this as a requirement to begin with. Still, by converting Theorem 2 into a definition, we could technically drop the assumption of losses being ≥1. We leave this as an aisle for future research. As a justification of the restrictions as stated, note that most risk management in the IT domain is based on categorical terms (see [23–27]), which naturally map into integer ranks ≥1. Thus, our assumption seems mild, at least for IT risk management applications (applications in other contexts like insurance [28] are not discussed here and constitute a possible reason for dropping the lower bound in future work).

3.3 Distributions with Unbounded Tails

Theorem 2 tells that distributions with thin tails would be preferred over those with fat tails. However, catastrophic events are usually modeled by distributions with fat, heavy or long tails. The boundedness condition in definition 1 rules out many such distributions relevant to risk management (e.g., financial risk management [29]). Thus, our next step is extending the ordering by relaxing some of the assumptions that characterize .

The ⪯-relation cannot be extended to cover distributions with heavy tails, as those typically do not have finite moments or moment generating functions. For example, Lévi’s α-stable distributions [30] are not analytically expressible as densities or distribution functions, so the expression E(ϕ(X)) could be quite difficult to work out for the usual stochastic order. Conversely, resorting to moments, we can work with characteristic functions, which can be much more feasible in practice.

Nevertheless, such distributions are important tools in risk management. Things are, however, not drastically restricted, for at least two reasons:

1. Compactness of the support is not necessary for all moments to exist, as the Gaussian distribution has moments of all orders and is supported on the entire real line (thus violating even two of the three conditions of assumption 1). Still, it is characterized entirely by its first two moments, and thus can easily be compared in terms of the ⪯-relation.
2. Any distribution with infinite support can be approximated by a truncated distribution. Given a random variable X with distribution function F, then truncated distribution is the conditional likelihood .
   By construction, the truncated distribution has the compact support [a, b]. More importantly, for a loss distribution with unbounded support [1, ∞) and given any ε > 0, it is easy to choose a compact interval [a, b] large enough inside [1, ∞) so that for all x. Hence, restricting ourselves to distributions with compact support, i.e., adopting assumption 1, causes no more than a numerical error that can be made as small as we wish.

More interestingly, we could attempt to play the same trick as before, and characterize a distribution with fat, heavy or long tails by a sequence of approximations to it, arising from better and better accuracy ε → 0. In that sense, we could hope to compare approximations rather than the true density in an attempt to extend the preference and equivalence relations ⪯ and ≡ to distributions with fat, heavy or long tails.

Unfortunately, such hope is an illusion, as a distribution is not uniquely characterized by a general sequence of approximations (i.e., we cannot formulate an equivalent to lemma 1), and the outcome of a comparison of approximations is not invariant to how the approximations are chosen (i.e., there is also no alike for lemma 2). To see the latter, take the quantile function F⁻¹(α) for a distribution F, and consider the tail quantiles . Pick any sequence (α_n)_{n → ∞} with α_n → 0. Since lim_{x → ∞} F(x) = 1, the tail quantile sequence behaves like , where the limit is independent of the particular sequence (α_n)_{n → ∞}, but only the speed of divergence is different for distinct sequences.

Now, let two distributions F₁, F₂ with infinite support be given. Fix two sequences α_n and ω_n, both vanishing as n → ∞, and set (4) Let us approximate F₁ by a sequences of truncated distributions with supports [1, a_n] and let the sequence approximate f₂ on [1, b_n]. Since a_n < b_n for all n, it is easily verified that the sequence of moments of the distributions truncated to [1, a_n] and [1, b_n] implies that the respective moment sequences diverge so that ultimately. However, by replacing the “<” by a “>” in Eq (4), we can construct approximations to F₁, F₂ whose truncated supports overlap one another in the reverse way, so that the approximations would always satisfy . It follows that the sequence of approximations cannot be used to unambiguously compare distributions with infinite support, unless we impose some constraints on the tails of the distributions and the approximations. The next lemma (see the supporting information S1 Proofs for a proof) assumes this situation to simply not occur, which allows to give a sufficient condition to unambiguously extend strict preference in the way we wish.

Lemma 3. Let F₁, F₂ be two distributions supported on [1, ∞) with continuous densities f₁, f₂. Let be an arbitrary sequence with a_n → ∞ as n → ∞, and let for i = 1,2 be the truncated distribution f_i supported on [1, a_n].

If there is a constant c < 1 and a value such that f₁(x)<c ⋅ f₂(x) for all x ≥ x₀, then there is a number N such that all approximations satisfy whenever n ≥ N.

By virtue of lemma 3, we can extend the strict preference relation to distributions that satisfy the hypothesis of the lemma but need not have compact support anymore. Precisely, we would strictly prefer one distribution over the other, if all truncated approximations are ultimately preferable over one another.

Definition 3 (Extended Preference Relation ≺). Let F₁, F₂ be distribution functions of nonnegative random variables that have infinite support and continuous density functions f₁, f₂. We (strictly) prefer F₁ over F₂, denoted as F₁ ≺ F₂, if for every sequence a_n → ∞ there is an index N so that the approximations for i = 1,2 satisfy whenever n ≥ N.

The ≻-relation is defined alike, i.e., the ultimate preference of F₂ over F₁ on any sequence of approximations.

Definition 3 is motivated by the above arguments on comparability on common supports, and lemma 3 provides us with a handy criterion to decide the extended strict preference relation.

Example 1. It is a matter of simple algebra to verify that any two out of the three kinds of extreme value distributions (Gumbel, Frechet, Weibull) satisfy the above condition, thus are strictly preferable over one another, depending on their particular parametrization.

Definition 3 can, however, not applied to every pair of distributions, as the following example shows.

Example 2. Take the “Poisson-like” distributions with parameter λ > 0, It is easy to see that no constant c < 1 can ever make f₁ < c ⋅ f₂ and that all moments exist. However, neither distribution is preferable over the other, since finite truncations to [1, a_n] based on the sequence a_n := n will yield alternatingly preferable results.

An occasionally simpler condition that implies the hypothesis of definition 3 is (5) The reason is simple: if the condition of definition 3 were violated, then there is an infinite sequence for which f₁(x_n) ≥ c ⋅ f₂(x_n) for all c < 1. In that case, there is a subsequence for which lim_{k → ∞} f₁(x_nk)/f₂(x_nk) ≥ c. Letting c → 1, we can construct a further subsequence of to exhibit that limsup_{n → ∞}(f₁(x_n)/f₂(x_n)) = 1, so that Condition (5) would be refuted. Observe that Eq (5) is similar to the definition of a likelihood ratio order [22] in the sense that it implies both, a likelihood ratio and ≺-ordering. Note that, however, a likelihood ratio order does not necessarily imply a ≺-order, since the former only demands f(t)/g(t) to be increasing, but not a <-relation among the densities.

Remark 1. It must be emphasized that the above line of arguments does not provide us with a mean to extend the ⪯- or ≡-relations accordingly. For example, an attempt to define ⪯ and ≡ as above is obviously doomed to failure, as asking for two densities f₁, f₂ to satisfy f₁(x) ≤ c₁ ⋅ f₂(x) ultimately (note the intentional relaxation of < towards ≤), and f₂(x) ≤ c₂ ⋅ f₁(x) ultimately for two constants c₁, c₂ < 1 is nonsense.

A straightforward extension of ⪯ can be derived from (based on) the conclusion of lemma 3:

Definition 4. Let F₁, F₂ be two distributions supported on the entire nonnegative real half-line with continuous densities f₁, f₂. Let be a diverging sequence towards ∞, and let for i = 1, 2 denote the density F_i truncated to have support [1, a_n]. We define F₁ ⪯ F₂ if and only if for every sequence there is some index N so that for every n ≥ N.

More compactly and informally spoken, definition 4 demands preference on all approximations with finite support except for at most finitely many exceptions near the origin.

Obviously, preference among distributions with finite support implies the extended preference relation to hold in exactly the same way (since the sequence of approximations will ultimately become constant when a_n overshoots the bound of the support), so definition 4 extends the ⪯-relation in this sense.

3.4 Comparing Distributions of Mixed Type

The representation of a distribution by the sequence of its moments is of the same form, for discrete, categorical and continuous random variables. Hence, working with sequence representations (hyperreal numbers) admits to compare continuous to discrete and categorical variables, as long as there is a meaningful common support. The framework itself, up to the results stated so far, remains unchanged and is applied to the category’s ranks instead. The ranking is then made in ascending order of loss severity, i.e., the category with lowest rank (index) should be the one with the smallest damage magnitude (examples are found in IT risk management standards like ISO 27005 [31] or the more generic ISO 31000 [32] as well as related standards).

A comparison of mixed types is, obviously, only meaningful if the respective random variables live in the same (metric) space. For example, it would be meaningless to compare ordinal to numeric data. Some applications in natural risk management define categories as numeric ranges (such as [23–26]), which could make a comparison of categories and numbers meaningful (but not necessarily so).

4.1 Deciding ⪯ between Categorical Variables

Let F₁, F₂ be two distributions over a common support, i.e., a common finite set of categories, hereafter denoted in descending order as c₁ > c₂ > … > c_n. Let be the corresponding probability mass functions. For example, these can be normalized histograms (empirical density functions) computed from the available data to approximate the unknown distributions F₁, F₂ of the random variables X, Y.

Letting the category c_i correspond to its rank n − i + 1 within the support, it is easy to check that the expectation of X ∼ F₁, Y ∼ F₂ by definition is a sequence whose growth is determined by whichever distribution puts more mass on categories of high loss. Formally, if p₁ > q₁, then , since the growth of either sum is determined by the largest term (here being ). Upon the equality p₁ = q₁, we can retract the respective terms from both sums (as they are equal), to see whether the second-largest term tips the scale, and so on.

Overall, we end up observing that ⪯-comparing distributions is quite simple, and a special case of another common ordering relation:

Definition 5 (lexicographic ordering). For two real-valued vectors x = (x₁, x₂, …) and y = (y₁, y₂, …) of not necessarily the same length, we define x<_lex y if and only if there is an index i₀ so that x_i0 < y_i0 and x_i = y_i whenever i < i₀.

Our discussion from above is then the mere insight that the following is true:

Theorem 3. Let F₁, F₂ be two categorical random variables with a common ordered support Ω = {c₁ > c₂ > … > c_n}, and let f₁,f₂ be the respective (empirical) density functions. Then F₁ ⪯ F₂ ⇔ f₁<_lex f₂, where .

For illustration, we will apply Theorem 3 to two concrete example data sets #1 and #2 in section 5.

4.2 Deciding ⪯ between Continuous Variables

Let us assume that the two random variables R₁ ∼ F₁, R₂ ∼ F₂ have smooth densities f₁, f₂ ∈ C^∞([1, a]) for some a > 1. Under this assumption, we can switch to yet another useful sequence representation: (6) Given two distributions f₁, f₂ ∈ C^∞, e.g., constructed from a Gaussian kernel (cf. remark 2 below), let the respective representations according to Eq (6) be f₁,f₂. Then, it turns out that the lexicographic ordering of f₁,f₂ implies the same ordering w.r.t. ⪯, or formally:

Lemma 4 ([15]). Let f, g ∈ C^∞([1, a]) for a real value a > 1 be probability density functions. If then f ⪯ g.

Lemma 4 will be demonstrated on our example data set #3, in connection with a kernel density estimate, in section 5. Practically, we can thus decide the ⪯-relation by numerically computing derivatives of increasing order, until the decision is made by the lexicographic ordering (which, for our experiments, happened already at zeroth order in many cases).

Remark 2. The assumption on differentiability is indeed mild, as we can cast any integrable density function into a C^∞-function by convolution with a Gaussian density k_h with zero mean and variance h. Clearly, f ∗ k_h ∈ C^∞ by the differentiation theorem of convolution. Moreover, letting h → 0, we even have L¹-convergence of f ∗ k_h → f, so that the approximation can be made arbitrarily accurate by choosing the parameter h > 0 sufficiently small. Practically, when the distributions are constructed from empirical data, the convolution corresponds to a kernel density estimation (i.e., a standard nonparametric distribution model). Using a Gaussian kernel then has the additional appeal of admitting a closed form of the k-th derivatives (f ∗ k_h)^(k), involving Hermite-polynomials.

4.3 Comparing Deterministic to Random Effects

In certain occasions, the consequence of an action may result in perfectly foreseeable effects, such as fines or similar. Such deterministic outcomes can be modeled as degenerate distributions (point- or Dirac-masses). These are singular and thus outside by Definition 1. Note that the canonic embedding of the reals within the hyperreals represents a number by the constant sequence (a, a, …). Picking up this idea would be critically flawed in our setting, as any such constant sequence would be preferred over any probability distribution (whose moment sequence diverges and thus overshoots a inevitably and ultimately).

However, it is easy to work out the moment sequence of the constant X = a as E(X^k) = E(a^k) = a^k for all . In this form, the ⪯-relation between the number a and the continuous random variable Y supported on Ω = [1, b] can be decided as follows:

1. If a < b, then a ⪯ Y: to see this, choose ε < (b − a)/3 so that f is strictly positive on a compact set [b − ε, b − 2ε] (note that such a set must exist as f is continuous and the support ranges until b). We can lower-bound the k-th moment of Y as Note that the infimum is positive as f is strictly positive on the compact set [b − 2ε, b − ε]. The lower bound is essentially an exponential function to a base larger than a, since b − 2ε > a, and thus (ultimately) grows faster than a^k.
2. If a > b, then Y ⪯ a, since Y – in any possible realization – leads to strictly less damage than a. The formal argument is now based on an upper bound to the moments, which can be derived as follows: It is easy to see that for k → ∞, this function grows slower than a^k as a > b, which leads to the claimed ⪯-relation.
3. If a = b, then we apply the mean-value theorem to the integral occurring in to obtain an ξ ∈ [1, a] for which for all k. Hence, Y ⪯ a in that case. An intuitive explanation stems from the fact that Y may assign positive likelihood to events with less damage as a, whereas a deterministic outcome is always larger or equal to anything that Y can deliver.

4.4 On the Interpretation of ⪯ and Inference

The practical meaning of the ⪯-preference is more involved than just a matter of comparing the first few moments. Indeed, unlike for IT risk preferences based on Eq (1), the first moment can be left unconstrained while ⪯ may still hold in either direction.

For general inference, the comparison of two distributions provides a necessary basis (i.e., to define optimality, etc.). For example, (Bayesian) decision theory or game theoretic models can be defined upon ⪯, via a much deeper exploration of the embedding of into the hyperreals (by mapping a distribution to its moment sequence), such as the induced topology and calculus based on it. In any case, however, we note that the previous results may help in handling practical matters of ⪯ inside a more sophisticated statistical decision or general inference process. For practical decisions, some information can be obtained from the value x₀ that Theorem 2 speaks about. This helps assessing the meaning of the order, although the practical consequences implied by ≤_st or ⪯ are somewhat similar. The main difference is Eq (2) holding only for values ≥x₀ in case of ⪯. The threshold can hence be found by numerically searching for the largest (“right-most”) intersection point of the respective survival functions; that is, for two distributions F₁ ⪯ F₂, a valid x₀ in Theorem 2 is any value for which 1 − F₁(x) ≤ 1 − F₂(x) for all x ≥ x₀. An approximation of x₀, e.g., computed by a bisective search in common support of both distributions, then more accurately describes the “statistically best” among the available actions, since losses >x₀ are more likely for all other options. A practical decision, or more general inference based on ⪯, should therefore be made upon computing x₀ as an explicit auxiliary information, in order to assign a quantitative meaning to “extreme events” in the interpretation underneath Theorem 2. Further issues of practical decision making in the context of IT risk management are discussed along the first empirical example found in section 5.2.

Section 5 will not discuss (statistical) inference since the details are beyond the scope of this work (we leave this to follow up work). Instead, the following section will be dedicated to numerical illustrations of ⪯ only, without assigning any decisional meaning to the ⪯-preferred distributions. For each example, we will also give an approximation (not the optimal) value of x₀.

5.1 Comparing Parametric Models

We skip the messy algebra tied to the verification of the criteria in Section 3.3, and instead compute the moments numerically to illustrate the growth/divergence of moment sequences as implied by Lemma 2.

Example 3 (different mean, same variance). Consider two Gumbel-distributions X ∼ F₁ = Gumbel(31.0063, 1.74346) and Y ∼ F₂ = Gumbel(32.0063, 1.74346), where a density for Gumbel(a, b) is given by where and b > 0 are the location and scale parameter.

Computations reveal that under the given parameters, the means are E(X) = 30, E(Y) = 31 and Var(X) = Var(Y) = 5. Fig 1 plots the respective densities of F₁ (dashed) and F₂ (solid line). The respective moment sequences evaluate to thus illustrating that F₁ ⪯ F₂. This is consistent with the intuition that the preferred distribution gives less expected damage. The concrete region about which Theorem 2 speaks is at least for damages >x₀ = 25 (cf. Theorem 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Comparing distributions with different means.

https://doi.org/10.1371/journal.pone.0168583.g001

Example 4 (same mean, different variance). Let us now consider two Gumbel-distributions X ∼ F₁ = Gumbel(6.27294, 2.20532) and Y ∼ F₂ = Gumbel(6.19073, 2.06288), for which E(X) = E(Y) = 5 but Var(X) = 8 > Var(Y) = 7.

Fig 2 plots the respective densities of F₁ (dashed) and F₂ (solid line). The respective moment sequences evaluate to thus illustrating that F₂ ⪯ F₁. This is consistent with the intuition that among two actions leading to the same expected loss, the preferred one would be one for which the variation around the mean is smaller; thus the loss prediction is “more stable”. The range on which damages under F₂ are less likely than under F₁ begins at x > x₀ ≈ 5.5 (cf. Theorem 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparing distributions with equal means but different variance.

https://doi.org/10.1371/journal.pone.0168583.g002

Example 5 (different distributions, same mean and variance). Let us now consider a situation in which the expected loss (first moment) and variation around the mean (second moment) are equal, but the distributions are different in terms of their shape. Specifically, let X ∼ F₁ = Gamma(260.345, 0.0373929) and Y ∼ Weibull(20, 10), with densities as follows:

Fig 3 plots the respective densities of F₁ (dashed) and F₂ (solid line). The respective moment sequences evaluate to thus illustrating that F₂ ⪯ F₁. In this case, going with the distribution that visually “leans more towards lower damages” would be flawed, since F₁ nonetheless assigns larger likelihood to larger damages. The moment sequence, on the contrary, unambiguously points out F₂ as the preferred distribution (the third moment tips the scale here; cf. [13, 20, 21]). The statistical assurance entailed by Theorem 2 about an interval in which high damage incidents are less likely (at least) includes losses >x₀ ≈ 10.3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comparing distributions with matching first two moments but different shapes.

https://doi.org/10.1371/journal.pone.0168583.g003

5.2 Empirical Test Data and Methodology

To demonstrate how the practical matters of comparing distributions work, we will use three sets of empirical data, based on qualitative data from risk estimation, and based on simulating a malware outbreak using percolation.

Test Data Set #1 – IT Risk Assessments.

The common quantitative understanding of risk by the Formula (1) is easily recognized as the expectation (i.e., first moment) of a loss distribution. Although being standard in quantitative IT risk management, its use is discouraged by the German Federal Office of Information Security (BSI) [33] for several reasons besides the shortcomings that we discussed here (for example, statistical data may be unavailable at the desired precision and an exact formula like Eq (1) may create the illusion of accuracy where there is none [33]).

Best practices in risk management (ranging up to norms like the ISO27005 [31], the ISO31000 [32] or the OCTAVE Allegro framework [34]) usually recommend the use of qualitative risk scales. That is, the expert is only asked to utter an opinion about the risk being “low/medium/high” or perhaps using a slightly more fine-grained but in any case ordinal scale. In a slight abuse of formalism, these categories are then still carried into an evaluation of Eq (1) (cf. [35]) towards finding the decision with the “least” risk in qualitative terms as Eq (1) gives.

Categorical risk assessments are heavily used in the IT domain due to their good systematization and tool support. Our first test data set is thus a risk assessment made in terms of the Common Vulnerability Scoring System (CVSS) [27]. The CVSS ranks risks on a scale from 0 to 10, as a decimal rounded up to one place behind the comma. Usually, these CVSS values come from domain experts, so there is an intrinsic ambiguity in the opinions on grounds of which a decision shall be made. Table 1, taken from [36] (by kind permission of the author A. Beck), shows an example of such expert data for two security system installments being assessed by experts in the left and right part of the table (separated by the double vertical line). The ⪯-ordering shall now help to choose the better of the two options, based on the ambiguous and even inconsistent domain expert inputs. For simplicity of the example, we did not work with the fine-grained CVSS scores, but coarsened them into three categories, i.e., intervals of scores low = [0, 3) (L), medium = [4, 8) (M) and high = [8, 10] (H). We remark that the categorial assessment was added in this work, and is not from the source literature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Example CVSS Risk Assessment [36].

https://doi.org/10.1371/journal.pone.0168583.t001

Test Data Set #2 – Malware Outbreaks.

Computer malware infections are continuously reported in the news, with an early and prominent example having been the Stuxnet worm in 2008 [37], which infected the Iranian uranium enrichment facilities. Ever since, the control and supervision of cyber-physical systems has gained much importance in risk management, since attacks on the computer infrastructure may have wide effects ranging up to critical supply infrastructures such as water supply, power supply, and many others (e.g., oil, gas or food supply networks, etc.).

The general stealthiness of such infections makes an exact assessment of risk difficult. A good approach to estimate that risk is to apply outbreak simulation models, such as, for example, using percolation theory [38, 39]. These simulations provide us with possible infection scenarios, in which the number of infected nodes (after a fixed period of time), can be averaged into a probability distribution describing the outcome of an infection. Repeating the simulation with different system configurations yields various outcome distributions. An example for a network with 20 nodes and 1000 repetitions per simulation is displayed in Table 2. The ⪯-relation shall then help deciding which configuration is better in minimizing the risk of a large outbreak.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Simulated malware infection.

https://doi.org/10.1371/journal.pone.0168583.t002

5.3 Comparing Empirical Distributions

With the three data sets as described, let us now look into how decisions based on the empirical data can be made.

Categorical Data – Comparing Normalized Histograms.

Compiling an empirical distribution from the example CVSS data in Table 1 gives the histograms shown in Fig 4. Clearly, scenario 1 is preferable here, as it is less frequently rated with high damage than scenario 2. On the contrary, the decision is much less informed than in the case where the full numeric data would have been used. We will thus revisit this example later again.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparing Empirical Distributions – Test Data #1 in Nominal Scale.

Outcome: “scenario 1” ≺ “scenario 2” (i.e., scenario 1 has lower security risk, based on the coarsened data), for medium and high categories (x₀ = ‘M’).

https://doi.org/10.1371/journal.pone.0168583.g004

For the simulated malware infection data in Table 2, the empirical distribution of the number of affected nodes as shown in Fig 5 is obtained by normalization of the corresponding histograms. In this case, configuration 2 is preferable as the maximal damage of 20 node has occurred less often than in configuration 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Test Data #2 – (Simulated) Empirical Distribution of Malware Outbreak Sizes under two Configurations.

Outcome: “config. 2” ≺ “config. 1” (i.e., the second configuration is more secure w.r.t. extreme outcomes, i.e., infections of >x₀ = 9 nodes).

https://doi.org/10.1371/journal.pone.0168583.g005

Continuous Data – Comparing Kernel Density Estimates.

If the data itself is known to be continuous, then a nonparametric distribution estimate can be used to approximate the unknown distribution.

In the following, let us write to mean a general kernel density estimate based on the data (observations) x₁, …, x_n, of the form (7) where K(x) is the chosen kernel function, and h > 0 is a bandwidth parameter, whose choice is up to any (of many existing) heuristics (see [40, 41] among others). Computing a kernel density estimate from data is most conveniently done by invoking the density command within the R statistical computing software [42].

This comparison of two kernel density estimates (KDE) is illustrated in Fig 6. For that purpose, we divided the test dataset #3 (data(Nile) in R) into observations covering the years 1871-1920 and 1921-1970. For both sets the density is estimated with a Gaussian kernel and the default bandwidth choice nrd0 (Silverman’s rule [41]) yielding a KDE with bandwidth h₁ = 79.32 for the years 1871-1920 and a KDE with bandwidth h₂ = 45.28 for the years 1921-1970. Further we have the maximal observed values x_n1 = 1370 and y_n2 = 1170, and see that Therefore (and also by visual inspection of Fig 6), the density for the period from 1871 until 1920 has had higher likelihoods for a high water level, which became less in the period from 1921-1970, thus indicating a “down-trend” by .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of two kernel density estimates – Test Data #3 (Nile Water Level).

Outcome: “1921-1970” ≺ “1871-1920” (indicating that floodings at a water level >x₀ ≈ 220 (Theorem 2) have been more likely in the years before 1921).

https://doi.org/10.1371/journal.pone.0168583.g006

Using Gaussian Kernels.

Commonly, the kernel density approximation is constructed using Gaussian kernels per default, in which case K takes the form . Definition 1 is clearly not met, but there is also no immediate need to resort to the extended version of ⪯ as given by Definition 4. Indeed, if we simply truncate the KDE at any point a > 1 into the distribution and remember that K ∈ C^∞, the truncated kernel density estimate is again a C^∞-density, as required by Lemma 4.

Returning to the CVSS example data in Table 1, we constructed Gaussian kernel density estimates f₁, f₂, with bandwidths h₁ ≈ 0.798 and h₂ ≈ 0.346 (using the default Silverman’s rule in R); plots of which are given in Fig 7. Using the criterion of Lemma 4 in connection with the lexicographic ordering, we end up finding that f₂ ⪯ f₁, in contrast to our previous finding. This is, however, only an inconsistency at first glance, and nevertheless intuitively meaningful if we consider the context of the decision and the effect of the nonparametric estimation more closely:

• Since scenario 2 is based on more data than scenario 1, the bandwidth h₂ is less than h₁. This has the effect of the distribution being “more condensed” around higher categories, as opposed to the distribution for scenario 1, whose tail is much thicker. Consequently, the decision is to prefer scenario 2 is implicitly based on the larger data set, and considers the higher uncertainty in the information about scenario 1.
• The Gaussian kernel has tails reaching out to +∞, which also assigns positive mass to values outside the natural range of the input data (1…10 in case of CVSS). In many contexts, observations may not be exhaustive for the possible range (e.g., monetary loss up to the theoretical maximum may – hopefully – not have occurred in a risk management process in the past). By construction, the KDE puts more mass on the tails the more data in this region is available. From a security perspective, this mass corresponds to zero-day exploit events. Thus, such incidents are automatically accounted for by ⪯.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Comparing Nonparametric Distribution Models – Test Data #1 (directly used).

Outcome: “scenario 2” ≺ “scenario 1” (given a more refined view that in Fig 4, scenario 1 has less security).

https://doi.org/10.1371/journal.pone.0168583.g007