On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness

As simple as it sounds, the first option is not to distribute any weights to the indicators, otherwise called an ‘attributes-based weighting system’ (see e.g. Slottje 1991, pp. 686–688). This system may have two consequences. First, the overall score (index) could simply be the non-weighted arithmetic average of the normalised indicators (Booysen 2002; Singh et al. 2009; Karagiannis 2017). A common problem that appears here, though, is that of ‘double counting’³ (Freudenberg 2003; OECD 2008). Of course, this issue might partially be moderated by averaging the collinear indicators as well prior to their aggregation into a composite (Kao et al. 2008). The second alternative in the absence of weights is that the composite index is equal to the sum of the individual rankings that each unit obtains in each of the sub-indicators (e.g. see the Information and Communication Technologies Index in Saisana and Tarantola 2002, p. 9). By relying solely on aggregating rankings, this approach fails to achieve the purpose of vastly improving the statistical information, as it does not benefit from the absolute level of information of the indicators (Saisana and Tarantola 2002).

Equal weighting is the most common scheme appearing in the development of composite indicators (Bandura 2008; OECD 2008). It is important to note here that the difference between distributing equal weights and not distributing weights at all (e.g. the ‘non-weighted arithmetic average’ discussed above) is that equal weighting schemes could be applied hierarchically. More specifically, if the indicators are grouped into a higher order (e.g. a dimension) and the weighting is distributed equally dimension-wise, then it does not necessarily mean that the individual indicators will have equal weights (OECD 2008). For instance, ISTAT (2015) provides the ‘BES’, a broad data set of 134 socio-economic indicators for the 20 Italian regions. These are unevenly grouped into 12 dimensions. If equal weights are applied to the highest hierarchy level (e.g. dimensions) a priori, then the sub-indicators are not weighted equally due to the different number of indices in each dimension. In general, there are various justifications for most applications choosing equal weights a priori. These include: (1) simplicity of construction, (2) a lack of theoretical structure to justify a differential weighting scheme, (3) no agreement between decision makers, (4) inadequate statistical and/or empirical knowledge, and, finally, (5) alleged objectivity (see Freudenberg 2003; OECD 2008; Maggino and Ruviglioni 2009; Decancq and Lugo 2013). Nevertheless, it is often found that equal weighting is not adequately justified (Greco et al. 2017). For instance, choosing equal weights due to the ‘simplicity of the construction’,⁴ instead of an alternative scheme that is based on a proper theoretical and methodological framework, bears a huge oversimplification cost, especially in certain aggregation schemes (Paruolo et al. 2013). Furthermore, we could argue that, conceptually, equal weights miss the point of differentiating between essential and less important indicators by treating them all equally. In any case, the co-operation of experts and the public in an open debate might resolve the majority of the aforementioned justifications (Freudenberg 2003). Finally, considering equal weights as an ‘objective’ technique (relative to the ‘subjective’ exercise of a developer who sets the weights arbitrarily) is far from being undisputable. Quoting Chowdhury and Squire (2006, p. 762), setting weights to be equal ‘[seems] obviously convenient, but also universally considered to be wrong’. Ray (2008, p. 5) and Mikulić et al. (2015) claim that equal weighting is not only wrong—as it does not convey the realistic image—but also an equally ‘subjective judgement’ to other arbitrary weighting schemes in existence. This last argument prepares the scene for the consideration of a plurality of weighting systems, mainly related to the representation of the preferences of a ‘plurality of individuals’ (see e.g. Greco et al. 2017).

Understandably, the decision maker could choose from a range of weighting schemes, depending on the structure and quality of the data or her beliefs. More specifically, in the first case, higher weighting could be assigned to indicators with broader coverage (as opposed to those with multiple cases of treated missing data) or those taken from more trustworthy sources, as a way to account for the quality of the indicators (Freudenberg 2003). However, an issue here is that this could result in a ‘biased selection’ in favour of proxies that are not able to identify and capture properly the information desired to measure (see e.g. Custance and Hillier 1998, pp. 284–285; OECD 2008, p. 32). Moreover, indicators should be chosen carefully a priori and according to a conceptual and quality framework (OECD 2008). Otherwise, a ‘garbage in–garbage out’ outcome may be produced (Funtowicz and Ravetz 1990), which in this case is that of a composite indicator reflecting ‘insincere’ dimensions in relation to those desired (Munda 2005a). When the weighting scheme is chosen by the developer of an index, naturally this means that it is conceived as ‘subjective’, since it relies purely on the developer’s perceptions (Booysen 2002). There are several participatory approaches in the literature to make this subjective exercise as transparent as possible. These involve a single or several stakeholders deciding on the weighting scheme to be chosen. Stakeholders could be expert analysts, policymakers, or even citizens to whom policies are addressed. From a social viewpoint, the combination of all of them in an open debate could be an ideal approach theoretically (Munda 2005b, 2007),⁵ but it is only viable if a well-defined framework for a national policy exists (OECD 2008). Indeed, if one could imagine a framework on which policies will be based, enlarging the set of decision makers to include all the participants’ preferences is probably the desired outcome (Munda 2005a). However, if the objective is not well defined or the number of indicators is very large and it is probably impossible to reach a consensus about their importance, this procedure could result in an endless debate and disagreement between the participants (Saisana and Tarantola 2002). Moreover, if the objective involves an international comparison, in which no doubt the problem is significantly enlarged, common ground is even harder to achieve or simply ‘inconsistent outcomes’ may be produced (OECD 2008, p. 32). For instance, one country’s most important objective could be different from another country’s (e.g. economy vs environment). In general, participatory methods are seen as a conventional way for transparent and subjective judgements, and they could be effective and of great use when they fulfil the aforementioned requirements. However, since these techniques may yield alternative weighting schemes (Saisana et al. 2005),⁶ one should carefully choose the most suitable according to their properties, of which we provide a brief overview in the following subsections.

In the aftermath of participatory approaches, this ‘subjectivity’ behind the arbitrariness in decision makers’ weight selection is dismissed by other statistical methods that claim to be more ‘objective’.¹⁰ This property is increasingly claimed to be desirable in the choice of weights (Ray 2008), thus stirring up interest in approaches like correlation analysis or regression analysis, principal component analysis (PCA) or factor analysis (FA), and data envelopment analysis (DEA) models and their variations. These so-called ‘data-driven techniques’ (Decancq and Lugo 2013, p. 19), as their name suggests, emerge from the data themselves under a specific mathematical function. Therefore, it is often argued that they potentially do not suffer from the aforementioned problems of ‘manipulation’ of the results (Grupp and Mogee 2004, p. 1382) and the subjective, direct weighting exercise of various decision makers (Ray 2008, p. 9). However, these approaches bear a different kind of criticism, deeply rooted in the core of their philosophy. More specifically, Decancq and Lugo (2013, p. 9) distinguish these techniques from the aforementioned ones based on the ‘is–ought’ distinction that is found in the work of a notable philosopher of the eighteenth century, David Hume. In the authors’ words: ‘it is impossible to derive a statement about values from a statement about facts’ (p. 9). In other words, they claim that one should be very cautious in deriving the importance of a concept (e.g. indicator/dimension) based on what the data ‘consider’ to be a fact, as this appears to be the ‘is’ that we observe but not the ‘ought’ that we are seeking. After all, statistical relationships between indicators—for example in the form of correlation—do not always represent the actual influence between them (Saisana and Tarantola 2002). This appears to be one side of the criticism that these approaches receive, and it is related to the philosophical aspect underlying their use. Further criticism appearing in the literature is focused on their specific properties, which we will examine individually in the following subsections.

Among the compensatory aggregation approaches, the linear one is the most commonly used in composite indicators (Saisana and Tarantola 2002; Freudenberg 2003; OECD 2008; Bandura 2008, 2011). Two general issues must be considered in this additive utility-based approach. The first is that it assumes ‘preferential independence’ among indicators (OECD 2008, p. 103; Fusco 2015, p. 621), something that is conceptually considered as a very strong assumption to make (Ting 1971). Second, there is a chasm between the two perceptions of weights, translated into importance measures and trade-offs. More specifically, if one sets the weights by considering them as importance measures for the indicators, one will soon find that this is far from actually happening in this aggregation setting, and this situation is the norm rather than the exception (Anderson and Zalinski 1988; Munda and Nardo 2005; Billaut et al. 2010; Rowley et al. 2012; Paruolo et al. 2013). Quoting the latter (p. 611): ‘This gives rise to a paradox, of weights being perceived by users as reflecting the importance of a variable, where this perception can be grossly off the mark’. This happens because the weights in this setting should be perceived as trade-offs between pairs of indicators and therefore assigned as such from the very beginning. Decancq and Lugo (2013) stress this point by showing how weights in this setting express the marginal rates of substitution among pairs of indicators. Understandably, this trade-off implies constant compensability between indicators and dimensions; thus, a unit could compensate for the loss in one dimension with a gain in another (OECD 2008; Munda and Nardo 2009). This, however, is far from desirable in certain cases. For instance, Munda (2012, p. 338) considers an example of a hypothetical sustainability index, in which economic growth could compensate for a loss in the environmental dimension in the case of a compensatory approach. Of course, this argument could easily be extended to applications in other socio-economic areas,²³ albeit with the following point: constant compensation is always assumed in linear aggregation at the rate of substitution among pairs of indicators (e.g. w_a/w_b) (Decancq and Lugo 2013, p. 17). That is something that should be taken into consideration at the very beginning of the construction stage, the theoretical framework (OECD 2008).

One partial solution to that issue could be to use geometric aggregation instead. This approach is adopted when the developer of an index prefers only ‘some’ degree of compensability (OECD 2008, p. 32). While linear aggregation assumes constant trade-offs for all cases, geometric aggregation offers inferior compensability for indices with lower values (diminishing returns) (van Puyenbroeck and Rogge 2017). This makes it far more appealing in a benchmarking exercise in which, for instance, regions with lower scores in a given dimension will not be able to compensate fully in other dimensions (Greco et al. 2017). Moreover, the same regions could be even more motivated to increase their lower scores, as the marginal increase in these indicators will be much higher in contrast to regions that already achieve high scores (Munda and Nardo 2005). Therefore, under these circumstances, a switch from linear to geometric aggregation could even be considered both appealing and more realistic. One such case is that of probably the most well-known composite index to date, the Human Development Index (HDI). Having received paramount criticism (Desai 1991; Sagar and Najam 1998; Chowdhury and Squire 2006; Ray 2008; Davies 2009), the developers of the HDI switched the aggregation function from linear to geometric in 2010, addressing one of their main methodological criticisms. More specifically, in their yearly report (UNDP 2010, p. 216), they state the following: ‘It thus addresses one of the most serious criticisms of the linear aggregation formula, which allowed for perfect substitution across dimensions’. There is no doubt that, compared with the linear type of aggregation, geometric is the solid first step towards a solution to the issue of an index’s compensability. In fact, it is argued that, under such circumstances, it provides more meaningful results (see e.g. Ebert and Welsch 2004). However, this still appears to be only a partial solution or a ‘trade-off’ between compensatory and non-compensatory techniques (Zhou et al. 2010, p. 171). Therefore, if complete ‘inelasticity’ of compensation, or the meaning of weights to be interpreted solely as ‘importance coefficients’, is the actual objective of a composite index, a non-compensatory approach is ideal and strongly suggested to be reconsidered (Paruolo et al. 2013, p. 632).

Non-compensatory aggregation techniques (Vansnick 1990; Vincke 1992; Roy 1996) are mainly based on ELECTRE methods (see e.g. Figueira et al. 2013, 2016) and PROMETHEE methods (Brans and Vincke 1985; Brans and De Smet 2016). Given the weights for each criterion (interpreted as ‘importance coefficients’ in this exercise) and some other preference parameters (e.g. indifference, preference, and veto thresholds), the mathematical aggregation is divided into the following steps: (1) ‘pair-wise comparison of units according to the whole set of indicators’ and (2) ‘ranking of units in a partial, or complete pre-order’ (Munda and Nardo 2009, p. 1516). The first step creates the ‘outranking matrix’²⁴ (Roy and Vincke 1984), which essentially discloses the pairwise comparisons of the alternatives (e.g. countries) for each criterion (Munda and Nardo 2009). Moving to the second step (i.e. the exploitation procedure of the outranking matrix), an approach must be selected regarding the proper aggregation. The exploitation procedures can mainly be divided into the Condorcet- and the Borda-type approach (Munda and Nardo 2003). These two are radically different²⁵ and as such yield different results (Fishburn 1973). Moulin (1988) argues that the Borda-type approach is ideal when just one alternative should be chosen. Otherwise, the Condorcet-type approach is the most ‘consistent’ and thus the most preferable for ranking the considered alternatives (Munda and Nardo 2003, p. 10). A big issue with the Condorcet approach, though, is that of the presence of cycles,²⁶ the probability of which increases with both the number of criteria and the number of alternatives to be evaluated (Fishburn 1973). A large amount of work has been carried out with the aim of providing solutions to this issue (Kemeny 1959; Young and Levenglick 1978; Young 1988). A ‘satisfying’ one is for the ranking of alternatives to be obtained according to the maximum likelihood principle,²⁷ which essentially chooses as the final ranking the one with the ‘maximum pair-wise support’ (Munda 2012, p. 345). While this approach enjoys ‘remarkable properties’ (Saari and Merlin 2000, p. 404), one drawback is that it is computationally costly, making it unmanageable when the number of alternatives increases considerably (Munda 2012). Nevertheless, the C–K–Y–L approach is of great use for the concept of a non-compensatory aggregation scheme, and it could be used as a solid alternative solution to the common practice of linear aggregation schemes. Munda (2012) applies this approach to the case of the Environmental Sustainability Index (ESI), produced by Yale University and Columbia University in collaboration with the World Economic Forum and the European Commission (Joint Research Centre). According to the author, there are noticeable differences in the rankings between the two approaches (linear and non-compensatory), mostly apparent in the countries ranked among the middle positions and less apparent among those ranked first or last.

Despite its desirable properties, judging from the number of applications existing in this literature, the non-compensatory multi-criteria approach (NCMCA) is not met hugely popular. This could be attributed to the simplicity of construction of other methods (e.g. linear or geometric aggregation) or the issue of being computationally costly to calculate. Furthermore, NCMCA approaches are so far used to provide the developer with a ranking of the units evaluated; thus, one can only follow the rankings through time (Saltelli et al. 2005, p. 364), swapping the absolute level of information in possession with an ordinal scale. Despite these drawbacks, Paruolo et al. (2013, p. 631) urge developers to reflect on the cost of oversimplification that other techniques bear (e.g. linear), and, whenever possible, to use NCMC approaches, in which the weights exhibit the actual importance of the criteria. Otherwise, the authors suggest that the developers of an index should inform the audience to which the index is targeted that, in the other settings (e.g. linear or geometric aggregation), weights express the relative importance of the indicators (trade-offs) and not the nominal ones that were originally assigned.

Owing to the unresolved issues of choosing a weighting and an aggregation approach, several methodologies appear in the literature, dealing with these steps in different manners. These methodologies are hybrid in the sense that they do not particularly fit into one category or the other both weighting- and aggregation-wise. This is because they use a combination of different approaches to solving the aforementioned issues. These are discussed further below.

Robustness analysis is usually accomplished through ‘uncertainty analysis’, ‘sensitivity analysis’, or their ‘synergistic use’ (Saisana et al. 2005, p. 308). These are characterised as the ‘traditional techniques’ (Permanyer 2011, p. 308). Putting it simply, uncertainty analysis (UA) refers to the changes that are observed in the final outcome (viz. the composite index value) from a potentially different choice made in the ‘inputs’ (viz. the stages to construct the composite index). On the other hand, sensitivity analysis (SA) measures how much variance of the overall output is attributed to those uncertainties (Saisana et al. 2005). It is often seen that these two are treated separately, with UA being the most frequent kind of robustness used (Freudenberg 2003; Dobbie and Dail 2013). However, both are needed to give the developer, and the audience to which the index is referred, a better understanding.²⁹ By solely applying uncertainty analysis, the developer may observe how the performance of a unit (e.g. ranking) deviates with changes in the steps of the construction phase. This is usually illustrated in a scatter plot, with the vertical axis exhibiting the country performance (e.g. ranking) and the horizontal axis exhibiting the input source of uncertainty being tested for (e.g. alternative weighting or aggregation scheme) (OECD 2008). To gain a better understanding, however, it is also important to identify the portion of this variation in the rankings that is attributed to that particular change. For instance, is it the weighting scheme that mainly changes the rankings, is it the aggregation scheme that affects them, or is it a combination of these changes in the inputs (interactions) that has a greater effect on the final output? These questions are answered via the use of sensitivity analysis, and they are generally expressed in terms of sensitivity measures for each input tested. More specifically, they show by how much the variance would decrease in the index if that uncertainty input were removed (OECD 2008). Understandably, with the use of both, a composite index might convey a more robust picture (Saltelli et al. 2005), and it can even be proven useful in dissolving some of the criticisms surrounding composite indicators (e.g. see Saisana et al. 2005, for an example using the Environmental Sustainability Index). Having discussed the concept of robustness analysis through the use of uncertainty and sensitivity analyses, we will now briefly discuss how these are applied after the construction of a composite index.³⁰

The first step in uncertainty analysis is to choose which input factors will be tested (Saisana et al. 2005). These are essentially the choices made in each step (e.g. selection of the indicators, imputation of missing data, normalisation, and weighting and aggregation schemes) where applicable. Ideally, one should address all sources of uncertainty (OECD 2008). These inputs are translated into scalar factors, which, in a Monte Carlo simulation environment, are randomly chosen in each iteration. Then the following outputs are captured and monitored accordingly: (1) the overall index value; (2) the difference in the values of the composite index between two units of interest (e.g. countries or regions); and (3) the average shift in the rank of each unit.

Unlike UA, sensitivity is applied to only two of the above-mentioned outputs, which are relevant to the evaluation of the quality of the composite. These are (2) and (3) as mentioned in the previous paragraph (Saisana et al. 2005). According to the authors, variance-based techniques are more appropriate due to the non-linear nature of composite indices. For each input factor being tested, a sensitivity index is computed, showing the proportion of the overall variance of the composite that is explained, ceteris paribus, by changes in this output. These sensitivity indices are calculated for all the input factors via a decomposition formula (see Saisana et al. 2005, p. 311). To obtain an even better understanding, it is also important to identify the interactions between the considered inputs (e.g. how a change in an input factor interacts with a change in another). For this exercise, total sensitivity indices are produced. According to the authors, the most commonly used method is the one by Sobol (1993), in a computationally improved form given by Saltelli (2002).

Stochastic multi-attribute acceptability analysis (SMAA; Lahdelma et al. 1998; Lahdelma and Salminen 2001) has become popular in multiple criteria decision analysis for dealing with the issue of uncertainty in the data or the preferences required by the decision maker during the evaluation process (e.g. see Tervonen and Figueira 2008). SMAA has recently been introduced in the field of composite indicators as a technique to deal with uncertainties in the construction process. More specifically, Doumpos et al. (2016) use this approach to create a composite index that evaluates the overall financial strength of 1200 cross-country banks in different weighting scenarios.³¹ SMAA can prove to be a great tool in the hands of indices’ developers, and it can extend beyond its use as an uncertainty tool. For instance, Greco et al. (2017) propose SMAA to deal with the issue of weighting in composite indicators by taking into consideration the whole set of potential weight vectors. In this way it is possible to consider a population in which preferences (represented by each vector of weights) are distributed according to a considered probability. In a complementary interpretation, the plurality of weight vectors can be imagined as a representative of the preferences of a plurality of selves, of which each individual can be imagined to be composed (see e.g. Elster 1987). On the basis of these premises, SMAA is applied to the ‘whole space’ of weight vectors for the considered dimensions, obtaining a probabilistic ranking. Essentially, this output illustrates the probability that each considered entity (a country, a region, a city, etc.) attains the first, the second and so on position, as well as the probability that each entity is preferred to another one. Moreover, Greco et al. (2017) introduce a specific SMAA-based class of multidimensional concentration and polarisation indices (the latter extending the EGR index) (Esteban and Ray 1994; Esteban et al. 2007), measuring the concentration and the polarisation of the probability of a given entity being ranked in a given position or better/worse (e.g. the concentration and the polarisation to be ranked in the third or a better/worse position).

The use of SMAA as a tool that extends beyond its standard practice (e.g. dealing with uncertainty) is a significant first step towards a conceptual issue in the construction of composite indicators: representative weights. More specifically, constructing a composite index using a single set of weights automatically implies that they are representative of the whole population (Greco et al. 2017). Quoting the authors (p. 3): ‘[…] the usual approach considering a single vector of weights levels out all the individuals, collapsing them to an abstract and unrealistic set of ‘representative agents”’. Now one can imagine a cross-country comparison using a single set of weights that act as a representative set for all the countries involved. Understandably, it is a rather difficult assumption to make, given Arrow’s theorem (Arrow 1950). Decancq and Lugo (2013, p. 10) describe this fundamental problem with a simple example of a theoretical well-being index. According to the authors, the literature is well documented with respect to the variation of personal opinions on what a ‘good life’ is. Therefore, following the same reasoning, how can a developer assume that a set of weights acts as a representative of all this variation? Quoting the authors (p. 10): ‘Whose value judgements on the “good life” are reflected in the weights?’ This is a classic example of a conflictual situation in public policy, arising due to the existence of a plurality of social actors (see e.g. Munda 2016). This issue of the representative agent (see e.g. Hartley and Hartley 2002) has long been criticised in the economics literature, one of the most well-known criticisms being made by Kirman (1992). According to Decancq et al. (2013), inevitably there are many individuals who are ‘worse off’ when a policymaker chooses a single set of weights. On the one hand, SMAA extends above and beyond the issue of representativeness by providing the developer of an index with the option to include all possible viewpoints. However, for every viewpoint taken into account, a different ranking is produced; thus, a choice has to be made afterwards regarding how to deal with these outcomes. Usually, the mode ranking is chosen, obtained by the ranking acceptability indices (see e.g. Greco et al. 2017). Moreover, in its current form, SMAA can only provide the developer with a ranking of the units evaluated. Thus, it still suffers from the same issue as other non-compensatory techniques: swapping the available information in possession with an ordinal scale in the form of a ranking.

Several other approaches appear in the literature, with which the robustness of composite indices may be evaluated or which may simply provide more robust rankings. An example of the latter is given by Cherchye et al. (2008), presenting a new approach according to which several units may be ranked ‘robustly’ (i.e. rankings are not reversed for a wide set of weighting vectors or aggregation schemes). To achieve this, they propose a generalised version of the Lorenz dominance criterion, which leaves to the user the choice of how ‘weak’ or ‘strong’ the dominance relationship will be for the ranking to be considered robust. This approach can be implemented via linear programming, an illustrative application of which is given with the well-known HDI. In regard to the robustness evaluation, Foster et al. (2012) present another approach,³² in which several other weight vectors are considered to monitor the existence of rank reversals. In essence, by changing the weights among the indicators, this approach measures how well the units’ rankings are preserved (e.g. in terms of percentage). In an illustrative application, the authors examine three well-known composite indices, namely the HDI, the Index of Economic Freedom, and the Environmental Performance Index. Similar to Foster et al. (2012), Permanyer (2011) suggests considering the whole space of weight vectors, though the objective is slightly different this time. The author proposes to find three sets of weights according to which: (1) a unit, say ‘α’, is not ranked below another unit, say ‘β’; (2) units ‘α’ and ‘β’ are equally ranked; and (3) ‘β’ dominates ‘α’. Essentially, the original intended weight vector set by the developer can fairly be considered to be ‘robust’ the further it is from the second subset (viz. the set of weights according to which ‘α’ is equal to unit ‘β’), because the closer to it that it is, the more possible it is for a rank reversal to happen. This intuitive approach is further extended to multiple examples and specifications, details of which can be found in Permanyer (2011, pp. 312–316). An illustrative example is provided using the well-known HDI, the Gender-related Development Index, and the Human Poverty Index.

While still considering the robustness evaluation, Paruolo et al. (2013) propose another approach, which is mainly concerned with the perception of weights and the actual effect that they have on the final output. More specifically, the authors stress how far off the mark a weighting scheme might be when it is assigned in comparison with the actual effect that it has on the overall index (what they call the ‘main effect’, p. 610). This effect is notably apparent in the case of linear aggregation. They propose to measure this effect via Karl Pearson’s correlation ratio, often applied in sensitivity analysis as a first-order measure. According to the authors, this measure can potentially fill a gap in the criticism regarding the difference between the stated importance (given by the weighting) and the actual importance achieved (after the aggregation has taken part) in the case of compensatory aggregation. In a recent study, Becker et al. (2017) extend this area of research by introducing three tools to aid the developers of composite indices in gaining a better insight into the effect that weights have on the final synthetic measure. The first tool is based on Paruolo et al. (2013), estimating the main effect of weights using either Gaussian processes or penalised splines, depending on the size of the considered data set and thus the computational cost. The second tool relates to the isolation of indicators’ correlation in the main effect measured by Karl Pearson’s correlation ratio. Using a regression-based approach, the correlation effect can be isolated from this first-order measure so that the developer has an insight into the pure effect of the weights on the composite index, regardless of the correlation among indicators. Finally, the authors propose a third tool allowing stated weights to be aligned perfectly with their actual importance in the final index.

Undeniably, robustness analysis in any form, ‘traditional’ or otherwise, may act as a quality assurance tool. This exhibits the strength of an index by delineating all its potential forms in the case of different choices made in the inputs. However, one of the first points stressed in the OECD’s Handbook is that one cannot interpret an assessment of robustness as the validation of a ‘sensible’ index (OECD 2008, p. 35). Rather, it is the creation of a sound theoretical framework that determines whether the index is actually sensible. Robustness might only help the developer to answer the questions related to the fit of the model and the meaning of its concept (OECD 2008). Unfortunately, but no doubt reasonably, the Handbook cannot provide any form of aid to the developer regarding which theoretical framework fits best. Quoting the authors (OECD 2008, p. 17): ‘[…] our opinion is that the peer community is ultimately the legitimate forum to judge the soundness of the framework and fitness for purpose of the derived composite’. However, they do urge developers to bear in mind that, whichever framework is used, transparency is of the utmost importance. Making an effort to reduce this uncertainty stemming from the creation of the theoretical framework, Burgass et al. (2017) suggest the following actions: first, the use of systems modelling (either quantitative or qualitative) to aid the developers of indices to make the proper choices; second, the promotion of open discussions among modellers, experts, and stakeholders to construct a sound theoretical framework that works for all.