1 Introduction
ROI-MOB (Measuring return on investment from EU VET mobility) is a European project that aims to represent the final outcomes of an Erasmus + mobility experience using a single, complex indicator (Fabbris and Boetti 2019). ROI-MOB focuses on the Vocational Education and Training (VET) experience, which consists of an internship undertaken by a student or apprentice at a workplace in another country. Schools, companies, or other intermediate bodies organize, in agreement with the hosting bodies, the internship of individuals or small groups of participants. The indicator measures the quality of this complex phenomenon jointly experienced by the young participant and all other bodies involved in the process. The indicator is a composite index of elementary indicators, each capturing the viewpoint of an assessor, aggregated using an appropriate ‘importance’ weighting. In particular, the surveys collected data from four groups of assessors: (i) the participants directly involved in VET international mobility, either residing or hosted in one of four countries (Germany, Italy, Portugal, and Spain); (ii) the schools and training centres facilitating the participation of students or apprentices, either by sending them abroad or hosting them; (iii) the companies and public bodies that sent or hosted the participants; and (iv) representatives of the national or European institutions promoting VET international mobility. For the sake of simplicity, only the data from the first three groups of assessors are processed in this paper.
This paper aims to estimate the importance weights to attach to the final evaluations from three groups of assessors surveyed after a set of mobility experiences that occurred in 2017 and 2018 in the four focal countries. For each assessor, the final evaluation is assumed to be an overall evaluation measure of the outcomes of the pertinent VET mobility experience. To estimate the weights, we applied a set of procedures to the rankings of beneficiaries from the VET international mobility scheme, as expressed by the assessors, and defined an optimum way to transform rankings into scores. The procedures rely on the hypothesis that the mobility experience is more important for those categories of assessors who receive greater benefits and, therefore, the evaluation of the experience of these categories must weigh more in the composite indicator representing the overall evaluation.
Advertisement
Technically, the ROI-MOB indicator is a combination of the final evaluations independently given by the mobility actors, weighted according to the recognised ‘benefit’ the actor received from the mobility experience. All actors are assumed to be equally informed of the process. This prevents the argument be made that the weights attached to the experience assessments should be proportional to the level of information possessed by the assessor, which is unknown. However, we do know the level of benefit the assessors obtain through the process.
In the following, we deal just with the aim of estimating the weights for the indicator construction, which we will call scores so as not to confuse them with the weights that will be used in their estimation. The rest of the paper is organised as follows. Section 2 describes in detail the available data, the analytical models used in the estimation of weights, and the criteria for model selection. Section 3 describes the results, and these are then discussed in Sect. 4.
2 Data and Methods
2.1 The Data
The data came from three surveys conducted between March and August 2018 in four European countries (Germany, Italy Portugal, and Spain). The questionnaires were sent to samples of participants, schools, and companies whose email addresses were provided by the project partners. All participants, companies, and schools involved in a VET mobility experience organised by one of the project partners in the last two years were included in the sample. Each sample aimed to represent a national context. After two pilot rounds of data collection, the questionnaires, written in the four national languages of the project (Italian, German, Portuguese, and Spanish), plus English, were administrated to over 5,000 potential respondents through a Computer Assisted Web-based Interviewing (CAWI) system. The participation-level was good, with a response rate of 31% (1,545 questionnaires).
The national representativeness of the samples does not provide European representativeness in the overall sample. However, the presence in the sample of both Mediterranean and Central Europe countries allows for broad cross-national inference. Moreover, the four focal countries account for a substantial proportion of the international students and trainees mobilised by the Erasmus + Programme (Fabbris and Boetti 2019).
Advertisement
The basic questions posed in the surveys to evaluate the ranking of beneficiaries slightly differed according to the mobility actor. The questions posed only to participants was:
-
‘Finally, which are the two categories that get the highest benefits from Erasmus + mobility? (Please, click the first and the second category of possible recipients)’. The option was given to specify a first and second selection out of five possible beneficiaries: students/apprentices, schools and training centres, companies (both sending and hosting), the labour market, and the European Union as an institution. The beneficiaries were ranked on a three-level ordinal scale.The question posed only to the schools and companies was:
-
‘Finally, which are the categories of possible recipients that get the highest benefits and the ones that get the lowest ones from Erasmus + mobility? Please, order the categories from 1 (highest) to 5 (lowest benefits)’. The same categories presented to the participants were used here. The beneficiaries were ranked on a five-level ordinal scale.
The collected data contained nonresponses. In some cases, people did not respond to the question at all; in other cases, people indicated just the first position or just few initial positions. All given rankings, even those that were incomplete, were processed.
It is worth highlighting that the three categories of assessors evaluated themselves as a possible recipient of mobility benefits. Thus, when a respondent evaluated the relative positions of the mobility actors, they were expected to operate a sequence of mental comparisons, first pinpointing the top beneficiary and then selecting a second best among the residual categories, then a third, and so on. All the possible beneficiaries were well known to the assessors as actors of the represented mobility process.
2.2 Models and Methods
In the following, we deal with the evaluation systems in which each unit of a sample of\(n\) independent assessors ranks \(A^{\prime}\) alternatives from a set of \(A\) total alternatives, \(A^{\prime} \le A\), from the first to last possible place. Our aim is to find an optimum strategy to aggregate the n rankings and transform them into scores to assign to the \(A^{\prime}\) alternatives.1 Without a loss of generality, we assume that—as occurred in our surveys—all alternatives are ranked, e.g.\(A^{\prime} = A.\)
Let us consider the information contained in the rankings given by a group of assessors. To estimate the scores, the information can be conceived at different levels:
-
We can use just the top position of each distribution. Thus, the score estimates are proportional to the frequencies of the top positions obtained by the \(A\) alternatives. This estimation method uses basic information and does not require particular statistical techniques. In the following, this procedure is called ‘first position’.
-
We can deduce scores from the averages of frequency distributions. This estimation method uses the full frequency distribution as if each ranking was an empirical occurrence of a random variable distributed with mean \(\left( {A + 1} \right)/2\) and variance \(\left( {A^{2} - 1} \right)/12\). Henceforth, we will call this method the ‘univariate’ procedure.
-
We can model the relations between the \(A\left( {A - 1} \right)/2\) distinct pairs of alternatives by comparing their ranking positions and defining the level of ‘dominance’ of alternative \(a\) over alternative \(b\). Thus, this weighting procedure is called ‘bivariate’. This estimation method requires the construction of a ‘dominance matrix’, which reflects the bivariate relationships between alternatives. The way such a matrix is defined and how it can be processed for weight estimation purposes is described later in this section.
The univariate approach uses the (absolute or relative) frequency distribution of the rankings given by the \(n\) respondents. For the \(a\) th alternative, the absolute frequency distribution is the vector \({\mathbf{f}}_{a} = \left\{ {f_{a1} ,...,f_{ah} ,...,f_{aA} } \right\}\) whose elements add up to \(n\). To give an example, \(f_{a1}\) is the number of respondents who put the \(a\) th alternative in the first position. The score for alternative \(a\), Ya, is obtained as a linear combination of the frequencies at all rank positions of that alternative,where: \(f_{ai}\) is the frequency of assessors who placed alternative \(a\) in position \(i \left( {i = 1,...,A} \right)\); and \(W_{i}\) is a weight assigned to rank \(i\), such that \(\sum \nolimits_{i = 1}^{A} W_{i} = 1\). For the sake of simplicity, the score is assumed to be normalised, so as to add up to one across all alternatives, \(\sum \nolimits_{a = 1}^{A} Y_{a} = 1\) and \(0 \le Y_{a} \le 1\).
$$Y_{a} = \mathop \sum \limits_{i = 1}^{A} f_{ai} W_{i} \;\;\;\;\;\;\left( {W_{i} \ge 0;\;\;a = 1, \ldots , A} \right)$$
(1)
For score estimation purposes, the bivariate approach refers to the analysis of dominances, tournaments, or round-robins (Ahn and Park 2008). This analysis requires a three-step procedure:
1)
Estimation of a dominance matrix, \({\mathbf{P}} = \left\{ {p_{ab} \left( {a \ne b = 1,...,A} \right); \;p_{aa} = 0 \;\; \forall \; a, b} \right\}\), through the rankings expressed by the respondents. The dominance relations can be recorded in a ‘generalised’ tournament matrix, whose entries vary between zero and one, with extremes included (Moon and Pullman 1970; Tanino 1988). \({\mathbf{P}}\) is a square skew-symmetric matrix of cardinality \(A\) with \(p_{ab} = 1 - p_{ba} \left( {a \ne b = 1,...,A} \right)\) and \(p_{aa} = 0\) for \(\left( {a = 1,...,A} \right)\). The entry \(p_{ab}\) can be interpreted as a measure of the dominance of \(a\) over \(b\), and the more this measure is close to one, its maximum, the larger the score assigned to a with respect to b.
2)
The computation of the right eigenvector \(\user2{w }\) corresponding to the first eigenvalue \(\gamma\) of matrix \({\mathbf{P}}\) is as follows:
$${\mathbf{P}}{\varvec{w}} = \gamma {\varvec{w}}$$
(2)
imposing \({w^{\prime}w} = 1\). The \(a\) th entry of this eigenvector is proportional to the score of the \(a\) th alternative.
3)
Normalisation of the values of \({\varvec{w}} = \left\{ {w_{a} \left( {a = 1,...,A} \right)} \right\}\) is accomplished by dividing each element with the sum of the \(A\) elements of the eigenvector \(\sum \nolimits_{a}^{A} w_{a}\) so that the normalised values \(w_{a}^{*}\) add up to one across the \(A\) alternatives, (\(\sum \nolimits_{a}^{A} w_{a}^{*} = 1\)). So, in our case, the normalised values estimate how much the actors benefitted from the international mobility on a 0–1 scale so that the total benefit of the \(A = 5\) possible beneficiaries is \(100\%\).
Therefore, for bivariate analysis, we estimate the dominance relations through the following formula:where \(n\left( {a > b|a = i} \right)\) denotes the number of times the alternative \(a\) is at rank \(i\) and the alternative \(b\) follows in the sequence, say from \(\left( {i + 1} \right)\) to \(A\); \(m\) is the total number of comparisons between \(a\) and \(b\); and \(W_{i}\) is a weight associated with the number of times \(a\) dominates \(b\) when alternative \(a\) is at rank \(i \left( {i = 1,...,A - 1} \right)\).
$$P_{ab}^{*} = \mathop \sum \limits_{i = 1}^{A - 1} \frac{{n\left( {a > b|a = i} \right) W_{i} }}{m} = \mathop \sum \limits_{i = 1}^{A - 1} f\left( {a > b|a = i} \right) W_{i} \;\;\;\;\;\;\;\;\; \left( {a \ne b = 1, \ldots , A} \right)$$
(3)
For \(P_{ab}^{*}\) to be an element of the \({\varvec{P}}\) matrix it requires further refinement:
$$p_{ab} = P_{ab}^{*} /\left( {P_{ab}^{*} + P_{ba}^{*} } \right) \;\;\;\;\;\;\;\;\;\left( {a \ne b = 1, \ldots , A} \right)$$
so that \(p_{ab} = 1 - p_{ba} .\)
In defining the weights, both in the univariate and the bivariate approach, we will refer to various scoring functions. Let us start with the plurality and the Borda rules (Borda 1784; Brams and Fishburn 2002). The plurality rule is the simplest way of weighting since it assumes that \(W_{1} = 1\) and \(W_{h} = 0\) for \(h = 2,...,A\). This rule is equivalent to considering only the first position (i.e. the univariate approach) and only the comparisons involving the first ranking position for each alternative in the bivariate approach. In these cases, the scores assigned to the \(A\) alternatives are given by the vector \({\mathbf{y}} = \left\{ {\hat{Y}_{1} ,...,\hat{Y}_{a} ,...,\hat{Y}_{A} } \right\}\) in which the score estimate of alternative \(a\) is given simply by the relative frequency of the first positions assigned to the alternative \(a\) by the \(n\) assessors.
The application of the Borda rule implies different weights, \(W_{i} \left( {i = 1,...,A} \right)\), according to the position of the alternative \(a\) in the ranking. For a univariate analysis, the rule assigns a weight of \(A\) to the first position, \(A - 1\) to the second position, and so on until the weight of one is given to the last position. So, the standardised form of weight \(W_{i}\) is a linear function of the distance of alternative \(a\) from the first position of the ranking:
$$W_{i}^{^{\prime}} = W^{\prime}\left( {a = i} \right) = \frac{A - i + 1}{{\mathop \sum \nolimits_{j}^{A} A - j + 1}} = \frac{{2\left( {A - i + 1} \right)}}{{A\left( {A + 1} \right)}} \;\;\;\;\;\;\;\;\; \left( {i = 1, \ldots , A} \right)$$
(4)
which is decreasing in \(i\), the position of the alternative in the ranking. The weight has a maximum of \(2/\left( {A + 1} \right)\), corresponding to the first position, and a minimum of \(2/\left( {A^{2} + A} \right)\), corresponding to the bottom position:
$$\frac{2}{{A\left( {A + 1} \right)}} \le W_{i}^{{\prime }} \le \frac{2}{{A + 1}}.$$
The weights obtained according to formula (4) are based on the idea that the rank order should be reflected directly in the weights and are known in the literature as rank sum (RS) weights (\(W_{i}^{^{\prime}} = W_{i}^{RS}\): Stillwell et al. 1981). Another way for converting the rankings into scores is the rank reciprocal (RR) rule, in which weights are proportional to the reciprocals of the rank order of alternative I (Barron 1992; Stillwell et al. 1981):
$$W_{i}^{RR} = W^{RR} \left( {a = i} \right) = \frac{1/i}{{\mathop \sum \nolimits_{j = 1}^{A} 1/j}} \;\;\;\;\;\;\;\;\;\left( {i = 1, \ldots , A} \right)$$
(5)
Variants of the rank reciprocal rule are the so called rank order centroid (ROC), in which weights are proportional to the sum of rank reciprocals from position i till the bottom position of the ranking:
$$W_{i}^{ROC} = W^{ROC} \left( {a = i} \right) = \frac{1}{A}\mathop \sum \limits_{i = i}^{A} \frac{1}{i}\;\;\;\;\;\;\;\;\; \left( {i = 1, \ldots , A} \right)$$
(6)
and the Sum-Reciprocal (SR) rule as proposed in Danielson and Ekenberg (2014), in which weights are intermediate between the linear and the reciprocal ones:
$$W_{i}^{SR} = W^{SR} \left( {a = i} \right) = \frac{{\frac{1}{i} + \frac{A + 1 - i}{A}}}{{\mathop \sum \nolimits_{j = 1}^{A} \frac{1}{j} + \left( {\frac{A + 1 - j}{A}} \right)}} \;\;\;\;\;\;\;\;\;\left( {i = 1, \ldots , A} \right)$$
(7)
Both RS and ROC approaches have sound statistical bases, both relying on distributional properties and being subject to random error (Jia et al. 1998). When numerical weights are estimated through functions, instead of directly observed, they are also called surrogate weights (Danielson and Ekenberg 2017).
The literature on how independent rankings can be merged into one or more summarizing objects of the same ordinal (ranking)2 or metric (score) nature makes reference to the feasible consensus, compromise theory in social choices, or visual representations of the rankings in a one-dimensional normalised scale (Honda et al. 1981; Jensen 1986; Hudry and Monjardet 2010). A metric solution requires the introduction of the notion of distance: the consensus scores are those minimizing the remoteness from the basic rankings, and the remoteness depends on the imposed metric. In what follows, to account for the possible dependence between the solutions and the adopted metrics, we applied various metric rules.
Therefore, we conjectured that the further we go down the ranking, the more the distance from the first position should more than linearly reduce the weight of the alternative. Formula (4) then changes as follows:where \(\delta\) is the power of the distances from the first position. Ahn and Park (2008) call this rule ‘rank exponent’ (RE) weight. It is easy to see that if \(\delta = 0\), all weights are equal, and equal to 0.2, and if \(\delta = 1\), Formula (8) coincides with Formula (4), which corresponds to the linearly scaled weights. For \(\delta = 2\), the frequencies close to the top position weigh more than linear and diverge from linearity as we approach \(A\), the bottom of the ranking. In fact, in case \(A = 5\), the vector of linear weights is \(\left[ {0.333; 0.267; 0.200; 0.133; 0.065} \right]\) while the quadratic vector is \(\left[ {0.454; 0.291; 0.164; 0.073; 0.018} \right]\). Moreover, the vector of RR weights is \(\left[ {0.438; 0.219; 0.146; 0.109; 0.088} \right]\), the vector of ROC is \(\left[ {0.457; 0.257; 0.157; 0.090; 0.040} \right]\), and the vector of SR is \(\left[ {0.381; 0.248; 0.178; 0.124; 0.076} \right]\).
$$W^{\prime\prime\prime} = W^{\prime\prime\prime}\left( {a = i} \right) = \frac{{\left( {A - i + 1} \right)^{\delta } }}{{\mathop \sum \nolimits_{i}^{A} \left( {A - i + 1} \right)^{\delta } }} \;\;\;\;\;\;\;\;\; \left( {i = 1, \ldots , A; \delta \ge 0} \right)$$
(8)
To weigh a comparison between alternatives a and b, the rules vary according to assumptions. Weights can be computed simply as a function of the distance of alternative a from the first position: if alternative a is in the first position, we assign the largest weight to the comparison between \(a\) and \(j\), where \(j\) can be any other alternative \(\left( {j = b,...,A} \right)\); if \(a\) is in the second position, the weight will be one less than previously, until the next to last position, when the weight will be \(1\); and for \(a = A\), the weight will be zero. The standardised form of weight \(W_{i}\) for a bivariate analysis is a linear function of the distance of the first alternative from the top position in the ranking3:
$$W_{i}^{\prime\prime} = W^{\prime\prime} \left( {a = i} \right) = \frac{A - i}{{\mathop \sum \nolimits_{i}^{A} A - i}} = \frac{{2\left( {A - i} \right)}}{{A\left( {A - 1} \right)}}\;\;\;\;\;\;\;\;\; \left( {i = 1, \ldots , A - 1} \right)$$
(9)
This weighting system has a maximum of \(2/A\) for \(i = 1\) and a minimum of \(2/\left( {A\left( {A - 1} \right)} \right)\) for \(i = A - 1\):
$$\frac{2}{{A\left( {A - 1} \right)}} \le W^{\prime\prime} \le \frac{2}{A}{.}$$
It is easily seen that Formula (9) is the same as Formula (4) in cases where i varies between 1 and (A—1). Formulae from (5) to (7) can also be adapted to weigh pair comparisons of alternatives by leaving i to vary between 1 and (A—1) and substituting A with (A—1) in the formulae.
Other possible rules refer either to how many ranks separate the two alternatives or which are the alternatives that occur between the two at stake. For instance, if we compare two alternatives whose positions are \(a = 1\) and \(b = 4\), should it matter if another alternative \(c\) is in the second or the fifth position? From Arrow’s theorem (Arrow 1951), we know it does because the rest of the world may in some way influence, even significantly, the relationship between the two compared alternatives. The degree to which it matters and in which direction is unknown. In what follows, we model the between-alternative relations according to various hypotheses on the distance between their ranks. We ignore the possible dependence of the magnitude of a dominance relationship on the position of the remaining alternatives.4 As a matter of fact, we rely on the local independence of the irrelevant alternatives (Young 1995), namely the independence between the two compared alternatives and the other A—2 alternatives not involved in the comparison, an assumption that is reasonable in a case, such as ours, in which the alternatives are social entities just functionally interacting with each other but likely independent in their judgements of the mobility process.
Analogous to the univariate case, we conjectured that the further we go down the ranking, the more the distance from the first position should more than linearly affect the weight of the alternative. So, the distance rule can be powered as:
$$W^{IV} = W^{IV} \left( {a = i} \right) = \frac{{\left( {A - i} \right)^{\delta } }}{{\mathop \sum \nolimits_{i}^{A} \left( {A - i} \right)^{\delta } }}\;\;\;\;\;\;\;\;\; \left( {i = 1, \ldots , A - 1; \;\; \delta \ge 0} \right)$$
(10)
If \(A = 5\), the vector of weights in the linear case is \(\left[ {0.4; 0.3; 0.2; 0.1; 0} \right]\) and in the quadratic case is \(\left[ {0.533; 0.300; 0.133; 0.033; 0} \right]\). Analogously, the vector of weights is \(\left[ {0.48; 0.24; 0.16; 0.12; 0} \right]\) in the RR weighting system, \(\left[ {0.521; 0.271; 0.146; 0.062; 0} \right]\) in the ROC system, and \(\left[ {0.436; 0.273; 0.182; 0.109; 0} \right]\) in the SR system.
Also, if alternatives \(a\) and \(b\) are apart in a ranking, we could pose for ourselves the question: should we equally weight a comparison between \(a\) and \(b\) if, for instance, \(a\) is first and \(b\) is second, instead of when \(a\) is first and \(b\) is in any other non-contiguous position? Indeed, Bradley and Terry (1952) and Kemeny (1959) suggested to account for how far apart one alternative is from another in a comparison. This question is not addressed in this paper.
2.3 Criteria for Choosing the Best Approach
The literature on how to transform a set of available rankings into a set of scores (as defined in Sect. 2.2) is sparse, and there is no agreed upon solution for comparing these or similar approaches. Therefore, we propose two methods that may help in selecting the best among the proposed methods: where \(\hat{Y}_{mga}\) denotes the estimate of the score for alternative \(a\) from the group of assessors \(n\) with method \(m\); and Y̿ga the average score for alternative \(a\) from group \(g\) across the compared methods.
a)
Variability of rankings given by assessors around their aggregate ranking. This criterion assumes that the aggregate ranking is what the assessors evaluated and that a ranking delivered by an assessor is a trial of the ranking we expected to measure. Therefore, for each group of assessors, \(\left( {g = 1,...,G} \right)\), we have \(n_{g}\) rankings whose variability is proportional to the distance between each individual ranking and the group ranking:
where \(\left| \cdot \right|\) denotes the absolute distance of the argument; \(R_{mg0a}\), the mean rank of the \(a\) th alternative obtained with method \(m \left( {m = 1,...,M} \right)\) at group \(g\); and \(R_{mgka}\), the ranking of the same alternative in the ranking delivered by assessor \(k \left( {k = 1,...,n_{g} } \right)\) belonging to group \(g\). The difference \(R_{mgka} - R_{mg0a}\) measures how many ranks alternative \(a \left( {a = 1,...,A} \right)\) has to move to be in the same position as the group ranking; \(\delta\) is the power to which the distances between rankings are raised; and all the other symbols have the same meanings as in Sect. 2.2. It can be easily understood that \(\delta = 1\) gives the between-rank absolute distance, and \(\delta = 2\) gives the Euclidean distance.5 The former distance is known to smooth the differences between the compared rankings, while the latter one gives much more relevance to large deviations.
$$D_{mg} = \left\{ {\frac{1}{{n_{g} }}\mathop \sum \limits_{k = 1}^{{n_{g} }} \mathop \sum \limits_{a = 1}^{A} \left| {R_{mgka} - R_{mg0a} } \right|^{\delta } } \right\}^{1/\delta } \left( {m = 1, ..., M; \;\;g = 1, ...,G; \;\;\delta \ge 1} \right)$$
(11)
By construction, \(D_{mg}\) is always positive. It can be standardised to compare the ranking variability from different studies. A possible standardisation is the ratio between the empirical value and its maximum, which is obtained when one half of assessors give rankings in reverse order to the other half of assessors.
In case \(\delta = 1\), \(\text{Max}(D_{mg} ) = A^{2} /2\) if \(A\) is even and \(\left( {A^{2} - 1} \right)/2\) if \(A\) is odd. In case \(\delta = 2\), \(\text{Max}(D_{mg} ) = A\left( {A - 1} \right)\left( {A + 1} \right)/3\). It can be noticed that both maxima are a constant function of \(A\), so they can be used to standardise both single rankings and the aggregate one.
For a group of assessors, the standardised distance, \(D_{m}^{*} = D_{m} /{\text{Max}}\left( {D_{mg} } \right)\), can be conceived as the proportion of within-ranking positions that disagree with respect to a mean ranking. For instance, a standardised distance evaluated as \(0.3\) means that, as a whole, \(30\%\) of positions in the analysed rankings differ from the reference positions.
The method with the minimum standardised distance has a ranking that best represents the rankings delivered by the \(n\) assessors, i.e. the ranking varies the least.
b)
Absolute distance between the final scores generated by a model and the aggregated one mediated over all models. We assume the aggregated score is that to which all models should conform. In our case, the reference score is the one resulting from the average of the scores estimated for all models. Therefore, the absolute distance between all models and their mean is an estimate of the bias of the models:
$$B_{mg} = \frac{1}{A} \mathop \sum \limits_{a = 1}^{A} \left| {\hat{Y}_{mga} - \overline{\overline{Y}}_{ga} } \right| \;\;\;\;\;\;\left( {m = 1, \ldots , M;\;\;g = 1, \ldots , G} \right)$$
(12)
3 An Application to Mobility Beneficiaries
The models presented in Sect. 2.2 were applied to the rankings delivered by three groups of assessors: (a) the participants in international mobility; (b) the schools and training centres that either sent or hosted participants; and (c) the companies that either sent or hosted participants.6
In this application, the alternatives (henceforth, the actors) are the five possible beneficiaries of VET international mobility: participants; schools and training centres; companies (both sending and hosting); the labour market; and the European Union (EU) as an institution. To merge the scores of the assessing groups, the responses obtained from the three groups of assessors were mediated through an (unweighted) arithmetic mean, since, for the purpose of representing the mobility process, the responses given by the three groups are equally informative.
The models are ordered according to computational difficulty:
1.
First, we analysed the distribution of the main beneficiaries of international mobility that occurred in the first positions in the assessors’ rankings. The frequency gained by actor \(a \left( {a = 1,...,A} \right)\) was the estimate of their benefit score. This model is named Model 1. It could be computed for each group of assessors and, in a mediated form, for all groups of assessors.
2.
Then, we computed the linear combination of the frequencies at all rank positions for each mobility actor according to Formula (4), in its standardised form. Formula (4) applies constant weights to the ranking positions. This model, which can be considered a weighted synthesis of each frequency distribution, is named Model 2, and was computed for each group of assessors. Estimates for \(\delta = 1\) and \(\delta = 2\) and for ROC, RR, and SR rules were computed.
3.
We also applied the bivariate analysis to estimate the benefit scores. This application was based on the estimates of the dominance degree of a generic actor a over another generic actor b for all possible combinations of actor pairs. This allowed the construction of a dominance matrix \({\mathbf{P}}\) (Formula 3) of which we computed the right eigenvector associated with the first positive eigenvalue (Formula 2). The standardised elements of the eigenvector estimated the benefit scores of the mobility actors. This is presented as Model 3.
4.
Finally, we applied the same procedure adopted to compute pair comparisons between actors using weights given in Formula (9). The standardised scores deriving from the first eigenvector of this new matrix are presented as Model 4. Weights were powered both with \(\delta = 1\) and \(\delta = 2\) and also ROC, RR, and SR rules were applied.
The main results are presented in Table 1, in which the estimates of the benefit scores for the actors of international mobility are compared to each other according to the generating model and the group of assessors.
Table 1 shows that estimates differed significantly according to the applied model. The overall mean scores of the alternatives by assessor group and the average of the mean scores are presented at the bottom of Table1 . The results show the following:
-
The difference in estimates using just the first positions with respect to the methods based on the full distribution of evaluations is notable: the average of the participants’ scores computed over all assessors was almost 80%, while the average of all other methods was 36.9% and the median was 36.7%. This suggests that the estimates based on just the first position of the rankings should be considered a shortcut, as it is not influenced by the relations between beneficiaries. Since the estimation procedure based on just the first positions is an outlier among the applied models, it will not be considered further here.
-
Among the other computational methods, Model 2 rules were computationally the least troublesome. The difference between the most extreme score-profiles obtained by adopting the quadratic distance of alternatives and that obtained applying the absolute distance with the same RS rule is relevant: the quadratic strategy produced scores that were, averaging the benefit perceived by all assessors for participants as beneficiaries, more than 10% larger (42.4% instead of 31.6%) than those obtained with the absolute weighting rule. The scores assigned to the other beneficiaries were reduced proportionally to the score assigned to the first beneficiary but were consistent with each other. The score assigned to the top beneficiary according to the ROC rule follows immediately that of the RS with quadratic weighting rule (40.9% on average), then the RR (36.7%) and the SR (34.1) scores. These results are consistent with the literature.
-
The weighting rules seemed not so important if scores were estimated through the analysis of dominances. The quadratic procedure yielded results similar to the linearly-weighted analysis of dominances: the mean difference was lower than 0.6% whichever assessing-group was considered. Furthermore, the differences between assessors using either linear or quadratic powers for weight estimation were less important than those obtained in the univariate analysis. The application of any weighting rule to the analysis of dominances (Model 4) led to score profiles that did not diverge with each other. Actually, the first-beneficiary scores maintained the ordering highlighted in case of univariate analysis (Model 2): the largest value was obtained by the quadratic weighting (37.0% on average), ROC (36.9%), RR (36.8%), SR (36.7%), and RS (36.7%) rules.
-
We ascertained just three cases of equal scores between alternatives: once when mediating the scores obtained by three groups of assessors with Model 3 and twice when applying linear weights to Model 4. This may mean that even a flatting rule applied together with the analysis of dominances tends to assign to alternative scores that differ from each other. This result is in line with the literature (e.g.Landau 1951; Hemelrijk, Wantia and Gygax 2005).
-
Whatever the weighting rule, there was a clear hierarchy that placed participants at the top of the ranking with an endorsement of approximately 37%, then schools and companies at approximately 19% each, and finally, the labour market and the EU as an institution with approximately 12% each.
Table2 presents the results of the application of the two statistical criteria suitable for the evaluation of the capability of the models to give adequate estimates. The criteria were a measure of ranking variability, Formula (11), and a measure of bias, Formula (12).
The distance measures in Table2 refer to the variability of the delivered rankings from the final ranking, computed for each assessing group and model. The mean absolute distances were computed from the values marked with an asterisk at the bottom of Table 1. Moreover, for practical purposes, only complete rankings have been analysed. The bias was computed as the mean absolute deviance from a mean score, computed for each assessing group and for the assessors altogether. The Table2 estimates allow us to state the following:
-
The estimates of the distance between the collected rankings and the reference ranking did not differentiate among models because the reference ranking of the single models was about the same for all models. Thus, the measure of distance for Models 2 and 3 were the same for every group of assessors, but only for the RR rule. Small differences, instead, were found between Models 3 and 4 regardless of the weighting rule. The relative invariance of the distances (no difference exceeded 2.5%) was caused by both the near-linearity of the assessed alternatives and the shortness of the ranking.
-
Models 3 and 4, which were based on the analysis of dominances, were able to produce biased estimates of the scores in a lower proportion than Model 2, which was based on just the univariate analysis of frequencies.
-
Model 3, which was based on an equal-weight system, performed better in terms of variability than Model 4, for which we introduced various articulated weighting systems, but its bias was higher than any of the weighting systems in Model 4.7
-
All weighting rules applied to dominance analysis showed the same level of variability. Instead, the bias slightly differed among the applied rules: the ROC and RR rules showed the lowest bias, while the SR and the RS with linear weighting rules showed an 8% higher bias. The highest bias (almost twice that of the ROC rule) was found for the RS rule with quadratic weighting.
4 Discussion and Conclusion
In this paper, we presented a set of rules to score a common profile starting from large sets of rankings. Both rankings and scores concerned five categories of possible recipients of benefits from VET international mobility. For score estimation, we applied various weighting procedures that included at least three levels of information present in the collected rankings: (a) just the top recipient as stated by assessors; (b) the full distribution of frequencies derived from the rankings delivered by the assessors; and (c) the between-alternative dominance relations that can be deduced from the assessed rankings.
We showed that the more information we used in the scoring procedure, the more stable and accurate were the estimates. So, the estimates based on just the first position were shown to be an outlier with respect to the other estimation procedures, and this method would only be recommended for an initial estimation of scores, but not for a definitive estimate. The estimation procedure based on the analysis of univariate frequency distributions, which is intermediate in terms of quantity of processed information, gave estimates that were also intermediate regarding both variability and bias. Models that use the bivariate relations between alternatives on top of the frequency distribution of assessments were the best-performing models. This result is consistent with the literature (see, among others, Poisbleau, Jenouvrier and Fritz 2006), where it has been highlighted that the between-alternative dominance relations takes into account both the identity of the ranked alternatives and the bivariate interactions between alternatives present in the assessor’s minds.
Regarding the optimality of the weighting rules, the comparison between EW, RS, RR, ROC, SR, and RE rules within the processed informative models, showed conflicting results: in both univariate and bivariate analyses the variability criterion highlighted that rules whose configuration is steeper (RR and ROC) perform somewhat worse than or as well as linear or near-linear (RS and SR) rules, while the bias criterion showed the opposite, with RR and ROC slightly better than the linear and quasi-linear one. The quadratic weighting intensifies the steepness of the configuration so much that both the variability and bias criteria highlight its inadequacy as a behavioural model. On the contrary, the equal-weight rule provide a less adequate fit to the basic rankings than any position-based rule. Our data conclusively shows that the first position is to be weighted somewhat more than with a simple linear weighting.
Also, the solution of weighting through an SR model—which is intermediate between the linear and the RR rule—is as adequate as RR in the bivariate analysis and somewhat more adequate than RR in the univariate analysis if we refer to the variability criterion, but is less adequate if the bias criterion is considered. If we adopt the bias criterion, the best rule is RR, while if the variability criterion is adopted, there is no difference between the ROC, RS, and SR rules. In other words, although the between-rule differences are little, the highlights depend on the chosen evaluation criterion.
Our results are in line with the mainstream literature. In fact, Barron and Barrett (1996) compared RS, RR, ROC, and EW rules and stated that the ROC appears to perform better that the others. Sarabando and Dias (2009), applying a variety of rules, found that ROC rule is the best if just the first alternative or the top ranked alternatives are retained. Jia et al. (1998) compared EW, RS, and ROC rules, and also direct rating of alternatives in case the basic rankings are affected by assessment error. They found that direct rating is better if error is small, while RR and ROC rules are better if error is large, but the differences between linear and steeper rules are small. In addition, Jia et al. (1998) found that the accuracy of the rules is sensitive to the number of alternatives and to the sign and level of correlation among basic rankings. In particular, the authors found that if the rankings’ fluctuation is high and the number of alternatives is five, the RR rule gives more accurate results than the ROC, and the latter more accurate results than RS. However, the between-rule differences are small. Furthermore, they showed that if the correlation among the rankings is positive and high—as it is in our case—the between-rule differences are even smaller.
So, which weighting method should one use? From our results, the application of a dominance analysis technique gives more accurate estimates than just the univariate analyses. The accuracy of the choice of the weighting rule depends on the steepness of the “true” weights. In summary, EW is one extreme of the horn, that in which weighting is irrelevant because either we purposively ignore the elicited rankings or consider the basic data purely random. On the contrary, the RR, ROC, and RE (with δ > 1) rules highlights that the first alternative is nearly more important than the others and implies steep weight functions. In between, RS and SR rules provide flatter weight functions. Tversky et al. (1988) and Fischer and Hawkins (1993) found that in behavioural analyses and in preference-governed people’s choices, functions are often quite steep. This seems true in our case, because in all assessors’ minds, participants are those who benefit the most from international mobility, while the EU as an institution and the labour market benefit just indirectly from the scheme and are, consequently, positioned by all assessors at the bottom of the ranking.
In conclusion, our results support the value of surveys on social preferences for elucidating hidden social hierarchies and values orders, and for pinpointing the social categories that are particularly associated with a phenomenon (i.e. those that are often observed at the top of a ranking, versus others that occur further down the rankings and are only vaguely associated with a phenomenon). In cases where ranks are transformed into scores, these results imply that a more-than-linear weight should be assigned to the top position. That is why the RR and ROC rules applied together within a dominance analysis seem to better fit our mobility data.
Table 1
Percent estimates of the benefit scores as assessed by participants, by type of model and category of beneficiaries from international mobility (estimates add up to 100 by row).
Models by assessor and weighting power | Beneficiaries of mobility (Alternatives) | ||||
|---|---|---|---|---|---|
Participants | Schools and training c. | Companies (send/host) | Labour market | EU as an institution | |
1—Participants | 69.7 | 11.0 | 9.3 | 3.6 | 6.4 |
1—Schools | 90.7 | 2.3 | 2.3 | 0.0 | 4.7 |
1—Companies | 75.8 | 6.4 | 8.4 | 2.0 | 7.4 |
1—Mean of Assessor Groups | 78.7 | 6.6 | 6.7 | 1.9 | 6.1 |
2—Participants, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 31.6 | 18.3 | 18.5 | 15.5 | 16.1 |
2—Participants, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 43.9 | 16.3 | 16.5 | 11.0 | 12.3 |
2—Schools, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 32.8 | 21.3 | 18.3 | 15.1 | 12.5 |
2—Schools, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 43.8 | 20.3 | 15.8 | 11.2 | 8.9 |
2—Companies, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 30.5 | 19.1 | 21.6 | 15.2 | 13.6 |
2—Companies, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 39.5 | 17.3 | 21.3 | 11.5 | 10.4 |
2—Assessors Average, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 31.6 | 19.6 | 19.5 | 15.2 | 14.1 |
2—Assessors Average, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 42.4 | 18.0 | 17.9 | 11.2 | 10.5 |
2—Participants ROC | 39.5 | 16.9 | 17.0 | 12.8 | 13.8 |
2—Schools ROC | 43.8 | 18.9 | 15.5 | 11.7 | 10.1 |
2—Companies ROC | 39.4 | 16.9 | 20.2 | 12.1 | 11.4 |
2—Assessors Average ROC | 40.9 | 17.6 | 17.5 | 12.2 | 11.8 |
2—Participants RR | 32.8 | 17.9 | 17.8 | 15.4 | 16.1 |
2—Schools RR | 39.8 | 20.0 | 16.6 | 12.0 | 11.6 |
2—Companies RR | 37.8 | 16.6 | 18.9 | 13.1 | 13.6 |
2—Assessors Average RR | 36.7 | 18.2 | 17.8 | 13.5 | 13.8 |
2—Participants SR | 32.2 | 18.1 | 18.2 | 15.4 | 16.1 |
2—Schools SR | 36.7 | 19.7 | 17.0 | 14.1 | 12.5 |
2—Companies SR | 33.6 | 18.0 | 20.5 | 14.3 | 13.6 |
2—Assessors Average SR | 34.1 | 18.6 | 18.6 | 14.6 | 14.1 |
3—Participants | 33.7 | 19.0 | 19.7 | 13.1 | 14.5 |
3—Schools | 39.7 | 20.2 | 16.7 | 12.7 | 10.7 |
3—Companies | 35.0 | 18.3 | 21.3 | 13.3 | 12.1 |
3—Average of Assessors | 36.1 | 19.2 | 19.2 | 13.1 | 12.4 |
3—All assessors, P matrix | 35.8 | 19.2 | 19.3 | 13.2 | 12.5 |
4—Participants, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 34.4 | 19.2 | 19.2 | 12.7 | 14.5 |
4—Participants, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 34.8 | 19.4 | 19.0 | 12.1 | 14.7 |
4—Schools, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 39.9 | 20.2 | 16.7 | 12.2 | 11.0 |
4—Schools, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 39.9 | 20.1 | 16.7 | 11.7 | 11.6 |
4—Companies, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 35.8 | 18.0 | 21.2 | 12.5 | 12.5 |
4—Companies, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 36.2 | 17.9 | 21.1 | 11.8 | 13.0 |
4—Assessors Average, RS \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 36.7 | 19.1 | 19.0 | 12.5 | 12.7 |
4—Assessors Average, RS \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 37.0 | 19.1 | 18.9 | 11.9 | 13.1 |
4—Participants ROC | 34.7 | 19.3 | 19.1 | 12.3 | 14.6 |
4—Schools ROC | 39.8 | 20.1 | 16.7 | 11.8 | 11.6 |
4—Companies ROC | 36.1 | 17.8 | 21.0 | 12.0 | 13.1 |
4—Assessors Average ROC | 36.9 | 19.1 | 18.9 | 12.0 | 13.1 |
4—Participants RR | 34.4 | 19.2 | 19.2 | 12.7 | 14.5 |
4—Schools RR | 39.8 | 20.0 | 16.6 | 12.0 | 11.6 |
4—Companies RR | 36.0 | 17.8 | 20.9 | 12.3 | 13.0 |
4—Assessors Average RR | 36.8 | 19.0 | 18.9 | 12.3 | 13.0 |
4—Participants SR | 34.4 | 19.2 | 19.2 | 12.7 | 14.5 |
4—Schools SR | 39.9 | 20.1 | 16.7 | 12.1 | 11.2 |
4—Companies SR | 35.9 | 17.9 | 21.1 | 12.4 | 12.7 |
4—Assessors Average SR | 36.7 | 19.1 | 19.0 | 12.4 | 12.8 |
Mean Score: Participantsa | 35.1 | 18.4 | 18.5 | 13.2 | 14.7 |
Mean Score: Schoolsa | 39.6 | 20.1 | 16.7 | 12.4 | 11.2 |
Mean score: Companiesa | 36.0 | 17.8 | 20.8 | 12.8 | 12.6 |
Mean Score: Overalla | 36.9 | 18.8 | 18.7 | 12.8 | 12.8 |
Table 2
Mean absolute standardised distance between the delivered rankings and the final ranking and bias, by model and group of assessors (estimates add up to 100 by row; Model 1 excluded).
Model | Participants | Schools | Companies | Overall | ||||
|---|---|---|---|---|---|---|---|---|
Distance | Bias | Distance | Bias | Distance | Bias | Distance | Bias | |
2, \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 0.538 | 0.0147 | 0.294 | 0.0273 | 0.337 | 0.0219 | 0.498 | 0.0212 |
2, \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 0.538 | 0.0351 | 0.294 | 0.0176 | 0.337 | 0.0160 | 0.498 | 0.0220 |
2, ROC | 0.538 | 0.0175 | 0.294 | 0.0167 | 0.337 | 0.0137 | 0.498 | 0.0160 |
2, RR | 0.558 | 0.0142 | 0.294 | 0.0023 | 0.337 | 0.0124 | 0.510 | 0.0065 |
2, SR | 0.538 | 0.0142 | 0.294 | 0.0132 | 0.378 | 0.0108 | 0.498 | 0.0121 |
3 | 0.538 | 0.0071 | 0.294 | 0.0020 | 0.337 | 0.0061 | 0.484 | 0.0049 |
4, \(\delta \hspace{0.17em}=\hspace{0.17em}1\) | 0.538 | 0.0059 | 0.294 | 0.0017 | 0.337 | 0.0024 | 0.510 | 0.0027 |
4, \(\delta \hspace{0.17em}=\hspace{0.17em}2\) | 0.538 | 0.0059 | 0.294 | 0.0029 | 0.378 | 0.0039 | 0.510 | 0.0037 |
4, ROC | 0.558 | 0.0059 | 0.294 | 0.0025 | 0.378 | 0.0031 | 0.510 | 0.0025 |
4, RR | 0.538 | 0.0059 | 0.294 | 0.0023 | 0.378 | 0.0019 | 0.510 | 0.0025 |
4, SR | 0.538 | 0.0059 | 0.294 | 0.0013 | 0.378 | 0.0018 | 0.510 | 0.0027 |
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.