Visual management of performance with PROMETHEE productivity analysis

A large number of methods of solving multi-criteria problems have been developed, and this trend seems set to continue. Wallenius et al. (2008) showed that the number of academic publications related to MCDA is steadily increasing. This proliferation is not only due to researchers’ impressive productivity but also to the development of MCDA methods specific to certain types of problems. Roy (1981) has described four types of problem:Additional problem formulations have also been proposed:PROMETHEE (Brans and Mareschal 1994, 2005) is a multi-criteria method that belongs to the family of the outranking methods. It has been easily adapted to solve all the problems formulations (Mareschal et al. 2010). The aim of this paper is to present a MCDA-based solution to a new type of problem, namely the productivity problem. The majority of MCDA tools allow to assess the effectiveness: namely an alternative is said to be effective when it meets the output target. The MCDA methods allow to collapse multi-dimensional outputs into a single index that can be used as measure of effectiveness. The major shortcoming of effectiveness is that it is based just on the levels of output, and it does not deal with resource used to produce the output. For instance, an alternative could be ineffective because it has got a very limited resource. This is the reason why the most commonly used measure of performance is productivity (Ray and Chen 2015).The usual measure of productivity is a single indicator like output per worker. However, when there is more than one input (like labour and capital) and more than one output, the ratio Output/Workers fails to account for the use of all the inputs used and all the output produced. In order to overcome this shortcoming, we propose a MCDA method to build an aggregate measure of the inputs, and an aggregate measure of the outputs, and we assess productivity by means of relation between these two indices.

In the next subsections, we explain how we adapted PROMETHEE to solve this problem.

As with any other multi-criteria method, we consider a set of n possible actions \(A=\{ {a_1 ,a_2 ,\ldots ,a_n }\}\) which are evaluated according to a set of k criteria \(C=\{ {c_1 ,c_2 ,\ldots ,c_k }\}\). The main difference between PPA and the other methods in the PROMETHEE family is that the criteria are split into two groups: input and output criteria. The processing of the data is kept separate. For each criterion \(c_i\), and for each pair of actions ( a, b), the decision-maker expresses his/her preference by means of a preference function \(P_i\): the preference degree \(P_i( {a,b})\) is a number between 0 and 1 that indicates the extent to which action a is preferred to action b based on criterion \(c_i\). Six typical preference function shapes are proposed (Brans and Vincke 1985): usual function, U-shape, Level (Fig. 2), V-shape, V-shape with indifference (Fig. 1) and Gaussian (Fig. 3). The usual (\({p} = {q} = 0\)), V-shape (\({p} = 0\)) and U-shape (\({p} = {q}\)) are particular cases of the V-shape with indifference (Fig. 1), where p is the indifference and q the preference threshold on the axis d, which represents the score of an action on the given criterion.

A multi-criteria preference index is then calculated as a weighted sum of the single-criterion preference degrees:The weights \(w_i\) represent the relative importance of each criterion in the decision.

As each action is compared with the other \(n-1\) actions, the two following preference flows are defined:where n is the number of actions in set A.

This score represents the global strength of action a relative to all the other actions and the aim is to maximise it.This score represents the global weakness of a relative to all the other actions and the aim is to minimise it.

The net flow is the balance between the two previous flows and is given by:The positive and negative flows are used to build the PROMETHEE I partial ranking, whilst the net flow is the basis for the PROMETHEE II complete ranking: all the actions can be ranked in order of net flow value.

As PPA is based on PROMETHEE, it inherits its advantages. In particular, weights and preference functions can be assigned to criteria. If the criteria weights are known a priori by the decision-maker, this is important information which should be added to the model. Applying a preference function can also result in better representation of reality because changes in inputs (resources) and outputs (production) do not always have a linear effect on productivity. An increase in facility spending of 1000 Euros on an existing 10,000 Euro bill does not necessarily have the same effect as an increase of 1000 Euros on a 1000,000 Euro bill. Furthermore, in PROMTHEE all criteria can be expressed in their own units and thus there is no scaling effect. No normalisation of the scores is required, which avoids the drawback of the ranking depending on the choice of normalisation method (Tofallis 2008; Ishizaka and Nemery 2011). A detailed, technical discussion of the normalisation effect in the specific case of universities is given in Tofallis (2012).

The interpretation of net flows in PPA depends on the set of output and input criteria used. The higher the net flows of an action’s outputs and the lower the net flows of its inputs, the better it is.

In order to evaluate performances on the basis of the net flows of outputs and inputs, we define the PPA production possibility set \(({{\Psi }_\mathrm{PPA}})\) as:where \({\Phi }_\mathrm{I}\) is the net input flow, and \({\Phi }_\mathrm{O}\) is the net output flow.

In line with the DEA variable return to scale production possibility set (Banker et al. 1984) we assume that (Cooper et al. 2004, p 42):The production possibility set is a polyhedral convex set whose vertices correspond to all the actions that are not dominated by another action, i.e. no action simultaneously has higher net output flow and lower net input flow. The PPA frontier can easily be represented graphically (Sect. 2.4).

It worth noting that, unlike the original DEA (Charnes et al. 1978; Banker et al. 1984), inputs and outputs are not explicitly considered in the production possibility set (5). Similar approaches can be found in more recent DEA applications. One of the most popular of these approaches is PCA-DEA (see Ueda and Hoshiai 1997; Adler and Golany 2001, 2002; Adler and Yazhemsky 2010), in which DEA is used on the Principal Components of the original variables.

A second difference from the original DEA is that if one action changes its inputs and outputs, it can modify the inputs and outputs of other actions in the production possibility set (5). This feature is also displayed in other DEA model, for example by the model of Muñiz (2002), in which indexes of efficiency obtained by DEA in the first stage are used as inputs in the second stage. Other examples of DEA models sharing this feature are all the DEA applications on data pre-treated with specific normalisations (such as max–min, Nardo et al. 2008), for instance the model used in Mizobuchi (2014).

To measure the distance to the frontier in PPA, we use an algorithm based on the standard additive model introduced by Charnes et al. (1985) and elaborated by Banker et al. (1989). The algorithm is based on two steps. The first step measures the input distance as:where \({{\varvec{\Delta }}}_{{{\Phi }_{{I}}} _{\varvec{k}}}\) is the input distance to the frontier for the actions k under evaluation;In the second step we measure the output distance as:where \({{\varvec{\Delta }} }_{{{\Phi }_{\mathrm{O}}}_{\varvec{k}}}\) is the output distance to the frontier for the actions k under evaluation.

Finally, the minimum distance to the frontier defines the PPA inefficiency as:

The productivity of the actions can be depicted in a two-dimensional graph (Fig. 4). The range of the two axes is \({-}\,1\) to 1, and they represent the net flows of the inputs and outputs. Four categories of action can be defined:

Any action that is not dominated in terms of Input/Output net flows lies on the PPA frontier (shown in red on Fig. 4). Actions that are not on the frontier can be improved by taking real action(s) that are closest to the frontier as an example (e.g. in Fig. 4 action F can be improved by looking at the example of action C).

An action’s productivity ranking is determined by measuring the distance between it and the productivity frontier. This distance can be measured in various ways, e.g.:PPA has been implemented in two stages analysis:

Measuring universities’ performance is a fundamental necessity for society as it shows how well taxpayers’ money is being used, but it is not an easy task. The productive process of a university is a complex, multi-dimensional system with more than one objective (Agasisti and Haelermans 2016). Recently globalisation has increased the pressure on universities to improve their standards (Steiner et al. 2013). Performance measures have a significant impact on universities’ ability to attract the top scholars, bright students and research funding (Kaycheng 2015).

Comparisons of the quality of universities are regularly published in the form of ranking lists. Some UK examples are The Complete University Guide, The Guardian University Guide and the Sunday Times University Guide, all of which include leagues tables based on publicly available statistical data, e.g. from the Higher Education Statistical Agency (HESA) and the National Student Survey (NSS). These rankings have a sizeable impact on universities as they are one indication of prestige and have a direct influence on the number and quality of applicants (Hazelkorn 2007). As published rankings have proliferated so have criticisms of them (Yorke 1997; Bowden 2000; Oswald 2001; Vaughn 2002; Marginson 2007; Taylor and Braddock 2007; Tofallis 2012; De Witte and Hudrlikova 2013). The method used to rank universities is usually a simple weighted sum of the evaluation criteria, where the aggregation is fully compensatory and does not differentiate between universities with strengths in different areas. Moreover, as each criterion is measured in a different unit, the values need to be transformed into equivalent units to enable them to be summed. There are numerous ways of standardising data (commercial rankings generally uses z-transformation), and they often lead to different final rankings (Pomerol and Barba-Romero 2000). To avoid these problems, Giannoulis and Ishizaka (2010) used an outranking method. Another criticism is that input and output data are treated in the same way, which rewards inefficiency (Bougnol and Dulá 2015). For example, consider two universities with exactly the same input and output levels. The following year one of them reduces its input (e.g. facilities spend), but they continue to have the same output. The published lists will assign a lower ranking to the one which has decreased its input, although it has become more efficient. It would therefore be wise in the MCDA methodology to minimise input data and to maximise output data. Universities are facing funding cuts, so their input resources are becoming more restricted and therefore productivity is crucial. This means that efficiency should be considered alongside rankings based on various measures of perceived quality.

The data used in the analysis were obtained from the Sunday Times University Guide 2014 (http://​www.​thesundaytimes.​co.​uk/​sto/​University_​Guide/​). It evaluated British universities on the basis of nine criteria (Table 1).

Two of the criteria in Table 1 can be used as proxy of the two standard factors of production: capital (“Services/facilities spent”) and labour (“Staff”). As shown in several analysis of efficiency in Higher Education sector (among others see Thanassoulis et al. 2011) the “production” of university is multi-dimensional and often intangible. This is the reason why not only quantitative, but also qualitative measure of production should be taken into account. Based on this framework “The total number of students” can be used as a proxy of quantity production, and the remaining six criteria can be used as proxies of quality of production (“Student satisfaction”, “Research quality”, “UCAS entry points”, “Graduate prospects”, “Firsts/2:1s”, and “Completion rate”).

From a technical perspective, several of these criteria are scale-independent. Figure 5 shows the universities’ production process based on Table 1 variables. In our case, the universities use one scale-dependent and one scale-independent input to produce one scale-dependent and six scale-independent outputs.

Scale independence is not a problem in PPA as the preference function transforms all differences to a 1 to \({-}\,1\) scale. In this case study V-shape preference functions have been selected as all criteria are numerical. The preference thresholds have been chosen proportionally to the standard deviation observed on each criterion. This is a statistical approach which is consistent with the normalisation (z-transformation) used in the Sunday Times analysis. Of course other choices could be made taking into account the preferences of evaluators. The weights of the criteria are the same as in the Sunday Times university ranking, i.e. 1.5 for student satisfaction and research quality, 1 for all other criteria. In the analysis, the criterion “student–staff ratio” has been replaced by two separate criteria: the numbers of students (an output) and staff (an input) as suggested by (Thanassoulis et al. 1995).

Table 2 summarises the data. The last two columns show that some universities perform well in terms of the scale-independent outputs (Bath, Cambridge, Imperial College and Oxford). Some universities do best in terms of the scale-dependent output (Manchester). Finally, some of the small universities are best at keeping their costs low (Highlands and Islands, London Metropolitan).

In order to measure the universities’ productivity, we used the algorithms presented in Sects. 2.2 and 2.3. In particular, we first estimate the input and output net flows by Promethee, and then we measure inefficiencies by means of Eqs. (6), (7), and (8). Four universities are on the PPA frontier (i.e. \({{\varvec{\Delta }} }_{{{\Phi }_\mathrm{I}}} ={{\varvec{\Delta }} }_{{{\Phi }_\mathrm{I}}} =0)\): two of them are old universities, Cambridge and Bath, and two of them are among the most recent universities, Bishop Grosseteste and Arts Bournemouth. The distance to PPA frontier (Sect. 2.4) can be interpreted as the amount of potential improvement. Table 3 in Appendices summarises the results: \({{\varvec{\Delta }} }_{{{\Phi }_\mathrm{I}}}\) is the inefficiency inputs, \({{\varvec{\Delta }}}_{{{\Phi }_\mathrm{O}}}\) represents the inefficiency outputs and \(\min {\Delta }\) is the minimum distance to the frontier defined by the four efficient universities.

As Table 3 in Appendices is difficult to read, the PPA method allows for an intuitive, visual representation of the results, this highlights unexpected relationships that might lead to important insights.

Higher education institutions vary considerably in history, objectives and operating practices (Shattock 2013). For our analysis we separated universities into three groups:The red line in Fig. 6 represents the PPA frontier. We have used the same criterion weights as Sunday Times university ranking, i.e. 1.5 for student satisfaction and research quality, 1 for all other criteria. Only the best actions are on the frontier (Cambridge, Bath, Bishop Grosseteste and Arts Bournemouth). It is interesting to note that the new-new universities are generally located on the bottom left: they use a frugal production system. None of post-1992 universities have a positive net flow; they are not effective. None of the old universities are in the worst quadrant (bottom-right). The former polytechnics are generally located towards the bottom right and thus are generally less efficient than the other universities. A clear exception is London Metropolitan University whose position suggests that its working practices are much closer to those of the new-new universities.

In policy terms, effectiveness is expensive and requires long-term strategies; however, effectiveness is the only guarantee of the prestige that allows an institution last more than a single generation (Huisman et al. 2012; Middlehurst 2013; Filippakou and Tapper 2015). In conflicts with other authorities it is only the quality of their achievements that has enabled universities to maintain their position (Lenartowicz 2015). Moreover, in the recursive production of themselves, universities have specialised in two basic activities: research and education of students. Both are ‘axes of construction’ for self-reproduction. What is new today will be known tomorrow and will form the basis for further discoveries. Similarly, today’s students will be tomorrow’s teachers.