Academic logics in changing performance measurement systems: An exploration in a university setting

2.1 Performance measurement in universities

Since the 1990s, a radical growth of new managerial models and methods inspired by NPM have impacted university operations, with changes in governance structures, accountability mechanisms and decision-making processes (Parker, 2002). PMS, quality assessment and audit systems have been implemented in universities under the auspices of NPM reforms (Kallio et al., 2017; Parker, 2012; Ter Bogt and Scapens, 2012). Literature on this topic has grown in parallel, investigating the effects of these changes on academic work. Two types of approaches of inquiry have emerged:

1. one that has investigated the effects of these systems at a sector-wide level; and

2. the other that has looked at the effects of changes on individuals at the micro-level within organisations.

The analysis hereinafter follows these two main categories of studies.

Performance measurement takes place in different countries through mechanisms of external assessment of the research quality of universities, which have then been linked to funding to support efficient allocation of resources. A number of authors have studied various country systems. For example, the first exercise of external assessment of research – RAE: Research Assessment Exercise (currently REF: Research Excellence Framework) – took place in the UK in 1986 under Margaret Thatcher’s government (Rebora and Turri, 2013). Other European governments, such as Denmark (Opstrup, 2017), Finland (Kallio et al., 2016; Kallio et al., 2017), Ireland (Morrissey, 2013), Italy (Rebora and Turri, 2013), Portugal (Melo et al., 2010) and Sweden (Modell, 2003) have developed their own systems for measuring research performance. Other non-European countries have also introduced research assessment systems, such as Australia (Neumann and Guthrie, 2002) and the USA. Audit mechanisms have also been introduced, providing accreditation of institutions and programmes as well as the measurement of teaching quality.

In response to a growing audit culture and related government policies, and competition created by comparison of measures between institutions, universities have adapted their internal structures and management styles and developed incentives organised around the idealised model of corporate performance to enhance research excellence and impact. These changes have gradually transformed the institutional logic from “universitas” to the “entrepreneurial university” (Czarniawska and Genell, 2002; Parker and Guthrie, 2005; Pop-Vasileva et al., 2011; Saravanamuthu and Tinker, 2002; Winter et al., 2000).

Another important consequence of this transition is represented by the increase in administrative power. This effect has been associated with the implementation of a hierarchical management model which replaced the traditional committee-based model. This new model is “characterized by a significant increase in the number of ‘professional’ management appointees in the central and faculty/school bureaucracy as opposed to elected senior academics who have traditionally functioned as deans” (Neumann and Guthrie, 2002, p. 725). Parker (2011, p. 441) claims that “universities have increasingly moved toward the redefinition of university vice-chancellors or principals as chief executive officers (CEOs), with government councils being downsized and composed more in the nature of corporate boards, with a pre-dominant membership drawn from industry and commerce”.

The shift of university management to this more corporate style is, also, seen as a direct threat to the academic freedom (Taylor et al., 1998; Melo et al., 2010) of academic staff as they are used to having a greater degree of flexibility and autonomy in their work (Bellamy et al., 2003). Innovation and novelty are tempered by the constraints of the control system; this produces a strengthening of conformity and superficiality, especially in certain fields such as social sciences (as opposed to physical sciences), creating constraints on the development of multivocality and leading to ambiguity (Gedron, 2008). Nevertheless, despite the shift towards corporatisation, collaborative and collegial practices still remain in place within universities (Christopher, 2012).

With regard to individuals, several studies have looked at how the growth of performance measurement practices in academia has generated significant effects on individual academics’ behaviour.

The introduction of competitive funding mechanisms in HE is found to generally increase research productivity across all types of institutions (Creamer, 1998, Leech et al., 2015). Great emphasis is now given to outputs such as refereed journal publications (Saravanamuthu and Tinker, 2002). For this reason, journal rankings have proliferated in the realm of HE from 1999 onwards and have become one of the most central concerns within business schools (Gray et al., 2002; Parker and Guthrie, 2005; Parker et al., 1998).

Gendron (2008, p. 100) underlines that “the performance measurement scheme based on journal rankings tends to become increasingly influential in a growing number of disciplines, putting significant pressure on researchers to publish in ‘top’ journals to ensure that they can show a displayable productivity, otherwise their careers are at risk of perishing”.

One of the dysfunctional effects of these systems, for example, is that “publication is now viewed as the objective of research, rather than the dissemination of the knowledge contained within it” (Steele in Bazeley, 2010, p. 67).

In addition to the effects on researchers’ agendas, there are also individual/psychological effects.

Gendron (2008) has pointed out that such changes lead to a revolution in the academic world; as academics are no longer professors, they become performers (Gendron, 2008).

The need to deal with and adapt to the new performance management culture has led academics to feel overworked, pressured, demoralised and frustrated (Pop-Vasileva et al., 2011). For example, research performance rankings can be dangerous because they pit institutions and individuals against each other. Ter Bogt and Scapens (2012) focused on the implementation of more quantitative PMS within two European universities highlighting an increase in the use of quantitative measures of performance. These practices can induce anxiety, uncertainty and can inhibit creativity and innovation (Ter Bogt and Scapens, 2012).

The consequence of PMS is stress in the academic work, which is mainly attributed to meeting deadlines, frequent interruptions, excessive paperwork, work hours and an increase in conflicts (Dobija et al., 2019; Pop-Vasileva et al., 2011, p. 409, Thorsen, 1996; Tytherleigh et al., 2005).

Previous literature (Anderson, 2006, Anderson et al., 2002) also adds that increased workloads in respect to teaching, research and administration have had an adverse effect on the job satisfaction of academic staff.

Using a different perspective, a few studies have investigated the operational aspects of PMS within universities. In particular, Agyemang and Broadbent (2015) analyse how the management control system is connected with the processes of organisational change, and in particular, how this system is influenced by the design of management control systems in the organisation. Important context variables are, therefore, the external environmental regulators, organisational context and actions taken by the individuals who work in the organisation.

These scholars argue that all these variables interact in explaining the effects of management control systems on academics. In particular, they point to a case in which the internal management control systems developed by academics might amplify rather than weaken the controls imposed by external research performance assessment. Moreover, in their theoretical conceptualisation, Agyemang and Broadbent (2015) argue that these connections lead to a sort of “symbolic violence” process that led individuals and organisations towards a “reorientation”, often associated with individual “gaming” of the system. Dobija et al. (2019) investigate the different uses and users of PMS within universities, as resulting from external factors (isomorphic pressure) and endogenous factors. They show how uses of PMS in universities are strongly dependent on the attitudes and “reactions” of the various internal actors involved. According to the authors, when individual actors deny the utility of PMS, they are inclined to develop “resistance” strategies.

Sutton and Brown (2016) study how universities exert management control without threatening the autonomous motivation of their researchers, and in particular, how the use of PMS could prevent negative effects on academic work. Their study represents an attempt to understand the operational implications of some macro-level issues such as journal ranking, funding policies and performance assessment schemes, by examining the nature and operational consequences of management control systems within universities. They argue that, to avoid unintended effects on research, management control systems should “operate with an illusion of no control”. This illusion is achieved by designing incentive systems that favour long-term control, and that value the autonomy of researchers. Therefore, a better understanding is required regarding how PMS are designed within universities and what effects they have on individual behaviours.

This study adopts a management and accounting perspective to gain a better understanding of the case examined, as is explained in the following section.

2.2 Theoretical framework

New institutional theory (Grenwood et al., 2008) has been used in recent years to study performance measurement practices and accounting changes in organisations, in particular in HE (Dobija et al., 2019; Modell, 2001; Parker, 2011; Rebora and Turri, 2013; Saravanamuthu and Tinker, 2002; Tucker, 2016), and seems to be a reasonable theoretical basis for the purposes of this study. Neo-institutionalism used in accounting studies views PMS as “institutions” – i.e. rules and routines (Burns and Scapens, 2000) – located in a social context that have influence on peoples’ behaviours within organisations.

The term ‘institutions’ also includes the taken-for-granted assumptions within an organisation that inform and shape the actions of individuals, while these assumptions themselves are outputs of social action.

In particular, attention has been given in the literature to the institutional context and the institutionalisation process by which PMS as social practices “come to take on a rule-like status in social thought and action” (Meyer and Rowan, 1977, p. 341).

One of the most important concepts developed in the institutional theory is isomorphism, or the organisation’s tendency to confirm and homogenise itself with its institutional context in response to various external pressures in order to secure social approval or legitimacy.

Tolbert and Zucker (1983) highlight the concept of legitimacy in their study, and identify two different motivations for adoption of changes: the motivation of early adopters to improve operations, and the motivation of later adopters to appear modern, efficient, and rational, to obtain “secure social legitimacy” (Tolbert and Zucker, 1983; see also Brunsson and Sahlin-Andersson, 2000). In contrast, Fligstein (1987), instead, offers, what is defined by Greenwood et al. (2008, p. 9) as “an exogenous shock model”, showing how disruptive changes in legal frameworks enable shifts in organisational behaviour by altering patterns of incentives and opportunities (Fligstein, 1987).

The literature has identified three types of isomorphism (DiMaggio and Powell, 1983): coercive, whereby external powerful organisations force an organisation to adopt a new rule; normative, whereby pressure is exerted by a professional organisation; and mimetic, whereby an organisation imitates another because it wants to be more rational and to avoid appearing deviant or backward. Some studies on PMS in universities have recognised only one of the three types of isomorphism in response to external pressures (Carmona et al., 1998; Carruthers, 1995; DiMaggio and Powell, 1983; Fogarty, 1996); in other cases, some characteristics of combined isomorphism can be identified (Parker, 2011). The adoption, for example, of accounting standards can be considered both as a result of external coercive pressures and also the need to incorporate within each organisation the standards established and introduced by professionals (normative isomorphism) (Modell, 2001). Similarly, organisational members may take on board the legal and cultural rules and expectations of the society around them through a process that combines both coercive and mimetic isomorphism (Parker, 2011).

However, the theory has emphasised that organisations do not always mechanically follow institutional pressures, but that there are different organisational responses to the same institutional pressures (Oliver, 1991), including conformity or resistance to the new norms and values (Hyvonen et al., 2009; Jarvinen, 2006). Hence, PMS may be acted upon in different ways in universities (Dobija et al., 2019) according to the different organisational responses.

Burns and Scapens (2000) developed a framework for interpreting how management accounting changes can be (or not be) institutionalised in the organisation. According to their model, the process of institutionalisation starts with the encoding of new principles into rules which consist of formalised statements of procedures. Then, these rules must be translated into new behaviours by organisational actors so that they become institutionalised. This process requires changes in “ways of doing things”, and so it may be subject to resistance if the new rules and behaviours challenge existing values and beliefs.

Recent works in the institutional theory have emphasised the importance of variables that explains the relationship between institutions and individual actions. This evolution recognises the importance of decision-making, the power of individual actors/groups within the organisation to impose new rules, and the role played by institutional logic. The latter is defined as the “broader cultural beliefs and rules that structure cognition and guide decision-making in a field” (Lounsbury, 2007, p. 289). The notion of institutional logic in organisational studies is connected to the set of rules and practices, rewards, and sanctions that are developed and socialised by individuals within an organisation so that specific behaviours become legitimated and expected (Thornton and Ocasio, 2008).

Institutional logics, therefore, are experienced by individuals; such logics influence individuals’ mind-sets and define the behavioural roles by which actors carry out activities and take decisions.

Universities are regularly characterised by and subject to different and competing institutional logics (Ahrens and Khalifa, 2015; Bastedo, 2009; Grossi et al., 2019a; Kitchener, 2002; Kilfoyle and Richardson, 2011; Lounsbury, 2007; Narayan et al., 2017). Thus, the institutional logic approach may usefully integrate Burns and Scapens’s framework because it allows a better understanding of how individual and organisational behaviours interact in a particular academic-driven organisational context.

Institutional logic in academia has attributes that are connected these different types of with the autonomy of research, collegiality, and lack of central control (Grossi et al., 2019b). The typical values and norms that characterise “academic logic” are drawn primarily from the model of science that emphasises research freedom, the openness of research results, and rewards in the form of peer recognition (Merton, 1973). It can be considered a particular representation of the societal-level professional logic by means of which academia advances fundamental knowledge in society (Conrath-Hargreaves and Wustemann, 2019).

Institutional logics are difficult to observe directly; however, they manifest themselves in particular organisational forms, managerial practices and individual decisions (Greenwood et al., 2010; Thornton and Ocasio, 2008). Institutional logics are often conceptualised in the form of abstract sets of norms and rules that facilitate classification and comparison. For example, scholars have contrasted co-existing academic and business logics within universities. Grossi et al. (2019a, pp. 5-6) argue that academic logic “focuses on the specific nature of research activities and the interests of the academic community”, in contrast with business logic, which focuses on managerialism, audit culture, and performance control.

Academic and business logics co-exist in universities as a combination of professional/academic and managerial/administrative values, both at organisational and individual levels. These multiple logics can be in competition and may be difficult to reconcile, resulting in ambiguous goals and rules for individuals, who react by either maintaining or changing their behaviour.

This study contributes to this theoretical categorisation by adding that academic logic, within a given university setting, can be further articulated into multiple kinds of academic logics that are connected to the traditions of different disciplines. The article suggests that a deeper understanding of academics’ responses to PMS can be gained by considering the potential differences between disciplines that influence the notion of academic work, i.e. the operating values and norms applied in carrying out research, teaching, and third-mission activities in universities.

The notion of combining these three activities in the academic work is commonly rooted in academic ideology and institutionalised in academic practices. The usual assumption is that academic activities play complementary roles and are mutually beneficial (Shils, 1983). However, this representation oversimplifies the relationship and interactions between these three activities. Different academic logics are taken within the organisation insofar as the three categories of activities are viewed as distinct, rather than unitary, and as competitive and conflicting, rather than complementary. Moreover, the operational features of conducting each academic activity, as well as the rules for measuring their performance, may differ across disciplinary traditions, which can broadly be referred to as the hard sciences and the social sciences (Borlaug and Langfeldt, 2019). For instance, the hard science favours quantitative research approaches, dissemination of research results by means of journal articles, use of bibliometric indicators, spin-out commercialisation companies and patents for measuring research performance; the social sciences favour qualitative research approaches, dissemination via books, peer-review assessment and various forms of engagement for third-mission activities. More specifically, individual academics are socialised into their own discipline’s norms and rules during the lifecycle of their professional career. As a consequence, academics taking tenure and roles within universities are expected to take decisions and behave according to those specific values.

The broad concept of institutional logic is important in studies of management accounting in HE because it affects the creation of PMS, but it is less commonly investigated how institutional logics are affected by PMS. Scholars have pointed out the symbolic value of accounting rules in organisational settings and showed that these rules contribute jointly with other belief systems in creating specific logics for individual action (Dent, 1991; Hines, 1988; Hopwood, 1987; van Helden and Reichard, 2019). In particular, the use of PMS involves certain rules that operationalise symbolic values and pressures (e.g. action assumptions) that can affect individual behaviours and thus change organisational actions and routines. Within this process, actions and routines, in turn, contribute dynamically to the consolidation or the affirmation of a new particular institutional logic (Burns and Scapens, 2000).

However, it is not self-evident how external performancer measurement pressures’ are translated into organisational and individual academic responses in a university context characterised by different logics and pressures. Moreover, it is not well known how individual responses are related to the specific operational features of PMS. Increased understanding about the use of PMS in a university setting may offer insights into the effects of performance measurement in the public sector. Thus, the present study examines and categorises external pressures, the features of PMS as a response to these pressures, and the related institutional logics.

Figure 1 describes the conceptual framework used in the case analysis. Notwithstanding the fact that both academic and business logics co-existing and are in conflict in a university setting (Grossi et al., 2019a), in order to shed light on how PMS impact on individual academic behaviours, the proposed framework considers only academic logics.

External NPM pressures increase the use of PMS within organisations. In particular, academic actors are affected by pressures and logics of their discipline, but they change and translate new rules when bringing them into practice, and this occurs at two levels. At the “organisational” level, external pressures are filtered by the governing bodies of the organisation, which decide the features of PMS and create decision-making routines associated with their use. At the individual level, PMS affect individual actions and behaviours. The outcome at both levels, depends on the pressures, institutional logics, and rules acted upon in the organisation. In Figure 1, the two-way arrows indicate mutual interaction between pressures, institutional logics, and organisational responses.

3.1 Case method

Given the exploratory aim of this article, this study uses a case study approach (Stake, 1995; Yin, 2014). The university considered in this study (hereinafter, “university”) is a young and multi-discipline public university located in Italy with student numbers in the range of 30,000 to 40,000 and a dual focus on teaching and research activities. About 62 per cent of students are enrolled in social sciences programmes, the remainder in mathematical, physics and natural sciences programmes. The university is divided into four schools and 14 departments, and there are currently 954 tenured academics and 800 administrative staff employed. The university has strong research performance; it was well ranked in Italy’s Valutazione della Qualità della Ricerca (VQR) research assessment conducted in 2012 and 2016 and also achieved several “departments of excellence” rankings, winning funding in the last 2018 competition.

The university was selected because of the emphasis on quantitative metrics in its internal PMS. It represents an extreme case for two reasons: first, the university designed the PMS by replicating exactly the indicators used by the government for external performance assessment and funding; second, the university assigned the highest weight to research performance metrics in comparison with other Italian universities, as reported in a previous study (Francesconi and Guarini, 2018).

To obtain perceptions from different research cultures (Borlaug and Langfeldt, 2019; Hammarfelt and Rushforth, 2017; Horta and Santos, 2019), this study has considered a purposive-convenience sample of eight departments (out of 14). Departments were selected by type of scientific discipline –“big or hard sciences” (HSc)/social sciences (SoSc) – and their performance in the last national competition for funding departments’ research excellence by classifying them under the categories of “excellent and funded” (hereinafter “excellent”) and “not excellent or not funded” (hereinafter “not excellent”). The latter criteria enable examination of similarities and differences between faculties that exhibited relatively high or low research performance as “department of excellence” funds are granted on the basis of the previous five-year research performance (VQR) and of quality of the research development plan. Hard sciences included all scientific disciplines such as mathematics, physics and natural sciences.

Among the eight departments, four were selected from the hard science type and four from the social science type; then, they were classified according to their performance in the “department of excellence” competition. These variations in sampling were expected to elicit a range of responses in the way PMS were expected to be received, reflecting the variety of institutional logics present, e.g. organisational circumstances and discipline traditions.

Table I shows features of the sample.

In terms of composition, the selected departments ranged in size from 37 to 64 members, with interviewees from different scientific disciplines and with varying tenure: 18 of the 32 respondents were male (56 per cent) and 14 were female (44 per cent); 15 were full professors (47 per cent), 12 were associate professors (37 per cent) and five were assistant professors (16 per cent).

3.2 Data sources

Individual face-to-face interviews were considered the optimal method for collecting data on academics’ views in response to the metrics applied by the university for research assessment. This approach is useful because it offers a rich account of the interviewee’s experiences, knowledge, ideas and impressions in a particular social context (Alvesson, 2003; Chandler, 2008; Qu and Dumay, 2011). The study also benefited from participant observation and our own lived experience in the analysed case.

To obtain different perspectives, two types of interviews were developed: one for the department heads and the other for faculty members. The department heads are professors elected by peers; they have an important role in the design and implementation of PMS as their delegates seat in the governance of the university, and they are responsible for coordination of research, teaching and third-mission activities in their unit. The interviews for department heads were aimed at capturing the main features of PMS at the university level and actions taken in response to institutional pressures. Tenured professors were asked about how they give value to research, teaching or the third mission; the performance requirements as members of the faculty and the metrics that most strongly influence their behaviour and perceptions of the effects of their choices.

In this study, 32 interviews were conducted with full-time tenured professors (24) and department heads (eight) of the eight departments of the university during the period from July to December 2018. All interviewees had research and teaching appointments as required by legislation.

To introduce the topic, interviewees were asked to describe their familiarity with features of PMS in place in Italian HE and within the university. Thereafter, questions focused on understanding the extent to which their academic behaviour is related to performance metrics.

In order to avoid potential bias related to personal cognisance of the interviewees, they were selected randomly. Sampling continued until the authors reached saturation. For the purpose of this study, saturation means the saturation of knowledge (Strauss and Corbin, 1998), in that authors of this study recognised emerging patterns in the experiences of the interviewees as long as the interviews confirmed what was already perceived from previous interviews.

Most interviews were conducted jointly by the authors, and lasted for approximately 30-45 min each. Interviews were conducted in an informal manner in the offices of the interviewees as per their choice, and were recorded and transcribed for later coding and analysis. The interview data were complemented by other sources including public data and the university’s internal data (accessed by the authors, see Appendix 1), to ensure data triangulation and validity of the empirical evidence (Stake, 1995; Yin, 2014).

3.3 Data analysis

Data analysis used a reflexive and inductive process to allow categories to emerge from the data rather than relying on an a priori theoretical framework (Strauss and Corbin, 1998).

Applying a manual open coding process, the authors classified data through a multiple-step process. In the first step, transcripts were studied independently by each author, and data were initially grouped into codes for each interview. In the second step, emerging code categories were compared and iteratively agreed through multiple readings and interpretations.

In the third step, similarities and differences across departments and scientific fields (i.e. hard sciences vs social sciences) were explored and overarching issues identified. The emerging issues were triangulated with empirical data, which included university’s financial reports (2008-2018) and publications data (2009-2017). To ensure privacy, the authors dis-associated names from responses during the coding process (Creswell, 2014). A coding matrix (Appendix 2) was used to code emerging categories and themes, which have been summarised in a concept map (Wheeldon and Ahlberg, 2012) to provide a visual representation of findings of this study (Figure 2).

The following subsection briefly analyses the context of performance management in Italian universities.

5.1 Organisational response: features of performance measurement systems

Since the university’s establishment, its governance body has placed strong emphasis on the quality of research and teaching, at both at the departmental and individual level; however, the departments themselves have essentially been left to decide what to do and how to do it within a traditional logic of autonomy. In the past, the assessment of professors’ performance has been solely related to the specific values and rules of each discipline, and the allocation of resources to departments has been managed collegially on a democratic basis, but with the rector (who had a scientific background in HSc) exerting a strong influence on decision-making. Traditionally, however, management and governance at the university has been conducted in a collegial manner with the involvement of professors in decision-making at different levels, which contributed to the recognition of the university as a good employer.

This situation started to change in 2013 when a new rector (with a scientific background in HSc) assumed office along with a new team of pro-rectors. The new governance was particularly inclined towards externally funded research and incentivised academics to apply for and win research grants. In particular, an ad hoc research fund was established to sustain grant submissions that achieved good external evaluation but not enough to be funded.

The rector also presented a new vision of excellence based on performance, followed by a re-centralisation of departmental processes and the development of a PMS for assessing research and teaching activities.

Research performance measures started to gain importance in decision-making in 2013 following the publication of results of the first round of VQR 2004-2010, in which the university achieved a high ranking. A relevant external driver was the decision of the central government to start using VQR research performance data for allocation of resources to universities (PBF). After this decision was announced, there was a considerable emphasis from the university’s governance body on the rearrangement of internal research management control systems and incentives for academics to be successful in VQR. It seems that here, the alignment between NPM’s external pressures and the dominant values of academics sitting in the governing bodies has favoured the inclusion of control mechanisms and performance metrics in the academic logic. The new pro-rector for research, a highly ranked and internationally well-known scientist, led the introduction of a new PMS for faculty staff allocation to departments that was applied in 2014 and is still in use at time of writing. The allocation was based on an internal ranking of departments resulting from a performance formula that precisely replicated the research and the teaching indicators considered in the PBF, but gave higher weight to the former. In particular, 50 per cent of resources were proportionally distributed on the basis of departments’ VQR research scores, 30 per cent on the basis of indicators of teaching efficiency (number of regular students, completion rate and students’ international mobility) and 20 per cent on turnover needs of departments. Because national-level VQR scores are only renewed every five years, in practice, for five consecutive academic years, departments that had achieved high VQR scores in the most recent national exercise inevitably attracted more professors and postdoctoral research positions, until new national level VQR research scores became available. Other perspectives related to teaching outcomes, department’s research and teaching needs and objectives, or third-mission activities are not considered. The new rules contributed to shape the academic logic towards placing an increasing value on research performance.

In parallel with the measurement of research activities, the university implemented a QA system for teaching activities based on output performance indicators required by ANVUR. The university’s governing bodies placed strong emphasis on measuring customer satisfaction for teaching and, since 2017, a mandatory minimum threshold of teaching performance score has been required for the triennial salary increase of tenured professors. Student satisfaction data are publicly disclosed on the university website at departmental and course level. However, as it will be discussed in the following sections, it appeared that teaching performance metrics were not perceived by professors to be strong incentivising mechanisms, because these were neither considered for departmental resource allocation nor for career advancement decisions. Table II summarises the features of the PMS identified in the case.

The design of PMS within the university has been developed by its governing bodies in a way that it recreated within the organisation the same external factors, i.e., – the same measurement systems implemented by the Italian government, that it sought to deal with. This approach was not only the result of adaptive isomorphism (DiMaggio and Powell, 1983; Parker, 2011) to external pressures emerging from funding mechanisms, but also a result of the specific prevailing academic logic and the way academic power was exerted by professors who held positions in the governance bodies.

The process of isomorphic adaptation took place over a number of different steps. VQR results and the related PBF created disturbances to the established internal value system of the university. To cope with these changes and maintain its financial position in this environment, the governing bodies had an incentive (coercive isomorphism) to develop an internal PMS by linking the research performance, which they wished to control, to the same metrics used by the external VQR and performance-based funding. The intention of the new system was to align department goals with the university’s priority of succeeding in VQR grant competition, as a response to external pressures. Because Italian universities cannot increase tuition fees, changes in the size of research block grants can have significant effects on the budget of any university. It should not be surprising that the university sought to align the two systems because in designing performance management systems, organisations take into account the key factors relevant to their success (Ferreira and Otley, 2009; Johanson et al., 2019).

The following extract from the university strategic plan provides evidence of how the PMS is shaped by the external environment as a condition for survival and achievement of the university mission:

The internal allocation of resources is based on performance and transparency. We follow the principle of accountability so that public money achieved by the university for research performance will be re-allocated to departments according to their results. This approach will strengthen the virtuous cycle between scientific productivity, reward, and reinvestment in research capabilities to be successful in grant competition. On the basis of this premise, we aim to develop new interdisciplinary research to create value for research and teaching innovation (University Strategic Plan 2018-2020).

In such a way, the external performance assessment mechanism of VQR was internalised into the university PMS by creating a system of financial incentives and rewards targeted at departments. The internal ranking of departments for the allocation of FTE faculty staff became so a calculative mechanism at the disposal of the rector for legitimising the new set of values (i.e. to be successful in research and funding), thus facilitating the incorporation of some aspects of the business logic into the academic logic.

In 2017 the university was again ranked among the top universities in Italy in several disciplines according to the VQR 2011-2014, which reinforced the perception held by the rector that giving greater emphasis to research performance metrics in faculty-related decisions was the right strategy. Again, the PMS provided some illusion of objectivity for the governing bodies and made the management of performance easy to control:

Our performance measurement system is designed and developed collectively and is grounded on evidence. For example, the assessment of scientific performance follows the criteria of VQR and, as such, stated by an external body at national level (University Strategic Plan 2018-2020).

Moreover, it seems that the exercising of power may also be implicated in this process (Hoffman, 2011). Leading actors in the university may try to impose their personal vision of academic work by claiming legitimacy for the PMS based on its relationship with the external environment (Moll and Hoque, 2011):

Our PMS is designed along the metrics used in PBF. Part of the state grant achieved on an FTE cost per student basis is useful to fund operations related to the number of students, but it is not performance-based. We are a research university, so we can’t allocate FTE faculty hiring according to the number of students in each department. Our teaching activity is driven by research (Authors interview, Francesconi and Guarini, 2018).

Power may also be instrumental for specific academic groups sitting in governing bodies to take advantage of the PMS to obtain more resources. Academics in governance roles hold a managerial responsibility and as such, their institutional decisions are naturally influenced by their own academic logics. This kind of influence seems to have occurred at the university in decisions related to the design of the PMS.

It is worth noting that the eight interviewed department heads declared that they were not involved in the design of the formula-based mechanism for resource allocation. The analysis of the board meeting minutes reveals that the decision about the higher weight to be assigned to research metrics in the formula was driven by the rector and other professors from hard sciences backgrounds sitting on the board, and no official discussion about the effectiveness of such system was reported. This issue also underlines how external pressures have been incorporated into the academic logics and how the process has been influenced by the power held by key actors.

Moreover, most of the university’s revenues are related to the number of students, rather than being money coming from research performance (Figure 3), which means that teaching is now cross sub-subsidising research at the university.

University’s turnover was €235.3m in 2018, made up of €90.4m (38 per cent) from the government block-grant, €44.7m (19 per cent) in research income (VQR and other competitive grants), €79.7m from teaching (34 per cent), with about €34.1m of this teaching income funded by the Italian government (FTE cost per student grant), and €20.5m is from miscellaneous other sources (9%) (authors’ calculation based on university financial report, 2018).

Likewise, about 39 per cent of FTE faculty hiring authorisation awarded by the central government to the university is driven by the number of students enrolled (i.e. tuition fees and the state grant related to FTE cost per student), whereas research performance-related revenues count for about 13 per cent of government authorisation (authors’ calculation based on central government data).

Hence, in contrast with previous literature that has argued that the emphasis on performance measurement within universities was an effect of NPM logic and tools adopted by university managers (Neumann and Guthrie, 2002; Parker, 2011), in this case, there is evidence that such emphasis may occur even when some professors – those who are members of the university governance bodies (e.g. head of departments, and academic staff representatives) are fully in charge of the design of the PMS. Previous studies (Kurunmäki, 2004) have suggested that in organisational contexts characterised by the dominance of professionals, competencies in the design of PMS can also be developed by non-accountants (Orton and Weick, 1990). This was not the case here because professors involved in the university’s governance were not involved in any learning initiative aimed at developing competences associated with university PMS.

5.2 Individual responses to performance measurement systems

Previous research (Kallio et al., 2016) has underlined the need to understand how academics react to and make sense of the shift caused by managerialism and PMS within HE. The following sub-sections analyse and discuss the respondents’ explanations of how they responded to external pressures and the internal demands exerted by PMS and the extent to which their behaviour was related to PMS.

The way in which academics in the present study have dealt with institutional pressures seems to have been guided by their individual career expectations. Indeed, “individuals plan and prioritize different academic activities to reach the maximum output” (Grossi et al., 2019a, p. 16). Interviewees highlighted the importance of career advancement in driving their research choices, with differences noted between HSc and SoSc faculties:

In my field of research there are very few top-ranked journals. I tried to publish many of my research outputs in these journals. In my opinion other journals would have been more relevant but I had to choose a top-ranked journal. Maybe when I’ll become a full professor, I will change my behaviour and avoid these “quantitative cages”. I use the term “cage” because the metrics are conditioning me in all my choices (male associate professor – SoSc).

For our career promotion, we have to demonstrate a number of articles ranging from 6 to 11, depending on the discipline, and a minimum number of citations (female assistant professor – HSc).

The relevance of PMS was observed by all faculty members who claimed to have familiarity with the system and a good knowledge of the way performance measures are used within the university with regard to individual research performance requirements.

Although interviewees recognised that the emphasis on performance measurement has greatly increased the transparency and objectivity of academic performance considerations in career advancement, as opposed to practices of nepotism, they have also expressed criticism of procedures and methods (e.g. use of journal rankings). Interviewees emphasised that the new PMS is affecting their academic priorities and actions with different intensities and nuances, depending on their specific traditions of their discipline and their individual career expectations. Because the university incorporated into its PMS the performance metrics used for VQR and funding assessment, it was difficult to distinguish between individual responses driven by external quantitative metrics and those driven by internal organisational demands, which sometimes appeared to have been overlapping.

Three different types of individual reactions emerged from this analysis. The first type of reaction, which has been labelled as “detachment”, has been reported by two interviewees who claimed a lack of interest in changing their behaviour as a consequence of new performance metrics. They described their motivation in terms of their inherent notion of academic work, or in terms of criticism of the current research assessment systems:

I think that managing operations as a coordinator of a degree programme, as well as finding solutions to everyday problems, is for me more important than QA performance metrics per se (female full professor – SoSc).

[…] I do exactly the job I have to do, unfortunately I did not get the qualification to become a full professor. I don’t trust VQR and its research performance metrics. Publications are valued with different metrics in VQR and in the national qualification assessment (male associate professor – HSc department).

The second type of reaction has been categorised as “business as usual”, identified in seven interviewees from HSc departments who did not report any special change in their research behaviour. Consistent with previous research (Reale and Seeber, 2011), this reaction was associated by the respondents with the fact that they were already acquainted with the use of quantitative measures in research evaluation, or because their performance was rated excellent in VQR and the university’s internal ranking for resource allocation. It therefore seems that PMS implemented by the university were consistent with their academic logic and did not require any relevant modification to their behaviour.

Third, several other academics, from both HSc (six) and SoSc (seventeen), reported a radical or partial change in their ways of doing research and of prioritising academic activities as “a consequence of VQR and PMS implemented by the university”. This reaction has been categorised as “epistemic reorientation” because of the nature of the influences that quantitative metrics are exerting on the academic practices of knowledge production. The features of this response are described hereinafter.

Comments made by interviewees and analysis of data were compared to check whether there was any difference between different disciplines/departments. This analysis revealed that there were consistent patterns in responses made by interviewees from HSc and SoSc working in departments with different levels of research performance. Indeed, interviewees that reported a radical reorientation of their behaviour (twenty-three) were primarily from SoSc-“not excellent” departments (ten) with low scores in both the VQRs and the university’s internal ranking, and from HSc-“not excellent” departments (three). “Reorientation” also characterised interviewees from “excellent” SoSc (seven) and HSc (three) departments, especially assistant and associate professors with career expectations (seven). In line with the proposed theoretical framework (Figure 1), it could not be excluded that reorientation leads to changes in the individual’s academic logic which, in turn, further shape individual responses to become “business as usual” behaviour.

Although it is not possible to identify causality using the current research design, these findings suggest that PMS have to some extent re-oriented academics and departments as a consequence of those individuals’ efforts to align their performance with the new metrics, whilst attempting to maintain access to resources and maximise individual career outcomes. This is not surprising because the PMS aims to re-orientate individual behaviours and the analysed case has been selected because of the strong importance allocated to research assessment, which also contributes to the explanation of why the most predominant reaction to the changes introduced by the PMS is represented by reorientation.

Previous research by Sutton and Brown (2016, p. 595) has underlined that in resource-constrained organisations, such as universities, incentive systems that are based on research work-related rewards (e.g. increased research funding and staffing, career advancement) are very effective in driving academic behaviour towards undertaking greater volumes of research-related activities. In the analysed case, the strong linkage between VQR research performance metrics and departmental resource allocation, combined with research performance requirements for career progression, have created a “closed incentive system” that values only individual research performance. In such a system, performance metrics become an instrumental device for valuing time allocation to research, teaching and third-mission activities. Although this incentive scheme may be rewarding for highly motivated researchers, it may surreptitiously alter the commonly accepted notions of academic work and the university mission. The following quotes are indicative of this change in academic logic among academics when deciding priorities in their academic agenda:

In the past I spent a lot of time supervising students, but now I have to focus my efforts on research […] I had to study a lot to change my research subject, so I had to reduce teaching activities (female assistant professor – HSc).

I am fully involved in many activities; teaching is the last of my thoughts, unfortunately. I’m not satisfied, but on the other hand I cannot dedicate more time (female full professor – SoSc).

Now, it is more difficult to find colleagues willing to collaborate in teaching activities as these are not very valued in internal resource allocation (female full professor – HSc).

The current teaching performance indicators are mainly related to outputs and do not appear to be useful for evaluating the effects of this reorientation in attitude. When respondents were asked about departmental demand for their individual performance, all of them mentioned only VQR and research productivity metrics. This was particularly underlined by academics from departments in low positions in the internal FTE faculty allocation ranking, suggesting awareness of the increasing internal pressures related to the use of research metrics in university decisions. VQR aims to assess the research quality of departments and universities, rather than of individuals. However, because VQR metrics have been institutionalised within the university as a basis for resource allocation, these measures also affect the behaviour of individual academics.

Teaching performance was mainly assessed based on student satisfaction rates and the target did not appear to be challenging, nor one with many consequences if it was not met. In this case, use of rhetoric prevails over substance. It is worth emphasising here that although this metric was included in the salary advancement rules for faculty members, the required threshold was not challenging for academic staff to meet. Moreover, there is not yet any formal linkage between individual teaching performance and career promotion metrics:

Teaching performance is not very relevant either at departmental or university level. Above all, nobody cares if you get low performance scores (female associate professor – SoSc).

I would say that we are perhaps more accountable for research, because there is no real accountability on the level of teaching at the department level. Yes, we have measures of student satisfaction for each course, but you are required to take action only if you get low scores (male full professor – SoSc).

We have learned the hard way that VQR’s metrics do matter. My department got a low performance score in the last VQR so that now we can’t hire new staff because of the University’s resource allocation system. I had to raise awareness of the importance of dealing with research performance metrics above all. Now professors have started to be aware, also because their results impact the entire department (female department head – SoSc).

Similar comments were made regarding third-mission activities, with interviewees stating that these are not valued for either career advancement or performance funding, and performance metrics are still under development.

VQR research metrics also seem to have been altering the process of knowledge production in terms of the types of publications produced and how research is conducted. Changes in publication behaviours have been identified by interviewees from SoSc – either “excellent” or “not excellent” departments – who have reported the abandonment of publishing books in favour of journal articles. These changes have been attributed to the assessment criteria adopted by VQR, which conflicted with their discipline’s evaluation traditions:

I focus more on ranked journals. This seems to me a nonsense. In the past, books were well considered. Now, a research monograph is valued less than an article published in a journal. Writing a good book takes a lot of time. In the past this effort was valued; now it is not (male assistant professor– SoSc).

There is also evidence that these metrics are influencing the selection of which journals to publish in of academics, an issue that has also emerged in other contexts (Gray et al., 2002; Parker et al., 1998; Parker and Guthrie, 2005; Saravanamuthu and Tinker, 2002). For example, the choice of publishing outlet is influenced by journal rankings and related metrics:

My publishing choices are influenced by journal ranking. Sometimes I choose well-known open access journals if I need to publish quickly (male assistant professor – HSc).

[….] there is an obsessive attention on where to publish (female full professor – SoSc).

In my research group, we look for the most cited journals; it’s like a business. In the past, you got a research result and just wanted to publish it, now you start with a publication strategy (male full professor – HSc).

Because academics have an incentive to choose topics that maximise their research performance indicators (Martin, 2001), the use of metrics has important epistemic consequences for the planning of research projects:

It is very likely that researchers are oriented to follow more fashionable research topics, i.e., more fertile research streams in terms of publications (male associate professor – HSc).

In our field there are fashionable topics which you can rely on in order to publish in top-ranked journals (female full professor – SoSc).

I had to switch my previous research topic because it was not in the mainstream. Now I’m fine, I managed to get in touch with colleagues I work well with, but I know other academics who have not been as successful as I was (female assistant professor – HSc).

Again, it seems that research performance metrics also bias the dynamic of research collaborations, in that early-career scholars have an incentive to select only research groups and topics that maximise the chance to publish and obtain research funds, to ensure career progression. The current research requirements for career development in Italy, for which publishing in top-ranked international journals is pivotal for certain disciplines, is acting as an important coercive isomorphism factor influencing individual academic responses:

Strong research groups always tend to become stronger and stronger and influence the career of young scholars. My research group is small and led by an assistant professor, so it is more difficult to proceed with career advancement (male assistant professor – HSc).

Research groups are also important. Small research groups have been marginalised. If you do research in a strong group of colleagues, then your research productivity improves a lot (male assistant professor – HSc).

In the past I was not in a strong research group and so I’m still an assistant professor (male assistant professor – HSc).

We must also consider that it is very important to accumulate citations in a short time. The more people participate in the work, the more quotes you can get (female assistant professor – HSc).

Metrics are also very important for funding; you do not get funds if you are not well placed in a group, you have to jump on the “wagon of the winners” to be able to publish well and get funded (female assistant professor – HSc).

Despite the fact that choosing journals by looking only at rankings is believed to be counterproductive to the advancement of knowledge and also somewhat detrimental to knowledge dissemination (“the third mission”), metrics lead such decisions regardless of the selection bias. The following statements illustrate this contradiction:

In my field of research there are very few top-ranked journals. I tried to publish many of my research outputs in these journals. In my opinion other journals would have been more relevant but I had to choose a top-ranked journal. Maybe when I become a full professor, I will change my behaviour and avoid these “quantitative cages”. I use the term “cage” because the metrics are conditioning me in all my choices (male associate professor – SoSc).

Educating future leaders is our most important responsibility. As a researcher I have to publish in top-ranked journals. I do not publish any more in any other journals, so I don’t do science, I don’t create culture. Here is the paradox: we create less culture; culture is produced independently of the fact that I publish in top-ranked journals. I can develop culture by writing in a newspaper, doing dissemination, making conferences, seminars, etc. (male department head – SoSc).

Another argument raised by interviewees from SoSc departments is that the higher scores attributed to high-impact international journals in the VQR have led to the progressive abandonment of research topics considered more relevant at the national, rather than international, level, as well as to the downfall of important Italian journals:

[…] there has been an impoverishment of research activity useful to our industries and banks in the national context. We are focused only on international mainstream research topics (female full professor – SoSc).

We do not publish any more research in Italian. We only publish for international English-language journals. Nothing else (male associate professor – SoSc).

Each discipline has different research peculiarities. The main issue in our field [banking and finance] is that we have a very limited number of top-ranked journals in comparison with other management fields. Moreover, it has become increasingly difficult to publish in these few top journals, whereas good Italian journals are low-ranked according to the H-index (male associate professor – SoSc).

This type of “reorientation” seems also to be encouraged by SoSc department heads, who have reported that they are allocating department funds for proof editing services in order to support publications in English-language journals. This is a clear example of how certain social and normative values embedded in PMS change actors’ organisational routines and, at the same time, reinforce the prevailing institutional logic.

Furthermore, reorientation was as a response to the demand for increased research productivity, which was reported by academics either from HSc or SoSc “not excellent” departments (Figure 4). Consistent with previous studies (Creswell, 1985; Creamer, 1998; Edgar and Geare, 2013; Gendron, 2008; Leech et al., 2015; Martin-Sardesai and Guthrie, 2018), reorientation has resulted in an increased quantity of publications:

The existence of a research evaluation system pushes you to work harder. Since the VQR has been implemented, I have set quantitative targets that have led me to publish more (male full professor – SoSc).

My research productivity has increased in recent years, compared to what it was in the past years. I well remember that some years ago there were tenured colleagues who did not publish at all (female full professor – SoSc).

Imagine, for example, that we have data for a very significant research work which is very risky and uncertain in terms of results or two medium-quality works that are fast and easy to publish, what should we do? We decide on two medium-quality works (female full professor – HSc).

Finally, what emerged from the interviews is that PMS have affected faculties from social sciences more significantly than those from hard sciences, who appeared more at ease with the system. This also shows that the academic logic, although characterised by common values (such as collegiality), bears different nuances in different fields of research.