The UK Teaching Excellence Framework (TEF): The Development of a New Transparency Tool

Abstract

The results of the first UK Teaching Excellence Framework (TEF) were published in 2017, with each HEI receiving a Gold, Silver, or Bronze award. The TEF is comprised of three components: teaching quality, including student satisfaction; the institutional environment in which students learn; and student outcomes, including the performance of under-represented groups. This new classification, solely assessing the ‘teaching mission’ of the university produced an unfamiliar hierarchy of institutions. This chapter provides an overview of the rationales for the TEF as set out by policymakers. It also accounts for the development process of a new transparency tool which involved defining and measuring ‘teaching excellence’. It argues that the TEF can be viewed as a multi-purpose instrument which seeks to provide accountability (by providing performance data to inform applicant choice and fee rises) and drive improvement (through raising the importance of teaching and learning within HEIs). The analysis explores where the TEF is situated relative to established Quality Assurance agendas on the one hand, and wider government ambitions to reconfigure the higher education sector on the other. The paper concludes with a summary of what the wider EHEA community can learn from the implementation of the TEF.

Context

Transparency Tools and Quality Assurance

Hazelkorn (2015) notes the explosion over the previous decade of “what are euphemistically called transparency and accountability instruments and tools, operating in tandem but differing with respect to purpose, policy orientation and user”. The calls for greater transparency are a “reaction to the increasingly more inquisitive and demanding environment in which higher education operates” (Hazelkorn 2012, p. 348). This is where political and societal support for higher education can only be maintained through the availability of greater information to provide individual and public investor confidence (Hazelkorn 2012).

Van Vught and Ziegele (2011) define transparency tools as “instruments that aim to provide information to stakeholders about the efforts and performance of higher education and research institutions” (p. 25). The authors point out that transparency tools are related to quality assurance processes which also produce “information to stakeholders (review reports, accreditation status) and in that sense helps to achieve transparency” (p. 25). Moreover, Harman (2011) finds there has been increased interaction between quality assurance systems and transparency tools, such as institutional rankings, in recent years.

It is useful to establish how “transparency tools” differ to “quality assurance”. In origin, quality assurance comes from within the higher education community whereas transparency tools tend to be imposed from outside. Quality assurance is rooted in an ethos of institutional autonomy and the principles of peer review undertaken by self-regulating professionals. It is focused on assurance, and increasingly concerned with enhancement, rather than performance measurement and comparison (Neave 2014). Transparency tools, alternatively, may serve agendas and stakeholders outside the academic community, and they typically have characteristics more akin to external audit and public scrutiny. They intend to measure and compare performance and then share this information to expose good and bad provision.

This chapter charts the development of a new transparency tool in the United Kingdom (UK): the Teaching Excellence Framework (TEF). Following a discussion of the context, the second section provides an overview of the framework. The third section features an analysis of the TEF, which is followed by a conclusion.

Transparency Tools in a Governance Context

Recent policy developments in the UK, most notably the launch of the TEF, can be seen as another chapter in the application of New Public Management (NPM) ideas to higher education. NPM is a policy paradigm providing a series of principles which have been used to reform public sector organisations since the early 1980s (Gunter et al. 2016; Radnor et al. 2016). It is an amalgam of theories within New Institutional Economics and a strand of management theory that has been generically applied across the public sector. The consensus emanating from this amalgam represents a compromise between the interventionist bias of the market failure paradigm—where it is the duty of the state to intervene to prevent market failure, and the case made by the public choice variant of the government failure literature—there should be limited government to enable consumer choice in the market. NPM, therefore, guides policymakers towards an appropriate role of government in a market economy and the most socially desirable forms of government intervention (Wallis and Dollery 1999).

The policy prescriptions emanating from NPM seek to empower the consumer in a competitive market, ending the dominance of “producer capture” (Trowler 2001). The resulting operating environment involves complex systems of evaluation and national schemes of performance indicators implemented by the “evaluative” or “regulatory” state (Neave 2012). Within higher education institutions (HEIs), this has resulted in the rise of strategic planning and more managerial forms of organisation (Khvatova and Dushina 2017). This is often at the expense of the professional autonomy of academics and represents a shift from more collegial forms of governance (Yates et al. 2017). For policymakers, this is justified, as the “black box” of the university needs to be “opened up” to public scrutiny and regulatory oversight. In furthering this agenda, we can see how transparency tools can be a very useful addition to the policymaker’s toolbox. This is where such tools can be used to extract information on institutional performance to enhance accountability and inform decision making.

An Evolving European Policy Context

Since 2000, the process of open coordination has accelerated the formation and visibility of an EU Learning Area (Lawn 2006). Here, both quality assurance and enhancement have been promoted by the European Higher Education Area (EHEA) (Gvaramadze 2008). Recent developments within policy show a growing interest in both transparency tools and the pursuit of teaching excellence.

A review of transparency tools published by the European Association for Quality Assurance in Higher Education by Vercruysse and Proteasa (2012) identified that delivering transparency should be regarded as an important aspect of higher education policies. It recommended transparency tools should be based on evidence of users’ information needs, and their capacities to process information. According to Vercruysse and Proteasa (2012), the primary function of transparency tools is to provide information to a diverse range of users including students, businesses, academics, policymakers, and institutional leaders. The authors argue each of these groups has diverse information needs and expectations, probably making it impossible for transparency tools to meet all individual demands at once (Vercruysse and Proteasa 2012, p. 11).

The report also recommended governments should encourage the development of new indicators and processes to enable HEIs to identify and communicate their various profiles, especially the teaching and third missions which are marginalised in existing transparency tools (Vercruysse and Proteasa 2012, p. 11). An example of this is the U-Multirank project funded by the European Commission, which highlights how existing international rankings tend to focus on a single dimension of university performance—research (Boulton 2011; Van Vught and Westerheijden 2012). The project developed a new multidimensional ranking designed to recognise the diverse range of activities—including teaching—that take place within HEIs (Van Vught and Ziegele 2011, p. 17).

Sin (2015) charts how learning and teaching have migrated from the margins to the core of European higher education policy. This is because policymakers increasingly regard teaching as being critical in delivering the utilitarian and economic objectives assigned to higher education in policy discourse (p. 338). The massification of university-level education, and the need to find a way to finance this, also accounts for the increased attention paid to teaching quality and measuring graduate outcomes. The rising cost of higher education and concerns about the sustainability of the graduate salary premium have exacerbated this trend. There are, therefore, legitimate reasons, beyond the imperatives of NPM, which account for the need to subject the teaching mission to greater transparency.

Transparency Tools and the Pursuit of Teaching Excellence

The enhanced importance of teaching is evident in various reports. For example, in 2013 the European Commission received the report Improving the quality of teaching and learning in Europe’s higher education institutions from the High Level Group on the Modernisation of Higher Education (McAleese et al. 2013). The report made a series of recommendations for public policy makers and HEIs concerning creating policy and institutional environments and professional practices to foster quality teaching. Specifically, it recommended public authorities “should ensure the existence of a sustainable, well-funded framework to support institutions’ efforts to improve the quality of teaching and learning” (McAleese et al. 2013, p. 64).

In 2014 the European Association for Quality Assurance in Higher Education published the report Concept of Excellence in Higher Education by Brusoni et al. (2014). The report argued:

The development of a commonly accepted “framework for excellence” would provide a basis for a more strategic approach to quality improvement, allowing institutions to measure their performance against defined criteria and facilitating the comparative analysis of institutional performance as an alternative to league tables and rankings (Brusoni et al. 2014, p. 37).

The 2015 revisions to the Standards and Guidelines for Quality Assurance in the EHEA set the future direction of quality assurance in the region. The report makes a more explicit link between quality assurance and the learning and teaching processes within institutions (ESG 2015), highlighting the need for more attention to be paid to the latter.

These reports follow a range of recent policy activity and academic literature in making the case for national policymakers to nurture teaching excellence, through producing frameworks which influence institutional behaviour. We can see from reports such as these the direction of travel in policy thinking and the role of government-driven transparency tools in achieving these objectives. Policy change was driven by a growing realisation that existing transparency tools and popular rankings—such as the Academic Ranking of World Universities (ARWU), and the Leiden Ranking (LR)—had inappropriately overemphasised the research mission of the university. This has negative ramifications for the teaching mission which was gaining greater recognition. New transparency tools were, therefore, required to meet the information needs of a range of stakeholders and enable them to navigate the more complex and competitive higher education landscape. In particular, there was a perceived need to provide prospective students with better intelligence on teaching quality and graduate outcomes. In this context, there is a role for state intervention, as Ciolan et al. (2015) argue, to verify and standardise all information and ensure it is usable and comparable (p. 27). The policy debates within the UK and rationales for the TEF are situated within this direction of travel.

Teaching Excellence and the Teaching Mission

There is no one definition of teaching excellence. As Skelton (2005) points out, teaching excellence is a contested, value-laden concept located within a shifting social, economic and political context (p. 4). Definitions can be nebulous and contradictory and are connected to a wide range of contested and disputed, long-running academic debates (Greatbatch and Holland 2016). This presents many challenges when developing any kind of evaluative framework for the teaching mission of the university.

Assessing the quality of teaching has been a long-standing issue for higher education systems internationally. It remains more challenging than assessing research quality. This is because there is greater agreement around the definition of research excellence and the indicators that can be used to assess this (Trigwell 2011, p. 165). This measurement issue also explains the dominance of research in university rankings globally. Courtney (2014) identifies three main reasons for the scarcity of teaching assessment exercises worldwide: first, the conceptual and practical difficulties in defining, operationalising, and measuring either excellence or basic standards; second, a lack of consensus on the difference between excellent teaching, teachers, learning, and learners; and third, the problematic conflation of competence and excellence. The idea of “excellence”—as opposed to competence or simply meeting expectations—is particularly challenging. Brusoni et al. (2014), who advocated the “framework for excellence”, acknowledge that relating the methods of quality assurance to excellence is probably one of the most difficult tasks facing quality assurance agencies (p. 37).

The complexities within these enduring educational debates present obstacles when developing an assessment framework. There is also an additional dimension: the difference between assessing “teaching” in a university and the “teaching mission” of the university—a sibling to the “research” and “third” missions. The former is largely concerned with pedagogy, while the latter is a much broader entity, incorporating the inputs, processes, outputs and outcomes stages of the whole student lifecycle. The teaching mission includes: applicant information, entry requirements and widening participation; the student experience and teaching quality on the course, as well as the resources available to learners both online and on campus; outputs such as student retention and degree classifications; and outcomes including graduate destinations and personal development (Gunn 2018, p. 135).

The next section considers the development of a new transparency tool to measure the teaching mission of the university. Although the TEF is a UK-wide initiative, the analysis presented here applies only to England.¹

The Framework

Policymaker’s Rationales for the Teaching Excellence Framework

The UK Government has four stated objectives for the TEF; being introduced as a way of:

• (a) Better informing students’ choices about what and where to study

• (b) Raising esteem for teaching

• (c) Recognising and rewarding excellent teaching

• (d) Better meeting the needs of employers, business, industry and the professions (DfE 2016, p. 5).

The TEF responds to several concerns within government regarding the higher education sector and how it serves the economy. To address these concerns, the teaching mission of the university needed to be subject to greater transparency. For example, the government sees information about the quality of teaching as being “vital to UK productivity” (DBIS 2015, p. 19). This is because in the absence of information about the quality of courses and skills gained, it is difficult for employers to recruit graduates with the right skills and harder for providers to know how to improve their courses. Improved information, it is argued, may also reduce the mismatch² between graduate jobs and the skills within the graduate cohort (DBIS 2016a, p. 11).

There was also the perception that applicants were poorly-informed about the content and teaching quality on courses, as well as the job prospects they can expect; which can lead to poor decisions by the student as to which course and institution to attend. This was explained by the lack of information, which meant there was little pressure on institutions to differentiate themselves, or fully and accurately advertise their offerings. The government also believed “information, particularly on price and quality, is critical if the higher education market is to perform properly. Without it, providers cannot and students cannot make informed decisions” (DBIS 2016a, p. 11). A 2017 National Audit Office³ report supported the government’s position, finding prospective students in England to be in “a potentially vulnerable position”—lacking adequate information to make informed choices, and, in effect, victims of “mis-selling” (NAO 2017).

Through the TEF, the government sought to introduce two things to English higher education for the first time. First, is the introduction of a national framework to bring “sector-wide rigour to the assessment of teaching excellence” (DBIS 2016a, p. 44). Second, is a shift to financing teaching on a quality basis, something which occurred for research funding three decades ago.⁴ This is based on the view that “for too long, we have funded teaching on the basis of quantity, not quality. This is in sharp contrast to research, with its quality-driven funding stream allocated through the Research Excellence Framework” (DBIS 2016a, p. 43). This would be achieved by allowing HEIs with good TEF results to raise their undergraduate fees.

Developing a New Transparency Tool

Paying deference to the higher education community, the TEF was not simply designed and imposed upon the sector. Development of the new framework began in 2015, going through a series of policy development stages—known as Green and White papers⁵—each of which provided opportunities for comment (DBIS 2015 and 2016a, b). Further technical consultations⁶ and a process of “co-development” between the government and academy not only helped develop the novel methodology, but also allowed the academic community to become stakeholders in the process, thus giving the TEF greater legitimacy.

Through this process, a new assessment framework was developed to encompass the breadth of the teaching mission. This comprised of three components. The official descriptors of these are set out below:

Teaching Quality: includes different forms of structured learning that can involve teachers and academic or specialist support staff. This includes seminars, tutorials, project supervision, laboratory sessions, studio time, placements, supervised on-line learning, workshops, fieldwork and site visits. The emphasis is on teaching that provides an appropriate level of contact, stimulation and challenge, and which encourages student engagement and effort. The effectiveness of course design, and assessment and feedback, in developing students’ knowledge, skills and understanding are also considered. The extent to which a provider recognises, encourages and rewards excellent teaching is also included in this aspect.

Learning Environment: includes the effectiveness of resources such as libraries, laboratories and design studios, work experience, opportunities for peer-to-peer interaction and extra-curricular activities in supporting students’ learning and the development of independent study and research skills. The emphasis is on a personalised academic experience which maximises retention, progression and attainment. The extent to which beneficial linkages are made for students between teaching and learning, and scholarship, research or professional practice (one or more of these) is also considered.

Student Outcomes and Learning Gain: is focused on the achievement of positive outcomes. Positive outcomes are taken to include:

• acquisition of attributes such as lifelong learning skills and others that allow a graduate to make a strong contribution to society, economy and the environment,

• progression to further study, acquisition of knowledge, skills and attributes necessary to compete for a graduate level job that requires the high level of skills arising from higher education (DfE 2016, p. 19).

Taken together, the three aspects are designed to provide a balanced view of learning and teaching quality and to enable diverse forms of teaching and learning excellence to be identified. It is not the intention of the TEF to constrain or prescribe the form that excellence should take.

Methodology and Metrics

Assessments are made against the criteria, based on both core and split metrics supplemented by a written submission; and carried out by assessors and panellists, comprised of experts in teaching and learning, widening participation specialists, students and employer representatives. The “splitting” of metrics by key variables reveals the extent to which positive outcomes are achieved for all students, including those from disadvantaged backgrounds. The recommendations of each group of assessors are received by a central TEF Panel, which decides the outcome for each HEI (HEFCE 2016a). Four ratings are available: Gold, Silver, Bronze, and Provisional.⁷ The TEF only assesses undergraduate education, but it is envisioned it will eventually be extended to postgraduate taught courses.

The first full TEF (known as “Year Two”, as it followed an introductory year) assessed institutional performance in the 2016/17 academic year and drew upon three main metrics: The National Student Survey⁸ (NSS), used to measure teaching quality, assessment and feedback provision, and academic support; the non-continuation data from the Higher Education Statistics Agency; and the number of graduates in employment/further study or highly-skilled employment/further study from the Destination of Leavers from Higher Education Survey (DLHE) (DfE 2016, p. 18).

These metrics are not beyond reproach. The response of The Royal Statistical Society to the TEF technical consultation identified several limitations of the metrics and criteria adopted. For example, they point out that innovative forms of teaching, which challenge pre-conceptions or force students to engage in new ways, often score low student satisfaction ratings, despite these methods often being highly effective in enhancing student learning (RSS 2016, p. 1). This questions the validity of using student satisfaction data, such as the NSS, as a measure of teaching quality. Criticisms of the NSS more broadly include a study by Lenton (2015) which suggests raw scores should not be used as a method of ranking, while Bennett and Kane (2014) find students ascribe disparate meanings to the wording of particular questions which undermines the comparability of the results.⁹

The development of the TEF has followed its own “learning curve”, as the Year Three specification was revised following the lessons learned from Year Two (DfE 2017a). For example, an analysis of metrics resulted in the weighting given to the NSS in the assessment being halved (DfE 2017b). The Year Three specification indicates an increased emphasis on student outcomes (DfE 2017c).

The initial design of the TEF was constrained by the metrics available. The results, therefore, only reflect what is captured by three metrics which represent a limited cross-section of the total teaching mission. The problem with any metric-driven methodology is how it privileges things that can be measured in numbers. This can provide an incomplete or distorted perspective. This problem is exacerbated if the number of metrics used is small.

This limitation will be reduced in future years by the incorporation of additional metrics. Alongside the implementation of the TEF, work was undertaken developing new metrics which assess additional aspects of the teaching mission. Possibly the most innovative, and potentially the most transformative, is Learning Gain, which measures:

the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time. This allows for a comparison of academic abilities and how participation in higher education has contributed to such intellectual development (McGrath et al. 2015).

There is a myriad of ways in which Learning Gain can be applied and measured, presenting methodological challenges. A series of projects will scope out how new metrics can be developed to achieve this, such as looking at how non-subject specific learning gain may be measured at scale (HEFCE 2017a; Kandiko Howson 2016).

A new input metric being piloted is Teaching Intensity (Johnson 2017a). This will be assessed using a method that weights the number of contact hours taught by the student-staff ratio of each taught hour. This will produce a “Gross Teaching Quotient” (GTQ) score to allow comparisons within subject groups (DfE 2017d, p. 63). However, academic research questions the relationship between class size and student satisfaction (Gannaway et al. 2017); and challenges the notion that contact hours are a valid measure of teaching quality, finding that indicators which measure the engagement which occurs within those hours has greater potential for capturing teaching excellence (HEA 2016).

Further supplementary metrics include Longitudinal Educational Outcomes¹⁰ (LEO), which reveals earnings one, three and five years after graduation, by degree subject studied and HEI attended (DfE 2017e). A limitation of LEO is that it is a retrospective metric in a dynamic labour market, where the earnings of past graduates may not be a good predictor of future graduate salaries. There is also a new metric to identify Grade Inflation overtime. The number of first-class degrees awarded in the UK has increased considerably in recent years (Coughlan 2017); which the government regards as a pressing problem that needs to be addressed (Johnson 2017b). However, previous studies of efficiency—which consider the conversion of inputs to outputs—have disputed the extent to which genuine grade inflation is actually present (see Bachan 2017; Johnes and Soo 2017).

Results of the Teaching Excellence Framework

The results of the first full TEF were released in June 2017. They produced an unfamiliar hierarchy of institutions and made a distinctive contribution to the rankings debate hitherto driven by research. The complete focus on the teaching mission removed the advantage enjoyed by the longest established universities, with their income and esteem derived from research.

Participation in the TEF is voluntary, but over 80% of the sector chose to take part. Of the 295 participating providers, 134 were HEIs, 106 were further education colleges offering higher education, and 55 were alternative providers.¹¹ On this new “level teaching playing field”, 60 higher education providers were rated Gold, 115 Silver, 53 Bronze and 65 Provisional (HEFCE 2017b). Several universities, based on their metrics alone, would have achieved a lower rating than their actual award. This is because their performance in the written submission component helped elevate them to a higher rating. This also indicates the judgment of the panel in making their assessment is a consequential part of the process.

A Higher Education Academy¹² report found the content of submissions varied widely. Because this was the first year of the exercise, there were no successful submissions from previous rounds for institutions to benchmark. Some submissions concentrated on qualitatively describing the context of their provision while others directly addressed the TEF metrics. Submissions awarded Gold were more likely to be structured around the TEF criteria (Moore et al. 2017).

Gold was obtained by universities with a wide range of histories. Among the Russell Group of “research intensive universities”, eight out of 21 institutions were awarded the Gold rating, 10 got Silver, three got Bronze. The results didn’t correlate with institutional age or type, research income, or the socio-economic background of the student cohort; although, there was a slight correlation by entry tariff, where institutions with higher entry requirements did slightly better but this wasn’t statistically significant (DfE 2017f).

Analysis

Between Quality Assurance and Market Reconfiguration

As a framework to assess teaching quality, the TEF features many debates familiar to scholars and practitioners of quality assurance—as it seeks to assess, assure and enhance. Two of the government’s four stated objectives for the TEF—raising the esteem of teaching and helping to recognise and reward excellent teaching—are both closely allied to quality assurance activities that already take place within institutions.

The TEF can be seen as an annexe to, but not a replacement of, the UK’s long-standing Quality Assessment¹³ arrangements, which ensure providers meet baseline expectations for academic quality and standards. These procedures were revised alongside the early development of the TEF (HEFCE 2016c); which ensures the mutual co-existence of the two schemes that draw on some of the same data (HEFCE 2016d, pp. 9–10). We can see the boundary between transparency tools and quality assurance in the UK is not as clear-cut as it once was. This interconnected quality and transparency agenda acknowledges recent developments in the quality debate (the greater emphasis on student outcomes and accountability to stakeholders, for example) while also providing the transparency of a ranking system through the TEF with its simple scores.

However, ideas typically found in the quality assurance literature do not fully explain the agendas surrounding the implementation of the TEF. For example, the other two of the government’s stated objectives for the TEF—informing student choice and better meeting the needs of employers—are part of a wider set of government priorities. There is also the issue of the means (the form of evaluation used, the incentives unleashed, and the financial and reputational rewards) by which the TEF seeks to achieve its four stated objectives and deliver any additional government goals.

To fully understand the TEF, we need to appreciate how it is situated relative to established quality assurance agendas, on the one hand, and wider government ambitions to reconfigure the higher education sector, on the other. The latter is more effectively explained by the literature on approaches to governance (for example, Moran 2003; Jordana and Levi-Faur 2004). This is because the TEF, as a new transparency tool, was not developed in a political vacuum to merely improve teaching quality. Rather, it is an integral part of a reform process which seeks to reconfigure the higher education marketplace in two respects: first, to change the dynamics within the market, and second, to change the parameters of the market itself.

The first point refers to changing consumer behaviour through subjecting HEIs to greater scrutiny to obtain better product information, which applicants can use to make an informed choice. The second point refers to removing longstanding barriers to market entry to allow profit-making providers into the higher education sector to compete alongside existing universities. These new alternative providers will offer students additional choices. Both of these moves seek to foster greater competition between institutions.

These points also show how these changes move well beyond earlier incarnations of NPM reform which UK higher education has experienced since the 1980s (Ferlie et al. 2008; Paradeise et al. 2009). This can be seen in the desire to broaden opportunity for the profit-making sector and create an environment favourable for private investment; the willingness to end long-standing public sector monopolies and professional regulatory capture; and a fervent belief in the power of competition to deliver better services at lower costs.

The TEF can be understood as part of the new regulatory governance arrangements, where, according to policymakers, the “market needs to be re-oriented and regulated proportionately—with an explicit primary focus on the needs of students, to give them choices about where they want to study, as well as what and how” (DBIS 2016a, p. 11). The TEF is, therefore, designed to bring transparency to all HEIs in a more liberalised higher education market which includes a greater plurality of providers. The TEF achieves this re-orientation and regulation role through using both “punishments and rewards” to induce behaviour change. Success in the TEF results in direct benefits, including the reputation rewards of receiving a highly publicised badge of excellence, and the financial rewards of being allowed to raise fees. It will also inform consumer choices, conferring further benefits. Flowing from this, the government envisages dynamics whereby:

those institutions and courses that do best within the TEF framework attract more student applications and, through their greater ability to raise fees, reinvest in the quality of their teaching and grow their capacity to teach more students. …Those institutions that receive a lower assessment either seek to raise their teaching standards in order to maintain student numbers and/or raise fees (DBIS 2015, p. 84).

The government also foresees a situation where institutions may need to differentiate themselves as a lower cost or specialist provider or even leave the market entirely.

Influencing Institutions

The TEF has attracted criticism from many educationalists on the grounds it doesn’t accurately measure teaching quality. For example, Gibbs (2017) argues the assessment method, a small panel of experts making quality judgments, is not well founded. Furthermore, Rust (2017) argues the metrics used to measure retention, employment outcomes, and student satisfaction within the TEF do not assess teaching excellence with any validity. Arguments such as these underscore a common criticism levelled at the TEF—teaching quality is not measured directly but by proxies.

These factors are critical in determining the effectiveness of the TEF as an accurate measurement tool. However, as a transparency tool, there is another perspective from which to analyse the TEF. This considers how the framework influences the performance of individual institutions, and the higher education system as a whole, to deliver the intended benefits of the reforms. This is concerned with whether the TEF provides the right signals and incentives in the context within which it operates.

There is early evidence the TEF is making progress on realising two of its stated objectives: raising esteem for teaching, and recognising and rewarding excellent teaching. The presence of a high-profile national evaluation programme has raised the perceived importance of teaching within the academy, and, crucially, among university leaders and strategic planners. For example, at a high-level conference on research-based education in June 2017 (UCL 2017), the Vice-Provost (Education) of Imperial College London said the TEF had forced leaders of research-intensive institutions to start paying close attention to the quality of teaching for the first time, and support these efforts with real money, thus placing “teaching at the centre, where it should be” (Bothwell 2017). Moreover, research by Universities UK¹⁴ (2017) found many leadership teams were paying more attention to core TEF metrics and monitoring their performance to track student success. Changes such as these have cascading consequences on the nature of academic leadership within HEIs.

Considering the third objective—to better inform students’ choices about what and where to study—there are indicative reports the TEF does influence applicant behaviour. For example, research commissioned by a consortium of UK students’ unions which surveyed 9000 students found 50% would have reconsidered or not applied to their university if they had known it would be rated Bronze (Trendence UK 2017). In the international student market, there are reports that Gold rated HEIs are attracting more applicant interest (Kennedy 2017). A survey of 3335 international students interested in studying in the UK revealed applicants would select a university with a Gold TEF award over one placed in the UK top 25 in a ranking; although respondents were more likely to choose a university with a high ranking above one with a Silver or Bronze award. This survey also revealed some misunderstanding of the TEF results: 24.5% of students aware of the TEF believed that a Bronze award means teaching quality is “unsatisfactory”, which is not the case (Raybould 2017).

Regarding this objective, one contention is the Olympic medal inspired awards reduce the whole exercise down to only three ratings. Although easy to understand, these may be too crude to adequately inform choice or have the reputation effects to provide clear signals to incentivise improved performance. Another issue is the underlying assumption that more information is beneficial, which may or may not be true—it depends on how relevant it is and how it is used. The UK Government has commissioned new research to identify the information needs of applicants and the usability of the TEF. This will contribute to the future development of the framework.¹⁵

The fourth objective of the TEF—to make higher education better serve the needs of employers, business, industry and the professions—will require a longer-term assessment of to what extent, and in what ways, universities have pivoted towards serving external stakeholders—and the consequences of this.

Incentives Alignment

The examples in the previous section suggest the incentives emanating from the TEF are effective. However, the examples above are all instances where the incentives are straightforward—others are multifaceted and don’t neatly align. Considering the effects on competition and prices, the incentives structures are complex. For example, the TEF is designed to realise the government’s ambition to finance teaching on a quality, not quantity basis, as is the case for research. However, in practice, funding teaching on a quality basis works differently. When universities perform well in the Research Excellence Framework, they get more public research income. Alternatively, universities that achieved an award in the first TEF were allowed¹⁶ to generate more income by raising their undergraduate fees¹⁷—thus increasing the indebtedness of their future students. The relationship between good performance and financial reward is therefore questionable.

The initial intention was that in future years HEIs with a higher award would be permitted to increase their undergraduate fees by more than those with a lower award.¹⁸ Here, “performance related prices” would produce differentiated fees between institutions. However, following consultation (DBIS Commons Select Committee 2016), attaching such significant financial rewards to the results was shelved until the TEF had been subject to a robust assessment following its implementation (DBIS 2016a, p. 44).

Moreover, linking the TEF results to fees assumed the government would allow fees to rise every year by at least inflation.¹⁹ The financial incentives associated with the TEF, therefore, became enmeshed with the tuition fee debate. Fee increases are politically unpopular in England and were frozen by the Prime Minister in October 2017, pending a review (May 2017). A review of fees and funding may change the environment in which the TEF operates. This means the financial rewards attached to the TEF may need to be reconfigured. Although, the government minister responsible for higher education identified there are a “variety of ways” of linking funding to the TEF (Morgan 2017).

There is also the issue of how the TEF fosters competition, and through this, influences prices and efficiency. In England, there is competition between HEIs for the quantity and quality of students, not on price. This is because domestic undergraduates don’t pay their fees at the point of use; rather, the government pays, and the students are given a loan which they pay back after graduation contingent on their income (Bolton 2016). This results in “low price sensitivity” and no real price competition. Here, undergraduate education remains a prestige, not a price, market; where universities are reluctant to lower prices as this may undermine their prestige and ability to attract the best students. The TEF, therefore, provides product information to inform choice in a quasi-market without variable price signals. Because there is no price competition, HEIs are subject to minimal pressure to increase efficiency or reduce student fees.

Factors such as those outlined here identify the complexity when simultaneously incentivising so many things through one framework. They also question the extent to which the TEF contains a coherent set of incentives which align to improve outcomes and whether all the benefits of the reforms, as purported by NPM doctrines, will materialise.

Universities as “Institutions” and Subjects as “Nested Institutions”

The first full TEF operated at the level of each higher education provider. The submission, assessment and results were for HEIs as a whole, with no internal breakdown within them. If we assume universities are internally homogenous, this is fine. But if we believe that universities are in fact internally diverse, this is a large limitation on the effectiveness of the TEF as a transparency tool.

Returning to the ideas within the New Institutional Economics, we can see the university (or any HEI) as an “institution”. This institution will have its own long-term, stable—yet internally evolving—collection of rules, logics, activities and identities (Meyer et al. 2008; Olsen 2007). These not only shape the activities that take place within it but also its performance. However, we can also see the university as containing a series of “nested institutions”; where there are various subject areas and organisational divisions “nesting” within the larger institutional structure. A study of nested institutions focuses on the inter-relationships among systems at different levels. The core idea of “nested institutions” is that the behaviour of units in any subsystem is inexorably affected by the structure of the larger system within which the subsystem is nested. In the theory, there may be full conformity between the encompassing and the encompassed institution. Alternatively, a nested institution may deviate from the rules and order established by the overarching institution. It may also behave and perform differently from other institutions nesting alongside it in the same overarching institution (Blavoukos and Bourantonis 2017).

The theory of nested institutions informs us that an analysis at either the overarching institution or subsystem level alone may fail to provide an accurate and comprehensive account. In the context of the university, there may be varying degrees of good and bad practice within one institution. If it received one grade, reflecting the aggregate performance, this lack of granularity could make the TEF a “blunt tool”—by not recognising excellence or identifying areas of poor performance. It also doesn’t provide sufficiently detailed information to inform student choice as there is no break down by subject areas or comparable courses, one of the main rationales for the TEFs introduction.

The UK government does believe there are differences both between and within institutions. To address this, the TEF will move to the subject level from 2020. This means the framework will assess sub-institutional level performance. This approach, as explained by the literature on nested institutions, is clearly superior. But to operationalise such an evaluation creates many methodological difficulties. For example, the internal composition of each university—faculties, schools, departments, etc.—does not follow the same subject groupings, making comparisons between institutions along these lines impossible. Moreover, there are around 38,000 undergraduate courses in the UK, meaning assessment at individual course level is not practical. To make the data manageable, courses will have to be placed into a small number of subject groups; however, there will have to be sufficient granularity to inform student choices.

Policymakers in the UK are proceeding through piloting the TEF exercise at the subject level and testing the usability of various subject group classifications (DfE 2017d). The development and implication of the subject level TEF will have a large bearing on the future success of the TEF as a transparency tool. This section illustrates the methodological difficulties in selecting the “unit of analysis”, and developing an effective tool to provide accurate transparency at that level.

Conclusion

To conclude, the TEF can be understood as a multi-purpose instrument which seeks to provide accountability (by providing performance data to inform applicant choice and fee rises) and drive improvement (through raising the importance of teaching and learning within HEIs).

Considering the effectiveness of the TEF as a transparency tool, some of the difficulties outlined may be reduced or resolved as the TEF evolves—through the implementation of new metrics, changes to the weighting of the various components, and the move to sub-institutional level—while others may not. We can also conclude the long-term sustainability of the TEF depends on how it can influence institutional performance and how it works within the new regulatory governance arrangements.

This chapter considers the development of the TEF up to the publication of the evaluation of the TEF Year Two in 2017. It is too early for an analysis of the full effects of the new framework. From its initial inception, the government saw the TEF as a venture that would come to fruition over the longer term. It was the government’s expectation that the impact of the TEF would grow over time, as institutions “respond to its stronger incentives to focus on and improve the quality of teaching they offer, and as students are able to exercise better informed choices” (DBIS 2015, p. 85). Moreover, in accordance with standard practice for all new UK government policy initiatives, a series of evaluations will be undertaken to gauge of the extent to which the TEF has “impacted students’ choices and teaching practices” in reality (DfE 2016, p. 56).

From this study of the TEF, the following underlying issues of relevance to the wider EHEA have been distilled. These include methodological challenges of wider applicability. First, there is a tension when implementing onto autonomous institutions a national framework designed to provide public scrutiny. A balance needs to be struck between two imperatives. On the one side, the state will act as industry regulator in the interests of the consumers, graduate employers, and taxpayers. On the other, is the need to respect academic freedom and the role of HEIs in assuring their own quality.

Second, there is a difficulty in striking the right balance when producing results. It needs to be simple enough for the public—who are not higher education specialists—to sufficiently comprehend and use it to inform their choices. Yet, it also needs to be detailed enough to be meaningful and have credibility and the confidence of those within the academy. This presents a trade-off. Related to this, is the need to produce more granular data at the sub-institutional level.

Third, the development of a new framework to assess the teaching mission of the university requires not only defining what teaching excellence is and working out how to measure it, but also having a political and policy context which enables this agenda to be operationalised. The suitability of the TEF as a benchmark model for other systems in the EHEA will depend on a number of factors, including: the extent to which there is a perceived need to improve product information and/or teaching quality; the presence and scope for competition in the sector; and the maturity of the quality assurance arrangements in a given higher education system.