Theorising a contextual framework for moderation of internal assessment: development and opportunities

Policy change in education is complex. Policies themselves overlay and interact, and organisations enact—and react to—policy change in diverse ways. Thus, what is envisioned by policymakers is not always what occurs in practice. Instead, policies are interpreted and translated in ways that vary from organisation to organisation and, often, from the original policy intent (e.g. see Ball, Maguire, and Braun, 2012, for contextual factors that influence policy enactment in secondary schools). This article proffers a framework designed to provide a mechanism through which to understand variations between policy as intended and as enacted.

Policy that addresses the quasi-autonomous application of complex processes across multiple organisations is particularly susceptible to enactment variation (Ball et al., 2012; Keddie, 2019). The policy domain discussed in the present article—internal, high-stakes educational assessment of learner performance—is a salient example. In its broadest sense, the term internal assessment refers to any assessment judgement of learner performance made by an educator involved in the programme of learning being assessed, while external assessment refers to judgements made under the auspices of a superordinate authority. In some cases of internal assessment, it is not only assessment judgements that are made by the educator; responsibility for the assessment activities and grading systems might also be devolved. In many jurisdictions, internal assessment is used for credentialing, making it a high-stakes process subject to substantial political scrutiny. In these cases, quality assurance processes that maintain the integrity of the assessment process are essential to support the reliability, validity and credibility of qualifications that are based on those assessments. Most often, quality assurance of internal assessment takes the form of ‘moderation’—processes designed to ensure an acceptable level of comparability between the assessments conducted by different assessors and organisations, and therefore between qualifications awarded on the basis of those assessments (Klenowski & Wyatt-Smith, 2014).

In this article, we develop a new contextual framework for moderation comprising two elements. The first is a modification of a heuristic developed by Ball et al. (2012), to represent the intra- and extra-organisational contexts of policy enactment. The second is a modification of Biggs’ (1993) systems-theory representation of tertiary education. We combine these to describe a hierarchy of embedded and interacting systems within which moderation policy is enacted. This new framework contributes theoretically and empirically by providing an analytical device for shedding light on ways in which policies are enacted within and between the levels of a systems hierarchy, and illuminating contextual factors that exist for entities within and external to individual organisations. We illustrate the framework by discussing its key features in relation to a range of studies. Finally, we consider potential uses of the contextual framework.

Internal assessment may be defined as formal judgements of students’ learning made by assessors (e.g. teachers) within an education organisation (Crooks, 2011; Harlen, 2007). Internal assessment has also been called classroom (Brookhart, 2003), coursework (Crisp, 2013), school-based (Klenowski & Wyatt-Smith, 2014) and teacher (Harlen, 2007) assessment. Internationally, internal assessment is used at most education levels (Klenowski & Wyatt-Smith, 2014), including primary and middle school (Adie, 2012; Ward & Thomas, 2016), secondary school (Crisp, 2017) and tertiary—including in vocational education and training (Halliday-Wynes & Misko, 2013), workplace training (Vaughan, Gardiner, & Eyre, 2012) and university (Crimmins et al., 2016).

Globally, internal assessment is widely used for summative purposes to credential students (i.e. to ascertain whether their demonstrated knowledge, skills and attitudes meet specified requirements of a standard or course; Harlen, 2007), including the award of formal qualifications (Crisp, 2013). To be credible, qualifications—and the assessments that contribute to them—must be robust and trustworthy (Broadfoot, 2007). Moderation assists with calibrating judgements across assessors and professional learning, and can be characterised as a quality assurance process. Construed as such, moderation focuses specifically on the quality and integrity of summative internal assessment (Harlen, 2007; Newton & Shaw, 2014).

Moderation is used as a quality-assurance process in many countries including New Zealand (Crooks, 2011), the UK (Crisp, 2017; Grant, 2012), Australia and Canada (Klenowski & Wyatt-Smith, 2014), the Netherlands (Johnson, 2013) and Switzerland (Mottier Lopez & Pasquini, 2017). The quality-assurance regimes to which moderation contributes are situated within the political and cultural environments of the jurisdictions, societies, sectors and organisations in which they occur. As such, the forms and foci of moderation processes are generally tailored to their contexts (Misko, 2015). In jurisdictions including Australia (Australian Skills Quality Authority, n.d.; Tertiary Education Quality & Standards Agency, 2021), England (Crisp, 2017) and New Zealand (New Zealand Qualifications Authority, 2021a, 2021c), it is mandatory for organisations to have in place systems of internal moderation or cross-organisation (i.e. moderation conducted within or between organisations, respectively) if they are to conduct summative assessment for credentialing purposes (Bloxham et al., 2016; Misko, 2015). In some jurisdictions (e.g. New Zealand), education organisations that conduct internal assessment against sector-based or nationally-specified standards for credentialing purposes are additionally required to engage with moderation conducted by external quality assurance bodies (Hipkins, Johnston, & Sheehan, 2016; New Zealand Qualifications Authority, 2021b).

Individual organisations respond to and enact policy and regulatory requirements—including those pertaining to quality assurance regimes and moderation—in contextualised, creative and diverse ways. The enactment of organisation-specific policies (those that are established and enacted within an organisation in situated and idiosyncratic ways) is similarly diverse: such enactment is influenced by the contexts and cultures of organisations and the wider environments within which they are located. Hence, context shapes policy enactment (Ball et al., 2012; Keddie, 2019). Far from being solely the stage-set upon which organisations (or sub-organisation entities) function, ‘… context is an “active” force … [which] initiates and activates policy processes and choices … in relation to policy imperatives and expectations’ (Ball et al., 2012, p. 24). Therefore, to better understand the complexity of policy enactment within and between organisations, ongoing attention to contextual factors is necessary (Keddie, 2019).

Ball and colleagues (2012) developed and tested four inter-linking contextual dimensions (situated contexts, material contexts, professional cultures and external contexts) as a heuristic to assist understanding of policy enactment at the organisation level in secondary schools in England. Situated contexts encompass aspects of organisations associated with geographic location and history. Ball et al. included the student intake of an organisation and the community the organisation serves as components of its situated context. Material contexts refers to the infrastructure of an organisation, including its facilities, buildings, information and communications technologies, budgets, staffing levels and other resources. The professional cultures dimension comprises the professional expertise, experiences, ethos, values, attitudes, philosophies and motivations of teachers and other staff. The external contexts dimension encompasses regulatory and legal requirements, and the expectations and pressures of local communities and wider society.

Systems theory is the multidisciplinary study of broadly applicable principles and theoretical constructs that pertain to entities—systems—comprising organised and interacting components, interactions and relationships between those components, and interactions between systems and their environments (Boulding, 2003; Midgley, 2003; von Bertalanffy, 2003). A component of a system ‘may [itself] be [a system] of a lower order’ (Hall & Fagen, 2003, p. 68) such that lower-order (micro- or sub-) systems can be embedded within higher-order (macro- or superordinate) systems. Interactions between system components describe ways in which changes in one component impact on other components and on the system as a whole. Likewise, interactions of a system with its environment—or a higher-order system of which the initial system is a constituent part—describe ways in which the changes in the system impacts on the environment and vice-versa. Nonetheless a feature of many systems as they thus respond to forces from within and without, is that they self-adjust towards equilibrium by absorbing pressure, or by adapting or evolving through feedback mechanisms towards either an established or new equilibrium. If feedback mechanisms are insufficiently effective for the system to adapt or evolve, that system may breakdown and fail (von Bertalanffy, 2003).

Biggs (1993) applied systems thinking to tertiary education and conceptualised multiple levels of systems operating in relation to teaching and learning: a hierarchy of ‘macro- and constituent micro-systems’ (p. 78). Working from the micro to the macro, a system might comprise a student, a classroom, a department or faculty, an institution or a community. Biggs represented this hierarchy as a set of concentric circles, with constituent micro-systems nested within a macro-system, which may also be a micro-system within a larger macro-system.

The inner-most component in Biggs’ systems hierarchy is the individual student. The next level is the classroom, which comprises its students and teacher, the teaching context and content, teaching and learning activities, students’ perceptions of those activities, the teacher’s perceptions of the students, and students’ learning outcomes. Classroom systems are themselves embedded in department or faculty systems, with components comprising staff, structures, policies, practices and classrooms. The next-most superordinate system is the education institution, which comprises the (preceding) constituent micro-systems, as well as organisation-wide policies and processes, staff, resources and infrastructure. The most ‘macro’ system in Biggs’ hierarchy is the community, which includes the expectations and requirements imposed on tertiary education organisations (Biggs, 1993). Further applying systems thinking, Biggs proposed that the internal components and micro-systems of tertiary education systems interact to hold those systems in equilibrium, and that each system also works to maintain its superordinate systems in steady states. Thus, each micro-system in the macro-system of tertiary education ‘has its degree of autonomy, but also its interdependence with adjacent systems’ (Biggs, 1993, p. 83). He concluded, therefore, that whenever attention is focused on one aspect of the teaching or learning context, for example with an intent to effect change, it is necessary to consider both the micro-system of which that aspect is a constituent element and the adjoining systems in order to obtain a ‘[more] fruitful understanding of teaching and learning’ (Biggs, 1993, p. 83).

As an example of a system adapting to maintain equilibrium in response to external forces, we can apply systems thinking to a period of profound change in the New Zealand education and qualifications system. The introduction of the National Certificates of Educational Achievement (NCEA) school-leaving qualifications in the early 2000s marked a major assessment reform in New Zealand, with a shift from a norm-referenced model of assessment to a standards-based (criterion-referenced) model. NCEA comprises a mix of external (predominantly examination-based) and internal assessment against discrete assessment standards, and the New Zealand Qualifications Authority (NZQA) is the central government agency with responsibility for managing the setting and marking of external assessment and the moderation of internal assessment.

In the initial execution of the reform, any plan that NZQA may have had in place for managing the reliability of large-scale standards-based assessment proved inadequate; for a full account, refer to Hipkins et al. (2016). However, in summary, over the period 2002–2004, there was such a level of year-on-year variability in external examination results for individual standards that public and stakeholder confidence in the NCEA qualifications system was severely shaken. The variability contributed to a public and political outcry, which precipitated the New Zealand State Services Commission reviewing NZQA’s performance in delivering the secondary school qualifications. The Commission’s findings and recommendations led to the introduction of measures to better ensure reliability—in terms of consistency of results over time, and across subjects, standards and organisations—for NCEA external and internal assessments.

Hipkins and colleagues observed that, although the new measures were imperfect, they were ‘much better than nothing; without their inception for the 2005 examination round, the variability problems would have persisted, and it is most unlikely that NCEA would have survived at all’ (Hipkins et al., 2016, p. 92). Through the lens of systems thinking, we can recognise the assessment reform, the public and political disquiet and the State Services Commission report as external forces emanating from the superordinate system of central government and the NZ public. Through this lens we can also recognise the education and qualifications system as responding to those forces; NZQA as the component that initially exercised inadequate control mechanisms to maintain stability through the period of transition; the measures introduced in response to feedback in the form of the State Services Commission report; and adaptation in the utilisation of these measures for the 2005 examination round and beyond. This period in the early years of NCEA is therefore an example of a system correcting and adapting in order to survive. When viewed through the lens of systems thinking, the successive adjustments to NCEA in the ensuing years (also as described by Hipkins et al. (2016) suggest that these types of dynamics have been ongoing.

The hierarchy of nested systems in tertiary education described by Biggs (1993) provides us with a structure within which to conceptualise embedded contexts that we can adapt and apply to moderation systems. His hierarchy offers a way of conceptualising the layers of sub-systems, systems and contexts within which internal assessment—and moderation—occurs, using a series of increasingly expansive lenses.

While the heuristic developed by Ball et al. (2012) takes both intra-organisational factors (the material and professional cultures dimensions) and extra-organisational factors (largely the situated contexts and external contexts dimensions) into account, it always takes an education organisation to be the focal unit of analysis. However, moderation practices may differ within an organisation (Adie et al., 2013), between organisations (Bloxham et al., 2016) and across entire education sectors (Crisp, 2017; O’Connell et al., 2016). Thus, a comprehensive analysis of moderation policy and practice requires focus both on entities that are subordinate to an organisation and on those that are superordinate. In our view, reconceptualisation of the dimensions of Ball and colleagues’ device is necessary in order to support such a comprehensive analysis.

We propose a two-dimensional conceptualisation of context that distinguishes intrinsic and extrinsic contexts on one dimension and human and material contextual factors on the other.

Intrinsic contexts are those that are internal or subordinate to the unit of analysis, whereas extrinsic contexts are those that are external or superordinate. Note that, whether a context is intrinsic or extrinsic is relative to the unit of analysis. For example, if an analysis is at a level that focuses on government education agencies with responsibility to oversee the embedding of policy across the compulsory education sector, individual schools are each intrinsic contexts, because they are subordinate to the agencies. On the other hand, if an analysis is at the level of an individual school, (other) schools are extrinsic contexts because they are external to the school that is the focus of the analysis—as are government agencies because the agencies are superordinate to that school.

Human contextual factors comprise the people within a system, and their knowledge, constructs and experiences. Human contextual factors largely combine the professional cultures- and external-context categories in Ball and colleagues’ heuristic—that is, the professional expertise, experience, ethos, values, attitudes, philosophies and motivations of people within a system, as well as expectations and regulatory and legal requirements. We also include students, communities and historical contexts in this category, which Ball and colleagues (2012) included in their situated dimension. Our material contextual factors comprise locale (the remaining constituent of the situated dimension of Ball et al.), as well as their material contexts.

Our proffered framework for conceptualising moderation as a hierarchy of nested and contextualised systems is depicted in Fig. 1. The diagram comprises a set of concentric circles representing the layers of systems. Students are represented at the centre, in recognition of their centrality in education. Each system (layer) that surrounds this core contains both human and material contextual factors, as per our conceptualisation of context (above).

Whether each layer of the system represents an intrinsic or an extrinsic context depends on the layer at which analysis is focussed. The layer that surrounds students represents the learning context they experience (i.e. the teaching–learning–assessment cycle), and the courses or programmes of learning involved. We have called this the ‘students’ education’ system-layer, similar to Biggs’ (1993) immediate classroom micro-system. Human contextual factors in this system-layer include teachers and assessors involved with that learning context, as well as pedagogy and learning behaviour. Material contextual factors comprise the physical and electronic resources, as well as the facilities and infrastructure, of that learning environment.

In our framework, the next-most superordinate system is the individual ‘organisation’. The human contextual factors involved in this system include academic and other staff or personnel (e.g. teachers), academic processes and practices, timetabling, internal quality management systems, human resources and performance management processes. Material contextual factors involved include teaching and learning resources, buildings and facilities, budgets, information communication and technology (ICT) software and equipment, and other infrastructure.

Encompassing the organisation system in our framework is a layer representing the ‘education- and qualifications systems’ within which individual organisations operate. This layer comprises the human contextual factors of curricula, training schemes, assessment standards and qualifications, regulatory and quality-assurance bodies, standard-setting bodies and other education organisations. Material contextual factors include the budgets, and ICT and physical infrastructure of those bodies and organisations, and other infrastructure such as computer equipment and software supporting online databases. The macro-system in our framework is ‘society’, which comprises government, community, families, prospective students, graduates, industry and employers. The broad policy directions determined by government originate from this contextual layer and determine or influence what occurs in each of the subordinate layers of the framework. Thus, the macro-system within which moderation occurs has many layers of embedded sub-systems. Furthermore, each layer of this hierarchy has its own contextual influences. Internal assessment—the primary focus of moderation—features in the second, students’ education layer. The systems that encompass this layer are increasingly peripheral to assessment itself.

There is flexibility in how the organisation and education- and qualifications systems layers of this framework can be conceptualised; each can be tailored to better reflect the nuances and characteristics of the individual organisation or jurisdiction of interest. The organisation layer is frequently stratified, containing subordinate layers—such as in an organisation where departments or faculties warrant being considered as distinct embedded systems within an overall organisational context. The configuration of the education- and qualifications layer is influenced by variables including the jurisdiction, its legislative structure, and the reach of regulatory and quality-assurance bodies relevant to internal assessment. Depending on these variables, the education- and qualifications systems layer may be conceptualised as singular or stratified (comprising distinct sub-systems where one leads to another, e.g. pre-compulsory education, compulsory education), and as uniform (i.e. applying throughout the jurisdiction) or as comprising overlaying parallel sub-systems (that do not necessarily articulate, e.g. that are state or district-specific).

To illustrate two configurations of education- and qualifications systems and how the framework can be made relevant to different international settings, we now apply it to the neighbouring countries of New Zealand and Australia with their contrasting parliamentary and legal systems. New Zealand is a unitary parliamentary democracy with a central government (Eagles et al., 2006) responsible for national policy, regulation and governance pertaining to the moderation of internal assessment in formal education. The Ministry of Education and NZQA have national regulatory, quality assurance and standard-setting roles. Quality-assured qualifications, including NCEA, are registered on the comprehensive New Zealand Qualifications Framework (Ministry of Education, 2015). From an internal assessment and moderation perspective, the New Zealand secondary education sector can thus be conceptualised as one uniform (nation-wide) sub-system forming an internal context of the broader stratified education- and qualifications systems.

In contrast, Australia is a commonwealth of federated states and self-governing territories with a federal parliamentary constitution (Eagles et al., 2006). The education- and qualifications systems of the framework are stratified and comprise both uniform aspects and overlaying parallel internal contexts (sub-systems). For example, uniform aspects include that national education and training policies for all levels of education are set at the federal level by the Department of Education, and there is a unitary national qualifications framework, the Australian Qualifications Framework (Department of Education, n.d.). The school-sector sub-system itself comprises further overlaying parallel state- or territory-level sub-systems. The delivery of the Australian Curriculum, and the development, assessment and quality assurance of state or territory senior secondary qualifications, are responsibilities of state or territory education authorities, for example, the Queensland Curriculum and Assessment Authority (Australian Curriculum, Assessment and Reporting Authority, 2016).

We now apply the contextual framework for moderation to a selection of contemporary studies addressing moderation of internal assessment to identify some of the contextual factors and system layers evident in each. The selection has been chosen from empirically-based studies of moderation that involved ‘social’ or ‘expert’ approaches, or a hybrid of the two (Klenowski & Wyatt-Smith, 2014), as identified through the lead author’s doctoral literature review (Willliams, 2019). The studies have been purposefully chosen to provide variation (Creswell, 2012) across each characteristic of: focal unit of analysis level (individual course to whole-of-jurisdiction); education sector (primary school to university); moderation approaches (social, expert, hybrid); internal, inter-organisation, or external quality assurance body moderation; and country. Where more than one study was identified as an example of particular characteristics (e.g. state-wide social moderation at the middle school level), the study that illustrated a specific contextual factor was chosen (e.g. information and communications technology infrastructure; per Adie, 2012). This discussion starts with studies focusing on elements near the centre of the framework, at the students’ education system-layer, where internal assessment actually occurs, and moves outwards to consider studies addressing more peripheral systems, such as those that set or enact policy or engage in cross-organisation moderation.

As might be expected in studies that deal with moderation of internal assessment, the assessment aspect of the teaching–learning–assessment cycle—itself comprised largely of human contextual factors and intrinsic to the students’ education system-layer of the framework—is evident in all studies about moderation that we considered. For example, internal assessment for a first-year (tertiary) communications course was the focus of the moderation initiative on which Crimmins et al. (2016) reported. Another example comes from an evaluation by O’Connell et al. (2016) of a multi-institutional intervention in the Australian university sector that involved nationally-defined accounting standards. The internal assessments at the centre of the intervention were components of the teaching–learning–and–assessment–cycle within the accounting courses at the various participating organisations.

Other aspects of the cycle from the students’ education layer of our framework are represented in Crimmins and colleagues' study. Crimmins et al. (2016) found that the scheduling of some moderation activities (e.g. moderation meetings to calibrate assessment judgement occurring early in a course) enabled staff to incorporate into their teaching what they had learned through those moderation activities, and thus, to better prepare students for assessment. Further aspects of the cycle are also evident in the King’s Oxfordshire summative assessment project on which Black et al. (2011) reported. This was a 3-year, multi-stage intervention involving teachers from three schools in the lower-secondary sector in England that explicitly aimed, simultaneously, to enhance learning and summative assessment, and to improve comparability between schools. Over the life of the project, the assessment activities involved underwent multiple cycles of use and refinement to better achieve both aims. This process clearly situated assessment as being integrally linked to teaching and learning.

In noting that what suits one classroom or course may not suit another, Black and colleagues (2011) also alluded to the nuances and situated nature of each layer of the framework. During the King’s Oxfordshire project, these researchers observed that a portfolio assessment approach exposed a tension between flexibility and comparability. In response to this tension, measures such as a clearer definition of the required portfolio composition, and use of intra- and inter-school moderation processes, were instigated. These measures were intended to enhance the comparability of assessor judgements while maintaining sufficient flexibility to enable the selection of appropriate evidence from the different learning contexts involved: different courses and programmes of learning; different classrooms with different teachers; and students in different schools.

The organisation layer of the framework is evident in many of the studies that we considered. Teachers, their professional learning, their development of shared understandings, their perceptions about moderation and the professional learning resulting from participation in moderation are all intrinsic, human contexts for organisations and featured strongly in the studies of Adie et al. (2013) and Grant (2012). In exploring how the staff in one Australian university faculty viewed moderation, Adie and colleagues set the focus of analysis at the level of organisational systems and, in particular, at the faculty sub-system level. The specific focus was on the human contextual factors within that sub-system—the academic teaching staff.

In another example, Grant (2012) reported on a 4-year initiative that involved staff from all primary and secondary schools under the jurisdiction of one Scottish educational authority. Like the study of Adie et al. and various other studies that we have already introduced (Adie, 2012; Black et al., 2011; Crimmins et al., 2016), Grant’s study focussed on intrinsic, human factors associated with teachers’ professional learning, including the development of shared understandings. The initiative aimed to provide professional development for teachers and to develop sustainable assessment approaches in schools and a sustainable inter-sectoral community of practice. Both Grant and Adie (2012) also made mention of other human contexts intrinsic to the organisation layer, reporting that individual teachers applied their learnings to their own practice in an idiosyncratic fashion, and that teachers took their professional learning from moderation activities and then disseminated the knowledge in their own communities of practice.

Academic systems and processes, which are also human contexts intrinsic to organisational systems, have also featured in several studies that we have referred to here. Internal moderation processes designed to enhance the comparability of assessor judgements within organisations featured in the interventions of Black et al. and Crimmins et al. Organisations’ assessment and moderation practices—as reported by assessors within those organisations—were a focus for Halliday-Wynes and Misko (2013) in their Australian vocational education and training sector study. These included the widespread practice of organisations developing or modifying assessment instruments for their individual contexts, despite a lack of assessment competence and expertise amongst the assessors involved.

A material context intrinsic to the organisation layer is that of the facilities and infrastructure that support any organisational system. Information and communications technology infrastructure (both extrinsic and intrinsic to the organisation) played an important role in Adie’s (2012) study. The social moderation meetings forming the basis of the intervention that she investigated were conducted in an online environment, using an online technology platform (which itself was an extrinsic material context). For teachers to participate, they needed access to an internet-connected computer and a telephone (intrinsic material factors). In some project meetings, multiple teachers from one school ‘were clustered around one computer’ (Adie, 2012, p. 46).

The education- and qualifications systems system-layer of the framework is also evident in a number of the studies that we examined. Various contexts for that system layer are apparent, including intrinsic human contexts such as education organisations or sector- and jurisdiction-wide standards, qualifications, and curricula. Extrinsic human contexts include regulatory and quality-assurance bodies.

The studies of Adie (2012), Black et al. (2011) and O’Connell et al. (2016) all involved participants from multiple organisations in social moderation approaches, facilitating the development of cross-organisation comparability of assessment and shared understandings between teachers of standards and assessment requirements. The initiative that Grant (2012) reported on was both cross-organisational and cross-sector, participants from all primary and secondary schools in the jurisdiction of an education authority were involved. Furthermore, the local education authority itself, another intrinsic human context for the education- and qualifications systems system-layer, was integrally involved in the initiative; the authority commissioned Grant’s research and had staff involved as participants. Yet another human contextual sub-system that is intrinsic to this layer and evident in Grant's (2012) study comprised the national guidelines used as standardised criteria for moderation activities.

Another example in which education- and qualifications systems are visible is the Australian Research Council Linkage project, reported on by Adie (2012). This project was instituted in response to a state-wide policy initiative involving the introduction of a standards-based curriculum and assessment approach, both intrinsic, human contexts of the system-layer. Sector- or jurisdiction-wide comparability of assessment and student achievement against national standards, also intrinsic, human contexts for this layer, were aims of the moderation systems reported on by O’Connell et al., (2016; sector-wide), Crisp (2017; jurisdiction-wide), and audited by the Controller and Auditor-General (2012; also jurisdiction-wide). The moderation systems utilised by NZQA (Controller and Auditor-General), and those at the centre of Crisp’s study (external moderation of the General Certificate of Secondary Education—GCSE—coursework in England) differed from those used in the other studies discussed here: both used expert moderation approaches involving external moderators from the relevant quality assurance bodies. In contrast, other studies used social moderation approaches involving teachers, as well—in the case of Crimmins and colleagues—as expert moderation with more senior academic staff in moderator roles. Thus, the moderation systems of NZQA and those that Crisp reported on reside only in the jurisdiction-wide education- and qualifications systems layer of the framework (although their impacts reverberate through the subordinate system-layers). In contrast, the moderation involved in the studies of Black et al. (2011) and Crimmins et al. (2016) were conducted within education organisations, which are intrinsic contexts to this superordinate layer. The studies of Adie (2012), Grant (2012) and O’Connell et al. (2016) reported on moderation process conducted between organisations (which Black and his colleagues also did)—each of which is an intrinsic context to this layer.

The macro-system of our framework, society, is evident, or intimated, in a number of studies that we examined, two of which we will discuss here. In outlining the purpose of her audit into the performance of NZQA, the Controller and Auditor-General (2012) made specific reference to the importance of NZQA’s work to members of society. The stated aims of her audit were to provide assurance that NZQA was able to effectively assure the quality of internal assessment and, thereby, the credibility and consistency of NCEA qualifications. The stakeholders in this case included the New Zealand Parliament, the parents, caregivers and future employers of students, the students themselves and tertiary education providers. By stating that an aim of her audit was to ‘provide assurance … about whether NZQA has procedures to ensure that there is no such manipulation’ (Controller & Auditor-General, 2012, p. 11), the Controller and Auditor-General addressed a common criticism of internal assessment as being vulnerable to manipulation, and highlighted the function of her audit in maintaining public confidence in the ability of NZQA to quality-assure the internal assessment for NCEA. Thereby, the audit also functioned to maintain public confidence in moderation, which is itself—in part at least—a quality assurance process for internal assessment and qualifications. In terms of the framework for moderation, these functions inhabit the society system layer. Likewise, the Controller and Auditor-General and the stakeholders who were intended to be served by the audit are contextually intrinsic to this layer. Drawing from systems theory, the Controller and Auditor General’s audit is an example of a feedback mechanism (von Bertalanffy, 2003) exerting adaptive pressure on NZQA as a result of the loss of public confidence in NCEA and in the wake of the 2005 State Services Commission report. In response to this pressure, NZQA increased the accountability focus of its moderation system for internal assessment (Hipkins et al., 2016).

In her study focussing on the consistency of internally assessed coursework for GCSE in England, Crisp (2017) also alluded to the societal context. She reported that, at the time of her study, there was continuing debate over the robustness and trustworthiness of internal assessment in England, which are similar to the criticisms raised by the Controller and Auditor-General in New Zealand. Crisp observed that, subsequent to her data collection, coursework was replaced with ‘controlled assessment’, a more restricted version of internal assessment. Implicit in questions about the credibility of internal assessment for senior secondary qualifications is public confidence in the credibility and legitimacy of internal assessment for high-stakes purposes, and therefore, in the credibility of the qualifications themselves. Public confidence is a human context of society—the macro-system of our framework. One function of moderation and other quality assurance processes in education sectors is to maintain public trust and confidence in qualifications and to answer—if not prevent—questions about the integrity of those qualifications (Misko, 2015). Thus, the society system layer is always present in, albeit not always the focus of, studies and analyses of moderation of internal assessment—at least when that assessment is used for high-stakes summative purposes.

Of the studies that we examined, five evince, or at least connote, pressure from a superordinate system that stimulates a response in adjacent intrinsic contexts (i.e. subordinate systems). The initiative reported by Crimmins et al. (2016) was prompted, at least in part, by changes in federally-mandated policy in the Australian university sector, which required greater emphasis on the quality assurance of assessment and learning outcomes. O’Connell et al. (2016) reported on a project that was also established in response to those changes in federally-mandated policy. While we have already argued that the project reported on by O’Connell and colleagues focussed on the education- and qualifications systems layer, decisions and actions were required within the intrinsic contexts of subordinate organisations to allow, enable and facilitate participants’ involvement in the project. Those in the control group needed to be able to undertake the pre-test and post-test tasks and required time and resources to do so. Those in the treatment group needed not only to undertake the pre-test and post-test tasks, but also to engage in the intervention: what appeared to be full-day in-person workshops, which, given the geographic spread of participants, would have likely involved travel and accommodation. Thus, components of the organisation layer are also evident in O’Connell and colleagues’ (2016) study, albeit less explicitly than the adjacent and superordinate education- and qualifications systems layer. Similarly, while we have used the initiative reported on by Grant (2012) to illustrate the contexts of the education- and qualifications systems layer, participants’ involvement in the study emanates from the organisation layer. The resources (and presumably logistics, including timetabling and teacher-release) that would have been required for teachers to participate over the lifespan of the project demonstrates substantial commitment on the part of each primary and secondary school, and each teacher, involved.

Other examples of human contexts within a superordinate system—which exert pressure on intrinsic contexts—are the roles that stakeholders and the public play; stakeholder and public confidence are strong drivers for moderation regimes. The Controller and Auditor-General (2012) emphasised the importance of the public having confidence in an education and qualifications system and in the education and qualifications gained by students. Moreover, stakeholder feedback can prompt debates and policy considerations, for instance, a lack of confidence expressed by stakeholders in the training and assessment delivered by parts of the Australian vocational education and training sector, which led to those parts being identified by the federal-level regulatory and quality assurance body for the vocational education and training sector, the Australian Skills Quality Authority (Department of Education, n.d.), as being at ‘high risk’ (Halliday-Wynes & Misko, 2013, p. 13).

Our new framework contributes to the theoretical literature. Through applying the framework to a selection of studies we have shown that it is applicable to different organisations, sectors and jurisdictions, and to studies with a range of foci. Across these studies we have elucidated all systems layers of the framework, and the multitude of human and material contexts involved. The involvement of multiple layers of focus in many of the studies that we examined, including those of Adie (2012), Black et al. (2011), Crimmins et al. (2016), Grant (2012) and O’Connell and colleagues (2016), highlights the capacity of our framework to shift between systems and sub-systems as frames of reference, with an associated shift in which (other) systems constitute intrinsic and extrinsic contexts for each frame. Further, the framework enables contextual relationships within and between system layers to be made visible, and to remain so irrespective of shifts in frame-of-reference, revealing the tensions, pressures, forces, actions and reactions present in, or engendered by, those relationships. In this way the framework makes explicit the structure of hierarchies, authority and influence inherent in all social systems.

We now turn our attention to whom, and for what, this contextual framework for moderation could be of use. We suggest that the framework could be fruitfully used by education organisations, central agencies (e.g. regulatory or quality assurance bodies) and researchers as an empirical analytic tool for designing and evaluating policy initiatives. Taking advantage of its capacity to shift between frames of reference, likely pitfalls and unintended consequences of their enactment could be identified. The framework illuminates the human and material contextual factors within each system layer, allowing the roles that these factors play in policy initiatives and enactments to be made explicit. Furthermore, in drawing from systems theory, the framework could assist in highlighting that an attempt to effect change in one system layer is more likely to be effective if attention is given to interventions in the intrinsic and extrinsic contexts of that layer that might support the intended change. Those involved in policy work within education organisations and central agencies, and researchers, would find such insights to be of great value to their work.

In this article, we have proposed a new contextual framework for moderation, the development of which was driven by literature and theory. We have also trialled the framework in post hoc analyses of other studies. Notwithstanding these previous applications, refinements to the framework may be required for other jurisdictions or sectors, just as it is anticipated that the organisational system-layer of the framework would be tailored for individual organisations.

The framework draws together various participants, policies and artefacts that are directly and indirectly involved in moderation. It represents the complex layers of contextual systems within which any given frame of reference for a moderation process is situated. Thus, the framework could inform thinking about moderation, and the design and evaluation of moderation policies, systems and practices. It could be used as an empirical tool to help make sense of discussions about moderation, including those about research findings reflecting on the effectiveness or otherwise of particular moderation systems and processes.

Furthermore, drawing on systems theory, the framework offers a structure that illuminates ways in which contextual systems interact with a system selected as a frame of reference, and with each other. Thus, it brings into focus the multiple embedded systems within—and on—which moderation functions. If we consider a process of policy enactment from only one frame of reference without sufficient attention to the contextual roles of other systems that interact with that frame of reference, we heighten the risk of widespread variation in that enactment, which can result in egregious unintended consequences. Our new framework is intended as a mechanism to help to prevent and mitigate such consequences.