The future of online testing and assessment: question quality in MOOCs

Although some of the hope, hype and hysteria of the MOOC phenomenon may have abated, the number of courses running and of learners enrolling continues to climb (Class Central, 2018). The “gadarene rush” to MOOCs described by Daniel (2012), now looks less of a gold-rush or educational technology bubble. Despite their persistence, however, the final place that the MOOCs will claim in the future higher educational landscape is still uncertain. It may be that MOOCs find their best purpose as an “innovation platform”, as a place for institutions to learn and experiment with pedagogy in an environment very different from their traditional institutional educational provision (Brown et al., 2015). Hence faculty who hold traditional pedagogical or teaching beliefs may be compelled to transform their practices through teaching MOOCs (Freitas & Paredes, 2018). A recent study by Zhu, Sari and Lee (2018) of the methodological approaches to MOOC research, for instance, highlights the breadth of the research into teaching and learning at scale and the ambition to innovate traditional practice.

To achieve the greatest impact, MOOC pedagogical innovations must be brought home and mainstreamed. The affordances of MOOCs will need to be harnessed and married to the more traditional forms and mechanisms of higher education. To this end, there have been efforts to posit MOOCs as a bridge to more formal learning qualifications. This can potentially take many forms, one of which is designing systems through which MOOC learning can itself be accredited (Bralić & Divjak, 2018). Most of the major MOOC platforms such as edX, Coursera and FutureLearn have announced initiatives, promising to allow learners to attain accredited degrees through MOOCs (Baker et al., 2018). The Georgia Tech Masters in Computer Science through edX is one of the most prominent, and perhaps also successful, of these efforts to use MOOCs as a platform to deliver degrees. A study by Goodman et al. (2016) claimed that the MOOC had effectively opened new opportunities to gain formal degrees to learners who would otherwise not have gone to college, which is one of the few studies that give hard evidence that a MOOC has widened access to education in a significant way.

As the focus in the future turns from informal and free, to formal, accredited and paid - greater scrutiny will of course be brought to bear on the quality of the courses themselves. There have been empirical studies into the Quality of MOOCs overall (Lowenthal & Hodges, 2015; Margaryan et al., 2015), including the various assessment methodologies within (Sandeen, 2013; Balfour, 2013; Conole, 2016). Creating, and subsequently grading, valid assessments for courses with student cohorts numbering well into the thousands, is challenging. Individual grading by teachers or faculty becomes unfeasible, and so other approaches, such as automated marking and peer-assessment are employed, the latter relying heavily on learners’ active participation (Admiraal et al., 2014; Balfour, 2013; Meek et al., 2017). However, a notable gap exists in that this prior research has not examined Multiple Choice Questions (MCQs), a key component of many MOOCs which employ automated assessments, in any way.

MCQs are omnipresent in education, and are a time-efficient method of assessment, enabling representative sampling of broad areas of course content(Van Der Vleuten & Schuwirth, 2005). They are utilised in most domains of study, but are particularly prevalent in disciplines such as medicine and the sciences, where an extensive body of literature now exists regarding their use (Schuwirth & van Der Vleuten, 2004). Moreover, mathematical models for assessing the reliability of this examination format, and item metrics such as difficulty and discrimination, are well-established (De Champlain, 2010). In addition to their use in summative examinations, tests (or quizzes) incorporating MCQs are also frequently found in formative assessments. They may be used in conjunction with Classroom Response Systems, or within innovative peer assessment systems such as PeerWise, which facilitates the development of student-generated learning tools, test-enhanced learning and peer assessment via MCQs (Denny et al., 2008; Larsen et al., 2008).

As with any assessment tool, there are potential criticisms and disadvantages regarding the MCQ format, particularly if used in isolation (Downing, 2002a; Schuwirth & van Der Vleuten, 2004). Simply-phrased questions may only test simple factual recall, and be answered by simple repetition or recitation of memorised facts, perhaps with minimal actual understanding of content (Schuwirth & van Der Vleuten, 2004). This testing of factual recall may be an appropriate and valid component of assessment at some levels, but the ability to test application of knowledge is essential at higher cognitive levels. The incorporation of layered and detailed scenarios (contextual vignettes) within the stimulus format, followed by a ‘lead-in’ question designed to test the understanding or application of information thereof, allows testing of these higher cognitive levels and problem-solving abilities (Case & Swanson, 2002; Schuwirth & van Der Vleuten, 2004). In addition, it is essential that care is taken when writing multiple choice questions in order to minimise the possibility of cueing, the effects of which may be either positive or negative. Most of these flaws can be avoided with appropriate staff training and quality control procedures (Jozefowicz et al., 2002; Schuwirth et al., 1996; Tarrant et al., 2006a, 2006b).

While various formats have been employed since the introduction of MCQs over the past century, those most commonly seen in current use fall into two main categories, true/false or one-best-answer, which pose very different tasks for the examinee (Haladyna et al., 2002; Case & Swanson, 2002). Within the true/false format, the candidate is required to decide whether each option provided is true or false. This frequently means that they must also make a value judgment as to what extent or degree an option is correct; this may be straightforward, or may involve also trying to anticipate what the examiner had in mind when phrasing the question (Case & Swanson, 2002). In order to reduce potential ambiguity, item writers may default to assessing the recall of precise details and minutiae, instead of testing the understanding of broader, but perhaps more nuanced, concepts. In comparison, for one-best-answer MCQs, the candidate is simply required to rank the options in order, and then choose the best or most appropriate option from the list. Current guidelines from both North American medical licensing institutions with regard to writing MCQs advise the use of one-best-answer formats, for example either single best answer (SBA) or extended matching questions (EMQ) (Case & Swanson, 2002; Wood & Cole, 2001). The number of response options within each format is not entirely fixed, and institutions may vary this slightly according to their assessment strategy (Rodriguez, 2005; Swanson et al., 2008; Tarrant & Ware, 2010). That being said, most institutions will provide between 3 and 5 response options when writing SBAs, and 10 to 15 options within the EMQ format.

At a most basic level, MCQs may be required to undergo peer-review prior to use, or may be more formally assessed using a simple rubric (Jozefowicz et al., 2002); under certain circumstances, students themselves may be involved in this process and often prove to be discerning evaluators (Denny, Luxton-Reilly & Simon, 2009; Purchase et al., 2010). More comprehensive manuals and publications are also available to educators, advising on the design of appropriate assessment strategies, and offering recommendations and guidelines regarding the writing of high-quality MCQs. The canonical source is generally held to be that from the US National Board of Medical Examiners, but other published guidelines also offer excellent advice, in varying contexts (Case & Swanson, 2002; Tarrant et al., 2006a, 2006b; Wood & Cole, 2001; Haladyna et al., 2002). Despite the availability of these resources, the prevalence of flawed items in high-stakes examinations is an ongoing problem (Jozefowicz et al., 2002; Rodríguez-Díez et al., 2016; Tarrant et al., 2006a, 2006b). This is of concern due to indications that that flawed MCQs may be confusing to examination candidates, penalizing some examinees (particularly non-native speakers), and thus potentially reducing the validity of the examination process (Downing, 2005; Tarrant et al., 2009). At their most egregious, item flaws may lead to MCQ assessments being guessable or gameable by ‘test-wise’ candidates, who otherwise have little content knowledge (Poundstone, 2014; Case & Swanson, 2002; Haladyna et al., 2002). Given the extent to which MCQs are relied upon to provide an accurate assessment of candidates’ competence in both undergraduate and postgraduate medical education, there is a high demand for high-quality MCQs. Nonetheless, research over the past decade indicates a high prevalence of flawed items, in numerous contexts; this prevalence typically ranges from 33% to 72%, but with one publication reporting that all items reviewed contained at least one flaw (DiSantis et al., 2015; Rodríguez-Díez et al., 2016; Stagnaro-Green & Downing, 2006; Tarrant et al., 2006a, 2006b; Downing, 2002b; Pais et al., 2016).

The question then arises as to the impact that flawed or ambiguous items may have on candidates. As previously stated, most authors group the potential effects into two main categories; those that introduce irrelevant difficulty into the question, and those that enable test-wise candidates to perform well. Within Table 1, we have categorized these flaws according to where they appear within the item, and outlined their potential effects as described within the evidence-base. These effects are not minor; one study examining the impact of item writing flaws demonstrated that 33–46% of MCQs were flawed in a series of basic science examinations; the authors concluded that ‘perhaps as many as 10 – 15% of the examinees were incorrectly graded as failing’, when they should in fact have passed, due to the presence of these flawed items (Downing, 2002b; Downing, 2005). However, interaction between flawed items and student achievement can be complex, and the effect of flawed items may not be consistent. Another publication examined MCQs used within undergraduate nursing examinations, and determined that 47.3% of all items were flawed (Tarrant & Ware, 2008). In contrast to Downing, they demonstrated that borderline students benefited from these flawed items, which allowed a number of borderline students to pass examinations that they would otherwise have failed, had the flawed items been removed (Tarrant & Ware, 2008). They also concluded that flawed items negatively impacted the high-achieving students in examinations, lowering their scores.

MCQs are currently a key element of MOOC assessments. The ease with which they can be deployed at scale, with automated marking, makes them particular well-suited for this medium. The results that participants receive from these assessments frequently contribute to their summative grade, and hence ultimately towards a certificate or credentials that they receive upon completion or participation, whether this be formal or informal in nature. Given this integral role, the question arises as to their quality. This issue is critically important if MOOCs are to fulfil aspirations to deliver formal learning that can contribute towards recognized awards. Therefore, in this study we sought to determine the prevalence of flawed items in a sample of MOOC MCQ assessments. Our study makes an important contribution by being the first empirical study to our knowledge that has critically examined MCQs in MOOCs.

In the remainder of this paper we will outline the methods of our study which include our how we collected the data that comprised our sample. We then describe how we drew on relevant literature to develop our evaluation framework. The analysis of these results by the evaluators is then presented and followed by a discussion of the findings and their significance.

In the following section we detail how we set about the collection of the data that comprise this study. We detail how we drew on the relevant literature to develop framework for an analysis of that data and how this analysis was carried out both computationally and by the two human evaluators.

In total, 204 questions were reviewed and a total of 202 item writing flaws were detected (Table 2). In 112 (54.9%) of the questions analysed there was at least one flawed item present, while 57 questions (27.9%) contained two or more flawed items. These were not evenly distributed among the MOOCs; when grouped by source, three MOOCs were found to have only a single flawed item among their sampled MCQs, but all the other courses had more than one flawed item in their sampled questions and every revised MOOC exhibited flawed MCQ items.

A large number of these flaws could be attributed to the use of older or outmoded formats; over 10% of questions (22 of 204) were written using the true/false format, and 17 were written in an overly complex manner (Tables 1 & 2). Similarly, a high number of items were either ambiguous or unfocussed in their phrasing of information provided (n = 19) or the question asked (n = 11), potentially leaving candidates unsure as to how to answer the question. Other flaws frequently observed were the inclusion of absolute terms (n = 30), negatively worded stems (n = 21), implausible distractors (n = 15) and the provision of more than one, or no correct answer (n = 18). Some flaws were infrequent, such as the inclusion of gratuitous or unnecessary information in the stem (n = 0), or the inclusion of the options ‘all of the above’ (n = 2) or ‘none of the above’ (n = 2).

In addition to the above flaws, the length of the options in each MCQ was calculated, by counting the number of characters contained within it, and we then examined how frequently the longest option was also the correct option, by means of Chi—squared analysis. The longest option was found to be correct significantly more often than would be expected by chance (χ² [1, N = 204] = 10.75, p < 0.01). With regard to calculating the position of the correct option, this was somewhat complicated by the fact that the number of options that may be provided with an MCQ is not fixed (Rodriguez, 2005; Swanson et al., 2008; Tarrant & Ware, 2010). Therefore, in order to examine the position or distribution of correct options, we limited our analyses to those MCQs which had four options, which gave us a total of 113 MCQs to consider (Fig. 1). While a relatively even distribution was observed between the second, third and fourth positions, the first option was infrequently correct; however, these differences did not reach statistical significance (χ² [1, N = 113] = 7.46, p = 0.06).

This descriptive study examined the prevalence or frequency of item writing flaws within MOOC MCQ assessments. While a high number of flawed items were identified, these results are comparable with previous research in other domains, although this is not perhaps a particularly reassuring fact (DiSantis et al., 2015; Haladyna et al., 2002; Pais et al., 2016; Rodríguez-Díez et al., 2016; Stagnaro-Green & Downing, 2006; Tarrant et al., 2006a, 2006b;Tarrant & Ware, 2008; Downing, 2005). Additional studies examining the effect of flawed items on aspects of assessment such as item psychometrics and candidate performance, suggest that these flawed items reduce the validity of the examination process, penalizing some examinees (Downing, 2005; Tarrant & Ware, 2008).

A large number of these flaws can be attributed to the use of older or outmoded formats; while some authors still hold a preference for the true/false format, these can be problematic, particularly if attempting to assess application of knowledge in context, as opposed to simple, explicit, factual recall (Case & Swanson, 2002). Likewise, overly complex item formats increase irrelevant item difficulty, and may disproportionally affect learners who are non-native speakers of English (Tarrant & Ware, 2008). If MOOCs are to realise ambitions of opening up access to education on a global scale assessment formats which disadvantage non-native speakers of the primary language of the MOOC present a problem.

A number of other item flaws were identified, which hold the potential to increase the construct irrelevant variance, or irrelevant difficulty, of the questions. These flaws may make the item harder to answer correctly, but are unrelated to the content purportedly being examined, and so are of little benefit in discriminating between candidates of high- or low-ability (Downing, 2002a; Rodríguez-Díez et al., 2016). For example, the use of a negatively worded stem was observed in 21 MCQs (10.3%) in our dataset. These items may often be quicker to construct, and so there may be reluctance from some writers to move away from this phraseology (Haladyna et al., 2002). Indeed, this is a commonly occurring item flaw, typically observed in up to 13% of MCQs in recent studies (Stagnaro-Green & Downing, 2006; Tarrant et al., 2006a, 2006b; Downing, 2005). However, students may find the wording of these items confusing, particularly if double-negatives or additional exemptions are introduced, and so this wording is discouraged in current item-writing guidelines (Tarrant et al., 2006a, 2006b; Case & Swanson, 2002; Haladyna et al., 2002; Wood & Cole, 2001).

One frequent criticism of the MCQ format is that the questions may be phrased in such a way as to be easily guessable to a ‘test-wise’ candidate (Poundstone, 2014). The inclusion of implausible distractors, seen in 15 MCQs within our dataset (7.4%), increases the possibility that a candidate may guess the correct option by random chance. The presence of logical cues or word repeats in both the stem and options may guide a candidate towards the correct option, even if they possess little or no intrinsic content knowledge (Case & Swanson, 2002; Tarrant et al., 2006a, 2006b). The most frequently occurring ‘test-wise’ flaw observed in our dataset was the presence of absolute terms, which was detected in 30 items (14.7% of MCQs; Table 2). While some hard sciences may lend themselves to absolutes, in general variance and individuality are intrinsic parts of life; question writers cannot always account for every possible circumstance (‘never’ might hold true today but not tomorrow). A further complication is that even supposedly absolute terms such as ‘always’ or ‘never’ may be interpreted differently (Holsgrove & Elzubeir, 1998). Test-wise candidates will be aware that the elimination of options containing the words ‘always’ and ‘never’ greatly improves their probability of choosing the correct option by chance. For these reasons, the use of absolute terms, in either the stem or options, is strongly discouraged (Case & Swanson, 2002; Holsgrove & Elzubeir, 1998; Tarrant et al., 2006a, 2006b). Best practice also recommends against using relative or absolute frequency terms (e.g. adverbs such as ‘frequently’, ‘sometimes’), again due to the multiple interpretations that may be inferred (Case, 1994; Holsgrove & Elzubeir, 1998; Tarrant et al., 2006a, 2006b). The potential for learners to be able to guess answers and become test-wise raises a question mark over the current fitness of these tests for formal learning. As such it represents a threat to the trustworthiness of MOOC accreditation which has the potential to undermine their future role in this regard.

It is beyond the scope of this study to say the MCQs analysed, even those functioning ones, contributed to anything beyond factual recall. Valuable lessons that can be drawn from this study are the clear importance of proper faculty training in MCQ writing, and the importance of quality assurance and review. Detailed guidelines with regard to the writing of high-quality MCQs do exist, and ample evidence now exists which shows the extensive, but often unpredictable effects which item flaws have on assessment psychometrics and student performance (Downing, 2005; Tarrant & Ware, 2008; Pais et al., 2016). All question writers are prone to cognitive biases and errors, which proper training should alleviate but may not always overcome, and for this reason additional peer review and statistical analysis of MCQs is considered best practice. This is of course a time-consuming and expensive activity. However, even some simple procedures could be included in MOOC MCQ engines in the future to obviate obvious flaws; the randomisation of option position is a simple matter when assessments are delivered online, but one which surprisingly few MOOCs seem to enforce. Many other common flaws could be detected through algorithmic means (Vasiliki et al., 2015). Within this study, we simply counted the number of characters within each option in order to identify the longest one, but others have used computational techniques to look for the most linguistically complex option instead (Brunnquell et al., 2011). Post-test statistical analysis of MCQs by established methods such as Classical Test Theory, would provide additional information regarding which items are poorly discriminating, and so potentially assist human evaluators in identifying item flaws. Our work here contributes by alerting researchers developing future testing and assessment in MOOCs to common pitfalls they may encounter.

The lack of these psychometric data was a limitation of this study. Hence, we are confined to a descriptive study, a simple representation of the prevalence of item flaws within a convenience sample of MOOCs. However, in the absence of any previous studies of this nature, this publication represents an opening of an important dialogue regarding the quality of these assessments, and the implications should they be used for formal accreditation into the future. The evidence from other domains is conclusive: item flaws are a threat to assessment validity (Downing, 2002a; Downing, 2005; Pais et al., 2016; Tarrant et al., 2006a, 2006b; Tarrant & Ware, 2008). If the MCQs from MOOCs analysed within our dataset were used in formal accredited assessments, it is credible that undesirable effects might occur on student achievement, with some students passing tests beyond their inherent ability, simply because of the presence of flawed items within the tests. It is vital that these assessments are fit-for-purpose, and we believe that rich bodies of research exist that can help define, develop and ensure quality in future online courses. All assessment methods have varying strengths and weaknesses, it is appropriate that multiple constructs and multiple data points are employed, particularly in respect of high-stakes decisions, if MOOCs are to be integrated within formal, accredited, degrees and courses (Sandeen, 2013; Epstein, 2007). So, MCQs may be best deployed as but one component of an overall assessment design that draws on several other existing assessment types such as self-assessment and peer assessment particularly for formative assessment purposes (Admiraal et al., 2015) or that make use of semantic web technologies for more open ended questions (Del Mar Sánchez-Vera and Prendes-Espinosa, 2015). Balfour (2013) gives a useful analysis of the two assessment types comparing and contrasting semantic technologies, in this case automated essay grading, with calibrated human peer review in MOOCs. However, for all of the above approaches, authentication of identity, and potentially proctoring of examinations, remain challenges at the scales involved (Sandeen, 2013).