Human vs. Machine Marking: A Comparative Study of Chemistry Assessments

Lev Vygotsky’s constructivist learning theory served as the foundation for this study. Vygotsky (1978) asserted that knowledge is developed through social interactions and that learning occurs within the Zone of Proximal Development (ZPD) through scaffolding. This study, which compares human and machine marking of chemistry assessments, aligns closely with constructivist learning theory, emphasizing the significance of active engagement and knowledge construction by learners (Blikstein & Worsley, 2016; Siemens & Long, 2011). Constructivist learning theory also underscores the value of personalized learning environments, acknowledging the diversity of learners and their distinct cognitive processes. It advocates for adapting content and instructional strategies to meet individual student needs (Dede, 2010; Russell & Norvig, 2010). Machine learning algorithms can analyze student data to identify strengths, weaknesses, and learning preferences (Jackson, 2024). AI-driven assessment tools function as virtual scaffolds by offering real-time feedback, hints, and guidance tailored to a student’s current level of understanding. Vygotsky highlighted the role of social interaction in learning. In this context, AI technologies can facilitate active learning by providing interactive, problem-solving scenarios, immediate feedback, adaptive assessments, and scaffolded learning experiences. These tools enable students to actively engage in the learning process, constructing their understanding through exploration and collaboration (Anderson et al., 1995, cited in Jackson, 2024). Additionally, AI can enhance social interactions by incorporating intelligent agents such as chatbots and virtual tutors that engage students in meaningful discussions, guide them through problem-solving tasks, and encourage collaborative knowledge construction (D’Mello & Graesser, 2014). Overall, Vygotsky’s constructivist learning theory provides a solid framework for integrating AI into education, particularly in assessment. AI-driven tools can strengthen formative assessment by identifying students’ ZPD, offering scaffolding, and fostering interactive learning experiences.

Assessment plays a crucial role in education with marking as a critical component. Marking plays a vital role in the teachers’ involvement in the assessment process since written responses are an essential method for providing feedback to students and assisting teachers in evaluating students’ comprehension (Elliott et al., 2016). The process of marking is essential for accuracy in assessing student learning, providing feedback, and guiding instructional decisions. Marking is defined as the process of assessing the quality of written work of students by assigning values for correct responses (Kumari, 2022). Williamson (2018) sees marking as a process where assessors assign numerical scores to candidates’ responses, guided by specific marking criteria. Marking provides both teachers and students with an opportunity to assess and evaluate academic progress in a structured, supportive, and personalized manner (Huddersfield Grammar School, 2016). The researcher defines marking as a measurement process of assigning numerical values to written responses of students to questions using a marking guide, which provides the correct answers and descriptors that clarify how the marks should be distributed. Although marking serves several purposes (Brookhart, 2013) not excluding provision feedback to students on their academic achievement and progress and challenging students’ critical thinking to promote learning. Additionally, it serves as a basis for evaluating students’ attainment of learning objectives, and making decisions about their academic progression, such as promotion, graduation, or certification; it has the potential to be hugely time-consuming and laborious.

Marking is done manually by human experts. Human expert marking involves the evaluation of student work by teachers or other qualified educational professionals providing feedback on student learning outcomes and informing instructional decisions (Elliott et al., 2016). Its role includes ensuring that assessments are fair, reliable, and valid by taking into account the learning context, student entry behavior, and potential mitigating circumstances, interpreting complex responses, recognizing subtle distinctions in student understanding, and using explicit feedback to guide student improvement and learning (Ramesh & Sanampudi, 2022; Black & Williams, 1998 cited in Rawi, 2023). Teachers use their professional judgment to evaluate student performance based on established criteria, which helps in understanding students’ strengths and areas for improvement. Human expert marking relies on the judgment of trained educators to mark students’ work, providing detailed and contextually sensitive assessments (Sadler, 2012). Chartered Institute of Educational Assessors (2019) asserted that human expert markers typically consider student accuracy in demonstrating a correct understanding of the concepts and skills being assessed, application of knowledge and skills to solve problems or complete tasks, work presentation clearly, concisely, and organized, and student critically analyzing information and drawing well-supported conclusions.

In all, effective assessment still requires human expertise to provide insightful feedback, judge complex tasks, and ensure fair and equitable outcomes for all students. Regardless of these advantages of human expert marking, major challenges and inefficiencies have confronted the marking process (Menya, 2018). Human marking is fraught with marker’s subjectivity, marking consistencies, individual marker’s biases, time constraints, and scalability issues (Kazmi, 2022; Ramesh & Sanampudi, 2022). These factors can influence the assessment process and the quality of feedback given to students; hence, the need is for machine marking to overcome the challenges.

Machine marking, also known as automated marking or AI-driven assessment, involves using machines to mark students’ assessments as a result of machine learning (ML) technology (Pearson & Penna, 2023). Machine learning is a subset of AI, which is a process of computers learning and acting like humans. It uses data and algorithms to mimic human perception, decision-making, and the way that humans learn, gradually providing its accuracy in completing a task (Jimenez & Boser, 2021). In other words, ML enables machines to automatically learn from large datasets (such as student responses) and past experiences, recognizing patterns to make predictions with minimal human involvement. Automated marking systems employ ML techniques such as supervised machine learning (SML) and unsupervised machine learning (UML) (McNulty, 2023). The SML involves training algorithms using labeled datasets, allowing them to classify data or make predictions accurately. The model adjusts itself as new input data are introduced until it finds the best fit. In contrast, UML employs ML algorithms to examine and categorize unlabeled datasets, detecting patterns or clusters without the need for labeled data or human guidance. These approaches enable AI-driven automated marking tools to enhance their accuracy and provide more precise feedback, as the systems can learn and improve over time, especially when processing large, complex datasets (Jimenez & Boser, 2021; McNulty, 2023). Although AI based on machine learning is still in its early stages, it has already demonstrated remarkable effectiveness in handling complex tasks that are not rule-based, such as evaluating students’ written answers or analyzing extensive, intricate datasets (Jimenez & Boser, 2021).

There are key benefits of machine marking. First, machine marking is efficient, marking large volumes of assessments rapidly, which significantly saves time and can lead to substantial cost reductions for institutions (Pearson & Penna, 2023; Slimi, 2023). Second, Ramaswami et al. (2023) contended that machine marking applies consistent criteria across all assessments, ensuring uniformity in marking and highlighting that learner-facing dashboard and automated data storytelling tools can provide consistent feedback, reducing the subjectivity associated with human marking. Last, sophisticated automated grading systems can deliver comprehensive and actionable feedback by utilizing natural language processing (NLP) and other AI techniques to examine student responses. They generate feedback that not only helps students recognize their errors but also provides suggestions for improvement, thereby enhancing the effectiveness of student learning support (Paiva et al., 2022). Despite the advantages, there are challenges associated with the implementation of machine marking. One major concern is the accuracy of these systems in understanding and evaluating complex, open-ended responses. While automated systems perform well with structured SAQs and objective questions, their ability to accurately mark essays, creative writing, or nuanced answers is still developing and they cannot provide personalized feedback (McNulty, 2023). Boud and Dawson (2021) pointed out that teacher feedback literacy is crucial in effectively integrating automated tools, as these tools can sometimes misinterpret student responses, leading to incorrect marking. Machine marking algorithms may exhibit bias and inequity depending on how they were trained (McNulty, 2023). Acceptance and trust in machine marking systems to accurately and fairly mark assessments may vary among stakeholders; students need to be assured that their work is being evaluated fairly and that the feedback they receive is meaningful (McNulty, 2023; Pearson & Penna, 2023).

Both human expert marking and machine marking offer unique strengths and drawbacks in assessment; whereas human markers provide nuanced and contextually sensitive evaluations, machine marking systems offer efficiency, consistency, and scalability advantages. Considering the complementary nature of these approaches to marking and addressing the challenges related to validity, reliability, and fairness, educators can leverage both human expertise and technological innovations to enhance assessment practices and support student learning.

Chemistry education is an aspect of STEM education that studies the teaching and learning of chemistry (National Academic Press, 2012). Assessment is essential to effective teaching and learning of chemistry that in turn involves presenting students with tasks and giving feedback. With the increasing population of students in secondary schools, it becomes imperative to introduce machine marking to speed up the process of marking and offer immediate feedback. Machine marking or automated marking, primarily, at the earlier stage of large stake assessment was applied only to objective-type questions. However, AI-driven models in NLP and ML have taken the center stage as SAQs, problem-solving task, and essay can be understood and evaluated seamlessly with automated marking systems (Zhai, 2021.).

Several studies explored different model machine marking in different disciplines including chemistry in higher education. The current study is in secondary education. Zhang et al. (2022) worked on auto-grading of off-line handwritten organic cyclic compound structure. The researchers highlighted that achieving automatic grading of handwritten chemistry assignments necessitates solving the issue of recognizing chemical notations. To address this, they proposed a method based on component detection, which includes detecting components, recognizing text components, and interpreting structures. The model initially identifies the components of offline handwritten organic cyclic compound structures as objects and detects them using the deep learning detector YOLOv5. This is followed by an enhanced attention-based encoder-decoder model for text recognition and a comprehensive algorithm for interpreting the spatial structure of the handwritten content. Similarly, Rahaman and Mahmud (2022) suggested a deep-learning framework for the automated grading of handwritten essay scripts. This architecture combines a CNN with BiLSTM, enabling it to recognize and grade handwritten answers with the same accuracy as a human expert. Zhai (2021) presented ML-based science assessments as cutting-edge technologies that are increasingly involved in innovative assessments in science education including construct, functionality, and automaticity. Zhai argued that automated assessment enables the evaluation of intricate, varied, and structural elements. It facilitates the elicitation of performance, provides methods for gathering, observing, and interpreting evidence, and ultimately aids in prompt and complex decision-making and actions. Reina et al. (2024) developed PLATA, an online automated platform for chemistry undergraduates that allows students to engage in problem-solving practice. “PLATA is programmed to randomly modify not only numeric values but also chemical substances, chemical reactions, and equilibrium constants that are related with one another and cannot vary independently” (Reina et al., 2024:1033). PLATA offers several advantages that include but not limited to considering problems with numerical values representing chemical quantity as correct only when presented with proper units, offering detailed feedback for each attempt, and enabling students to identify the origins of their errors. Moore et al. (2022) assessed the quality and cognitive level of student-generated online short answer questions (SAQs) in college-level chemistry using both human and automated methods. The findings revealed that experts rated 32% of the student-generated questions as high quality and 23% as evaluating higher cognitive processes. However, automatic evaluation with the ChatGPT- 3 model was considered suboptimal because of the overestimation of student-generated SAQs and misclassifying the cognitive processes of Bloom’s taxonomy. These studies are relevant to the current study which explored chemistry test items comprising SAQs.

In another study recently conducted by Hollis-Sando et al. (2023) to investigate medical students’views on computer-based grading and evaluation of the accuracy of deep learning (DL), a subset of machine learning, in assessing medical short answer questions (SAQs). After completing an online survey to gather their opinions on computer-based marking, the responses were initially graded by human evaluators, followed by a deep learning analysis utilizing convolutional neural networks, which are a deep learning algorithm used for image recognition, object detection, and image segmentation. The study found that automated marking of 1-mark SAQs achieved consistently high classification accuracy, with a mean accuracy of 0.98. However, for 2-mark and 3-mark SAQs, which required multi-class classification, the accuracy was lower, with mean accuracies of 0.65 and 0.59, respectively. The use of deep learning in Hollis-Sando et al.’s study demonstrates the potential of advanced machine learning techniques for improving automated marking precision. This can be relevant to the current study, as it suggests that more sophisticated machine learning models might be able to overcome some of the limitations detected in ChatGPT. In addition, Hollis-Sando et al.’s recommendation to combine human and machine marking aligns with the potential benefits of a hybrid approach in assessment as posited in the current study.

Two separate studies found that machine-generated scores generally align with human scores just as closely as scores from one human expert align with those from another human expert when using e-rater and ChatGPT, respectively (Bridgeman et al., 2012; Morjaria et al., 2024). Likewise, Pearson and Penna (2023) employed the NUMBAS e-assessment tool to explore best practices for designing and marking longer questions in surveying modules taken by engineering students at Newcastle University. Their investigation focused on two methods for automated marking of extended computational questions: awarding follow-through marks and breaking down a question into interim steps. The study found that follow-through marks should constitute 25% or 50% of the total available marks to ensure a normal distribution. Additionally, breaking longer calculation questions into too many parts resulted in unnaturally high marks due to excessive guidance provided to students.

From the foregoing review, automated or machine marking is applied to every discipline for marking of SAQs, mathematical tasks, and essays using various AI-driven models. Studies that explored machine marking of handwritten responses first subjected them to a text recognition platform before the actual machine that does the marking. The studies available in the literature to the author were all in higher education and outside the Nigerian context. The current study aimed at comparing machine marking and human expert marking of SAQs of secondary school chemistry.

Figure 1 is the marking framework, which presents the paths that compare the machine and human marking.

Full size image

Figure 1 is meant to compare the human expert marking and machine marking of SAQs in chemistry. In comparative marking according to Saunders and Topping (2023), human experts mark assessments guided by a marking rubric. Comparative analysis of the submissions of the markers can account for quality feedbacks and account for the nuances in the markings in a more effective way. The current study compared human experts marking submissions on an assessment task with the submissions of ChatGPT using the same marking guide, a method of assigning marks for correct responses. “A marking guide provides broad outlines for success and allocates a range of marks for each component” (Federation University Australia, 2024:1). The marking framework as designed involves three steps. First is the preparation of the marking guide indicating where and how marks will be awarded. Second, the responses to the chemistry SAQs and the marking guides are given to human experts for marking; similarly, ChatGPT is prompted to perform the same task. The last stage is comparing the feedback from human marking with feedback from machine marking using ChatGPT by correlating the scores.