Technology Enhanced Assessment

Automated generation of assessment items can provide large item pools for formative assessments with little effort. However, if the generation process produces self-contained items, these need to be updated or re-generated each time the data source used for generation changes. This paper describes and discusses an alternative approach that dynamically retrieves item content from Wikidata using SPARQL queries. The paper compares four different examples and discusses both benefits and limitations of this approach. Results show that the approach is usable for a broad range of different items for formative assessment scenarios and that limitations are manageable with acceptable effort.

Modern technologies increasingly make use of personal data to provide better services. Technologies using biometric data for identity and authorship verification in the context of e-assessment are a case in point. Previous studies in e-health described a privacy paradox in relation to consent to personal data use: even when people consider protection of their personal data important, they consent fairly readily to personal data use. However, the new European Data Protection Regulation (GDPR) assumes that people give free and informed consent. In the context of e-assessment, this study investigates students’ attitudes towards personal data sharing for identity and authorship verification purposes with the aim of optimising informed consent practice. Students with special educational needs or disabilities (SEND) were included as a specific target group because they may feel more dependent on e-assessment. The findings suggest that a privacy paradox exists in the context of e-assessment as well. Furthermore, the results indicate that students are more reluctant to share video recordings of their face than other personal data. Finally, our results confirm the effect found in previous studies on e-health: those feeling a stronger need for technologies, in this case SEND students, are more inclined to consent to personal data use. Implications for informed consent practice are discussed.

Rubrics support students in learning complex generic (21^st century) skills, as they provide textual descriptions of skills’ mastery levels with performance indicators for all constituent subskills. If students know their current and strived-for mastery level, they can better determine subsequent learning activities towards skills mastery. However, text-based rubrics have a limited capacity to support the formation of mental models of a complex skill. Video-enhanced rubrics (VER) with video modeling examples have the potential to improve and enrich mental model formation, feedback quality, and thus improve students’ performance.

In the Viewbrics-project we therefore developed, through design-based research, a methodology for the formative assessment of complex skills with Video-Enhanced Rubrics (VER), precipitated in an online tool. This paper describes the features of the first prototype of this online tool and the results of a stakeholder evaluation of its perceived usefulness and usability, by means of a questionnaire and card-sorting exercise, with 7 teachers and 21 students of two secondary schools.

The evaluation of this first prototype showed that both teachers and students evaluated the online tool and formative assessment methodology as handy, usable, helpful and feasible for learning complex skills, although some recommendations were made to further improve the design of the tool.

This paper presents a retrospective analysis of learning in a MOOC as reconstructed from the conversations that learners conducted in MOOC group forums while performing the course tasks. A mixed method approach was applied to analyze the quantity and the quality of these conversations. Two activity patterns were distinguished – in groups with higher activity levels, there were more individual contributions (posts) on more course themes and these contributions were broader spread throughout the course. In high activity groups there was also more interaction between participants, i.e., more questions, answers, explanations and elaborations. The presented study demonstrates how modeling interactions in group forums helps to elicit individual and emerging group knowledge construction and thus supports defining MOOC learning, informs MOOC design and provides insights on how assessing MOOC learning can be automated.

To cope with changing demands from society, higher education institutes are developing adaptive curricula in which a suitable integration of workplace learning is an important factor. Automated feedback can be used as part of formative assessment strategies to enhance student learning in the workplace. However due to the complex and diverse nature of workplace learning processes, it is difficult to align automated feedback to the needs of the individual student. The main research question we aim to answer in this design-based study is: ‘How can we support higher education students’ reflective learning in the workplace by providing automated feedback while learning in the workplace?’. Iterative development yielded (1) a framework for automated feedback in workplace learning, (2) design principles and guidelines and (3) an application prototype implemented according to this framework and design knowledge. In the near future, we plan to evaluate and improve these tentative products in pilot studies.

This exploratory study examines the experience and views of students about 3D Virtual Reality Glasses (3DVRG) and e-authentication systems. The authors developed the “Virtual Reality Classroom” App, which is an Open Educational Resource based on 360 photos of the renowned “Bletchley Park”. Participants were 2 groups of students from the UK and Brazil who explored in pairs this App using a 3DVRG in the classroom and also completed a formative assessment activity using the TeSLA face recognition system. Our research question focuses on whether the students’ interactions through the 3DVRG enhance learning and assessment of inquiry skills for Responsible Research and Innovation (RRI). Findings suggested that the combination of authentic scenario, interactive tasks and assessment-in-context helped students acquire new information and connect with their existing knowledge and practice the inquiry skills together. These interactions enhanced their immersion, particularly for those who found the activities fun as they did not experience motion sickness. Three types of interactions were identified between students with: the virtual space (1), their peer (2) and the topic (3). These three interactions propitiated, respectively, students virtual, social and cognitive presence, which supported their experiential learning.

Conversational interfaces (also called chatbots) have recently disrupted the Internet and opened up endless opportunities for assessment and learning. Formative feedback that provides learners with practical instructions for improvement is one of the challenging tasks in self-assessment settings and self-directed learning. This becomes even more challenging if a user’s personal information such as learning history and previous achievements cannot be exploited for data protection reasons or are simply not available. This study seeks to explore the opportunities of providing formative feedback in chatbot-based self-assessment. Two main challenges were faced: the limitations of the messenger as an interface that restricts visual representation of the quiz questions, and zero information about the user to generate adaptive feedback. Two types of feedback were investigated regarding their formative effect: immediate feedback, which was given after answering a question, and cumulative feedback detailing strengths and weaknesses of the user in each of the topics covered along with the directives for improvement. A chatbot called SQL Quizbot was deployed on Facebook Messenger for the purposes of this study (Try out the prototype at https://​www.​messenger.​com/​t/​2076690849324267​). A survey conducted to disclose users’ perception of the feedback reveals that more than 80% of the users find immediate feedback helpful. Overall this study shows that chatbots have a great potential as an aiding tool for e-learning systems to include an interactive component into feedback in order to increase user motivation and retention.

This paper reports on practical experiences with the two e-assessment tools AlephQ and JACK, explains their key features and sketches usage scenarios from two different universities. Using a lecture in accountancy as a concrete example, the paper then presents a successful concept for improving a lecture by introducing both e-assessment systems. Conclusions are drawn on how to improve a lecture by selecting and combining the most suitable features from different tools.

Digital exams are rather uncommon in engineering education because general e-assessment platforms lack the ability to use advanced item types that mimic general engineering problem-solving processes and award partial scores. However, it is possible to develop such advanced items with Maple T.A.. We describe how such items are structured in scenarios and developed for a second year bachelor’s-level material science course that ran three times at the Delft University of Technology. We evaluate how these items function in practice, are scored and perform from an educational measurement perspective. The paper discusses the results of the study and future directions for development of digital exams in engineering courses.

Item banks in higher education can be regarded as important assets to increasing the quality of education and assessment. An item bank allows for the flexible administration of computer-based achievement tests for summative purposes, as well as quizzes for formative purposes. Developing item banks, however, can require quite an investment. A well-worked-out business case can help with convincing stakeholders to start an item bank development project. An important part of such a business case should be the increase in item quality and the estimated reduction in costs, particularly for the collaborative development of an item bank. However, a theoretical underpinning of a business case, incorporating considerations based on classical test theory is lacking in the literature. Therefore, a model is described to make estimations of reductions in misclassifications and per-unit costs. Examples are presented of the likelihood of reducing misclassifications and cost per unit based on findings in the literature. Implications for research and practice are discussed.

Automatic Assessment Systems empowered by mathematical engines allow the development of online assignments for Mathematics, which goes beyond multiple-choice modality. Automatically assessed assignments, used with formative purposes, can support teaching and learning from several perspectives, such as conceptual and procedural understanding, metacognition, enactment of adaptive strategies, and teachers’ management of the class. This paper reports on an experimentation where automatic assessment has been used in a blended modality according to a model of formative assessment and interactive feedback to enhance learning. The experiment involved a total number of 546 students of 8th grade in the town of Turin (Italy). The use of the automatic assessment is shown and exemplified. Data from learning tests, questionnaire and platform usage are analyzed and used to show the effectiveness of the interactive materials for enhancing mathematical understanding and self-assessment skills. Moreover, a profile of the students who did not use the online opportunities, defined as “reluctant users”, is drawn and discussed.

This design research was focused on developing standards covering the entire process of examination to limit the chances of security risks (e.g., the prevention of exam fraud as much as possible, and detection by means of data forensics), together these standards form the Educational Data Forensics Protocol. Two research questions guided this study. The first question was, which standards regarding preventing and detecting fraud in the process of examination need to be included into the EDF protocol? In addition, practitioners must be able to act on indications of exam fraud based on these standards. Therefore, a second research question was formulated, namely which conditions must be considered during development of the EDF protocol to support practitioners in detecting possible gaps in the security of their examination process?

The EDF protocol was developed and validated in five consecutive steps. This study analyses on the theoretical base of developing the EDF protocol (Step 1) and the considerations for developing a prototype (Step 2). The prototype was being validated (e.g., establishing correctness of the content) through seven semi-structured interviews with content experts in the field of either test security or data forensics (Step 3). Statements from these interviews were used to adjust the prototype into a final version of the EDF protocol (Step 4). Finally, to determine the practical value, the final version of the EDF protocol was used to flag gaps in the security of the exam process and determine possible security risks for one of eX:plain’ s exam programs (Step 5).

This paper proposes gesture performance as one main channel for assessing collaboration skills, while multiple users solve a problem collaboratively on a tangible user interface. Collaborative problem solving incorporates two dimensions, complex problem solving and collaboration. Thus, the technology-based assessment of collaborative problem solving includes assessing both problem solving and collaboration skills. Particularly, for assessing collaboration skills, we consider gesture performance as an important indicator. We differentiate between physical 3D mid-air gestures and manipulative gestures; for the latter, we developed a gesture recognition application using Kinect. The method we follow for object and gesture recognition is to merge the logging files from our tangible interface software framework (object recognition) with the Kinect log files (gesture recognition) in one file. The application can analyze the number of object manipulations with respect to timing axis, subject/participant, and handedness.