main-content

## Über dieses Buch

This book constitutes the refereed proceedings of the 17th International Conference on Artificial Intelligence in Education, AIED 2015, held in Madrid, Spain, in June 2015.

The 50 revised full papers presented together with 3 keynotes, 79 poster presentations, 13 doctoral consortium papers, 16 workshop abstracts, and 8 interactive event papers were carefully reviewed and selected from numerous submissions. The conference provides opportunities for the cross-fertilization of approaches, techniques and ideas from the many fields that comprise AIED, including computer science, cognitive and learning sciences, education, game design, psychology, sociology, linguistics, as well as many domain-specific areas.

## Inhaltsverzeichnis

### Is a Dialogue-Based Tutoring System that Emulates Helpful Co-constructed Relations During Human Tutoring Effective?

We present an initial field evaluation of Rimac, a natural-language tutoring system which implements decision rules that simulate the highly interactive nature of human tutoring. We compared this rule-driven version of the tutor with a non-rule-driven control in high school physics classes. Although students learned from both versions of the system, the experimental group outperformed the control group. A particularly interesting finding is that the experimental version was especially beneficial for female students.

Patricia Albacete, Pamela Jordan, Sandra Katz

### Educational Question Answering Motivated by Question-Specific Concept Maps

Question answering (QA) is the automated process of answering general questions submitted by humans in natural language. QA has previously been explored within the educational context to facilitate learning, however the majority of works have focused on text-based answering. As an alternative, this paper proposes an approach to return answers as a concept map, which further encourages meaningful learning and knowledge organisation. Additionally, this paper investigates whether adapting the returned concept map to the specific question context provides further learning benefit. A randomised experiment was conducted with a sample of 59 Computer Science undergraduates, obtaining statistically significant results on learning gain when students are provided with the question-specific concept maps. Further, time spent on studying the concept maps were positively correlated with the learning gain.

Thushari Atapattu, Katrina Falkner, Nickolas Falkner

### A Study of Automatic Speech Recognition in Noisy Classroom Environments for Automated Dialog Analysis

The development of large-scale automatic classroom dialog analysis systems requires accurate speech-to-text translation. A variety of automatic speech recognition (ASR) engines were evaluated for this purpose. Recordings of teachers in noisy classrooms were used for testing. In comparing ASR results, Google Speech and Bing Speech were more accurate with word accuracy scores of 0.56 for Google and 0.52 for Bing compared to 0.41 for AT&T Watson, 0.08 for Microsoft, 0.14 for Sphinx with the HUB4 model, and 0.00 for Sphinx with the WSJ model. Further analysis revealed both Google and Bing engines were largely unaffected by speakers, speech class sessions, and speech characteristics. Bing results were validated across speakers in a laboratory study, and a method of improving Bing results is presented. Results provide a useful understanding of the capabilities of contemporary ASR engines in noisy classroom environments. Results also highlight a list of issues to be aware of when selecting an ASR engine for difficult speech recognition tasks.

Nathaniel Blanchard, Michael Brady, Andrew M. Olney, Marci Glaus, Xiaoyi Sun, Martin Nystrand, Borhan Samei, Sean Kelly, Sidney D’Mello

### Teachable Agents with Intrinsic Motivation

Dynamic communication between Teachable Agents (TA) and students is crucial for educational effectiveness of the TA, as dynamic interaction is the vital part throughout the teaching and learning processes. Existing TA design mainly focuses on the functions and features to ensure the TA to be taught by students rather than bi-directional interaction. However, according to reciprocity theory in social psychology, if the TA can offer friendly actions, students in response will be much more cooperative and motivated. In order to improve quality of communication and seize the interest of students, we propose a need modeling approach to enable TAs to have “intrinsic motivations”. In this way, the TA can proactively carry out dynamic communication with students so that the TA can adapt to students’ changing behaviors and sustain a good human-agent relationship. Our field study showed that students were highly attracted by the TA with dynamic needs. They statistically completed more tasks. Also, better results were obtained on students’ learning efficiency and attitude towards TA’s informational usefulness and affective interactions.

Ailiya Borjigin, Chunyan Miao, Su Fang Lim, Siyao Li, Zhiqi Shen

### Temporal Generalizability of Face-Based Affect Detection in Noisy Classroom Environments

The goal of this paper was to explore the possibility of generalizing face-based affect detectors across multiple days, a problem which plagues physiological-based affect detection. Videos of students playing an educational physics game were collected in a noisy computer-enabled classroom environment where students conversed with each other, moved around, and gestured. Trained observers provided real-time annotations of learning-centered affective states (e.g., boredom, confusion) as well as off-task behavior. Detectors were trained using data from one day and tested on data from different students on another day. These cross-day detectors demonstrated above chance classification accuracy with average Area Under the ROC Curve (AUC, .500 is chance level) of .658, which was similar to within-day (training and testing on data collected on the same day) AUC of .667. This work demonstrates the feasibility of generalizing face-based affect detectors across time in an ecologically valid computer-enabled classroom environment.

Nigel Bosch, Sidney D’Mello, Ryan Baker, Jaclyn Ocumpaugh, Valerie Shute

### Transfer Learning for Predictive Models in Massive Open Online Courses

Data recorded while learners are interacting with Massive Open Online Courses (MOOC) platforms provide a unique opportunity to build predictive models that can help anticipate future behaviors and develop interventions. But since most of the useful predictive problems are defined for a real-time framework, using knowledge drawn from the past courses becomes crucial. To address this challenge, we designed a set of processes that take advantage of knowledge from both previous courses and previous weeks of the same course to make real time predictions on learners behavior. In particular, we evaluate multiple transfer learning methods. In this article, we present our results for the stopout prediction problem (predicting which learners are likely to stop engaging in the course). We believe this paper is a first step towards addressing the need of transferring knowledge across courses.

Sebastien Boyer, Kalyan Veeramachaneni

### Mind the Gap: Improving Gender Equity in Game-Based Learning Environments with Learning Companions

Game-based learning environments hold great promise for engaging learners. Yet game mechanics can initially pose barriers for students with less prior gaming experience. This paper examines game-based learning for a population of middle school learners in the US, where female students tend to have less gaming experience than male students. In a pilot study with an early version of E

ngage

, a game-based learning environment for middle school computer science education, female students reported higher initial frustration. To address this critical issue, we developed a prototype learning companion designed specifically to reduce frustration through the telling of autobiographical stories. In a pilot study of two 7

th

grade classrooms, female students responded especially positively to the learning companion, eliminating the gender gap in reported frustration. The results suggest that introducing learning companions can directly contribute to making the benefits of game-based learning equitable for all learners.

Philip Sheridan Buffum, Kristy Elizabeth Boyer, Eric N. Wiebe, Bradford W. Mott, James C. Lester

### Comparing Representations for Learner Models in Interactive Simulations

Providing adaptive support in Exploratory Learning Environments is necessary but challenging due to the unstructured nature of interactions. This is especially the case for complex simulations such as the DC Circuit Construction Kit used in this work. To deal with this complexity, we evaluate alternative representations that capture different levels of detail in student interactions. Our results show that these representations can be effectively used in the user modeling framework proposed in [2], including behavior discovery and user classification, for student assessment and providing real-time support. We discuss trade-offs between high and low levels of detail in the tested interaction representations in terms of their ability to evaluate learning and inform feedback.

Cristina Conati, Lauren Fratamico, Samad Kardan, Ido Roll

### Games are Better than Books: In-Situ Comparison of an Interactive Job Interview Game with Conventional Training

Technology-enhanced learning environments are designed to help users practise social skills. In this paper, we present and evaluate a virtual job interview training game which has been adapted to the special requirements of young people with low chances on the job market. The evaluation spanned three days, during which we compared the technology-enhanced training with a traditional learning method usually practised in schools, i.e. reading a job interview guide. The results are promising as professional career counsellors rated the pupils who trained with the system significantly better than those who learned with the traditional method.

Ionut Damian, Tobias Baur, Birgit Lugrin, Patrick Gebhard, Gregor Mehlmann, Elisabeth André

### Predicting Comprehension from Students’ Summaries

Comprehension among young students represents a key component of their formation throughout the learning process. Moreover, scaffolding students as they learn to coherently link information, while organically constructing a solid knowledge base, is crucial to students’ development, but requires regular assessment and progress tracking. To this end, our aim is to provide an automated solution for analyzing and predicting students’ comprehension levels by extracting a combination of reading strategies and textual complexity factors from students’ summaries. Building upon previous research and enhancing it by incorporating new heuristics and factors, Support Vector Machine classification models were used to validate our assumptions that automatically identified reading strategies, together with textual complexity indices applied on students’ summaries, represent reliable estimators of comprehension.

Mihai Dascalu, Larise Lucia Stavarache, Philippe Dessus, Stefan Trausan-Matu, Danielle S. McNamara, Maryse Bianco

### A Tutorial Dialogue System for Real-Time Evaluation of Unsupervised Dialogue Act Classifiers: Exploring System Outcomes

Dialogue act classification is an important step in understanding students’ utterances within tutorial dialogue systems. Machine-learned models of dialogue act classification hold great promise, and among these, unsupervised dialogue act classifiers have the great benefit of eliminating the human dialogue act annotation effort required to label corpora. In contrast to traditional evaluation approaches which judge unsupervised dialogue act classifiers by accuracy on manual labels, we present results of a study to evaluate the performance of these models with respect to their performance within end-to-end system evaluation. We compare two versions of the tutorial dialogue system for introductory computer science: one that relies on a supervised dialogue act classifier and one that depends on an unsupervised dialogue act classifier. A study with 51 students shows that both versions of the system achieve similar learning gains and user satisfaction. Additionally, we show that some incoming student characteristics are highly correlated with students’ perceptions of their experience during tutoring. This first end-to-end evaluation of an unsupervised dialogue act classifier within a tutorial dialogue system serves as a step toward acquiring tutorial dialogue management models in a fully automated, scalable way.

Aysu Ezen-Can, Kristy Elizabeth Boyer

### Positive Impact of Collaborative Chat Participation in an edX MOOC

A major limitation of the current generation of MOOCs is a lack of opportunity for students to make use of each other as resources. Analyses of attrition and learning in MOOCs both point to the importance of social engagement for motivational support and overcoming difficulties with material and course procedures. In this paper we evaluate an intervention that makes synchronous collaboration opportunities available to students in an edX MOOC. We have implemented a Lobby program that students can access via a live link at any time. Upon entering the Lobby, they are matched with other students that are logged in to it. Once matched, they are provided with a link to a chat room where they can work with their partner students on a synchronous collaboration activity, supported by a conversational computer agent. Results of a survival model in which we control for level of effort suggest that having experienced a collaborative chat is associated with a slow down in the rate of attrition over time by a factor of two. We discuss implications for design, limitations of the current study, and directions for future research.

Oliver Ferschke, Diyi Yang, Gaurav Tomar, Carolyn Penstein Rosé

### Who Needs Help? Automating Student Assessment Within Exploratory Learning Environments

This article describes efforts to offer automated assessment of students within an exploratory learning environment. We present a regression model that estimates student assessments in an ill-defined medical diagnosis tutor called Rashi. We were pleased to find that basic features of a student’s solution predicted expert assessment well, particularly when detecting low-achieving students. We also discuss how expert knowledge bases might be leveraged to improve this process. We suggest that developers of exploratory learning environments can leverage this technique with relatively few extensions to a mature system. Finally, we describe the potential to utilize this information to direct teachers’ attention towards students in need of help.

Mark Floryan, Toby Dragon, Nada Basit, Suellen Dragon, Beverly Woolf

### Moody Agents: Affect and Discourse During Learning in a Serious Game

The current study investigated teacher emotions, student emotions, and discourse features in relation to learning in a serious game. The experiment consisted of 48 subjects participating in a 4-condition within-subjects counter-balanced pretest-interaction-posttest design. Participants interacted with a serious game teaching research methodology with natural language conversations between the human student and two artificial pedagogical agents. The discourse of the artificial pedagogical agents was manipulated to evoke student affective states. Student emotion was measured via affect grids and discourse features were measured with computational linguistics techniques. Results indicated that learner’s arousal levels impacted learning and that language use is correlated with learning.

Carol M. Forsyth, Arthur Graesser, Andrew M. Olney, Keith Millis, Breya Walker, Zhiqiang Cai

### Examining the Predictive Relationship Between Personality and Emotion Traits and Learners’ Agent-Direct Emotions

The current study examined the relationships between learners’ (

N

= 124) personality traits, the emotions they experience while typically studying (trait studying emotions), and the emotions they reported experiencing as a result of interacting with two Pedagogical Agents (PAs - agent-directed emotions) in MetaTutor, an advanced multi-agent learning environment. Overall, significant relationships between a subset of trait emotions (trait anger, trait anxiety) and personality traits (agreeableness, conscientiousness, and neuroticism) were found for three agent-directed emotions (pride, boredom, and neutral) though the relationships differed between the two PAs. These results demonstrate that some trait emotions and personality traits can be used to predict learners’ emotions toward specific PAs (with different roles). Suggestions are provided for adapting PAs to support learners’ (with certain characteristics) experience of positive emotions (e.g., enjoyment) and minimize their experience of negative emotions (e.g., boredom). Such an approach presents a scalable and easily implemented method for creating emotionally-adaptive, agent-based learning environments, and improving learner-PA interactions to support learning.

Jason M. Harley, Cassia C. Carter, Niki Papaionnou, François Bouchet, Ronald S. Landis, Roger Azevedo, Lana Karabachian

### Evaluating Human and Automated Generation of Distractors for Diagnostic Multiple-Choice Cloze Questions to Assess Children’s Reading Comprehension

We report an experiment to evaluate DQGen’s performance in generating three types of distractors for diagnostic multiple-choice cloze (fill-in-the-blank) questions to assess children’s reading comprehension processes. Ungrammatical distractors test syntax, nonsensical distractors test semantics, and locally plausible distractors test inter-sentential processing. 27 knowledgeable humans rated candidate answers as correct, plausible, nonsensical, or ungrammatical without knowing their intended type or whether they were generated by DQGen, written by other humans, or correct. Surprisingly, DQGen did significantly better than humans at generating ungrammatical distractors and slightly better than them at generating nonsensical distractors, albeit worse at generating plausible distractors. Vetting its output and writing distractors only when necessary would take half as long as writing them all, and improve their quality.

Yi-Ting Huang, Jack Mostow

### Machine Learning for Holistic Evaluation of Scientific Essays

In the US in particular, there is an increasing emphasis on the importance of science in education. To better understand a scientific topic, students need to compile information from multiple sources and determine the principal causal factors involved. We describe an approach for automatically inferring the quality and completeness of causal reasoning in essays on two separate scientific topics using a novel, two-phase machine learning approach for detecting causal relations. For each core essay concept, we initially trained a window-based tagging model to predict which individual words belonged to that concept. Using the predictions from this first set of models, we then trained a second stacked model on all the predicted word tags present in a sentence to predict inferences between essay concepts. The results indicate we could use such a system to provide explicit feedback to students to improve reasoning and essay writing skills.

Simon Hughes, Peter Hastings, Mary Anne Britt, Patricia Wallace, Dylan Blaum

### Learning to Diagnose a Virtual Patient: an Investigation of Cognitive Errors in Medical Problem Solving

Although cognitive errors (i.e., premature closure, faulty data gathering, and faulty knowledge) are the main reasons for making diagnostic mistakes, the mechanisms by which they occur are difficult to isolate in clinical settings. Computer-based learning environments (CBLE) offer the opportunity to train medical students to avoid cognitive errors by tracking the onset of these errors. The purpose of this study is to explore cognitive errors in a CBLE called BioWorld. A logistic regression was fitted to learner behaviors that characterize premature closure in order to predict diagnostic performance. An ANOVA was used to assess if participants who were highly confident in their wrong diagnosis engaged in more faulty data gathering via confirmation bias. Findings suggest that diagnostic mistakes can be predicted from faulty knowledge and faulty data gathering and indicate poor metacognitive awareness. This study supports the notion that to improve diagnostic performance medical education programs should promote metacognitive skills.

Amanda Jarrell, Tenzin Doleck, Eric Poitras, Susanne Lajoie, Tara Tressel

### Studying Student Use of Self-Regulated Learning Tools in an Open-Ended Learning Environment

This paper discusses a design-based research study that we conducted in a middle school science classroom to test the effectiveness of SimSelf, an open-ended learning environment for science learning. In particular, we evaluated two tools intended to help students develop and practice the important regulatory processes of planning and monitoring. Findings showed that students who used the supporting tools as intended demonstrated effective learning of the science topic. Conversely, students who did not use the tools effectively generally achieved minimal success at their learning tasks. Analysis of these results provides a framework for redesigning the environment and highlights areas for additional scaffolding and guidance.

John S. Kinnebrew, Brian C. Gauch, James R. Segedy, Gautam Biswas

### Situated Pedagogical Authoring: Authoring Intelligent Tutors from a Student’s Perspective

We describe the Situated Pedagogical Authoring (SitPed) system that seeks to allow non-technical authors to create ITS content for soft-skills training, such as counseling skills. SitPed is built on the assertion that authoring tools should use the learner’s perspective to the greatest extent possible. SitPed provides tools for creating tasks lists, authoring assessment knowledge, and creating tutor messages. We present preliminary findings of a two-phase study comparing authoring in SitPed to an ablated version of the same system and a spreadsheet-based control. Findings suggest modest advantages for SitPed in terms of the quality of the authored content and student learning.

H. Chad Lane, Mark G. Core, Matthew J. Hays, Daniel Auerbach, Milton Rosenberg

### Two Modes are Better Than One: a Multimodal Assessment Framework Integrating Student Writing and Drawing

We are beginning to see the emergence of advanced automated assessment techniques that evaluate expressive student artifacts such as free-form written responses and sketches. These approaches have largely operated individually, each considering only a single mode. We hypothesize that there are synergies to be leveraged in multimodal assessments that can integrate multiple modalities of student responses to create a more complete and accurate picture of a student’s knowledge. In this paper, we introduce a novel multimodal assessment framework that integrates two techniques for automatically analyzing student artifacts: a deep learning-based model for assessing student writing, and a topology-based model for assessing student drawing. An evaluation of the framework with elementary students’ writing and drawing assessments demonstrate that 1) each of the framework’s two modalities provides an independent and complementary measure of student science learning, and 2) together, the multimodal framework significantly outperforms either uni-modal approach individually, demonstrating the potential synergistic benefits of multimodal assessment.

Samuel Leeman-Munk, Andy Smith, Bradford Mott, Eric Wiebe, James Lester

### To Resolve or not to Resolve? that is the Big Question About Confusion

Positive relationships between confusion and learning have been found for the last decade. Most theoretical foundations for confusion hypothesize that it is not the mere occurrence of confusion, but rather the successful resolution that benefits learning. Empirical research has provided some support for this hypothesis, but investigations of the confusion resolution process are still sparse. The present work is a preliminary investigation of the confusion resolution process within two learning environments that experimentally induce confusion (false feedback, contradictory information). Findings showed that learners did benefit from confusion resolution compared to when confusion was unresolved, but it was not merely from increased effort. The nature of the confusion induction method also influenced the positive impact of confusion resolution on learning. Implications for intelligent tutoring systems are discussed.

Blair Lehman, Art Graesser

Making effective problem selection decisions is an important yet challenging self-regulated learning (SRL) skill. Although efforts have been made to scaffold students’ problem selection in intelligent tutoring systems (ITS), little work has tried to support students’ learning of the transferable problem selection skill that can be applied when the scaffolding is not in effect. The current work uses a user-centered design approach to extend an ITS for equation solving,

Lynnette

, so the new designs may motivate and help students learn to apply a general, transferable rule for effective problem selection, namely, to select problem types that are not fully mastered (“Mastery Rule”). We conducted user research through classroom experimentation, interviews and storyboards. We found that the presence of an Open Learner Model significantly improves students’ problem selection decisions, which has not been empirically established by prior work; also, lack of motivation, especially lack of a mastery-approach orientation, may cause difficulty in applying the Mastery Rule. Based on our user research, we designed prototypes of tutor features that aim to foster a mastery-approach orientation as well as transfer of the learned Mastery Rule when the scaffolding is faded. The work contributes to the research of supporting SRL in ITSs through a motivational design perspective, and lays foundation for future controlled experiments to evaluate the transfer of the problem selection skill in new tutor units where there is no scaffolding.

Yanjin Long, Zachary Aman, Vincent Aleven

### SNS Messages Recommendation for Learning Motivation

Setting goals for learning enhances motivation and performance. This research shows that observing learning goals from peers on social networks allows learners to specify new learning purposes and to enhance the perception of their own expertise. This study consists of: 1) a model recommending goal-based messages from peers with diverse textual contents (i.e. purpose) for a same goal (e.g. mastering English), and 2) a Web-based implementation using an LDA (Latent Dirichlet Allocation) model, known as a highly accurate text latent topic model. The experiment was conducted by university students who expressed and evaluated their goals before observing similar/diverse messages from other peers. Results showed that observing the diversity of peers’ learning purposes is an important factor positively affecting intrinsic motivational attributes such as goal specificity and confidence to achieve the goal.

Sébastien Louvigné, Yoshihiro Kato, Neil Rubens, Maomi Ueno

### How Spacing and Variable Retrieval Practice Affect the Learning of Statistics Concepts

This research investigated key factors in learning conceptual material about statistics, and tested the effect of variability during retrieval practice. The goal was to build a model of learning for schedule-based interventions. Participants (

n

= 230) completed multiple reading and test trials with fill in the blank sentences about basic statistics concepts. The experiment was a 2 (trial type: read or drill) × 3 (learning trial spacing: wide medium, or narrow) × 2 (fill-in term during learning: variable or constant) × 2 (fill-in term during posttest: variable or constant) within-subjects design. The model of the results captures the data with recent and long-term components to explain posttest transfer and the testing and spacing effects. These results, and data on the conceptual confusions amongst statistical terms, are discussed with respect to implications for future intelligent learning systems.

Jaclyn K. Maass, Philip I. Pavlik, Henry Hua

### Leveraging Multiple Views of Text for Automatic Question Generation

Automatic question generation can play a vital role in educational applications such as intelligent tutoring systems. Prior work in question generation relies primarily on one view of the sentence provided by a parser of a given type, such as phrase structure trees or predicate argument structure. In contrast, we explore using multiple views from different parsers to create a tree structure which represents items of interest for question generation. This approach resulted in a 17% reduction in the error rate compared with our prior work, which achieved a 44% reduction in the error rate compared to state-of-the-art question generation systems. Additionally, the work presented in this paper generates with greater question variety than our previous work, and creates 21% more semantically-oriented versus factoid questions.

Karen Mazidi, Rodney D. Nielsen

### Mind Wandering During Learning with an Intelligent Tutoring System

Mind wandering (zoning out) can be detrimental to learning outcomes in a host of educational activities, from reading to watching video lectures, yet it has received little attention in the field of intelligent tutoring systems (ITS). In the current study, participants self-reported mind wandering during a learning session with Guru, a dialogue-based ITS for biology. On average, participants interacted with Guru for 22 minutes and reported an average of 11.5 instances of mind wandering, or one instance every two minutes. The frequency of mind wandering was compared across five different phases of Guru (Common-Ground-Building Instruction, Intermittent Summary, Concept Map, Scaffolded Dialogue, and Cloze task), each requiring different learning strategies. The rate of mind wandering per minute was highest during the Common-Ground-Building Instruction and Scaffolded Dialogue phases of Guru. Importantly, there was significant negative correlation between mind wandering and learning, highlighting the need to address this phenomena during learning with ITSs.

Caitlin Mills, Sidney D’Mello, Nigel Bosch, Andrew M. Olney

### DeepStealth: Leveraging Deep Learning Models for Stealth Assessment in Game-Based Learning Environments

A distinctive feature of intelligent game-based learning environments is their capacity for enabling stealth assessment. Stealth assessments gather information about student competencies in a manner that is invisible, and enable drawing valid inferences about student knowledge. We present a framework for stealth assessment that leverages

deep learning

, a family of machine learning methods that utilize deep artificial neural networks, to infer student competencies in a game-based learning environment for middle grade computational thinking, E

ngage

. Students’ interaction data, collected during a classroom study with E

ngage

, as well as prior knowledge scores, are utilized to train deep networks for predicting students’ post-test performance. Results indicate deep networks that are pre-trained using stacked denoising autoencoders achieve high predictive accuracy, significantly outperforming standard classification techniques such as support vector machines and naïve Bayes. The findings suggest that deep learning shows considerable promise for automatically inducing stealth assessment models for intelligent game-based learning environments.

Wookhee Min, Megan H. Frankosky, Bradford W. Mott, Jonathan P. Rowe, Eric Wiebe, Kristy Elizabeth Boyer, James C. Lester

### Learning Mental Models of Human Cognitive Processing by Creating Cognitive Models

We investigated how creating cognitive models enhances learners’ construction of mental models on human cognitive information processing. Two class practices for undergraduates and graduates were performed, in which participants were required to construct a computational running model of solving subtraction problems and then develop a bug model that simulated students’ arithmetic errors. Analyses showed that by creating cognitive models, participants learned to identify buggy procedures that produce systematic errors and predict expected erroneous answers by mentally simulating the mental model. The limitation is that this benefit of creating cognitive models was observed only in participants who successfully programmed a computational model.

Kazuhisa Miwa, Nana Kanzaki, Hitoshi Terai, Kazuaki Kojima, Ryuichi Nakaike, Junya Morita, Hitomi Saito

### A Player Model for Adaptive Gamification in Learning Environments

Many learning environments are swiftly abandoned by the learners, even if they are effective. Gamification is as a recent game-based learning approach that can enhance the learners’ motivation. However, individual expectations and preferences towards game-like features may be very different from one person to another. This paper presents a model to adapt gamification features according to a player profile of the learners. Two version of this model are evaluated within a gamified online learning environment. The first version comes from experts’ judgment, and the second one is induced from empirical data. Our experiments confirm that the first version can be efficient to predict the player’s preferences among the gamification features.

Baptiste Monterrat, Michel Desmarais, Élise Lavoué, Sébastien George

### Exploring the Impact of a Learning Dashboard on Student Affect

Research highlights that many students experience negative emotions during learning activities, and these can have a detrimental impact on behaviors and outcomes. Here, we investigate the impact of a particular kind of affective intervention, namely a learning dashboard, on two deactivating emotions: boredom and lack of excitement. The data comes from a study we conducted with over 200 middle school students interacting with an intelligent tutor that provided varying levels of support to encourage dashboard use. We analyze the data using a range of techniques to show that the learning dashboard is associated with reduced deactivating emotions, but that its utility also depends on the way its use is promoted and on students’ gender.

Kasia Muldner, Michael Wixon, Dovan Rai, Winslow Burleson, Beverly Woolf, Ivon Arroyo

### Cognitive Tutor Use in Chile: Understanding Classroom and Lab Culture

As technological capabilities flourish around the world, intelligent tutoring systems are being deployed globally to provide learners with access to quality educational interventions. Such systems have been widely studied in in-vivo deployments in the Western world, allowing for the development of sophisticated models of behavior within the system that have been shown to accurately represent and support learning. Yet, these models have recently been shown not to reliably transfer across cultures. In this paper, we report on our quantitative field observations of student behaviors in two different schools (urban and rural) and two different learning contexts (ITS lab and the math classroom) in central Chile. We observed that students across schools exhibit different behaviors in the ITS lab vs the classroom, especially with respect to student interaction, movement, and on-task behavior, yet these students behave altogether differently from previously observed U.S. student populations. These results have implications for future modeling efforts of help-seeking and engagement in advanced learning technologies in new global contexts.

Amy Ogan, Evelyn Yarzebinski, Patricia Fernández, Ignacio Casas

### TARLAN: a Simulation Game to Improve Social Problem-Solving Skills of ADHD Children

Attention Deficit Hyperactivity Disorder (ADHD) is a developmental disorder which can impact different aspects of sufferers’ lives, including their social skills. We present TARLAN, a game that teaches social problem-solving skills to ADHD children. TARLAN is a simulation game with 40 scenarios based on children’s everyday experiences. We conducted a study to investigate how the social problem-solving skills of ADHD children are affected by interactions with TARLAN. Forty children with ADHD aged 8-12 were randomly allocated to two conditions: a computer-based intervention wherein children worked with TARLAN, and the control group with the standard psychological intervention. Another group of 20 children without ADHD but with inadequate social skills also worked with TARLAN. Results show that TARLAN significantly improved social problem-solving skill of the ADHD children compared to their peers who were in the psychological intervention. The game is also beneficial for chil-dren who have social skills deficit but who are not diagnosed with ADHD.

### Blocking Vs. Interleaving: Examining Single-Session Effects Within Middle School Math Homework

The benefit of interleaving cognitive content has gained attention in recent years, specifically in mathematics education. The present study serves as a conceptual replication of previous work, documenting the interleaving effect within a middle school sample through brief homework assignments completed within ASSISTments, an adaptive tutoring platform. The results of a randomized controlled trial are presented, examining a practice session featuring interleaved or blocked content spanning three skills: Complementary and Supplementary Angles, Surface Area of a Pyramid, and Compound Probability without Replacement. A second homework session served as a delayed posttest. Tutor log files are analyzed to track student performance and to establish a metric of global mathematics skill for each student. Findings suggest that interleaving is beneficial in the context of adaptive tutoring systems when considering learning gains and average hint usage at posttest. These observations were especially relevant for low skill students.

Korinn Ostrow, Neil Heffernan, Cristina Heffernan, Zoe Peterson

### Impact of Adaptive Educational System Behaviour on Student Motivation

In this work we try to connect research on student modeling and student motivation, particularly on the relation between task difficulty and engagement. We perform experiments within widely used adaptive practice system for geography learning. The results document the impact of the choice of a question construction algorithm and target difficulty on student perception of question suitability and on their willingness to use the system. We also propose and evaluate a mechanism for a dynamic difficulty adjustment.

### Understanding Student Success in Chemistry Using Gaze Tracking and Pupillometry

Eye tracking allows us to identify visual strategies through gaze behavior, which can help us understand how students process content. Furthermore, understanding which visual strategies are successful can help us improve educational materials that foster successful use of these visual strategies. Previous studies have demonstrated the predictive value of eye tracking for student performance. Chemistry is a highly visual domain, making it particularly appropriate to study visual strategies. Eye tracking also provides measures of pupil dilation that correlate with cognitive processes important to learning, but have not yet been assessed in any realistic learning environments. We examined the gaze behavior and pupil dilation of undergraduate students working with a specialized ITS for chemistry: Chem Tutor. Chem Tutor emphasizes visual learning by focusing specifically on graphical representations. We assessed the value of over 40 high-level gaze features along with measures of pupil diameter to predict student performance and learning gains across an entire chemistry problem set. We found that certain gaze features are strong predictors of performance, but less so of learning gains, while pupil diameter is marginally predictive of learning gains, but not performance. Further studies that assess pupil dilation with higher temporal precision will be necessary to draw conclusions about the limits of its predictive power.

Joshua Peterson, Zachary Pardos, Martina Rau, Anna Swigart, Colin Gerber, Jonathan McKinsey

### AttentiveLearner: Improving Mobile MOOC Learning via Implicit Heart Rate Tracking

We present AttentiveLearner, an intelligent mobile learning system optimized for consuming lecture videos in both Massive Open Online Courses (MOOCs) and flipped classrooms. AttentiveLearner uses on-lens finger gestures as an intuitive control channel for video playback. More importantly, AttentiveLearner implicitly extracts learners’ heart rates and infers their attention by analyzing learners’ fingertip transparency changes during learning on today’s unmodified smart phones. In a 24-participant study, we found heart rates extracted from noisy image frames via mobile cameras can be used to predict both learners’ “mind wandering” events in MOOC sessions and their performance in follow-up quizzes. The prediction performance of AttentiveLearner (accuracy = 71.22%, kappa = 0.22) is comparable with existing research using dedicated sensors. AttentiveLearner has the potential to improve mobile learning by reducing the sensing equipment required by many state-of-the-art intelligent tutoring algorithms.

Phuong Pham, Jingtao Wang

### Distractor Quality Evaluation in Multiple Choice Questions

Multiple choice questions represent a widely used evaluation mode; yet writing items that properly evaluate student learning is a complex task. Guidelines were developed for manual item creation, but automatic item quality evaluation would constitute a helpful tool for teachers.

In this paper, we present a method for evaluating distractor (i.e. incorrect option) quality that combines syntactic and semantic homogeneity criteria, based on Natural Language Processing methods. We perform an evaluation of this method on a large MCQ corpus and show that the combination of several measures enables us to validate distractors.

Van-Minh Pho, Anne-Laure Ligozat, Brigitte Grau

### Interpreting Freeform Equation Solving

Learners’ step-by-step solutions can offer insight into their misunderstandings. Because of the difficulty of automatically interpreting freeform solutions, educational technologies often structure problem solving into particular patterns. Hypothesizing that structured interfaces may frustrate some learners, we conducted an experiment comparing two interfaces for solving equations: one requires users to enter steps in an efficient sequence and insists each step be mathematically correct before the user can continue, and the other allows users to enter any steps they would like. We find that practicing equation solving in either interface was associated with improved scores on a multiple choice assessment, but that users who had the freedom to make mistakes were more satisfied with the interface. In order to make inferences from these more freeform data, we develop a Bayesian inverse planning algorithm for diagnosing algebra understanding that interprets individual equation solving steps and places no restrictions on the ordering or correctness of steps. This algorithms draws inferences and exhibits similar confidence based on data from either interface. Our work shows that inverse planning can interpret freeform problem solving, and suggests the need to further investigate how structured interfaces affect learners’ motivation and engagement.

Anna N. Rafferty, Thomas L. Griffiths

### ITS Support for Conceptual and Perceptual Connection Making Between Multiple Graphical Representations

Connection making between representations is crucial to learning in STEM domains, but it is a difficult task for students. Prior research shows that supporting connection making enhances students’ learning of domain knowledge. Most prior research has focused on supporting one type of connection-making process:

conceptual

reasoning about connections between representations. Yet, recent research suggests that a second type of connection-making process plays a role in students’ learning:

perceptual

translation between representations. We hypothesized that combining support for both conceptual and perceptual connection-making processes leads to higher learning gains on a domain-knowledge test. We tested this hypothesis in a lab experiment with 117 undergraduate students using an intelligent tutoring system for chemistry. Results show that the combination of conceptual and perceptual connection-making supports leads to higher learning outcomes. This finding suggests that the effectiveness of educational technologies can be enhanced if they combine support for conceptual and perceptual connection-making processes.

Martina A. Rau, Sally P. W. Wu

### Discovering Individual and Collaborative Problem-Solving Modes with Hidden Markov Models

Supporting students during learning tasks is the main goal of intelligent tutoring systems, and the most effective systems can adapt to students based on a model of their current state of knowledge or their problem-solving actions. Most tutoring systems focus on individual students, but there is growing interest in supporting student pairs. However, modeling student pairs involves considerations that may differ from individual students. This paper reports on hidden Markov models (HMMs) of student interactions within a visual programming environment. We compare HMMs for individual students to those of student pairs and examine the different approaches the students take. The resulting models suggest that there are some important differences across both conditions. There is potential for using these models to predict problem-solving modes and support adaptive tutoring for collaboration in problem-solving domains.

Fernando J. Rodríguez, Kristy Elizabeth Boyer

### Improving Student Problem Solving in Narrative-Centered Learning Environments: a Modular Reinforcement Learning Framework

Narrative-centered learning environments comprise a class of game-based learning environments that embed problem solving in interactive stories. A key challenge posed by narrative-centered learning is dynamically tailoring story events to enhance student learning. In this paper, we investigate the impact of a data-driven tutorial planner on students’ learning processes in a narrative-centered learning environment, C

rystal

I

sland

. We induce the tutorial planner by employing

modular reinforcement learning

, a multi-goal extension of classical reinforcement learning. To train the planner, we collected a corpus from 453 middle school students who used C

rystal

I

sland

in their classrooms. Afterward, we investigated the induced planner’s impact in a follow-up experiment with another 75 students. The study revealed that the induced planner improved students’ problem-solving processes—including hypothesis testing and information gathering behaviors—compared to a control condition, suggesting that modular reinforcement learning is an effective approach for tutorial planning in narrative-centered learning environments.

Jonathan P. Rowe, James C. Lester

### Filtering of Spontaneous and Low Intensity Emotions in Educational Contexts

Affect detection is a challenging problem, even more in educational contexts, where emotions are spontaneous and usually subtle. In this paper, we propose a two-stage detection approach based on an initial binary discretization followed by a specific emotion prediction stage. The binary classification method uses several distinct sources of information to detect and filter relevant time slots from an affective point of view. An accuracy close to 75% at detecting whether the learner has felt an educationally relevant emotion on 20 second time slots has been obtained. These slots can then be further analyzed by a second classifier, to determine the specific user emotion.

Sergio Salmeron-Majadas, Miguel Arevalillo-Herráez, Olga C. Santos, Mar Saneiro, Raúl Cabestrero, Pilar Quirós, David Arnau, Jesus G. Boticario

### Contextual Recommendation of Educational Contents

This paper proposes a recommendation engine for educational contents in the organizational context of a user. The novelty in this paper lies in creating a context model for a user incorporating the role and the tasks assigned to him, and its application to recommendation problem. The recommendations are made on the basis of the estimated gap that exists between an employee’s current knowledge level and the skill-set required in his job-context. A probabilistic reasoning framework is used for recommendations, to account for the inexact specifications of user competencies and requirements of the job context.

Nidhi Saraswat, Hiranmay Ghosh, Mohit Agrawal, Uma Narayanan

### Coherence Over Time: Understanding Day-to-Day Changes in Students’ Open-Ended Problem Solving Behaviors

Understanding students’ self-regulated learning (SRL) behaviors in open-ended learning environments (OELEs) is an on-going area of research. Whereas OELEs facilitate use of SRL processes, measuring them reliably is difficult. In this paper, we employ

coherence analysis

, a recently-developed approach to analyzing students’ problem solving behaviors in OELEs, to study how student behaviors change over time as they use an OELE called

Betty’s Brain

. Results show interesting patterns in students’ day-to-day transitions, and these results can be used to better understand the individual student’s characteristics and the challenges they face when learning in OELEs.

James R. Segedy, John S. Kinnebrew, Gautam Biswas

### From Learning Companions to Testing Companions

Experience with a Teachable Agent Motivates Students’ Performance on Summative Tests

In three quasi-experimental studies, we investigated the effects of placing a Teachable Agent (TA) from a math game in a digital summative test. We hypothesized that the TA would affect test performance, even without actual “teachability”, by social influence on the test situation. In Study 1 (N=47), students did a pretest, played the math game for seven weeks, and did a posttest either with or without the TA. In Study 2 (N=62), students did not play the game but were introduced to a TA directly in the posttest. In Study 3 (N=165), the game included a social chat with the TA, and the posttest offered a choice of more difficult questions. Results showed significant effects of the TA on choice and performance on conceptual math problems, though not on overall test scores. We conclude that experience with a TA can influence performance beyond interaction and informative feedback.

Björn Sjödén, Agneta Gulz

### Negotiation-Driven Learning

Negotiation mechanisms used in the current implementations of Open Learner Models are mostly position-based and provide minimal support for learners to understand why their beliefs contradict with that of the system. In this paper, we propose the paradigm of Negotiation-Driven Learning with the aim to enhance the role of negotiations in open learner models with special emphasis on affect, behavior and metacognitive abilities of the learners.

Raja M. Suleman, Riichiro Mizoguchi, Mitsuru Ikeda

### From Heterogeneous Multisource Traces to Perceptual-Gestural Sequences: the PeTra Treatment Approach

This paper presents PeTra, a framework proposed for representing and treating multi-source heterogeneous traces from simulated learning environments. We tested our proposition on traces from TELEOS, a simulation-based ITS dedicated to percutaneous orthopedic surgery. This ITS captures learners interactions from three different and independent sources. The conducted experiment demonstrated that the sequences generated by PeTra fostered efficiently: 1) the learning analytics task of evaluating the influence of visual perceptions on learners’ errors; 2) the extraction of interesting association rules potentially reusable for tutoring services production. However, its genericity has not been tested and it will need to be evaluated at a larger scale.

Ben-Manson Toussaint, Vanda Luengo, Francis Jambon, Jérôme Tonetti

### Probability Based Scaffolding System with Fading

We propose a scaffolding system that provides adaptive hints using a probabilistic model, i.e., item response theory (IRT). First, we propose an IRT for dynamic assessment, whereby learners are tested under dynamic conditions of providing a series of graded hints. We then propose a scaffolding system that presents adaptive hints to a learner according to the estimated ability of IRT from the learner response data. The system provides hints so that the learner’s correct response probability is 0.5. It decreases the number of hints (amount of support) automatically as a fading function according to the learner’s growth capability. We conducted some experiments with students. The results demonstrate that the proposed system is effective.

Maomi Ueno, Yoshimitsu Miyasawa

### Understanding Students’ Use of Code-Switching in a Learning by Teaching Technology

Personalized learning systems have shown significant learning gains when used in formal classroom teaching. Systems that use pedagogical agents for teaching have become popular, but typically their design does not account for multilingual classrooms. We investigated one such system in classrooms in the Philippines to see if and how students used code-switching when providing explanations of algebra problem solving. We found significant amounts of code-switching and explored cognitive and social factors such as explanation quality and affective valence that serve as evidence for code-switching motivations and effects. These results uncover complex social and cognitive interactions that occur during learning interactions with a virtual peer, and call for more affordances to support multilingual students.

Evelyn Yarzebinski, Amy Ogan, Ma. Mercedes T. Rodrigo, Noboru Matsuda

### Improving Learning Maps Using an Adaptive Testing System: PLACEments

Several efforts have been put forth in finding algorithms for identifying optimal learning maps for a given cognitive domain. In (Adjei, et. al. 2014), we proposed a greedy search algorithm for searching data fitting models with equally accurate predictive power as the original skill graph, but with fewer nodes/skills in the graph. In this paper we present PLACEments, an adaptive testing system, and report on how it can be used to determine the strength of the prerequisite skill relationships in a given skill graph. We also present preliminary results that show that different learning maps need to be designed for students with different knowledge levels.

Seth Akonor Adjei, Neil T. Heffernan

### Domain Module Building From Textbooks: Integrating Automatic Exercise Generation

DOM-Sortze

is a framework for the semiautomatic generation of

Domain Modules

from textbooks. It identifies not only topics and relationships between topics but also Learning Objects (e.g., definitions, examples, problem-statements) included in an electronic document.

ArikIturri

is a NLP-based system designed to automatically generate test-based exercises from corpora. To enrich the Learning Object Repository of

DOM-Sortze

with new test-based exercises, both systems have been integrated. The experiment conducted to verify the validity of the proposal is described throughout the paper.

Itziar Aldabe, Mikel Larrañaga, Montse Maritxalar, Ana Arruarte, Jon A. Elorriaga

### The Beginning of a Beautiful Friendship? Intelligent Tutoring Systems and MOOCs

A key challenge in ITS research and development is to support tutoring at scale, for example by embedding tutors in MOOCs. An obstacle to at-scale deployment is that ITS architectures tend to be complex, not easily deployed in browsers without significant server-side processing, and not easily embedded in a learning management system (LMS). We present a case study in which a widely used ITS authoring tool suite, CTAT/TutorShop, was modified so that tutors can be embedded in MOOCs. Specifically, the inner loop (the example-tracing tutor engine) was moved to the client by reimplementing it in JavaScript, and the tutors were made compatible with the LTI e-learning standard. The feasibility of this general approach to ITS/MOOC integration was demonstrated with simple tutors in an edX MOOC “Data Analytics and Learning.”

Vincent Aleven, Jonathan Sewall, Octav Popescu, Franceska Xhakaj, Dhruv Chand, Ryan Baker, Yuan Wang, George Siemens, Carolyn Rosé, Dragan Gasevic

### Predicting Misalignment Between Teachers’ and Students’ Essay Scores Using Natural Language Processing Tools

We investigated linguistic factors that relate to misalignment between students’ and teachers’ ratings of essay quality. Students (n = 126) wrote essays and rated the quality of their work. Teachers then provided their own ratings of the essays. Results revealed that students who were less accurate in their self-assessments produced essays that were more causal, contained less meaningful words, and had less argument overlap between sentences.

Laura K. Allen, Scott A. Crossley, Danielle S. McNamara

### Am I Wrong or Am I Right? Gains in Monitoring Accuracy in an Intelligent Tutoring System for Writing

We investigated whether students increased their self-assessment accuracy and essay scores over the course of an intervention with a writing strategy intelligent tutoring system, W-Pal. Results indicate that students were able to learn from W-Pal, and that the combination of strategy instruction, game-based practice, and holistic essay-based practice led to equivalent gains in self-assessment accuracy compared to heavier doses of deliberate writing practice (offering twice the amount of system feedback).

Laura K. Allen, Scott A. Crossley, Erica L. Snow, Matthew E. Jacovina, Cecile Perret, Danielle S. McNamara

### Predicting Students’ Emotions Using Machine Learning Techniques

Detecting students’ real-time emotions has numerous benefits, such as helping lecturers understand their students’ learning behaviour and to address problems like confusion and boredom, which undermine students’ engagement. One way to detect students’ emotions is through their feedback about a lecture. Detecting students’ emotions from their feedback, however, is both demanding and time-consuming. For this purpose, we looked at several models that could be used for detecting emotions from students’ feedback by training seven different machine learning techniques using real students’ feedback. The models with a single emotion performed better than those with multiple emotions. Overall, the best three models were obtained with the CNB classifier for three emotions: amused, bored and excitement.

Nabeela Altrabsheh, Mihaela Cocea, Sanaz Fallahkhair

### Using Artificial Neural Networks to Identify Learning Styles

Adaptive learning systems may be used to provide personalized content to students based on their learning styles which can improve students’ performance and satisfaction, or reduce the time to learn. Although typically questionnaires exist to identify students’ learning styles, there are several disadvantages when using such questionnaires. In order to overcome these disadvantages, research has been conducted on automatic approaches to identify learning styles. However, this line of research is still in an early stage and the accuracy levels of current approaches leave room for improvement before they can be effectively used in adaptive systems. In this paper, we introduce an approach which uses artificial neural networks to identify students’ learning styles. The approach has been evaluated with data from 75 students and found to outperform current state of the art approaches. By increasing the accuracy level of learning style identification, more accurate advice can be provided to students, either by adaptive systems or by teachers who are informed about students’ learning styles, leading to benefits for students such as higher performance, greater learning satisfaction and less time required to learn.

Jason Bernard, Ting-Wen Chang, Elvira Popescu, Sabine Graf

### Measuring Argumentation Skills with Game-Based Assessments: Evidence for Incremental Validity and Learning

Cognitive scientists and assessment developers have long been concerned with creating comprehensive, authentic measures–especially which elicit evidence of proficiency on one or more constructs under conditions of focus and engagement of test takers reflecting their true performance level. This challenge is particularly arduous for complex constructs, including 21

st

century skills, that can be highly contextualized and involve the interplay of multiple skills. The current work describes the recent development and evaluation of a game-based assessment on argumentation skills, called Mars Generation One (MGO). Our results show that the in-game process data can substantially improve the measurement of argumentation compared to non-interactive multiple-choice tests. Lastly, students’ show high levels of engagement and improve their argumentation skills during gameplay.

Maria Bertling, G. Tanner Jackson, Andreas Oranje, V. Elizabeth Owen

### Student Performance Prediction Using Collaborative Filtering Methods

This paper shows how to utilize collaborative filtering methods for student performance prediction. These methods are often used in recommender systems. The basic idea of such systems is to utilize the similarity of users based on their ratings of the items in the system. We have decided to employ these techniques in the educational environment to predict student performance. We calculate the similarity of students utilizing their study results, represented by the grades of their previously passed courses. As a real-world example we show results of the performance prediction of students who attended courses at Masaryk University. We describe the data, processing phase, evaluation, and finally the results proving the success of this approach.

Hana Bydžovská

### Steps Towards the Gamification of Collaborative Learning Scenarios Supported by Ontologies

The Computer-Support Collaborative Learning (CSCL) script is an effective approach to support meaningful interactions and better learning. Unfortunately, in some situations, scripted collaboration demotivates students, which makes more difficult its use over time. To deal with this problem, we propose the use of gamification to positively change learners’ motivation and engagement. Nevertheless, the adequate application of gamification is a complex task that requires deeper knowledge about game design and their impact on collaborative learning (CL). Thus, we develop an ontology called OntoGaCLeS to provide a formal systematization of the knowledge about gamification and its correct application. In this paper, we focus in the formalization of concepts relate to gamification as persuasive technology.

Geiser Chalco Challco, Riichiro Mizoguchi, Ig Ibert Bittencourt, Seiji Isotani

### Towards the Development of the Invention Coach: a Naturalistic Study of Teacher Guidance for an Exploratory Learning Task

We describe a study of naturalistic teacher guidance for an exploratory learning activity called Invention. Our study illustrates a specific pedagogical style, whereby the teacher offers little feedback and few explanations, but largely poses questions, to help students identify and remedy their own errors. These findings have informed the design of a computerized Invention Coach and may apply more broadly to other exploratory learning environments.

Catherine C. Chase, Jenna Marks, Deena Bernett, Melissa Bradley, Vincent Aleven

### Adaptive Representation of Digital Resources Search Results in Personal Learning Environment

The massive explosion of digital resources available in the user’s personal environment creates many issues. Users aim to select among a mass of heterogeneous digital resources, the best one to use in the activity. Traditionally, this process is time consuming and requires a lot of effort for the user to optimize selecting parameters. That often makes unexploitable digital resources available in repositories or digital libraries. In this paper, we proposed an approach that allows a user to have new ways of interpreting the resource search results. We proposed a method for adaptive visual representation of these results based on the context of use and the user profile. This approach use an adaptive tf-idf scoring and adaptive visual representation to allow relevant digital resources selection. This study was conducted as part of the design of a personal environment for consolidated digital resource management called PRISE ( PeRsonal Interactive research Smart Environment).

Daouda Sawadogo, Cyrille Suire, Ronan Champagnat, Pascal Estraillier

### Towards Investigating Performance Differences in Clinical Reasoning in a Technology Rich Learning Environment

Technology Rich Learning Environments (TREs) are increasingly used to support scholastic activities. BioWorld is an example of a TRE designed to support the metacognitive activities of learners tasked with solving virtual patient cases. The present paper aims to examine the performance differences of novice physicians in diagnosing cases in BioWorld. We present an empirically guided line of research concerning the performance differences: (1) across three endocrinology cases, (2) between genders, (3) between goal orientations, and (4) in diagnosis correctness.

Tenzin Doleck, Amanda Jarrell, Eric Poitras, Susanne Lajoie

### Emotional, Epistemic, and Neutral Feedback in AutoTutor Trialogues to Improve Reading Comprehension

We manipulated three types of short feedback (emotional, epistemic, and neutral) in an intelligent tutoring system designed to help struggling adult readers improve reading comprehension strategies. Although participants self-reported a preference for emotional feedback, there were no differences in individual motivation or usefulness ratings between emotional and epistemic feedback. Analysis from coded facial emotions indicated that participants tended to be more sensitive to epistemic feedback than emotional feedback.

Shi Feng, Janay Stewart, Danielle Clewley, Arthur C. Graesser

### Comparison of Expert Tutors Through Syntactic Analysis of Transcripts

In this paper we show that the C4.5 machine learning algorithm, applied to a number of syntactic features in transcripts, can be used to accurately differentiate between two expert human tutors. Although these tutors had taught together for years and explicitly discussed their tutoring style with one other, an analysis based on frequency of parts of speech and higher-level syntactic constructs was able to easily separate their productions.

Reva Freedman, Douglas Krieghbaum

### Exploring Through Simulation an Instructional Planner for Dynamic Open-Ended Learning Environments

Modern online courses can be characterized as

dynamic open-ended

learning environments (DOELEs). For instructional planning to work in DOELEs, an approach is needed that does not rely on data structures such as prerequisite graphs that would need to be continually rewired as the LOs change. A promising approach is collaborative filtering based on learning sequences (CFLS) using the ecological approach (EA) architecture. We developed a CFLS planner that compares a given learner’s most recent path of LOs (of length

$$b$$

) to other learners to create a neighbourhood of similar learners. The future paths (of length

$$f$$

) of these neighbours are checked and the most successful path ahead is recommended to the target learner, who then follows that path for a certain length (called

$$s$$

). An experiment with simulated learners was used to explore what are the best values of

$$b$$

,

$$f$$

and

$$s$$

. Results showed that the CFLS planner should avoid sending a learner any further ahead (

$$s$$

) than they have been matched in the past (

$$b$$

), a prediction that can be applied to the real world.

Stephanie Frost, Gord McCalla

### Modeling Self-Efficacy Across Age Groups with Automatically Tracked Facial Expression

Affect plays a central role in learning. Students’ facial expressions are key indicators of affective states and recent work has increasingly used automated facial expression tracking technologies as a method of affect detection. However, there has not been an investigation of facial expressions compared across age groups. The present study collected facial expressions of college and middle school students in the C

rystal

I

sland

game-based learning environment. Facial expressions were tracked using the Computer Expression Recognition Toolbox and models of self-efficacy for each age group highlighted differences in facial expressions. Age-specific findings such as these will inform the development of enriched affect models for broadening populations of learners using affect-sensitive learning environments.

Joseph F. Grafsgaard, Seung Y. Lee, Bradford W. Mott, Kristy Elizabeth Boyer, James C. Lester

### Adapting Feedback Types According to Students’ Affective States

Affective states play a significant role in students’ learning behaviour. Positive affective states can enhance learning, while negative ones can inhibit it. This paper describes the development of an affective state reasoner that is able to adapt the feedback type according to students’ affective states in order to evoke positive affective states and as such improve their learning experience. The reasoner relies on a dynamic Bayesian network trained with data gathered in a series of ecologically valid Wizard-of-Oz studies, where the effect of feedback on students’ affective states was investigated.

Beate Grawemeyer, Manolis Mavrikis, Wayne Holmes, Sergio Gutierrez-Santos

### Can Young People with Autism Spectrum Disorder Benefit From an Open Learner Model?

This paper describes the evaluation of Maths Island Tutor - an intelligent tutoring system for children with autism spectrum disorder (ASD). The tutor includes an open learner model (OLM). In order to benefit from this feature, the learner needs to be able to process metacognitive attributes, which can be impaired in ASD. In order to address the needs of this specific population, young people with ASD were involved in the design of the software for their use, including the OLM. A preliminary study evaluating the system demonstrated that young people with ASD did initiate access to their OLM, could correctly reproduce details from their OLM, and could also highlight the location of (study-intended) errors within their OLMs giving rise to suggestions about their abilities to remember and potentially meta-cognitively reflect on their learning.

Beate Grawemeyer, Hilary Johnson, Mark Brosnan

### Affect Matters: Exploring the Impact of Feedback During Mathematical Tasks in an Exploratory Environment

We describe a Wizard-of-Oz study that investigates the impact of different types of feedback on students’ affective states. Our results indicate the importance of matching carefully the affective state with appropriate feedback in order to help students transition into more positive states. For example when students were confused affect boosts and specific instruction seem to be effective in helping students to be in flow again. We discuss this and other effective ways to and implications for the development of our system and the field in general.

Beate Grawemeyer, Manolis Mavrikis, Wayne Holmes, Alice Hansen, Katharina Loibl, Sergio Gutiérrez-Santos

### How Do Learners Behave in Help-Seeking When Given a Choice?

We describe the results of a study that investigated learners’ help-seeking behavior using two feedback options implemented in an ITS for Java programming. The 25 students had the choice between asking for feedback on errors in their programs and feedback on possible next steps in the solution process. We hypothesized that learners’ choices would depend on correctness of their programs and their progress in problem-solving. Surprisingly, this hypothesis was not confirmed.

Sebastian Gross, Niels Pinkwart

### Modeling Children’s Pedestrian Safety Skills in an Intelligent Virtual Reality Learning Environment

This work presents an intelligent virtual reality environment for training child traffic safety. Key pedestrian skills are discussed. The overall system design is described together with a set of implemented practical exercises. An evaluation study shows that the approach is well accepted and that children struggle with the same skills in the virtual environment as in the real world.

Yecheng Gu, Sergey Sosnovsky, Carsten Ullrich

### Measuring Misconceptions Through Item Response Theory

In this paper we propose an assessment model to measure both student knowledge and misconceptions through testing. For this purpose we use a well-founded psychometric theory, i.e. the Item Response Theory (IRT). Our proposal is an extension of our previous work in this field and permits, in the same test, the data-driven evaluation of knowledge and several misconceptions, thereby more efficiently using the evidence provided by the students, while solving a test, to enrich student perturbation models.

Eduardo Guzmán, Ricardo Conejo

### No Child Behind nor Singled out? – Adaptive Instruction Combined with Inclusive Pedagogy in Early Math Software

We describe a unique play-&-learn game for early math, designed to provide adaptive instruction with respect to support and challenge as well as to cater for an inclusive pedagogy where no child, whether far behind or far ahead, is exposed as being “different”.

Magnus Haake, Layla Husain, Erik Anderberg, Agneta Gulz

### An Integrated Emotion-Aware Framework for Intelligent Tutoring Systems

This conceptual paper integrates empirical studies and existing conceptual work describing emotion regulation strategies deployed in intelligent tutoring systems and advances an integrated framework for the development and evaluation of emotion-aware systems.

Jason M. Harley, Susanne P. Lajoie, Claude Frasson, Nathan C. Hall

### Purpose and Level of Feedback in an Exploratory Learning Environment for Fractions

This paper reports on our progress on a systematic approach to operationalizing support in Fractions Lab – an exploratory learning environment for learning fractions in primary education. In particular, we focus on the question of what feedback to provide and consider in detail the implementation of feedback according to two dimensions: the purpose of the feedback, depending on the task-specific needs of the student, and the level of feedback, depending on the cognitive needs of the student. We present early findings from our design-based research that includes Wizard-of-Oz studies of the intelligent feedback system and student perception questionnaires.

Wayne Holmes, Manolis Mavrikis, Alice Hansen, Beate Grawemeyer

### Off the Beaten Path: The Impact of Adaptive Content Sequencing on Student Navigation in an Open Social Student Modeling Interface

One of the original goals of intelligent educational systems is to guide every student to the most appropriate educational content. Exploring both knowledge-based and social guidance approaches in past work, we learned that each of these approaches has weak sides. In this paper we follow the idea of combining social guidance with more traditional knowledge-based guidance to support more optimal content navigation. We proposed a greedy sequencing approach that maximizes student’s level of knowledge and tested it in a classroom. Results indicated that this approach positively impacts students’ navigation.

Roya Hosseini, I-Han Hsiao, Julio Guerra, Peter Brusilovsky

### Alleviating the Negative Effect of Up and Downvoting on Help Seeking in MOOC Discussion Forums

Through the lens of Expectancy Value Theory, we examine the effect of help giver badges, information about helper expertise, and up- and downvoting on help seeking in a MOOC discussion forum. Results show that badges alleviated the negative impact on help seeking introduced by up- and downvoting.

Iris Howley, Gaurav Tomar, Diyi Yang, Oliver Ferschke, Carolyn Penstein Rosé

### Challenges of Using Observational Data to Determine the Importance of Example Usage

Educational interventions are often evaluated with randomized control trials, which can be very expensive to conduct. One of the promises of “Big Data” in education is to use non-experimental data to discover insights. We focus on studying the impact of example usage in a Java programming tutoring system using observational data. For this, we compare different formulations of a recently proposed generalized Knowledge Tracing framework called FAST. We discover that different formulations can have the same predictive performance; yet their coefficients may have opposite signs, which may lead researchers to contradictory conclusions. We discuss implications of using fully data-driven approaches to study non-experimental data.

Yun Huang, José P. González-Brenes, Peter Brusilovsky

### Clique Algorithm to Minimize Item Exposure for Uniform Test Forms Assembly

Educational assessments occasionally require “ uniform test forms” (or parallel test forms), in which each test form consists of a different set of items, but the forms are equivalent (i.e., equivalent quality based on test information function of item response theory). However, the construction of uniforms tests often suffers bias of item exposure frequency. Ideally, the item exposure frequency should have a uniform and low distribution. For this purpose, we propose a clique algorithm for uniform test forms assembly with low item exposure. We formalize this test assembly as a searching the clique that has minimum item exposure in the maximum cliques. As the results, the proposed method utilizes the item pool more efficiently than traditional methods do. We demonstrate the effectiveness of the proposed method using simulated and actual data.

Takatoshi Ishii, Maomi Ueno

### Game Features and Individual Differences: Interactive Effects on Motivation and Performance

To optimize the benefits of game-based practice within Intelligent Tutoring Systems (ITSs), researchers examine how game features influence students’ motivation and performance. The current study examined the influence of game features and individual differences (reading ability and learning intentions) on motivation and performance. Participants (

n

= 58) viewed lesson videos in iSTART-2, an ITS designed to improve reading comprehension skills, and practiced with either a game-like activity or a minimally game-like activity. No main effects of game environment were observed. However, there was an interaction between game environment and pretest learning intentions in predicting students’ self-reported effort. The correlation between learning intentions and self-reported effort was not significant for students who practiced with the more game-like activity, whereas it was for students who practiced in the less game-like activity. We discuss the implications for this interaction and how it might drive future research.

Matthew E. Jacovina, Erica L. Snow, G. Tanner Jackson, Danielle S. McNamara

### Gamification of Online Learning

The gamification of online learning has been a subject of interest lately. This study attempts to explore two things in particular, the effects of gamification on learning and the moderating effects of user characteristics. The results demonstrate that the gamification elements contribute to higher learning outcomes while two user characteristics, agreeableness and pre-training motivation, are important moderators of the links between the gamification elements and learning outcomes. The study findings indicate that a gamified system in consideration of user characteristics is an effective means to improving the efficacy of the e-learning environment.

Jincheul Jang, Jason J. Y. Park, Mun Y. Yi

### Examining the Relationship Between Performance Feedback and Emotions in Diagnostic Reasoning: Toward a Predictive Framework for Emotional Support

The purpose of this research is to understand achievement emotions resulting from performance feedback in a medical education context where 30 first and second year medical students learned to diagnose virtual patients in an intelligent tutoring system (ITS), BioWorld. We found that students could be organized into groups using cluster analyses based on the emotions they reported after receiving performance feedback: a positive emotion cluster, negative emotion cluster, and low overall emotion cluster. Medical students in the positive achievement emotion cluster had the highest performance on the diagnostic reasoning cases; those in the negative achievement emotion cluster had the lowest performance; and students categorized as belonging to the low overall achievement emotion cluster had mean performance levels that fell between the two. From the results we propose critical performance thresholds that can be used to predict emotions following performance feedback.

Amanda Jarrell, Jason M. Harley, Susanne Lajoie, Laura Naismith

### Learning, Moment-by-Moment and Over the Long Term

The development of moment-by-moment learning graphs (MBMLGs), which plot predictions about the probability that a student learned a skill at a specific time, has already helped to improve our understanding of how student performance during the learning process relates to robust learning [1]. In this study, we extend this work to study year-end learning outcomes and to account for differences in learning on original questions and within knowledge-construction scaffolds. We discuss which quantitative features of moment-by-moment learning in these two contexts are predictive of the longer-term outcomes, and conclude with potential implications for instruction.

Yang Jiang, Ryan S. Baker, Luc Paquette, Maria San Pedro, Neil T. Heffernan

### When Is It Helpful to Restate Student Responses Within a Tutorial Dialogue System?

Tutorial dialogue systems often simulate tactics used by experienced human tutors such as restating students’ dialogue input. We investigated whether the amount of tutor restatement that supports student inference interacts with students’ incoming knowledge level in predicting how much students learn from a system. We found that students with lower incoming knowledge benefit more from an increased level of these types of restatement while students with higher incoming knowledge benefit more from a decreased level of such restatements.

Pamela Jordan, Patricia Albacete, Sandra Katz

### Quality of LOD Based Semantically Generated Questions

This research aims to automatically generate questions to support history learning. The questions are generated semantically with natural language patterns using Linked Open Data (LOD). The generated questions are designed to reinforce history learning by supporting acquisition of new information and encouraging learners to think about their knowledge. In this paper, we describe an evaluation assessing the capability of the system to generate questions of a quality high enough to support learning. The evaluation had two main results: first, the questions generated by the system cover 87% of the questions generated by humans. Second, we confirmed that the system can generate questions that enhance history thinking of the same quality as human generated questions.

Corentin Jouault, Kazuhisa Seta, Yuki Hayashi

### New Opportunities with Open Learner Models and Visual Learning Analytics

This paper compares approaches to visualising data for users in educational settings, contrasting visual learning analytics and open learner models. We consider the roots of each, and identify how each field can learn from experiences and approaches of the other, thereby benefiting both.

Judy Kay, Susan Bull

### The Relationship Between Working Memory Capacity and Students’ Behaviour in a Teachable Agent-Based Software

The current study investigated if and how students’ behaviour when using a teachable agent-based educational software were related to their working memory capacity. Thirty Swedish students aged 11–12, participated in the study. Results showed that differences in behaviour such as time spent on an off-task activity, time spent on interactive dialogues, and the number of tests that students let their TA take, were associated with differences in working memory capacity.

Lisa Palmqvist, Camilla Kirkegaard, Annika Silvervarg, Magnus Haake, Agneta Gulz

### Lesson Discovery Support Based on Generalization of Historical Events

Historical events include lessons of good and bad behaviors of human beings that can be readily applied to the modern world. To discover these lessons, one must generalize the basic attributes of multiple historical events, so that one can perceive the underlying patterns that commonly occur. This paper proposes a novel scheme for uncovering the typical patterns that emerge from multiple historical events by generalizing historical characters. We then construct a learning system that supports the generalization and discovery of common patterns based on the proposed scheme.

Tomoko Kojiri, Yusuke Nogami, Kazuhisa Seta

### Predicting Student Performance from Multiple Data Sources

The goal of this study is to (i) understand the characteristics of high-, average- and low-level performing students in a first year computer programming course, and (ii) investigate whether their performance can be predicted accurately and early enough in the semester for timely intervention. We triangulate data from three sources: submission steps and outcomes in an automatic marking system that provides instant feedback, assessment marks during the semester and student engagement with the discussion forum Piazza. We define and extract attributes characterizing student activity and performance, and discuss the distinct characteristics of the three groups. Using these attributes we built a compact decision tree classifier that is able to predict the exam mark with an accuracy of 72.69% at the end of the semester and 66.52% in the middle of the semester. We discuss the most important predictors and how such analysis can be used to improve teaching and learning.

Irena Koprinska, Joshua Stretton, Kalina Yacef

### Automated Generation of Self-Explanation Questions in Worked Examples in a Model-Based Tutor

A framework is proposed for automated generation of self-explanation questions in worked examples. In the framework, in addition to the questions, the correct answer, distracters and feedback are also automatically generated. The framework is based on model-based generation of worked examples, and is domain-dependent rather than problem-specific.

Amruth N. Kumar

### First Evaluation of the Physics Instantiation of a Problem-Solving-Based Online Learning Platform

Problem solving is a commonly used learning activity around which a large number of state-of-the-art Intelligent Tutoring Systems are developed and evaluated. In this paper, we present our problem-solving-based online learning platform and discuss a preliminary laboratory trial of this platform. While the platform itself is domain independent, for this evaluation, it was instantiated with a collection of problems from the unit of Electricity and Magnetism taught in high-school-level physics. Results indicate pedagogical effectiveness of problem solving in the Physics instance of the platform, with 41% of participants exceeding the stringent reliable change index.

Rohit Kumar, Gregory K. W. K. Chung, Ayesha Madni, Bruce Roberts

### FARMA-ALG: An Application for Error Mediation in Computer Programming Skill Acquisition

The central problem of using students’ errors as a fundamental part of teaching and learning computer programming is presented in a critical manner. The literature review points out that previous research works have not accounted for the dynamic nature of mediation in educational interactions of the referred domain. To fill such a gap, an application named FARMA-ALG is introduced. The application aims to promote error mediation in teaching and learning computer programming, with effective teacher participation. The application details and preliminary tests are shown, highlighting relevant progress to error mediation.

Alexander Robert Kutzke, Alexandre I. Direne

### The Role of Peer Agent’s Learning Competency in Trialogue-Based Reading Intelligent Systems

This paper investigates how the peer agent’s learning competency affects English learners’ reading, engagement, system self-efficacy, and attitudes toward the peer agent in a trialogue-based intelligent tutoring system (ITS). Participants learned a summarizing reading strategy in the compare-contrast text structure in the ITS. Results detected the significant main effect of the peer agent’s learning competency on learners’ performance and on engagement.

Haiying Li, Qinyu Cheng, Qiong Yu, Arthur C. Graesser

### Teaching a Complex Process: Insertion in Red Black Trees

Red black trees (and all balanced trees) are an important concept in computer science with many applications. This paper describes a new approach using an example tracing tutor and our experince in using it to teach insertion in red black trees.

C. W. Liew, F. Xhakaj

### Learner-Adaptive Pedagogical Model in SIAL, an Open-Ended Intelligent Tutoring System for First Order Logic

SIAL is an Intelligent Tutoring System (ITS) for learning Computational Logic. It teaches classical refutation by resolution concepts using Robinson’s Binary Resolution Rule. Furthermore, SIAL can be considered a Model-Based System, as its Domain Model is an Automated Theorem Prover (ATP) for First Order Logic (FOL). This allows SIAL to accept several solutions to the proposed exercise, all of which are correct, providing a kind of open-ended feature to the ITS. The Domain Model is in charge of carrying out the error diagnosis, identifying, in many cases, the misunderstandings. The focus of this paper is to describe the Pedagogical Model of SIAL that takes advantage of the error diagnosis capabilities of the Domain Model to offer a learner-adaptive tutorial action, according to the user cognitive profile.

### Predictive Knowledge Modeling in Collaborative Inquiry Learning Scenarios

The ongoing EU project Go-Lab provides a generalized interface and tool set to enable and structure learning activities with online laboratories. In this context, we have studied collaborative inquiry learning activities using various tools in blended learning scenarios. Former research indicates that the composition of heterogeneous vs. homogeneous groups in terms of student competencies or skills has an effect on the learning gain. This has been investigated using a theory-driven approach for predictive modeling based on Markov logic with data from a recent classroom study.

Sven Manske, Tobias Hecking, H. Ulrich Hoppe

### Worked Examples are More Efficient for Learning than High-Assistance Instructional Software

The ‘assistance dilemma’, an important issue in the Learning Sciences, is concerned with how much guidance or assistance should be provided to help students learn. A recent study comparing three high-assistance approaches (worked examples, tutored problems, and erroneous examples) and one low-assistance (conventional problems) approach, in a multi-session classroom experiment, showed equal learning outcomes, with worked examples being much more efficient. To rule out that the surprising lack of differences in learning outcomes was due to too much feedback across the conditions, the present follow-up experiment was conducted, in which feedback was curtailed. Yet the results in the new experiment were the same: there were no differences in learning outcomes, but worked examples were much more efficient. These two experiments suggest that there are efficiency benefits of worked example study. Yet, questions remain. For instance, why didn’t high instructional assistance benefit learning outcomes and would these results hold up in other domains?

Bruce M. McLaren, Tamara van Gog, Craig Ganoe, David Yaron, Michael Karabinos

### Domain Model for Adaptive Blended Courses on Basic Programming

Basic

Programming

is a mandatory course that covers the fundamentals of programming in Computer Engineering degrees. During the last years, the authors have experimented different approaches to improve the course. For example, they have included Lego Mindstorms robots and visual programming environments in their lectures. However, the heterogeneity of the students in the course significantly affects the course development. To overcome this problem, the next step entails the adoption of adaptive learning systems in the frame of Blended Learning (B-Learning). In this context, the OWLish generic architecture has been defined. This paper centers on the adaptation of the Domain Model of OWLish to meet the requirements of programming courses.

Mikel Larrañaga, Ainhoa Álvarez

### Learning Bayesian Networks for Student Modeling

In the last decade, there has been a growing interest in using Bayesian Networks (BN) in the student modelling problem. In order to develop a Bayesian Student Model (BSM), it is necessary to define the structure (nodes and links) and the parameters. Usually the structure can be elicited with the help of human experts (teachers), but the difficulty of the problem of parameter specification is widely recognized in this and other domains. In the work presented here we have performed a set of experiments to compare the performance of two Bayesian Student Models, whose parameters have been specified by experts and learnt from data respectively. Results show that both models are able to provide reasonable estimations for knowledge variables in the student model.

Eva Millán, Guiomar Jiménez, María-Victoria Belmonte, José-Luis Pérez-de-la-Cruz

### Tutorial Dialogue Modes in a Large Corpus of Online Tutoring Transcripts

Building on previous work in this area, we provide a description and justification for a new way of identifying modes and mode switches in tutorial dialogues, part of a coding scheme involving 16 modes and 125 distinct dialogue acts. We also present preliminary results from an analysis of 1,438 human-annotated transcripts, consisting of more than 90,000 turns. Among other findings, this analysis shows subtle differences in the “mode architecture” of successful vs. less successful sessions, as judged by expert tutors.

Donald M. Morrison, Benjamin Nye, Vasile Rus, Sarah Snyder, Jennifer Boller, Kenneth Miller

### Data-Driven Worked Examples Improve Retention and Completion in a Logic Tutor

Research shows that expert-crafted worked examples can have a positive effect on student performance. To investigate the potential for data-driven worked examples to achieve similar results, we generated worked examples for the Deep Thought logic tutor, and conducted an experiment to assess their impact on performance. Students who received data-driven worked examples were much more likely to complete the tutor, and completed the tutor in less time. This study demonstrates that worked examples, automatically generated from student data, can be used to improve student learning in tutoring systems.

Behrooz Mostafavi, Guojing Zhou, Collin Lynch, Min Chi, Tiffany Barnes

### Improving Engagement in an E-Learning Environment

Student engagement indicators, such as behavior and affective states, are known to impact learning. This study uses an established quantitative field observation method to evaluate engagement during students’ use of a new version of an online learning system (Reasoning Mind’s Genie 3). Improvements to Genie 3’s design intended to increase engagement include: using virtual small-group tutoring environment, separating text and speech, and using indicators to focus students’ attention. In this study, Genie 3 classrooms outperformed a traditional classroom on key indicators of engagement, including time on-task, engaged concentration, and boredom. These results have important implications for further improvements to Reasoning Mind, for the design of other online learning systems, and for general pedagogical practices.

Kevin Mulqueeny, Leigh A. Mingle, Victor Kostyuk, Ryan S. Baker, Jaclyn Ocumpaugh

### Using Eye Tracking to Identify Learner Differences in Example Processing

In this paper, we focus on how students with different levels of knowledge study worked examples. In order to comprehend SQL examples, the learner needs to understand the database which is used as the context. We analysed eye movements collected from a quasi experiment, and found a significant difference in the amount of attention students paid to database schemas.

Amir Shareghi Najar, Antonija Mitrovic, Kourosh Neshatian

### The Design Rationale of Logic-Muse, an ITS for Logical Reasoning in Multiple Contexts

This paper describes the design and implementation of Logic-Muse, an Intelligent Tutoring System (ITS) that helps learners develop reasoning skills on various contents. The study was conducted jointly with the active participation of logicians and reasoning psychologists. Logic-Muse’s current version was internally validated. It is focused on propositional logic and supports learners reasoning in a wide range of situations.

Roger Nkambou, Clauvice Kenfack, Serge Robert, Janie Brisson

### Evaluating the Effectiveness of Integrating Natural Language Tutoring into an Existing Adaptive Learning System

This paper reports initial results of an evaluation for an ITS that follows service-oriented principles to integrate natural language tutoring into an existing adaptive learning system for mathematics. Self-explanation tutoring dialogs were used to talk students through step-by-step worked solutions to Algebra problems. These worked solutions presented an isomorphic problem to a preceding Algebra problem that the student could not solve in an adaptive learning system. Due to crossover issues between conditions, experimental versus control condition assignment did not show significant differences in learning gains. However, strong dose-dependent learning gains were observed that could not be otherwise explained by either initial mastery or time-on-task.

Benjamin D. Nye, Alistair Windsor, Phillip Pavlik, Andrew Olney, Mustafa Hajeer, Arthur C. Graesser, Xiangen Hu

### Adapting Collaboration Dialogue in Response to Intelligent Tutoring System Feedback

To be able to provide better support for collaborative learning in Intelligent Tutoring Systems, it is important to understand how collaboration patterns change. Prior work has looked at the interdependencies between utterances and the change of dialogue over time, but it has not addressed how dialogue changes during a lesson, an analysis that allows us to investigate the adaptivity of student strategies as students gain domain knowledge. We address this question by analyzing the shift in types of collaborative talk occurring within a single session and in particular how they relate to errors for 26 4

th

and 5

th

grade dyads working on a fractions tutor. We found that, over time, the frequency of interactive talk and errors both decrease in dyads working together on conceptual problems. Although interactive talk is often held as a gold standard in collaboration, as students become more proficient, it may not be as important.

Jennifer K. Olsen, Vincent Aleven, Nikol Rummel

### The Role of Student Choice Within Adaptive Tutoring

While adaptive tutoring systems have improved classroom education through individualization, few platforms offer students preference in regard to their education. In the present study, a randomized controlled trial is used to investigate the effects of student choice within ASSISTments. A problem set featuring either text feedback or matched content video feedback was assigned to a sample of 82 middle school students. Those who were able to choose their feedback medium at the start of the assignment outperformed those who were randomly assigned a medium. Results suggest that even if feedback is not ultimately observed, students average significantly higher assignment scores after voicing a choice. Findings offer evidence for enhancing intrinsic motivation through the provision of choice within adaptive tutoring systems.

Korinn S. Ostrow, Neil T. Heffernan

### Identifying Affective Trajectories in Relation to Learning Gains During the Interaction with a Tutoring System

This paper presents the identification of sequences of affective states and its consequential impact to learning in an intelligent tutor for Mathematics at secondary level. These trajectories are represented as time series obtained by DAE 1.0 a software capable of detecting and labeling points in human faces in relation to affective states. Data was collected from students (N=44) in one secondary school, in a semirural town in Veracruz, Mexico. The students were asked to interact with the tutoring system for 40 minutes and were photographed by DAE 1.0 at a pace of 1 picture each 5 seconds. Based on a dataset consisting of 480 pictures per student, we employed the SAX algorithm to make the data discrete and facilitate the interpretation of the time series. The results of classifying the data using ID3 showed an accuracy of 62.85% in identification of affective trajectories related to higher learning gains. Future studies will seek to test this algorithm on a different data set with the aim of predicting performance towards personalizing affective interventions in the tutoring system.

### A Predictive Model of Learning Gains for a Video and Exercise Intensive Learning Environment

This work approaches the prediction of learning gains in an environment with intensive use of exercises and videos, specifically using the Khan Academy platform. We propose a linear regression model which can explain 57.4% of the learning gains variability, with the use of four variables obtained from the low level data generated by the students. We found that two of these variables are related to exercises (the proficient exercises and the average number of attempts in exercises), and one is related to both videos and exercises (the total time spent in both) related to exercises, whereas only one is related to videos.

José A. Ruipérez-Valiente, Pedro J. Muñoz-Merino, Carlos Delgado Kloos

### Integrating Learning Progressions in Unsupervised After-School Online Intelligent Tutoring

We present the design of a novel conversational intelligent tutoring system, called DeepTutor. DeepTutor is based on cognitive theories of learning, the framework of Learning Progressions proposed by the science education research community, and deep natural language and dialogue processing techniques and principles. The focus of the paper is on the role of Learning Progressions on the design of DeepTutor. Furthermore, we emphasize the role of Learning Progressions in guiding macro-adaptivity in conversational ITSs. We conducted a large-scale, after-school experiment with hundreds of high-school students using DeepTutor. Importantly, these students interacted with the system totally unsupervised, i.e. without any supervision from an instructor or experimenter. Our work so far validates the Learning Progressions theory.

Vasile Rus, Arthur Graesser, Nobal Niraula, Rajendra Banjade

### When More Intelligent Tutoring in the Form of Buggy Messages Does not Help

This paper reports on the null results from two large scale randomized controlled trials that were run in the ASSISTments online tutoring system. Both studies attempted to use reactive buggy messages to help students learn; one in the form of short 20–30 second videos and another in the form of large color-coded text. Bug messages were supplied for common wrong answers for one-step equation problems in both studies. Despite the large amount of prior research done on error analysis, both interventions using the predicted common wrong answers were unsuccessful at helping students.

Douglas Selent, Neil Heffernan

### Supporting Students’ Interactions over Case Studies

Having students analyze case studies is one of the common tutoring strategies for ill-defined domains. This paper presents a system, Umka, that supports learners as they form arguments about case studies in a professional ethics class. The system creates visualizations of the differences between students’ case positions, and suggests to students certain peers to interact with who might help to provide deeper and broader insight. The system has been run in several iterations. Experiments with the latest version confirm the benefits of the approach in stimulating productive interactions among students, and causing students to reconsider and broaden their initial positions on a case study.

Mayya Sharipova, Gordon McCalla

### A Framework for Automated Generation of Questions Based on First-Order Logic

In this work, questions are tasks posed to students to help them understand a subject, or to help educators assess their level of competency in it. Automated question generation is important today as content providers in education try to scale their efforts. In particular, MOOCs need a continuous supply of new questions in order to offer educational content to thousands of students, and to provide a fair assessment process. In this paper we establish first-order logic as a suitable formal tool to describe question scenarios, questions and answers. We apply this approach to the domain of mechanics (physics) in high school education.

Rahul Singhal, Martin Henz , Shubham Goyal

### Fine-Grained Analyses of Interpersonal Processes and Their Effect on Learning

Better conversational alignment can lead to shared understanding, changed beliefs, and increased rapport. We investigate the relationship in peer tutoring of convergence, interpersonal rapport, and student learning. We develop an approach for computational modeling of convergence by accounting for the horizontal richness and time-based dependencies that arise in non-stationary and noisy longitudinal interaction streams. Our results, which illustrate that rapport as well as convergence are significantly correlated with learning gains, provide guidelines for development of peer tutoring agents that can increase learning gains through subtle changes to improve tutor-tutee alignment.

Tanmay Sinha, Justine Cassell

### Promoting Metacognitive Awareness within a Game-Based Intelligent Tutoring System

Metacognitive awareness has been shown to be a critical skill for academic success. However, students often struggle to regulate this ability during learning tasks. The current study investigates how features designed to promote metacognitive awareness can be built into the game-based intelligent tutoring system (ITS) iSTART-2. College students (n=28) interacted with iSTART-2 for one hour, completing lesson videos and practice activities. If students’ performance fell below a minimum threshold during game-based practice, they received a pop-up that alerted them of their poor performance and were subsequently transitioned to a remedial activity. Results revealed that students’ scores in the system improved after they were transitioned (even when they did not complete the remedial activity). This suggests that the pop-up feature in iSTART-2 may indirectly promote metacognitive awareness, thus leading to increased performance. These results provide insight into the potential benefits of real-time feedback designed to promote metacognitive awareness within a game-based learning environment.

Erica L. Snow, Danielle S. McNamara, Matthew E. Jacovina, Laura K. Allen, Amy M. Johnson, Cecile A. Perret, Jianmin Dai, G. Tanner Jackson, Aaron D. Likens, Devin G. Russell, Jennifer L. Weston

### Student Performance Estimation Based on Topic Models Considering a Range of Lessons

This paper proposes a prediction framework for student performance based on comment data mining. Given the comments containing multiple topics, we seek to discover the topics that help to predict final student grades as their performance. To this end, the paper proposes methods that analyze students’ comments by two topic models: Probabilistic Latent Semantic Analysis (PLSA), and Latent Dirichlet Allocation (LDA). The methods employ Support Vector Machine (SVM) to generate prediction models of final student grades. In addition, Considering the student grades predicted in a range of lessons can deal with prediction error occurred in each lesson, and achieve further improvement of the student grade prediction.

Shaymaa E. Sorour, Kazumasa Goda, Tsunenori Mine

### Towards a Model of How Learners Process Feedback

It is well known that learners using intelligent learning environments (ILEs) make different use of the feedback provided by the ILE, exhibiting different patterns of behavior. The field of educational neuroscience offers the opportunity to study how learners process the feedback they receive in an ILE. Based on a literature review of what is known about the processing of feedback from cognitive psychology and neuroscience perspective, a model of how learners process feedback in ILEs is presented. The model represents how learners notice, process, and understand feedback. We are in the process of conducting a study to test the model. Preliminary evidence indicates that the model may be valid, but that further study must be conducted using other techniques such as eyetracking and EEG to fully validate the model.

Michael Timms, Sacha DeVelle, Ursula Schwantner, Dulce Lay

### Item Response Model with Lower Order Parameters for Peer Assessment

Peer assessment has become popular in recent years. However, in peer assessment, a problem remains that reliability depends on the rater characteristics. For this reason, some item response models that incorporate rater parameters have been proposed. However, in previous models, the parameter estimation accuracy decreases as the number of raters increases because the number of rater parameters increases drastically. To solve that problem, this article presents a proposal of a new item response model for peer assessment that incorporates rater parameters to maintain as few rater parameters as possible.

Masaki Uto, Maomi Ueno

María-Ángeles Serrano, Eduardo Vidal-Abarca, Ignacio Máñez, Carmen Candel

### Personalized Expert Skeleton Scaffolding in Concept Map Construction

Concept maps have been widely used in educational contexts to facilitate meaningful learning. Recent research has examined how concept mapping tools assist students in summarizing, relating, and organizing concepts. Our goal is to explore how personalized scaffolding can be applied to concept map construction. We provide personalized scaffolding in the form of an adaptive expert skeleton map based on student prior knowledge. We conducted a study comparing the adaptive map to a fixed map and to unscaffolded concept mapping. In an exploratory analysis, we examine the possible impacts of adaptive scaffolding on student learning processes.

Shang Wang, Erin Walker, Rishabh Chaudhry, Ruth Wylie

### Using Eye Gaze Data to Explore Student Interactions with Tutorial Dialogues in a Substep-Based Tutor

We used eye gaze data to investigate student interactions with tutorial dialogues in EER-Tutor. The results show that tutorial dialogues are effective as they enable students to correct their mistakes. However, some students do not take advantage of opportunities to reflect on what they have learnt. We identify several possible improvements to EER-Tutor, as well as future directions of work on using eye-tracking for on-line adaptation.

Amali Weerasinghe, Myse Elmadani, Antonija Mitrovic

### UML-IT: An ITS to Teach Multiple Modelling Tasks

Modelling software systems using Unified Modelling Language (UML) is a core skill expected from a software engineer. This involves modelling a software system using multiple diagrams. Students find it very difficult to develop this skill due to the open-ended nature of this task: the final outcome is defined in abstract terms but there is no well-defined procedure to achieve the outcome. Students also find it difficult to understand the different modelling conventions used to represent multiple perspectives of a particular system and the consistencies need to be maintained between these diagrams. We believe an ITS that teaches these multiple modelling tasks will be able to support learners to develop the skill of UML modeling efficiently and effectively.

Amali Weerasinghe, Bernard Evans

### Virtual Teams in Massive Open Online Courses

Previous work on MOOCs highlights both that the current MOOCs fail to provide the kind of social environment that is desired and that social interaction and exchange of support is important for slowing down attrition over time. However, little is known about how to support virtual teams in a MOOC context. In this paper, we demonstrate what factors distinguish successful and nonsuccessful virtual teams in NovoEd MOOCs, where team collaboration is an integral part of the course design. In particular, we find team leaders play a central role in determining team performance. We discuss implications for continued work towards intelligent support for team leaders in MOOCs.

Miaomiao Wen, Diyi Yang, Carolyn Penstein Rosé

### Promoting Self-regulated Learning in an Intelligent Tutoring System for Writing

The Writing Pal (W-Pal) is an intelligent tutoring system that was developed to improve students’ writing proficiency; however, it remains relatively unclear whether and how this system promotes self-regulated learning. In previous studies, we have begun to investigate the

characteristics

of students’ self-assessments, as well as whether interactions with W-Pal can lead to more accurate self-assessments and better writing performance. Here, we propose a series of three experiments that will test whether and how W-Pal can be used to enhance students’ self-regulated learning strategies and, consequently, influence their writing habits and the overall quality of their essays. These studies will examine the role of self-assessment in the writing process, and investigate how explicit instruction on writing criteria and self-regulated learning strategies can improve students’ writing processes and, ultimately, their writing proficiency.

Laura K. Allen, Danielle S. McNamara

### Exploring Missing Behaviors with Region-Level Interaction Network Coverage

We have used a complex network model of student-tutor interactions to derive high-level approaches to problem solving. We also have used interaction networks to evaluate between-group differences in student approaches, as well as for automatically producing both next-step and high-level hints. Students do not visit vertices within the networks uniformly; students from different experimental groups are expected to have different patterns of network exploration. In this work we explore the possibility of using frequency estimation to uncover locations in the network with differing amounts of student-saturation. Identification of these regions can be used to locate specific problem approaches and strategies that would be most improved by additional student-data, as well as provide a measure of confidence when comparing across networks or between groups.

Michael Eagle, Tiffany Barnes

### Educational Technologies to Support Linguistically Diverse Students, and the Challenges of Classroom Integration

Though one of the main benefits of educational technologies is their ability to provide personalized instruction, many systems are still built with a one-size-fits-all approach to culture. In our work, we’ve demonstrated that there may be learning benefits when technologies use the same non-standard dialects as students, but that educators are likely to be initially resistant to technologies that bring non-standard dialect practices into the classroom. Based on what we have uncovered about teachers’ needs and expectations regarding this type of classroom technology, our future work will investigate how systems designed to align with these needs may be able to support both students and teachers in this complex educational problem and promote a positive classroom culture.

Samantha Finkelstein

### Developing Self-regulated Learners Through an Intelligent Tutoring System

Intelligent tutoring systems have been developed to help students learn independently. However, students who are poor self-regulated learners often struggle to use these systems because they lack the skills necessary to learn independently. The field of psychology has extensively studied self-regulated learning and can provide strategies to improve learning, however few of these include the use of technology. The present proposal reviews three elements of self-regulated learning (motivational beliefs, help-seeking behavior, and meta-cognitive self-monitoring) that are essential to intelligent tutoring systems. Future research is suggested, which address each element in order to develop self-regulated learning strategies in students while they are engaged in learning mathematics within an intelligent tutoring system.

Kim Kelly, Neil Heffernan

### Building Compiler-Student Friendship

Previous studies have shown that compilers positively influence students when they are designed to build connections with students. In this paper, I propose to study the use of a friendly compiler for young novice programmers. This study involves designing compiler messages that incorporate a friendship model. The goal is to make students view compiler as a friend, instead of as an error-picking authority. I hypothesize that a good compiler-student relationship will change students’ attitude, self-efficacy and motivation towards programming, as well as change students compilation behaviors.

Zhongxiu Liu, Tiffany Barnes

### Toward Combining Individual and Collaborative Learning Within an Intelligent Tutoring System

Collaborative and individual learning appear to have complementary strengths; however, the best way to combine these learning methods is still unclear. While previous work has demonstrated the effectiveness of Intelligent Tutoring Systems (ITSs) for individual learning, collaborative learning with ITSs is much less frequent – especially for young students. In this paper, we discuss our prior and future work with elementary school students that aims to investigate how to best combine individual and collaborative learning using their complementary strengths within an ITS. Our previous findings demonstrate that ITSs are able to support collaboration, as well as individual learning, for this population. In addition, we propose future research to understand how to best combine individual and collaborative learning within an ITS.

Jennifer K. Olsen, Vincent Aleven, Nikol Rummel

### Motivating Learning in the Age of the Adaptive Tutor

My research is rooted in improving K-12 education through novel approaches to motivation and individualization via adaptive tutoring systems. In an attempt to isolate best practices within the science of learning, I conduct randomized controlled trials within ASSISTments, an online adaptive tutoring system that provides students with immediate feedback and teachers with powerful assessment. This paper examines two facets of my research: the optimization of feedback delivery and the provision of student autonomy. For each tenet, the basis for research is examined, contributions thus far are presented, and directions for future work are outlined.

Korinn Ostrow

### Creating Data-Driven Feedback for Novices in Goal-Driven Programming Projects

Programming environments that afford the creation of media-rich, goal-driven projects, such as games, stories and simulations, are effective at engaging novice users. However, the open-ended nature of these projects makes it difficult to generate ITS-style guidance for students in need of help. In domains where students produce similar, overlapping solutions, data-driven techniques can leverage the work of previous students to provide feedback. However, our data suggest that solutions to these projects have insufficient overlap to apply current data-driven methods. We propose a novel subtree-based state matching technique that will find partially overlapping solutions to generate feedback across diverse student programs. We will build a system to generate this feedback, test the technique on historical data, and evaluate the generated feedback in a study of goal-driven programming projects. If successful, this approach will provide insight into how to leverage structural similarities across complex, creative problem solutions to provide data-driven feedback for intelligent tutoring.

Thomas W. Price, Tiffany Barnes

### Towards Multimodal Affective Detection in Educational Systems Through Mining Emotional Data Sources

This paper introduces the work being carried out in an ongoing PhD research focused on the detection of the learners’ affective states by combining different available sources (from physiological sensors to keystroke analysis). Different data mining algorithms and data labeling techniques have been used generating 735 prediction models. Results so far show that predictive models on affective state detection from multimodal-based approaches provide better accuracy rates than single-based.

Sergio Salmeron-Majadas, Olga C. Santos, Jesus G. Boticario

### Promoting Metacognition Within a Game-Based Environment

Metacognition refers to students’ ability to reflect upon what they know and what they do not know. However, many students often struggle to master this regulatory skill. We have designed and implemented two features to promote metacognition within the game-based system iSTART-2. These two features have been tested and shown to have positive impacts on students’ ability to reflect upon their performance. Future work is being planned to further explore the most effective way to implement these features and the ultimate impact they have on learning outcomes. We are seeking advice and feedback on the methodology and metacognitive feature design that will be included in a series of follow-up studies. The implications of this work for both iSTART-2 and the AIED field are discussed.

Erica L. Snow, Matthew E. Jacovina, Danielle S. McNamara

### Negotiation-Driven Learning: A New Perspective of Learning Using Negotiation

Negotiation mechanisms used in the current implementations of Open Learner Models are mostly position-based and provide minimal support for learners to understand why their beliefs contradict with that of the system. In this paper, we propose the paradigm of Negotiation-Driven Learning with the aim to enhance the role of negotiations in open learner models with special emphasis on affect, behavior and metacognitive abilities of the learners.

Raja M. Suleman, Riichiro Mizoguchi, Mitsuru Ikeda

### Supporting K-5 Learners with Dialogue Systems

Interactive learning environments have been built to support various audiences from preschool to university students. However, it is not yet known how to bring the great promise of tutorial dialogue systems, which engage students in rich natural language, to bear for young learners such as those in grades K-5. This doctoral consortium paper presents our goal of developing a dialogue system in the form of an interactive spoken dialogue agent with embedded assessment to support K-5 students in learning computer science. It discusses the challenges faced so far and how we plan to solve those challenges to bring individualized dialogue systems technology to young learners.

Jennifer Tsan, Kristy Elizabeth Boyer

### Sharing Student Models That Use Machine Learning

Sharing student models has long been a problem of interest for the AIED community. Current proposals can use student models that use machine learning, but can’t modify them. We propose a multi-agent architecture for decoupled student models that enables machine learning components to: train with different data, choose what data is more reliable, and to compensate in case its sources of information are missing. The architecture uses a fragmented user model approach. The expected contributions are the architecture for sharing student models and its implementation, guidelines for decoupling or extracting implemented models from intelligent tutoring systems and intelligent learning environments, and an analysis of the portability of current state of the art student models.

Benjamin Valdes, Carlos Ramirez, Jorge Ramirez

### Backmatter

Weitere Informationen