Joseph F. Grafsgaard
Eric N. Wiebe
ABSTRACT
Detecting learning-centered affective states is difficult, yet crucial for adapting most effectively to users. Within tutoring in particular, the combined context of student task actions and tutorial dialogue shape the student's affective experience. As we move toward detecting affect, we may also supplement the task and dialogue streams with rich sensor data. In a study of introductory computer programming tutoring, human tutors communicated with students through a text-based interface. Automated approaches were leveraged to annotate dialogue, task actions, facial movements, postural positions, and hand-to-face gestures. These dialogue, nonverbal behavior, and task action input streams were then used to predict retrospective student self-reports of engagement and frustration, as well as pretest/posttest learning gains. The results show that the combined set of multimodal features is most predictive, indicating an additive effect. Additionally, the findings demonstrate that the role of nonverbal behavior may depend on the dialogue and task context in which it occurs. This line of research identifies contextual and behavioral cues that may be leveraged in future adaptive multimodal systems.
- Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K. and Christopherson, R.M. 2009. Emotion Sensors Go To School. 14th International Conference on Artificial Intelligence in Education, 17--24. Google Scholar
Digital Library
- Cooper, D.G., Muldner, K., Arroyo, I., Woolf, B.P. and Burleson, W. 2010. Ranking Feature Sets for Emotion Models used in Classroom Based Intelligent Tutoring Systems. Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization, 135--146. Google ScholarDigital Library
- D'Mello, S.K. and Graesser, A.C. 2010. Multimodal Semiautomated Affect Detection From Conversational Cues, Gross Body Language, and Facial Features. User Modeling and UserAdapted Interaction. 20, 2, 147--187. Google ScholarDigital Library
- D'Mello, S.K. and Kory, J. 2012. Consistent but Modest: A Meta-Analysis on Unimodal and Multimodal Affect Detection Accuracies from 30 Studies. Proceedings of the 14th ACM International Conference on Multimodal Interaction, 31--38. Google ScholarDigital Library
- D'Mello, S.K., Lehman, B., Pekrun, R. and Graesser, A.C. 2014. Confusion Can Be Beneficial for Learning. Learning & Instruction. 29, 153--170.Google ScholarCross Ref
- Ekman, P., Friesen, W. V. and Hager, J.C. 2002. Facial Action Coding System. A Human Face.Google Scholar
- Grafsgaard, J.F., Fulton, R.M., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2012. Multimodal Analysis of the Implicit Affective Channel in Computer-Mediated Textual Communication. Proceedings of the 14th ACM International Conference on Multimodal Interaction, 145--152. Google ScholarDigital Library
- Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2013. Automatically Recognizing Facial Expression: Predicting Engagement and Frustration. Proceedings of the 6th International Conference on Educational Data Mining, 43--50.Google Scholar
- Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N. and Lester, J.C. 2013. Automatically Recognizing Facial Indicators of Frustration: A Learning-Centric Analysis. Proceedings of the 5th International Conference on Affective Computing and Intelligent Interaction, 159--165. Google ScholarDigital Library
- Harrigan, J.A. and O'Connell, D.M. 1996. How Do You Look When Feeling Anxious? Facial Displays of Anxiety. Personality and Individual Differences. 21, 2, 205--212.Google ScholarCross Ref
- Hart, S.G. and Staveland, L.E. 1988. Development of NASATLX (Task Load Index): Results of Empirical and Theoretical Research. Human Mental Workload. P.A. Hancock and N. Meshkati, eds. Elsevier Science. 139--183.Google Scholar
- Joshi, J., Goecke, R., Parker, G. and Breakspear, M. 2013. Can Body Expressions Contribute to Automatic Depression Analysis? Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1--7.Google Scholar
- Kapoor, A. and Picard, R.W. 2005. Multimodal Affect Recognition in Learning Environments. Proceedings of the 13th Annual ACM International Conference on Multimedia, 677--682. Google ScholarDigital Library
- Littlewort, G., Bartlett, M.S., Salamanca, L.P. and Reilly, J. 2011. Automated Measurement of Children's Facial Expressions during Problem Solving Tasks. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 30--35.Google Scholar
- Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J.R. and Bartlett, M.S. 2011. The Computer Expression Recognition Toolbox (CERT). Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 298--305.Google Scholar
- Marx, J.D. and Cummings, K. 2007. Normalized Change. American Journal of Physics. 75, 1, 87--91.Google ScholarCross Ref
- O'Brien, H.L. and Toms, E.G. 2010. The Development and Evaluation of a Survey to Measure User Engagement. Journal of the American Society for Information Science and Technology. 61, 1, 50--69. Google ScholarDigital Library
- Scherer, S., Stratou, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A. and Morency, L.-P. 2013. Automatic Behavior Descriptors for Psychological Disorder Analysis. Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 1--8.Google Scholar
- Symonds, M.R.E. and Moussalli, A. 2010. A Brief Guide to Model Selection, Multimodel Inference and Model Averaging in Behavioural Ecology using Akaike's Information Criterion. Behavioral Ecology and Sociobiology. 65, 1, 13--21.Google ScholarCross Ref
- Vail, A.K. and Boyer, K.E. 2014. Adapting to Personality Over Time: Examining the Effectiveness of Dialogue Policy Progressions in Task-Oriented Interaction. Proceedings of the 15th Annual SIGDIAL Meeting on Discourse and Dialogue, 41--50.Google Scholar
- Vail, A.K. and Boyer, K.E. 2014. Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes. Proceedings of the 12th International Conference on Intelligent Tutoring Systems, 199--209.Google Scholar
- VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A. and Rosé, C.P. 2007. When Are Tutorial Dialogues More Effective Than Reading? Cognitive Science. 31, 1, 3--62.Google ScholarCross Ref
- Wiebe, E.N., Lamb, A., Hardy, M. and Sharek, D. 2014. Measuring Engagement in Video Game-based Environments: Investigation of the User Engagement Scale. Computers in Human Behavior. 32, 123--132. Google ScholarDigital Library
- Woolf, B.P., Burleson, W., Arroyo, I., Dragon, T., Cooper, D.G. and Picard, R.W. 2009. Affect-Aware Tutors: Recognising and Responding to Student Affect. International Journal of Learning Technology. 4, 3--4, 129--164. Google ScholarDigital Library
- Wu, T., Butko, N.J., Ruvolo, P., Whitehill, J., Bartlett, M.S. and Movellan, J.R. 2012. Multi-Layer Architectures for Facial Action Unit Recognition. IEEE Transactions on Systems, Man and Cybernetics, Part B: Cybernetics. 42, 4, 1027--1038.Google ScholarDigital Library
Index Terms
- The Additive Value of Multimodal Features for Predicting Engagement, Frustration, and Learning during Tutoring
Recommendations
Predicting Learning and Engagement in Tutorial Dialogue: A Personality-Based Model
ICMI '14: Proceedings of the 16th International Conference on Multimodal InteractionA variety of studies have established that users with different personality profiles exhibit different patterns of behavior when interacting with a system. Although patterns of behavior have been successfully used to predict cognitive and affective ...
Multimodal Analysis and Modeling of Nonverbal Behaviors during Tutoring
ICMI '14: Proceedings of the 16th International Conference on Multimodal InteractionDetecting learning-centered affective states is difficult, yet crucial for adapting most effectively to users. Within tutoring in particular, the combined context of student task actions and tutorial dialogue shape the student's affective experience. As ...
Multimodal analysis of the implicit affective channel in computer-mediated textual communication
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interactionComputer-mediated textual communication has become ubiquitous in recent years. Compared to face-to-face interactions, there is decreased bandwidth in affective information, yet studies show that interactions in this medium still produce rich and ...
Comments