Emotions as implicit feedback for adapting difficulty in tutoring systems based on reinforcement learning

In tutoring systems, a pedagogical policy, which decides the next action for the tutor to take, is important because it determines how well students will learn. An effective pedagogical policy must adapt its actions according to the student’s features, such as knowledge, error patterns, and emotions. For adapting difficulty, it is common to consider student knowledge but not the other features as emotions. Reinforcement learning (RL), which is a machine learning framework, fits well for adapting to difficulty; however, the known ways of considering emotions into RL like through states or reward-shaping functions are not enough. Then, to find the pedagogical policy that maximizes the student learning gain, we propose considering emotions as implicit feedback through both the reward and the exploration–exploitation strategy, using the circumplex model to represent emotions and the flow theory to select the appropriate difficulty level. Our approach follows three design considerations: pursuing positive emotions, managing unwanted (anxiety and boredom) emotions, and anticipating unwanted emotions. We simulate interactions with users based on real data from publicly available datasets to quantitatively compare our approach with others that adapt difficulty. Also, we qualitatively compare our approach with others that consider emotions in different contexts. Quantitative results show that our approach is better than the others that adapt difficulty to foster learning gain in students because it allows getting higher values all the studied time (200 tasks). Qualitative comparisons show that although other approaches pursue positive emotions or manage unwanted emotions, our approach does so as well and additionally anticipates unwanted emotions. We conclude that our approach is useful in tutoring systems for adapting difficulty because it allows high learning gains in students in a few interactions.