Sentiment analysis in learning resources

Emotions affect the way we think, act and behave. In the psychological literature, there exists empirical evidence of the influence of the affective state in cognitive processes, including memorizing and decision-making (Leony et al., 2012; Cordero et al., 2020; Perozo et al., 2012). The emotional state influences the way humans learn, as claimed by Santos et al. (2014): "There is an agreement in the literature that affect influences learning". Since the emotional state is an influential factor in learning, it must be considered for enhancing the learning process. Authors of Shen et al. (2009); Aguilar et al. (2015) found evidence of the superiority of emotion-aware systems against non-emotion aware systems, which suggests that emotional data can improve the performance of e-learning systems. Several other works evidencing the advantages of emotion-aware systems in education can be found in the literature (Fatahi, 2019; Pekrun, 1992; Faria et al., 2017; Le et al., 2018; Cordero et al., 2019). Some of the main benefits in the education domain are the improvement of the teaching-learning process and students’ performance, as well as the reduction in attrition (Mite-Baidal et al., 2018).

One important aspect that can be extracted from textual contents is their emotions. This emotion can be deduced from the reviews, comments, and opinions that other persons write of the text (Garcia & Schweitzer, 2011; Ullah et al., 2016; Mohammad, 2016; Sánchez et al., 2019). The other way is to carry out a sentiment analysis in the text to identify and extract its affective states. In this work, we focus on the utilization of Machine Learning (ML) techniques for this task.

This work is interested in the emotion evoked by the text in the learning resources (sentiment analysis) because it affects the learning process of students, as has been determined in previous works (Cordero et al., 2019; Lajoie et al., 2020). Thus, this paper exploits the textual information in the digital learning resources, but also, analyzes their titles and descriptions contained in their metadata (they describe each learning resource), among other things. This information can be useful for modifying and adapting the learning process, especially in virtual learning environments (Imani & Montazer, 2019; Yadegaridehkordi et al., 2019; Alatrash et al., 2021; Sharma et al., 2021; Aguilar, 2001; Pekrun et al., 2002; Chamba-Eras & Aguilar, 2017; Aguilar, 1998).

Specifically, our aim is to predict the general or "average" perceived emotion from various readers, focusing on the common information. We do this with the objective of resembling the emotion induced by the text, assuming that the average emotion perceived by the audience is quite similar to the evoked emotion by the text. We rely on this assumption given that one way to calculate the emotion induced by content is by recognizing the emotion that it generates in several people. Then, we can obtain the similarities between them in search of a "general" emotional state that the content provokes in people. In the case of teachers, a tool based on this work is very important because, as emotions influence the learning process, they could use it as a service to determine when and in what learning activities they could rely on learning resources according to their emotional charge (Leony et al., 2012; Shen et al., 2009; Sánchez et al., 2016). For example, a learning resource that provokes negative feelings could be used by the teacher in discussion activities, with a lot of interaction, to minimize its negative impact. On the other hand, highly positively charged learning resources could be used by the teacher in individual activities, particularly when the student has negative feelings and requires motivation.

In the literature, two groups of models have been commonly used to represent emotions, categorical models and continuous models (Ekman, 1992; Cambria et al., 2015, 2018; Colombetti, 2009; Plutchik, 1980) Categorical models try to represent emotions with discrete values, such as happiness, surprise, sadness, and disgust. Categorical models start from the principle that humans have an innate set of recognizable and distinguishable basic emotions. In general, six basic, universally accepted emotions have been proposed (Ekman, 1992; Colombetti, 2009; Plutchik, 1980): happiness, anger, sadness, surprise, disgust and fear. Continuous models propose dimensions that constitute a continuous space, in which a specific emotional state is represented by a point in said space (Cambria et al., 2015, 2018; Plutchik, 1980). Thus, continuous models of emotions attempt to conceptualize human emotions in one or several dimensions. Each dimension corresponds to a variable that describes an emotion, such as valence and/or arousal or intensity. In these models, depending on the value of these variables, one or another emotion appears. Several dimensional models of emotion have been developed, some of them are the circumflex model, the vector model, and Senticnet. In this work, we use a continuous model for representing emotions: the arousal valence dimensional model. Each one of the dimensions has two opposite poles, aroused-calmed for arousal, and pleasure-disgust for valence.

The main objective of this work is to develop a method for recognizing the emotions in learning resources, which can be later used in other applications like educative recommendation systems. The main contributions of this work are: (1) propose a new method for automatic sentiment analysis in textual contents, in terms of arousal and valence and; (2) compare various feature extraction strategies for text, and ML techniques, in the task of sentiment analysis in learning resources.

The rest of the work is structured as follows: Sect. 2 explores the related works; the proposed approach for the recognition of emotions from textual contents in learning resources is detailed in Sect. 3; the next section describes the experiments. Then, Sect. 5 presents the analysis of the results, and finally, Sect. 6 presents the conclusions of the work.

In the literature, there exist many works focusing on extracting emotions from texts, and not only the extraction of emotions has been investigated, but other affective information like cognitive appraisals and sentiments have been also widely studied (Garcia & Schweitzer, 2011; Ullah et al., 2016; Mohammad, 2016). However, the sentiment analysis from formal texts has received less attention from the experts in computational intelligence. The area of sentiment analysis has focused mainly on informal textual contents like email, chat, SMS, and online user reviews.

In this section, we will analyze recent works related to the sentiment analysis in multimedia content, to strategies of identification and/or recognition of emotions, emphasizing the works dedicated to such analysis in formal texts, intensive learning environments in technology, and/or multimedia learning material.

Perceived and induced (evoked) emotions from contents have been mainly investigated for multimedia content: videos, images, presentations, music, etc. Authors of Chen and Picard (2016) propose using 3D convolutional neural networks for extracting spatio-temporal features from animated GIFs, in order to predict crowd-sourced intensity scores of 17 emotions. In Koelsch (2015), various principles underlying the evocation of emotion with music are described, including their relevance for music therapy. The investigation presented in Tian et al. (2017) explores the relationship between perceived and induced emotions from affective movie content. The main difference between those works and our work is that they recognize emotions from multimedia contents (GIFs, music and movies), and we predict emotions from the textual content, specifically, from the textual contents of learning resources.

In the context of emotion recognition, Acheampong et al. (2020) review the area of emotion detection from texts and presents the main approaches used for the design of text-based emotion detection systems. Also, emotion-labeled text data sources are presented. Batbaatar et al. (2019) propose a neural network architecture, called SENN (Semantic-Emotion Neural Network) that utilizes both semantic/syntactic and emotional information for emotion detection and recognition from text. SENN model uses two sub-networks, a first sub-network based on the bidirectional Long-Short Term Memory (BiLSTM) technique to capture contextual information and semantic relationships, and the second sub-network based on the convolutional neural network (CNN) to extract emotional features and emotional relationships between words from the text. The authors of Paltoglou et al. (2013) propose predictive models of the emotional responses for informal texts in social media using a real-valued scale on the two affective dimensions of valence and arousal. The results show that the prediction of emotional states is possible. In Rodriguez et al. (2012), Rodriguez et al. describe different approaches for automatic extraction of emotions from texts, which are essays written by students. Also, they describe its utilization in a context-based adaptive e-learning system for the dynamic recommendation of activities according to the student’s emotions. The paper (Aguilar et al., 2020) studies the semantic representation of educational digital resources. They propose an extraction mechanism of features/characteristics from the digital resources using Best Matching 25, Latent Semantic Analysis, Doc2Vec, and Latent Dirichlet allocation techniques. For this analysis, they test two types of educational digital resources: scientific publications and learning objects.

In the paper, Perikos and Hatzilygeroudis (2016), the authors describe a sentiment analysis system for automatic recognition of emotions in text, using an ensemble of classifiers. The ensemble schema is based on three classifiers; one is a knowledge-based tool performing deep analysis of the natural language sentences, and the other two are statistical (a Naïve Bayes and a Maximum Entropy learner). Also, Chatterjee et al. (2019) define a Deep Learning-based approach to detect emotions in textual dialogues. They combine both semantic and sentiment-based representations for more accurate emotion detection. On the other hand, Su et al. (2018) propose a long-short term memory (LSTM)-based approach to text emotion recognition based on semantic word vector and emotional word vector of the input text. The works in the previous two paragraphs either do not do so for learning resources, or do not do so from an emotional feature extraction perspective. We combine both aspects in this work.

In the context of formal texts, in the paper Mensink (2021) is analyzed the seductive details in scientific texts, which can potentially evoke a strong emotional response. They carried out experiments to analyse the effects when participants read scientific texts related, with and without embedded seductive details. They conclude that the presence of seductive details that produce negative emotions reduced recall performance. On the other hand, Tan and Zhang (2008) carry out a sentiment categorization on Chinese documents using different feature selection methods and machine learning methods on 1021 documents. The experimental results indicate that the quality of the classifiers is severely dependent on domains or topics. In Ahmad et al. (2020), an emotional state classification system for poetry text is proposed using Deep Learning. For this purpose, they use the C-BiLSTM model on the poetry corpus.

There is a recent special issue where the contributors address the importance of understanding and measuring emotions in technology-intensive learning environments (Lajoie et al., 2020). These works consider that emotions play an important role in fostering learning, for what analyze different domains, including mathematics, medicine, history, among others. Finally, the closer to our proposition is proposed by Moreno-Marcos et al. (2018), where they compared several ML techniques for sentiment analysis to determine the learners’ emotions when they use a MOOC (Massive Open Online Course). For that, they considered the forum messages in MOOCs as the source of information, which were studied using a sentiment analysis approach to detect complex emotions, such as frustration or boredom. The works of the two previous paragraphs show the feasibility of doing sentiment analysis in formal documents, and also, the relevance of the sentiment analysis of learning resources for virtual educational media.

Finally, several works start from the premise that the emotional design of multimedia learning material can evoke positive emotions in learners, which in turn facilitate the learning process. For example, Heidig et al. (2015) explore the potential of an emotional design of multimedia learning resources. They define an approach to deduce emotionally relevant design features, adopting concepts from web design. They consider nine conditions, classified according to two design factors (classical vs. expressive aesthetics) and a usability factor (high vs. low usability). They conclude that the perceived aesthetics and usability positively affected the emotional states of the learners, including the motivation to continue working with the resource. Newman and Joyner (2018) used a sentiment analysis tool, called VADER (Valence Aware Dictionary and sEntiment Reasoner), to analyze the positivity/negativity of comments of the students about a course using three sources: a forum with comments of the course, an unofficial “reviews” site of the students and the official evaluations. Too, Plass and Kaplan (2016) propose a model integrating cognitive and affective aspects of learning with digital media for making designs of multimedia learning. Their model combines elements of the affective computing paradigm and the cognitive-affective theory, among other elements. On the other hand, Peng et al. (2021) investigate how the visual aesthetics of interface design influenced learners’ cognitive processes and emotional valences. As well, Rodrigues and Silva (2022) review the articles about emotional design in multimedia learning, considering several aspects like emotional design features most frequently found, effects of emotional design on the learning process, among others. Finally, the paper Stark et al. (2018) proposes a method based on eye-tracking for the emotional design in multimedia learning for textual parts of multimedia resources. In their experiments, the positive emotional text design led to better learning outcomes. Further, learners in the group with the negative emotional text design showed a worse emotional state after learning.

As we can see previously, there is an important number of works dedicated to the analysis of emotions in texts, particularly in informal texts. Additionally, various studies have shown the affective impact of multimedia learning. Particularly, the multimedia didactic material can evoke positive emotions in the learners, which improves their learning processes. Despite the investigations for multimedia contents, or on the analysis of sentiment in the educational context, studies for extracting emotions in learning resources were not found. Thus, there are no studies that have been dedicated exclusively to the analysis of the emotions evoked by the textual content in learning resources. Only the work of Moreno-Marcos et al. (2018) is close to our proposal, but they do not propose a methodological scheme, or a strategy for the extraction of characteristics/variables, to be able to determine the emotions in the learning resources. Our research is innovative because it points to a goal that has not been explored in the literature: the emotions extraction in learning resources using a feature engineering approach Pacheco et al. (2014). This is the main contribution of our work, which could later be used by recommender systems based on emotions Monsalve-Pulido et al. (2020); Salazar et al. (2021). With this work, we aim to open the door to the investigation of this topic, showing the evidence that the extraction of this kind of affective information can be performed with high accuracy.

Our approach has two main steps: In the first step are automatically labeled the learning resources, and in the second are defined the sentiment analysis models for learning resources. These steps have sub-steps, as can be observed in Figure 1, where the first step is in blue and the second in yellow.

This section provides details of the experiments performed. It is divided in the two steps presented in the previous section.

This work presents a method for automatic sentiment analysis in learning resources. Additionally, it compares various feature extraction strategies and ML techniques in the task of sentiment analysis from textual data in learning resources.

The automatic annotation of emotional states presents distributions similar to the original annotations, demonstrating good consistency and excellent similarity to the true annotations. Thus, the automatic annotation of emotional states presents the same nature as the original dataset distribution, which is biased to the center of the arousal-valence space. With higher quality data, a more adequate predictor can be built. Also, the proposal for the sentiment analysis in learning resources is promising, the results are good, reflecting the viability of the approach. They demonstrate that is possible to extract the emotions in the textual information in learning resources, with very good performance.

The proposed process for extracting the emotions using these ML techniques is very domain-sensitive. When applying the learned models to other domains, metrics go to the ground, reflecting the great attachment of these models to the domain of the data with which they were trained, showing low generalization towards other domains. This is a common problem with ML techniques, various of them suffer from low generalization when testing with data of different nature Aguilar et al. (2020). Despite that, it was found that the RF technique overperformed PLS for this task. Also, there was not a dominant feature set between the four analyzed, the results showed great dependence on the ML technique used for training.

A tool of this type is very useful in a smart classroom, in the management of a learning process by a teacher, since it allows improving the management of learning resources, being aware of emotions (Shen et al., 2009; Pekrun, 1992; Lajoie et al., 2020). For example, a learning object recommendation system can adapt to the emotional context (Leony et al., 2012; Santos et al., 2014). We have considered the different types of learning resources contained in the Merlot repository (videos, textbooks, videogames, etc.) but we have not considered scientific works, patents, etc., that may be useful in learning processes (Aguilar et al., 2020; Heidig et al., 2015). Also, we have considered the title and description of the learning resources established in the metadata of the IEEE-LOM standard, but it should be analyzed that other metadata may be useful to improve our approach (Aguilar et al., 2020; Heidig et al., 2015). Finally, two machine learning techniques were tested as sentiment analysis methods.

Among the most relevant limitations of this work are that the emotional annotation of learning resources is required, a very arduous task due to the large number of resources of this type that exist. Another limitation is that it used a specific continuous emotional model; its behavior must be analyzed for other models, even discrete ones, to see if they improve its performance. Also, the extraction of meta-characteristics from the learning resources in Merlot involved developing a specific script, which should be done for each learning resource repository to be used. Finally, a case study in a real learning environment is required using a tool such as the one proposed in this work, to analyze its effect on the emotional behavior of students and their learning processes.

For future works, more complex ML models, such as deep neural networks, can be explored for obtaining adequate generalization between textual domains. Another future work is the utilization of other feature sets, and the definition of a feature selection process for solving the problem of fusing the feature sets, which showed worse results than considering the feature sets single.