Impact of students evaluation of teaching: a text analysis of the teachers qualities by gender

This study proposes the Educational Process and Data Mining (EPDM) model to demonstrate the capabilities of text mining and its related technologies within the educational domain. The main research questions are as follows:

Accordingly, the study develops a set of constructs and uses it to perform the investigations and analyses as follows:

Based on the stated research questions and effort to provide answers to the identified objectives, this study makes the following contributions to knowledge:

The remainder of this paper is structured as follows: The Background Information section discusses the relevant state-of-the-art in this topic area, especially as it relates to analyzing SET outcomes by gender of the teachers. The Methodology section introduces the proposed method, the EDPM model and its main components, the participants’ information, instruments, data analyses, and case study implementation. In the Discussion section, this work covers the outcomes and the impact of the EDPM method, and then concludes and points out the direction for future studies in the Conclusion section.

Currently, existing research has increasingly questioned the impact and validity of SET as an indicator of teachers’ performance and assessment (Boex, 2000; Boring, 2017; Drowling, 2000; Rivera & Tilcsik, 2019). For example, Rivera and Tilcsik (2019) note that researchers have paid little attention to the architecture of assessment and how biased evaluations play a primary role in sustaining gender inequalities. Boring (2017) observes that SET recently has not just been used to assess teachers’ performance but has also led to other studies related to teaching dimensions or qualities that students value in their teachers. For instance, examining them in terms of gender stereotypes. Perhaps, most of the scientific research has focused on SET and its impact on educational systems due to an increase in the rate at which program data is collected over the academic semesters (Bianchini et al., 2013). Indeed, there is a need to explore the potential of the information from the SET datasets for decision-making purposes. Along these lines, we show that analyzing the textual data in SET allows us to identify the students’ learning expectations, to translate the speech or text into meaningful insights about the teaching-learning processes, and to discover patterns and correlation of words. All of these, in turn, can provide the ability to make effective educational-process-related decisions (e.g., monitoring of the teaching-learning process, and content recommendation, etc).

Moreover, contemporary studies provide new evidence on gender bias in teaching evaluations (Boring, 2017; Heath et al., 2019; Islahi & Nasreen, 2013; Kafedžić, Maleč, & Nikšić, 2018; Laube, Massoni, Sprague, & Ferber, 2007; Mengel, Sauermann, & Zölitz, 2019; Rivera & Tilcsik, 2019; Sabbe & Aelterman, 2007). In those studies and the different contexts, we note several assumptions about differences in teaching qualities and performances of the male and female teachers. For example, Sabbe and Aelterman (2007) note that male and female teachers are often presumed to differ in teaching styles, capacities, and effects considering the students’ perspectives and teaching as a career. A majority of the studies argue that females receive systematically lower teaching evaluations than their male counterparts (Boring, 2017; Mengel et al., 2019). Boring (2017) notes that male teachers are perceived by both male and female learners as being more knowledgeable and having more persuasive class leadership skills, even though the students appear to learn more from the females than from male teachers.

Interestingly, this observation aligns with the results and findings of our study, as discussed in detail in the Results and Discussion section. Furthermore, Kafedžić et al. (2018) measure competencies related to teaching skills for the different genders. Islahi and Nasreen (2013) note the need for both substantive and methodological focus on gender-specific responsibilities and requirements. Such methodologies would enable stakeholders like educators, researchers, and learners to benefit from a more effective instruction by both male and female teachers. Likewise, Rivera and Tilcsik (2019) highlight that minor technical performance and aspects of the teachers can have a high impact on teacher evaluations when analyzed by gender.

On the other hand, Laube et al. (2007) note that a gender-view of SET is an artifact in itself and further state that quantitative measures can only mask underlying gender bias. We note that most existing literature has focused on quantitative analysis. Few studies emphasize quantifiable linguistic methods to determine the differences between the male and female teachers through the analysis of free-text comments (Heath et al., 2019) such as the method introduced in this current study.

Text mining methods, including sentiment analysis and opinion mining, are theoretical and technological accomplishments of the last decade that are closely related to natural language processing (NLP) (Pandey & Pandey, 2019), otherwise referred to as language technologies (Wen, Yang, & Rosé, 2014). The benefits of such methods and analyses can be understood through the different views or patterns that are contained in the databases of the organizations in question (Binali, Wu, & Potdar, 2009; El-Halees, 2011; Jones, 2019; Tur, Marín, & Carpenter, 2017). Interestingly, Wen et al. (2014) note that text mining can be adopted to reveal relationships between the actual processes (the learning processes in social networks, government, businesses, for example) and the intended stakeholders. The reactions or intentions of the stakeholders are seen in teacher-student interactions, the mood of Twitter tweets, political opinions, and even stock market fluctuations, etc. (Bollen, Mao, & Zeng, 2011).

For example, the experiments conducted by Wen et al. (2014) showed that the forums and online discussion platforms for students in the massive open online courses (MOOCs) could provide ample opportunity to engage learners, even in social processes (Brinton et al., 2014; Crues et al., 2018). For this purpose, they proposed a survival modeling approach to analyze the impact of students’ opinions over time in MOOCs. The model was envisioned as a predictive and monitoring vehicle for determining the effectiveness of certain language behaviors or human expressions on the probability of an event occurring. Similarly, our study shows that through automated modeling and processing of the SET datasets, text mining is capable of providing new and vital information about the patterns or expectations of the undergraduate students for improvement. Besides, there may not even be the necessity to rely on the results of the performance assessments or the post-course evaluations.

There has been a lot of progress in the current literature about using data obtained from the educational settings to improve the intended processes (Wang & Zhu, 2019; Abu Zohair, 2019, Exter et al., 2019; Munro, 2018; Dommett, Gardner, & van Tilburg, 2019; Alizadeh, Mehran, Koguchi, & Takemura, 2019). It is noteworthy that existing research has demonstrated that HEIs are increasingly data-hungry, with the stakeholders (students, teachers, and administrators, for example) seeking data and study results for their own needs and use (Clark, 2015; Williamson, 2018). According to Williamson (2018), the Data Future program will pave the way for new modes of quantification and standardization. Indeed, programs that are allied to business intelligence systems, adaptive learning tools and environments, educational/learning analytics which include several “smart learning tools” and dashboards can be utilized by educators to support the different decision-making strategies and processes (Perrotta & Williamson, 2018). Moreover, nowadays, datasets are increasingly being captured about the teaching-learning process, promoting the advancement of educational data mining techniques. These, in turn, have led to what HEIs today could refer to as technology-focused quality education. For example, Daniel (2015) notes that digitalization of the educational processes and services has led to significant innovations and has paved the way for the new notion of datafication. Meanwhile, Williamson (2018) notes that HEIs have experienced transformations in the past as new forms of digital data were being generated, analyzed, and used to support their decision-making processes. In other words, datafication plays a vital role in understanding and enabling development within the broad context of the higher education sectors.

As presented in this study, one of the main areas where datafication and its associated technologies (such as the text mining technique) are currently being applied and are gaining attention in recent years is in the educational innovation teaching models. The new technologies are useful during the implementation of modern educational models because they can leverage methodologies that develop disciplinary and transversal competencies. Examples include challenge-based learning, memorable university experience, and inspiring professors. The technologies provide flexibility as to how, when, and where learning occurs to achieve the goals of HEI (TEC, 2018).

This work shows that the EPDM model can help provide innovative practices within the HEI setting that range from the adoption of intelligent methodologies to the transformation of students’ learning experiences by enabling the so-called three-dimensional expressive-communication-relational skills. The series of experiments and analyses presented in this paper is useful for improvement of the teaching practices that students value, primarily because it addresses gender stereotypes and teacher-student engagement (Crues et al., 2018; Sánchez et al., 2019). The model we present is a Competency-Based Education (CBE) model (UNESCO, 2015; Yadav & Berges, 2019), and it is adequate and effective in supporting the new educational process initiatives and practices (TEC, 2018).

Several institutions have had to integrate emerging educational technologies in their different activities to maintain or sustain the credibility and quality of their business models or operations in general (Kori, Pedaste, & Must, 2018). Therefore, recent research (Wang & Zhu, 2019; Abu Zohair, 2019, Exter et al., 2019; Munro, 2018; Dommett et al., 2019; Alizadeh et al., 2019; Kori et al., 2018) has focused on overcoming some of the challenges within the educational processes to provide opportunities for innovative teaching and learning methods (e.g., through educational data mining).

Typically, Wang and Zhu (2019) compared the learning performances of users in a MOOC-based, flipped classroom with those in a traditional class. They observed that adopting advanced network technologies for teaching can support the transformation and pave the way for in-depth construction of high-quality education (Crues et al., 2018). Also, Abu Zohair (2019) notes that accurate data analysis has not only become essential for improving the students’ performance and experience (Crues et al., 2018; Kori et al., 2018; Weston, Dubow, & Kaminsky, 2019) but it also serves as a mechanism for elevating the university’s ranking and reputation. The results of the study by Abu Zohair (2019) show that support vector machines or algorithms can be used to train and analyze the readily available datasets to produce acceptable learning-process-classification accuracy and reliability test rates (Liao et al., 2019).

Accordingly, Mayer-Schönberger and Cukier (2014) note that the datafication of learning, i.e., the integration of information technologies and education, has opened new innovative methods for HEIs and brought about modern educational models. Along these lines, Jones (2019) shows that by tracking, aggregating, and analyzing the student profiles and behaviors recorded in the information systems or databases (Kori et al., 2018), the universities are adopting modern educational methods or strategies. The technologies employed include educational data mining, machine learning, and learning analytics technologies. Moreover, those new methods are useful in the evaluation of the quality of educational models and processes (Alizadeh et al., 2019; Dommett et al., 2019; Munro, 2018). For instance, Dommett et al. (2019) found that the teachers and students quite often have different perspectives about the teaching-learning process. They argue that information technologies could potentially serve as the bridge from observed behaviors to a useful educational model. Nonetheless, existing studies have shown that the students, educators, and administrators have benefited significantly through the use of information technologies to predict academic achievements. These studies include views of the students’ perceptions of learning and the mediating roles of academic engagement, including digital readiness within the HEIs. (Exter, Caskurlu, & Fernandez, 2018; Kim, Hong, & Song, 2019; Kori et al., 2018; Sánchez et al., 2019).

Early research has examined how best to apply the educational models or strategies to provide students with productive and positive learning experiences (Altrabsheh, 2016; Crues et al., 2018; Mackness, Mak, & Williams, 2010; Wen et al., 2014). On the one hand, Altrabsheh (2016) shows that there is a need to explore the best method to visualize and understand the performance and results of the teaching-learning processes. On the other hand, Wen et al. (2014) investigate what sentiment analysis can tell us about the students’ opinions with regards to the learning experiences (Crues et al., 2018). Notably, Moshinskie (2001) states that it is essential to monitor the students’ feelings or emotions because learners with a positive attitude have been seen as more confident and motivated to learn (Weston et al., 2019). Besides, considering the students as consumers of the educational systems and curricula is one of the main reasons to use digital learning strategies as a way to personalize learning experiences (Munro, 2018; Romero, Saucedo, Caliusco, & Gutiérrez, 2019).

Nowadays, text mining techniques are being applied within the educational sector, allowing researchers to analyze textual data such as formal documents, e-mails, chat messages, digital notes, and online discussions for teaching-learning improvement and decision-making purposes. For example, Kumakawa (2017) notes that to exploit the goldmine of educational data and its use to understand better how the students progress with learning, researchers should essentially employ data mining techniques.

Figure 1 shows the Educational Process and Data Mining model (EPDM) proposed in this paper. Likewise, Altrabsheh (2016) observes that most of the existing works within the educational domain have only focused on sentiment analysis in the broad spectrum of e-learning (Kechaou, Ammar, & Alimi, 2011; Ortigosa, Martín, & Carro, 2014; Song, Lin, & Yang, 2007; Troussas, Virvou, Espinosa, Llaguno, & Caro, 2013), with fewer studies being carried out on feedback or SET results (Munezero, Montero, Mozgovoy, & Sutinen, 2013). Interestingly, Ortigosa et al. (2014) studied the polarization of emotions in social networks in the context of e-learning: The positive classifications had good or optimistic feelings, whereas the negative classes were considered bad (aversive).

Thus far, one of the main contributions of the EPDM model is the text and sentiment analysis of comments given in SET. This study introduces the EPDM model for determining and analyzing the underlying sentiments (i.e., emotional valence) of the students based on their comments in SET, as demonstrated in the subsequent sections of this paper.

Figure 1 illustrates the architecture of the EPDM model that this study introduces for the text analysis of the educational data and case study implementation.

The EPDM model uses data from the educational domain and defines a method that allows for contextual-based information analysis, which is useful for improving the teaching-learning processes. The model is grounded on the following main components and design framework:

This study uses data from the Student Opinion Surveys to carry out the experimentations and practical implementation of the EPDM model. ECOA is a system designed for the collection of information about students’ opinions about the course outcomes including the teaching and academic services offered throughout the various campuses of the host institution where this research took place. Specifically, we analyzed the data collected by ECOA from the undergraduates about their teachers’ performances through a three-semester period (Semester Jan-May 2017, Semester Aug-Dec 2017, and Semester Jan-May 2018). The survey was carried out across six different campuses of the North Region of the host country, Mexico; covering around 14 Divisions or Schools, 78 Departments, and 1082 Courses. Therefore, a wide variety of students’ views and opinions were represented. Although the ECOA contains some quantitative SET data, for this work, we focused on the comments provided by the students in response to the question, “Why would you recommend or not recommend the teacher,” which was also asked in the SET. To get practical, unbiased evaluations, ECOA does not record the names or the gender of the participants. Therefore, the dataset includes the gender of the teachers but not the gender of the participating students. From the ethical point of view, the students who provided the comments were informed about the purpose of the questions. However, they did not directly participate in the analyses performed in this study. The comments were provided by the students at the end of their respective courses when completing the ECOA survey.

Statistically, the dataset analyzed in this paper includes a total sample of n = 93,915 participants in the survey. However, after narrowing the data by filtering out the students who commented in the data, we noted a sample of n₁ = 52,218 comments for male teachers and n₂ = 41,072 for female teachers. However, this work uses the first 41,072 comments for the male teachers and the n₂ = 41,072 for the females to conduct its analysis. This was done to distribute the data equally based on teachers’ gender (variables) and for comparison purposes and validation of the results. In summary, considering the gender of the teachers and the students who answered the survey, we used a total sample of n_t = 82,144 comments (i.e., n₁ = 41,072 for male teachers, and n₂ = 41,072 for female teachers) throughout the analysis and experiments presented in this paper.

This study implements the EPDM model by first determining the frequency of each of the top twenty qualities that the students perceived as necessary in the evaluation of the teachers’ performance across the data (ECOA). Technically, the sets of qualities were analyzed in terms of male and female teachers by determining the different clusters of words, frequencies, and correlation of words. The word frequencies were presented in highest to lowest order. The word correlations were determined by measuring the probability of the students using the associated terms in the data. The measurements fall between 0 and 1, representing 0% to 100%. The results are shown in Figs. 2 and 3 and Table 4 in appendix.

® Frequencies and word correlations are reported in Table 4 and Appendix.

The EPDM model applied to the comments of the students allowed us to determine the top twenty qualities that the students considered necessary when evaluating the teachers’ performance. In Fig. 2, we list them in two Word Clouds (English and Spanish), classified according to the gender of the teachers. In Fig. 3, we plot them in a bar chart. It is noteworthy that 19 of those terms are similar, and only one was different for the genders. Henceforth, “problems” only appear differently for males, while “patience” only appears for females. Whereas, “problems” can be described as tools or mechanisms presented by the teachers to improve and evaluate the learning processes, “patience” is a term related to the personal abilities and performance of good teachers. Table 4 in appendix reports the word frequency in decreasing order, along with their correlations. The most mentioned terms are “conocimiento” (knowledge) and explicar (to explain); the performance of the teachers associated with these terms are dependent statistically by gender. “knowledge” is the most mentioned for male teachers, while “explain” is the most indicated for the females.

As previously mentioned, the total number of comments evaluated were 41,072 for each gender. Thus, Table 4 and the Appendix show the word “knowledge” appearing in 17.9% of the comments associated with male teachers and 15.4% of those with female teachers. The term “explain” appears in 16.8% of female teachers’ comments and 17.0% of the male teachers. Interestingly, even though the word “experience” appears in both lists, for male teachers, it occupies the 9th place, and for female teachers, it holds the 20th place.

The outcome of the analysis of the comments shows that the students in the broad spectrum consider particular teachers’ qualities as essential, independent of the SET scores. Although those qualities seem to be closely related in both genders, the frequency of use or the word correlations vary accordingly. In Fig. 2 and Table 4 in appendix, we note a plethora of essential words that were used by the students to describe the qualities of the teachers. Table 4 in appendix also reports the correlation of words, which is the probability (or likelihood) of students using the associated words in their comments. Consistently, for teachers of both genders, the word “conocimiento” (knowledge) appears in phrases like “transmits knowledge,” “wide knowledge,” and “a lot of knowledge.” The word “explicar” (infinitive of explain) appears in phrases expressing that the teacher makes something clear or more evident. Moreover, the first row in Table 4 in appendix shows that such phrases correlate significantly with the highest-rated word for male teachers, “conocimiento” (knowledge). This denotes that the students, on the one hand, consider the male teachers as more knowledgeable or intellectually deeper than their female counterparts. For female teachers, the words “clara” and “claras” (singular and plural feminine for the English word “clear,” from “clarify”) significantly correlated with the most common term associated with the females, namely “explicar” (to explain). This means that the students hold the teaching styles and abilities to pass across information; of the female teachers in higher regard than their male counterparts.

Furthermore, this study considers it essential to determine the weight (emotional valence) of the different combinations of words/comments given by the students. To do it, we determined the average sentiment of the individual student’s comments, as shown in Table 1, and then established the foundation shape, represented as a graphical impact view of the different comments (see: Fig. 5). Typically, the EPDM allows us to analyze the sentiment scores, which refers to the sentiment of the individual comments by each student represented in each matrix in the corpus. Thus, the resultant values represent the average sentiment of the sentences of individual students, as shown in Tables 1 and 2, and Fig. 4.

Technically, we utilize the get_sentiment and get_nrc_sentiment function in R (RStudio, 2018) to determine the word count and the average sentiment scores with standard deviation. The sentiment scores are represented as interval values between − 2 and 2, denoting the level of impact for each of the comments, as shown in Table 1. For example, in Table 1, we show the first and last five comments in the datasets to expand on the method, whereby the comments with positive interval values represent a positive (goodness) sentiment, and comments with negative values mean an aversive feeling. Overall, as shown in Table 2 and Fig. 4, we found that male teachers received the most negative comments (sentiments) with a min value of − 1.12 in comparison to − 1.00 received by the females. However, the female teachers also received the most positive sentiments with a max value of 1.817 as opposed to the male counterparts who received an average of 1.570, respectively.

Henceforth, we assume that by visualizing the frequency or projection of the positive (attractiveness/goodness), neutral, and negative (aversiveness/badness) emotions of the students, we can identify the significance and/or the differences of impact among the male and female teachers. The visualization of those results is displayed in Fig. 5. There, we can identify positive, neutral, and negative emotions among the comments of the students, although it can sometimes be tough to discern any trend or behavior across the data. By utilizing the Syuzhet package with the simple plot function in R, we were able to visualize the data to soften (smooth out) and normalize it (i.e., between − 1 to 1, as shown in the upper and lower plots of Fig. 5. Interestingly, we observe that most of the comments are centered evenly around the zero value, as noted earlier in Fig. 4.

Similarly, as seen in the lower plot in Fig. 5, the majority of the comments are clustered around the zero-point on the scale, which means that a large number of the comments provided by the students are considered neutral (see: Fig. 6). This observation was found for both genders of teachers. Nevertheless, considering the simplified macro shape (presented in the upper part of the graph) and the scatter plot (lower part of the graph), we found that the male teachers consistently received highly positive (+) comments, as opposed to the female teachers, who received these in an up and down manner. Although from the same graph (Fig. 5), it can be said that the teaching performance of females was slightly more favorable than those corresponding to males given the margin of comments, which are considered to be positive. In other words, as gathered in Table 2 and Fig. 5, the female teachers received the highest number of positive comments (sentiment) on average (max 1.817). In any case, there were some comments below the zero margin, which are considered as negative (−) for both genders.

To quantify the levels of emotional intensity for purposes of confirming and validating the results, we applied the EPDM model to establish the valence scores, otherwise referred to as the measures of the emotional valence. To do this, we applied the sentimentr miner (get_nrc_sentiment) algorithm in R to extract the different scores for each comment about the male and female teachers. This is obtained from the intensity of the words used in the text by using positive (++), neutral (0) and negative (−-) values, as shown in Fig. 6.

Although the intensity element represents attractiveness (goodness), neutral, and aversiveness (badness) classifications of the comments, respectively, the scale also shows the intensity of emotions. As an example of the scores considering the different genders (i.e., male and female teachers), in Table 3, we show the emotional valence scores of the first 323 comments. The Comments column, [1] to [307], represents the id of the first individual comment within each row. The comments with positive values indicate an attractive valence (emotion), whereas the negative values indicate an aversive valence, and the zeros represent a neutral comment. Valence scores for male teachers rank between − 4 to 8, while those given to females rank between − 4 to 6. We summarize the different valence scores in Fig. 7, revealing that both genders received or showed similar distributions. In Fig. 8, we report some examples of specific comments and their associated emotional valence.

In Fig. 9, we report the overall sentiment analysis of the students’ comments about their teachers’ performance, by gender of the teachers. We look at both genders and the different qualities that the students deem as crucial in the SET evaluation. There have been several definitions of emotional perception and analyses of these in the literature, especially highlighting its application in various domains (Kort, Reilly, & Picard, 2001; Litman & Forbes-Riley, 2004; Nosu & Kurokawa, 2006; Shen, Wang, & Shen, 2009; Tian, Zheng, & Zheng, 2010). Primarily, the EPDM model applies the sentiment classifications within the educational domain, as observed in Litman and Forbes-Riley (2004). It is noteworthy that the feelings are slightly the same for both genders. While some elements of anxiety, resentment, dissatisfaction, and uncertainty appear in most of the optional comments or remarks, motivation, contentment, and amazement appear at a lower extent for both genders. The analysis reveals that the students choose whether or not to write comments about their teachers, but often these may reflect negative sentiments. Interestingly, another notable result is that for male teachers, the remarks or comments most frequently reflect confidence, which tends to appear less often for female teachers.