Beyond Frequency: Using Epistemic Network Analysis and Multimodal Traces to Understand Temporal Dynamics of Self-Regulated Learning

During SRL processing, learners actively pursue desired learning goals through planful, evaluative, and reflective processes, dynamically shifting their focus between broader learning phases and more fine-grained moment-to-moment actions (Schunk & Greene, 2018). In SRL research, studies employing event-based methods (Winne & Hadwin, 1998) to examine SRL processes have established coded think-aloud protocol (TAP) data to identify specific, micro-level SRL processes (Bannert et al., 2014; Greene & Azevedo, 2007, 2009). In a think-aloud protocol (Ericsson & Simon, 1998), participants are asked to verbalize their thoughts, feelings, and actions, which researchers then code for micro-level SRL processing. Some of these micro-level codes have also been expanded to include aspects such as valence (Greene et al., 2018). These micro-level SRL processing codes, such as judgment of understanding, can then be aggregated into macro-level processes like monitoring (Greene & Azevedo, 2009). By collecting concurrent data of students’ reflections as they occur, think-aloud protocols offer fine-grained measurement and relatively accurate picture of SRL processing. However, this measurement method may not capture non-conscious or non-verbalized aspects of cognitive processes (Saint et al., 2020).

Given the limitation of any single method of capturing SRL, there is a growing interest in integrating multichannel and multimodal methods to investigate SRL processing. For example, digital trace data (Bernacki, 2018) can automatically capture and timestamp SRL behaviors that occur during the learning process (e.g., Azevedo et al., 2022). Researchers then can integrate data collected across other modalities such as eye tracking or affective data, to capture a more detailed, more varied, and accurate representation of SRL processing. However, using multichannel and multimodal data for understanding SRL processing presents challenges. These challenges include temporal alignment of multiple data streams, handling variations in data granularity, difficulty in interpretation, and other methodological and analytics challenges (Azevedo & Gašević, 2019). A primary challenge, however, stems from the inherently temporal and sequential nature of SRL (Azevedo, 2014).

Early SRL research commonly employed frequency-based modeling to investigate students’ SRL patterns (Saint et al., 2022). Conceptually, frequency serves as a proxy for the total occurrence of various self-regulatory processes, such as goal-setting, monitoring, and reflection. Methodologically, frequency is typically assessed by counting observable occurrences of SRL-related verbal traces or behaviors over a given period. Research has shown that frequencies of certain SRL behaviors correlate positively with better learning outcomes (e.g., Greene & Azevedo, 2009; Winters et al., 2008). However, frequency-based analyses do not capture the full complexity of the SRL process. Other factors such as the duration, sequence, and context of learning behaviors also shape SRL processing and subsequent learning outcomes (Maldonado-Mahauad et al., 2018). Recognizing this, time-based measurements in SRL research have emerged. One such approach is duration-based modeling, which assesses the amount of time spent on particular SRL processes, offering a proportional sense of their importance in SRL processing. Research has shown that the proportion of time allocated to specific SRL processes can predict learning outcomes (e.g., Taub et al., 2014), highlighting the importance of temporal considerations in understanding SRL. Another approach in time-based measurements is sequence-based modeling. This type of modeling focuses on particular common sequences of multiple SRL behaviors or events and how often they occur over time (Molenaar, 2014). Thus, frequency-based, duration-based, and sequence-based modeling of the SRL process are all being used to investigate the dynamic interplay of SRL processing (Saint et al., 2022).

In the systematic review by Saint et al. (2022), which focused on temporally and sequentially oriented analyses of SRL, the most common approach identified was the use of frequency and time metrics to examine SRL processes. This demonstrates a missed opportunity where single-event-based methods are being used to study a temporal, multi-event process. Addressing this, Saint et al. (2022) indicated that among studies focused on temporality and sequence (n = 53), 13% employed ENA to examine co-temporality of the events in SRL processing and related strategies (e.g., Fan et al., 2021; Uzir et al., 2020). ENA is an analytical technique that identifies and quantifies connections among cognitive elements in discourse, both in individual and collaborative settings (Shaffer et al., 2016). It offers insights into temporal interconnections among learning behaviors by modeling co-occurrence of the events within recent temporal context (for more details, see Shaffer, 2017). ENA has been widely applied to activity-based learning analysis, including SRL research (Saint et al., 2022). In particular, multimodal traces, which encompass both conscious and non-conscious thoughts and actions during learning via TAPs and digital traces, are particularly suited for ENA modeling of the sequence and co-occurrence of these events compared to methods like process mining or ordered network analysis (ONA; Tan et al., 2022)), which typically assumes directional sequences. The temporal relationship among these multimodal traces in SRL processes can be less linear and more contextually intertwined. For instance, a student’s digital trace, such as submitting a quiz response, can occur before, during, or after think-aloud verbal utterances related to their SRL processes. In this context, ENA’s capability to account for the co-occurrences of multimodal traces within a recent temporal context using a time window is adept at capturing these complex interactions in a non-linear temporal framework. This approach enables us to explore how various SRL events may occur simultaneously or in close temporal proximity, providing insights into the dynamic and intertwined nature of SRL processes during learning activities.

However, when using ENA, researchers must consider several analytic choices to optimize its utility. These include data segmentation, event granularity (i.e., unified or varied analytic time unit), temporal window size, and guarding against model overfitting. As these analytic choices can significantly impact ENA results (e.g., Siebert-Evenstone et al., 2017), researchers must possess a thorough understanding of the data and its context. Additionally, ENA tends to emphasize stronger connections, potentially downplaying less frequent events or connections. Considering these factors throughout the process, researchers should employ an iterative and multi-method approach to data analysis and interpretation by looping back and forth between the data and the model.

We recruited 49 undergraduate students from two parallel sections of the same introductory biology course offered in the Spring 2020 semester at a large public university in the southeastern United States. Participants were recruited through in-person announcements during lectures, weekly class emails, and the learning management system (LMS), Sakai, which was used in the course. Participants received a $75 gift card for their time upon completion, which typically lasted for an average of 75 min. One participant was excluded from the final dataset due to difficulties in verbalizing their thoughts aloud. Consequently, our analyses comprised data from 48 participants, aged between 18 and 26 years old. The majority of participants were first-year students (N = 40), and out of the total, 36 identified as female and 9 as biology majors (see Table 1 for detailed demographic characteristics of participants).

This study closely followed ethical guidelines, obtaining written informed consent from participants prior to their involvement in any research activities. The research protocol was approved by the Institutional Review Board (IRB), ensuring participant confidentiality. Participants were informed of their right to withdraw at any time, and data were securely stored and accessible only to the research team.

Upon arrival at the lab, participants received an overview of the session and were asked to review and sign consent forms. Following this, they were instructed on the think-aloud protocol and practiced. The lab sessions were conducted individually, with each participant working one-on-one with one or two research assistants. Primarily, one researcher managed the session, ensuring that participants verbalized their thoughts effectively. In instances where two researchers were present, one served as the primary facilitator, while the second provided assistance as needed. During the lab session, participants were provided with learning materials from Lesson 23: How Populations Evolve II, part of an introductory biology course. The introductory biology course employed a high-structure active learning (Eddy & Hogan, 2014) course design across pre-class, in-class, and after-class phases, incorporating various learning activities. High-structure active learning is a student-centered educational practice demanding students’ intentional engagement in learning activities for knowledge construction (Bonwell & Eison, 1991; Lombardi et al., 2021). This course design was mirrored in the lab study. In the pre-class phase, participants were told to prepare for class as they typically would. During the in-class phase, participants watched a recorded video and answered practice questions on Learning Catalytics. Following this, in the after-class phase, participants completed a quiz on Mastering Biology. At the end of the session, participants filled out a demographics survey.

Results (see Table 3) revealed that students in the progressing group (M = 152.55, SD = 42.58) exhibited statistically significantly higher overall SRL-related code occurrences than those in the mastery group (M = 129.92, SD = 33.89), with a moderate effect size (t = 2.05, p < 0.05, Cohen’s d = 0.58). Specifically, we found that students classified in the progressing group produced statistically significantly higher occurrences of task-specific (t = 2.50, p < 0.01, Cohen’s d = 0.73), assessment strategies (t = 2.03, p < 0.05, Cohen’s d = 0.58), and domain-general strategies (t = 2.53, p < 0.01, Cohen’s d = 0.71) in TAP macro-level codes, all with a moderate effect size. This shows that, on average, students in the progressing group, who exhibited relatively lower performance in in-task quizzes, tended to verbalize more about task-specific processes (e.g., restating a sub-task or sub-goal), assessment strategies (e.g., guessing), and domain-general strategies (e.g., re-reading, summarizing) compared to those in the mastery group (for t-test results on the raw occurrences of macro- and micro-level codes to see the results without normalization, see the Appendix). Furthermore, students in the progressing group exhibited statistically significantly higher occurrences of pre-class tasks: incorrect (t = 2.73, p < 0.01, Cohen’s d = 0.77) events but lower on guided reading questions (t =  − 2.11, p < 0.05, Cohen’s d =  − 0.62) events in Trace macro-level codes, both with a moderate effect size, indicating that they were more likely to submit more incorrect responses in their pre-class homework submissions, whereas also less inclined to revisit guided reading questions compared to those in the mastery group.

The t-test results (see Table 4) revealed that students in the progressing group (M = 30.93%, SD = 9.78%) spent statistically significantly more time producing TAP or digital traces during the task compared to those in the mastery group (M = 26.39%, SD = 5.02%), with a medium effect size (t = 2.09, p < 0.05, Cohen’s d = 0.56). Specifically, students in the progressing group allocated statistically significantly higher proportion of time spent on task-specific, with a moderate effect size (t = 2.27, p < 0.05, Cohen’s d = 0.66), indicating that they proportionally devoted more time to verbalizing task-specific processes (e.g., restating a sub-task or sub-goal) compared to those in the mastery group. Moreover, students in the progressing group exhibited statistically significantly higher proportion of time spent on pre-class tasks: incorrect (t = 2.73, p < 0.01, Cohen’s d = 0.77), but lower on guided reading questions (t =  − 2.11, p < 0.05, Cohen’s d =  − 0.62) in macro-level codes of Trace data compared to the mastery group. This shows that students in the progressing group spent proportionally more time on addressing errors in their pre-class homework submissions but less time on engaging guided reading questions for review, which aligns with the findings from RQ1.

In this paper, we investigated an understudied aspect of SRL theory: how temporal SRL processing, as captured by multimodal traces, differs between mastery and progressing groups. We examined count-based, duration-based, and sequence-based models of SRL processing to unveil diverse dimensions of the process across different achievement levels. Specifically, for the sequence-based model, we employed a multi-method approach, incorporating ENA to analyze the temporal and sequential dynamics of multimodal events during SRL processing and then qualitative analysis to delve deeper into those dynamics.

Our findings from RQ1, focusing on conventional frequency-based modeling of SRL processing (Model 1), revealed that students in the progressing group, characterized by relatively lower in-task quiz performance, exhibited more frequent occurrences of overall SRL-related utterances and log events compared to those in the mastery group. The verbalizations of progressing group students primarily focused on restating sub-goals, guessing an answer, and re-reading and summarizing. These tendencies appear to be associated with their increased efforts to correct homework submissions, as captured in digital traces, which prolonged their individual task times. Despite normalization of raw counts based on session durations and average session time across all students, frequency-based modeling remained highly sensitive to fluctuations in task durations and the frequency of learning activity attempts, often influenced by performance-related re-attempts. This suggests that if individual student’s task session durations fluctuate based on performance, frequency-based modeling may not provide an accurate depiction of SRL, nor fully capture the temporal and sequential aspects of the process (Saint et al., 2022).

For RQ2, focusing on duration-based modeling of SRL processing (Model 2), we found statistically significant differences in the time allocation for multimodal traces between the two groups. Specifically, students in the progressing group devoted a greater proportion of time to verbalizing task-specific processes, such as restating sub-tasks or sub-goals, or addressing errors in homework submissions, whereas they dedicated less time reviewing guided reading questions. Although some findings from Model 2 echoed those of Model 1, the statistically significant differences in the frequency of two TAP macro-level codes observed in Model 1, assessment strategies and domain-general strategies, vanished when adjusting for the proportions of time spent on SRL events. The discrepancy between the two models implies that although there may be differences between the mastery and progressing groups in the frequency of SRL events, especially for TAP codes that extend over time, these differences may not be reflective of the actual proportion of time allocated to those events during the learning process. In essence, differences in frequency may not necessarily translate into differences in the actual time spent on those events. This highlights the importance of examining SRL processing dynamics from both frequency and duration perspectives when utilizing multimodal traces, which encompass various time spans for occurrence.

For RQ3, focusing on sequence-based modeling of SRL processing using ENA (Model 3), statistically significant differences emerged in the patterns of multimodal traces during SRL processing between the progressing and mastery groups. Although the TRACE codes themselves did not present substantial differences, suggesting that the primary distinctions in SRL processing were more evident in verbalizations rather than digital trace events, important nuances were still observed. The mastery group showed stronger connections between digital trace events, such as guided reading questions and in-class tasks, and their SRL verbalizations, indicating a more intentional use of these digital resources to support their SRL processes. In contrast, the progressing group more frequently linked their SRL verbalizations with incorrect pre-class responses, suggesting that their SRL efforts were often focused on addressing the errors they encountered. Interestingly, when examining TAP codes, both groups frequently verbalized monitoring-related processes—the most prevalent among the SRL codes—yet no difference in frequency was detected between the two groups. However, the way this specific SRL process temporally and sequentially connected to other processes differed between the groups: it more frequently co-occurred with domain-specific strategies in the mastery group, but with domain-general strategies in the progressing group. Students in the mastery group demonstrated that they tended to primarily engage in monitoring their learning during mathematical problem-solving processes, contributing to their superior performance on in-task quizzes. In contrast, those in the progressing group appeared to be more actively engaged in monitoring their learning while accessing information from learning materials, primarily focusing on evaluating the relevance of the content to their overall learning task, goal, or question, rather than monitoring their own learning processes. Furthermore, their evaluations of these contents tended to be negative, often following struggles to fully comprehend the contents. These insights from our multi-method approach underscore the inherently temporal and sequential nature of SRL (Azevedo, 2014). As the SRL process unfolds over time (Bernacki, 2018), it is crucial to analyze these data in ways that take into account the temporal and sequential nature of SRL to gain an accurate understanding of how they dynamically interact and influence SRL processing.

In addition, our work contributes to advancing theoretical understanding of the mechanisms underlying SRL processes between mastery and progressing students in technology-enhanced STEM learning. By leveraging multichannel and multimodal data to assess and understand SRL processing (Azevedo & Gašević, 2019), which capture both conscious and non-conscious thoughts and actions during learning through think-aloud protocols and digital traces, we offer a nuanced understanding of mastery and progressing students’ SRL processing in STEM learning within technology-enhanced learning environments—the way specific SRL processes temporally and sequentially connect to other processes differed between the two groups, which lead to disparities in their academic performance. Doing so reveals how assumptions about differences in SRL processing between novices and experts (e.g., Zimmerman, 2002) require data collection and analysis techniques that go beyond what occurred, and instead can model how sequences of processing differ. This is evidence for more sophisticated modeling of the phenomenon of SRL, comprising more than individual processes (Greene, 2022). In addition, our findings underscore the need for targeted interventions to guide and assist low achieving students in effectively utilizing SRL strategies at various stages of the learning process to enhance their learning outcome.

Our study also made contributions to methodologies for capturing and analyzing SRL. We conducted empirical investigations into various facets of the temporal SRL processing by collectively examining frequency-based, duration-based, and sequence-based modeling approaches in a unified manner. To our knowledge, no previous empirical study has undertaken such a comprehensive approach, and doing so allowed us to demonstrate how each modeling approach reveals unique aspects about temporal SRL processing.

Another methodological contribution is our empirical demonstration of implementing a novel analytic approach to model the temporal dynamics of SRL processes employing the interleaving approach and ENA. We systematically applied the interleaving approach to accurately model multimodal events occurring during the learning process, reflecting the temporal interconnections among them through temporal alignment of multiple data streams. By detailing how we operationalized the multimodal trace data, representing a “temporally entangled” data structure (Sung et al., 2022), our analytic approach provides insights into one method for modeling the dynamic interplay among multimodal interactions during the SRL process. Moreover, our study leverages ENA as a potent tool to model and visualize SRL process patterns while retaining sequential and temporal information of multimodal traces. Through a multi-methods approach that integrates quantitative findings with qualitative insights, we empirically showcased the analytical effectiveness of our approach, employing the interleaving and ENA, in temporally focused SRL analysis.

This study has several limitations. First, due to the highly structured course design for individual learning, there were limited instances of interleaving between verbalization and digital traces during the learning process, which may have diminished the impact of the interleaving approach in modeling multimodal interactions. Given that the interleaving approach has shown its usefulness in modeling multimodal discourse in computer-supported collaborative learning context and revealing different patterns of collaborative multimodal interactions among students (e.g., Sung & Nathan, 2024), future researchers should explore its applicability in more typical technology-enhanced collaborative learning settings, where there are much more instances of students interacting with peers while exploring digital learning resources contemporaneously. Secondly, the current treatment of digital traces as momentary events ignores their duration. Future researchers should consider extracting temporal information, such as the duration of student engagement with digital resources, to better understand their impact and influence on the learning process.