Finds documents with both search terms in any word order, permitting "n" words as a maximum distance between them. Best choose between 15 and 30 (e.g. NEAR(recruit, professionals, 20)).
Finds documents with the search term in word versions or composites. The asterisk * marks whether you wish them BEFORE, BEHIND, or BEFORE and BEHIND the search term (e.g. lightweight*, *lightweight, *lightweight*).
The article combines Natural Language Processing (NLP) with Epistemic Network Analysis (ENA) to study student knowledge integration within an AI dialog. Students write initial and revised explanations for the Energy Story activity, with an AI avatar providing adaptive prompts. NLP models score the KI levels of the explanations and detect ideas, while ENA visualizes how students' connections among ideas change. The research aims to understand how students refine their ideas through revision, providing valuable insights into the learning process and the effectiveness of AI-driven educational tools.
AI Generated
This summary of the content was generated with the help of AI.
Abstract
In this study, we used Epistemic Network Analysis (ENA) to represent data generated by Natural Language Processing (NLP) analytics during an activity based on the Knowledge Integration (KI) framework. The activity features a web-based adaptive dialog about energy transfer in photosynthesis and cellular respiration. Students write an initial explanation, respond to two adaptive prompts in the dialog, and write a revised explanation. The NLP models score the KI level of the initial and revised explanations. They also detect the ideas in the explanations and the dialog responses. The dialog uses the detected ideas to prompt students to elaborate and refine their explanations. Participants were 196 8th-grade students at a public school in the Western United States. We used ENA to represent the idea networks at each KI score level for the revised explanations. We also used ENA to analyze the idea trajectories for the initial explanation, the two dialog responses, and the final explanation. Higher KI levels were associated with more links and increased frequency of mechanistic ideas in ENA representations. Representation of the trajectories suggests that the NLP adaptive dialog helped students who started with descriptive and macroscopic ideas to add more microscopic ideas. The dialog also helped students who started with partially linked ideas to keep linking the microscopic ideas to mechanistic ideas. We discuss implications for STEM teachers and researchers who are interested in how students build on their ideas to integrate their ideas.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Introduction
Epistemic Network Analysis (ENA) offers a way to represent the connections among student ideas that align with the goals of the Knowledge Integration (KI) framework. In this research, we explore the use of ENA to represent the ideas in students’ initial and revised explanations for a KI activity called the Energy Story. The Energy Story activity asks students to explain, “How does energy from the sun help animals to survive?” Students write an initial explanation, respond to an AI dialog with an avatar, and write a revised explanation (Fig. 1). We use learning analytics in the form of Natural Language Processing (NLP) models to score the KI level of the initial and final explanations and to detect the ideas in the responses to the avatar. The dialog with the avatar features researcher-designed prompts that target idea(s) detected in the students’ prior responses. We apply ENA to the output from the NLP learning analytics to represent how the connections students make among their ideas change before and after the AI dialog.
Fig. 1
NLP adaptive dialog between Ada (a thought-buddy) and Sophia (a 7th-grade student)
The KI framework emphasizes that students intentionally make sense of their everyday experiences, instruction, and interactions with others to develop numerous ideas about any phenomenon (diSessa, 2018; Linn & Eylon, 2011; Sayary et al., 2015). As a result, students develop a repertoire of ideas that includes broad, intuitive, mechanistic, and non-normative ideas. Consistent with constructivist views, the KI framework offers design principles for instruction that build on student ideas, encouraging them to explore their ideas, discover new ideas, distinguish their ideas from those of others, and reflect on the varied perspectives they hold (Linn et al., 2023). When teaching for KI, teachers affirm student ideas and invite them to explore the connections among their ideas. They encourage students to analyze their own ideas rather than describing ideas as misconceptions (Gerard, Bradford, et al., 2022a, 2022b; Otero, 2006). KI assessments measure the degree to which students combine their ideas into coherent arguments to explain complex scientific phenomena. KI levels reward students for using evidence to explain scientific concepts and phenomena such as global climate change and photosynthesis (Bradford et al., 2023; Gerard et al., 2022a; Ryoo & Linn, 2015). These assessments are scored for the level of KI on a 1 to 5 scale.
Advertisement
Learning Analytics Using NLP Models for KI Activities
Researchers have developed learning analytics to analyze students’ responses to KI activities, using the state-of-the-art Natural Language Processing (NLP) techniques for automated scoring and identifying discourse-level expressions of ideas that align with the Next Generation Science Standards (NGSS). In one series of studies, researchers have developed KI scoring models (Liu et al., 2016) that automatically score student-written responses using the KI rubric. Across numerous KI items, the NLP models achieve Quadratic Weighted Kappa (QWK) agreement with human raters greater than 0.80 (Holtmann et al., 2023; Riordan et al., 2020). Recently, researchers have developed idea detection models that identify multiple distinct ideas students express in their explanations of specific scientific phenomena. Across numerous topics, NLP idea detection models have achieved overall micro-averaged F-scores greater than 0.68 (Holtmann et al., 2023; Li et al. 2024). This accuracy is sufficient for use in classroom dialogs (cf. Schulz et al., 2018, 2019). We use these KI-based NLP learning analytics to establish the KI level and detect student ideas in the KI activity studied in this research.
Epistemic Network Analysis
We designed uses of ENA to represent changes in the relationships among the detected ideas in the responses in the KI activity. To connect to the KI level, we compared the revision trajectories of students whose initial explanation was at KI level 1 or 2 to students whose initial explanation was at KI level 3 (the highest score on the initial explanation). We built on prior work with ENA (Shaffer et al., 2016). ENA is rooted in the theory of epistemic frames. It has been used to visualize how elements of learning such as cultural behaviors, discourse, and topics relate to each other and form a complex web of meanings (Shaffer, 2018). Recent studies have explored students’ epistemic ideas in science (Peters-Burton et al., 2023), their conceptions of science topics such as food webs (Rachmatullah & Wiebe, 2022), their perceptions regarding how technology can facilitate science learning (Chang & Tsai, 2023), their connections between science concepts and real-life experiences (Talafian & Kang, 2023), and other topics (Akumbu et al., 2023; Sun et al., 2022).
Thus, this research studies how students respond to a Knowledge Integration activity that features a dialog with an avatar to promote revision of the initial explanation. Student-written responses to the adaptive prompts in the dialog were collected and analyzed by the combined, serial use of the KI-based NLP models and ENA. By investigating this combination, we aimed to understand students’ conceptual models and the evolution of their ideas.
Our research question is as follows:
Advertisement
How does ENA help represent how students refine their ideas as they revise in a KI activity in response to a dialog with an avatar?
Background
Knowledge Integration Dialogs
We use the Knowledge Integration (KI) framework to design the AI dialog and the adaptive guidance. The dialog supports students to engage in the KI processes of adding ideas and distinguishing among them (Linn & Eylon, 2011). The design builds on investigations of curriculum design for various science topics and develops adaptive guidance to help students improve their explanations (Gerard et al., 2015; Vitale et al., 2016). Prior studies have shown the effectiveness of adaptive guidance. Gerard et al. (2022b) showed that the adaptive KI prompts encouraged students to reflect on prior experiences, consider new variables, and raise scientific questions. Li et al. (2023a) explored how the adaptive KI prompts in the Energy Story Activity facilitated middle school students’ integrated revision. Bradford et al. (2023) showed two rounds of KI eliciting and distinguishing prompts in the climate change adaptive dialog supported all students to broaden their ideas.
The studies using the KI adaptive guidance revealed how students improved the coherence of science understanding with KI-based scaffolding. Researchers mentioned the necessity to investigate the detailed revision trajectories of how students add new ideas and distinguish among their ideas with adaptive guidance in the AI dialogs (Li et al., 2023b).
NLP and KI
Advances in technology can help capture and display patterns in students’ ideas, and then respond to their ideas. NLP uses machine learning to enable computers to understand text and speech (Chowdhary, 2020). One important application of NLP in science education is the automated assessment of student explanations (Zhai et al., 2020). Prior research has demonstrated the value of using the KI framework to design an NLP model that generates a holistic score for the overall coherence and accuracy of student responses. Ryoo and Linn (2015) developed a KI scoring rubric for an Energy Story item measuring understanding of photosynthesis, while Gerard and Linn (2016) showed how automated guidance based on a KI scoring model of Energy Story, in conjunction with teacher guidance, enhanced classroom inquiry activities.
Prior research has also used NLP to develop Intelligent Tutoring Systems (ITSs), like AutoTutor (Nye et al., 2014), to evaluate students’ work and give them timely feedback (Graesser et al., 2004). Many AutoTutor models used a bag-of-words approach (VanLehn, 2011) called Latent Semantic Analysis to compare student responses with anticipated correct and incorrect answers (Graesser et al., 2007). AutoTutor has been updating their NLP tools. They use dialogs that model the problem-solving process and give hints instead of diagnosing the accuracy of their answers (Graesser et al., 2004; Paladines & Ramirez, 2020).
Focusing on middle school student science explanations, Riordan et al. (2020) applied the cutting-edge multi-label NLP model for idea detection in KI activities using a token classification approach (Riordan et al., 2020). This model automatically identifies discourse-level expressions of ideas according to pre-defined rubrics. The multi-label idea detection models have been employed in multiple KI activities (Holtmann et al., 2023; Bradford et al., 2023; Li et al. 2024) to design adaptive dialogs that assign researcher-designed personalized guidance to help students make integrated revisions. Prior work shows that students need specialized help to make integrated revisions to their science explanations by reformulating the connections among ideas to increase the coherence of a response (Gerard & Linn, 2022), which is different than revising language or adding more evidence.
The rapid development of NLP applications to education has raised questions about how students use adaptive guidance to learn, including discovering new ideas, distinguishing between their initial and new ideas from instruction, and constructing links to form an integrated understanding (Linn & Eylon, 2011). A recent study using Epistemic Network Analysis combined NLP-generated data to capture temporal changes within the AI dialog (Li et al., 2023c). In this study, all the codes within the dialog require laborious human coding. Our study that applied the NLP learning analytics provides ENA with automatically generated idea codes that scale up the human expert annotation. In our study, we propose to use ENA to visualize the NLP analytics to show how students develop their ideas as they form and revise science explanations through interaction with an NLP dialog.
ENA in STEM Education
We use the output data from the NLP models as the direct input data for ENA to represent how students connect their ideas as they revise their responses during the Energy Story activity. We explore the potential added value of these representations for illustrating learning progress.
ENA has been used in education for a variety of purposes. For instance, coding frameworks have been developed to examine students’ cognitive and metacognitive processes during science inquiry, enabling researchers to identify patterns in learning behaviors such as students’ scientific reasoning and metacognitive regulation patterns (Omarchevska et al., 2021; Sun et al., 2022). These studies utilized ENA to analyze group conversations or individual think-aloud and behavior data within a time frame of around 1 h to reveal students’ discourse or interaction patterns (Akumbu et al., 2023; Omarchevska et al., 2021; Sun et al., 2022). They demonstrate the benefit of ENA in understanding the interconnected nature of students’ discourse and interaction in STEM learning. Recent studies have also begun to examine individual student performances on written assessments to investigate student views or ideas within the frame of the assessment task (Chang & Tsai, 2023; Rachmatullah & Wiebe, 2022). Through ENA, each student’s network model can be generated. A certain group’s network model can be defined and then aggregated from individual student models to show an averaged model for the group. Models among different groups can therefore be compared.
We utilized ENA in this study in line with previous research that employed ENA to investigate student ideas in their written responses. ENA provided new perspectives for KI studies with detailed visualizations of student idea networks at the KI levels. We further explore an innovative approach of combining ENA with KI-based NLP models, which exemplifies a type of learning analytics that combines NLP and visualization analysis techniques. We have only found one study that demonstrated a similar combination (Ferreira et al., 2018). Ferreira et al. (2018) employed Latent Dirichlet Allocation (LDA), a topic-based NLP method, to detect topics from students’ textual messages in asynchronous discussions in an online learning environment. They then used ENA to examine the relationship between cognitive presence and discussion topics. LDA detected topics based on occurrences of words rather than semantics, such as the one used in this study. In addition, they used human experts to code cognitive presence. However, in this study, we use the automated generated idea labels from the NLP models as the idea nodes input for ENA, exploring the possibility of advanced automation technologies for future learning analytics to understand student ideas in STEM education.
The Energy Story Activity
Design of the Energy Story Adaptive Dialog
The Energy Story Activity studied in this research was embedded in an open-source Web-based Inquiry Science Environment (WISE) unit in a start-of-year inventory. The activity does not include instruction. Students typically complete the activity in 5 to 10 min. The Energy Story activity was designed by a Research Practice Partnership (RPP; Gerard, Bradford, et al., 2022a, 2022b) including middle school science teachers, learning scientists, and computer scientists. The activity features an NLP adaptive dialog that prompts students to link ideas about energy transfer and transformation during photosynthesis and cellular respiration. Students write an initial explanation to the question, “How does energy from the Sun help animals to survive?” The KI scoring model automatically assigns a KI score to their initial explanation, and the idea detection model automatically identifies the ideas in the explanation. Subsequently, students have a dialog with an avatar, serving as their thought buddy. Finally, the KI scoring model automatically assigns a KI score to the revised explanation.
Application of the KI Scoring Model
The Energy Story KI scoring model was developed and validated in previous studies (Holtmann et al., 2023; Riordan et al., 2020; Ryoo & Linn, 2015). Because the previously developed KI scoring model (see the 5-level rubric in Fig. 2) had a high agreement with human raters (QWK = 0.809), in this study, we used it to score students’ initial revised explanations. Details for the model development approach can be found in (Riordan et al., 2020). To evaluate the overall impact of the dialog, we compare the KI scores of their initial to revised explanations using paired t-tests.
Fig. 2
Energy Story KI level description and student examples with idea codes
We developed and applied an NLP model for idea detection using a token classification approach (Riordan et al., 2020; Schulz et al., 2019). The model was trained to predict an idea category for each word token in the student response data. Sequential words predicted to represent the same idea were grouped as an idea span. Given the potential overlap of ideas, we employed a multi-label idea detection model. The model architecture consisted of a pre-trained transformer backbone, succeeded by a single-layer bidirectional GRU-based RNN and a final linear projection (Li et al. 2024). During model development, we experimented with various learning rates and adopted SciBERT (Beltagy et al., 2019) as the backbone, benefiting from its pretraining on 1.1 million scientific papers for wordpiece vocabulary creation, based on the BERT language model (Devlin et al., 2019).
To build the NLP model (See details in Li et al. 2024), two researchers led the RPPs to identify the range of distinct ideas expressed by students in their explanations (19 ideas; see Fig. 5 for a detailed description of each idea). The researchers collaboratively annotated ideas for 20% of the total 1206 student responses, discussing and refining the idea rubrics until achieving satisfactory inter-rater reliability (Cohen’s Kappa = 0.79). Subsequently, each researcher individually annotated half of the remaining 80% of student responses, labeling distinct ideas. This annotated dataset served as the training data for the idea detection NLP model. Before deploying the idea detection NLP models in the activity, the model achieved an overall micro-averaged F-score of 0.7297, which was acceptable for deployment (cf. Schulz et al., 2018, 2019).
Once the model was developed and validated, we used it to assign personalized KI-based guidance in the dialog. Applying the KI pedagogical framework, the first prompt is designed to elicit further reasoning about the student’s idea. It might ask one question like “What else…” or “What happens next…”. In the next round to help the student connect their idea with evidence or mechanism the prompt encourages distinguishing among ideas, by asking one question like “How is this different from…” or “What is the process…”. After two rounds of adaptive guidance, students were prompted to revise their initial explanation using their newfound ideas.
Methods
Participants
We implemented the Energy Story Activity as an inventory activity with 196 8th-grade students using a Chromebook in their science classroom at the beginning of the school year with two science teachers at a public school. 33% of students in this school have a home language other than English. Each student wrote an initial explanation (E1), responded to two adaptive prompts from the avatar (Dialog 1 and 2 as D1 and D2), and wrote a final explanation (E2). Based on our prior experience where the second KI adaptive prompt had a small impact, we limited the conversation to two adaptive prompts (Gerard & Linn, 2022). We designed the avatar to implement the KI processes of eliciting ideas for the first dialog and distinguishing ideas for the second dialog. The data contained 784 student responses, with each of the 196 students contributing four responses. The responses were analyzed and scored using the KI scoring and idea detection models, as elaborated in the preceding section.
ENA Representation of Connections Among Student Ideas
An output dataset from the computer-logged NLP dialog, formatted as a CSV file, was prepared for ENA. The data preprocessing procedure is summarized in Fig. 3. Students’ explanations at E1 and E2 were scored with KI levels. Their explanations at E1 and E2, as well as each student’s response at D1 and D2, were coded as 1 or 0 for the presence or absence of each idea. 132 responses were labeled as “non-scorable” from the NLP model with no idea labels identified. One researcher who created the idea rubric reviewed those responses and assigned codes to them. If the response did not belong to any code, the research categorized them into new codes. During this process, 67 responses (8.55% of all responses) were categorized as “off-topic” to the main question, such as “idk.” We excluded the code “offtopic” from ENA modeling because students did not express any idea related to the science topic. The researcher assigned 54 responses (6.89%) to existing idea labels. Eleven responses (1.40%) mentioned animals use energy sources other than the Sun to survive, such as “animals also get energy from the water and soil.” Therefore, we added this idea to our codes for ENA, resulting in 20 nodes in ENA models (see Fig. 5).
To represent student ideas using ENA, we used the ENA1.7.0 Web Tool (Marquart et al., 2018) to visualize the mean structure of the students’ ideas, showing on average how ideas occur and co-occur within each response for a particular group of students. The 20 ideas (Fig. 5) are the nodes in the ENA models. Edges reflect the relative frequency of co-occurrence of these codes within each response that the student submitted to the NLP adaptive dialog. The unit of analysis is each individual student. Setting the unit of analysis as each student allows us to examine an overall network model for each student or an averaged model for a specific group of students during a specific round (by including data from only a specific round) or considering all four rounds (by including data from all four rounds). We set the stanza size to 1 since there was no student–student interaction (Shaffer, 2014; Zörgő et al., 2021). A larger-sized node indicates that more students demonstrated this idea. A thicker line indicates that the connection between the two nodes occurred more frequently (Wooldridge et al., 2018).
ENA utilizes a dimensional reduction method that identifies two dimensions accounting for the greatest variation in the data and positions the nodes by the dimensions (Shaffer et al., 2016). This approach enables us to identify the two most significant factors underlying the distribution and structure of students’ ideas. We also compared idea network models of student responses among different KI levels, as well as between students who started with intuitive ideas and partial knowledge integration. T-tests were conducted to determine whether the two student groups’ models significantly differed in their positions on the X and Y dimensions.
Results
Overall, students made progress in explaining how energy from the sun helps animals survive. We report those results and explore how ENA representations of the connections among student ideas clarify student trajectories.
Initial and Revised Explanation Scores
During the Energy Story activity, from their initial to revised explanations, students made significant KI gains (MInitial = 2.46, SDInitial = 0.37; MRevised = 2.76, SDRevised = 0.27; t(195) = 6.42, p < 0.001). They mainly started at KI 2 or 3. Many students improved to KI 3 or 4 (see Fig. 4). Thus, the adaptive dialog motivated students to rethink their initial response and make gains in integrating their ideas.
Fig. 4
The frequencies of each KI level at initial and revised explanations
Students added ideas from their initial (N = 415 total ideas, see Fig. 5) to the final explanation (N = 593 total ideas). We first explored the frequencies of each idea node in the Initial explanation (E1) and Revised explanations (E2). The most frequent ideas in both explanations are “the Sun helping plants survive” (SunPlt), “Animals eating plants as food and nutrition” (Foodchain), and “Energy transfer” (EngTran). Among the remaining ideas, the most frequently added are “Animals use glucose/food for energy” (AnimalGrw), “Animal uses cellular respiration to release energy” (AnimalCellResp), and “Energy transformation during photosynthesis” (PhotoEngTransfer).
Fig. 5
The idea label description and frequencies of each idea code in the dialog
Students expressed fewer ideas in response to the two dialog prompts, with 271 ideas at Dialog 1 (D1) and 245 at Dialog 2(D2), consistent with the design of the prompt to focus on one idea. The most frequent idea at D1 is “Energy transfer” (EngTran), while the most frequent idea at D2 is “Animals eating plants as food and nutrition” (Foodchain).
Using ENA to Compare Idea Structures Among Different KI Levels
We employed ENA models to examine how ideas are linked to the initial ideas across the dialog, and how the students sort out the ideas they have added in their final explanations. First, we compared the ENA models on the final explanation (E2) for students at the four KI levels in the dataset (Fig. 6). The ENA model accounted for 29.6% of the total variance, a level deemed acceptable for social science (Ozili, 2023). The ENA model had co-registration correlations of 0.92 (Pearson) and 0.92 (Spearman) for the first dimension (x-axis) and co-registration correlations of 0.85 (Pearson) and 0.84 (Spearman) for the second dimension (y-axis). These measures indicate a strong fit between the visualization and the model.
Fig. 6
Idea network ENA models for responses at each KI level on the final explanation (idea labels are shown in Fig. 5)
We interpreted the axes based on the node placement and relationships to domain knowledge. We relied on nodes at the extreme edges of the space to provide more information for labeling the axis (Shaffer et al., 2016). We described the X dimension (accounting for 17.1% of the variance) as representing macroscopic to microscopic ideas about ecosystems and organisms. Macroscopic ideas such as the Sun helping plants survive (SunPlt) and Animals eating plants as food and nutrition (Foodchain) characterized the macroscopic edge. Nodes at the microscopic edge included ideas about invisible energy transfer (EngTran), and transformation within photosynthesis (PhotoEngTr). Nodes in the middle of the x-axis, nodes are related to the chemical reactions within cells that connect the macroscopic to the microscopic level, such as reactant and product of photosynthesis (CO2.Water, O2.Sugar) and cellular respiration (AnimCellResp).
We described the Y dimension (accounting for 12.5% of the variance) as representing descriptive to mechanistic ideas. Towards the top of the y-axis, nodes contain descriptive language that addresses specific aspects of the question, such as the Sun’s role in plant survival (SunPlt), energy transfer from plants to animals (EngTran), and descriptive explanations of the food chain, including the conservation (EngCon) or decrease (EngDown) of energy along the chain. Towards the bottom of the y-axis, nodes feature mechanistic language with detailed reasoning regarding how the Sun’s energy aids plants in supporting animals’ survival. This includes detailed descriptions of how animals consume plants for food (Foodchain) and the mechanism of energy transformation in photosynthesis to provide chemical energy for animals (PhotoEngTr).
The four idea networks in Fig. 6 including axes and node locations are based on all students’ final explanations. Only the node size and weight of the lines change across the KI levels. The KI 1 and 2 networks show largely disconnected ideas with somewhat larger node sizes for descriptive ideas. Some connections occur at KI 2. The KI 3 and 4 networks show emerging connections among the ideas. To test the differences among KI groups, we applied a two-sample t-test assuming unequal variance in the distribution of the centroids of the KI 2 and KI 3 groups. The results showed that responses at KI 2 (M = 0.10, SD = 0.33, N = 50; t(49.00) = 2.16, p = 0.04, Cohen’s d = 0.31) were statistically significantly different from responses at KI 3 (M = − 0.02, SD = 0.36, N = 138; t(92.91) = 2.25, p = 0.03, Cohen’s d = 0.36). KI 3 connections are weighted towards the mechanistic-microscopic direction. KI 4 shows connections between the microscopic level and mechanistic ideas. The networks represent that students who score at partial compared to those who score at a full link (KI 3 versus KI 4) link a mechanistic idea to another idea, consistent with the KI scoring rubric. Across the four ENA idea networks, higher scores are associated with more links and increased frequency of mechanistic ideas.
Using ENA Idea Models to Compare Students Who Start with Intuitive Ideas to Those Who Start with Partially Linked Ideas
We explored the ability of ENA to capture the idea links for the groups of students who started with intuitive ideas and those who started with partially linked ideas. We divided students based on the level of their initial explanation: students who started with Intuitive Ideas, as indicated by their initial explanations scored as KI 1 and 2, and students who started with Partially Linked Ideas, as indicated by their initial explanations scored as KI 3 (no students scored at KI 4 or 5 in initial explanations). This resulted in 93 students in the Starting with Intuitive Ideas group and 103 students in the Starting with Partially Linked Ideas group. We used ENA to model the connections among ideas for all students’ responses on explanations and dialogs. Our model had co-registration correlations of 0.94 (Pearson) and 0.94 (Spearman) for x-axis and co-registration correlations of 0.74 (Pearson) and 0.75 (Spearman) for the y-axis. A t-test on the mean positions of the centroids of the two groups showed that overall the group that starts with Partially Linked Ideas (M = − 0.05, SD = 0.29, N = 412) was statistically significantly different at the alpha = 0.05 level from the group that starts with Intuitive Ideas (M = 0.05, SD = 0.23, N = 372; t(773.89) = − 5.57, p = 0.00, Cohen’s d = 0.39) along the X dimension.
The ENA model accounted for 12.7% of the variance on the y-axis and 15.3% of the variance on the x-axis, totaling 28.0% of the variance (Fig. 7), slightly less than the model using only the final explanations. Based on the location of the nodes, the x-axis and y-axis are described as having the same dimensions as represented for the KI levels using the final explanations. We described the X dimension as representing macroscopic to microscopic ideas about ecosystems and organisms. We observed that nodes that capture the scientific language and reasoning of student responses define the y-axis.
Fig. 7
ENA models for the students who started with intuitive ideas and partially linked ideas (idea labels are shown in Fig. 5)
We examined the differences between the models for the four rounds of the dialog to understand how the connections among ideas in students’ responses changed as they progressed through the activity (Fig. 7). In the initial explanation, both groups made the common link between the idea that the Sun helps plants survive (SunPlt) and the idea that Animals eat plants for food and nutrition (Foodchain). When students link these two ideas, their explanation usually includes a common entry point, “Plants get energy from the sun and then animals eat plants.” The ENA representations show that students form multiple links in both groups, as they respond to prompts in the dialog (Fig. 7). The group starting with intuitive ideas ends up with a higher KI average score (Intuitive Group, MInitial = 1.87, MFinal = 2.54). In contrast, the group starting with partial links does not gain in KI average score (Partially Linked Group, MInitial = 3.00, MFinal = 2.95).
For students who started with Partially Linked Ideas, the ENA representation captures additions of microscopic and mechanistic ideas. Some of the students who started with Partially Linked Ideas had connections to microscopic ideas. For this group, more students made links to macroscopic ideas during the dialog as shown by additional thin lines between the ideas.
For instance, in Fig. 7-Partial Trajectory-D1, they used scientific mechanisms of light energy being transformed into chemical energy during photosynthesis (PhotoEngTr) to connect to how animals get energy from plants when they eat them (EngTran). The links of these mechanistic ideas are thicker compared to E1. This suggests that some students link to the micro level by adding details of energy transfer after receiving an adaptive prompt. In Fig. 7-Partial Trajectory-D2, students connect the micro and macro levels more often compared to D1, such as the link between Energy Transfer (EngTran) and the Sun helps plants survive (SunPlt) and Energy transformation in photosynthesis (PhotoEngTr) and Foodchain. To connect these ideas, students think about both the transfer of matter and the transformation of energy along the food chain when animals eat plants. In Fig. 7-Partial Trajectory-E2, students continued adding more microscopic ideas to the links but did not continue linking mechanistic ideas. The ENA shows that the students made many links between macro and micro levels, using descriptive language rather than mechanistic language, which explains why the mean for this group remains at the KI 3 level.
In the group that starts with Intuitive Ideas, the strongest connections in Fig. 7-Intuitive Trajectory were between the Sun helping plants survive and then animals eating plants (SunPlt and Foodchain) at all four-time points. Both ideas are at the macroscopic level. Between the initial and final explanation, the NLP adaptive prompts at Dialog 1 and 2 helped students build on this macroscopic level link by connecting micro-level ideas about Energy transfer (EngTran) and mechanistic language about energy transformation during photosynthesis (PhotoEngTr), as evidenced by new thin lines between these ideas after then second prompt and for the final explanation. For students who started with Intuitive Ideas, the dialog provided various scaffolds to help them add ideas. The ideas could be linked at a microscopic-descriptive level, such as “The Sun helps plants survive then animals eat the plants to get their energy” (SunPlt and EngTran, KI 3), or a macroscopic-mechanistic level, such as “The Sun helps plants survive by photosynthesis (SunPlt and Photo, KI 3).
In summary, comparing the ideas added by students who started with intuitive ideas to those who started with partial links, the ENA shows that the intuitive group and the partial group both add some ideas. The intuitive group primarily adds ideas to the final explanation as shown by node size. The partial group adds ideas more gradually as shown by the increasing size of the nodes. Comparing the links added by students who started with intuitive ideas to those who started with partial links, the intuitive group has fewer added links than the partial group as shown in the number and thickness of the lines between ideas. Thus, the group that was already linking ideas added more links and the group that did not have many links also added a few links. This suggests that students who have not identified links may need a model of how to connect ideas as we have found in other research. Specifically, an annotator that provides a model of how to link ideas resulted in students who initially started with mainly intuitive ideas making significant progress (Gerard & Linn, 2022).
Discussion
This study uses ENA to illustrate how the adaptive dialog supports students in revising their explanations by building on their intuitive ideas and everyday experiences to add ideas (Nordine et al., 2011; Smith et al., 1993). The results align with previous research on Knowledge Integration (KI) pedagogy, indicating that students often hold multiple descriptive, mechanistic, microscopic, and macroscopic ideas simultaneously (e.g., Ryoo & Linn, 2012). ENA representations demonstrate that students who achieve KI levels 3 and 4 have more idea connections than students who achieve KI levels 1 and 2. These connections include descriptive language at the macroscopic level as well as mechanistic language at the microscopic level. In this activity, which does not include any instruction, students generate ideas in response to dialog prompts. The resulting set of ideas is consistent with prior research showing that students often have fragmented knowledge including a mixture of personal experiences and scientific terms from prior instruction (diSessa & Sherin, 1998).
The ENA idea models for KI levels offer a snapshot of the impact of the adaptive dialog. Adaptive prompts scaffolded students to add microscopic ideas. Many students also connect the microscopic ideas to mechanistic explanations leading to higher KI scores. The KI-based adaptive guidance is akin to a conversation with a knowledgeable peer who motivates their partner to express more ideas and distinguish among the ideas. This aligns with the description of the zone of proximal development as described by Vygotsky (1978). In the Vygotsky work, as in this activity, there is no instruction. The intervention prompts students to consider more of their ideas and to think about how the ideas fit together.
Future research could investigate potential adjustments to the adaptive dialog to better support students who start with partially linked ideas. The results show that students who did not progress often lacked mechanisms to explain the phenomena. However, since the activity was not designed to introduce mechanisms, it may be that the students did not have the disciplinary knowledge. Another question concerns the length of the adaptive dialog. Future research could explore ways to add additional interactions with the avatar.
Methodologically, this study investigates the novel combination of ENA and NLP learning analytics. We use ENA to analyze datasets comprising automatically generated analyses of student explanations. The ENA representations illustrate the sorts of connections that arise as students respond to adaptive prompts. In previous ENA studies, researchers reported that developing idea rubrics and coding discourse data is time-consuming (Elmoazen et al., 2022). Our study takes advantage of state-of-the-art automation assessment tools to generate codes and scale up the application of ENA. We provide an early-stage example of the use of ENA to represent trajectories of student KI and to capture the links characteristic of students at each KI level. Future work could explore how ENA can be integrated with automation technologies to represent idea networks in real time in classrooms.
Limitations
The results of this study are limited to an analysis of students from one school and two teachers. This is due to the ENA web tool’s limit of 1000 conversations with stanza size 1. As a result, the findings can mainly be generalized to this population. Nonetheless, the results offer valuable insights into ways that ENA can represent differences in the relationships among ideas in KI scores and for revision trajectories of students starting with intuitive or partially linked ideas. Further, ENA visualizes students’ idea models along two dimensions, potentially overlooking additional dimensions that may stem from the complexity of understanding photosynthesis along with the impact of students’ diverse languages and cultures (e.g. Holtmann et al., 2023).
Conclusions
In this study, we utilized ENA to represent idea connections across revisions of knowledge integration explanations. We compared the idea network models of student explanations at different KI levels and their responses to adaptive prompts during the activity. Additionally, ENA helped identify significant dimensions among the students’ ideas about photosynthesis and cellular respiration as captured in the axes for the models. The ENA also captures the links that were added as students integrated their ideas about energy transfer in photosynthesis and cellular respiration. For instance, when learning energy transfer, as scores increased, there were more links between microscopic and mechanistic ideas, capturing the ways that student ideas become integrated. The ENA also reinforced KI theory indicating that there are multiple pathways from intuitive to integrated understanding. Specifically, KI 1 and 2 groups primarily focus on intuitive and broad concepts at the macroscopic level, while KI 3 and 4 groups integrated ideas may center on core microscopic mechanisms. Future applications could involve directly integrating NLP output into ENA and embedding this learning analytics within instruction technologies, empowering both teachers and students to access idea network models and trajectories for tailored instruction.
Declarations
Ethical Approval
All procedures performed in studies involving human participants received Institutional Review Board (IRB) of the University of California Berkeley approval for human subject research (IRB No. 2021–06-14389).
Consent to Participate
Informed consent was obtained from all participants and was obtained in alignment with the IRB approval.
Consent for Publication
Not applicable.
Competing Interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Akumbu, R. V., Lux, K., Schulz, D., Espino, D., & Hamilton, E. (2023). Cultural impact on a global virtual STEM project. In G. Arastoopour Irgens & S. Knight (Eds.), Advances in Quantitative Ethnography (pp. 96–111). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-47014-1_7
Bradford, A., Li, W., Riordan, B., Steimel, K., & Linn, M. C. (2023). Adaptive dialog to support student understanding of climate change mechanism and who is most impacted. In Proceedings of the 17th International Conference of the Learning Sciences-ICLS 2023, pp. 816–823. International Society of the Learning Sciences. https://repository.isls.org//handle/1/10333
Chang, H.-Y., & Tsai, C.-C. (2023). Epistemic network analysis of students’ drawings to investigate their conceptions of science learning with technology. Journal of Science Education and Technology,32(2), 267–283. https://doi.org/10.1007/s10956-022-10026-9CrossRef
Chowdhary, K. (2020). Natural language processing. In Fundamentals of artificial intelligence (pp. 603–649). Springer.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understandinghttps://doi.org/10.48550/arXiv.1810.04805
diSessa, A. A. (2018). A friendly introduction to “knowledge in pieces”: Modeling types of knowledge and their roles in learning. In G. Kaiser, H. Forgasz, M. Graven, A. Kuzniak, E. Simmt, & B. Xu (Eds.), Invited Lectures from the 13th International Congress on Mathematical Education (pp. 65–84). Springer International Publishing. https://doi.org/10.1007/978-3-319-72170-5_5
Elmoazen, R., Saqr, M., Tedre, M., & Hirsto, L. (2022). A systematic literature review of empirical research on epistemic network analysis in education. IEEE Access,10, 17330–17348.CrossRef
Ferreira, R., Kovanović, V., Gašević, D., & Rolim, V. (2018). Towards combined network and text analytics of student discourse in online discussions. In C. Penstein Rosé, R. Martínez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, & B. du Boulay (Eds.), Artificial Intelligence in Education (pp. 111–126). Springer International Publishing.
Gerard, L. F., & Linn, M. C. (2016). Using automated scores of student essays to support teacher guidance in classroom inquiry. Journal of Science Teacher Education,27(1), 111–129. https://doi.org/10.1007/s10972-016-9455-6CrossRef
Gerard, L., Bradford, A., & Linn, M. C. (2022b). Supporting teachers to customize curriculum for self-directed learning. Journal of Science Education and Technology,31(5), 660–679. https://doi.org/10.1007/s10956-022-09985-wCrossRef
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers,36(2), 180–192. https://doi.org/10.3758/BF03195563CrossRef
Graesser, A. C., Penumatsa, P., Ventura, M., Cai, Z., & Hu, X. (2007). Using LSA in AutoTutor: Learning through mixed initiative dialogue in natural language. In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 243–262). Erlbaum.
Holtmann, M., Gerard, L., Li, W., Linn, M. C., & Riordan, B. (2023). How does an adaptive dialog based on natural language processing impact students from distinct language backgrounds? In P. Blikstein, J. Van Aalst, R. Kizito, & K. Brennan (Eds.), Proceedings of the 17th International Conference of the Learning Sciences—ICLS 2023 (pp. 1350–1353). International Society of the Learning Sciences. https://doi.org/10.22318/icls2023.921177
Li, W., Liao, Y., Steimel, K., Bradford, A., Gerard, L., & Linn, M. (in press, 2024). Teacher-informed expansion of an idea detection model for a knowledge Integration assessment. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (L@S ’24), July 18–20, 2024, Atlanta, Georgia, USA. ACM, New York, NY, USA. https://doi.org/10.1145/3657604.3664687
Li, M., Kamaraj, A. V., & Lee, J. D. (2023). Modeling trust dimensions and dynamics in human-agent conversation: A trajectory epistemic network analysis approach. International Journal of Human–Computer Interaction, 0(0), 1–12. https://doi.org/10.1080/10447318.2023.2201555
Li, W., Bradford, A., & Gerard, L. (2023). Responding to students’ science ideas in a natural language processing based adaptive dialogue [poster presentation]. 2023 American Educational Research Association (AERA) Annual Meeting
Li, W., Gerard, L., Lim-Breitbart, J., Bradford, A., Linn, M. C., Riordan, B., & Steimel, K. (2023). Explaining thermodynamics: Impact of an adaptive dialog based on a natural language processing idea detection model. In P. Blikstein, J. Van Aalst, R. Kizito, & K. Brennan (Eds.), Proceedings of the 17th International Conference of the Learning Sciences—ICLS 2023 (pp. 1306–1309). International Society of the Learning Sciences. https://doi.org/10.22318/icls2023.199424
Linn, M. C., Donnelly-Hermosillo, D., & Gerard, L. (2023). Synergies between learning technologies and learning sciences: Promoting equitable secondary school teaching. Routledge.CrossRef
Linn, M. C., & Eylon, B.-S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. Routledge.
Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching,53(2), 215–233. https://doi.org/10.1002/tea.21299CrossRef
Nordine, J., Krajcik, J., & Fortus, D. (2011). Transforming energy instruction in middle school to support integrated understanding and future learning. Science Education,95(4), 670–699. https://doi.org/10.1002/sce.20423CrossRef
Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education,24(4), 427–469. https://doi.org/10.1007/s40593-014-0029-5CrossRef
Omarchevska, Y., Wörner, S., Pavlou, Y., Papaevripidou, M., Zacharia, Z., Puntambekar, S., ..., & Linn, M. C. (2021). Science learning with virtual experiments. Proceedings of the 15th International Conference of the Learning Sciences (ICLS).
Ozili, P. K. (2023). The acceptable R-square in empirical modelling for social science research. In Social Research Methodology and Publishing Results: A Guide to Non-Native English Speakers (pp. 134–143). IGI Global. https://doi.org/10.4018/978-1-6684-6859-3.ch009
Paladines, J., & Ramirez, J. (2020). A systematic literature review of intelligent tutoring systems with dialogue in natural language. IEEE Access,8, 164246–164267.CrossRef
Peters-Burton, E. E., Dagher, Z. R., & Erduran, S. (2023). Student, teacher, and scientist views of the scientific enterprise: An epistemic network re-analysis. International Journal of Science and Mathematics Education,21(2), 347–375. https://doi.org/10.1007/s10763-022-10254-wCrossRef
Rachmatullah, A., & Wiebe, E. N. (2022). Building a computational model of food webs: Impacts on middle school students’ computational and systems thinking skills. Journal of Research in Science Teaching,59(4), 585–618. https://doi.org/10.1002/tea.21738CrossRef
Ryoo, K., & Linn, M. C. (2012). Can dynamic visualizations improve middle school students’ understanding of energy in photosynthesis? Journal of Research in Science Teaching,49(2), 218–243.CrossRef
Sayary, A. M. A. E., Forawi, S. A., & Mansour, N. (2015). STEM education and problem-based learning. Routledge.
Schulz, C., Eger, S., Daxenberger, J., Kahse, T., & Gurevych, I. (2018). Multi-task learning for argumentation mining in low-resource settingshttps://doi.org/10.48550/arXiv.1804.04083
Schulz, C., Meyer, C. M., & Gurevych, I. (2019). Challenges in the automatic analysis of students’ diagnostic reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1) 1 https://doi.org/10.1609/aaai.v33i01.33016974
Shaffer, D. W. (2018). Big data for thick description of deep learning. Routledge.CrossRef
Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics,3(3), 3. https://doi.org/10.18608/jla.2016.33.3CrossRef
Shaffer, D. W. (2014). Formatting data for epistemic network analysis. Games and Professionals Imulations (Gaps) Technical Report Series, 1, 1–15.
Smith, E. L., Blakeslee, T. D., & Anderson, C. W. (1993). Teaching strategies associated with conceptual change learning in Science. Journal of Research in Science Teaching,30(2), 111–126. https://doi.org/10.1002/tea.3660300202CrossRef
Sun, M., Wang, M., Wegerif, R., & Peng, J. (2022). How do students generate ideas together in scientific creativity tasks through computer-based mind mapping? Computers & Education,176, 104359. https://doi.org/10.1016/j.compedu.2021.104359CrossRef
Talafian, H., & Kang, H. (2023). Modeling students’ performances in physics assessment tasks using epistemic network analysis. In C. Damşa & A. Barany (Eds.), Advances in Quantitative Ethnography (pp. 285–298). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-31726-2_20
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist,46(4), 197–221.CrossRef
Vitale, J. M., McBride, E., & Linn, M. C. (2016). Distinguishing complex ideas about climate change: Knowledge integration vs specific guidance. International Journal of Science Education,38(9), 1548–1569. https://doi.org/10.1080/09500693.2016.1198969CrossRef
Vygotsky, L. L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Wooldridge, A. R., Carayon, P., Shaffer, D. W., & Eagan, B. (2018). Quantifying the qualitative with epistemic network analysis: A human factors case study of task-allocation communication in a primary care team. IISE Transactions on Healthcare Systems Engineering,8(1), 72–82. https://doi.org/10.1080/24725579.2017.1418769CrossRef
Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020). Applying machine learning in science assessment: A systematic review. Studies in Science Education,56(1), 111–151. https://doi.org/10.1080/03057267.2020.1735757CrossRef
Zörgő, S., Swiecki, Z., & Ruis, A. R. (2021). Exploring the effects of segmentation on semi-structured interview data with epistemic network analysis. In A. R. Ruis & S. B. Lee (Eds.), Advances in Quantitative Ethnography (pp. 78–90). Springer International Publishing. https://doi.org/10.1007/978-3-030-67788-6_6
