Utilizing Deep Learning AI to Analyze Scientific Models: Overcoming Challenges

Data for this study were collected in the years of 2022 and 2023 from an NSF funded project. The project examines the effect of a formative assessment system that automatically generates feedback based on students’ open-ended assessment responses in chemistry and physics consistent with a previously developed learning progression that describes the successively more complex understandings students can develop about electrical interactions.

In this study, students engaged in scientific modeling through a multimodal online platform that allows them to construct models using a combination of visual and textual representations. The platform is designed to support multiple modes of expression to enhance accessibility and cognitive engagement in modeling practices.

Alongside their visual models, students were prompted to provide written explanations to articulate their reasoning. These textual responses allowed students to elaborate on the relationships between model components, explain causal mechanisms, and clarify any conceptual aspects that might not have been fully conveyed in their drawings. This integration of textual and visual representation aligns with multimodal learning theories, which emphasize that combining different modes of expression can enhance cognitive processing and deepen conceptual understanding (Kress, 2009; Schwarz et al., 2017).

This multimodal design aims to provide students with flexible options for constructing models while also enabling researchers to examine how different representational choices influenced students’ ability to express their scientific reasoning. The combination of structured (icon-based) and open-ended (freehand drawing and text) modalities was particularly useful in capturing a range of student modeling practices and challenges, as discussed in our analysis of AI-human scoring discrepancies.

We focus on one modeling task, named “Electroscope modeling¹” (Fig. 1), in the curriculum materials for high school students. Students responded to the task during their unit learning process. Aligned with the Next Generation Science Standards (NGSS Lead States, 2013), this task was designed to assess students’ proficiency on the performance expectation (PE): HS-PS2-4, with specific focus on the Coulomb’s law part of the PE. The three dimensions of scientific knowledge reflect in this item include: Types of interactions (DCI), Developing and using models (SEP), and Cause and effect (CCC). The item was designed to assess students’ usable knowledge of constructing a model to represent what causes neutral objects to become charged when put in contact with the charged objects and how the magnitude of the charge on the charged object affects the observations.

Full size image

The assessment task was designed using an evidence-centered design (ECD, Mislevy & Haertel, 2006) approach to gather performance-based evidence to assess students’ usable knowledge. For the electroscope modeling task, students were first provided a video that shows when a charged rod gets close to or touches a metal ball on the electroscope, the foil leaves of the electroscope move apart. Then the video presents two scenarios: Scenario A shows when a charged rod touched the ball, the foil leaves of the electroscope move away from each other, while in Scenario B shows when a charged rod touches the ball and makes the leaves move much further part (see Fig. 1). The electroscope modeling task asks students to draw a model to show and explain what the differences are in the rod and electroscope in the two scenarios that cause the observations.

We designed a holistic rubric to understand students ‘ levels of knowledge-in-use proficiencies aligned with a validated learning progression (Kaldaras et al., 2021). For the electroscope modeling task, students’ models should show point charge transfer from the charged rod to all components of the electroscope and the magnitude of the repulsive force between the leaves to be larger for scenario B compared to A. To comprehensively assess students’ performance, we designed an analytic rubric for the electroscope modeling task by adapting a rubric deconstruction process for three-dimensional (3D) explanations that should integrate DCIs, CCCs, and SEPs (Kaldaras et al., 2022). The analytic rubric was developed based on ECD approach to capture the nuances challenges of model development, which ensures that each rubric category specifies evidence that reflects students’ understanding of the underlying scientific concepts (Kaldaras et al., 2023, 2024). The rubric aims to capture meaningful evidence from students’ models, not to evaluate their responses against an ideal solution. For this modeling task, we focused on identifying how students represent charge distribution and static equilibrium using scientific principles, while acknowledging that students can develop various appropriate models.

We developed a 13-category analytic rubric (Table 1) to capture students’ knowledge-in-use performance on this task, among which categories 1–10 are designed to identify presence or absence of critical components of the model as detailed in the ECD argument. In our context, this is represented by charges on each rod and electroscope part (rod, sphere, hook, leaves) in scenarios A and B respectively (Categories 1–4 & 6–9). Categories 6–10 emphasize the relationship between components that leads to the causal mechanism, specifically that there should be more charges on each part of the electroscope in B as compared to scenario A. Categories 5 and 10 reflect presence or absence of model components and their relationship between among which categories 1–10 identify presence or absence of charges on each rod and electroscope part (rod, sphere, hook, leaves) in scenarios A and B respectively (Categories 1–4 & 6–9). Categories 6–10 emphasize that there should be more charges on each part of the electroscope in B as compared to scenario A. Categories 5 and 10 reflect presence or absence of model components indicating direction and magnitude of the force between the electroscope leaves. Categories 11–13 capture inaccuracies of students’ responses to identify common mistakes in models and for further feedback design. For instance, category 13 is about “Either the rod in scenario A is not charged or the whole electroscope is not charged in scenario A.” This is a common representation from students, although scientifically inaccurate, especially when learning the necessary ideas. To address concerns about whether the 13 categories are distinct, we carefully reviewed the rubric during its iterative development. Each category captures distinct features of the model, such as the presence and placement of charges or relationships between components. These distinctions were validated through expert reviews and scoring training practices, which confirmed that the categories address key aspects of student performance without overlap.

Our analytic rubric emphasizes the use of evidence-based reasoning rather than defining a perfect scientific model. For example, it evaluates whether students correctly depict the movement of charges and their effects on other components, even if their representations are incomplete or imprecise. By capturing several common ideas or inaccuracies, such as inconsistent charge placement or misunderstandings of charge interactions, the rubric helps identify areas where students need support.

To ensure the reliability of human scoring for validating automated systems, we initially evaluated approximately 200 student models using the rubric. This initial step confirmed the rubric's effectiveness in assessing student modeling proficiency. Following minor adjustments and a comprehensive review by all authors, the refined rubric—enriched with detailed scoring categories and model examples—was used for rater training.

A scoring team, led by the first author and including three research assistants with science education backgrounds, was trained extensively. Training encompassed a detailed rubric walkthrough, the project's background, and the scoring objectives, supplemented by scoring exercises using student models. The scoring process was structured into two phases: training and formal scoring. In the training phase, two rounds of scoring were conducted. Round 1 involved the facilitator presenting 30–35 preselected responses to familiarize raters with the rubric categories, followed by a group scoring session of 25 responses to refine their understanding (Bejar et al., 2006). Round 2 introduced 25 randomized responses for independent scoring by the raters, allowing for a critical evaluation and adjustment of the rubric based on the scoring outcomes (Nehm et al., 2012).

The formal scoring of the students’ electroscope modeling task applied the rubric across 13 categories (detailed in Table 1). Binary scores (0 or 1) were assigned based on the rubric criteria. Krippendorff's alpha was used to measure interrater reliability; responses were further reviewed until an alpha of ≥ 0.8 was consistently achieved, reflecting a reliable consensus (Krippendorff, 2011). Discrepancies were resolved through team discussions, ensuring the scoring reflected collective evaluative standards (Bejar et al., 2007; Nehm et al., 2012). The final human agreement rates on the rubric categories are documented in Table 2.

In this study, we employ Convolutional Neural Networks (CNNs) to analyze and score students' models. Drawing inspiration from the biological visual systems of animals, CNNs are adept at mimicking the visual system's capability to discern spatial hierarchies of image features, from basic to complex patterns (Fukushima & Miyake, 1982; Krizhevsky et al., 2017; LeCun et al., 1998). Their architecture is particularly suited for computer vision tasks such as image classification (Krizhevsky et al., 2017) and object detection (Girshick et al., 2014), making them an ideal choice for our application. A CNN architecture comprises several layers, each performing specific functions (Xu et al., 2023). Convolutional layers, responsible for feature extraction, use small, learnable filters to detect features such as edges and textures. Pooling layers then reduce these feature maps in size, enhancing computational efficiency and the network’s robustness to feature variations. The identified high-level features are used by fully connected layers to generate predictions or classifications.

Our CNN model, detailed in Fig. 3, is a feedforward network optimized for image recognition. It leverages a structured layer arrangement allowing deep, multi-level abstraction of image information. This capability is particularly advantageous for evaluating students' drawn models, where the CNNs identify and analyze model components, their relationships, and explanations. This approach enables a comprehensive and nuanced analysis of student work, ensuring assessments are both detailed and objective. By using CNNs, we can achieve a more reliable and nuanced analysis of students' drawn models, reflecting their understanding and skill level in a way that traditional scoring methods might miss.

Full size image

To respond to the RQ1, we first developed a CNN-based algorithm based on the architecture presented above in Fig. 3. Our data analysis process includes a training stage which uses tenfold cross validation and independent testing stages.

To respond to RQ2, we performed a three-step experiment to explore the potential of overcoming the challenges in scoring educational data that measures modeling practice. Figure 4 presents the mechanism of strategies. We evaluated cross-validation and testing accuracies for three training strategies to assess their effectiveness in addressing the challenges of automated scoring of student models.

Full size image

To investigate where and why the AI system diverged from human analysis (RQ3), we conducted a structured qualitative analysis of scoring discrepancies. This approach was chosen because analyzing knowledge-in-use proficiency requires evaluating sensemaking novelty rather than localized visual features, rendering pixel-attention methods (e.g., Grad-CAM) ineffective for explaining misalignments. Our CNN-based model, designed to extract hierarchical structural patterns (e.g., diagram composition, semantic relationships), inherently prioritizes statistically frequent features over unconventional expressions.

Using a predefined threshold informed by pilot studies of human inter-rater variability, we identified cases where AI scores substantially deviated from human ratings. To ensure representativeness, the research team randomly sampled these cases from the full dataset after stratifying them by rubric category. Three researchers independently analyzed discrepancies using the constant comparative method (Glaser, 1965). They began with open coding to generate initial themes (e.g., "AI penalizes unconventional diagrams"), progressed to axial coding to group themes into broader categories (e.g., "bias toward normative structures"), and concluded with selective coding to synthesize core explanations (e.g., "prioritization of statistically frequent patterns"). The team resolved disagreements through iterative discussions until achieving consensus.

To systematically investigate these limitations, we focused on outputs from the most effective AI strategy (Strategy 3), analyzing inconsistencies between human and machine scores. We first identified inconsistent cases within each of the 13 rubric categories, systematically documenting divergences where AI scores deviated from human judgments. Within each category, we detected recurring patterns in these discrepancies—for example, observing how the AI penalized unconventional diagram layouts that human raters praised as innovative, or misclassified metaphorical reasoning as vagueness. After establishing category-specific patterns, we broadened our analysis to synthesize common themes across all 13 categories. This cross-category synthesis revealed systemic evaluation gaps, such as the AI’s inability to recognize context-dependent creativity in interdisciplinary models or its overreliance on lexical complexity as a proxy for scientific rigor.

By anchoring our analysis in Strategy 3’s outputs—the highest-performing AI approach—we ensured observed discrepancies stemmed from inherent algorithmic limitations rather than suboptimal model training. This two-tiered process (category-specific pattern detection followed by cross-category synthesis) allowed us to pinpoint how and why the AI’s hierarchical feature extraction—prioritizing edges, shapes, and statistically frequent compositional norms—clashed with human evaluators’ emphasis on relational novelty (e.g., analogies, speculative hypotheses).

This case analysis approach directly contrasts with heatmap-based methods, which focus on localized spatial attention rather than conceptual relationships. For instance, while a heatmap might highlight the AI’s attention to technical terms like “photosynthesis,” it cannot explain why the system undervalued a student’s creative analogy linking photosynthesis to “nature’s recipe book.” Our methodology, by systematically dissecting discrepancies across 13 categories, provides actionable insights into the AI’s failure to align with human creativity assessment paradigms.

We employed a tenfold cross-validation process during the training phase and tested the developed algorithms using independent testing datasets. We examine several critical performance metrics for both phases by comparing the human and machine assigned scores to each response: accuracy, precision, recall, and F1 score (see Table 4), which are vital for evaluating the efficacy of machine learning algorithms.

Accuracy is defined as the proportion of total correct predictions (both true positives and true negatives) made by the algorithm relative to the total number of cases. High accuracy, typically above 0.85, indicates substantial agreement between the model and human raters, signifying that the algorithm effectively replicates human judgment patterns (Japkowicz & Shah, 2011). In our training phase, the models displayed high accuracy across all categories, with values ranging from 0.87 to 0.97. However, reliance on accuracy in highly imbalanced categories, such as Category 11, may not fully capture the model’s effectiveness due to the predominance of the majority class. For instance, despite Category 11 achieving a training accuracy of 0.93 with a narrow 95% confidence interval (CI) of (0.92, 0.93), this metric alone could obscure the model's potential shortcomings in handling minority class predictions. During the testing phase, while accuracy remained robust overall, ranging from 0.83 to 0.97, this measure exhibited more variability across the categories, which highlights some of the challenges of generalizing some models to new data. Specifically, Category 1 maintained a high accuracy from training to testing, with a testing accuracy of 0.92 with a 95% CI of (0.87, 0.97). This indicates the developed algorithm has better stable performance in more balanced data conditions found in Category 1. These examples underscore the importance of tuning algorithms specifically for the diversity of data distributions encountered in educational assessments. The observed variations in the 95% confidence intervals between the training and testing phases indicate increased uncertainty in the algorithm's performance during the testing phase. This increased uncertainty could stem from differences in sample distributions or the presence of unaccounted-for variables in the testing data, which might not be as controlled as the training set. Consequently, while the training results suggest a high degree of confidence in the algorithm’s predictive accuracy, the broader confidence intervals during testing suggest a lower level of certainty in these estimates when the model is applied to new data.

Precision measures the accuracy of positive predictions, calculated as the ratio of true positives to the sum of true positives and false positives (Sokolova & Lapalme, 2009). This metric fundamentally reflects the algorithm’s reliability in making correct positive identifications. During the training phase, our algorithms showcased impressive precision, with Category 3 using SMOTE strategy achieving as high as 0.97, underscoring the algorithm’s reliability in a controlled setting. However, during the testing phase, precision generally showed a slight decline, suggesting areas for potential enhancement in algorithm robustness. For instance, Category 12 without augmentation displayed the lowest precision in the testing phase at 0.45, significantly lower than its training precision of 0.62. This indicates a substantial drop and highlights the model’s challenges in accurately finding true cases of this feature in unseen data. Conversely, Category 5 with SMOTE maintained high precision during testing at 0.94, nearly mirroring its training performance, which suggests robust generalization capabilities for this category.

Recall or sensitivity, a critical metric, quantifies the proportion of actual positives accurately identified by the model, reflecting its ability to detect all relevant instances (Powers, 2011). During the training phase, the recall was impressively high across the first ten categories, showcasing the algorithms' proficiency in capturing pertinent positive cases. For example, Category 2 achieved the highest recall rate at 0.96 when SMOTE was applied, demonstrating superior sensitivity in identifying positive instances. However, Categories 11 and 12 exhibited lower recall when using approaches without augmentation and with general augmentation. Implementing SMOTE significantly improved recall in these categories, indicating its effectiveness in handling highly imbalanced datasets. Despite these gains in training, the testing phase presented challenges, with recall dropping to 0.55 and 0.65, respectively. This variability highlights the models' decreased ability to consistently identify relevant cases in uncontrolled and diverse data sets. Particularly in highly imbalanced categories like Category 12, recall fell dramatically in the testing phase to 0.65 from 0.92 during training, underscoring difficulties in generalizing learned patterns to new data. In contrast, Category 5 with SMOTE exhibited a stable performance, maintaining a high testing recall of 0.94, closely mirroring its training recall of 0.95. This suggests better adaptability of the developed algorithm in recognizing positive instances under varied testing conditions.

F1 score, the harmonic mean of precision and recall, serves as a crucial metric in assessing an algorithm's balanced accuracy, especially in scenarios where the classes are unevenly distributed (Van Asch & Daelemans, 2016). Throughout the training phase of applying SMOTE, the F1 scores demonstrated consistent robustness, ranging from 0.91 to 0.97, indicating that the models effectively recognized true positives while effectively minimizing false positives. For example, Category 3 with SMOTE achieved an F1 score of 0.97 during training, highlighting the algorithm's ability to maintain a balance between precision and recall, even though moderately imbalanced. During the testing phase, however, the F1 scores exhibited substantial variability, spanning from 0.56 to 0.94. This variation underscores the challenges faced when applying the trained models to new, uncontrolled data environments. For instance, Category 12 with SMOTE observed a drop in the F1 score to 0.68 during testing due to decreases in both precision and recall values. This F1 testing score is significantly lower than its training performance. This decrease can be attributed to the model's difficulty in generalizing the training insights to new datasets, particularly in highly imbalanced settings. Conversely, Category 5 with general augmentation maintained a high F1 score of 0.94 during testing, closely mirroring its training performance (F1 score = 0.95) and demonstrating effective generalization and reliability in more balanced conditions.

To address RQ2, we assess the efficacy of three AI strategies—no data augmentation, general data augmentation, and SMOTE—across 13 categories of student model data characterized by varying levels of imbalance. Each strategy was deployed to optimize the machine learning algorithms' performance in automatically analyzing students' scientific models, focusing particularly on how these strategies improved accuracy, precision, recall, and F1 scores given the unique challenges of each category's data distribution. Our analysis found that AI strategies are crucial in enhancing machine performance, especially where data imbalances might otherwise skew assessment outcomes. By tailoring AI techniques to the specific imbalances present, our algorithms exhibited substantial improvements across key performance metrics essential for gauging the efficacy of educational assessments.

For categories with severe data imbalances—Categories 8, 11, 12, and 13, where positive instances were well below the 20% mark—the implementation of SMOTE was vital. This strategy involves generating synthetic samples from the minority class, thereby balancing the training datasets and significantly boosting algorithm recall and overall accuracy. Prior to SMOTE implementation, these categories exhibited low recall (0.52 on average), indicating that the model frequently misclassified minority-class instances as negative cases. After SMOTE, recall improved substantially across these categories, increasing from 0.52 to 0.89, demonstrating the model’s improved ability to detect underrepresented responses.

In addition to recall, F1 scores also showed notable improvements, particularly in categories where positive cases were scarce. For example, in Category 11, the F1 score increased from 0.47 to 0.86, reflecting a more balanced performance between precision and recall. Importantly, these improvements were achieved without a significant drop in precision, suggesting that the synthetic samples generated by SMOTE effectively preserved the meaningful structure of student responses without introducing noise. The empirical evidence highlighted substantial improvements in recall and F1 scores, illustrating a reduced bias towards the majority class and fostering a fairer evaluation of minority class examples.

In categories with moderate data imbalances—such as Categories 2, 3, 7, and 9, where positive cases exceeded 10% but did not exceed 20%—we employed a combination of moderate data augmentation and class weight adjustments. These strategies were crucial in maintaining fairness and accuracy in algorithm predictions. Fairness refers to the ability of the algorithm to predict minority and majority classes with high accuracy, thus ensuring that the algorithms learn equitably from both class distributions. Particularly in Category 2, the application of SMOTE escalated recall from 0.83 to 0.93 during testing, and the F1 score improved to 0.94, underscoring the effectiveness of this approach in addressing skewed data distributions.

For Categories 4, 5, 6, and 10, where cases were fairly balanced with positive cases ranging between 20 and 30%, targeted interventions such as SMOTE learning for positive and negative examples. This improved the models’ accuracy and fairness, as seen in Category 4, where SMOTE raised testing precision and recall to 0.90 each. Conversely, categories that were near balanced or highly balanced—such as Category 1 —required fewer or no intensive balancing interventions. In these instances, even minor data augmentation strategies optimized performance. For instance, in Category 1, general data augmentation improved testing accuracy to 0.95 and elevated the F1 score to 0.95, demonstrating that subtle modifications in data handling could lead to significant gains in algorithm performance across various metrics.

We conducted constant comparative thematic analysis to identify instances where machine and human scores diverged, aiming to investigate the areas of disagreement between human and machine evaluations. Building on the findings reported for RQs 1 and 2, we identified several themes across categories and cases with discrepant scores.

Although the system was designed so that students could use stamps instead of free-style drawing, some students still chose to draw freely. In examining where AI and human ratings disagree with each other in analyzing student scientific models, we identified a significant challenge in interpreting students' free-style drawings. See the sample student responses in Fig. 5. Across all 13 categories, discrepancies between machine and human ratings were predominantly found in the interpretation of free-style drawing models. While drawings is an effective means of presenting creativity and higher-order thinking skills, they can present substantial challenges in automated scoring systems due to broad diversity. This discrepancy highlights the limitations of automated scoring systems in dealing with unstandardized and highly individualized student responses. Specifically, automated scoring systems often fail to accurately recognize and value expressions that human experts consider creative or insightful. For instance, in cases involving symbolic or abstract representations (see models in Fig. 5a, b, and c), automated systems might assign lower scores due to their inability to understand the context and deeper meanings of these visual elements. Moreover, the consistency and reliability of machine scoring are challenged by the diverse visual styles students use to express the same scientific concept. For instance, for category 1, students have different approaches to show charges on the rod using their own approaches instead of using the pre-designed icons as part of the online curriculum system interface (e.g., 5a and 5b), which increases the likelihood of the machine's algorithm to appropriately analyze the ideas. For most of the models in the dataset, students used the pre-designed icons to indicate important features relevant to the scenario. These provide a more consistent representation for the AI algorithm to learn but also result in hand-drawn elements for the same features that are more diverse and less represented within the same dataset. Figure 5 shows a subset of models that include hand-drawn features to represent various model elements, with some models also including pre-defined elements. For instance, model 5c utilizes pre-designed arrow icons and hand-drawn arrows in orange to depict the repulsive forces between the foil leaves due to like charges, whereas model 5b employs drawn lines to represent fields resulting from charges. Although these hand-drawn features might have similar sketch lines, they convey different meanings in scientific models, which could influence the machine's interpretation.

Full size image

The second significant theme that emerged is the mixed representation of both neutral, negative and positive charges in students' models (see Fig. 6). This theme encompasses various situations. For instance, Fig. 6a shows one situation where the model includes both types of charges, but each scenario features only one type of charge—neutral in scenario A and negative in scenario B. In Fig. 6b, within each scenario, the model includes both types of charges, but similar charges are located within discrete parts of the electroscope. In contrast, Fig. 6c presents a situation where the model displays both types of charges across scenarios a and b, with the two types of charges mixed across different parts of the electroscope. Variations in how students depict these charges can lead to significant discrepancies in machine versus human scoring. The difficulty lies in the machine's ability to interpret these mixed representations, often due to a lack of sufficiently varied examples in the training data. This theme is especially pronounced in categories 11 through 13, designed to understand students' incomplete or inaccurate ideas. In these categories, the machine should be able to recognize the placement of charges, the type of charge, and whether the type of charge is consistent throughout the model. Thus, the AI model must synthesize diverse information to determine the correct scores, which can be challenging for the automated system because these student ideas are not well-represented in the training set. Take, for example, category 11, which focuses on models showing both types of charges in various parts of the electroscope across different scenarios. However, this category includes an exceptional situation when human coders analyze student models; they realize that “This can be ignored if positive and negative charges are not accumulated in specific locations.”, a nuance that automated systems frequently miss. This and similar cases in categories 11 to 13 highlight the importance of designing coding rubrics that capture or identify these unusual ideas from students’ models rather than excluding this specific type of response. These responses showcase students' diverse ideas and plausible misunderstandings, which teachers need to explore further to provide appropriate support. For AI, the limited number of such responses results in insufficient data for training, leading to suboptimal performance. To ensure these students' ideas are not excluded, it is crucial to have a rubric that captures these responses, perhaps in another defined rubric category, and to use data augmentation to enhance these cases for further AI algorithm development. Enhancing the dataset in this way is crucial for improving the accuracy and consistency of machine assessments and ensuring that automated systems can effectively support an equitable and effective educational assessment environment.

Full size image

A third significant theme identified in our thematic analysis involves the unclear depiction of charge locations in student models. This ambiguity poses a substantial challenge for the algorithms, which rely on clear indicators to assess and credit student responses accurately. This theme emerged particularly in scenarios where students were expected to demonstrate their understanding of charge distribution and its effects (e.g. categories 1–4 and 6–9). In many instances, students' models did not clearly specify the locations of positive or negative charges, leading to uncertainty and error in machine scores. In cases where charges should be distinctly marked to indicate the charges’ location on rod, metal ball, hook, or foil leaves (Fig. 7), vague or ambiguous placement made it difficult for the machine to determine whether the student's response was correct or flawed. For example, the student model in 7a shows a point charge between the ball and the hook of the electroscope device. Similarly, student model 7b shows point charges near and in between the foil leaves. These examples may even lead to disagreements between humans about whether a point charge is located “on” a specific part of the device as opposed to “near” a given part. This lack of clarity often resulted in the machine either wrongly crediting or failing to credit responses that might have been recognized by human scorers who could infer the intended meaning from the context or subtle cues within the drawings.

Full size image

Our research demonstrates that deep learning AI can replicate human judgment with high fidelity across accuracy, precision, recall, and F1 scores, reinforcing the potential of AI in enhancing educational assessments (Lee et al., 2021; Liu et al., 2016; Wilson et al., 2024). Despite the high accuracy rates ranging from 87 to 97% during the training phase, the study identified significant variability in accuracy during testing, especially in categories with high data imbalance. These finding calls attention to the challenges in generalizing AI models to new, diverse data sets, resonating with concerns raised by Li, Adah Miller, and He (2024) about the limitations of traditional metrics like accuracy in imbalanced scenarios. To counteract this, we emphasize the importance of alternative metrics such as precision, recall, and F1 scores, which provide a more nuanced understanding of model performance, particularly in handling minority class instances.

The study also explores the role of AI strategies, particularly data augmentation and SMOTE, in mitigating the effects of data imbalance, which are common in educational data sets. We found that AI strategies can enhance machine performance, especially where data imbalances might otherwise skew assessment outcomes. By tailoring AI interventions to specific imbalances, our algorithms showed significant improvements in key performance metrics, supporting the conclusions of Zhai, He, & Krajcik (2022) on the adaptive capabilities of AI. This strategic customization enhances the fairness and robustness of the resulting machine algorithm and ensures that AI can adapt to the complexities of educational data, thereby supporting more accurate and equitable evaluations of student models. These findings align with prior studies that have applied SMOTE in high-stakes educational assessments (Abu Zohair, 2019) and computer vision for student engagement analysis (Khan et al., 2021). While SMOTE was instrumental in improving performance for highly imbalanced categories, we also recognize its limitations: it does not inherently address potential conceptual differences in synthetic samples. Future research should explore complementary techniques such as adaptive synthetic sampling (ADASYN) (He et al., 2008) or generative data augmentation methods (Goodfellow et al., 2014) to further refine model generalizability.

This study focuses on developing a robust and reliable automated analysis system for evaluating student scientific models. The primary objective is not merely to enhance neural network performance but to ensure that the system reliably analyze student responses across diverse data distributions. Addressing data imbalance is critical in this context, as models trained on imbalanced datasets tend to disproportionately favor majority-class responses while failing to recognize nuanced ideas present in underrepresented student groups. To mitigate this issue, we applied SMOTE not simply to improve standard classification metrics (e.g., accuracy, recall, and F1 score) but to enhance the algorithm’s stability and generalizability in assessing a broad spectrum of student responses.

Importantly, SMOTE did not alter the scoring criteria/rubric; rather, it increased the diversity and balance of training samples, allowing the model to more accurately approximate human evaluative standards. For example, in Category 8, where positive cases comprised less than 10% of the data, the recall rate improved from 0.70 to 0.86 after SMOTE augmentation (see Table 4). This improvement indicates that the model became significantly more effective at identifying valid yet infrequent student responses that human raters had already recognized. Ensuring that automated scoring aligns with human judgment, particularly for rare but meaningful responses, is essential for achieving fair and accurate evaluations.

Regarding concerns about potential bias introduced by synthetic data augmentation, our approach ensured that SMOTE-generated samples adhered strictly to the original rubric criteria and human-scored ground truth. Prior research (Abu Zohair, 2019; Chawla et al., 2002) has demonstrated that SMOTE, when applied in educational settings, can improve fairness by reducing the model's tendency to overfit to majority-class responses. Additionally, our post-SMOTE case analyses (Figs. 5, 6, 7 and 8) confirmed that the model's evaluations remained consistent with human scores across both majority and minority-class responses, indicating no distortion of evaluation criteria.

While this study does not explicitly measure fairness metrics such as demographic parity or equalized odds, future research should further examine the impact of SMOTE on subgroup performance to ensure that no unintended biases emerge. Nonetheless, within the scope of this work, SMOTE proved instrumental in developing an automated scoring system that fair evaluates all student responses, rather than disproportionately favoring the most common ones. By addressing dataset imbalances, our approach enhances the reliability of AI-empowered assessments while maintaining alignment with human evaluative standards.

By analyzing students' models, we gain insights into their grasp of core disciplinary ideas, like charge distribution and static equilibrium. For instance, students often represented charges on the rod in the initial state (category 1) and often indicated differences in forces as the electroscope leaves moved (categories 5 & 10). By identifying these strengths in student models, one can provide instructional supports that extend these ideas to other elements and relationships in the model. On the other hand, students who struggled with charge placement often demonstrated misunderstandings about electrostatic interactions or an incomplete understanding of static equilibrium principles. These findings suggest that targeted teaching strategies, such as scaffolding students' understanding of these concepts, could address these gaps and improve scientific modeling practices. The study also demonstrates how AI-based analysis can provide targeted feedback on specific aspects of students' models. For example, AI systems can identify evidence of proficiency, such as accurate charge placement and equilibrium representations, and highlight areas needing improvement, like ambiguous charge locations. This aligns with the broader goal of using assessments to support learning by providing actionable insights to improve students' scientific understanding.

This study has highlighted discrepancies between human and machine assessments of student scientific models, particularly freestyle drawing interpretations, variability in icon size, mixed charge representations, and ambiguous charge locations. The first two types of challenges—freestyle drawing interpretations and variability in icon size—are generic features and commonly occur in student modeling in science. These challenges may apply to other automated system designs. The last two themes—mixed charge representations and ambiguous charge locations—are likely more task-specific challenges related to explicit task measurement purposes within this study. Specifically, these last two themes more accurately relate to modeling proficiency. Students not proficient in modeling will likely place charges ambiguously or use alternative ways of representing charge magnitude. For instance, in our modeling task for the electroscope, we particularly care whether students demonstrate their understanding by placing charges in specific locations on the rod and electroscope. We also found that the image recognition algorithms frequently misinterpret models with mixed charge representations due to a lack of varied examples in the training data, highlighting a gap in the AI's learning that can lead to significant scoring inaccuracies. Moreover, the challenge of ambiguously located charges in student models reveals the AI's limited ability to utilize contextual clues that human experts might employ to infer appropriate interpretations. Other assessment tasks with similar goals (e.g., labeling of specifically located elements or using differing charges) may also need to consider these challenges when designing their systems or rubrics. Additional studies will determine whether these ways of modeling relate to a lower understanding of the content (charges specifically), a lack of proficiency in modeling practice, or both.

We also identified two other themes from discrepant scores, challenges resulting from student hand-drawn components, and icon sizes. These are likely relevant in applying AI evaluation to a broad range of modeling tasks in science assessment. These discrepancies point out the challenges AI faces in accurately analyzing complex and creative student responses that are standard in scientific modeling tasks. For instance, AI systems struggle to understand the nuanced and contextual elements of free-style drawings, often failing to recognize the creativity and depth such representations may convey compared to human raters. The issue with scoring hand-drawn models is twofold. First, AI often fails to recognize proficiency in detailed and accurate hand-drawn models because it cannot interpret the creativity and depth that human raters can. Second, the variability in icon size students use to represent charge strengths often leads to misinterpretations by AI systems. AI may ignore the implied intensity differences intended by the students, resulting in inaccurate assessments. Therefore, the discrepancies highlight the need to define what constitutes proficiency in student hand-drawn models for both AI and human raters. Some hand-drawn models show high proficiency and detail but are still not scored accurately by AI. This indicates that the problem is not just the ambiguity or lack of proficiency in the models but also the AI's limitations in evaluating them. However, recent AI approaches are increasingly more adept at interpreting handwritten text and hand drawn sketches and should continue to be explored in the context of scientific practices like modeling (see below; Xu et al., 2022).

Based on the outcomes of this study, in which students used a set of electronic drawing tools, models that consistently used the supplied pre-defined elements were mis-scored less often by the AI system. Thus, some constraints to completely free drawings seem beneficial for later AI scoring applications. However, using these tools could constrain students’ ability to draw and model as they desire and limit their representations and creativity. Thus, some balance between constrained electronic tools and free-hand drawings should be realized. To address the nuanced challenges identified, AI algorithms must incorporate advanced data interpretation capabilities. This could involve integrating machine learning techniques for multimodal data processing and context-sensitive analysis. As educational assessments increasingly utilize AI tools, ensuring these technologies can support and value student creativity in developing scientific models becomes critical. This involves technological advancements in AI and adjustments in educational practices to accommodate and nurture diverse expressions of student understanding.

These findings suggest an urgent need for enhancing AI algorithms to better handle the complexities inherent in student-generated scientific models. This includes expanding training datasets to encompass a wider variety of student responses and integrating more sophisticated, context-aware machine learning techniques that can adapt to the diverse ways students express their scientific understanding. Such advancements are crucial for improving the accuracy and fairness of AI assessments and ensuring that AI-supported educational tools can enhance learning outcomes by accurately evaluating and supporting all students’ work.

This study contributes to the broader field of educational technology by highlighting specific areas where AI application in science education can be refined and by providing an explicit analysis of the limitations current systems face, thereby guiding future developments in AI-assisted educational assessments. It not only demonstrates AI's capacity to replicate human-like judgment but also reveals critical gaps in its ability to process complex and creative student inputs. By addressing these gaps, future research can pave the way for more reliable, fair, and innovative uses of AI in education, ensuring that it supports a broad spectrum of student abilities and expressions.

Generative AI (GenAI) models, such as Gemini and GPT-4, have demonstrated significant potential in assessing multimodal inputs, including handwritten work, as highlighted in previous studies (Kortemeyer et al., 2025). Despite their strengths, GenAI models face notable challenges, particularly with data privacy issue, stability and consistency. For example, their outputs can vary significantly based on the phrasing of prompts, which poses challenges for reliable performance in specific, high-stakes educational contexts. In this study, we prioritized the development of a stable and consistent algorithm capable of accurately analyzing student models across diverse datasets and instructional contexts. To achieve this goal, we selected a CNN-based deep learning approach. This method enabled task-specific tuning and demonstrated superior reliability in detecting and analyzing detailed features within students’ scientific models. While GenAI models remain promising for handling diverse and unstructured tasks, the specific requirements of our study—focused on physics-based modeling proficiency within a predefined curriculum—favored the use of a Deep Learning approach for its stability and adaptability. We encourage further research to explore the applications of GenAI in science education, particularly in handling multimodal inputs (Yang et al., 2024). However, in the context of this study, the DL approach offered a more contextually appropriate and reliable solution for achieving our research objectives.