Improving Writing Feedback for Struggling Writers: Generative AI to the Rescue?

The writing process may be challenging for some students for many reasons. For example, planning is the first step of writing, but many students don’t systematically brainstorm. Instead, they move directly into drafting their sentences which may, in turn, be disjointed and not effectively communicated (Evmenova & Regan, 2019). Students, particularly those with high-incidence disabilities may not produce text or compose limited text, struggling with content generation, vocabulary, and the organization of ideas (Chung et al., 2020). While multilinguism is an asset, we have observed similar challenges with writing among English Language Learners in our research (Hutchison et al., 2024). The cognitive demands needed for drafting a response leave many students at no capacity to then edit or revise their work (Graham et al., 2017). Therefore, teachers should provide scaffolds to break down the complex process of writing so that it is sequential and manageable, progressing from simple to more complex concepts and skills.

Instruction for struggling writers is typically characterized as systematic and explicit (Archer & Hughes, 2011; Hughes et al., 2018). In order to provide explicit instruction, teachers should be guided by ongoing student data. Specifically, special and general education teachers of writing should collaboratively, systematically, and continuously monitor and responsively adjust instruction based on student progress (Graham et al., 2014). Formative assessments of writing inform feedback that a teacher provides a learner. McLeskey et al., (2017) describes:

Effective feedback must be strategically delivered, and goal directed; feedback is most effective when the learner has a goal, and the feedback informs the learner regarding areas needing improvement and ways to improve performance… Teachers should provide ongoing feedback until learners reach their established learning goals. (p. 25)

Various formative assessments are available to guide feedback in writing, with rubrics being one frequently used method, which we will explore in the following section.

School-aged students are required to show progress towards mastery of writing independently in order to be successful at school, future work, and in their personal lives (Graham, 2019). Thus, educators continuously look for tools to increase and support learner agency and independence including in writing (Edyburn, 2021). Over the past decade, the authors have developed a digital tool to support learner autonomy, access, and independence during essay composition as part of a federally funded, design-based research project referred to as WEGO: Writing Effectively with Graphic Organizers (Evmenova et al., 2018–2023). This tool is a technology-based graphic organizer (or TBGO) that embeds numerous evidence-based strategies and universally designed supports for students as well as an analytic rubric for teachers to evaluate student products and providing feedback. A detailed description of the tool can be found elsewhere (students’ features: Evmenova et al., 2020a; teachers’ features: Regan et al., 2021).

The TBGO was developed to support upper elementary and middle school students with and without high-incidence disabilities to compose multiple genres of writing including persuasive (Evmenova et al., 2016), argumentative (Boykin et al., 2019), and/or personal narrative writing (Rana, 2018). The TBGO has also been effectively used by English Language Learners (Day et al., 2023; Boykin et al., 2019). In addition, it includes a dashboard that allows a teacher or caregiver to personalize instruction: assign prompts and support features embedded in the TBGO. After the student has an opportunity to write independently, the teacher can engage in what we refer to as data-driven decision making (or DDDM; Park & Datnow, 2017; Reeves and Chiang, 2018).

A common formative assessment of writing used in classrooms is a rubric. In order to facilitate the DDDM process within the TBGO, various data are collected by the tool and provided to teachers including final writing product, total time spent actively using the tool, video views and duration, text-to-speech use and duration, audio comments use and duration, transition words use, total number of words, number of attempts to finish. A teacher first evaluates those data as well as student’s writing using a 5-point rubric embedded in the teacher dashboard of the TBGO (a specific rubric is available at https://​wego.​gmu.​edu). Based on the rubric, a teacher identifies an area of need organized by phases of the writing process: Planning (select a prompt; select essay goal; select personal writing goal; brainstorm); Writing (identify your opinion, determine reasons, explain why or say more, add transition words, summarize, check your work); and Reviewing: Revise and Edit (word choice, grammar/spelling, punctuation, capitalization, evaluate). Then, a teacher provides specific instructional suggestions when the students’ score does not meet a threshold (e.g., content video models, modeling, specific practice activities). Once teachers select a targeted instructional move that is responsive to the identified area on the writing rubric, they record their instructional decision in the TBGO dashboard. The student’s work, the completed rubric, and the instructional decision is stored within the teacher dashboard. Recent investigations report that teachers positively perceive the ease and usability of the integrated digital rubric in the TBGO (see Regan et al., 2023a; b). Although promising, the teachers in those studies used DDDM with only a few students in their inclusive classes.

The current version of the TBGO relies on teachers or caregivers to score student writing using an embedded rubric and to subsequently provide the student(s) with instructional feedback. In a classroom of twenty or more students, scoring individual essays and personalizing the next instructional move for each student is time consuming, and teachers may not regularly assess or interpret students’ writing abilities in the upper grades, especially (Graham et al., 2014; Kiuhara et al., 2009). Generative AI or chatbots are arguably leading candidates to consider when providing students with instructional feedback in a more time efficient manner (Office of Educational Technology, 2023). For example, automated essay scoring (AES) provides a holistic and analytic writing quality score of students’ writing and a description as to how the student can improve their writing. Recent research on classroom-based implementation of AES suggests its potential; but questions have been raised as to how teachers and students perceive the scores, and how it is used in classroom contexts (Li et al., 2015; Wilson et al., 2022). Other investigations remark on the efficiency and reliability among AES systems (Wilson & Andrada, 2016) and the consistency of scores with human raters (Shermis, 2014). More recently, a large-language model (specifically, GPT-3.5 version of ChatGPT) was prompted to rate secondary students’ argumentative essays and chatbot’s responses were compared to humans across five measures of feedback quality (see Steiss et al., 2023). Although GPT-3.5 included some inaccuracies in the feedback and the authors concluded that humans performed better than ChatGPT, the comparisons were remarkably close.

A greater understanding of what generative AI tools can do to support classroom teachers is needed. First, leveraging technology, with the use of automated systems, or logistical tools, can potentially improve working conditions for both general and special education teachers (Billingsley & Bettini, 2017; Johnson et al., 2012). Also, although educators see the benefits of AI and how it can be used to enhance educational services, there is urgent concern about the policies needed around its use and how it is ever evolving. For example, when writing this manuscript, GPT-4 evolved, but at a cost, this latter version may not be widely accessible for educators or students. With the fast adoption of AI, the Office of Educational Technology states that “it is imperative to address AI in education now to realize and mitigate emergent risks and tackle unintended consequences” (U.S. Department of Education, 2023, p. 3). A first step in addressing AI in education is to understand what AI can do, and how its use supports or hinders student learning and teacher instruction. In this case, we focus on teachers’ writing instruction and feedback.

As we learn more about AI tools, it becomes obvious that AI literacy skills will need to be developed as part of digital skills by both teachers and students (Cohen, 2023). The importance of how we use chatbots, how we prompt them, and what parameters we use to direct the responses of chatbots becomes paramount.

Thus, the purpose of this study was to explore feedback and instructional suggestions generated by different AI tools when using prompts providing varying specificity (e.g., a generic 0–4 rating vs. analytic rubric provided) to help guide teachers of writing in their use of these tools. The purpose of including two versions of ChatGPT was not to criticize one and promote the other; but rather to understand and leverage their similarities and differences, given the same prompt. The research questions were:

The 34 essays used in this study were written by 21 girls and 13 boys. Students ranged in age 8–13 and were in grades 3–7. The majority (59%) were White, 21% were Hispanic, 3% were African American, and 17% were other. Among the students, 41% were identified with high-incidence disabilities (learning disabilities, ADHD); 24% were English language learners (with a variety of primary languages); and 35% were struggling writers as reported by teachers. Teachers identified struggling writers as those who consistently demonstrated writing performance below grade level expectations (e.g., needing extra support with writing mechanics, cohesive and well-organized ideas).

The data used in this study were collected in two separate settings: two inclusive classrooms in a suburban, private day school and an after-school program in a community center serving economically disadvantaged families. The same essay writing procedures were used in both settings. All students were first asked to write a persuasive opinion-based essay in response to one of two prompts validated by previous research (Regan et al., 2023b). Examples of the prompts included:

After the pretest, students were introduced to the technology-based graphic organizer (TBGO) with embedded evidence-based strategies and supports. The instruction lasted 5–6 lessons. Then students were asked to use the TBGO to practice independent essay writing without any help from the teachers. As the TBGO is a Chrome-based web application and works on any device with a Chrome browser installed, each student used their own device/laptop and individual login credentials to access the TBGO. After completing the independent writing, teachers reviewed students’ products and completed the analytic rubric built into the TBGO’s teacher dashboard. They identified one primary area of need and determined an instructional decision that should take place in order to address the existing area of need. The instructional decisions included whole- and small-group activities (especially in those cases when multiple students demonstrated the same area of need); independent activities (including watching video models embedded within the TBGO); as well as individual teacher-student check-ins to discuss the area of need and future steps. A posttest with the TBGO and a delayed posttest without the TBGO were later administered. The essays used in the current study were from an independent writing phase since those included teachers’ DDDM. On average, essays had 133.44 (SD = 57.21; range 32–224) total words written. The vast majority included such important persuasive essay elements such as a topic sentence introducing the opinion, distinct reasons, examples to explain the reasons, summary sentence, and transition words. While this provides some important context, the quantity and quality of students’ writing products is not the focus of the current study and is reported elsewhere (Boykin et al., 2019; Day et al., 2023; Evmenova et al., 2016, 2020b; Regan et al., 2018, 2023b).

The existing 34 essays were imported into two different versions of the ChatGPT generative AI: GPT-3.5 version of ChatGPT (free version) and GPT-4 (subscription version). Four different prompts were used in both ChatGPT versions (see Table 1). As can be seen in Table 1, the different prompts included (1) using a specific analytic rubric (when a rubric from the TBGO was uploaded to ChatGPT); (2) asking for a generic 0–4 rating (without any additional specifics regarding scoring); (3) no rubric (asking to identify the area of need without any rubric); (4) no information (asking to provide generic feedback without any information about the student in the prompt). Each prompt type constituted its own GPT chat. Thus, eight sets of responses (or eight different chats) were generated by ChatGPT. A prompt tailored to include the student’s essay as well as the specific student characteristics and the essay topic when applicable (according to the prompt samples presented in Table 1) was pasted into the chat. After GPT had a chance to react and provide feedback, the next prompt was pasted into the same chat. Thus, each chat included a total of 34 prompts and 34 GPT outputs. Each chat was then saved and analyzed.

Inductive thematic analysis was used to explore how generative AI can be used to provide writing feedback and guide writing instruction for struggling writers (Guest et al., 2011). First, each set of ChatGPT responses (or each GPT chat) was analyzed individually, and reoccurring codes across responses were grouped into categories. The four members of the research team were randomly assigned to analyze two GPT sets each. Each member generated a list of codes and categories within a chat that were the shared with the team and discussed. During those discussions, the patterns within categories were compared across different sets to develop overarching themes in response to RQ1 and RQ2. The trustworthiness of findings was established by data triangulation across 34 writing samples and eight sets of feedback. Also, peer debriefing was used throughout the data analysis (Brantlinger et al., 2005).

To answer RQ3, frequencies were used to compare teachers’ and ChatGPT scores on the analytic rubric and suggested instructional decisions. First, two researchers independently compared teachers’ and ChatGPT scores and suggestions. Since the same language from the rubric was used to identify the area of need, the comparisons were rated as 0 = no match; 1 = match. For instructional suggestions, the scale was 0 = no match; 1 = match in concept, but not in specifics; and 2 = perfect match. Over 50% of comparisons were completed by two independent researchers. Interrater reliability was established using point-by-point agreement formula dividing the number of agreements by the total number of agreements plus disagreements and yielding 100% agreement.

In effort to answer RQ1 and explore the differences between responses generated by different ChatGPT versions when given prompts with varying specificity, we analyzed eight sets of responses. While the purpose was not to compare the sets in effort to find which one is better, several patterns have been observed that can guide teachers in using ChatGPT as the starting point for generating writing feedback to their struggling writers. The following are the six overarching themes that emerged from this analysis.

Instructional suggestions based on the evaluation of student writing was a focus of RQ2. Although we expected the responses from prompts that included specific student characteristics to differentiate the instructional suggestions in some way, this was not the case. In fact, none of the sets provided explicit instructional suggestions aligned with students’ characteristics (e.g., grade, disability, ELL). First, the suggestions for improving the writing of a 3rd grader’s essay were not distinct from those suggestions provided in response to a 7th grader’s writing (in Generic Rating GPT-3.5 and No Rubric GPT-3.5 sets). Also, there were no remarkable differences in the vocabulary used in the feedback for a 3rd grader vs. a 7th grader (in Generic Rating GPT-4 set). Only one set (Generic Rating GPT-4) offered a personalized message in a student-friendly format (without any additional prompting to do so).

Second, student characteristics were merely acknowledged in some sets. For example, Specific Analytic Rubric GPT-3.5 and GPT-4 only noted those characteristics in the summary section at the end of the feedback (e.g., “This is a well-written persuasive essay by your 7th-grade student with ADHD”). This was also observed in responses from the Generic Rating GPT-4 set, as well. For example, “This feedback emphasizes both the strengths of the student’s writing and the areas where improvement can be made, offering encouragement and guidance that is particularly important for a student with ADHD.” Finally, the No Rubric GPT-4 set also gave a mere nod to the additional context (e.g., Given student characteristics…). Although rare, connecting student characteristics with instruction was observed here: “Students with ADHD often struggle with organizing their thoughts in a coherent manner, and the flow of ideas in this student’s paragraph seems a bit disjointed….” Students’ characteristics were not mentioned in any other sets in which student information was included in the prompt (Generic Rating GPT-3.5 and No Rubric GPT-3.5).

Below is the description of how specific, broad, or no instructional suggestions were included in the ChatGPT sets (see Table 2).

GPT-3.5 matched the teacher’s recommendation for the area of writing need 17.6% of the time. For example, the teacher identified Word Selection as the area of need (e.g., high use of repeated words and lacking sensory words) and GPT-3.5 noted the same area of need (e.g., there is some repetition and awkward phrasing). When comparing teacher versus ChatGPTs instructional decisions, there was no perfect match; however, 26.5% were coded as a partial match. For example, both the teacher and GPT-3.5 suggested an instructional activity of modeling how to write a summary sentence.

GPT-4 matched the teacher’s recommendation for the area of writing need 23.5% of the time. Similarly, when comparing the teacher versus ChatGPT’s instructional decisions, 47.1% were coded as a partial match for instruction.

This study is an initial exploration. There are several limitations that need to be taken into consideration. First and foremost, the four prompts were designed to present the chatbots with varying levels of details and student information to consider when providing feedback about a student’s writing sample. For example, Specific Analytic Rubric prompt asked the chatbot to assess students’ writing using an uploaded rubric, while No Rubric prompt asked to identify one area of need for the student’s writing and offer one instructional suggestion to address it. In addition to providing the chatbots with varying information, we also used varying language throughout the prompts when seeking feedback and suggestions (e.g., “Identify areas of need for this student’s writing”; “Identify one area of need … and offer one instructional suggestion”; “what feedback and instructional suggestions…”). Chatbots are clearly sensitive to the word choices made; thus, a consistency of the language in prompts should be considered for any future investigations that aim at prompt comparison. The purpose of this work was not to compare the four prompts in effort to find the best possible one. We also were not looking specifically for the feedback that could be shared with students as is (even though some versions generated such feedback without additional prompting). Instead, we were trying to explore how the output might differ depending on the prompts with differing level of detail. So, some of the reported difference are logical. We also did not prompt the ChatGPT any further, which would most likely result in refined feedback and/or suggestions. There is an infinite number of prompts that we could have used in this analysis. In fact, a new field of prompt engineering is emerging right in front of our eyes as we learn to design inputs for generative AI tools that would produce optimal outputs. Further investigations of various prompts to feed ChatGPT are needed. Our hope is that this paper will inspire teacher to spend some time exploring different tools and prompts in effort to find the most appropriate output depending on their context and their students’ needs.

Also, there was a limited numbers of essays from each specific group of learners (e.g., certain age/grade, specific disability categories and other characteristics). While we reported meaningful findings for this initial exploratory analysis, future research should include writing products from more homogeneous groups. Finally, teachers’ DDDM was accomplished by evaluating a completed graphic organizer, while ChatGPT feedback was provided based on the final student essay copied and pasted from the TBGO. Future research should consider new features of generative AI tools (e.g., Chat GPT’s new image analysis feature) where an image of a completed graphic organizer can be uploaded and analyzed.