Introduction
Literature review
Research questions
-
In an intensive academic writing course, when the instructional hours and tasks are held constant, does the employment of ChatGPT as a feedback tool have any significant impact on undergraduate ESL students’ writing skills?
-
How do the experimental group students perceive the impact of ChatGPT as a feedback tool on their writing skills?
Methodology
Mixed methods intervention design
Participants
Criterion | Condition for inclusion | Condition for exclusion |
---|---|---|
Age | Between 18 and 19 years | Less than 18 and more than 20 |
Academic qualification | Undergraduate first year in engineering and science | Any other year of undergraduation and any other disciplines |
English as a second language | Who has English as their second language | Who has English as their first or foreign language |
Number of classes attended | 6 | Anything fewer than 6 |
Participation in the classroom | Who participated in all classroom activities on self-and peer assessment and offered feedback | Who did not fully participate in classroom activities and did not offer adequate feedback |
Methods of data collection
Procedure of data collection
-
We are using ChatGPT to assess our own and our peers’ writing. How would you describe your experience with it?
-
We are getting feedback on the topic sentence, supporting details, concluding sentence, use of signposts, appropriateness of content, and grammar. How beneficial do you think this exercise is?
-
Do you have any suggestions for making the employment of ChatGPT more impactful?
Analysis
Findings
Positive impact of ChatGPT as a feedback tool
Tests | Mean | SD | N |
---|---|---|---|
Pre-test | 12.581 | 0.833 | 37 |
Post-test | 19.216 | 2.485 | 37 |
Delayed post-test | 20.419 | 2.575 | 37 |
Cases | Sum of squares | df | Mean square | F | p | η2 |
---|---|---|---|---|---|---|
Score | 1318.473a | 2a | 659.236a | 330.704a | 5.146e−37a | 0.902 |
Residuals | 143.527 | 72 | 1.993 |
Cases | Sum of squares | df | Mean square |
---|---|---|---|
Residuals | 342.507 | 36 | 9.514 |
Mean difference | SE | t | Cohen's d | pbonf | ||
---|---|---|---|---|---|---|
Pre-test | Post-test | − 6.635 | 0.331 | − 20.070 | − 3.300 | 2.839e−30*** |
Delayed post-test | − 7.838 | 0.331 | − 23.708 | − 3.898 | 1.075e−34*** | |
Post-test | Delayed post-test | − 1.203 | 0.331 | − 3.638 | − 0.598 | 0.002** |
t | df | p | |
---|---|---|---|
Post-test scores | − 5.643 | 70 | 3.297e−7 |
t | df | p | |
---|---|---|---|
Score | − 9.371 | 70 | 5.544e−14 |
Group | N | Mean | SD | SE | ||
---|---|---|---|---|---|---|
Score | CG | 35 | 15.400 | 1.897 | 0.321 | |
EG | 37 | 20.419 | 2.575 | 0.423 |
Students’ positive perception of the impact
Themes and Sub-themes | Codes | References |
---|---|---|
1. Content 1.1 Positive | Generation of more ideas on the topic | 22 |
Focused information on the topic | 21 | |
Promotion of learner autonomy | 19 | |
Promotion of collaboration | 15 | |
Faster writing speed | 13 | |
Creation of strong topic sentences | 13 | |
Time-saving in brainstorming | 8 | |
1.2 Negative | Less motivation to think | 4 |
More machine dependence | 3 | |
2. Organization 2.1 Positive and specific | Easy to stay focused | 22 |
Strong connection between the main idea and the supporting details | 17 | |
Appropriate use of signposts | 11 | |
Reaffirming the main idea in the conclusion | 8 | |
Enabling collaborative organization | 7 | |
2.2 Negative | Imposing a pattern on writing | 2 |
Hindering creative organization | 2 | |
3. Grammar 3.1 Positive and specific | Improving accuracy in the sentence structure | 22 |
Reliable source of grammar | 21 | |
Providing explanations for language errors | 17 | |
Making vocabulary choices more accurate | 9 | |
Leading to discussions among peers | 8 | |
Promotion of explicit knowledge of grammar | 7 | |
3.2 Negative | Reduction in attention to grammatical accuracy | 5 |
Increase in machine-dependence | 3 |
The way it guides us in obtaining the required information, arranging our ideas, and writing correctly is surprising. I was never aware that it could be a writing buddy. (S2, FGD 1)You see, when we were asking it to help us with organizing the ideas in writing, it gave us some directions. I guess that’s quite helpful. It’s like someone is constantly there to oversee your writing process. (S4, FGD 1)It’s actually better than Grammarly in the sense that it explains the grammar issues when asked. You have a choice, and you can also learn from it. (S1, FGD 2)
The best part about ChatGPT is that you ask for information, and you get it. You can go as specific or detailed as you wish. This reduced our thinking time invested to get ideas and information. (S5, FGD 3)I felt that it was a handy support tool for writing without being dependent on anyone. When we write, we usually have queries regarding several aspects of writing. With ChatGPT, you have a reliable support system with you. I like that freedom. (S3, FGD 2)
When you use ChatGPT in a classroom with your classroom, you’re doing it with several people. So much talk going on simultaneously! It’s kinda cool. The conversations are so meaningful and without noticing, we are working together and writing. (S2, FGD 3)I absolutely love how we play with it together and how that fun is so productive. The process took much less time and there was this constant focused chatter which helped complete the tasks. We didn’t miss anything significant, for example, when cheating the topic sentence, because one or two people are working with me and giving me feedback on the topic and the strength of the controlling idea. (S4, SGD 2)
It might be a concern that my dependence might discourage me to do things on my own when writing. What if I won’t want to write on my own? It can be scary, but I don’t know. (S5, FGD 3)
We felt that it keeps us on our toes. You know it’s so easy to get diverted and include details unrelated to the topic. When you as ChatGPT, it tells you where you skid off the track. (S1, FGD 3)It’s unreal! You share the topic sentence and the supporting details and it tells you if and how you have adhered to the controlling idea in your details. Isn’t that cool? I don’t think we can do that so accurately on our own. (S3, FGD 3)
When you asked us to write that paragraph on AI use in education, I created a topic sentence and added the required details along with a concluding sentence. When I asked GPT to tell me if my concluding sentence is a good one for the topic sentence, the feedback surprisingly good. I did the same when sharing feedback on my friends’ writing. (S5, FGD 2)I’m happy that I have improved my signpost use. In fact, most of my classmates have too. It made me conscious about the choice of signposts. The thread connecting information and ideas suddenly felt more robust. Sometimes, the explicit explanation with examples helped. (S4, FGD 3)
I’m not sure, but sometimes it feels like I’m under a spell and I’m arranging information as directed. Though it’s my responsibility to choose and accept, I may be getting too lazy to use my own creativity to place things in order. (S2, FGD 3)
I’ve been using Grammarly for a while, but it provides explanations for the grammar queries or when it identifies an error. Nothing like knowing the details about the issue. We kinda get sucked into curiosity by asking it questions on sentence structure, tense use and other aspects of grammar. It provides detailed explanations with examples. Good alternative to Grammarly, dictionary and other such stuff. (S1, FGD 3)