Top

Technology, Knowledge and Learning

Published in:

05-04-2022 | Original research

An error-analysis study from an EFL writing context: Human and Automated Essay Scoring Approaches

Authors: Norah Almusharraf, Ph.D., Hind Alotaibi, Ph.D.

Published in: Technology, Knowledge and Learning | Issue 3/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Evaluating written texts is believed to be a time-consuming process that can lack consistency and objectivity. Automated essay scoring (AES) can provide solutions to some of the limitations of human scoring. This research aimed to evaluate the performance of one AES system, Grammarly, in comparison to human raters. Both approaches’ performances were analyzed quantitatively using Corder’s (1974) error analysis approach to categorize the writing errors in a corpus of 197 essays written by English as a foreign language (EFL) learners. Pearson correlation coefficient and paired sample t-tests were conducted to analyze and compare errors detected by both approaches. According to the study’s results, a moderate correlation between human raters and AES in terms of the total scores and the number of errors detected. Results also indicated that the total number of errors detected by AES is significantly higher than human raters and that the latter tend to give students higher scores. The findings encourage a more open attitude towards AES systems to support EFL writing teachers in assessing students’ work.

previous article Digitalisation in the Curricula of Vocational Schools: Text Mining as an Instrument of Curricula Analysis

next article Female Students in Quantitative Courses: An Exploration of their Motivational Sources, Learning Strategies, Learning Behaviors, and Course Achievement

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Al-Ahdal, A. (2020). Using computer software as a tool of error analysis: Giving EFL teachers and learners a much-needed impetus. International Journal of Innovation, Creativity, and Change, 12(2), 418–437. https://doi.org/10.3390/languages4010019

Alrashidi, O., & Phan, H. (2015). Education context and English teaching and learning in the Kingdom of Saudi Arabia: An overview. English Language Teaching, 8(5), 33–44. https://doi.org/10.5539/elt.v8n5p33

Alshakhi, A. (2019). Revisiting the writing assessment process at a Saudi English language institute: Problems and solutions. English Language Teaching, 12(1), 176–185. https://doi.org/10.5539/elt.v12n1p176

Attali, Y. (2013). Validity and reliability of automated essay scoring. In M. D. Shermis, & J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 181–198). Routledge

Attali, Y. (2016). A comparison of newly-trained and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115. https://doi.org/10.1177/0265532215582283

Attali, Y., Lewis, W., & Steier, M. (2013). Scoring with the computer: Alternative procedures for improving the reliability of holistic essay scoring. Language Testing, 30(1), 125–141. https://doi.org/10.1177/0265532212452396

Barkaoui, K. (2010a). Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44(1), 31–57. https://doi.org/10.5054/tq.2010.214047CrossRef

Barkaoui, K. (2010b). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54–74. https://doi.org/10.1080/15434300903464418

Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17, 9–17. https://doi.org/10.1002/j.2333-8504.1997.tb01734.xCrossRef

Burstein, J., & Chodorow, M. (2010). Progress and new directions in technology for automated essay evaluation. In R. B. Kaplan (Ed.), The Oxford handbook of applied linguistics (pp. 529–539). Oxford University Press, Inc. https://doi.org/10.1093/oxfordhb/9780195384253.013.0036

Chastian, K. (1990). Characteristics of graded and ungraded compositions. Modern Language Journal, 11, 367–383. https://doi.org/10.2307/327937CrossRef

Chukharev-Hudilainen, E., & Saricaoglu, A. (2016). Causal discourse analyzer: Improving automated feedback on academic ESL writing. Computer Assisted Language Learning, 29(3), 494–516. https://doi.org/10.1080/09588221.2014.991795CrossRef

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104CrossRef

Corder, S. P. (Ed.). (1967). The significance of learners’ errors. International Review of Applied Linguistics, 5, pp.161–170, and reprinted in J.C. Richards (Ed.) (1974). Error Analysis: Perspectives on Second Language Acquisition. London: Longman

Corder, S. P. (1981). Error analysis and interlanguage. Oxford University Press

Cotos, E. (2015). Automated writing analysis for writing pedagogy. Writing & Pedagogy, 7(2–3), 197–231. https://doi.org/10.1558/wap.v7i2-3.26381

Dembsey, J. M. (2017). Closing the Grammarly® gaps: A study of claims and feedback from an online grammar program. The Writing Center Journal, 36(1), 63–96. https://www.jstor.org/stable/44252638CrossRef

Dikli, S., & Bleyle, S. (2014). Automated essay scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22, 1–17. https://doi-org.sdl.idm.oclc.org/10.1016/j.asw.2014.03.006CrossRef

Douglas, D. (2010). Understanding language testing. Hodder Education

Gamper, J., & Knapp, J. (2002). A review of intelligent CALL systems. Computer Assisted Language Learning, 15(4), 329–342. https://doi.org/10.1076/call.15.4.329.8270CrossRef

Ghufron, M. A., & Rosyida, F. (2018). The role of Grammarly in assessing English as a foreign language (EFL) writing. Lingua Cultura, 12(4), 395–403. https://doi.org/10.21512/lc.v12i4.4582

Goh, T. T., Sun, H., & Yang, B. (2020). Microfeatures influencing writing quality: The case of Chinese students’ SAT essays. Computer Assisted Language Learning, 33(4), 455–481. https://doi.org/10.1080/09588221.2019.1572017CrossRef

Higgins, J. J. (1983). Computer-assisted language learning. Language Teaching, 16(2), 102–114CrossRef

Huang, S. J. (2001). Error analysis and teaching composition (Unpublished master’s thesis), National Tsing Hua University

Huang, S. J. (2014). Automated versus human scoring: A case study in an EFL context. Electronic Journal of Foreign Language Teaching, 11, 149–164. https://doi.org/10.1080/15434303.2016.1230121

Jayavalan, K., & Razali, A. B. (2018). Effectiveness of online grammar checker to improve secondary students’ English narrative essay writing. International Research Journal of Education and Sciences, 2(1), 1–6. http://psasir.upm.edu.my/id/eprint/14442

Ke, Z. (2019). Automated essay scoring: A survey of the state of the art [Paper presentation]. International Joint Conference on Artificial Intelligence 28th Annual Meeting. https://doi.org/10.24963/ijcai.2019/879

Kenning, M. J., & Kenning, M. M. (1983). Introduction to computer assisted language teaching. Oxford University Press

Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004

Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System, 44, 66–78. https://doi.org/10.1016/j.system.2014.02.007

Lu, M., Deng, Q., & Yang, M. (2019). September). EFL writing assessment: Peer assessment vs. automated essay scoring. In E. Popescu, H. Tianyong, T.-C. Hsu, H. Xie, M. Temperini (Eds.), International Symposium on Emerging Technologies for Education (pp. 21–29). Springer. https://doi.org/10.1007/978-3-030-38778-5_3

Morse, J. M. (1991). Strategies for sampling. Qualitative nursing research: A contemporary dialogue, 127–145

Nova, M. (2018). Utilizing Grammarly in evaluating academic writing: A narrative research on EFL students’ experience. Premise: Journal of English Education, 7(1), 80–97. https://doi.org/10.24127/pj.v7i1.1300

O’Neill, R., & Russell, A. M. T. (2019). Stop! Grammar time: University students’ perceptions of the automated feedback program Grammarly. Australasian Journal of Educational Technology, 35(1), 42–56. https://doi.org/10.14742/ajet.3795CrossRef

Park, J. (2019). An AI-based English grammar checker vs. human raters in evaluating EFL learners’ writing. Multimedia-Assisted Language Learning, 22(1), 112–131. https://doi.org/10.15702/mall.2019.22.1.112

Perdana, I., & Farida, M. (2019). Online grammar checkers and their use for EFL writing. Journal of English Teaching, Applied Linguistics, and Literatures, 2(2), 67–76. https://doi.org/10.20527/jetall.v2i2.7332

Prinsloo, D., & Bothma, T. (2020). A copulative decision tree as a writing tool for Sepedi. South African Journal of African Languages. 40(1). 85–97. https://doi.org/10.1080/02572117.2020.1733834

Polit, D. F., & Hungler, B. P. (1993). Study guide for essentials of nursing research: Methods, appraisal, and utilization. Lippincott, Williams, & Wilkins

Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653–674. https://doi.org/10.1080/09588221.2018.1428994CrossRef

Rao, Z., & Li, X. (2017). Native and non-native teachers’ perceptions of error gravity: The effects of cultural and educational factors. The Asia-Pacific Education Researcher, 26(1–2), 51–59. https://doi.org/10.1007/s40299-017-0326-5

Reilly, E. D., Stafford, R. E., Williams, K. M., & Corliss, S. B. (2014). Evaluating the validity and applicability of automated essay scoring in two massive open online courses. International Review of Research in Open and Distributed Learning, 15(5), 83–98. https://doi.org/10.19173/irrodl.v15i5.1857

Seker, M. (2018). Intervention in teachers’ differential scoring judgments in assessing L2 writing through communities of assessment practice. Studies in Educational Evaluation, 59, 209–217. https://doi.org/10.1016/j.stueduc.2018.08.003CrossRef

Sharma, C., Bishnoi, A., Sachan, A. K., & Verma, A. (2019). Automated essay evaluation using natural language processing. International Research Journal of Engineering and Technology, 5(6), 2055–2058. https://www.irjet.net/archives/V6/i5/IRJET-V6I5398.pdf

Shermis, M., & Burstein, J. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge

Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53–76. https://doi.org/10.1016/j.asw.2013.04.001CrossRef

Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. In E. Baker, B. McGaw, & N. S. Petersen (Eds.), International encyclopedia of education (3rd ed., pp. 75–80). Elsevier

Sparks, J. R., Song, Y., Brantley, W., & Liu, O. L. (2014). Assessing written communication in higher education: Review and recommendations for next-generation assessment (Issue No. 2). ETS Research Report Series. https://doi.org/10.1002/ets2.12035

Vojak, C., Kline, S., Cope, B., McCarthey, S., & Kalantzis, M. (2011). New spaces and old places: An analysis of writing assessment software. Computers and Composition, 28(2), 97–111. https://doi.org/10.1016/j.compcom.2011.04.004CrossRef

Wali, F. A., & Huijser, H. (2018). Write to improve: Exploring the impact of an automated feedback tool on Bahraini learners of English. Learning & Teaching in Higher Education: Gulf Perspectives, 15(1). https://doi.org/10.18538/lthe.v15.n1.293

Wang, J., & Brown, M. S. (2007). Automated essay scoring versus human scoring: A comparative study.Journal of Technology, Learning, and Assessment, 6(2). https://files.eric.ed.gov/fulltext/EJ838612.pdf

Wilson, J., & Roscoe, R. (2019). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87–125. https://doi.org/10.1177/0735633119830764CrossRef

Wiseman, C. S. (2012). Rater effects: Ego engagement in rater decision-making. Assessing Writing, 17(3), 150–173. https://doi.org/10.1016/j.asw.2011.12.001CrossRef

Wind, S. A., & Engelhard, G. Jr. (2013). How invariant and accurate are domain ratings in writing assessment? Assessing Writing, 18(4), 278–299. https://doi.org/10.1016/j.asw.2013.09.002

Wu, H., & Garza, E. V. (2014). Types and attributes of English writing errors in the EFL Context – A study of error analysis. Journal of Language Teaching & Research, 5(6), 125–141. https://doi.org/10.4304/jltr.5.6.1256-1262CrossRef

Title: An error-analysis study from an EFL writing context: Human and Automated Essay Scoring Approaches
Authors: Norah Almusharraf, Ph.D.
Hind Alotaibi, Ph.D.
Publication date: 05-04-2022
Publisher: Springer Netherlands
Published in: Technology, Knowledge and Learning / Issue 3/2023
Print ISSN: 2211-1662
Electronic ISSN: 2211-1670
DOI: https://doi.org/10.1007/s10758-022-09592-z

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 3/2023

CADA: a teacher-facing learning analytics dashboard to foster teachers’ awareness of students’ participation and discourse patterns in online discussions

Mediating Teacher Professional Learning with a Learning Analytics Dashboard and Training Intervention

Developing the Teacher Self-Efficacy Scale in the Use of ICT at Home for Pre-school Distance Education During Covid-19

Affective State Analysis During Online Learning Based on Learning Behavior Data

Readiness and Perception of Pacific Students to Mobile Phones for Higher Education

Lecturers’ Perceptions of Support for Integrating Information and Communication Technologies in Teaching and Learning

Premium Partners