Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Education and Information Technologies
Published in: Education and Information Technologies 2/2022

03-08-2021

Computerized testing in reading comprehension skill: investigating score interchangeability, item review, age and gender stereotypes, ICT literacy and computer attitudes

Author: Seyyed Morteza Hashemi Toroujeni

Published in: Education and Information Technologies | Issue 2/2022

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Score interchangeability of Computerized Fixed-Length Linear Testing (henceforth CFLT) and Paper-and-Pencil-Based Testing (henceforth PPBT) has become a controversial issue over the last decade when technology has meaningfully restructured methods of the educational assessment. Given this controversy, various testing guidelines published on computerized testing may be used to investigate the interchangeability of CFLT and PPBT mean scores to corroborate if test takers’ testing performance is influenced by the effects of testing administration mode; specifically, if validity and reliability of two versions of the same test are affected. This research was conducted to probe not only score interchangeability across testing modes but also to explore the role of age and gender stereotypes, item review, ICT literacy and attitudes towards computer use as moderator variables in test takers’ reading achievement in CFLT. Fifty-eight EFL learners homogeneous in both general English and reading skills assigned into one testing group participated in this study. Three different versions of TOEFL reading comprehension test, Computer Attitude Scale (CAS), and ICT literacy Scale of TOEFL Examinees were used in this crossover quasi-controlled empirical study with a common-person and pretest–posttest design to collect data. The findings demonstrated that although the reading scores of test takers were interchangeable in both CFLT and PPBT versions regarding testing administration modes, they were different regarding item review. Furthermore, no significant interaction was found between age, gender, and ICT literacy and CFLT performance. However, attitudes towards the use of computer led to a significant change in testing achievement on CFLT.
previous article Mathematics learning and assessment using MathE platform: A case study
next article Improving OER descriptions to enhance their availability, reuse, and enrichment

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Literature
ADB. (2018a). Asian development outlook 2018: How technology affects jobs. Asian Development Bank.
ADB. (2017b). Asian development outlook 2018: Transcending the middle-income challenge. Asian Development Bank.
Al-Amri, S. (2009). Computer-based testing vs. paper-based testing: Establishing the comparability of reading tests through the revolution of a new comparability model in a Saudi EFL context. Thesis submitted for the degree of Doctor of Philosophy in Linguistics. University of Essex (UK).
Al-Amri, S. (2007). Computer-based vs. Paper-based Testing: Does the test administration mode matter. Proceedings of the BAAL Conference.
Alghammas, A. (2020). Online Language Assessment during the COVID-19 Pandemic: University Faculty Members' Perceptions and Practices. Asian EFL journal, 27(4.4): 169–195.
American Educational Research Association. (2014). Standards for Educational and Psychological Testing. American Educational Research Association.
American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. American Psychological Association.
Ary, D., Jacobs, L., Irvine, C., & Walker, D. (2018). Introduction to research in education. Cengage Learning.
Aydemir, Z., Ozturk, E., & Horzum, M. B. (2013). The effect of reading from screen on the 5th grade elementary students’ level of reading comprehension on informative and narrative type of texts. Educational Sciences: Theory and Practice, 13(4), 2272–2276.
Backes, B., & Cowan, J. (2018). Is the pen mightier than the keyboard? The effect of online testing on measured student achievement. National Center for Analysis of Longitudinal Data in Education Research. Working Paper 190. April 2018.
Bartram, D., & Hambleton, R. K. (2016). The ITC guidelines: International standards and guidelines relating to tests and testing. In F. T. L. Leong, D. Bartram, F. M. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC international handbook of testing and assessment (p. 35–46). Oxford University Press. https://​doi.​org/​10.​1093/​med:​psych/​9780199356942.​003.​0004
Barzillai, M., & Thomson, J. M. (2018). Children learning to read in a digital world. First Monday, 23(10), 1–10.
Blackhurst, A. E. (2005). Historical perspectives about technology applications for people with disabilities. In D. Edyburn, K. Higgins, & R. Boone (Eds.), Handbook of special education technology research and practice. (pp. 1–27). Knowledge by Design Inc.
Boeve, A., Meijer, R., Albers, C., Beetsma, Y., & Bosker, R. (2015). Introducing computer-based testing in high-stakes exams in higher education: Results of a field experiment. PLoS One, 10(12), e0143616.
Brown, H. D., & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices. White Plains, NY: Pearson Education.
Burr, S. A., Chatterjee, A., Gibson, S., Coombes, L., & Wilkinson, S. (2016). Key Points to Facilitate the Adoption of Computer-Based Assessments. Journal of Medical Education and Curricular Development, 3, JMECD.S20379. https://​doi.​org/​10.​4137/​jmecd.​s20379
Chen, J., White, S., McCloskey, M., Soroui, J., & Chun, Y. (2011). Effects of computer versus paper administration of an adult functional writing assessment. Assessing Writing, 16, 49–71.
Clinton, V. (2019). Reading from paper compared to screens: A systematic review and meta-analysis. Journal of Research in Reading, 42(2), 288–325.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Lawrence Earlbaum Associates.MATH
Csapo, B., Ainley, J., Bennett, R., Latour, T., & Law, N. (2012). “Technological issues of computer-based assessment of 21st century skills,” in Assessment and Teaching of 21st Century Skills, eds P. Griffin, B. McGaw, and E. Care (New York, NY: Springer), 143–230.
Daghan, G. (2017). Views of Students about Technology, Effects of Technology on Daily Living and their Professional Preferences. TOJET: The Turkish Online Journal of Educational Technology, 16 (4).
Delgado, P., Vargas, C., Ackerman, R., & Salmeron, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23–38.
Eagly, A. H., & Shelly C. (1998). “Attitude Structure and Function.” In Handbook of Social Psychology, ed. D.T. Gilbert, Susan T. Fisk, and G. Lindsey, 269–322. New York: McGowan-Hill.
Eaves, R. C., & Smith, E. (1986). The effect of media and amount of microcomputer experience on examination scores. Journal of Experimental Education, 55, 23–26.
Ebrahimi, M.R, Hashemi Toroujeni, S.M. & Shahbazi, V. (2019). Score Equivalence, Gender Difference, and Testing Mode preference in a Comparative Study between Computer-Based Testing and Paper-Based Testing. International Journal of Emerging Technologies in Learning (iJET), 14(07).
Eignor, D. R., Taylor, C., Kirsch, I., & Jamieson, J. (1998). Development of a scale for assessing the level of computer familiarity of TOEFL examinees. Educational Testing Service.
Farinosi, M., Lim, C., & Roll, J. (2016). Book or screen, pen or keyboard? A cross-cultural sociological analysis of writing and reading habits basing on Germany, Italy, and the UK. Telematics and Informatics, 33(2), 410–421.
French, M. M. J., Blood, A., Bright, N. D., Futak, D., Grohmann, M. J., Hasthorpe, A., Heritage, J., Poland, R. L., Reece, S., & Tabor, J. (2013). Changing fonts in education: How the benefits vary with ability and dyslexia. The Journal of Educational Research, 106, 301–304.
Galindo-Aldana, G., Meza-Kubo, V., Castillo-Medina, G., Ledesma-Amaya, I., Galarza-Del-Angel, J., Padilla-Lopez, A., and Moran, A. L. (2018) Computer-Based Neuropsychological Assessment: A Validation of Structured Examination of Executive Functions and Emotion. In: Harris D. (eds) Engineering Psychology and Cognitive Ergonomics. EPCE 2018. Lecture Notes in Computer Science, vol 10906. Springer, Cham. https://​doi.​org/​10.​1007/​978-3-319-91122-9_​26
Gallagher, A., Bridgeman, B., & Cahalan, C. (2000). The effect of computer-based tests on racial/ethnic, gender, and language groups (GRE Board Professional Report No. 96–21P). Princeton, NJ: Education Testing Service.
García Laborda, J., & Alcalde Penalver, E. (2018). Constraining issues in face-to-face and Internet-based language testing. Journal for Educators, Teachers, and Trainers, 9(2), 47–56.
Garcia Laborda, J., Magal Royo, T., & Enríquez Carrasco, E. V. (2010). Teachers’ trialing procedures for Computer Assisted Language Testing Implementation. Eurasian Journal of Educational Research., 39, 161–174.
Golan, D. D., Barzillai, M., & Katzir, T. (2018). The effect of presentation mode on children’s reading preferences, performance, and self-evaluations. Computers & Education, 126, 346–358.
Hashemi Toroujeni, S.M. (2016). Computer-Based Language Testing versus Paper-and-Pencil Testing: Comparing Mode Effects of Two Versions of General English Vocabulary Test on Chabahar Maritime University ESP Students’ Performance. Unpublished thesis submitted for the degree of Master of Art in Teaching. Chabahar Marine and Maritime University (Iran).
Hashemi Toroujeni, S.M., Thompson, N., & Faghihi, A. (In Press). Computer in Education Assessment: Exploring Score Equivalence of Paper-Based versus Computer-Based Language Testing considering Individual Characteristics. Profile: Issues in Teachers’ Professional Development.
Hermena, E. W., Sheen, M., AlJassmi, M., AlFalasi, K., AlMatroushi, M., & Jordan, T. R. (2017). Reading rate and comprehension for text presented on tablet and paper: Evidence from Arabic. Frontiers in Psychology, 8, 257.
Hosseini, M. & Hashemi Toroujeni, S.M. (2017a). Validity and Reliability Examination in Onscreen Testing: Interchangeability of Scores in Conventional and Computerized Tests by Examining Two External Moderators. International Journal of English Language Teaching and Linguistics (IJELTL), Vol 1(2), 2017. http://​ijeltal.​com/​index.​php/​ijeltal/​article/​view/​17
Howell, D. C. (2002). Statistical methods for psychology. Pacific Grove, CA: Duxbury/Thompson Learning.
Hughes, A. (2003). Testing for language teachers. (2nd ed.). Cambridge University Press.
Jeong, H. (2012). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33, 410–422.
Jerrim, J. (2016). PISA 2012: how do results for the paper and computer tests compare? Assessment in Education: Principles, Policy & Practice, 23(4), 495–518.
Johnson, M., & Green, S. (2006). On-line mathematics assessment: The impact of mode on performance and question answering strategies. The Journal of Technology, Learning, and Assessment, 4(5).
Karay, Y., Schauber, S., Stosch, C., & Schüttpelz-Brauns, K. (2015). Computer versus paper: Does it make any difference in test performance? Teaching and Learning in Medicine, 27(1), 57–62.
Karim, M. R., & Hasan, M. (2020). Virtual Classes during COVID 19 Pandemic in Tertiary Level in Saudi Arabia: Challenges and Prospects from the Students’ Perspective. Asian EFL journal, 27(5.1): 205–219.
Katz, I. R., & Elliot, N. (2016). Information literacy in digital environments: Construct mediation, construct modeling, and validation processes. In Information literacy: Research and collaboration across disciplines (pp. 97–116).
Khoshsima, H., & Hashemi Toroujeni, S.M. (2017h). Computer-Based Testing: Score Equivalence and Testing Administration Mode Preference in a Comparative Evaluation Study. International Journal of Emerging Technologies in Learning (iJET), 12(10), 35–55. https://​doi.​org/​10.​3991/​ijet.​v12i10.​6875
Khoshsima, H., Hashemi Toroujeni, S.M., Thompson, N., & Ebrahimi, M. R. (2019). Computer-Based (CBT) vs. Paper-Based (PBT) Testing: Mode Effect, Relationship between Computer Familiarity, Attitudes, Aversion and Mode Preference with CBT Test Scores in an Asian Private EFL Context. Teaching English with Technology (TEwT), 19(1), 86–101.
Kong, Y., Seo, Y. S., & Zhai, L. (2018). Comparison of reading performance on screen and on paper: A meta-analysis. Computers & Education, 123, 138–149.
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. Routledge.
Leino, K. (2014). The relationship between ICT use and reading literacy: focus on 15-yearold Finnish students in PISA studies (Academic dissertation). Jyväskylä University Press.
Margolin, S. J., Driscoll, C., Toland, M. J., & Kegler, J. L. (2013). E-readers, computer screens, or paper: Does reading comprehension change across media platforms? Applied Cognitive Psychology, 27(4), 512–519.
Meyer, A., Innes, S., Stomski, N., & Armson, A. (2016). Student performance on practical gross anatomy examinations is not affected by assessment modality. Anatomical Sciences Education, 9(2), 111–120.
Nunnally, J. C. (1978). Psychometric theory. (2nd ed.). McGraw-Hill.
Oz, H., & Ozturan, T. (2018). Computer-based and paper-based testing: Does the test administration mode influence the reliability and validity of achievement tests? Journal of Language and Linguistic Studies, 14(1), 67–85. Submission Date: 23/11/2017.
Phillips, D. (2001). Longman Complete Course for the TOEFL Test: Preparation for the computer and paper tests. Longman.
Pieger, E., Mengelkamp, C., & Bannert, M. (2016). Metacognitive judgments and disfluency– Does disfluency lead to more accurate judgments, better control, and better performance? Learning and Instruction, 44, 31–40.
Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based tests. The Journal of Technology, Learning, and Assessment, 2(6).
Privitera, G. J. (2012). Statistics for the Behavioral Sciences. SAGE publication Inc.
Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013). How teachers are using technology at home and in their classrooms. Pew Research Center’s Internet & American Life Project.
Rasmusson, M. (2015). Reading paper–Reading screen: A comparison of reading literacy in two different modes. Nordic Studies in Education, 34(1), 3–19.
Rausch, A., Seifried, J., Wuttke, E., Kogler, K., & Brandt, S. (2016). Reliability and validity of a computer-based assessment of cognitive and non-cognitive facets of problem-solving competence in the business domain. Empirical Research in Vocational Education and Training. 8(9).  https://​doi.​org/​10.​1186/​s40461-016-0035-y
Register-Mihalik, J. K., Kontos, D. L., Guskiewicz, K. M., Mihalik, J. P., Conder, B., & Shields, E. W. (2012). Age-related differences and reliability on a computerized and a paper-pencil neurocognitive assessment battery. Journal of Athletic Training, 47(3), 297–305.
Retnawati, H. (2015). The Comparison of Accuracy Scores on the Paper and Pencil Testing vs. Computer-Based Testing. TOJET: Turkish Online Journal of Educational Technology, 14(4).
Revuelta, J., Ximénez, M. C., & Olea, J. (2003). Psychometric and psychological effects of item selection and review on computerized testing. Educational and Psychological Measurement, 63, 791–808.MathSciNet
Sandford, A. (2020). Coronavirus: Half of humanity now on lockdown as 90 countries call for confinement. www.​euronews.​com, 03/04/2020, accessed 25 April 2021.
Scheerens, J., Glas C., & Thomas, S. M. (2005). Educational evaluation, assessment, and monitoring: A systemic approach. Lisse: Swets & Zeitlinger B.V.
Shraim, K. (2019). Online Examination Practices in Higher Education Institutions: Learners’ Perspectives. Turkish Online Journal of Distance Education, 20(4), 185–196.
Singer, L. M., & Alexander, P. A. (2017). Reading across mediums: Effects of reading digital and print texts on comprehension and calibration. The Journal of Experimental Education, 85(1), 155–172.
Singer Trakhman, L. M., Alexander, P. A., & Berkowitz, L. E. (2019). Effects of processing time on comprehension and calibration in print and digital mediums. The Journal of Experimental Education, 87(1), 101–115.
Stobart, G. (2012). Validity in formative assessment. In J. Gardner (Ed.), Assessment and learning. (pp. 233–242). Sage.
Stricker, L. J., Wilder, G. Z., & Rock, D. A. (2004). Attitudes about the computer-based Test of English as a Foreign Language. Computers in Human Behavior, 20, 37–54.
Tahmasebi, S., & Rahimi, A. (2013). Computer-assisted assessment: Highlights and challenges. Teaching English with Technology, 13(2), 55–74.
Tavakol, M., & Dennick, R. (2011). Making Sense of Cronbach’s Alpha. International Journal of Medical Education, 2, 53–55.
Texas Education Agency. (2008). A review of literature on the comparability of scores obtained from examinees on computer-based and paper-based tests. Retrieved 28 December 2020, from https://​goo.​gl/​AAdc5o
Thurlow, M., Lazarus, S. S., Albus, D., & Hodgson, J. (2010). Computer-based testing: Practices and considerations (Synthesis Report 78). University of Minnesota, National Center on Educational Outcomes.
Vansickle, T. (2015). Test Reliability Indicates More than Just Consistency. QUESTAR ASSESSMENT, INC.
Wang, H., & Shin, C. (2010). Comparability of computerized adaptive and paper-pencil tests. Measurement and Research Service Bulletin, 13, 1–7.
Woodrow, J. (1991). A Comparison of Four Computer Attitude Scales. Journal of Educl. Computing Research, 7, 165–187.
World Health Organization. (2020). Coronavirus. Geneva.
Yurdabakan, I., & Uzunkavak, C. (2012). Primary school students’ attitudes towards computer-based testing and assessment in turkey. Turkish Online Journal of Distance Education, 13(3), 177–188.
Metadata
Title
Computerized testing in reading comprehension skill: investigating score interchangeability, item review, age and gender stereotypes, ICT literacy and computer attitudes
Author
Seyyed Morteza Hashemi Toroujeni
Publication date
03-08-2021
Publisher
Springer US
Published in
Education and Information Technologies / Issue 2/2022
Print ISSN: 1360-2357
Electronic ISSN: 1573-7608
DOI
https://doi.org/10.1007/s10639-021-10584-2

Other articles of this Issue 2/2022

Education and Information Technologies 2/2022 Go to the issue

OriginalPaper

Going virtual and going wide: comparing Team-Based Learning in-class versus online and across disciplines

OriginalPaper

Effect of instructional methods on pre-service science teachers learning outcomes: a meta-analysis

OriginalPaper

Designing and validating an assessment inventory for online language teacher education accountability

OriginalPaper

Transformation and digital literacy: Systematic literature mapping

OriginalPaper

An intelligent tutoring system architecture based on fuzzy neural network (FNN) for special education of learning disabled learners

OriginalPaper

Systemic model for technology integration in teaching

Premium Partner

    Image Credits