Top

Journal of Computing in Higher Education

Published in:

07-10-2021

Design of peer assessment rubrics for ICT topics

Authors: Lorena Bowen-Mendoza, Maricela Pinargote-Ortega, Jaime Meza, Sebastián Ventura

Published in: Journal of Computing in Higher Education | Issue 1/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Peer evaluation consists of the evaluation of students by their peers following criteria or rubrics provided by the teacher, where the way to evaluate students is specified so that they achieve the desired competencies. The quality of the measurement instrument must meet two essential criteria: validity and reliability. In this research, we explored the educational value of peer evaluation rubrics by analyzing the quality of the rubric through the study of content validity, reliability, and internal consistency. Our main purpose was to design an appropriate rubric to grade tasks in the field of information engineering, as well as performing content validation through a group of experts. It was carried out in three phases: 1) construction of a rubric, with its criteria, characteristics, and levels of achievement; 2) content validation by five experts in the field, and 3) application of the rubric to ascertain students' perceptions and satisfaction with its validity. The relevance of the criteria and the definition of their characteristics obtained a score higher than 3.75/4 on a Likert scale. The content validity values (CVR), content validity index (CVI), and general content validity index (GIVC) gave maximum values of + 1. The results obtained indicate that the rubric is adequate, with Aiken’s V higher than V 0.87 in all its criteria. The rubric was applied to 326 students of 4 subjects. Cronbach's alpha was used to calculate the reliability of the rubric, obtaining a value of 0.839. The students' perception of validity and satisfaction with the rubric was higher than 0.78. As future work, we intend to design a rubric validation engine according to the applied procedure.

previous article Grappling with professional ethics in instructional technology by participating in an online service-learning course

next article Instructors’ conceptualization and implementation of scaffolding in online higher education courses

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Aiken, L. R. (1980). Content Validity and Reliability of Single Items or Questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419CrossRef

Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066CrossRef

Alsina, Á., Ayllón, S., Colomer, J., Fernández-Peña, R., Fullana, J., Pallisera, M., & Serra, L. (2017). Improving and evaluating reflective narratives: A rubric for higher education students. Teaching and Teacher Education, 63(2000), 148–158. https://doi.org/10.1016/j.tate.2016.12.015CrossRef

ArchMiller, A., Fieberg, J., Walker, J. D., & Holm, N. (2017). Group peer assessment for summative evaluation in a graduate-level statistics course for ecologists. Assessment and Evaluation in Higher Education, 42(8), 1208–1220. https://doi.org/10.1080/02602938.2016.1243219CrossRef

Ashton, S., & Davies, R. S. (2015). Using scaffolded rubrics to improve peer assessment in a MOOC writing course. Distance Education, 36(3), 312–334. https://doi.org/10.1080/01587919.2015.1081733CrossRef

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808CrossRef

Bernat-Adell, M. D., Moles-Julio, P., Esteve-Clavero, A., & Collado-Boira, E. J. (2019). Psychometric evaluation of a rubric to assess basic performance during simulation in nursing. Nursing Education Perspectives, 40(2), E3–E6. https://doi.org/10.1097/01.NEP.0000000000000436CrossRef

Bowen, L., Pinargote, M., Meza, J., & Ventura, S. (2020). Trends the use of Artificial intelligence techniques for peer assessment. ACM International Conference Proceeding Series. https://doi.org/10.1145/3410352.3410837CrossRef

Cabero Almenara, J., del Llorente, M., & C. . (2013). La Aplicación del juicio de experto como técnica de evaluación de las tecnologías de la información y comunicación (TIC). Journal of Chemical Information and Modeling, 53(9), 1689–1699.

Capuano, N., Caballé, S., Percannella, G., & Ritrovato, P. (2020). FOPA-MC: Fuzzy multi-criteria group decision making for peer assessment. Soft Computing, 24(23), 17679–17692. https://doi.org/10.1007/s00500-020-05155-5CrossRef

Chia, D., Kulkarni, C., Cheng, J., Wei, K. P., Klemmer, S. R., Le, H., & Papadopoulos, K. (2014). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction, 20(6), 1–31. https://doi.org/10.1145/2505057CrossRef

Escobar-Pérez, J., & Cuervo-Martínez, Á. (2008). Validez de contenido y Juicio de Expertos: Una aproximación a su utilización. Avances En Medición, 6, 27–36. https://doi.org/10.1016/0032-3861(78)90049-6CrossRef

Espinoza Fernández, M. (2018). La evaluación de competencias clínicas en estudiantes de enfermería, un nuevo paradigma. Universitat Jaume I.

Fernández-Gómez, E., Martín-Salvador, A., Luque-Vara, T., Sánchez-Ojeda, M. A., Navarro-Prado, S., & Enrique-Mirón, C. (2020). Content validation through expert judgement of an instrument on the nutritional knowledge, beliefs, and habits of pregnant women. Nutrients, 12(4). https://doi.org/10.3390/nu12041136

Galicia Alarcón, L. A., Balderrama Trápaga, J. A., & Edel Navarro, R. (2017). Validez de contenido por juicio de expertos: propuesta de una herramienta virtual Content validity by experts judgment: Proposal for a virtual tool. Apert. https://doi.org/10.32870/Ap.v9n2.993

García Duarte, A. (2017). La rúbrica, en la evaluación eficiente, de las asignaturas de programación de la carrera de Ingeniería en Sistemas de Información (UNAN-Managua, FAREM-Estelí.; 151). Retrieved from http://repositorio.unan.edu.ni/id/eprint/4204

García-Ros, R. (2011). Análisis y validación de una rúbrica para evaluar habilidades de presentación oral en contextos universitarios. Electronic Journal of Research in Educational Psychology, 9(3), 1043–1062. https://doi.org/10.25115/ejrep.v9i25.1468

Grant, J. S., & Davis, L. L. (1997). Focus on Quantitative Methods: Selection and Use of Content Experts for Instrument Development. Research in Nursing and Health, 20(3), 269–274. https://doi.org/10.1002/(sici)1098-240x(199706)20:3%3c269::aid-nur9%3e3.3.co;2-3CrossRef

Guevara Rodriguez, G., Veytia Bucheli, M. G., & Sánches Macías, A. (2020). Validez y confiabilidad para evaluar la rúbrica analítica socioformativa del diseño de secuencias didácticas. Revistaespacios.Com, 41(2019), 12. Retrieved from https://www.revistaespacios.com/a20v41n09/20410912.html

Hernández Sampieri, R., Fernández-Collado, C., & Baptista-Lucio, P. (2014). Metodología de la Investigación (Sexta; S. de CV, Ed.). México: Mc Graw Hill.

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002CrossRef

Ketonen, L., Hähkiöniemi, M., Nieminen, P., & Viiri, J. (2020). Pathways through peer assessment: implementing peer assessment in a lower secondary physics classroom. International Journal of Science and Mathematics Education, 18(8), 1465–1484. https://doi.org/10.1007/s10763-019-10030-3CrossRef

Lacave Rodero, C., Molina Díaz, A., Fernández Guerrero, M., & Redondo Duque, M. (2016). Análisis de la fiabilidad y validez de un cuestionario docente. ReVisión, 9(1), 2.

Lawshe, C. H. (1975). A Quantitative Approach to Content Validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.xCrossRef

Lee, J. E., Recker, M., & Yuan, M. (2020). The validity and instructional value of a rubric for evaluating online course quality: An empirical study. Online Learning Journal, 24(1), 245–263. https://doi.org/10.24059/olj.v24i1.1949

Lindblom-Ylänne, S., Pihlajamäki, H., & Kotkas, T. (2014). Self-, peer- and teacher-assessment of student essays. 51–62. https://doi.org/10.1177/1469787406061148

Liu, J., Guo, X., Gao, R., Fram, P., Ling, Y., Zhang, H., & Wang, J. (2019). Students’ learning outcomes and peer rating accuracy in compulsory and voluntary online peer assessment. Assessment and Evaluation in Higher Education, 44(6), 835–847. https://doi.org/10.1080/02602938.2018.1542659CrossRef

Luaces, O., Díez, J., & Bahamonde, A. (2018). A peer assessment method to provide feedback, consistent grading and reduce students’ burden in massive teaching settings. Computers and Education, 126(March), 283–295. https://doi.org/10.1016/j.compedu.2018.07.016CrossRef

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386. https://doi.org/10.1097/00006199-198611000-00017CrossRef

Magdaleno Arreola, L., Dino-Morales, L. I., & Davis Vega, L. (2021). Diseño y validación de un instrumento para evaluar la práctica profesional docente desde la socioformación. Atenas , Revista Cientifico Pedagógico, 2 (54), 85–99. Retrieved from http://www.uajournals.com/campusvirtuales/journal/19/3.pdf

Montagner, I. S. (2019). An experience with peer assessment in the context of a Computer Systems course. IEEE Frontiers in Education Conference (FIE), 2019-Octob, 1–5. https://doi.org/10.1109/FIE43999.2019.9028508

Muñiz, J., & Fonseca-Pedrero, E. (2019). Ten steps for test development. Psicothema, 31(1), 7–16. https://doi.org/10.7334/psicothema2018.291CrossRef

Nicol, D., Thomson, A., & Breslin, C. (2013). Rethinking feedback in higher education: A peer reviewpersepective. Assessment and Evaluation in Higher Education., 39(1), 102–122.CrossRef

Nisbet, G., Jorm, C., Roberts, C., Gordon, C. J., & Chen, T. F. (2017). Content validation of an interprofessional learning video peer assessment tool. BMC Medical Education, 17(1), 1–10. https://doi.org/10.1186/s12909-017-1099-5CrossRef

Panadero, E., & Brown, G. T. L. (2017). Teachers’ reasons for using peer assessment: Positive experience predicts use. European Journal of Psychology of Education, 32(1), 133–156. https://doi.org/10.1007/s10212-015-0282-5CrossRef

Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129–144. https://doi.org/10.1016/j.edurev.2013.01.002CrossRef

Pedrosa, I., Suárez-Álvarez, J., & García-Cueto, E. (2013). Content Validity Evidences: Theoretical Advances and Estimation Methods. Acción Psicológica, 10(2), 3–18. Retrieved from http://dx.doi.org/https://doi.org/10.5944/ap.10.2.11820

Pedrosa, I., Suárez Álvarez, J., & García Cueto, E. (2013b). Evidencias sobre la Validez de Contenido: Avances Teóricos y Métodos para su Estimación. Acción Psicológica, 10(2), 4–11.

Planas Lladó, A., Feliu Soley, L., Fraguell Sansbelló, R. M., Arbat Pujolras, G., Pujol Planella, J., Roura-Pascual, N., & Montoro Moreno, L. (2014). Student perceptions of peer assessment: An interdisciplinary study. Assessment and Evaluation in Higher Education, 39(5), 592–610. https://doi.org/10.1080/02602938.2013.860077CrossRef

Polit, D. F., Hungler, B. P., Palacios Martinez, R., & Feher de la Torre, G. (2000). Investigación científica en ciencias de la salud : Principios y métodos (6a ed.). McGraw-Hill Interamericana.

Puerta Sierra, L., & Marín Vargas, M. E. (2015). Análisis de validez de contenido de un instrumento de transferencia de tecnología. XX Congreso Internacional de Contaduría, Administración e Informática.

Quintana, A. M. V., Rogado, A. B. G., Gavilán, A. B. R., Martín, I. R., Esteban, M. A. R., Zorrilla, T. A., & Izard, J. F. M. (2014). Application of new assessment tools in engineering studies: The rubric. Revista Iberoamericana De Tecnologias Del Aprendizaje, 9(4), 139–143. https://doi.org/10.1109/RITA.2014.2363008CrossRef

Raposo-Rivas, M., & Gallegos-Arrufat, M.-J. (2016). University students’ perceptions of electronic rubric-based assessment. Digital Education Review, 30(2013–9144), 220–233.

Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment and Evaluation in Higher Education, 35(4), 435–448. https://doi.org/10.1080/02602930902862859CrossRef

Song, Y., Hu, Z., Guo, Y., & Gehringer, E. F. (2016). An experiment with separate formative and summative rubrics in educational peer assessment. Proceedings - Frontiers in Education Conference, FIE, 2016-Novem(October). https://doi.org/10.1109/FIE.2016.7757597

Topping, K. J. (2009). Peer assessment. Theory into Practice, 48(1), 20–27. https://doi.org/10.1080/00405840802577569CrossRef

van Helvoort, J., Brand-Gruwel, S., Huysmans, F., & Sjoer, E. (2017). Reliability and validity test of a Scoring Rubric for Information Literacy. Journal of Documentation, 73(2), 305–316. https://doi.org/10.1108/JD-05-2016-0066CrossRef

Vaughan, B., Yoxall, J., & Grace, S. (2019). Peer assessment of teamwork in group projects: Evaluation of a rubric. Issues in Educational Research, 29(3), 961–978.

Vinces, L. (2018). Diseño y Validación de una e-Rúbrica para evaluar las Competencias Clínicas Específicas en Diagnóstico Diferencial en Pediatría Linna Vinces Balanzátegui. Universidad Casa Grande.

Weaver, K. F., Morales, V., Nelson, M., Weaver, P. F., Toledo, A., & Godde, K. (2016). The benefits of peer review and a multisemester capstone writing series on inquiry and analysis skills in an undergraduate thesis. CBE Life Sciences Education, 15(4), ar51.1-ar51.9. https://doi.org/10.1187/cbe.16-01-0072

Title: Design of peer assessment rubrics for ICT topics
Authors: Lorena Bowen-Mendoza
Maricela Pinargote-Ortega
Jaime Meza
Sebastián Ventura
Publication date: 07-10-2021
Publisher: Springer US
Published in: Journal of Computing in Higher Education / Issue 1/2022
Print ISSN: 1042-1726
Electronic ISSN: 1867-1233
DOI: https://doi.org/10.1007/s12528-021-09297-9

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2022

Instructors’ conceptualization and implementation of scaffolding in online higher education courses

The COVID-19 pandemic and E-learning: challenges and opportunities from the perspective of students and instructors

Digital literacy: a review in the South Pacific

Adopting a framework to support the process of critical reflection and understanding of online engagement

A differentiated learning environment in domain model for learning disabled learners

Exploring the cohesion of classroom community from the perspectives of social presence and social capital

Premium Partner