Abstract
There is great potential in making assessment and learning complementary. In this study, we investigated the feasibility of developing a desktop virtual reality (VR) laboratory simulation on the topic of genetics, with integrated assessment using multiple choice questions based on item response theory (IRT) and feedback based on the cognitive theory of multimedia learning. A pre-test post-test design was used to investigate three research questions related to: (1) students’ perceptions of assessment in the form of MC questions within the VR genetics simulation; (2) the fit of the MC questions to the assumptions of the partial credit model (PCM) within the framework of IRT; and (3) if there was a significant increase in intrinsic motivation, self-efficacy, and transfer from pre- to post-test after using the VR genetics simulation as a classroom learning activity. The sample consisted of 208 undergraduate students taking a medical genetics course. The results showed that assessment items in the form of gamified multiple-choice questions were perceived by 97% of the students to lead to higher levels of understanding, and only 8% thought that they made the simulation more boring. Items within a simulation were found to fit the PCM and the results showed that the sample had a small significant increase in intrinsic motivation and self-efficacy, and a large significant increase in transfer following the genetics simulation. It was possible to develop assessments for online educational material and retain the relevance and connectedness of informal assessment while simultaneously serving the communicative and credibility-based functions of formal assessment, which is a great challenge facing education today.
Similar content being viewed by others
References
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659–701.
Ai-Lim Lee, E., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education,55(4), 1424–1442. https://doi.org/10.1016/j.compedu.2010.06.006.
Almond, R. G., Mislevy, R. J., Steinberg, L., Yan, D., & Williamson, D. (2015). Bayesian networks in educational assessment. New York: Springer.
Andrich, D., Sheridan, B., & Luo, G. (2010). Rasch models for measurement: RUMM2030. Perth, Australia: RUMM Laboratory.
Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher,36(5), 258–267. https://doi.org/10.3102/0013189X07306523.
Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research,61(2), 213–238. https://doi.org/10.3102/00346543061002213.
Bayraktar, S. (2000). A meta-analysis on the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education,34(2), 173–189. https://doi.org/10.1080/15391523.2001.10782344.
Black, P. J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education,5(1), 7–73.
Bonde, M. T., Makransky, G., Wandall, J., Larsen, M. V., Morsing, M., Jarmer, H., et al. (2014). Improving biotechnology education through simulations and games. Nature Biotechnology,32(7), 694–697. https://doi.org/10.1038/nbt.2955.
Boud, D. (1995). Assessment and learning: Contradictory or complementary? In P. Knight (Ed.), Assessment for learning in higher education (pp. 35–48). London: Kogan.
Brown, P. C., Roediger, H. L., & McDaniel, M. A. (2014). Make it stick: The science of successful learning. Cambridge, MA: Harvard University Press.
Burbules, N. C. (2006). Rethinking the virtual. In J. Weiss, J. Nolan, J. Hunsinger, & P. Trifonas (Eds.), The international handbook of virtual learning environments (pp. 37–58). Dordrecht: Springer. https://doi.org/10.1007/978-1-4020-3803-7.
Butler, A. C., Karpicke, J. D., & Roediger, H. L., III. (2008). Correcting a metacognitive error: feedback increases retention of low-confidence correct responses. Journal of Experimental Psychology. Learning, Memory, and Cognition,34(4), 918.
Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition,36(3), 604–616.
Cranney, J., Ahn, M., McKinnon, R., Morris, S., & Watts, K. (2009). The testing effect, collaborative learning, and retrieval-induced facilitation in a classroom setting. European Journal of Cognitive Psychology,21, 919–940.
Cummings, J. J., & Bailenson, J. N. (2016). How immersive is enough? A meta-analysis of the effect of immersive technology on user presence. Media Psychology,19(2), 272–309. https://doi.org/10.1080/15213269.2015.1015740.
Dantas, A. M., & Kemm, R. E. (2008). A blended approach to active learning in a physiology laboratory-based subject facilitated by an e-learning component. Advances in Physiology Education,32, 65–75. https://doi.org/10.1152/advan.00006.2007.
De Jong, T., & Van Joolingen, W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research,68(2), 179–201.
Deci, E., Eghrari, H., Patrick, B., & Leone, D. (1994). Facilitating internalization: the self-determination theory perspective. Journal of Personality,62(1), 119–142.
DeVellis, R. F. (1991). Scale development: Theory and applications. Thousand Oaks, CA, USA: Sage.
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest,14, 4–58.
Embretson, S., & Reise, S. P. (2000). Item response theory for psychologists. Books.google.com. Retrieved from http://books.google.com/books?hl=en&lr=&ie=UTF-8&id=rYU7rsi53gQC&oi=fnd&pg=PP11&dq=ORDERED+LATENT+CLASS+MODELS+IN+NONPARAMETRIC+ITEM+RESPONSE+THEORY&ots=ZAESC95fcK&sig=Rvmsiq0-E7GYnGqw9ejqivzaKd4
Gardner, L., Sheridan, D., & White, D. (2002). A web-based learning and assessment system to support flexible education. Journal of Computer Assisted Learning,18, 125–136.
Gerjets, P., & Kirschner, P. (2009). Learning from multimedia and hypermedia. In S. Ludvigsen, et al. (Eds.), Technology-enhanced learning (pp. 251–272). Berlin: Springer.
Groth-Marnat, G. (2000). Visions of clinical assessment: Then, now, and a brief history of the future. Journal of Clinical Psychology,56(3), 349–365. https://doi.org/10.1002/(SICI)1097-4679(200003)56:3%3c349:AID-JCLP11%3e3.0.CO;2-T.
Hattie, J. (2009). Visible learning: A synthesis of 800 + meta-analyses on achievement. London: Routledge.
Johnson, C. I., & Priest, H. A. (2014). The feedback principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 449–463). Cambridge: Cambridge University Press.
Jones, N. (2018). The virtual lab: Can a simulated laboratory experience provide the same benefits for students as access to a real-world lab? Nature,562, S5–S7.
Kapur, M. (2008). Productive failure. Cognition and instruction,26(3), 379–424.
Khan, K. S., Davies, D. A., & Gupta, J. K. (2001). Formative self-assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Medical Teacher,23, 158e163.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin,119(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology,78(2), 311–328. https://doi.org/10.1037/0021-9010.78.2.311.
Kruglikova, I., Grantcharov, T. P., Drewes, A. M., & Funch-Jensen, P. (2010). The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity virtual-reality simulation: a randomised controlled trial. Gut,59(2), 181–185.
Labster. (2019). Labster—Cytogenetics lab. Retrieved January 16th 2019 from https://www.youtube.com/watch?v=_VsabhW1LkA.
Larsen, D. P., Butler, A. C., & Roediger, H. L. (2009). Repeated testing improves long-term retention relative to repeated study: A randomized controlled trial. Medical Education,43, 1174–1181.
Lee, E. A.-L., & Wong, K. W. (2014). Learning with desktop virtual reality: Low spatial ability learners are more positively affected. Computers & Education,79, 49–58. https://doi.org/10.1016/j.compedu.2014.07.010.
Lee, E. A.-L., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education,55(4), 1424–1442. https://doi.org/10.1016/j.compedu.2010.06.006.
Makransky, G., Bonde, M. T., Wulff, J. S. G., Wandall, J., Hood, M., Creed, P. A., et al. (2016). Simulation based virtual learning environment in medical genetics counseling: An example of bridging the gap between theory and practice in medical education. BMC Medical Education, 16, 98. https://doi.org/10.1186/s12909-016-0620-6.
Makransky, G., Borre-Gude, S., & Mayer, R. E. (2019a). Motivational and cognitive benefits of training in immersive virtual reality based on multiple assessments. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12375.
Makransky, G., & Lilleholt, L. (2018). A structural equation modeling investigation of the emotional value of immersive virtual reality in education. Educational Technology Research and Development,66, 1141–1164.
Makransky, G., Lilleholt, L., & Aaby, A. (2017). Development and validation of the Multimodal Presence Scale for virtual reality environments: A confirmatory factor analysis and item response theory approach. Computers in Human Behavior,72, 276–285. https://doi.org/10.1016/j.chb.2017.02.066.
Makransky, G., Mayer, R. E., Veitch, N., Hood, M., Christensen, K. B., & Gadegaard, H. (2019b). Equivalence of using a desktop virtual reality science simulation at home and in class. PLoS ONE,14(4), e0214944. https://doi.org/10.1371/journal.pone.0214944.
Makransky, G., & Petersen, G. B. (2019). Investigating the process of learning with desktop virtual reality: A structural equation modeling approach. Computers & Education. https://doi.org/10.1016/j.compedu.2019.02.002.
Makransky, G., Schnohr, C., Torsheim, T., & Currie, C. (2014). Equating the HBSC family affluence scale across survey years: A method to account for item parameter drift using the Rasch model. Quality of Life Research,23(10), 2899–2907. https://doi.org/10.1007/s11136-014-0728-2.
Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019c). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction,60, 225–236. https://doi.org/10.1016/j.learninstruc.2017.12.007.
Makransky, G., Wismer, P., & Mayer, R. E. (2018). A gender matching effect in learning with pedagogical agents in an immersive virtual reality science simulation. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12335.
Marcus, N., Ben-Naim, D., & Bain, M. (2011). Instructional support for teachers and guided feedback for students in an adaptive elearning environment. In Information Technology: New Generations (ITNG), 2011 Eighth International Conference on (pp. 626–631). IEEE.
Masters, G. N. (1982). A rasch model for partial credit scoring. Psychometrika,47(2), 149–174. https://doi.org/10.1007/BF02296272.
Mayer, R. E. (2008). Learning and instruction (2nd ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.
Mayer, R. E. (2011). Applying the science of learning. Boston: Pearson.
McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. I. I. I. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology,103(2), 399–414.
McDermott, K. B., Agarwal, P. K., D’Antonio, L., Roediger, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied,20, 3–21.
McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., & Wayne, D. B. (2011). Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Academic Medicine : Journal of the Association of American Medical Colleges,86(6), 706–711. https://doi.org/10.1097/ACM.0b013e318217e119.
McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010). A critical review of simulation-based medical education research: 2003–2009. Medical Education,44(1), 50–63. https://doi.org/10.1111/j.1365-2923.2009.03547.x.
Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education,70, 29–40. https://doi.org/10.1016/j.compedu.2013.07.033.
Meyer, O. A., Omdahl, M. K., & Makransky, G. (2019). Investigating the effect of pre-training when learning through immersive virtual reality and video: A media and methods experiment. Computers & Education. https://doi.org/10.1016/j.compedu.2019.103603.
Mislevy, R., J. (2016). Postmodern test theory. The gordon commission on the future of assessment in education. Retrieved from http://www.gordoncommission.org/rsc/pdf/mislevy_postmodern_test_theory.pdf
Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science,32(1/2), 99–113. https://doi.org/10.1023/B:TRUC.0000021811.66966.1d.
Moreno, R., & Valdez, A. (2005). Cognitive load and learning effects of having students organize pictures and words in multimedia environments: The role of student interactivity and feedback. Educational Technology Research and Development,53(3), 35–45. https://doi.org/10.1007/BF02504796.
National Research Council. (2011). Learning science through computer games and simulations. Washington: National Research Council.
Nitko, A. J. (1996). Educational assessment of students. Des Moines, IA: Prentice-Hall Order Processing Center.
Pallant, J. F., & Tennant, A. (2007). An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology,46(1), 1–18. https://doi.org/10.1348/014466506X96931.
Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.
Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: National Academies Press.
Perkins, D. (1994). Do students understand understanding? Education Digest,59(5), 21.
Pintrich, P. R. R., Smith, D., Garcia, T., & McKeachie, W. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). Ann Arbor, MI: University of Michigan.
Polly, P., Marcus, N., Maguire, D., Belinson, Z., & Velan, G. M. (2014). Evaluation of an adaptive virtual laboratory environment using Western Blotting for diagnosis of disease. BMC Medical Education,14(1), 222.
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin,138(2), 353–387. https://doi.org/10.1037/a0026838.
Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences,15(1), 20–27.
Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: The complexity of the tasks matters. Learning and Instruction,49, 142–156.
Ronen, M., & Eliahu, M. (2000). Simulation—a bridge between theory and reality: the case of electric circuits. Journal of Computer Assisted Learning,16(1), 14–26.
Rummer, R., Schweppe, J., Scheiter, K., & Gerjets, P. (2008). Lernen mit Multimedia: die kognitiven Grundlagen des Modalitätseffekts. Psychologische Rundschau,59(2), 98–107.
Rutten, N., Van Joolingen, W. R., & Van Der Veen, J. T. (2012). The learning effects of computer simulations in science education. Computers & Education,58(1), 136–153. https://doi.org/10.1016/j.compedu.2011.07.017.
Ryan, R. M., Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55 (1): 68–78. CiteSeerX10.1.1.529.4370. https://doi.org/10.1037/0003-066x.55.1.68.
Ryan, R. M., & Deci, E. L. (2016). Facilitating and hindering motivation, learning, and well-being in schools: Research and observations from self-determination theory. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (2nd ed., pp. 96–119). New York: Routledge.
Sackett, P. R., Borneman, M. J., & Connelly, B. S. (2008). High-stakes testing in higher education and employment. American Psychologist,64, 215–227. https://doi.org/10.1037/0003-066X.63.4.215.
Sadler, D. R. (1998). Formative assessment: revisiting the territory. Assessment in Education,5(1), 77e84.
Schraw, G., Mayrath, M. C., ClarkeMidura, J., & Robinson, D. H. (Eds.). (2012). Technology based assessments for 21st century skills: Theoretical and practical implications from modern research. IAP
Schunk, D. H., & DiBenedetto, M. K. (2016). Self-efficacy theory in education. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (pp. 34–54). New York: Routledge.
Shavelson, R. J., Young, D. B., Ayala, C. C., Brandon, P. R., Furtak, E. M., Ruiz-Primo, M. A., ··· & Yin, Y. (2008). On the impact of curriculum-embedded formative assessment on learning: A collaboration between curriculum and assessment developers. Applied Measurement in Education, 21(4), 295–314.
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research,78(1), 153–189. https://doi.org/10.3102/0034654307313795.
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer Games and Instruction,55(2), 503–524.
Shute, V. J., & Becker, B. J. (2010). Innovative assessment for the 21st century. New York, NY: Springer.
Shute, V. J., & Ke, F. (2012). Assessment in game-based learning. In D. Eseryel (Ed.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 43–58). New York: Springer. https://doi.org/10.1007/978-1-4614-3546-4.
Shute, V. J., Leighton, J. P., Jang, E. E., & Chu, M.-W. (2016). Advances in the science of assessment. Educational Assessment,21(1), 34–59. https://doi.org/10.1080/10627197.2015.1127752.
Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. Serious Games: Mechanisms and Effects,2, 295–321.
Smith Jr., E. V. (2002). Detecting and evaluating the impact of multidimensionality using item fit statistics and principal component analysis of residuals. Journal of Applied Measurement, 3(2), 205–231. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/12011501
Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2004). Classroom assessment for student learning: doing it right–using it well. Portland: Assessment Training Institute.
Strandbygaard, J., Bjerrum, F., Maagaard, M., Winkel, P., Larsen, C. R., Ringsted, C., ··· & Sorensen, J. L. (2013). Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. Annals of Surgery, 257(5), 839–844.
Tennant, A., & Conaghan, P. G. (2007). The Rasch measurement model in rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Care and Research,57(8), 1358–1362. https://doi.org/10.1002/art.23108.
Thisgaard, M., & Makransky, G. (2017). Virtual learning simulations in high school: Effects on cognitive and non-cognitive outcomes and implications on the development of STEM academic and career choice. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2017.00805.
Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education,81, 259–269. https://doi.org/10.1016/j.compedu.2014.10.013.
Uner, O., & Roediger, H. L. (2018). The effect of question placement on learning from textbook chapters. Journal of Applied Research in Memory and Cognition,7(1), 116–122.
Wigfield, A., Guthrie, J. T., Tonks, S., & Perencevich, K. C. (2004). Children’s motivation for reading: Domain specificity and instructional influences. Journal of Educational Research,97(6), 299–309. https://doi.org/10.3200/joer.97.6.299-310.
Williams, J. R. (2008). The declaration of Helsinki and public health. Bulletin of the World Health Organization. https://doi.org/10.2471/BLT.08.050955.
Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education,13(2), 181–208.
Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology,25(1), 82–91.
Funding
This research was funded by Innovation fund Denmark.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Mads Bonde is a co-founder of the simulation development company Labster that provided the simulation that was used in this study. Ainara Lopez Cordoba works at Labster. The remaining authors declare that they have no conflict of interest.
Informed consent
Ethical consent was obtained from all participants in accordance with the ethical regulations of the Health Research Ethics Committee in Denmark.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix 1: Questionnaire items and sources
Appendix 1: Questionnaire items and sources
Rights and permissions
About this article
Cite this article
Makransky, G., Mayer, R., Nøremølle, A. et al. Investigating the feasibility of using assessment and explanatory feedback in desktop virtual reality simulations. Education Tech Research Dev 68, 293–317 (2020). https://doi.org/10.1007/s11423-019-09690-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-019-09690-3