Top

Education and Information Technologies

Published in:

01-12-2015

Data science in educational assessment

Authors: David C. Gibson, Mary E. Webb

Published in: Education and Information Technologies | Issue 4/2015

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This article is the second of two articles in this special issue that were developed following discussions of the Assessment Working Group at EDUsummIT 2013. The article extends the analysis of assessments of collaborative problem solving (CPS) to examine the significance of the data concerning this complex assessment problem and then for educational assessment more broadly. The article discusses four measurement challenges of data science or ‘big data’ in educational assessments that are enabled by technology: 1. Dealing with change over time via time-based data. 2. How a digital performance space’s relationships interact with learner actions, communications and products. 3. How layers of interpretation are formed from translations of atomistic data into meaningful larger units suitable for making inferences about what someone knows and can do. 4. How to represent the dynamics of interactions between and among learners who are being assessed by their interactions with each other as well as with digital resources and agents in digital performance spaces. Because of the movement from paper-based tests to online learning, and in order to make progress on these challenges, the authors call for the restructuring of training of the next generation of researchers and psychometricians to specialize in data science in technology enabled assessments.

previous article Technology enhanced assessment in complex collaborative settings

next article Computational thinking in compulsory education: Towards an agenda for research and practice

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Al-diban, S., & Ifenthaler, D. (2011). Comparison of two analysis approaches for measuring externalized mental models. Educational Technology and Society, 14(2), 16–30.

Aldrich, C. (2005). Learning by doing : the essential guide to simulations, computer games, and pedagogy in e-learning and other educational experiences. San Francisco: Jossey-Bass.

Bar-Yam, Y. (1997). Dynamics of complex systems. Reading: Addison-Wesley.MATH

Bates, D., & Watts, D. (1988). Nonlinear regression analysis and its applications. John Wiley & Sons (Vol. 32). doi:10.2307/2289810.

Bennett, R. (2010). Cognitively based assessment of, for, and as learning (CBAL): a preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspective, 8(2–3), 70–91. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/15366367.2010.508686..

Black, P., Harrison, C., Hodgen, J., Marshall, B., & Serret, N. (2010). Validity in teachers’ summative assessments. Assessment in Education: Principles, Policy & Practice, 17(2), 215–232.

Bollen, J., Van de Sompel, H., Hagberg, A., Bettencourt, L., Chute, R., Rodriguez, M. A., & Balakireva, L. (2009). Clickstream data yields high-resolution maps of science. PloS One, 4, e4803. doi:10.1371/journal.pone.0004803.

Borgatti, S. P., & Halgin, D. S. (2011). On network theory. Organization Science, 22, 1168–1181. doi:10.1287/orsc.1100.0641.CrossRef

Brandes, U., & Erlebach, T. (2005). Network Analysis: Methodological Foundations. Lecture Notes in Computer Science (Vol. 3418). doi:10.1007/b106453.

Choi, Y., Rupp, A., Gushta, M., & Sweet, S. (2010). Modeling learning trajectories with epistemic network analysis: An investigation of a novel analytic method for learning progressions in epistemic games. In National Council on Measurement in Education (pp. 1–39).

Clarke-Midura, J., & Dede, C. (2010). Assessment, technology, and change. Journal of Research in Teacher Education, 42(3), 309–328.CrossRef

Clarke-Midura, J., Code, J., Dede, C., Mayrath, M., & Zap, N. (2012). Thinking outside the bubble: Virtual performance assessments for measuring complex learning. In Technology-based assessments for 21st Century skills: Theoretical and practical implications from modern research (pp. 125–148). Charlotte: Information Age Publishers.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302. doi:10.1037/h0040957.CrossRef

De Freitas, S. (2014). Education in computer generated environments. London: Routledge.

DeFreitas, S., Gibson, D., & Morgan, J. (2015). Will MOOCS transform teaching and learning in higher education? British Journal of Educational Technology, in press.

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–1101. doi:10.1037/0022-3514.92.6.1087.CrossRef

Gibson, D. (2012). Game changers for transforming learning environments. In F. Miller (Ed.), Transforming Learning Environments: Strategies to Shape the Next Generation (Advances in Educational Administration, Volume 16) (pp. 215 – 235). Emerald Group Publishing Ltd. doi:10.1108/S1479-3660(2012)0000016014

Gibson, D., & Clarke-Midura, J. (2013). Some Psychometric and Design Implications of Game-Based Learning Analytics. In D. Ifenthaler, J. Spector, P. Isaias, & D. Sampson (Eds.), E-Learning Systems, Environments and Approaches: Theory and Implementation. London: Springer.

Gibson, D., & de Freitas, S. (2015). Exploratory analysis in learning analytics. Technology, Knowledge and Learning. doi:10.1007/s10758-015-9249-5.

Gibson, D., & Jakl, P. (2013). Data challenges of leveraging a simulation to assess learning. CA: West Lake Village. Retrieved from http://www.curveshift.com/images/Gibson_Jakl_data_challenges.pdf.

Gibson, D., & Knezek, G. (2011). Game Changers for Teacher Education. In P. Mishra & M. Koehler (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2011 (pp. 929–942). Chesapeake: VA: AACE.: AACE.

Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: a meta-analysis from the angle of assessment. Review of Educational Research, 75, 27–61.CrossRef

Han, J., Cheng, H., Xin, D., & Yan, X. (2007). Frequent pattern mining: current status and future directions. Data Mining and Knowledge Discovery, 15(1), 55–86. doi:10.1007/s10618-006-0059-1.MathSciNetCrossRef

IBM. (2015). Big Data. Retrieved from http://www-01.ibm.com/software/au/data/bigdata/.

Ifenthaler, D., Eseryel, D., Ge, X., & Ge, X. (2012). In D. Ifenthaler & D. Eseryel (Eds.), Assessment in game-based learning. Lobdon: Springer.CrossRef

Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45, 65–90. doi:10.1007/BF02299613.CrossRef

Khan, S. (2011). Khan Academy. Educational Technology. Retrieved from http://www.khanacademy.org/.

Kline, P. (1998). The new psychometrics: Science, psychology, and measurement. London: Routledge.

Kopainsky, B., Pirnay-dummer, P., & Alessi, S. M. (2010). Automated assessment of learner’s understanding in complex dynamic systems. In 28th International Conference of the System Dynamics Society (pp. 1–39). Seoul.

Kozleski, E., Gibson, D., & Hynds, A. (2012). Changing complex educational systems: Frameworks for collaborative social justice leadership. In C. Gersti-Pepin & J. Aiken (Eds.), Defining social justice leadership in a global context (pp. 263–286). Charlotte: Information Age Publishing.

Mansell, W., James, M., & the Assessment Reform Group. (2009). Assessment in schools. Fit for purpose? A Commentary by the Teaching and Learning Research Programme. London. Retrieved from http://www.tlrp.org/pub/documents/assessment.pdf.

Margetts, H., & Sutcliffe, D. (2013). Addressing the policy challenges and opportunities of “big data”. Policy and Internet, 5(2), 139–146. doi:10.1002/1944-2866.POI326.CrossRef

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.CrossRef

Mislevy, R., Steinberg, L., & Almond, R. (1999). Evidence-centered assessment design. Educational Testing Service. Retrieved from http://www.education.umd.edu/EDMS/mislevy/papers/ECD_overview.html

Mislevy, R., Steinberg, L., & Almond, R. (2003a). On the structure of educational assessments. Russell the Journal of the Bertrand Russell Archives, 1(1), 3–62.

Mislevy, R., Wilson, M., Ercikan, K., & Chudowsky, N. (2003). Psychometric Principles in Student Assessment. In T. Kellaghan & D. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 489–531). Kluwer Int.

Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs: Prentice-Hall.

Patton, M. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: The Guilford Press.

Pirnay-Dummer, P., Ifenthaler, D., & Spector, M. (2010). Highly integrated model assessment technology and tools. Educational Technology Research and Development, 58(1), 3–18.CrossRef

PISA. (2013). Draft collaborative problem solving framework. OECD: Organization for Economic and Community Development.

Prensky, M. (2001). Digital game-based learning. New York: McGraw-Hill.

Psychometrics. (2014). In American Heritage Dictionary. Houghton-Mifflin Company

Quellmalz, E., Timms, M., & Schneider, S. (2009). Assessment of Student Learning in Science Simulations and Games. DC: National Research Council.

Quellmalz, E., Timms, M., Buckley, B., Davenport, J., Loveland, M., & Silberglitt, M. (2012). 21st century dynamic assessment. In M. Mayrath, J. Clarke-Midura, & D. Robinson (Eds.), Technology-based assessments for 21st Century skills: Theoretical and practical implications from modern research (pp. 55–90). Charlotte: Information Age Publishers.

Scheffel, M., Niemann, K., & Leony, D. (2012). Key action extraction for learning analytics. 21st Century Learning (pp.320–333). Retrieved from http://link.springer.com/chapter/10.1007/978-3-642-33263-0_25.

Schmidt, M., & Lipson, H. (2009). Symbolic regression of implicit equations. Genetic Programming Theory and Practice, 7, 73–85.

Shaffer, D., Hatfield, D., Svarovsky, G., Nash, P., Nulty, A., Bagley, E., & Mislevy, R. (2009). Epistemic network analysis: a prototype for 21st-century assessment of learning. International Journal of Learning and Media, 1(2), 33–53. Retrieved from http://www.mitpressjournals.org/doi/abs/10.1162/ijlm.2009.0013.

Sporns, O. (2011). Networks of the brain. Cambridge: MIT Press.

Sterman, J. D. (1994). Learning in and about complex systems. System Dynamics Review, 10(2 / 3), 291.CrossRef

Stevens, R. (2006). Machine learning assessment systems for modeling patterns of student learning. In D. Gibson, C. Aldrich, & M. Prensky (Eds.), Games and simulations in online learning: Research & development frameworks (pp. 349–365). Hershey: Idea Group.

Stevens, R., & Palacio-Cayetano, J. (2003). Design and performance frameworks for constructing problem-solving simulations. Cell Biology Education, 2(Fall), 162–179.CrossRef

Stevens, R., Johnson, D., & Soller, A. (2005). Probabilities and predictions: modeling the development of scientific problem-solving skills. Cell Biology Education, 4(1), 42–57. doi:10.1187/cbe.04-03-0036.CrossRef

Stiggins, R. J. (1995). Assessment Literacy for the 21st Century. Phi Delta Kappan, 77.

Sugihara, G., May, R., Ye, H., Hsieh, C., Deyle, E., Fogarty, M., & Munch, S. (2012). Detecting causality in complex ecosystems. Science (New York, N.Y.), 338(6106), 496–500. doi:10.1126/science.1227079.CrossRef

Tashakkori, A., & Teddlie, . (2003). Handbook of Mixed Methods in Social & Behavioral Research. SAGE. Retrieved from http://books.google.com/books?id=F8BFOM8DCKoC&pgis=1.

Thagard, P. (2010). How brains make mental models. Retrieved from cogsci.uwaterloo.ca/Articles/Thagard.brains-models.2010.pdf.

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning Analytics Dashboard Applications. American Behavioral Scientist. doi:10.1177/0002764213479363.

Voogt, J., & Knezek, G. (2008). International Handbook of Information Technology in Primary and Secondary Education. New York: Springer.CrossRef

Webb, M., & Gibson, D. (2015). Technology enhanced assessment in complex collaborative settings. Education and Information Technologies. doi:10.1007/s10639-015-9413-5.

Webb, M., Gibson, D., & Forkosh-Baruch, A. (2013). Challenges for information technology supporting educational assessment. Journal of Computer Assisted Learning, 29(5), 451–462. doi:10.1111/jcal.12033.CrossRef

Title: Data science in educational assessment
Authors: David C. Gibson
Mary E. Webb
Publication date: 01-12-2015
Publisher: Springer US
Published in: Education and Information Technologies / Issue 4/2015
Print ISSN: 1360-2357
Electronic ISSN: 1573-7608
DOI: https://doi.org/10.1007/s10639-015-9411-7

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 4/2015

Technology enhanced assessment in complex collaborative settings

Digital equity and intercultural education

Preparing teachers for schooling in the digital age: A meta-perspective on existing strategies and future challenges

Teachers’ professional development for ICT integration: Towards a reciprocal relationship between research and practice

Computational thinking in compulsory education: Towards an agenda for research and practice

Digital citizenship within global contexts

Premium Partner