Skip to main content
SpringerLink
Log in

Computer-based assessment of Complex Problem Solving: concept, implementation, and application

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Complex Problem Solving (CPS) skills are essential to successfully deal with environments that change dynamically and involve a large number of interconnected and partially unknown causal influences. The increasing importance of such skills in the 21st century requires appropriate assessment and intervention methods, which in turn rely on adequate item construction, delivery, and scoring. The lack of assessment tools, however, has slowed down research on and understanding of CPS. This paper first presents the MicroDYN framework for assessing CPS, which is based on linear structural equation systems with input and output variables and opaque relations among them. Second, a versatile assessment platform, the CBA Item Builder, which allows the authoring, delivery, and scoring of CPS tasks for scientific and educational purposes is introduced. Third, we demonstrate the potential of such a tool for research by reporting an experimental study illustrating the effect of domain specific content knowledge on performance in CPS tasks both on an overall performance and on a process level. The importance of accessible and versatile technical platforms not only for assessment and research but also for intervention and learning are discussed with a particular focus on educational contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abele, S., Greiff, S., Gschwendtner, T., Wüstenberg, S., Nickolaus, R., Nitschke, A., et al. (2012). Die Bedeutung übergreifender kognitiver Determinanten für die Bewältigung beruflicher Anforderungen. Untersuchung am Beispiel dynamischen und technischen Problemlösens [The importance of general cognitive determinants in mastering job demands. Some research on the example of dynamic and technical problem solving]. Zeitschrift für Erziehungswissenschaft, 15, 363–391.

    Article  Google Scholar 

  • Autor, D., & Dorn, D. (2009). This job is “Getting Old”: Measuring changes in job opportunities using occupational age structure. American Economic Review, 99, 45–51.

    Article  Google Scholar 

  • Autor, D. H., Levy, F., & Murnane, R. J. (2003). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118, 1279–1333.

    Article  Google Scholar 

  • Buchner, A. (1995). Basic topics and approaches to the study of complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 27–63). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Bunderson, C. V., Inouye, D. K., & Olsen, J. B. (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational Measurement (pp. 367–407). New York, NY: Macmillan Publishing.

    Google Scholar 

  • Chen, Z., & Klahr, D. (1999). All other things being equal: Acquisition and transfer of the Control of Variables Strategy. Child Development, 70, 1098–1120.

    Article  Google Scholar 

  • Danner, D., Hagemann, D., Holt, D. V., Hager, M., Schankin, A., Wüstenberg, S., et al. (2011). Measuring performance in a complex problem solving task: Reliability and validity of the Tailorshop simulation. Journal of Individual Differences, 32, 225–233.

    Article  Google Scholar 

  • Dörner, D. (1986). Diagnostik der operativen Intelligenz [Assessment of operative intelligence]. Diagnostica, 32(4), 290–308.

    Google Scholar 

  • Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. Journal of Problem Solving, 4(1), 19–42.

    Google Scholar 

  • Frensch, P. A., & Funke, J. (Eds.). (1995). Complex problem solving: The European perspective. Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Funke, J. (1992). Dealing with dynamic systems: Research strategy, diagnostic approach and experimental results. German Journal of Psychology, 16, 24–43.

    Google Scholar 

  • Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking and Reasoning, 7(1), 69–89.

    Article  Google Scholar 

  • Funke, J. (2003). Problemlösendes Denken [Problem solving]. Stuttgart: Kohlhammer.

    Google Scholar 

  • Funke, J. (2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11(2), 133–142.

    Article  Google Scholar 

  • Fyfe, E. R., Rittle-Johnson, B., & DeCaro, M. S. (2012). The effects of feedback during exploratory mathematics problem solving: Prior knowledge matters. Journal of Educational Psychology, 104, 1094–1108.

    Article  Google Scholar 

  • Gonzalez, C., Thomas, R. P., & Vanyukov, P. (2005). The relationships between cognitive ability and dynamic decision making. Intelligence, 33(2), 169–186.

    Article  Google Scholar 

  • Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45, 298–322.

    Article  Google Scholar 

  • Greiff, S., & Fischer, A. (2013). Der Nutzen einer Komplexen Problemlösekompetenz: Theoretische Überlegungen und empirische Befunde [Usefulness of Complex Problem Solving competency: Theoretical considerations and empirical results]. Zeitschrift für Pädagogische Psychologie, 27(1), 1–13.

    Article  Google Scholar 

  • Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic Problem Solving: A new measurement perspective. Applied Psychological Measurement, 36, 189–213.

    Article  Google Scholar 

  • Greiff, S., Wüstenberg, S., Molnar, G., Fischer, A., Funke, J., & Csapo, B. (in press). Complex Problem Solving in educational settings—something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology. doi:10.1037/a0031856.

  • Holyoak, K. J. (1985). The pragmatics of analogical transfer. In G. H. Bower (Ed.), The psychology of learning and motivation (pp. 59–87). New York, NJ: Academic Press.

    Google Scholar 

  • Kalyuga, S. (2009). Knowledge elaboration: A cognitive load perspective. Learning and Instruction, 19, 402–410.

    Article  Google Scholar 

  • Klahr, D. (2000). Exploring science: The cognition and development of discovery processes. Cambridge, MA: MIT Press.

    Google Scholar 

  • Kluge, A. (2008). Performance assessment with microworlds and their difficulty. Applied Psychological Measurement, 32(2), 156–180.

    Article  Google Scholar 

  • Kröner, S., Plass, J. L., & Leutner, D. (2005). Intelligence assessment with computer simulations. Intelligence, 33, 347–368.

    Article  Google Scholar 

  • Kyllonen, P. C. (2009). New constructs, methods, and directions for computer-based assessment. In F. Scheuermann & J. Björnsson (Eds.), The transition to computer-based assessment (pp. 151–156). Luxembourg: Office for Official Publications of the European Communities.

    Google Scholar 

  • Leutner, D. (2002). The fuzzy relationship of intelligence and problem solving in computer simulations. Computers in Human Behavior, 18, 685–697.

    Article  Google Scholar 

  • Mayer, R. E. (2003). Learning and instruction. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Mayer, R. E., & Wittrock, M. C. (2006). Problem solving. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 287–303). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Novick, L. R., & Bassok, M. (2005). Problem solving. In K. J. Holyoak & R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp. 321–349). Cambridge, NY: University Press.

    Google Scholar 

  • OECD (2010). PISA 2012 field trial problem solving framework. Accessed at: http://www.oecd.org/dataoecd/8/42/46962005.pdf.

  • OECD (2012). Better skills, better jobs, better lives. Highlights of the OECD skills strategy. Accessed at: http://www.oecd-ilibrary.org/docserver/download/fulltext/9112165e5.pdf.

  • Perkins, D. N., & Salomon, G. (1989). Are cognitive skills context-bound? Educational Researcher, 18(10), 16–25.

    Article  Google Scholar 

  • Raven, J. (2000). Psychometrics, cognitive ability, and occupational performance. Review of Psychology, 7, 51–74.

    Google Scholar 

  • Rölke, H. (2012). The item builder: A graphical authoring system for complex item development. in Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education (pp. 344–353). Chesapeake, VA: AACE.

  • Scheuermann, F., & Björnsson, J. (2009). The transition to computer-based assessment. Luxembourg: Office for Official Publications of the European Communities.

    Google Scholar 

  • Smith, M. C., & Reio, T. G. (2006). Adult development, schooling, and the transition to work. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 115–138). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Spitz-Oener, A. (2006). Technical change, job tasks, and rising educational demands: Looking outside the wage structure. Journal of Labor Economics, 24(2), 235–270.

    Article  Google Scholar 

  • Sternberg, R. J. (1995). Expertise in complex problem solving: A comparison of alternative concepts. In P. A. Frensch & J. Funke (Eds.), Complex problem solving. The European perspective (pp. 295–321). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Süß, H.-M. (1996). Intelligenz, Wissen und Problemlösen. Göttingen: Hogrefe.

    Google Scholar 

  • Triona, L. M., & Klahr, D. (2003). Point and click or grab and heft: Comparing the influence of physical and virtual instructional materials on elementary school students’ ability to design experiments. Cognition and Instruction, 21, 149–173.

    Article  Google Scholar 

  • Tschirgi, J. E. (1980). Sensible reasoning: A hypothesis about hypotheses. Child Development, 51, 1–10.

    Article  Google Scholar 

  • Vollmeyer, R., Burns, B. D., & Holyoak, K. J. (1996). The impact of goal specificity on strategy use and the acquisition of problem structure. Cognitive Science, 20, 75–100.

    Article  Google Scholar 

  • Williamson, D. M., Bejar, I. I., & Mislevy, R. J. (2006). Automated scoring of complex tasks in computer-based testing: An introduction. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Wirth, J., & Klieme, E. (2003). Computer-based assessment of problem solving competence. Assessment in Education: Principles, Policy & Practice, 10(3), 329–345.

    Article  Google Scholar 

  • Wittmann, W. W., & Süß, H. M. (1999). Investigating the paths between working memory, intelligence, knowledge and complex problem solving: Performances via Brunswik-symmetry. In P. L. Ackermann, P. C. Kyllonen, & R. D. Roberts (Eds.), Learning and individual differences: Process, trait and content (pp. 77–108). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving. More than reasoning? Intelligence, 40, 1–14.

    Article  Google Scholar 

Download references

Acknowledgments

This research was funded by a Grant of the German Research Foundation (DFG Fu 173/14-2), by the European Union (290683; LLLight'in'Europe), and by the German Federal Ministry of Education and Research (LSA004). We are grateful to the TBA group at DIPF (http://tba.dipf.de) for providing the authoring tool CBA Item Builder and technical support.

Author information

Authors and Affiliations

  1. EMACS Unit, University of Luxembourg, 6, rue Richard Coudenhove Kalergi, 1359, Luxembourg-Kirchberg, Luxembourg

    Samuel Greiff & Sascha Wüstenberg

  2. University of Heidelberg, Heidelberg, Germany

    Daniel V. Holt & Joachim Funke

  3. German Institute for International Educational Research (DIPF) and Centre for International Student Assessment (ZIB), Frankfurt, Germany

    Frank Goldhammer

Authors
  1. Samuel Greiff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sascha Wüstenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel V. Holt
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Frank Goldhammer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joachim Funke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samuel Greiff.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Greiff, S., Wüstenberg, S., Holt, D.V. et al. Computer-based assessment of Complex Problem Solving: concept, implementation, and application. Education Tech Research Dev 61, 407–421 (2013). https://doi.org/10.1007/s11423-013-9301-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-013-9301-x

Keywords

Search

Navigation