Assessing computational thinking: A systematic review of empirical studies

Highlights

• A review of current CT assessments about context, construct, assessment type, and psychometric evidence.

• Fewer CT assessments were examined in high school, college, and professional development than in elementary school.

• Most CT assessments focused on students' programming or computing skills.

• Traditional and portfolio assessments were often used to assess CT, and surveys were used to measure CT dispositions.

• Half of the studies reported reliability and validity evidence of their CT assessment.

Abstract

With the increasing attention to Computational Thinking (CT) in education, there has been a concomitant rise of needs and interest in investigating how to assess CT skills. This study systematically reviewed how CT has been assessed in the literature. We reviewed 96 journal articles to analyze specific CT assessments from four perspectives: educational context, assessment construct, assessment type, and reliability and validity evidence. Our review results indicate that (a) more CT assessments are needed for high school, college students, and teacher professional development programs, (b) most CT assessments focus on students' programming or computing skills, (c) traditional tests and performance assessments are often used to assess CT skills, and surveys are used to measure students’ CT dispositions, and (d) more reliability and validity evidence needs to be collected and reported in future studies. This review identifies current research gaps and future directions to conceptualize and assess CT skills, and the findings are expected to be beneficial for researchers, curriculum designers, and instructors.

Introduction

Computational Thinking (CT) has drawn increasing attention in the field of science, technology, engineering, and mathematics (STEM) education since Wing (2006) promoted it. Cuny, Snyder, and Wing (2010) defines CT as “the thought processes involved in formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an information-processing agent (Wing, 2011, p. 1, p. 1)”. Not only is CT grounded on concepts fundamental to computer science (CS), a field that has dramatically impacted society (Wing, 2006; 2008), but it is integral to modern research and problem-solving work of STEM (Henderson, Cortina, & Wing, 2007). Thus, CT should be embedded in the educational system as a substantial learning goal to prepare students with competency in their future life (Grover & Pea, 2013). The International Society for Technology in Education and the Computer Science Teachers Association (CSTA & ISTE, 2011) developed resources on approaches to bringing CT into K-12 settings. Meanwhile, the National Research Council (NRC) organized two workshops with scholars in CS and education around the scope and nature of CT and the pedagogical aspects of CT (National Research Council, 2010; 2011; 2012). These efforts together with others in the education community promoted a milestone in STEM education in 2013—the Next Generation Science Standards (NGSS Lead States, 2013) listed “using mathematics and computational thinking” as one of the Science and Engineering Practices that integrate both disciplinary core ideas and crosscutting concepts.

Drawing upon the growing interest in integrating CT in STEM education, the field dedicated many efforts to promote and examine students' CT skills. These efforts include, but are not limited to the following ones: (a) developing CT-integrated curriculum (e.g., Rich, Spaepen, Strickland, & Moran, 2019; Sung, 2019); (b) inventing CT-inspired teaching and learning tools (e.g., Bers, 2010; Grover, 2017a, 2017b; Weintrop et al., 2014); (c) building CT-embedded learning environment (e.g., Muñoz-Repiso & Caballero-González, 2019), and (d) developing assessments focusing on students' CT skills (e.g., González, 2015; Korkmaz, Çakir, & Özden, 2017a, 2017b). These studies generated a body of literature that help us understand the nature of CT, the CT integration in STEM classrooms, and the features of students’ performance in CT practices.

Some researchers have synthesized the work related to CT. Lockwood and Mooney (2018) summarized CT research in secondary education and provided information on the subjects used to teach CT, the tools used to teach and assess CT, and benefits and barriers of incorporating CT in secondary education. Hsu, Chang, and Hung (2018) explored teaching and learning activities and strategies when promoting CT. However, systematic reflection is lacking on the evaluation tools of student CT skills and performance used in these studies, making future research directions on promotion and assessment of CT practices unclear. In fact, both Lockwood and Mooney (2018) and Hsu et al. (2018) rendered it pressing to address the education community's uncertainty in how to best assess CT skills in their reviews.

Our study aims to systematically review CT assessment research in more detail than previous reviews regarding CT implementation contexts, CT and CT-related constructs, CT assessment tools, and their reliability and validity evidence across all educational levels. In particular, our review focuses specifically on the CT studies that apply assessments for the educational levels from kindergarten to college education and professional development for teachers. In addition, we reviewed these studies in terms of what educational contexts and subject domains these studies are conducted at, what types of the CT constructs are measured (we classified the constructs in terms of CT and CT-related learning outcomes), what assessment tools have been employed to measure these types of CT constructs, and finally what the reliability and validity evidence are being reported. To sum, Lockwood and Mooney (2018) provided a big picture of current CT studies in secondary education, while our review contributes more detailed information regarding the development and implementation of CT assessments.

In 1980, Seymour Papert first used the term “computational thinking” and suggested that computers might enhance thinking and change patterns of knowledge accessibility (Papert, 1980). He further emphasized that all children should have access to computers as a way to shape their learning and express their ideas (Papert, 1996). Later, in her influential article, Wing (2006) echoed these ideas by illuminating the concept of CT and broadcasting its applications in problem solving. She stated that CT should be at the core of K-12 curricula and called for research on effective ways of teaching CT to students. Since then, CT has drawn increasing attention from educators and educational researchers and has been identified as a critical competence that would equip students with foundational skills to learn STEM (Weintrop et al., 2014).

Despite of a research history of around 15 years, the field has not reached a consensus on the definition of CT (National Research Council, 2011). As such, we constructed a diagram to illustrate some of the well-cited definitions (See in Fig. 1). As shown in the figure, many researchers defined CT in a way of drawing from programming and computing concepts. For example, Brennan and Resnick (2012) developed a theoretical framework of CT that involves three key dimensions, in which one of them refers to computational concepts, including programming terms of sequences, loops, parallelism, events, conditionals, operators, and data. The other two dimensions are computational practices, including the processes of iteration, debugging, and abstraction, and computational perspectives, including expressing, connecting, and questioning. Another defining framework that originates from computing concepts were proposed by Weintrop et al. (2016). They classified CT into four major categories with 22 sub-skills: data practices, modeling & simulation practices, computational problem-solving practices, and systems thinking practices. Based on this framework, they developed a series of CT enhanced lesson plans for high school STEM classrooms. Denner, Werner, and Ortiz (2012) defined CT as a united competence, which is composed of three key dimensions of CT: programming, documenting and understanding software, and designing for usability.

Different from the skills in working with computing or programming activities, some researchers regarded CT as a set of competences requiring students to develop both domain-specific knowledge and problem-solving skills. For example, CSTA & ISTE, 2011 provided a list of vocabularies for CT: algorithms & procedures, automation, simulation, parallelization, algorithms and procedures, automation, simulation, parallelization. They suggested that those skills can be used to solve problems in everyday life, different subject domains, and across different grade levels. Selby and Woollard (2013) proposed operational definitions of CT skills including abstraction, decomposition, algorithmic thinking, evaluation, and generalization. As this operational definition is based on a meta-analysis of various studies of CT, it is broadly adopted in many studies (e.g., Atmatzidou & Demetriadis, 2016; Leonard et al., 2018). During an experiment that assessed the impact of CT modules on preservice teachers, Yadav, Mayfield, Zhou, Hambrusch, and Korb (2014) explained five CT concepts: problem identification and decomposition, abstraction, logical thinking, algorithms, and debugging with concrete examples from day-to-day life and related these concepts to preservice teachers’ personal experiences.

Despite the controversies surrounding the definition of CT, researchers have agreed that involving CT has an immense potential to transform how we approach subject domains in classrooms (e.g., Barr, Harrison, & Conery, 2011; Barr & Stephenson, 2011; Jona et al., 2014; Lee et al., 2011; Repenning, Webb, & Ioannidou, 2010; Wing, 2006). Weintrop et al. (2016) concluded with three main benefits of embedding CT into STEM classrooms: building a reciprocal connection between math, science, and CT; constructing a more accessible classroom context for teachers and students; and making math and science classrooms updated with current professional practices.

In addition to the theoretical discussion, many studies have attempted to integrate CT in classrooms. For example, Malyn-Smith and Lee (2012) have facilitated the exploration of CT as a foundational skill for STEM professionals and how professionals engaged CT in routine work and problem solving. Later, Lee, Martin, and Apone (2014) integrated CT into K-8 classrooms through three types of fine-grained computational activities: digital storytelling, data collection and analysis, and computational science investigations. Further, researchers have developed CT interventions on various subject domains such as biology and physics (Sengupta, Kinnebrew, Basu, Biswas, & Clark, 2013), journalism and expository writing (Wolz, Stone, Pearson, Pulimood, & Switzer, 2011; Wolz, Stone, Pulimood, & Pearson, 2010), science in general (Basu, Biswas, & Kinnebrew, 2017a, 2017b; Basu et al., 2016; Weintrop et al. 2016), sciences and arts (Sáez-López, Román-González, & Vázquez-Cano, 2016a, 2016b), and mathematics (Wilkerson-Jerde, 2014).

Taken together, researchers have illustrated the importance of teaching and learning CT and its integration in other subject domains. However, as indicated by Kalelioğlu, Gülbahar, and Kukul (2016) in their review on CT studies, the current CT concept and definition lacks scientific justifications. Not surprisingly, researchers would hold different perspectives when applying, interpreting, and assessing the proposed CT concept and definition.

Assessment plays a critical role when educators introduce CT into K-12 classrooms (Grover & Pea, 2013). Kalelioğlu et al. (2016) also advocated to have more discussions on how to assess students' mastery and application of CT skills in real-life situations. In this study, we categorized CT assessments according to McMillan’s (2013) paradigms of classroom assessment. Some CT studies employed selected-response and/or constructed-response tests, e.g., Shell and Soh (2013a, 2013b) developed a paper-pencil test to assess college students' CS knowledge and CT skills; also, Chen et al. (2017a, 2017b) designed an instrument with 15 multiple-choice questions and eight open-ended questions to assess students' application of CT skills to solve daily life problems. Performance or portfolio assessment is another major assessment tool. Researchers created programming or CT activities for students to complete and then employed a grading rubric to evaluate their work products, e.g., the Fairy Assessment of Alice programming, an environment which allows users to build 3D virtual worlds (Werner, Denner, Campe, & Kawamoto, 2012); the analysis of digital portfolios designed by students to complete e-textiles projects using CT (Fields, Shaw, & Kafai, 2018; Lui et al., 2019), and the evaluation of students' Scratch projects based on a visual programming environment (Garneli & Chorianopoulos, 2018). Questionnaires and interviews have also been used. For instance, Sáez-López et al. (2016a, 2016b) used a questionnaire to examine primary school students' perceptions of computational concepts after learning visual programming language on the Scratch platform.

Researchers have examined the quality of CT assessment. For example, to examine the reliability and validity of a self-efficacy perception scale for CT skills, Gülbahar, Kert, and Kalelioğlu (2018) conducted exploratory and confirmatory factor analysis. Weintrop et al. (2016) conducted interviews to probe students’ strategy on designing video games using block-based programming language.

A systematic review of specific CT assessments may yield insights into ways of improving and designing effective assessment tools so that two goals can be achieved: (a) the scholarly community and school practitioners can remain updated in the availability of CT assessments and their characteristics. (b) Researchers can continually add to this scholarship by investigating unexamined but important topics surrounding CT assessment, the most relevant constructs in CT assessment, appropriate assessment formats to measure CT, and the necessary reliability and validity evidence.

In this study, we conducted a systematic review with the purposes to reflect on prior studies and to identify the gaps by specifically focusing on one aspect of the literature—the assessment of CT. We analyzed the current state and features of studies exploring CT assessments and suggested future directions regarding how to assess CT for various purposes. The following research questions (RQ) formed the basis of this review:

• RQ1: What are the educational contexts to which CT assessments have been applied?

• RQ2: What CT constructs are measured by the CT assessments?

• RQ3: What assessment tools are used to assess CT constructs?

• RQ4: What is the reliability and validity evidence of these CT assessments?