Elsevier

Computers in Human Behavior

Volume 29, Issue 6, November 2013, Pages 2568-2572
Computers in Human Behavior

The validity of a game-based assessment of persistence

https://doi.org/10.1016/j.chb.2013.06.033Get rights and content

Highlights

  • We created an assessment of persistence based on timing data in the game.

  • Game-based assessment of persistence was validated.

  • Game-based assessment of persistence predicted learning.

  • Findings support the implementation of a formative assessment of persistence.

Abstract

In this study, 154 students individually played a challenging physics video game for roughly 4 h. Based on time data for both solved and unsolved problems derived from log files, we created a game-based assessment of persistence that was validated against an existing measure of persistence. We found that the game-based assessment of persistence predicted learning of qualitative physics after controlling for gender, video game experience, pretest knowledge and enjoyment of the game. These findings support the implementation of a real-time formative assessment of persistence to be used to dynamically change gameplay.

Introduction

There is growing evidence of video games and simulations supporting learning (e.g., Coller and Scott, 2009, Tobias and Fletcher, 2011; for a review see Wilson et al., 2009). An additional advantage of using video games and simulations in education is the vast amount of data that can be used for assessment purposes (Dede, 2005, DiCerbo and Behrens, 2012, Quellmalz et al., 2012, Shute and Ventura, 2013). Formative assessments embedded within a video game can enable us to more accurately provide feedback and change gameplay to maximize learning according to the ability level of the player.

In this paper, we focus on a game-based assessment for persistence, a facet of conscientiousness. Over the past 20 years or so, conscientiousness has emerged as one of the most important personality traits in predicting academic performance (e.g., Poropat, 2009) as well as in various life outcomes (e.g., Roberts, Kuncel, Shiner, Caspi, & Goldberg, 2007). Persistence (i.e., industriousness in Roberts, Chernyshenko, Stark, & Goldberg, 2005; achievement in Perry, Hunter, Witt, & Harris, 2010) is a facet of conscientiousness that reflects a dispositional need complete difficult tasks (McClelland, 1961), and the desire to exhibit high standards of performance in the face of frustration (Dudley, Orvis, Lebiecki, & Cortina, 2006). Perry et al. (2010) suggest that persistence may drive the predictive validity of conscientiousness and is the facet that consistently predicts a variety of outcomes (Dudley et al., 2006, Perry et al., 2010, Roberts et al., 2005) over other facets of conscientiousness.

Persistence can play an important role in learning in a video game due to the principle of challenge (Pausch, Gold, Skelly, & Thiel, 1994). That is, challenge entails adjusting the optimal level of difficulty for a player and is consistent with the theory of the zone of proximal development (Vygotsky, 1978) which states that learning takes place right at the outer edges of one’s abilities. The principle of challenge is pervasively used in video games and has been shown to engage attention and enhance learning (Lepper and Malone, 1987, Rieber, 1996, Sweetser and Wyeth, 2005). Thus video games can require persistence due to the design of progressive difficulty. This repeated exposure to challenge can positively affect persistence requiring a willingness to work hard despite repeated failure (for a review see Eisenberger, 1992, Ventura et al., 2012). For example, Eisenberger and Leonard (1980) showed that exposure to difficult tasks can improve persistence. Participants were randomly assigned to solve impossible, hard, or easy anagrams and then take the perceptual comparison task. Then participants were asked to detect as many differences as possible between two pictures. Participants in the impossible anagram condition spent the most time on the perceptual comparison task, followed by those in the hard anagram condition, and then those in the easy anagram condition. This provides evidence that exposure to difficult tasks can affect subsequent effort. The next section introduces a video game we developed that requires persistence due to its difficulty.

Research into what’s called “folk” physics demonstrates that many adults hold erroneous views about basic physical principles that govern the motions of objects in the world, a world in which people act and behave quite successfully (Reiner, Proffit, & Salthouse, 2005). The prevalence of these systematic errors has led some investigators to propose that incorrect performance on these tasks is due to specific “naive” beliefs, rather than to a general inability to reason about mechanical systems (McCloskey & Kohl, 1983). Recognition of the problem has led to interest in the mechanisms by which physics students make the transition from folk physics to more formal physics understanding (diSessa, 1982) and to the possibility of using video games to assist in the learning process (Masson et al., 2011, White, 1994).

One way to help remove misconceptions in physics is to illustrate physics principles with physical machines (Hewitt, 2009). In physics, a machine refers to a device that is designed to either change the magnitude or the direction of a force. Teaching about simple machines (e.g., lever, pulley, and wedge) is widely used as a method to introduce physics concepts (Hewitt, 2009). Research on science education also indicates that learners’ hands-on experience with such machines (both virtually and physically) support applicable understanding of important physics concepts (Hake, 1998).

We developed a PC video game called Newton’s Playground to help middle school students understand Qualitative Physics (Ploetzner & VanLehn, 1997). Qualitative physics as a nonverbal understanding of Newton’s three laws, balance, mass, gravity, conservation of momentum, potential and kinetic energy. Newton’s Playground is a 2D sandbox video game (i.e., a game design feature where the player can create objects) that requires the player to guide a green ball to a red balloon (inspired by the game Crayon Physics Deluxe). The player can nudge the ball to the left and right (if the surface is flat) but the primary way to move the ball is by drawing/creating simple machines on the screen that “come to life” once the object is drawn. Everything obeys the basic rules of physics relating to gravity and Newton’s three laws of motion.

The 74 problems in Newton’s Playground (NP) require the player to draw/create four simple machines: inclined plane/ramps, pendulums, levers, and springboards. All solutions are drawn with colored lines using the mouse. A ramp is any line drawn that helps to guide a ball in motion. A ramp is useful when a ball must travel over a hole. A lever rotates around a fixed point usually called a fulcrum or pivot point. Levers are useful when a player wants to move the ball vertically. A swinging pendulum directs an impulse tangent to its direction of motion. The pendulum is useful when the player wants to exert a horizontal force. A springboard (or diving board) stores elastic potential energy provided by a falling weight. Springboards are useful when the player wants to move the ball vertically. Fig. 1 displays a problem in NP. In this problem the player must draw a pendulum on a pin (i.e., little black circle) to make it swing down to hit the ball (surrounded by a heavy container hanging from a rope). In the depicted solution, the player drew a pendulum that will swing down to move the ball. To succeed, the player should manipulate the mass distribution of the club and the angle from which it was dropped to accomplish just the right amount of force to get the ball to the balloon.

NP consists of 7 playgrounds (each one containing 10–11 problems) that progressively get more difficult. Each problem is designed to elicit a particular simple machine (in the game we refer to them as “agents”). The difficulty of a problem is based on a number of factors including: relative location of ball to balloon, obstacles, number of agents required to solve the problem, and novelty of the problem. NP also includes tutorial videos that show the player how to create and use the various agents. During gameplay, students have the option to watch agent tutorial videos at any time.

NP displays silver and gold trophies in the top left part of the screen which represent progress in the game. A silver trophy is obtained for any solution to a problem. Players can also receive a gold trophy if a solution is under a certain number of objects (the threshold varies by problem, but is typically <3). A player can receive one silver and one gold trophy per problem.

NP automatically uploads log files to a server for each gaming session (i.e., log activity between login and logout). The code below displays what a session log looks like for one event of a problem. An event collects data for a particular visit to a problem. A player may revisit a problem multiple times thus logging multiple events. Fig. 2 displays a snapshot of the NP session event log. As can be seen the session event log reports several features of gameplay in a problem. For example, “game_time” reports the total time spent on this particular visit to the problem. “Silver” reports if a silver trophy was achieved in this visit to the problem.

Section snippets

Theory

The aim of this study is to describe how we used the log file data to develop a game-based assessment of persistence. We focus on two pieces of data from the log files to inform our game-based assessment of persistence (GAP): unsolved times and solved times on problems. That is, longer times spent on difficult problems (whether they were solved or not) should indicate greater persistence (Eisenberger and Leonard, 1980, Ventura et al., 2012).

We predict that the GAP will be positively correlated

Sample and procedure

Our sample consisted of 154 8th and 9th grade students (72 male, 82 female). Students were paid $25 for participation and were tested in groups of around 20 students per session. The students played NP for around 4 h (split into five 45-min sessions across two weeks) in a large computer lab. Students were not allowed to talk to other students or to look at other students’ computer screens.

We administered a qualitative physics pretest, a self-report persistence questionnaire, and a video game use

Results

Reliability for the physics test was acceptable (Form A: α = .72; Form B: α = .73). Reliability was good for the PMP (α = .80) as well as the self-report persistence items (α = .83). Alpha reliability could not be computed for the GAP since players did not solve or even attempt all problems. Additionally, if two players played the same problem one may have solved it while the other may have not solved it, thus the data for the 74 problems was sparse.

Table 1 displays the means for all the measures.

Discussion

These results suggest that a valid game-based assessment of persistence can be achieved in a video game. The GAP (unsolved and silver times) was correlated with the PMP, another measure of persistence. The relation between the GAP and PMP measures increased when looking at struggling players in NP. Thus we found evidence of construct validity of the GAP. Both of these measures are grounded on the premise that longer times spent on difficult problems indicate persistence (Eisenberger and

Conclusion

This study provides evidence of the validity of a game-based assessment of persistence and continues the emerging trend showing that video games and simulations can be useful for assessment purposes (e.g., Quellmalz et al., 2012, Shute and Ventura, 2013). The present data supports the GAP and the PMP as a more valid assessment of persistence than a self-report measure of persistence. Future research should focus on training studies using video games with embedded formative assessment designs to

Acknowledgements

We would like to thank the Bill and Melinda Gates Foundation for their funding of this project. We would also like to thank Matthew Small, Yoon Jeon Kim, and Lubin Wang for their work on this project.

References (30)

  • P.G. Hewitt

    Conceptual physics

    (2009)
  • M.R. Lepper et al.

    Intrinsic motivation and instructional effectiveness in computer-based education

  • M.E.J. Masson et al.

    Video-game training and naive reasoning about object motion

    Applied Cognitive Psychology

    (2011)
  • M. McCloskey et al.

    Naive physics: The curvilinear impetus principle and its role in interactions with moving objects

    Journal of Experimental Psychology: Learning, Memory, and Cognition

    (1983)
  • Pausch, R., Gold, R., Skelly, T., & Thiel, D. (1994). What HCI designers can learn from video game designers. In:...
  • Cited by (61)

    • Performance decline in a low-stakes test at age 15 and educational attainment at age 25: Cross-country longitudinal evidence

      2021, Journal of Adolescence
      Citation Excerpt :

      Concerns over the measurement properties of self-reports have led to the development of measurements that can supplement (or even replace) self-reports (Soland et al., 2019). Alternate instruments have been employed to assess students' achievement motives (Eisenberger & Leonard, 1980; Ventura & Shute, 2013). However, administering ad hoc instruments in large-scale settings is both impractical and costly.

    • Toward Asset-based Instruction and Assessment in Artificial Intelligence in Education

      2024, International Journal of Artificial Intelligence in Education
    • Games as stealth assessments

      2023, Games as Stealth Assessments
    View all citing articles on Scopus
    View full text