Gamifying education: what is known, what is believed and what remains uncertain: a critical review

In search for empirical research papers, that is, papers based on actual observations or experiments on educational gamification, we searched the following databases: Google Scholar, ACM Digital Library, IEEE Explore and ScienceDirect using the following search terms: (gamification OR gamify OR gameful) AND (education OR learning OR training) AND (since 2014). In the cases when the OR option was not available in the provided Boolean search functionality, an equivalent search strategy was carried out through multiple searches with alternative terms. This search yielded a total of 4998 results depicted in Table 1. We have chosen the definition of (Deterding et al., 2011) for gamification (“the use of game design elements in non‐gaming contexts”) to measure each found publication for relevance. Accordingly, publications discussing full-fledged games were filtered out. Peer-reviewed empirical research papers where no findings were reported were also excluded. For example, purely descriptive papers such as (Morrison & DiSalvo, 2014), which describes the implementation of gamification within Khan Academy, were not included. At the end of this step, all papers that appeared in the review presented in (Dicheva et al., 2015) were also filtered out. The review was restricted to papers appearing in the searched databases between June 30, 2014 and December 31, 2015. The result was a list of 51 empirical research papers. In sum, in the past one and a half years, several hundred articles pertaining to gamification in education have been published however only 51 studies met our criteria and are reviewed in this article.

For completeness of the review of the research in the field, we decided this time to include also theoretical papers dealing with gamification in education. Following (Seaborn & Fels, 2015), the “theoretical papers” category includes papers that propose an explanation of the underlying nature of gamification in education and such that propose relevant pedagogies or test already existing explanatory models from other domains with respect to gamification. We also added the published literature reviews to the group of theoretical papers. The end result was a list of 11 theoretical papers appearing in the searched databases between June 30, 2014 and December 31, 2015. Thus the final number of selected papers (empirical and theoretical) amounted to 63 in total. The last column of Table 1 shows the results after filtering out irrelevant papers and removing duplicates. For comparison, the total number of papers included in the previous review covering the period January 2010–June 2014 was 34.

Following the division empirical studies vs. theoretical papers, the first part of this review covers the published empirical research on the topic, while the second part surveys briefly publications targeting theoretical aspects of educational gamification.

A literature survey typically employs a framework for structuring the evaluation of the works in the targeted area. This framework captures the potential properties of interest and enables a comparison of the surveyed works and drawing meaningful conclusions. The use of gamification in learning involves a number of aspects, including game elements, educational context, learning outcomes, learner profile and the gamified environment. Gamification is receiving attention, particularly for its potential to motivate learners. Accordingly, our objective involving evaluation of the level of understanding of the motivational impacts of gamification in educational contexts has shaped our decision of what categories of information to be included in the framework for evaluating the surveyed works. More specifically, we looked for information that can facilitate the process of identifying and analyzing the empirical evidence demonstrating the motivational effects of gamification. Motivation as a psychological process that gives behavior purpose and direction is contextual. Not only are individuals motivated in multiple ways, but also their motivation varies according to the situation or context of the task. To provide support for analyzing the contextual aspect, the information collected from the studies include the educational level, academic subject, and type of the gamified learning activity. We also included the used game elements, mechanics and dynamics since they are inherently related to the success of a gamification application. A number of motivation measures have been used in attempts to establish the effect of gamification on student motivation. In addition to appropriate measures, the verification of the validity of reported results requires availability of relevant statistical information about the studies. In order to provide support for our decision on how conclusive the reported results of a study are, we added the following categories: study sample, study duration, method of data collection, and outcome. Thus the final structure of information to be derived from the reviewed studies included the following categories: game elements, educational level, academic subject, learning activity, study sample, study duration, data collection, and outcome.

Appendix 1 presents a description of the reviewed papers structured according to this framework. Obviously, the task of representing high-dimensional data in a table format is challenging, which implies a tradeoff between completeness and clarity.

Considering the educational level, the bulk of gamification studies in the survey period were conducted at university level (44 papers), with less attention to K-12 education (7 papers). At university level, 1 study has reported results involving graduate students (Nevin et al., 2014), while at K-12 level, 3 studies have reported results involving elementary school students (Boticki, Baksa, Seow, & Looi, 2015; Simoes, Mateus, Redondo, & Vilas, 2015; Su & Cheng, 2015) , 2 studies have reported results involving middle school students (Attali & Arieli-Attali, 2015; Long & Aleven, 2014 ) and 2 studies have reported results involving high school students (Davis & Klein, 2015; Paiva, Barbosa, Batista, Pimentel, & Bittencourt, 2015). A possible explanation of this disproportion is that perhaps it is easier for college instructors to experiment with using gamification in their own courses. This might be because they are better supported technically or have necessary computer-related skills, which allow them to implement some gamification features, e.g. an electronic leaderboard. Studies involving different demographic groups however are beneficial, as we cannot necessarily generalize the results of a study conducted with one demographic group to another demographic group.

The collection of papers covers a wide range of academic subjects (32) organized in six categories (see Table 2). The category “Others” includes studies with unspecified subjects, where the gamified activities are independent of a subject and the focus is on: the platform supporting gamification (Barrio et al., 2015; Chang & Wei, 2015; Davis & Klein, 2015; Lambruschini & Pizarro, 2015; Mekler et al., 2015), the game elements used (Boticki et al., 2015; Pedro et al., 2015a), a personal learning environment (Morschheuser et al., 2014), measurements (Simoes et al., 2015) or learners’ personalities (Tu et al., 2015).

One emerging area which is not an academic subject in its own but rather referring to a set of tools offering new affordances for enhancing students’ understanding of dynamic processes and systems is interactive simulations (dynamic computer-based models which can help students observe or interact with scientific phenomena). Although gamifying the use of such simulations can help overcome the problems with insufficient motivation and engagement, there is a lack of studies evaluating the effects of gamified simulation-based learning. In this context, the work of Bonde et al. (2014), who studied the effect of combining gamification elements with simulations for improving learning effectiveness and motivation of biotech students addresses a critical gap. The results show that a gamified laboratory simulation can increase both learning outcomes and motivation levels when compared with traditional teaching. Further research is needed to examine whether these results can be extrapolated to a general tendency of the effectiveness of gamified simulations.

As shown in Table 2, the vast majority of gamification studies are dealing with Computer Science (CS) and Information Technology (IT). This fact provokes the question: Are CS and IT more suitable to gamification than the other subjects? The present studies however do not provide conclusive answer to this question. In the lack of other evidences, speculative answers can be given similar to the ones for the observed disproportion in gamifying college vs. school level activities, namely that perhaps it is easier for CS and IT instructors to experiment in their own courses. In sharp contrast, gamification experiments targeting activities related to disciplines from humanity and social sciences are extremely limited, with only one example (Holman et al., 2015) touching this subject. Another interesting observation is the low proportion of studies on gamifying STEM disciplines, excluding CS/IT and mathematics, where reinforcement of motivation is particularly beneficial: only two out of thirty two (Bonde et al., 2014) and (Su & Cheng, 2015).

Formal learning typically involves a mix of instructional activities and supporting materials, such as lectures, tutorials, assignments, projects, labs, exercises, class discussions and team work. A sizable part of the papers (16) studied gamification of courses as a whole, which implies gamifying a range of learning activities. Half of these are studies of gamified online courses (Amriani et al., 2014; Bernik et al., 2015; Jang et al., 2015; Krause et al., 2015; Leach et al., 2014; Sillaots, 2014; Utomo & Santoso, 2015), while the remaining part are regular courses typically with web-based learning support. Online learning normally requires stronger motivation, which makes it a somewhat more promising field for applying gamification. Although this presumes a higher concentration of studies on gamified online learning our findings indicate the opposite.

As illustrated in Table 3, the majority of works (36) studied the effect of gamification on general class activities (16) or a particular learning activity, such as exercises (6), collaboration/discussion forums (4), projects/labs (6) or tests (4). Another part of the papers addresses activities with indirect effect on learning, such as engaging students in more regular interactions with the learning environment (11). The category “Others” includes perception studies (Davis & Klein, 2015), augmented game mechanics studies (Pedro et al., 2015a), a specific activity (Mekler et al., 2015) or platform dependent studies (Su & Cheng, 2015).

Although 6 studies are addressing “Exercises”, still limited attention is given to gamifying activities where students can learn through experimenting and retrying without fear of negative consequences. One observation that can be drawn from this distribution is that learning activities which involve tasks that are decomposable into simpler subtasks or tasks where performance is measurable (according to an obvious rewarding scheme or skills) are better candidates for gamification.

According to (Deterding et al., 2011) gamification is the use of game design elements in non-game contexts. In turn, game design elements which are used in the creation of gamification scenarios can be divided into three categories: dynamics, mechanics and components (Werbach & Hunter, 2012).¹ Dynamics represents the highest conceptual level in a gamified system. It includes constraints, emotions, narrative, progression and relationships. Mechanics are a set of rules that dictate the outcome of interactions within the system, while dynamics are users’ responses to collections of those mechanics. The game mechanics refer to the elements that move the action forward. They include challenges, chance, competition, cooperation, feedback, resource acquisition, rewards. Components are at the basic level of the gamification process and encompass the specific instances of mechanics and dynamics. They include: achievements, avatars, badges, collections, content unlocking, gifting, leaderboards, levels, points, virtual goods, etc. For instance, points (components) provide rewards (mechanics) and create a sense of progression (dynamics). However, we note that the gamification terminology is still unsettled and various variations of the introduced above terms exist. When there is no danger of confusion, we will use the terms mechanics and dynamics to refer also to their specific instances, that is, components. Also, for consistency with our previous studies (Dicheva et al. 2015), we will use the term game elements to refer to game components.

Most of the educational gamification studies and applications are driven by the presumption that gamification in education consists chiefly of incorporating a suitable combination of game elements within learning activities. However, our review shows that the empirical studies on understanding what kind of game elements under what circumstances can drive desired behavior are not quite systematic. In the reviewed collection, 11 papers report studies of the effect of a single game element, 8 papers study gamified systems using 2 game elements, 16 papers study gamified systems with 3 game elements, while the remaining 16 papers report results of gamifying systems by incorporating more than three elements (see Table 4).

In all reviewed works with the exception of (Tu et al., 2015), which investigates the relation between gamers’ personality and their game dynamics preferences, the gamification studies focus on the use of game elements (i.e. game components in terms of (Werbach & Hunter, 2012)). Typically, no justification is given for the selection of particular game elements. There is a need of more studies that can improve our understanding of how individual game elements are linked to behavioral and motivational outcomes and how they function in a given educational context. Without understanding the effect of individual game elements, it is difficult to identify their contribution in studies that mix several game elements together.

The majority of gamification studies feature a subset of the following game elements: points, badges, levels, leaderboards and progress bars. This is in line with the finding of other authors, e.g. (Nicholson, 2015) that the combination of points, badges and leaderboards (sometimes referred to as PBL) is the most used one (see Table 5).

In the absence of other justification for the overuse of points, badges and leaderboards, one possible explanation is that they somewhat parallel the traditional classroom assessment model and are also easiest to implement. This combination in its trivial form can be applied to almost any context, even if there isn’t a good reason to do so. Gamification with “deeper game elements” (Enders & Kapp, 2013) incorporating game design principles involving game mechanics and dynamics such as challenges, choice, low risk failure, role-play or narrative are still scarce. Only one work (Tu et al., 2015) among the reviewed studies addresses game dynamics explicitly. Studies utilizing to some extent “deeper game elements” are demonstrated in (Bonde et al., 2014; Boskic & Hu, 2015; Holman et al., 2015; Krause et al., 2015; Pettit et al., 2015). We believe that in addition to reward and feedback mechanisms, gamified systems should provide safe places where learners can gain experience without being judged or punished for failure, drawing upon approaches similar to the online learning environments proposed by (Hakulinen et al., 2015) and (Lehtonen et al., 2015), where students can improve their algorithmic skills by practicing with interactive exercises (Dichev et al. 2014).

Three questions related to the use of combinations of game elements remain open: “Do more game elements produce better results than less?”, “Is the task of identifying the right combination of game elements with respect to a given context and user group practically feasible?” and “How to balance points and rewards with play and intrinsic engagement?”. For answering these questions and for advancing the understanding of how to build successful gamified educational systems, there is a need for testing systems that support examining the effect of game elements and experimentally validating it. In particular, it implies the need of gamification platforms that support easy configuration of gamified learning prototypes with specific characteristics leveraging different game features and principles.

The available evidences indicate that in a learning context gamification is more than mapping game elements on to existing learning content. It should offer stronger ways to motivate students, rather than be simply a stream of extrinsic motivators.

The reviewed papers expand the scope of the empirical research on educational gamification, as compared to (Dicheva et al. 2015). Although the majority of empirical works still examine the impact of the gamification on students’ engagement, performance, participation or retention, they are widening and deepening the focus of their studies. A growing body of papers is exploring a range of learning and behavioral outcomes including:

Under the perceptual outcome category, we have included also some works that initiate a new line of studies - the impact of gamification on different demographic groups. For example, (Pedro et al., 2015b) reported that the game mechanics implemented in a virtual learning environment did not have any effect on motivation and performance of the female students. This findings are in line with the conclusions reported by (Koivisto & Hamari, 2014), who have shown in a more general context that women experience a greater effect when the gamification contains social aspects and men - when there is a sort of competition. (Christy & Fox, 2014), on the other hand, concluded that the use of leaderboards within educational settings may act to create stereotype threat (a belief that one may be evaluated based on a negative stereotype). The results of the study found that women in the female-dominated leaderboard condition demonstrated stronger academic identification than those in the control and male-dominated leaderboard conditions. These results suggest that the use of leaderboards in academic environments can, in some circumstances, affect academic performance of different demographics differently.

The motivational outcome category concerns concepts derived from motivational principles of games such as explicit goals, rules, a feedback system, and voluntary participation (McGonigal, 2011). Motivation is demonstrated by an individual’s choice to engage in an activity and the intensity of effort or persistence in that activity. Since video games are explicitly designed for entertainment, they can produce states of desirable experience and motivate users to remain engaged in an activity with unparalleled intensity and duration. Therefore, game design was adopted as an approach for making non-game activities more enjoyable and motivating. While gamification strives at its core to increase motivation, yet motivation is not a unitary phenomenon - different people may have different types and amounts of motivation, which can be shaped by the activity they are undertaking (Gooch et al., 2016). Additionally, success in one educational context does not guarantee that the same mechanism will be motivationally successful in another educational context.

An important distinction in the motivation research is that between intrinsic and extrinsic motivation (Ryan & Deci, 2000). While extrinsic motivation relies on incentives or expected consequences of an action, intrinsic motivation stems from fulfilling the action itself. According to the Self Determination Theory (Ryan & Deci, 2000), humans seek out activities to satisfy intrinsic motivational needs, such as competence, autonomy, or relatedness. More specifically, (Ryan et al., 2006) argue that the intrinsic appeal of games is due to their ability to satisfy the basic psychological needs for competence, autonomy, and relatedness. While self-determination theory provides a good theoretical starting point for studying the motivational dynamics of ‘gamified’ educational activities, further research is needed to bridge motivation to a more granular level of game elements and learners’ personalities. Although the connection between motivation and gamification design is demonstrated by a number of the reviewed studies, they do not add persuasive evidence confirming the effect of gamification as a motivational tool. The papers claiming to examine the motivational effects of gamification often report effects on learning outcomes instead on motivation.

The reviewed collection of empirical studies on gamifying education is very diverse with respect to the focus of the studies and the reported outcomes. This makes it difficult to find categorization that organizes the reviewed works in logical categories, captures the diversity and puts at the same time every work in a separate category. We selected a categorization with a focus on the effects of gamification on learners. It includes four categories: affective (A), behavioral (B), cognitive (C), and others. The intention with this grouping was to use it as an organizational structure for connecting outcomes with game elements and gamified activities. As under this categorization many outcomes fall into two categories, we extended it with behavioral and cognitive (B + C), affective and cognitive (A + C), and affective and behavioral (A + B) groupings. Table 6 presents the studies falling into a single category, organized in three sections: behavioral, affective, and cognitive, and connecting their outcomes with the corresponding game elements and gamified activities. Table 7 presents the studies falling into two categories, organized in the same way.

The two tables provide a more compact view, capturing the links between three key categorizing variables: game elements, gamified activities and reported outcomes. The more focused information extracted in the tables explicates data relevant to the questions guiding the study. Although the empirical work on applying gamification in educational contexts continues to grow, there is not sufficient evidence indicating noticeable progress based on collating and synthesizing the previous experiences. While the range of gamifying strategies is expanding, they are scattered across many different educational contexts and the aggregated information cannot confirm any emerging systematic approach yet. As it can be seen from the tables, the empirical research on gamified learning is quite fragmented. It covers studies on different configurations of game elements, used to gamify different activities and resulting in different outcomes, without any identifiable pattern of distribution. For example, the points-badges-leaderboard configuration is dominating, with 6 works studying its effect. However, the activities gamified with this configuration vary widely: project activities, course participation, online Java exercises, homework in high school LMS assessment and overall course activities. Within the category “Gamified activity” dominating is “Overall class activities” but again the configurations of game elements used to gamify it are very different: badges, leveling, autonomy, leaderboard, grade predictor; stamps, tokens, leaderboard; points, scoreboard, goals, avatar, feedback, levels, luck, competition; points, badges, leaderboard; points, leaderboard; badges, leaderboard, virtual coins, pseudonyms. The dearth of studies that build on the previous ones or parallel their efforts on exploring particular aspects of the effect of gamification on engagement and learning suggests a piecemeal approach. In the current studies that mix together points, badges, leaderboard, progress, status, etc. without a discernible systematic experimental approach, it is difficult to identify which game elements or configurations are most effective in promoting engagement and supporting learning for given activity and group of learners.

We noticed that in addition to the heterogeneous nature of the empirical research, the stated goals of the studies were not always in line with the reported outcomes. To provide an additional dimension for organizing and examining the links between the corresponding categorizing variables we further grouped the studies according to their stated goal (see Appendix 2, which lists the reviewed studies along with their goals). The two top categories for grouping the studies based on the study goals are: learner-centric and platform-centric (see Table 8). The bulk of works which expands and differentiates the earlier research on the effect of gamification on learners (e.g. (Dicheva et al., 2015)) falls in the first category (44 papers). This category includes 4 subcategories grouping further the studies as follows:

These four groups cover a wide variety of goals. Group A includes studies of the effectiveness of gamification in the classroom longitudinally (Hanus & Fox, 2015); the impact of gamification on retention and learning success (Jang et al., 2015; Krause et al., 2015), on participation and quality of online discussions (Smith et al., 2014), on reducing undesirable behaviors and increasing performance in virtual learning environments (Pedro et al., 2015b) and in personal learning environments (Lehtonen et al., 2015; Morschheuser et al., 2014); the effect of badges on student behavior (Hakulinen et al., 2015) and how they predict the student exam success (Boticki et al., 2015); the causal effect of gamifying a course project with leaderboards (Landers & Landers, 2015); the learning effectiveness of a gamified simulation (Bonde et al., 2014) and the effect of transforming a traditional course into a role-playing game (Boskic & Hu, 2015).

A progress has been made within the learner-centric category with explorations of psychological effects of gamification which can be summarized by the question: How students with different personalities, dispositions and learning styles are influenced by game elements? While in our first review the question shared between most of the papers was “Is gamification effective?”, now it appears in a more extended version, in combination with the questions “for what?” or “to whom?”.

Group B includes papers identifying learner types based on how students experience gamified courses (Barata et al., 2014) and how different learners perceive playfulness (Codish & Ravid, 2014), on the variation in motivation between learners with different gamification typologies (Herbert et al., 2014), on exploring whether points, leaderboards, and levels increase performance, competence, need satisfaction, and intrinsic motivation (Mekler et al., 2015), on involving Asian students in gamified course activities (Hew et al., 2016) and on the predictive effect of gaming personality on their game dynamic preferences (Tu et al., 2015). Even though the amount of papers addressing the question “to whom” is still limited, an emerging shared message particularly relevant to instructional designers recognizes that what one learner values, another may not, what one learner believes is achievable, another may not. Understanding differences in learners’ drivers, what they value and what they dislike is important to the design of reward, progress, and feedback systems with potential to achieve desired outcomes for the intended groups of learners.

Group C includes papers on students’ perceptions of simple game elements such as badges (Davis & Klein, 2015) or combination of points and badges (Paiva et al., 2015). It also includes studies on how students perceive a game-like course (Sillaots, 2015) and on profiling learners based on their gamification preferences (Knutas et al., 2014).

Another emerging topic in this category groups the works on measuring the impact of gamification (Group D). This group includes papers on the impact of gamification on students’ engagement and how to measure that impact (Simoes et al., 2015), papers on the effectiveness of gamification behavior patterns as a measure of playfulness (Codish & Ravid, 2015), and how predictive measurements can help students plan their pathways in gamified courses (Holman et al., 2015). While gamification is promoted as a motivational instrument, studies measuring its motivational effects are still limited.

In the second category we have placed 7 articles, which study the effect of incorporating selected game elements or game principles into specific learning platforms or experiment with conventional game elements by assigning them new roles. This category includes studies of employing gamification in audience response systems (Barrio et al., 2015; Pettit et al., 2015), in mobile learning systems (Su & Cheng, 2015), in Learning Management Systems (Lambruschini & Pizarro, 2015) and in MOOCs (Chang & Wei, 2015). Two papers explore creating badges as a tool for measuring students’ interest (Tvarozek & Brza, 2014) and the effect of collaborative badge creation on engagement and motivation (Pedro et al., 2015a). The papers listed in the platform-centric category do not cover all gamified platforms proposed in the reviewed papers. When according to our judgment the focus of a paper was on behavioral effects, as for example in (Krause et al., 2015), that paper was included in the first category. The availability of successful gamified platforms will help widen the scope of gamified educational activities and create a ground for broadening experimental studies towards developing evidence based practices.

One of the evolving goals of this review was to take a closer look at the supporting evidence for the ‘positive’ or ‘negative’ results of the empirical studies as reported by their authors. This was provoked by the fact that some of the papers studying the effect of gamification on learners reported a mix of positive and negative results, other were inconclusive, and yet other expressed a degree of caution, while the strength of the evidence backing the positive and negative results were varying significantly.

A common pattern observed in most studies is to design and develop a particular gamified course/activity/environment, test it in a pilot and assess users’ approvals and gains in performance. The reported outcome often concludes that the gamification produced the pursued learning gains and that the users appreciated the added gamification features. Irrespective of the goals of the studies, the works on gamifying education should be subject to the same level of skepticism and scrutiny that is applied to any other areas of empirical research. In order to improve our understanding and to offer a more realistic picture of the nature of the effects of using gamification in education, consistent with the presented evidence, we undertook a more in-depth examination of the reviewed papers with a focus on both the reported outcomes and how they have been obtained. The primary aim of this effort was twofold: (i) to provide a critical review, questioning the validity of some reported outcomes, and (ii) to offer a picture that avoids the harmful effects of an one-sided viewpoint.

Our decision on the validity of the gamification studies was guided by the following factors: the sample size, the number of study groups, the length of the study, how the data was collected, how the variables were controlled, how and by what statistical procedures the data was analyzed, how well the conclusions are supported by the data, and does the study give enough information to convince the reader in the correctness of the evaluation conclusions. The examination of the selected papers indicated that the empirical studies tended to use surveys and quasi-experimental designs, while the randomized controlled trials were less common. According to the nature of the empirical study, the papers were partitioned into two major categories: ABC studies, which target Affective/Behavioral/Cognitive outcomes, and non-ABC studies. The ABC studies were further partitioned into three subcategories: positive, negative and inconclusive, based on the reliability of the evidence for the reported ABC outcome. The outcomes were marked as “positive” if valid evidence confirms the claim and marked as “negative” if the evidence confirms its negation. The studies were marked as “inconclusive” if the presented evidence was judged as insufficient based on inadequacies, such as small sample sizes, lack of comparison groups, use of purely descriptive statistics, short experiment timeframes, and unreliable statistical evidence. For example, reported positive effects of gamification based on a two-week study could be attributed to the ‘novelty effect’ of the used tool or approach rather than to the added gamification features. In the inconclusive category we also included papers studying gamification in combination with some other factors, which make uncertain whether the observed effects can be attributed to the gamification or to the other variables, as well as papers where no positive effect was found but negative effect was not discernible either.

The classification of papers in accordance with our judgment of the degree of validity of the reported results is presented in Table 9 and the proportions of the resulting grouping of the ABC papers in Fig. 1.

The paper grouping, based on the strengths of the presented evidence, reveals that the high expectations for positive outcomes from gamified learning are not confirmed by the results of the reviewed empirical studies (see Fig. 1).

The examination of the papers shows that from the 41 ABC empirical studies only 15 present conclusive evidences for the reported outcomes. In those 15 papers, the findings related to the benefits of gamification are mixed: 12 studies present evidence for positive effects of gamification in educational settings, while 3 present evidence for negative effects. A surprising fact is that the vast majority of the empirical works (25 studies) report inconclusive outcomes, which means that there is no basis for confidence in the reported results. Such outcomes obscure the level of progress in the area of educational gamification. Table 10 and Table 11 below are obtained from Table 6 and Table 7, correspondingly, by eliminating the studies marked as inconclusive. With this relatively small number of (15) papers and a diverse specter of game elements and activities, the presented outcomes are insufficient to draw definitive conclusions on the effectiveness of gamification on students’ engagement, learning or participation. This judgement can be interpreted as an answer to the first guiding question about the existing empirical evidence for the impact of gamification on motivational processes and effectiveness of learning. Currently, there is a dearth of quality empirical evidence to support general claims of the impact of gamification on student’ learning and motivation. Whilst 12 studies report encouraging outcomes, they cover a range of specific combinations of game elements, specific activities and outcomes and thus do not support practical generalization. It would be short-sighted to assume that gamified implementations with the same configurations of game elements would function similarly across different educational contexts. For example, (Hakulinen et al., 2015) present convincing evidence that points, badges and leaderboard incorporated in online Java exercises increases the use of an open learning environment. However, with the current understanding of the motivational mechanisms afforded by gamification, we cannot generalize this claim to other activities, game element combinations or academic subjects. In general, studies reporting positive results from using a specific combination of game elements do not promote the understanding of the causal effect of the combination, as it is unclear whether the combination or a particular element led to the positive outcome (e.g. Bonde et al., 2014; Jang et al., 2015). Negative results such as those of Hanus and Fox (2015), who reported that badges, leaderboard, virtual coins, and pseudonyms incorporated in a communication course can have a detrimental effect on students’ motivation, satisfaction, and empowerment, help understand the limits of gamification. Again, the results obtained from such studies should be interpreted in a restricted manner, for the specific combinations of game elements, gamified activity, academic subject, and age group. The piecemeal approach observed in the reviewed studies slows down the advancement in the understanding of the effect of incorporating game elements in learning activities. From the 14 studies listed in the two tables, with 14 different combinations of game elements and 15 different gamified activities, it is difficult to derive useful information on how to gamify a new (different) activity with predictable outcomes. For example, two papers (Hakulinen et al., 2015) and (Landers & Landers, 2015) report positive outcomes for using single game elements, but one is for badges and the other one for leaderboards. On the other hand, two of the studies reporting negative results deal with Mathematics (Attali & Arieli-Attali, 2015; Long & Aleven, 2014). But, in these two cases, the game elements, the learning activities, the student level and the gender vary. In addition, the mix of badges, levels, leaderboards, progress, feedback, status and avatars used in the conclusive studies makes it hard to know which of these elements actually worked. Furthermore, the fundamental differences in the studied educational contexts hamper the transfer of experimented practices from one learning situation to another. All this suggests a need for a more systematic program of experimental studies.

We note that our judgment in studying inconclusiveness can be viewed as rather subjective. Therefore, Table 12 presents the papers judged as “inconclusive” along with a short explanation for placing them in this group. In several cases our judgment simply conveys the paper’s conclusion where the authors themselves acknowledge that the results of the study should be interpreted with caution.

While it seems apparent that gamification has the potential to create enhanced learning environments, there is still insufficient evidence that it (1) produces reliable, valid and long-lasting educational outcomes, or (2) does so better than traditional educational models. There is still insufficient empirical work that investigates the educational potential of gamification in a rigorous manner. Increasing the number of studies that use randomized controlled trials or quasi-experimental designs will increase the scientific robustness. The continued (and coordinated) collection of evidence, that is, data that substantiate the successes and failures of gamification, remains crucial for building an empirical knowledge base and consolidating best practices, extracting guidelines and eventually developing predictive theories. It is necessary to strengthen the methodical base of gamified learning and systematically enlarge the body of evidence that explains what factors and conditions produce desirable outcomes. The empirical research should thereby not just be fixated on the pros of gamified learning, but also be open to the cons and the conditions when gamification for learning should be avoided (Linehan et al., 2011; Westera, 2015).

Indirectly related to the conclusiveness of the reported results are the measurements used. A significant number of the studies (15) are using performance as a measure of the effect that gamification has on the studied activities. This is understandable for several reasons. First, the driving criterion for adopting any technology in education is whether and how much it can improve learning. Second, one can argue that high learner performance provides evidence of learners motivation since performance has been shown to correlate with learner’s motivation. However, such an approach is imperfect. Performance is an indirect measure of motivation that is influenced by many non-motivational factors such as ability, prior knowledge, and quality of instruction, while motivation is the actual driving force which makes individuals want to do something and help them continue doing it. Therefore, it is beneficial to understand the motivational triggers that engage learners. This suggests a need of studies that utilize more reliable measures of motivation and characterize better how gamification influences learner motivation and consequently how it improves learner engagement and outcomes. Motivation is associated with a number of learning related concepts such as engagement, effort, goals, focus of attention, self-efficacy, confidence, achievement, interest, etc. Improving our understanding of motivational aspects of gamification will enable us to predict its effect on the related concepts. In addition, it will help improve the gamification design, in particular, how to design an appropriate gamified experience that strengthens the motivation of a given population of learners and leads to desirable learning outcomes.