Gameful Learning for a More Sustainable World

In their set of goals for sustainable development, the UN listed targets for different areas of human and environmental wellbeing, one of which concerns waste, sustainable consumption and production (United Nations 2020). Acknowledging the insufficiency of the status quo in terms of waste management, the EU created a plan to raise EU-wide recycling to 55% and decrease landfill use to 10% by 2025 (European Parliament 2018). However, recent studies have shown that global progress is slow, partly due to a lack of appropriate legislation, insufficient financial resources, poor infrastructure, poor environmental attitudes and social norms and a lack of knowledge about what goes into which bin (Schultz et al. 1995; Thomas and Sharp 2013; Filho et al. 2016; Luo et al. 2018). A contributing factor is that many recycling and waste sorting facilities are as yet unable to reach maximum efficiency without pre-sorting measures (Bucciol et al. 2015; Hawlitschek 2020). Countries like Germany, Austria and Switzerland have tackled this issue by making domestic pre-sorting a citizen’s responsibility. However, incentivizing citizens to correctly and consistently dispose of their household waste continues to be a challenge for society, as it is a task requiring individuals to perform for the benefit of society often without rewards for compliance (Abdel-Shafy and Mansour 2018). Furthermore, successful compliance requires citizens to first gain the fundamental knowledge to fulfil the required task. Yet, municipal waste sorting authorities often fail in their education attempts, partly because of outdated measures of communication and information like analogue, paper-based flyers (Luo et al. 2018). Such materials are insufficient for knowledge transmission as they lack incentives to engage mentally, particularly given the amount and depth of information that people need to retain. Of the hundreds of potential waste items, more than 200 are listed on many websites for German waste management organizations as being fundamental to sufficient municipal waste sorting (e.g., Berlin and Hamburg)¹ While citizens do not have to know each item by heart, they need to understand the underlying principles that link different types of waste to their respective bin. To engrain the knowledge in the long term, such extensive amounts of information require adequate training measures.

As stated in an interview on “The Future of Waste Management,” Information Systems (IS) can teach citizens where exactly to dispose of different types of waste (Hawlitschek 2020). Multi-disciplinary research has shown that games in particular are successful educational tools and supplements (Fileni 1988; Van Eck et al. 2017). By applying gameful design to a real-life context, education can be effectively manipulated, whether as fully conceptualized games or as strategically implemented gamification affordances (Barata et al. 2013; Landers and Landers 2014). In their meta-analysis on digital games and learning, Clark et al. (2016) found significant correlations between quality of design and learning outcomes, highlighting the value of deliberation on specific design decisions.

We created a waste sorting training game based on best practices of game design as well as learning theories with the goal of addressing the prevalent lack of waste sorting knowledge. The game’s release was in 2015 and, as of April 2021, it was downloaded over 31,684 times on the Apple, Microsoft and Windows mobile app stores. As stated by Bellotti et al. (2013), a serious game’s purpose is twofold: to be fun and entertaining as well as educational. Thus, we must assess both aspects. While the field data allowed us to ascertain the game’s success in matters of game fun and engagement with a certain degree of external validity, we could not reliably infer the game’s efficacy in terms of the intended learning outcome. As this is the game’s primary aim, we prepared a lab experiment to measure the game’s learning outcome under the following research question: Does gameful design afford learning about correct sorting of waste items into their target bin?

Gamification/gameful design²) is a praxis that consists of designing suitable “service bundles” (Blohm and Leimeister 2013) by adding game design elements to the respective core offer – or core gameplay when the gamified product is a game in itself. The core interaction – core gameplay – of our game is based on a combination of sorting and feedback, the latter being particularly beneficial for knowledge transfer and player engagement (Sicart 2008; Bellotti et al. 2013). However, during the development of the first prototype, our user tests found that the core gameplay by itself did not engage players long enough to benefit from a long-term learning effect. We decided to add optional design elements that would offer players more choices on how to engage with the learning content. We based this decision on motivational theories that highlight autonomy as a fundamental factor of intrinsic motivation (Ryan and Deci 2000). We first chose a repetition element that would allow players to repeat a level – or wave – without penalty. The overall learning benefits of repetition are well-documented across different learning domains (Bygate 1996; Ahmadian 2012). However, as its inherent repetitiveness could interfere with the game fun, we wanted to gain insights into the potential detriments and benefits of including such a design element. We then added an index element, where waste items can be looked up penalty-free during the core gameplay (look-up element). This was inspired by testers frequently asking why certain items were assigned to bins other than expected. We conducted a literature review to find theoretical leads on the expected learning outcome of such an index-based design element. Lacking a related foundational theory, we analyzed research on related contexts: instructional explanations, dictionaries and help tools (Miller and Gildea 1987; Ryan and Shin 2011) finding mixed expectable outcomes.

The experiment consisted of five treatment groups to reflect both research questions. The first four learned to correctly sort waste items by playing the game. They differed with respect to whether participants played only the core gameplay, one or both design elements (repetition, look-up or combined). The fifth group completed the training with common, paper-based teaching materials on waste sorting. As we wanted to ensure long-term retention – long-term memory storage of the learned content – we measured learning outcomes 10 to 12 days after the participants had been trained. Also, while the training was conducted with a game, the learning outcome was supposed to be translated into real life. To test if participants successfully managed this knowledge transfer, we measured the learning outcome in three different ways: first, by testing knowledge retention within the training medium itself in a slightly altered version of the core gameplay designed to test each training item exactly once. Second, we measured knowledge transfer in an abstracted setting via a multiple-choice test featuring only the names (written words) of the trained items. Third, we measured knowledge transfer to real life through a sorting task with real-world waste items.

Our results showed that the treatment trained with the game significantly benefited with regard to learning outcomes of waste sorting knowledge compared to the treatment group given the non-game materials. This is especially remarkable, as, in contrast with other studies (Größler et al. 2000; Luo et al. 2018), we demonstrated that knowledge transfer to real life can be successfully achieved with a gameful application. We further found that the combination of the repetition and look-up game design elements showed significantly higher learning outcomes within the original content domain as well as the reduced setting. Interestingly, this combinatory effect was lost in transfer to real life.

We contribute to the literature and practice in several ways. While growing in numbers, assessments of the effects of specific game design elements are still rare (Bellotti et al. 2013; Kim and Shute 2015). This makes it difficult for both researchers and practitioners to derive informed design decisions from research. We identified a research gap with regard to expectable effects and outcomes of an optional look-up element and found that its implementation contributed to the learning outcome, especially when combined with a repetition-based design element. We also tested learning outcome in terms of long-term retention as well as knowledge transfer to ensure that an actual learning outcome was achieved. While our game successfully achieved the transfer of content to long-term memory, the differences of outcomes between the three different measures highlighted the importance of testing and ensuring successful knowledge transfer in multimedia contexts such as ours.

By constructing and testing a serious game on teaching correct waste sorting and detailing design rationales for future reproduction, we contributed to the ongoing effort of enhancing sustainable IS (see e.g., Elliot and Webster (2017) and Stanitsas et al. (2019)). Our results showed that successful learning outcomes can be achieved through meticulous gameful design even in less intrinsically motivated and attractive domains such as waste management and even outside a socially mediated learning context.

We based the theoretical foundations of this work on learning theories, particularly on models introduced by instructional design and the didactic method (Wittwer and Renkl 2008; Nitsch et al. 2016). Most learning theories have been developed in and for contexts where social interaction is interwoven into the learning process (e.g., Chi et al. 2001). However, multimedia learning is often designed to work outside of social interaction contexts. We thus built on learning theories and strategies that have proven effective outside of a socially embedded learning context, namely elaborative encoding (for Hypothesis 1) and repetition (for Hypothesis 2). While these strategies afford an empirical learning process, rationalization (the process of sense-making, or understanding “why”) is equally essential for providing meaning and context to learning matter (Wittwer and Renkl 2008). To offer explanations without overwhelming players, we implemented an optional, dictionary-inspired look-up element as a complementary game design element and elaborated on its supposed learning outcome in the development of Hypothesis 3.

Depending on context, the learning outcome can be measured with regard to different facets. In terms of memorization, the learning outcome often differs when tested immediately after the training phase rather than during a post-training transfer phase (Keith and Frese 2008). By measuring the learning outcome in terms of long-term retention, we wanted to ensure that the content was memorized in the long term to achieve successful change in real-life waste sorting behavior.

Furthermore, in contexts like ours, where the training medium differed from the application context, it was particularly important to assess the learning outcome with regard to knowledge transfer (Barnett and Ceci 2002). In language transfer theory, knowledge transfer happens within the context of translation, where words and meanings are connected between two languages, typically native and second (Mahmood and Murad 2018). In the didactics of mathematics, this transfer is referred to as “conversion,” or the mental merging of different representations – such as graphs and functions – of the same mathematical concept (Dreher et al. 2014). Expanding on the concept of conversion, Nitsch et al. (2016) differentiate two phases of the transfer process: identification (comparison of the incorporated information and identification of similarity with existing schema) and construction (transferability of the incorporated information into new situations). Differentiating the two is important because construction measures whether the content has been understood deeply enough for reapplication in new contexts. Therefore, as later explained in the section on the operationalization of our outcome variable, we measured knowledge transfer with different measurements that capture both identification and construction.

The first stage of the experiment was completed by 266 participants. Thirty-one participants did not complete all three stages of the experiment (17 participants did not start or finish Phase 2 and 14 more did not show up to Phase 3 in the lab). Of the 235 remaining participants, we had to exclude 14 further datasets because of transmission errors (e.g., the game data of the second or third stage of the experiment was missing) and one for failing a crucial control question. Finally, of the remaining 220 data sets, there was a minor data transmission error for 23 participants: not all single item sorts for the in-game performance had been transmitted completely. We decided to exclude the datasets where more than 30% of the item sorts were missing (five out of these 23). This decision was backed by a Kruskall-Wallis test that indicated that the performance of the 18 participants with more than 70% but less than 100% correctly transferred item sorts did not differ significantly from the participants with complete sets of item sorts. We thus decided to include them, leaving us with a total of 215 complete datasets. The average age of the participants was 22.72 years old (one person reported the age of 3, which we set as a missing value because this was either a typo or intentionally misreported), and the gender distribution was 66.05% of participants identifying as male vs 33.49% as female vs one person (0.47%) indicating “other.” Table 3 shows the descriptive statistics of the dependent measures for all treatments. For example, in the treatment with non-game materials, participants correctly sorted on average 70.8% of the items in the in-game performance measure, 59% in the multiple-choice test and 70.3% in the real-life sorting task. The pattern of having the lowest performance when measuring with the multiple-choice test compared to the other two learning outcome measures is stable over all treatments. The largest value of 78.8% was reached in the combined group for in-game performance. For more details on the descriptive statistics for both the dependent measures as well as the control variables, please see Tables 10 and 11 in the appendix.

For all statistical tests, we computed ordinary least square (OLS) regressions with the three continuous performance measures ranging between 0 (0% correctly sorted) and 1 (100% sorted correctly) as dependent variables. All our hypotheses were directed and therefore a test was significant if p of the two-tailed tests in the presented tables of the statistical tests was below 10%. Robust standard errors were used in all regressions to account for heteroscedasticity, based on the Breusch–Pagan test (Cohen et al. 2014). Furthermore, we bootstrapped the results with a sample size of 5,000 to account for non-normality of residuals (Tibshirani and Efron 1993; Pek et al. 2018).

To compute Hypothesis 1, we had to pool the treatments core gameplay, repeat element, look-up element and combined group into one group because Hypothesis 1 compared the games’ performance with the non-game materials. In contrast to the other two hypotheses, it did not focus on the effect of specific game design elements and their related individual treatments. We thus computed a binary variable “Game” that took the value 1 for all observations trained through the game (the pooled group) and the value 0 for the observations in the non-game material treatment. This binary variable was our only independent variable in this main analysis for Hypothesis 1. Table 4 shows the results of the three regressions of this binary variable on each of the three learning outcome measures. We found significant effects on all measures, which supported Hypothesis 1. When tested with the in-game performance measure, the game treatments were estimated to correctly sort 4.1% more items than non-game treatments. For the multiple-choice test, the effects were even larger: the game treatments were estimated to correctly sort 8.4% more items than non-game treatments. Finally, for the real-life sorting measure, the estimate was 6.9%. To sum up, we could fully support Hypothesis 1 for all three performance measures. The effect for in-game performance was surprisingly the weakest, although this was the measurement for which the medium (the digital game) of training and testing was the same.

In contrast to the analysis of Hypothesis 1, Hypotheses 2 and 3 focused on the effect of the examined design elements. Thus, we did not pool all game treatments but rather compared all five treatments with each other. We coded each treatment with a binary variable that took the value 1 if the observation belonged to the respective treatment. The reference category was the non-game material treatment which meant that all coefficients must be compared to the performance in the non-game material treatment.

Table 5 illustrates the results for Hypotheses 2 and 3. When comparing the in-game performance of the treatments with the non-game material treatment, we found a significantly increased learning outcome for the look-up element treatment (estimated increase of 4% of correct item sorts) and the combined one (8%). An additional Wald-test showed that the effect of the combined treatment was larger than that of the look-up treatment (p = 0.04). However, the effect for the look-up element was not significantly larger than for the repeat element treatment (again tested with Wald test, p = 0.56). Thus, the look-up treatment performed better than the non-game material treatment but not better than the treatment with only repetition. In sum, for the in-game performance measure, Hypothesis 3 was fully supported: we found better performance for both the groups that only had the look-up element by itself or the look-up element combined with the repetition element. Hypothesis 2 was only partially supported: we did not find a stronger performance when only playing with the repetition element. Hypothesis 2 was only supported if repetition was combined with the look-up element. For the multiple-choice test, we found even stronger results and could fully support Hypotheses 2 and 3: all four treatments trained through the game performed better in the multiple-choice test than the treatment trained with the non-game materials. The largest effect was measured for the combined treatment, where on average, participants sorted 11.9% more items correctly than the participants in the control treatment without game materials. For the real-life sorting task, we interestingly found weaker effects for the combined treatment. In detail, we found that only those treatments that had either one design element or neither of those two elements (the core game), performed significantly better than the treatment that did not play the game. Yet, the coefficients also showed that the effects for all four game treatments were rather similar, ranging between 6.3% for the combined treatment to 7.3% for the repeat element treatment. Thus, when conducting further Wald-tests comparing the coefficients of the game treatments with one another, one cannot claim that one game group performed better than another (all p > 0.8). Thus, all in all, we could support Hypotheses 2 and 3 and found that all game treatments did comparably well.

For game or gamification designers, it is interesting to compare the effects of game design elements not only to the non-game material group, but also to the core gameplay group to gain a better understanding of which design elements to include in their gameful applications. Therefore, we want to further focus in detail on the comparison of the different game treatments to the core gameplay group in Table 6.

We found that with the in-game performance measure and the multiple-choice test, the combined treatment achieved a significantly higher learning outcome than the treatment with only the core gameplay available during training, with an increase of 6.1% correctly sorted items with the in-game performance measure and 6.4% with the multiple-choice test. When comparing the groups within the real-life sorting measure, a significantly different learning outcome cannot be discerned. This is a result already highlighted in the analysis above: the game-treatments performed comparably well in the real-life sorting task. Thus, for real-life performance, the overall effect of the game itself was much stronger than that of adding the single design elements to the core gameplay.

To further assess the robustness of our results, we also computed robust OLS regressions with these control variables: age, gender, how long they lived in Germany (“Living in Germany”), how long they lived in the city the experiment was conducted in (“Living in XX city”), their gaming motivation, their general waste sorting motivation and the SUS (for details, see Tables 12, 13, 14 in the appendix). The results were robust regarding the inclusion of these control variables. However, there was one slight change: for Hypothesis 3, the effect on the repeat treatment became significant. Thus, for the statistics with control variables, we could now fully support Hypothesis 3. Regarding the significance of the control variables, we found that the longer participants lived in Germany, the better they performed in-game and in the multiple-choice test. This control variable can be seen as a proxy for prior knowledge about the participants’ waste sorting. Furthermore, the general waste sorting motivation value showed a tendency to positively affect the performance measures for all three measures (p ranges between 0.01 and 0.11). The SUS value of the game also had the tendency to positively influence the game performance (p = 0.064 for all three hypotheses).

In terms of our first and overarching research question, we found that the learning outcome for the groups given the game for training was significantly stronger than for the group given state-of-the-art paper-based information during the training phase. This held true across all three measures. Interestingly, the effect was weakest within the in-game performance measure (with 4.1% more items correctly sorted than by the non-game material group) and strongest in the multiple-choice test (with 8.4% more items correctly sorted). This outcome contrasts with the literature on context reinstatement, which suggests that information encoded in one mindset is more successfully retrieved in the same mindset (Fisher and Kraig 1988). This interesting finding was also apparent in the non-game material treatment, where performance in the in-game measure was significantly higher than in the multiple-choice test (Wilcoxon signed-rank test with z = 5.16; p < 0.01) although the games’ aesthetic and interaction were new to the non-game material treatment.

To gain further insight in this matter, we were interested in whether all participants generally performed better in one measure or the other. We found that performances, when measured in the game (Wilcoxon signed-rank test with z = 12.00; p < 0.01) and in real life (z = 7.85; p < 0.01), were significantly higher than when measured with the multiple-choice test. We also found that the game-trained group performed comparably well in the game and in real life (z = 1.95; p = 0.06) (for the descriptives, see Table 7). A potential explanation for this finding can be linked to the forming of memory connections: the multiple-choice test offered fewer memory connection items (offering only designated connections: words) than the game measure, which presented both iconic and designated connections (words and sticker-like icons) and the real-life measure, which provides real objects. Both the game and real-life objects offered more information items that could connect to existing schemata. This might have helped stimulate memories not activated by the fewer connections offered in the multiple-choice test. This finding is congruent with studies on word and picture learning (Kolers and Brison 1984; Mayer 2002) that found that learners performed better through a combination of words and pictures/objects than with words alone. Similarly, in the domain of mathematical didactics, studies have found that using more mathematical representations (like graphs, numbers, formulas) leads to an increased learning outcome (Ainsworth 2006). Our results showed that this effect works in both directions: learners retrieved formed memories more successfully if we offered more memory connections with their mental schemata.

In terms of Hypothesis 2 – adding repetition as a game design element increases the learning outcome – our results confirmed our conjectures. The group given the additional option to repeat waves showed a significantly higher learning outcome than the non-game material group in two of the three measures (multiple-choice and real-life). This also held true for the in-game measure when inserting control variables. However, when compared with the core gameplay group, the implementation of a repeat option by itself did not increase the learning outcome significantly. The game design elements enhanced learning potential; however, this manifested within the success of the combined design elements. This suggests that the repeat elements inherently lacking in fun can be compensated for better results. This is underpinned by a study by Kim and Shute (2015), who found that changes in just one design element “significantly impacted players' interactions with the game by changing players' mental ‘operational rules’ during play” (p.351). While the use of the repeat element was optional, it was generally well-received, as 63.95% of players who had the repeat element available used it at least once (mean: 3.88, min: 0, max: 24).

For Hypothesis 3 – the increase of the learning outcome through a look-up design element – our results showed that the group given this design element performed significantly better than the non-game material group in all three measures. In terms of usage, the players received it even better than the repeat element, as 67.86% of players who had the look-up element available used it at least once (mean: 14.42, min: 0, max: 85). These are relevant findings given that we found contradictory indicators on the potential outcome in our analysis of related literature (e.g., studies on help tools reported low usage as well as low effects Liu and Reed 1994; Größler et al. 2000; Aleven et al. 2003)). When compared to the core gameplay group, the prevalence of the look-up element by itself did not significantly enhance the learning outcome of the game. However, as mentioned above, in combination with the repeat element, this design element created a significantly stronger effect in the in-game and multiple-choice measures. This showed that look-up features should be considered as important design elements in learning-oriented gameful applications.

Literature on error management training (Chillarege et al. 2003; Keith and Frese 2008) provides a potential explanation for the success of the combination of these two design elements. In contrast to errorful and errorless learning, this method (EMT) consists of helping trainees understand why errors occur, indicating how they can be avoided (as afforded by the look-up element) and then applying that knowledge to solve the problem (as afforded by the repeat element). This offers positive indications that if both affordances are implemented at the same time, they could lead to an especially successful learning outcome. This can be further consolidated within the theory of learning styles. In a study conducted by Liu and Reed (1994), which considered affordance combinations in a hypermedia environment, learning was accomplished by offering a diverse set of tools and aides to groups of students with different learning styles. This suggests that offering different optional affordances benefits a diverse group of learners and leads to a stronger overall learning outcome. The combined effect could further assist in preventing the perception of cheating that could come with a help or hint-related design element (Clarebout and Elen 2009), as it allows players to test their own abilities in the first iteration of a wave before resorting to looking up the correct solution in the repeated wave.

In summary, the results showed that the core gameplay by itself already performed very well in comparison with the non-game materials. However, for the overall game to be more effective, it can be enhanced successfully by the two design elements that we suggested. Particularly, their combination showed their potential as building blocks for successful learning strategies by combining the mnemonic effect of repetition with easily accessible means for understanding.

When analyzing the control variables, we found that the number of years our participants had been living in Germany positively influenced their learning outcome in terms of the in-game and multiple-choice measures. This connection was expected since this particular control variable was implemented to passively enquire about prior waste sorting knowledge (to prevent priming, we decided against a full pre-measure of waste sorting knowledge – see Limitations Section). General waste sorting motivation also proved to have a significant influence over the learning outcome of the in-game measure alone. However, it is difficult to make sense of the fact that this effect was not replicated in the other measures – especially the real-life waste sorting measure. There could be influences in terms of cognitive dissonance of self-belief and self-actualization, but because of the setup of the experiment, we could not derive any personality-based indicators.

The central goal of our research is to contribute to the rise of sustainable behavior through gameful design, specifically with regard to waste management. This goal stands in line with point 12.8 of the UN catalogue of sustainable goals: “Ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature” (United Nations 2020). Our study showed that gameful design can successfully contribute to better municipal waste sorting, even with regard to a transfer of knowledge to real life. To the best of our knowledge, this is the first study to do so.

We further contributed to the ongoing efforts of investigating the potential of serious games and the implementation of gameful design as powerful teaching devices. In particular, the study showed significant positive learning outcomes within a domain that generally lacks incentives relating to direct personal interest – such as health or fitness-oriented games would offer – and that is hampered by disinterest or even disgust by their target group regarding the general topic. By successfully translating this into more desirable content matter, our research highlighted the benefits of gameful design for teaching under adverse conditions. In terms of theoretical contribution, by conducting a full assessment of design choices with regard to their different learning outcomes, our research added to the ongoing general efforts of methodically assessing learning through gameplay. In this, our study lined up with a growing amount of research dissipating still-existent doubts about the usefulness of game-based learning (Shute et al. 2009).

A factor that contributes to such doubts is that not all studies in gameful learning test the success of their artifact in connection with its transition to real-life knowledge and applicability (e.g., Kim and Shute 2015). This measure, however, is very important, as seen in, for instance, Größler et al. (2000), who found in their study on gamification of business simulators that “participants were not capable of accessing the knowledge gained outside the gaming context” (p. 271). Another example is Ball et al. (2002), who concluded that cognitive training may only improve skills that are specific to the trained cognitive domain. Also, Luo et al. (2018) conducted a study with a similar premise and goal to ours and did not manage to reinstate the learning outcome when measured in real life. In contrast, in our study, we found that our game did overall manage to overcome this hurdle. Despite this success, the difficulty of constructing knowledge could be seen in the differences in learning outcomes between the different testing media. Our study highlighted the importance of measuring in the training medium as well as the true context medium (real life) and proving that the transfer is manageable given good design choices (Van Eck et al. 2017).

We also identified a gap in the IS literature on the effectiveness of look-up/help-based design elements and added to the ongoing discussion by conducting an experimental setup that tested this element in an isolated and a combined treatment. Our results showed that affording an optional, learner-moderated look-up element can be a very promising learning-enhancing design element, especially if added to a repetition-based teaching setup. By intricately testing these specific game mechanics, we contributed to understanding how they function to produce meaningful learning experiences, which is a paradigm suggested by the Games, Learning, and Society initiative (Squire 2007). With regard to the general topic of sustainability in IS, our study was one of few to focus on challenges surrounding the domain of waste management. We hope to inspire further studies in this seminal area of research.

In terms of its practical contribution, we believe that if implemented into the teaching curriculum of sustainability classes, our artifact can have a beneficial impact on the topic of correct waste sorting. Our research aims to support the process of research informing practice and aide designers in optimizing their design decisions, as they have to make efficient decisions under time pressure (Stacey and Nandhakumar 2009). Furthermore, as stated by Clow (2013), educators need to be given insights about additional tools, as well as their strengths and limitations, which we provide in this manuscript. By affording detailed insights into the rationales behind the design decisions that went into the creation of our game and the design elements used, we facilitated easy means of reproduction for practitioners and researchers. While the mechanisms we looked at are embedded in the framework of a game, any learning or training context can serve as the foundation for the design mechanisms we analyzed in our study (Deterding 2016). Thus, we argued that in a playful setting that allows a certain degree of make-believe, a broad variety of teaching tasks (e.g., vocabulary, geography training, digital management training and onboarding) can benefit from applying the findings of our study.

One potential limitation concerns the fact that we omitted assessment of prior knowledge on waste sorting. Due to the three-phase setup of the experiment, we consciously decided against this assessment because of concerns about priming the participants and thus skewing the results. While it is common in the assessment of serious games to use pre- and post-testing, “the main problem with the pre- and post-test experimental design is that it is impossible to determine whether the act of pre-testing has influenced any of the results.” (Bellotti et al. 2013 p.3). By conducting a prior assessment like completing a survey-based multiple-choice test, we were concerned that participants would influence the actual results by looking up certain items they were unsure of before the first task. Instead, we measured “living in Germany” as a proxy indicator for prior knowledge, which turned out to have a significant influence on the learning outcome. Because we conducted an experiment by randomly assigning participants to treatments, we trust the internal validity of our results. Thus, the effect should be independent of confounders such as prior knowledge.

We believe the exclusion of prior knowledge as a predictor in our models, as well as the omission of measuring other variables that might influence waste sorting knowledge – e.g., participants’ exposure to the topic in school or other contexts or their families’ attitudes towards sustainability and eco-friendliness – are the main reasons for the rather low R² of our main models that included only the treatment variables. However, a low R² is not unusual for experimental research and does not harm the interpretation of the effect of the treatment variables. Our further analyses in the appendix also show that the inclusion of the control variables – e.g., the number of years that the participants have lived in German and general waste sorting motivation – helps reduce the unexplained variance substantially, yielding R² values around 0.2.

We further see that there could be an underlying cultural bias given the generally high range of results. Frese et al. (1991 p. 90) noted that errors may be perceived as especially stressful in German culture, “where perfectionism is highly valued.” For transferability to other cultures with different prior mentalities regarding correct waste sorting, future studies will be necessary to assess mentality as a moderating factor. Another important point is that we assessed the real-life measure with only seven items. While this arguably weakens comparability with the other two measures, practical concerns in terms of implementing a much larger number of items (limited setup and timeframe, participants’ resistance to interacting with certain items) limited our options for this measure. It should also be noted that within the non-game materials used during the training phase, one of the three flyers (the one showing examples of waste items for each bin in Fig. 10 in the appendix) featured more items than were presented to the game groups during the experimental task. When we designed the experiment, we wanted to approximate a real-life scenario and thus chose to use an unabridged set of standard materials provided by the local waste management (see in appendix “Non-Game Materials”). In hindsight, the overall experimental design would have been cleaner if we had reworked the flyer to feature the exact number of items that were trained in the game. However, it is important to note that the goal of the game was not to teach the specific relationship of each featured waste item to their bin but to help players understand and train them on the rules of the underlying waste systems. As all objects will eventually turn into a waste item, citizens need to learn how to correctly sort any object they encounter to the respective system by understanding and internalizing the underlying principles.

We wanted to test the learning outcome in a rigorous and controlled manner to obtain clear and interpretable results, so we decided to conduct a laboratory experiment to provide high internal validity. However, as our findings were based on an experimental setting including mostly students, any gained insights were only applicable for the tested age group (17–41). A future step would be to test whether the effects found are replicable in the field. Another facet of this relates to knowledge transfer. Even though we found that knowledge transfer to real life (construction) was successfully achieved in the game, we believe that this effect might be enhanced by transferring the game to a virtual/augmented reality environment by bringing the medium of training closer to the actual application context.

Finally, while we chose to separate learning from motivation to isolate our findings, this approach might have omitted an important influence on learning. On this basis and because of our overall goal to teach correct waste sorting and to boost the motivation to act upon that knowledge, we want to design and conduct another motivation-focused experiment to build on our findings and enhance gameful design-based learning even further.