Abstract
Affine Transformations (ATs) are a complex and abstract learning content. Encoding the AT knowledge in Game Mechanics (GMs) achieves a repetitive knowledge application and audiovisual demonstration. Playing a serious game providing these GMs leads to motivating and effective knowledge learning. Using immersive Virtual Reality (VR) has the potential to even further increase the serious game’s learning outcome and learning quality. This paper compares the effectiveness and efficiency of desktop-3D and VR in respect to the achieved learning outcome. Also, the present study analyzes the effectiveness of an enhanced audiovisual knowledge encoding and the provision of a debriefing system. The results validate the effectiveness of the knowledge encoding in GMs to achieve knowledge learning. The study also indicates that VR is beneficial for the overall learning quality and that an enhanced audiovisual encoding has only a limited effect on the learning outcome.
1. Introduction
Affine Transformations (ATs) are part of linear algebra, used for kinematic control [1], computer graphics [2], and development of Virtual Reality (VR) applications. In case of computer graphics, learners are challenged to develop an understanding how the theoretically grounded mathematical aspects result in an object’s transformation. ATs are expressed as matrices, usually of dimensionality , and their operations as matrix-matrix multiplications, each matrix representing one desired mapping. Hence, ATs are a very complex and abstract learning content that cannot easily be demonstrated.
The Gamified Training Environment for Affine Transformations (GEtiT) was specifically developed to address this problem. It intuitively requires the application of ATs and audiovisually demonstrates the underlying theoretical principles [3]. GEtiT yields a similar learning outcome to a traditional paper-based learning method while achieving a higher learning quality [4]. Also, GEtiT was developed as a demonstrator for the Gamified Knowledge Encoding model [5]. The Gamified Knowledge Encoding utilizes Game Mechanics (GMs) to directly encode a knowledge’s underlying principles as their internal game rules. This achieves a learning content’s repetitive application and audiovisual demonstration during the gameplay. GEtiT embeds the gameplay in complex problems, i.e., an escape scenario, to cause an intrinsic motivation in the learner to tackle the learning assignments.
The repetitive application of the encoded knowledge takes place on the skill-based or the rule-based layer of human performance [6] and leads to a compilation of mental models [7]. Mental models are complex mental constructs allowing for an internal visualization and are used for a knowledge application on the knowledge-based layer, i.e., a training transfer. Training transfer takes place when knowledge training in one context leads to an increased performance when applied in a different context [8]. In this way, the Gamified Knowledge Encoding defines how knowledge is learned with serious games.
However, it is unclear whether the audiovisual presentation of the encoded knowledge and the degree of the visual immersion has an effect on the learning effectiveness. For instance, immersive Virtual Reality (VR) has the potential to even further increase GEtiT’s learning outcome by presenting the learning content in a visually immersive and more natural way. Therefore, a specific GEtiT VR version implementing the same core GMs was developed [9]. This allows for a direct comparison between the two visualization technologies in respect to the learning outcomes of knowledge encoding using GMs.
This paper’s contribution is threefold: (1) comparison of the effectiveness as well as efficiency of GEtiT’s desktop and VR version, (2) validation of GEtiT’s learning outcomes [4], and (3) analysis of the effectiveness of an enhanced audiovisual encoding as well as the provision of a debriefing system. The present user study confirms the effectiveness of GEtiT by showing a similar learning outcome to a traditional paper-based learning method. Also, the results indicate a higher learning quality when using VR technology. Overall, this paper contributes to the ongoing research of analyzing the effectiveness of VR technology for educational purposes.
At first, an overview over the current state of research is given and the Gamified Knowledge Encoding is explained in detail. Subsequently, GEtiT is described and the study to compare the tested versions is explained. This is followed by the presentation of the study’s results and an in-depth discussion of the findings. Finally, the paper is concluded with a summary and an outlook for future research.
2. Related Work
Well-designed computer games automatically fulfill the conditions for optimal learning [10]. They present the encoded game knowledge in a highly engaging and immersive way. This achieves a high player motivation to tackle a game’s tasks and challenges. A game’s overall gameplay requires a repetitive application of the encoded knowledge, thus ultimately achieving a learning effect due to repetition [11]. Computer games periodically increase the gameplay’s difficulty to compensate for the learning effect and to continuously provide players with new tasks that keep them challenged [12]. In this way, a computer game requires preexisting knowledge and, over time, even requires the knowledge acquired during the gameplay. Computer games provide players with immediate feedback about the effects as well as the correctness of their actions and their progress towards solving a challenge. Simultaneously, a constant stream of new challenges paired with an immediate feedback increases a game’s flow-inducing aspects [13]. Flow is the central construct that mainly influences enjoyment and performance of gaming action [14]. Hence, it increases a player’s intrinsic motivation for knowledge learning [8].
2.1. Game-Based Learning
Computer games have already been implemented to learn complex sets of human skills such as leadership styles [15, 16], as well as skills of communication [17, 18] and cooperation [19, 20]. Video games were also used to train human abilities, such as the cognitive flexibility trait [21], the spatial visual attention [22], and the spatial resolution [23].
Game-based learning led to the development of serious games. Serious games feature an educational aspect and are not solely developed for entertainment [24, 25]. They are designed to educate players in a broad variety of topics like genetics [26] or biological consequences of alcohol abuse [27]. Also, serious games are not only used to teach about a specific knowledge, but also to motivate players to consider a science career [28].
In general, computer games encode specific knowledge being learned and mastered during the gameplay [29, 30]. Players periodically discover new challenges and multiple ways to solve them [31]. The immersive effect of playing a computer game [32, 33] can introduce players to ethical questions [34] and moral problems. This results in a training of moral decision making [35].
2.2. Game Mechanics
Each computer game consists of GMs encoding the underlying game rules. GMs are distinguished in player-bound GMs and game-bound GMs [5]. Game-bound GMs create the game world, provide challenges to a player and realize the overall narrative [36]. Player-bound GMs are executed by the player to achieve an interaction with game-bound GMs [37]. This interaction not only creates the gameplay, but also provides an immediate feedback about the effects of a player’s actions. Hence, GMs structure the gameplay, encode underlying principles, and define the game world as well as a player’s abilities [36].
For instance, a computer game might feature moving platforms on which a player is required to jump. The moving platform element is a game-bound GM as it is automatically executed and cannot be manipulated by the player. The ability to jump is a player-bound GM. Based on the outcome of the jump ability’s execution, players are provided with a clear feedback about their performance as they either hit or miss a platform.
The game-specific knowledge, i.e., the encoded game rules and principles, needs to be understood by the players to successfully play a game [11]. For example, the moving platforms GM encodes the platforms’ movement speeds and trajectories. The jump GM encodes the jump distance, the jump speed, and the actual action that needs to be performed, e.g., the key that needs to be pressed on a keyboard. Only when players have developed a basic understanding of this game-specific knowledge, they can master the challenges created by the GMs’ interaction.
2.3. Educational Use of VR
Learning of ATs requires an environment that visually demonstrates 3D geometrical problems. Computer games challenge a player’s skills of logic, memory, visualization, and problem-solving during the gameplay [38]. Fast-paced computer games, e.g., action-based computer games, improve cognitive abilities [39], thus enhancing a player’s ability to monitor and to observe task-relevant information [40]. More importantly, 3D action-based computer games train a player’s spatial abilities, such as the mental rotation skill [41], spatial visual attention [22], spatial resolution of vision [23], and spatial navigation [42]. This is crucial for GEtiT as a training of spatial abilities improves 3D geometry thinking [43]. Vice versa, training descriptive geometry assists the development of spatial abilities [44]. Thus, by visually demonstrating the AT knowledge in a 3D environment, the learning process is facilitated.
VR technology visually immerses a user in a 3D environment allowing for such a presentation of 3D geometry. As a result, designing a specific GEtiT VR version has the potential to enhance the learning effectiveness. VR technology provides the advantages of increasing a student’s motivation as well as engagement, achieves an immersive experience, and allows for a constructivist approach of learning [45, 46]. Also, a higher visual immersion and presence leads to a higher performance in case of a training scenario [47]. Spatial presence describes the subjective sensation of being in a real place, i.e., the virtual environment (VE), despite physically being in a different environment [48]. Presence has a mediating effect on the learning outcome as it affects a student’s intrinsic motivation and enjoyment, thus increasing the perceived learning quality and satisfaction [49]. Visual immersion is achieved with system properties reducing sensory inputs from the real world and replacing them with digital information, e.g., wearing a Head-Mounted Display (HMD) [50]. Utilizing HMD-VR allows users to easily change their perspectives which helps to analyze complex learning contents like 3D geometry [51]. Also, as an audiovisual presentation supports the compilation of mental models [52], a full visual immersion in such a presentation environment should further improve the learning outcome.
Therefore, designing a specific GEtiT VR version has the potential to increase the learning effectiveness and the learning quality.
2.4. Virtual Geometry Learning
Virtual learning of geometry was already approached with other projects. Construct3D represents an Augmented Reality application that allows students to collaboratively create and manipulate geometrical objects [53, 54]. Similarly, Mathland provides a learning platform that augments the real world with mathematical concepts like Newtonian physics, thus allowing for a learning in constructivistic ways [55]. In contrast to the present system, both applications are not gamified training environments that target a highly motivating knowledge learning.
3. Gamified Knowledge Encoding
The definition of the Gamified Knowledge Encoding [5] relies on the theoretically grounded concepts of knowledge [56, 57], human performance [6], mental models [7], and GMs [36, 58]. Declarative knowledge consists of information, facts, methods, and principles describing what a subject is, whereas procedural knowledge reflects motor or cognitive skills, hence describes how an action can be performed [56, 57]. The Gamified Knowledge Encoding maps the learning content as game rules to interacting GMs. In this way, the resulting gameplay creates learning affordances [59] for the knowledge to be learned. A learning affordance requires an interaction with the learning environment, i.e., an application of the knowledge, and simultaneously informs about the underlying principles [60]. The knowledge encoding is determined by the moderation, i.e., the degree to which knowledge rules are simplified, and mediation, i.e., the concrete realization of a GM. This section theoretically presents the proposed framework which then is demonstrated in Section 4.
3.1. Knowledge Encoding
Working with the Gamified Knowledge Encoding, players entrain the encoded knowledge on a skill-based and rule-based level of human performance during the gameplay. As a result, learners compile a mental model for the learning content [61] that allows them to transfer their knowledge to a different context, e.g., a real world application.
A direct knowledge encoding using the Gamified Knowledge Encoding is achieved by segmenting the learning content into smaller packages of which each describes a coherent part of the knowledge. Each knowledge package then is turned into a gameplay element requiring its application. For this purpose, the knowledge packages are transformed into clear and well-defined rules that are mapped to interacting GMs. This mapping process generates a gamification metaphor representing and requiring the learning content inside of a serious game. Player-bound GMs encode rules defining and requiring the actual knowledge application as game inputs. Game-bound GMs act as a verification system to check if a player’s inputs are correct or as a demonstration system to visualize the inputs’ effects. The interaction between a gamification metaphor’s GMs requires the knowledge’s application and informs about the underlying principles by providing immediate feedback.
3.2. Moderation and Mediation
Directly encoding the learning content in gamification metaphors might not necessarily result in an intuitive learning process. This especially is problematic in case of abstract knowledge which is hard to visualize and often escapes an intuitive approach. Therefore, the Gamified Knowledge Encoding also includes a knowledge moderation and a knowledge mediation to adjust the encoded knowledge’s level of abstraction. Also, the moderation and the mediation determine the knowledge presentation inside of a serious game.
The knowledge moderation scales the level of abstraction of the encoded knowledge by adjusting the accuracy and the selection of the sets of knowledge rules mapped to the gamification metaphor. Thus, the Gamified Knowledge Encoding creates a direct knowledge encoding that ranges from a non-moderated accurate simulation to a highly moderated simplified and intuitive knowledge application. By adjusting the moderation over time, the level of abstraction matches a learner’s knowledge gain. This relies on the game design principle of continuously increasing the difficulty to keep players challenged and in flow [12].
An abstract knowledge learning process can begin with a very intuitive demonstration of the learning contents. This is achieved by merely encoding a simplified set of rules, thus establishing a certain distance to the knowledge. Subsequently, as the learners progress through the gameplay, more complex sets of rules are mapped to the GMs. This reduces the initial distance to the knowledge over time. Finally, the complete and non-moderated set of rules is mapped to the GMs to completely close the distance and to achieve the knowledge’s simulation. When adjusted well, the game’s challenge and difficulty increase matches the current knowledge and/or skill level of the players. As a result, the gameplay’s flow-inducing aspects are created and maintained.
The knowledge mediation, i.e., the selection and the realization of GMs, partly depends on the degree of the knowledge moderation. A low degree of knowledge moderation requires GMs that accurately encode the knowledge rules, i.e., they remodel and simulate a particular real world application. In contrast, a high degree of knowledge moderation reduces the requirements and allows for GMs that represent complex knowledge rules with generalized and intuitive interactions.
For instance, a driving simulation can require an individual utilization of the clutch but also automatically include it during a shifting process. In the former version, two separate GMs are needed while in the latter implementation one GM combines both activities resulting in a more simplified knowledge presentation. Thus, the knowledge mediation can also scale the level of abstraction. It allows for a direct encoding of non-moderated knowledge rules in GMs that integrate and combine several sets of rules to achieve an intuitive application. In conclusion, the moderation and the mediation define a knowledge’s application and demonstration.
3.3. Optimal Knowledge Learning
Utilizing the Gamified Knowledge Encoding creates serious games that fulfill the conditions for optimal learning [10]. By encoding the learning content in interacting GMs, the serious game automatically provides learners with immediate feedback about the correctness of their inputs. By moderating the knowledge’s level of abstraction, highly motivating flow as well as a requirement for preexisting knowledge is created. Finally, a repetitive knowledge application is established by the requirement to frequently execute the gamification metaphor’s GMs during the gameplay.
The Gamified Knowledge Encoding describes the direct knowledge encoding in GMs and the resulting learning process during the gameplay. However, to ensure for a good playability, additional GMs targeting either entertaining aspects or providing further gameplay enhancements may be provided. For instance, the computer game Kerbal Space Program encodes knowledge of orbital mechanics in its core GMs [62]. As a result, players learn and practice this knowledge during the gameplay [63]. In addition to the orbital mechanics gamification metaphors, Kerbal Space Program implements further GMs to increase its playability, e.g., by realizing a career mode or by allowing players to plant flags on the surface of a celestial body. Thus, by providing further GMs in addition to the ones used in the gamification metaphors, a serious game’s overall entertaining and motivating aspects may be improved.
In conclusion, the Gamified Knowledge Encoding utilizes GMs as an educational tool by mapping knowledge rules to them, thus directly encoding the learning content (see Figure 1). The Gamified Knowledge Encoding utilizes the interaction between at least one game-bound GM and one player-bound GM to require the application of the learning content on a rule-based or skill-based level of human performance. Subsequently, learners are provided with immediate feedback about their learning progress. This learning process results in the compilation of a mental model for the knowledge. This mental model ultimately is utilized to apply the knowledge on a knowledge level, i.e., transferring it from the serious game to a real world context. The GMs that encode the knowledge’s rules and that interact with each other are metaphors for the learning content. They are responsible for a player’s knowledge gain by acting as learning affordances. We define such a gamification metaphor as knowledge’s gamified metamodel which can be fully internalized in the form of mental models.
4. Gamified Training Environment for Affine Transformations
GEtiT’s development followed the guidelines of the Gamified Knowledge Encoding. The main goals of this development process were (1) to transform the AT knowledge into game rules and (2) to realize GMs that mediate them. Subsequently, after demonstrating its effectiveness in its prototype version [4], GEtiT’s visual style was changed to a state-of-the-art style of modern computer games (see Figure 2). This major overhaul included the implementation of a background music and sound effects to provide learners with additional acoustic feedback. Also, GEtiT received a more advanced point system, an achievement system, a debriefing system, and a small built-in wiki. This section presents GEtiT’s design, describes the realization of the new features as well as the specific VR version, and demonstrates the Gamified Knowledge Encoding.
4.1. Design
4.1.1. Core Gameplay
Working with the Gamified Knowledge Encoding, the AT knowledge first was separated into the individual theoretically grounded mathematical operations and the resulting transformation effects. The mathematical operations were mapped as game-knowledge rules to a player-bound GM mediating each individual operation as a playable AT card. Activating a card displays a direct value configuration screen resembling the structure of a matrix that allows for the operation’s configuration (see Figure 3).
The AT cards, of which each can only be played once during a particular level, moderate the level of abstraction of the learning content. The degree of the moderation is controlled by providing four different difficulty levels: easy, medium, hard, and expert. Depending on the selected difficulty level, a card represents either a specific AT operation vector (easy), an empty transformation vector (medium), or an empty transformation matrix (hard, expert). Empty AT cards need to be defined via the direct value configuration screen that further moderates the level of abstraction by either resembling the structure of a vector or a matrix. The matrix only provides access to fields relevant for the selected AT operation type on hard difficulty and needs to be completely configured on expert difficulty. The cards, which are activated by clicking on them, are shown at the bottom of the user interface and display the predefined values as well as the transformation type. The transformation type is indicated with a symbol and a distinct color allowing for a fast and easy recognition (see Figure 4).
The transformation effects were mapped as knowledge rules to a manipulable game object, i.e., a game-bound GM, presented in the form of a cube. Configuring and subsequently playing a card internally propagates the entered values to the object GM that immediately changes its status. The object additionally casts an orange trail indicating the path on which it has translated. Thus, the object mediates the effects of an AT operation by providing an immediate feedback and visually demonstrating the underlying principles. The object’s position is displayed in GEtiT’s user interface to provide learners with concrete values they need to use to correctly compute further AT operations. In this way, GEtiT directly encodes the mathematical rules of matrix algebra and their utilization to express and to perform ATs (see Figure 5).
The application of ATs is required by GEtiT’s level design following the concept of an escape scenario [64]. Each individual level challenges a player to activate an exit portal by solving a spatial AT puzzle. The spatial puzzle is solved by transforming the object in such a way that it matches a level’s victory conditions. The victory conditions are presented in form of a semitransparent copy of the transformable object, i.e., a game-bound switch GM, that indicates the required position, rotation, and overall status of the object. GEtiT additionally displays the coordinates of the switch to allow learners to mainly focus on determining the correct mathematical solution instead of being challenged to locate the target position manually. As soon as the victory conditions are met, the exit portal is opened and the player can proceed to the next spatial puzzle (see Figure 6). The interaction between these three GMs creates a gamification metaphor for ATs.
The AT gamification metaphor creates a learning affordance for the AT learning content. Users are required to execute the AT cards GM during the gameplay, thus repetitively applying their AT knowledge on a rule-based level of human performance. Subsequently, they get visually informed about the underlying principles as the object immediately changes its state. This repetitive practice leads to a compilation of mental models for ATs. These mental models ultimately achieve a training transfer from the serious game to a real world application like utilizing ATs to create VR systems or simply solving the assignments of an exam.
4.1.2. Gameplay Enhancements
Aside from the three core GMs, GEtiT includes additional GMs to enhance the usability as well as the playability and to increase the learners’ motivation. For enhancing the usability, GEtiT displays the position of a level’s origin and the direction of a level’s axes. The former information is mostly needed when a rotation or reflection operation is desired. The latter information is relevant for every transformation operation. Also, GEtiT provides an undo function to allow learners to revert their last action in case of a wrong input. The serious game provides a small built-in AT wiki that informs about the underlying theoretically grounded mathematical aspects. The AT wiki keeps learners immersed when they need to look up further information to determine a spatial puzzle’s correct solution.
For the purpose of enhancing GEtiT’s motivational aspects and playability, an achievement and a point system got implemented. The point system is based on a performance rating system that challenges players to solve a level with a minimum amount of cards. Using the undo button keeps the draw counter unchanged to keep players from exploiting it. Beating a level with the minimum or small deviation from it rewards players with a performance dependent amount of points symbolized by stars. The points simultaneously provide users with feedback about their progress towards the completion of the game, i.e., stars earned for a particular level are displayed in the level selection menu. Also, the point system is used to create a ranking among all players when GEtiT is played in classroom mode. Here, GEtiT communicates with a database server to synchronize the points of all registered players. Achievements are unlocked by solving levels in a perfect way, completing all levels of a particular transformation type, or finding a hidden Easter-egg.
4.1.3. Debriefing
GEtiT displays a debriefing screen after a level was solved (see Figure 7). The debriefing system provides additional immediate feedback that allows learners to reflect on their computational results [65, 66]. The debriefing screen informs about the number of cards used, the level’s minimum, the stars achieved, the time needed, and a composite mathematical equation of the used ATs. The composite mathematical equation aims at the development of an understanding of different forms of expressing AT operations. This is critical as it directly integrates the theoretically grounded mathematical aspects into the gameplay. By displaying concrete matrix-matrix multiplications, learners can integrate this knowledge in their mental models. The debriefing screen also provides options to continue to the next puzzle, to retry the current puzzle, or to return to the level selection menu.
4.1.4. Audiovisual Encoding
Various sound effects were implemented in GEtiT to provide learners with acoustic feedback [64]. Each AT type received an individual sound effect that is played when an AT card is activated. This provides players with an acoustic feedback when a specific AT operation type successfully was applied. Furthermore, GEtiT provides sound effects for walking (footsteps), jumping, touching a card, using the undo button, and a general event indication. The game includes a dubstep-like background music to support its futuristic visual style.
4.2. GEtiT VR
GEtiT VR utilizes the same GMs as GEtiT but realizes them in a diegetic way [67] to increase the system’s naturalness, presence, and usability [68, 69]. Naturalness refers to the degree with which actions and effects in a VE correspond to the actions and effects in the real world [70]. The naturalness of an interaction depends on the degree with which it matches the task context [70]. Thus, naturalness is affected by the intuitiveness of the interaction [71]. This main design decision was made to allow for a comparison of the learning outcomes between the two different visualization technologies without confounding the results by implementing different GMs. GEtiT VR presents the AT cards as physical objects inside of the VE (see Figure 8). A moveable card holder gives players access to the cards. Selecting and configuring an AT card is realized with a selection and manipulation interaction technique. Selection and manipulation techniques are one of the three fundamental 3D interaction tasks [72]. Their realization is defined in terms of a user’s distance to the target element. The distance can either be remote requiring an artificial pointing metaphor, e.g., a virtual ray, or within arm’s reach allowing for a direct interaction [73]. The latter approach is a very natural interaction technique and can be realized with grasping metaphors simulating a user’s hand or controller inside of an VE [72].
Implementing a within arm’s reach grasping metaphor, players select a card by merely touching it with one of the game controllers (see Figure 9). A controller’s position is indicated with its 3D asset inside of the VE. Pulling the controller’s trigger button activates the selected AT card. Touching the controller’s trackpad displays the direct value configuration screen and allows players to configure a card using the second controller. This is done by selecting a value in the configuration screen. Subsequently, pulling the controller’s trigger button confirms the selected input.
The positions of the object and of the target are communicated via diegetic labels being directly attached to the objects inside of the VE. Other pieces of information, such as the level selection screen, the main menu, and the AT wiki, are presented in a diegetic way by providing a playing room (see Figure 10). Players can transition between the playing room and the spatial puzzle levels using a Virtual HMD metaphor [74]. This diegetic transition technique metaphor is very natural and provides a high degree of self-control. By slowly putting on or taking off the Virtual HMD, users are in full control over the actual transition. As GEtiT’s levels are normally larger than the tracking area, GEtiT VR implements the intuitive and easy Point & Teleport technology [75] to perform a locomotion inside of the VE aside from real walking [76, 77].
The development of this specific GEtiT VR version was mainly guided by the research goal to analyze if providing a full visual immersion while keeping the gamification metaphor the same leads to an increased learning outcome. GEtiT’s GMs were directly ported to VR and realized as diegetic and natural interfaces. This approach, however, neglected further adaptions to ensure a similar usability to GEtiT. Both GEtiT versions were compared in respect to their usability in a user study [78]. In particular, the study analyzed the games’ efficiency as well as flow-inducing aspects and the users’ satisfaction. The efficiency was evaluated by measuring the elapsed time and experienced task load when solving specific tasks, e.g., solving a particular level. The satisfaction was determined by assessing the games’ intuitive use and by analyzing the users’ preference. The results revealed slower times when using the direct value configuration screen as well as a higher task load in GEtiT VR. The intuitive use did not differ significantly between both versions and the majority of the participants favored GEtiT VR. Also, flow did not differ significantly between both versions. Thus, the results validated the overall design and the overall playability but indicated potential issues with the realization of the direct value configuration screen in VR. As a result, GEtiT VR is a mere prototype and potentially not directly comparable to GEtiT in respect to its learning effectiveness. A comparison of both systems still is very critical to gain insights into the overall feasibility of this approach and to draw technical design guidelines from the results.
4.3. Learning Approach
GEtiT fulfills some aspects of situated learning [79–81]. The serious game guides the learning process with a complex problem and embeds it in an authentic context. GEtiT provokes an intrinsic motivation in the learner to solve the learning assignments, i.e., to find a solution to the spatial puzzles, by providing an escape scenario. Targeting a training transfer to a computer graphics context [2], GEtiT creates an authentic context by requiring the application of ATs to transform a virtual game object inside the VE. This is achieved by providing the direct value configuration screen requiring the completion of matrices. However, the serious game lacks the aspects of collaborative construction and reflection of the learning content which is typically associated with the situated learning theory [82]. Also, GEtiT is designed to achieve a transfer-oriented learning of ATs instead of mainly linking the learning content’s application to the situations created during the gameplay.
GEtiT also fulfills some aspects of problem-based learning [83, 84]. Problem-based learning is self-directed learning being motivated with a complex problem [85] and being assisted with scaffolding that guides the learning process [86]. Solving the presented task provides learners with the opportunity to develop an understanding of the underlying principles and to acquire new knowledge. GEtiT acts as a tutorial system, provides learners with complex tasks and scaffolds them. In this way, GEtiT provides opportunities for a transfer-oriented learning.
4.4. Technology
GEtiT and GEtiT VR are developed with unity in the version 5.5.2p1 [87] for PC and Mac. The gameplay is rendered to the connected main monitor and, in case of GEtiT VR, to the HTC Vive HMD. The VR implementation of GEtiT VR is achieved using the SteamVR Plugin [88] in the version 1.2.0 which already provided functions for the point & teleport locomotion, controller-based system interaction, controller tooltips, and overall player controller. The playing room’s furniture was freely available on the unity asset store [89] or part of the unity standard assets.
5. Experimental Design
Due to the overall indications discussed in Section 2, the underlying design principles derived from Section 3, and the concrete implementation described in Section 4, we assume the following hypotheses:H1The learning outcome is improved when the mediation of the knowledge is audiovisually enhanced.H2The learning outcome is improved when a debriefing system is provided.H3The learning outcome is improved when the learning process takes place in immersive VR.
The experiment to test these hypotheses, to confirm GEtiT’s measured effectiveness, and to validate the Gamified Knowledge Encoding model consisted of two phases. The first phase was designed to analyze the effects of an audiovisual enrichment by comparing two different GEtiT versions. GEtiT in the enriched version utilized the aforementioned audiovisual encoding of the AT cards by providing a distinct symbol color and sound effect for each individual transformation type. The reduced version utilized the same color and provided the same sound effect for every transformation type. The first phase included a traditional paper-based learning method as a third condition. The second phase was designed to compare GEtiT with GEtiT VR in regard to their effectiveness and efficiency. Both GEtiT versions contained the debriefing system and the achievement system which were not implemented in phase 1.
Internally, both phases implemented the same experimental design to achieve comparability. The overall procedure was designed to follow the structure of a traditional class-based learning. The GEtiT-based learning began after the learning content was presented in an interactive computer graphics lecture and before it was fully discussed in the preceding session. In this way, the experiment simulated the implementation of GEtiT in the context of a regular curriculum at a university.
The experiment consisted of four 90-minute learning sessions taking place on a weekly basis. In the week preceding the last learning session, an AT knowledge assessment test was written. The participants who were assigned to one of the desktop-3D GEtiT groups or the paper-Group completed the sessions in the form of a traditional class. The vr-Group was split into smaller two-participant teams due to the amount of available HTC Vive systems in the lab. The vr-Group was required to take a break in the middle of their sessions to reduce the chances for an effect of cybersickness [90] and to avoid a strong effect of exhaustion.
6. Measures
All questionnaires were translated to the common language at the study’s location. For ensuring that all questions were understood properly, the participants’ language proficiency was assessed.
6.1. Simulator Sickness
During phase 2, the simulator sickness was measured for all participants assigned to GEtiT VR before, during the mandatory break, and after a playing session using the simulator sickness questionnaire (SSQ) [91]. The results were used to measure the overall quality of the VR simulation and to identify potential negative effects that could have affected the study’s results.
6.2. Effectiveness and Efficiency
The learning outcome was measured using a 16-assignment pen-and-paper exam assessing the participants’ overall AT knowledge. The assignments were designed to be of similar difficulty to the assignments given in a regular final exam of the interactive computer graphics lecture. Also, GEtiT recorded a participant’s solved levels to analyze the efficiency.
6.3. Learning Quality
The learning quality of the tested learning methods was measured using a self-designed questionnaire (1 = disagree; 5 = agree) following the idea of the assessment method used for the prototype version [4]. The questionnaire consists of two subcategories and specific questions relevant for each of the two phases. The Learning Quality subcategory consists of nine questions (Q1-Q9) and the system-specific Motivational Aspects subcategory consists of six questions (Q10-16). Q17 and Q18 were added to analyze the audiovisual encoding in phase 1. Q19 and Q20 were designed to assess the achievement system and the debriefing system added to the system in phase 2. For evaluating the results, the overall mean for the sum of a subcategory’s questions is computed.
Learning Quality Q1Did you enjoy playing GEtiT / solving the paper-based assignments?Q2Did GEtiT’s puzzles / the assignments help you to develop a better understanding of ATs?Q3Did you notice a knowledge gain while you were solving the GEtiT puzzles / the assignments?Q4Did the raise in the difficulty match your knowledge gain?Q5Were the tasks of the GEtiT puzzles / the assignments easy to understand?Q6Was the difficulty of the GEtiT puzzles / the assignments well adjusted?Q7Were you motivated by new challenges due to a raise in the difficulty?Q8Did you enjoy the class that was based on GEtiT / the paper-based assignments?Q9Was it interesting to solve the GEtiT puzzles / the assignments by using AT operations?
Motivational AspectsQ10Was the serious game-based learning method more enjoyable than traditional learning methods, e.g., paper-based assignments?Q11Would you prefer to utilize a serious game instead of visiting a regular class?Q12Did you notice a higher motivation to play GEtiT to practice your knowledge in contrast to other learning methods?Q13Were you motivated by the additional feedback mechanisms, such as highscores and the number of used operations?Q14Did the feedback mechanisms motivate you to try a particular level again to improve your performance?Q15Were you motivated by the indication of the needed time?Q16Were you motivated by the ranking system?
Phase 1Q17Did the color(s) of the AT cards help you to internalize the different AT operation types?Q18Did the sound effects of the AT cards help you to internalize the different AT operation types?
Phase 2Q19Did you find the possibility of unlocking achievements motivating?Q20Did the mathematical representation of your solution at the end of each level help you to develop a better understanding of ATs?
6.4. Participants
The participants were recruited from the students participating in the lecture on interactive computer graphics. They were offered credits mandatory for obtaining their Bachelor’s degree and bonus points for the lecture’s final exam. After being introduced to the experiment, the participants signed an informed consent form.
Phase 1. In total, 34 students volunteered to take part in the study. Unfortunately, 13 of them missed at least one session and had to be excluded from the sample. The remaining 21 participants (8 females; 13 males) had a mean age of years (). Based on self-report, 13 participants were frequent computer game players. They were randomly assigned to the enriched-Group (), the reduced-Group (), and the paper-Group ().
Phase 2. In total, 27 students volunteered to take part in the study. Unfortunately, 6 of them who were assigned to the GEtiT group missed at least one session and had to be excluded from the sample. The remaining 21 participants (6 females, 15 males) had a mean age of years (). Based on self-report, 13 participants were frequent computer game players. They were randomly assigned to the vr-Group () and the GEtiT phase 2 -Group ().
7. Results
In this section, the results of the user study are presented and evaluated according to the given hypotheses and the additional goals of this experiment. The results were compared by calculating either a one-way ANOVA or a two-sample t-test [92]. The effect size was determined using Cohen’s D. For determining a correlation, the Pearson’s product-moment correlation was computed.
7.1. Simulator Sickness
The participants of the vr-Group were asked to complete the SSQ before the start of the learning session (pre), right after they started their break (mid), and at the end of the session (post). As Table 1 displays, no significant change in the SSQ ratings was found for each of the practice sessions.
7.2. Effectiveness and Efficiency
Initially, the three different GEtiT conditions were compared in regard to the yielded test result (, ; see Table 2) and the number of successfully solved levels (, ; see Table 3) but no significant difference was found. Thus, to increase the accuracy of further analyses, the GEtiT groups were combined and called desktop-Group () in the remainder of this paper. The test results of the remaining three different conditions did not differ significantly (, ; see Figure 11). Further analyses revealed a significantly higher number of solved levels in the desktop-Group with a very large effect size (, , ; see Figure 12). No significant correlation was found for the vr-Group between the test result and the number of solved levels (, ). A significant correlation, however, was found for the desktop-Group between the test result and the number of solved levels (, ).
7.3. Learning Quality
At the end of the experiment, the participants were asked to rate the learning quality. In phase 1, 18 of the 21 participants filled in the questionnaire. In phase 2, all participants completed the learning quality questionnaire. A one-way ANOVA revealed no significant difference between the mean ratings of the learning quality subcategory (, ; see Table 4). Also, no significant difference was found between the individual four tested versions in regard to the motivational aspects subcategory (, ).
No difference was found between the reduced and the enriched version for Q17 (, ) and Q18 (, ) measuring the perceived educational effect of the audiovisual encoding in phase 1. Both visual approaches received a mean rating at the scale’s neutral midpoint. The mean rating for the acoustic encoding was below the scale’s neutral midpoint. The achievement system added in phase 2 received a mean motivational rating above the scale’s neutral midpoint for GEtiT VR and a mean motivational rating slightly below the scale’s neutral midpoint for GEtiT. The ratings were not significantly different (, ). The perceived learning effect of the debriefing system had a mean rating above the scale’s neutral midpoint for both GEtiT versions. The ratings were not significantly different (, ).
8. Discussion
Although a lack of statistical significance does not imply an equivalence, the results indicate that GEtiT achieves a similar AT knowledge learning outcome to traditional learning methods, i.e., by using paper-based assignments. Thus, the effectiveness measurements validate the findings of the initial prototype evaluation by confirming GEtiT’s transfer-oriented learning effects [4]. Also the significant correlation between the number of solved levels and the test result contributes to the ongoing validation of the Gamified Knowledge Encoding. By encoding the AT knowledge as game rules in GMs, a repetitive application of the learning content is achieved during the gameplay. This repetitive practice leads to an internalization of the AT knowledge in form of mental models. It also achieves a shift to a more pattern-driven application. The compiled mental models allow for a training transfer from GEtiT to a real world context. This was tested by implementing a pen-and-paper exam that only uses 2D pre- and post-images to visualize a desired AT operation. The participants of the GEtiT groups were not only required to solve the assignments, but also to transfer their knowledge from the 3D serious games to a 2D paper-based exam. As a result, the learning outcome of playing GEtiT could be even higher than using traditional learning methods.
8.1. Effectiveness and Efficiency
Interestingly, the learning outcome was not affected by the difference in the audiovisual encoding tested in phase 1 and the debriefing system provided in phase 2. The lack of an effect due to the audiovisual encoding is explainable by the fact that the two tested versions were only different in respect to the used AT card colors and sound effects. The overall gameplay and application of the AT knowledge remained the same. Participants potentially were only focused on finding the correct solution to the spatial puzzles without paying attention to the audiovisual realization of the knowledge application. Hence, the learning effect is mainly caused by the frequent application of the AT knowledge independent of the application’s enhanced audiovisual mediation. However, the lack of an increased learning outcome caused by the implementation of the debriefing system is surprising. The reason for this could be an issue with the realization of the debriefing system. Instead of only focusing on the mathematical equation, the screen also provides information about the overall gameplay-related performance. This additional information might have distracted learners from the actual learning content. The participants could also have been in a strong state of flow and hence immediately continued to the next spatial puzzle without analyzing the debriefing screen. A solution would be to directly display and to update the composite mathematical equation during the gameplay. As a result, learners would then be able to directly connect their gameplay actions with the changes in the mathematical equation. Also, separating the mathematical equations from the gameplay information in the debriefing screen could improve its effectiveness.
Therefore, H1 and H2 have to be rejected as no significant difference in the learning outcome was found.
Despite not being significantly different, the results indicate a tendency that GEtiT VR has a lower learning outcome in contrast to GEtiT's desktop version. This tendency is explainable by the significantly lower number of solved levels in the vr-Group. Despite having invested the same amount of time, the vr-Group was not able to complete as many spatial puzzles as the desktop-Group. A reason for this could be the complex interaction technique on higher difficulties. Instead of merely completing a matrix using mouse and keyboard, GEtiT VR requires the usage of both HTC Vive controllers to define an AT card. GEtiT VR’s learning outcome could potentially be improved by finding a more efficient input method for the direct value configuration screen. Thus, the analysis of GEtiT VR’s efficiency has not only confirmed an issue with the realization of the AT card GM in VR [78], but also revealed the importance of a high efficiency for serious games. This is a critical insight for developers and educators. It demonstrates that differences in the efficiency, i.e., an important usability factor, have a direct influence on the achieved learning outcome. In this way, it is of high importance to check for all usability factors during the development of a serious game. Overall, this leads to the outcome that both GEtiT versions cannot directly be compared in respect to their learning effectiveness. Also, it is not possible to draw generalizable insights about the effectiveness of VR technology for an AT knowledge learning based on this study’s results. Despite these limitations, the study indicated that using GEtiT VR leads to a successful training transfer and successfully demonstrated that the Gamified Knowledge Encoding is also valid for VR serious games. This is a valuable insight for scientists, game designers, and educators aiming at the development of serious games targeting HMD-VR.
Thus, H3 cannot be verified as both GEtiT versions ultimately were too different to be directly compared in regard to their learning outcome.
8.2. Learning Quality
The learning quality analysis validates the concept of developing GEtiT to achieve a higher learning quality when practicing the complex and abstract ATs. In this way, the present study also validates GEtiT’s design as well as playability. Although no significant difference was found in the learning quality subcategory between the tested learning methods, the results indicate a clear trend that GEtiT and GEtiT VR achieve a higher learning quality. This outcome is critical as all participants had to invest the same amount of time but felt more engaged when using the serious game. As a result, GEtiT not only yields effective knowledge learning, but also achieves a higher learning quality thus indicating its overall effectiveness. The results also align with previous research [45, 46] by showing the highest learning quality rating in the vr-Group. In this way, the user study confirms that using VR technology can be beneficial for the overall learning quality of a serious game. This result is supported by the behavior of the participants. Except for the vr-Group, all other conditions showed some drop outs. The vr-Group, however, even reported to have experienced a strong intrinsic motivation to attend every session, thus confirming the measured high learning quality.
The system-specific motivational aspects subcategory revealed that all tested GEtiT versions were perceived as an engaging and motivating learning method. Interestingly, GEtiT VR showed no trend to yield a higher motivation than the desktop version. This outcome is explainable by a habituation effect. Instead of playing GEtiT VR for a single learning session only, the participants used the system over the course of 4 weeks. As a result, the initial motivational benefit of providing an immersive VR version might have ceased over time. Interestingly, the implementation of an achievement system had no impact on the motivational aspects subcategory despite being rated as somewhat motivating. This could be a result of the general functionality of an achievement system. It rewards progress milestones but provides no constant feedback like the point system.
The specific questions targeting the audiovisual encoding tested in phase 1 revealed that the visual presentation of the core player-bound GMs requiring the knowledge application has only a limited effect on the perceived learning effect. This aligns with the assumptions drawn from the effectiveness measurement results. The results also show that acoustic effects are of lower priority when designing a serious game. This insight is important for designers who need to prioritize their development goals.
Finally, the perceived learning effect of the debriefing system was seen as helpful but not as a critical element relevant for knowledge learning. This aligns with the finding that the debriefing GM had no effect on the overall learning outcome.
9. Conclusion and Future Work
This paper presents two versions of GEtiT targeting a transfer-oriented learning of ATs. Both versions of the game implement the same core GMs to encode the learning content but use either desktop-3D or immersive VR to visualize the gameplay. In addition, a comprehensive presentation of the Gamified Knowledge Encoding is given for the first time. The two GEtiT versions were compared to a traditional paper-based learning method in regard to the learning outcome and learning quality. Also, the two versions were compared in respect to their efficiency. Lastly, this paper evaluates the effects of a debriefing system and of two different audiovisual encodings, i.e., reduced and enriched, of the learning content on the overall learning outcome and the perceived learning effects.
The results of the present study show that encoding and presenting complex knowledge using GMs leads to an effective transfer-oriented knowledge learning. Thus, the results validate the design of GEtiT and the underlying framework of the Gamified Knowledge Encoding. The effectiveness of the learning, however, was not affected by the audiovisual presentation or the provision of a debriefing system. Also, while showing VR technology being beneficial for the learning quality, the study revealed a flaw in GEtiT VR’s design negatively affecting its efficiency. Hence, no conclusions can be drawn from the comparison of the learning effectiveness of both versions. However, the study indicated a higher learning quality for the VR version. This is a critical insight for the ongoing research of VR-based education and an important finding for game designers who like to create effective serious games.
Future work needs to be aimed at further evaluations of the knowledge encoding in GMs as proposed with the Gamified Knowledge Encoding. Also, new methods to realize the AT card GM in GEtiT VR need to be implemented and tested. This would allow for a comparison of the different visualization techniques and potentially reveal new insights about knowledge learning in immersive VR. Finally, instead of assessing the learning outcome with a paper-based exam only, the measurement could additionally be performed inside of GEtiT. This would allow for a more in-depth analysis of its training transfer.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
We like to thank Mario Reutter and Jonathan Stahl for developing the first prototype of GEtiT. Also, we like to thank David Heidrich for his work in overhauling GEtiT, changing its visual style, and developing GEtiT VR. This publication was funded by the German Research Foundation (DFG) and the University of Würzburg in the funding programme Open Access Publishing.
Supplementary Materials
GEtiT - Gameplay shows the encoded AT operations and their visual effects. The video also provides an overview of GEtiT’s gameplay and of the knowledge learning process. GEtiT - Difficulties demonstrates the four different difficulty levels and shows how the AT knowledge is moderated and mediated. GEtiT VR showcases the specific VR version by showing the main menu and the successful completion of an easy as well as a hard spatial puzzle. (Supplementary Materials)