Usability Evaluation of Imikode Virtual Reality Game to Facilitate Learning of Object-Oriented Programming

Programming is the act of writing computer programs to solve a particular task. There is no gainsaying that computer programs have taken over the way things are done in today’s world. Hence, it is essential to leverage on this technology to students in tertiary institutions. For students at various universities to acquire the basic programming knowledge and skills, teaching and learning introductory programming becomes indispensable (Divna et al., 2015). According to Divna and colleagues, despite this lofty idea of catching them young by instilling programming skills early enough at the university level, students often find this subject very challenging. In addition, these courses titled introductory programming are often concealed under the title “computer science” in the literature (Radenski, 2006).

According to Jens et al. (2008), students' programming abilities are essentially a function of their mathematical prowess; hence, teachers leverage this method to develop a curriculum that will benefit students undertaking mathematical studies, while other areas such as problem solving skills are ignored in contrast to extant literature on the enormous benefits of programming in developing problem solving skills (Bile, 2022; Brown et al., 2009; De Freitas, 2018; Erhel & Jamet, 2019; Papastergiou, 2008; Peters et al., 2021; Pierre et al., 2020). A study by Divna et al. (2015) conducted a correlation analysis to unravel the relationship between mathematical abilities and students' programming prowess. However, the research found a significant relationship between the success of students in both mathematical and programming courses. Thus, if a student fails a mathematics subject, that student has a high probability of failing programming courses. They concluded by advocating a close examination of the programming curriculum to determine if the presented concepts and assignments favor a mathematical way of thinking. Ten factors affecting student performance in programming courses were emphasized by Hawi (2010).

According to Hawi (2010), some of the factors which became evident as a result of the interviews and observation based on their performance in a programming course included: “cheating”, “teaching method”, “lack of study”, “exam anxiety”, “lack of practice” and “learning strategy”. Some of these factors were also highlighted by Jens et al. (2008) in their research to predict students' success in a programming class. A study conducted by Sunday et al. (2020) used a classification method, such as the decision tree algorithm to analyze the performance of students in a programming class. Specifically, they used the ID3 & J48 classifier to analyze 239 instances in a programming course at Usmanu Danfodiyo University, Sokoto, Nigeria. The results of their study showed that students' massive failure rate in introductory programming courses was due to the low rate of class attendance by the students and the lack of infrastructure to support a large number of students in a class. Similarly, Dasuki and Quaye (2016) examined the factors responsible for students' massive failure in introductory programming in a Nigerian institution. According to Dasuki and colleagues, poor lecturer skills and behavior, peer influences, anxiety, lack of future expectations, and lack of intrinsic motivation are the primary factors contributing to the failure rate in programming.

To support and ease learning of programming, several VR tools/systems have been proposed/ developed to help students learn effectively. Table 1 shows a summary of the software that support learning programming.

According to usability.gov (2020), usability measures a user’s experience when interacting with products, systems, software, devices, etc. It measures the effectiveness or efficiency of the product and the overall user satisfaction. It also measures the intuitiveness of the error frequency, severity, design, efficiency of use, satisfaction, and ease of learning design. Therefore, this research aims to evaluate a VR game for learning programming to measure students' usability and satisfaction. Studies such as Mikropoulos and Natsis (2011) have concluded that 3D spaces (including VR) bring some benefits to the increasingly complex problem such as the ability to learn by experience, fostering collaborative learning and the willingness to tackle problems. This has enabled virtual reality applications to be well grounded in education, including teaching programming courses. 3D applications are often designed to support VR in education and enable the user to navigate the environment seamlessly (Segura et al., 2019). However, measuring this technology's effect, particularly for teaching programming, would significantly improve the learning process. Sutcliffe and Kaur (2000) evaluated the usability of the Virtual Reality user interfaces using the theory of interaction. However, this study does not consider the perspective of the users themselves. Another study by De Pace et al. (2019) measured the VR game interface's usability using a hybrid approach in which the first player interacts with AR while the second player interacts with VR. The manner in which users navigate through the process determines the usability. According to the authors, it is challenging to measure usability in this manner. Nonetheless, it is a unique approach that has been invented. In this work, another dimension had been taken to evaluate the usability by involving users. As the users are the system's ultimate users, their evaluation is better for determining the VR/AR system's usability.

At the Imikode application's instantiation, the virtual character “Bob” introduces the game environment and its objective to the player. This character guides the player throughout the game play in a conversational manner through a generated text and sound. A talking agent (Bob) teaches the players on what they will be learning and practicing within a particular time frame. For instance, the talking agent could inform the players on the need to learn classes and objects after which the players append the bubbled instructions on the whiteboard to show their level of assimilation. Bob guides the player to create objects in the virtual environment by using the knowledge of variable declaration and assignment statements. Furthermore, the virtual character guides the player in the area of method invocation using the setters and getters functions including method overriding, overloading and the artificial intelligence component that allows learners to receive intelligent and more interpretative feedback in terms of errors. If a player was able to append the write syntax of a portion of the code correctly, the virtual character responds with a smile pointing to the direction of the created object. For a function however, Bob responds by pointing the user to the created task. If the player, however, fails to append the right code on the whiteboard, Bob frowns at such a player and informs the player to repeat the task. For example, two trees and two houses can be built by assigning a value to a variable (tree = 2; house = 2) similar to the conventional programming languages coding style.

In the VR environment however, this is achieved by clicking and dragging each unit of the variable or value (tree = 2; house = 2) which is available on the bubbled list of the Imikode VR game to the whiteboard. To create an instance of an object such as a fox, the bubble command that a player will select and append to the whiteboard is fox = new Fox(). Each time a player selects the instruction correctly, an immediate execution is visible to the player to understand how this OOP concept behaves. If an incorrect instruction is selected, Bob informs the player to retry. The method calls to implement object behaviors are modeled in the Imikode OOP concept. For instance, to cause the fox object to walk, a method called fox.walk() must be selected and appended by the player. When this is done correctly, the player then sees the object fox walking in the simulated environment. If the player selects and appends the method fox.die() in the virtual space, the object fox immediately drops dead and after some time it leaves the space through garbage collection methods. It is worthy to note here that a method call precedes object instantiation and this must be taken into consideration when invoking methods. The detailed procedure of the research is given in Sect. 4.3.

Figure 1 presents a screenshot of a player creating trees, houses, and the instantiation of fox objects as explained above. The left part of Fig. 1 shows how the player creates these objects, while the right screen shows the output of executing each line of instruction.

More advanced OOP concepts and their applications are shown in Fig. 2. For instance, how to use the getter and setter methods to create variables and assign them values through a function with arguments is depicted in Fig. 2.

Diagnostic messages emanating from compilers and interpreters of programming languages help learners understand errors, warnings and other types of messages. Unfortunately, research has shown that compilers' error messages can be unclear to learners, especially novices (Becker et al., 2019). This problem motivated the Imikode VR game's improvement to include an artificial intelligence component that allows learners to receive more interpretive and intelligent feedback in terms of errors (see Fig. 3). Aside from the AI component in the improved Imikode, multiple game play levels can be dynamically added to give learners more opportunities to explore the VR environment for immersive learning experiences. In this improved Imikode VR game, users learn more as a result of more inbuilt functions and OOP concepts such as constructors, instant display of scores and instantaneous error presentation. If an error is committed by the player when instantiating objects or during method invocation, immediate feedback is provided to such players through the artificial intelligence component of the game.

Figure 3 shows a talking agent telling the players what they will learn.

Table 2 presents the objectives and the expected outcome of the Imikode VR game.

In this research, the USE questionnaire containing 30 items was utilized to collect information about students’ experiences of playing the VR game and its impact in enhancing the learning process of object introductory programming. The USE questionnaire administered in this research is a seven-point Likert scale consisting of four (4) variables: usefulness, ease of use, ease of learning, and satisfaction, as presented in Table 3.

There is also an option “NA” (not applicable) in this questionnaire for items that the students feel is not appropriate. The outline of the questionnaire is illustrated in Appendix I. Aside from the 30 questions; there is an open-ended option for the participants to provide comments regarding their VR game experience. Their comments in this part were used to canvass the collected quantitative data.

To examine the factor structure of our scale and the structural validity of the scale, both exploratory factor analysis (EFA) were performed (see Table 4) and confirmatory factor analysis (see Figs. 5 and 6). Furthermore and in line with Fornell and Larcker (1981) and Hair et al. (2006), composite reliability (CR) and average variance extracted (AVE) were conducted to further examine the validity of the scale (see Table 4).

From this table, we can observe that the KMO is greater than 0.5, which shows that there is no sample problem. Bartlett’s test shows that the test is significant, which implies that the variables are correlated. The initial factor loading is shown in Fig. 5.

In Fig. 5, the questions in the USE questionnaire were utilized to determine whether they would load in their respective factors. The figure depicts the initial factor loading of the individual questions and variables, and it was discovered that some factors were less than the threshold value of 0.7. These factors were dropped because the threshold for every item loading has been reached which is 0.7 (see Table 5) which is the reliability test of Cronbach’s alpha which explains the relationship between latent variables and manifest indicators (Hair et al., 2011; Witjaksono & Saputra, 2019). Table 5 represents the factor loading of constructs and their item measurements.

In this table, all the items loaded under their latent variables conformed to the threshold of 0.7. The final factor loading is shown in Fig. 6.

In Fig. 6, some factors were dropped because of their threshold values. The model quality metrics comprising cronbach’s alpha, effect size, f, composite reliability and average variance extracted were computed and presented in Table 6.

Table 6 shows the quality criteria for latent variables. It can be seen from the table that the Cronbach’s alpha values reached the threshold of 0.7, hence providing an acceptable level of reliability for the research instrument. Furthermore, the rho_A and composite reliability also satisfies the threshold of 0.7, signifying a valid scale for the instrument while the average variance extracted exceeds the benchmark of 0.5 which further validates the scale. The effect size was computed, f, to determine the level of intervention a variable has over another variable (Cohen, 1988). According to Cohen, the effect size f is low if f ≤ 0.2 and it is high if f > 0.5. According to the table, there exists a weak effect between usefulness, ease of use and satisfaction with the threshold (f ≤ 0.2) while there exists a large effect between ease of learning and satisfaction with the threshold (f > 0.50). The implication of this is that the ease at which the students found the Imikode virtual reality game in learning object oriented programming gave them more satisfaction compared to other constructs.

The bar chart depicting the correlation values of the research instrument in Fig. 7 is represented below.

In Fig. 7, all the correlation values for each item (Q1-Q30) were presented, and it was discovered that the values exceeded the r-value (0.25) at the level of 0.05 significance. Therefore, each correlated value is considered valid, satisfying the criteria. Hence, this instrument is useful for the evaluation of educational virtual reality games.

Participants in this study included undergraduate students in computer science studying introductory programming courses in their first year of studies at Usmanu Danfodiyo University, Department of Computer Science, Sokoto, Nigeria. The students had fundamental knowledge of computer studies from secondary school education. The participants comprising 103 undergraduate students (80 males and 23 females) between the ages of 16–20 years evaluated the VR game using the USE questionnaire (see Appendix I). The participants willingly availed themselves to be recruited for the study in order to experience the power of virtual reality technology in programming education.

The VR application was installed on a mobile device and then connected to a Google cardboard headset which was presented to all participants in this study to learn basic OOP concepts. In the first instance, a 30-min presentation of content using the traditional lecture method was presented to the students on virtual reality technology and basic OOP concepts such as objects, methods (behaviors), setters and getters methods, method arguments, and garbage collection to destroy unused objects. The role of VR in learning, specifically, how to use VR technologies to learn the OOP concept, was also discussed. Furthermore, the students were guided on how to insert the mobile device on the Google headset and launch the installed VR game. During the lecture, a presentation was conducted on how to use the VR headset to walk and navigate various scenes in the simulated environment. Finally, the students were admonished on the need to provide honest feedback when completing the questionnaire. During this participatory and discussion method, teachers and students asked questions and discussed the concept of VR for teaching and learning. Some of the questions from the students included “what will I see in the simulated world?”, “do I keep holding the VR headset while navigating the simulated environment?” “Is the simulated environment different from the real-world environment?” and finally, “what happens if something happens to me in the virtual environment?” The lecturer answered each of the students' questions to their satisfaction. In this question and discussion session, which took about 45 min, the teacher responded to the first question by informing the students that what they would see in the simulated world represents the real world.

For the second question, the lecturer told the students that it is not necessary to keep holding the headset as the Google cardboard headset was designed to fit appropriately to the position the user feels like placing it during the learning process. For the third question, the lecturer's response was that the simulated environment is a representation of the real-world environment and that they are not the same in both theory and practice. Finally, to the last question, the lecturer informed the students that since the virtual environment creates an illusion of the real-world environment, anything that happens to an individual in that environment does not affect such individuals in the real world. The learning process lasted for one (1) week, and each student played the Imikode VR game for 30 min. After each student completed the learning process, the USE questionnaire, which is a paper-based 7-point Likert scale instrument, was immediately administered to the student. Filling the questionnaire by each student, however, took approximately 10 to 15 min. It should be noted that the data were collected using the convenience sampling method because the students under consideration willingly availed themselves to participate in this research. In addition, the students were within the reach of the researchers, implying that the researchers experienced no difficulty bringing them to the fore to participate in the study.

The students were instructed to carefully read the questions and select just one option from seven options for each question that applies to them. The participants answered all 30 questions, including the questionnaire's open-ended questions, and their responses were collected immediately. However, it is worth noting that each response was anonymous and honest based on the actual utilization of the VR game in enhancing their academic performance concerning programming. The procedure of the research was illustrated in Fig. 8 and the different scenes of the learning process in Fig. 9a–d.

In Fig. 9a, the students demonstrated the utilization of the VR game inside the classroom; in this case, they wore the Google cardboard headset while holding the controller. As this happens to be the first time such a device is inserted, the students were busy viewing the simulated world and the beautiful environment. In addition, the students were also learning about the various features of the VR game and how to go about playing it.

Two different sets of students played the VR game in the computer laboratory as depicted in Fig. 6b; this time around, the students were busy creating objects and invoking methods to boost their programming skills. However, at this stage of the learning process, the students had already mastered navigating the virtual environment and various VR game features.

In Fig. 9c and d, however, the students used the Google headset to play the VR game directly facing other students in the classroom. Thus, showing all the students in the class how to put on the Google Cardboard headset, how to navigate in the virtual world, and finally, how to move in the simulated environment.

IBM SPSS statistics version 21 (IBM Corp., 2012) and Smart PLS applications for Windows operating systems were used for analyzing the quantitative data. The gleaned data was coded according to the construct categories into the SPSS and Smart PLS applications. In a bid to foster high quality data, three of the co-authors checked the data in their various computers in order to ensure that the integrity and consistency of the data is realized across platforms. Descriptive statistics, Analysis of variance (ANOVA) and multiple linear regression analysis were employed including the f & t test investigations on the data using these tools. The students’ responses were analyzed on the feedback form of the USE Instrument based on qualitative content analysis and the procedure outlined by Mayring (2014) were followed on content structuring and thematic analysis. In this case, the open-ended feedback forms were read on several instances in order to understand and classify each student’s responses based on some themes and in line with the research questions. Furthermore, duplicity of themes among the feedback forms were realized and were categorized according to the usefulness, ease of use, ease of learning & satisfaction constructs respectively. Some responses which did not fall nicely into these constructs were realized because it seems more of a recommendation and or drawbacks; hence, it was profiled and classified such responses under the recommendation and drawbacks theme of the VR tool. It was discovered from the analysis that out of the 103 students that responded to the open-ended feedback form, 10 students either forgot or deliberately decided not to answer the open ended questions in the USE questionnaire. The 93 remaining students offered various comments ranging from some positive comments (88) to providing a recommendation (1) while others provided some drawbacks of the technology (4). Finally, these themes were carefully selected and then segmented them into separate units (see Table 9).

Before analyzing the students’ responses to the questionnaire using descriptive statistics, a usability measurement score was carried out for Imikode educational virtual reality game. This measurement score represents the degree of acceptance or rejection of the Imikode VR game by the users hence, in this study each variable’s mean value was utilized to describe the usability measurement results, as suggested by Nielsen (1994). Also, each variable's mean score (see Table 7) was utilized by taking the average of the scores and grouping them according to each variable representation on a seven-point Likert scale. However, the direction of responses on this scale shows the level of acceptance or rejection, as put forward by Babbitt and Nystrom (1989). In this case, it was observed that the mean of each variable was within the acceptable range since no value was below 5.0, hence, each value draws closer to the peak of being firmly accepted. This means that as each mean score goes higher after satisfying the acceptance threshold of 5.0, the acceptance rate further increases (see Table 7). In addition, and similar to the work of Marreez et al. (2013) the seven-point Likert rating score were converted to two categories of “binomial data'' of acceptance and rejection according to the participants' responses of agree or disagree, where a score of 5 (somewhat agree), 6 (agree) and 7 (strongly agree) were grouped under the acceptance category. This was similarly done for the reject category, where score ratings of 1 (strongly disagree), 2 (disagree) and 3 (somewhat disagree) were merged and classified into the rejection category. However, the rating score for neutral was excluded from the analysis because it denotes uncertainty or neutrality. The reason the Likert score was converted and harmonized to 0–100 scores was to enable more analysis of the data from a binomial perspective where 0–49 indicates below average and 50–100 indicates acceptable value (Debevc & Bele, 2008). From Table 7, it can be observed that the trends of the constructs as the usefulness, ease of use, ease of learning and satisfaction constructs all exceeded the score value of 50, thus satisfying the criteria of being accepted.

Furthermore, when taken a cursory look at the average score of the four variables the score is 90.94, exceeding 50. Based on these results, it was concluded that the students found the Imikode VR educational game to be very useful for learning object oriented programming. Similarly, from the 0 to 100 score of the USE questionnaire in Table 7, it was discovered that 90.94 percent of the users were satisfied with its use as a means of boosting their programming skills. However, the implication of this is that, for instance, when there is a population of 100 students in the class, over 90 percent of students who play the Imikode VR educational game will be satisfied with it as an excellent means of learning programming. We present the model construct in Fig. 10 which shows the degree of usability of the Imikode VR game.

Furthermore, with regards to the distribution of the scores as illustrated in Table 8, it was observed that the scores were skewed negatively for all the variables, implying that the left tail of the distribution is longer relative to the right tail, and thus not symmetric. In addition, none of the students responded with N/A. However, this could be explained because the students were guided correctly on the different aspects of the USE questionnaire, including learning with the Imikode VR game.

This section explores multiple linear regression analysis assumptions: testing for normality test, checking for multicollinearity, and testing heteroscedasticity. Whenever multiple linear regression is used, it is essential to check if the assumption is satisfied. This study used multiple linear regression to examine the relationship between ease of use, satisfaction, ease of learning, and Imikode educational virtual reality game usefulness.

Multiple linear regression was conducted to show the linear relationship between students’ satisfaction with ease of use, ease of learning, and usefulness of educational VR games. Hariyanto et al. (2020) stated that the purpose of using multiple linear regression analysis is to understand the nexus between two or more independent variables and one dependent variable. Hence, in this research, the f-test distribution was explored to understand the ease of use, ease of learning, and usefulness variables. Likewise, the partial t-test distribution was conducted to establish each of the independent variables' contributions to the students' satisfaction using VR tool for learning.

The F-test is used to show the contributions of the ease of use, ease of learning and usefulness attributes of the Imikode educational VR game on students' satisfaction. Analysis of variance (ANOVA) were also carried out to determine the effect of these attributes of the VR educational tool on student satisfaction. From Table 12, the F-test showed that the variables were statistically significant (F = 77.508, p < 0.05). Usefulness, ease of use, and ease of learning affect students' satisfaction using the VR tool for educational purposes.

To understand whether there is a partial influence on the variables, a t-test investigation was conducted. Table 13 presents the partial influence of these variables on students' satisfaction using the VR tool for educational purposes. From Table 13, the t-test result showed a significant influence (t = 2.720, p < 0.05) between the usefulness and satisfaction of the students using the VR tool for educational purposes. The students found the Imikode educational VR game very useful, which made them satisfied anytime the game was played.

Furthermore, the t-test showed no significant influence (t = 1.100, p > 0.05) between ease of use and satisfaction of the students using the VR tool for educational purposes. This means that the ease of use of the Imikode VR game has no significant influence on students' satisfaction using the VR tool for educational purposes. Lastly, the t-test showed a significant influence (t = 7.072, p < 0.05) between ease of learning and students' satisfaction using the VR tool for educational purposes. This implies that the students were satisfied with the ease with which they used the Imikode VR tool to learn OOP.

To provide a solution to the first research question, reliability and validity analyses were conducted on the instrument used as a prerequisite for further analysis. The students interacted with the Imikode VR educational virtual reality game using the Google cardboard headset in classroom settings to learn object oriented programming. This is evident from the analyzed result of the USE questionnaire, as the usability measurement score which was converted to binomial data and harmonized to the 0–100 score in order to view the data from another dimension suggests that over 90 percent of the students accepted Imikode VR as another way of enhancing their coding skills.

In addition, from the open-ended feedback gathered from the participants, it was observed that more than 80% of the students alluded to the fact that the VR game enhanced their understanding of OOP which will in turn, lead to increased performance in tests and exams (see Table 9). This further shows that almost all the study participants confessed how usable the Imikode VR game is in enhancing their OOP skills. However, very few students provided some drawbacks while only one gave a recommendation of the VR technology based on their experiences with the virtual reality environment. The overall results of this analysis show that the instrument used in this study satisfied the requirements. Furthermore, it shows that the usability study using this instrument could further be used in subsequent studies as the factor loading, Cronbach’s alpha, composite reliability, effect size, average variance extracted and confirmatory test in Smart PLS using the factor algorithm including a bar chart of the correlation values that was used (see Fig. 7) all satisfied the minimum threshold of been accepted.

To check the correlation among the usability questionnaire attributes, multiple regression analysis was performed to show the relationship between the usefulness, ease of use, ease of learning, and satisfaction of the students using the VR tool for educational purposes. To achieve this, the multiple linear regression requirements derived from the classical assumption model of the tested variables were firstly analyzed. In this model, some tests were performed to ensure that the variables used were i. normally distributed ii. free from multicollinearity and iii. devoid of heteroscedasticity. This is in line with Hair et al. (2009), as conducting this test will lead to more trust in our results. A check if the data were normally distributed were confirmed by using a histogram and data plot and discovered that it satisfies the first assumption. A further check was carried out if the data were free of multicollinearity; the results also showed that the minimum benchmark was satisfied. To satisfy the third assumption, a test for heteroscedasticity with the aid of a scatter plot diagram was performed and it was discovered that it satisfies homoscedasticity as there was no pattern in the data.

Having satisfied the prerequisites of multiple linear regression, the F-test and t-test analyses were conducted. The F-test results show that usefulness, ease of use, and ease of learning affect students' satisfaction using the VR tool for educational purposes (F = 77.508, p < 0.05). There was a considerable correlation between the variables in the USE questionnaire. However, the t-test result shows that the students' satisfaction with the VR tool for educational purposes was influenced by both the usefulness and ease of learning variables (t = 7.072, p < 0.05; t = 2.720, p < 0.05). However, ease of use did not significantly influence students' satisfaction using the VR tool for educational purposes (t = 7.072, p < 0.05).

It should be noted however, that the normality test and homoscedasticity test focused on the satisfaction variable because they represent the dependent variable the authors presented earlier as outlined in the USE questionnaire (see Appendix I). ANOVA and t-test were conducted to determine the influence of the variables of the use questionnaire. The results of the t-test investigation showed no significant influence between the ease of use of the Imkode VR educational tool and students' satisfaction. However, the usefulness, ease of use, and ease of learning in tripartite were combined and analyzed concurrently using the F-test investigation and it was ascertained that there was a significant degree of influence on students' satisfaction.