Can serious games measure your cognitive profile in adults? An innovative proposal to evaluate and stimulate cognitive skills

Since their commercial breakthrough in the 1980s, video games have evolved, both in general and serious games in particular, gaining popularity to become one of the main entertainment options for children, young people, and adults, and displacing other, more classic forms of entertainment such as movies and music. According to data from the ESA (Entertainment Software Association) [1], 65% of American adults play serious games, more than other more creative hobbies such as drawing, writing or painting (52%), or playing a musical instrument (32%). The Spanish Video Game Association (AEVI) [2] annual report on the video game industry indicated that the number of video game players in Spain has increased year on year to more than 17 million in 2018 (one million more than the previous year), while frequency of play was estimated to be around 6 h on average per week per player [2]. Moreover, it is important to highlight that the purpose of serious games in the last few years has evolved; video games are no longer used only for personal interest and entertainment, but also for educational objectives [3].

The growth in the games industry is not stopping. It is an industry that is increasing in value every year, and other disciplines have started to emerge around it, such as electronic sports—commonly known as eSports—a phenomenon that is garnering increasing social, cultural, economic and scientific interest [4, 5].

These data reflect the increasing social and cultural impact of serious games on today's society, and justify why thousands of professionals are investigating alternative ways to use serious games—such as encouraging people to learn, increasing motivation in doing an activity, or demonstrating talent in a job selection test. One common example in this area is the serious game America's Army, a military simulation video game whose objective was to promote the US army and serve as a recruitment tool [6]. In addition, gamified applications such as Duolingo, which uses the elements of game design applied to language learning, have resulted in millions of people around the world learning through their mobile devices, breaking time and geographical barriers [7], or playing popular learning-based games such as Minecraft [8], Assassin's Creed [9] and Fortnite [10].

Although some studies have focussed on the negative effects of playing video games—such as addiction, violence and depression [11, 12]—in recent years, researchers have increasingly highlighted the possible benefits of playing video at the cognitive [13], motivational [14], emotional [15], and social level [16] in many different populations.

Unlike more traditional tools, serious games allow information and content to be presented in a wide variety of formats, which increases the chances that relevant information will be addressed [17]. Likewise, serious games also encourage the organisation of information through networks of interconnected concepts, just as our brains process information, which facilitates the rapid activation of content [18]. They also involve some dynamic behavioural factors that are capable of arousing interest, attracting attention, and inducing motivation, such as playful characters, progressive levels of difficulty, competitive situations and rewards [19].

Given the potential and benefits serious games may offer education and psychology, this study aims to analyse the viability of using serious games as tools to determine cognitive profiles. This would then allow teachers, psychologists, researchers and human resource managers to identify skills in a simple, fun and much more easily automated way than is currently available.

Within this context, it is worth noting that the measurement of intelligence and cognitive skills has aroused great interest throughout history because of its ability to predict work performance [20], academic performance [21], health [22] and socioeconomic status [23]. Nowadays, psychometric tests that analyse cognitive abilities are a continuing part of the day-to-day work of education and psychology professionals, who apply these tests to identify and inform interventions for learning difficulties, and to aid personnel selection.

However, despite the constant interest and numerous studies on the potential of using more progressive methods, most of the standardised tests used in the evaluation of cognitive skills remain based on traditional tools, and pencil and paper tests, such as the BADYG batteries [24] and the Wechsler scales [25]. These continue to be successful so, despite the potential of those more progressive methods, it is still difficult for educators, psychologists, and researchers to find tests that enable them to evaluate human skills and cognitive ability in ways that are simpler, more dynamic and more appealing for both evaluator and participant.

Serious games have definite potential as tools for evaluating and identifying cognitive profiles. An evaluation that is conducted by having the participant play a serious game allows evaluation objectives to be incorporated while also entertaining the participant, using an innovative and significant methodology [26]. The gameplay can also be recorded—without bias—and results can be automatically produced from the interactions between the participant and the evaluation tool.

Some success has already been had using so-called brain training video games, with positive results in the identification and improvement of basic cognitive abilities, such as processing speed, working memory, attention and reasoning [27]. However, although “brain-training” games have proven to be valuable, they take a very unitary vision of intelligence, and ignore other, more pluralistic approaches that have emerged during recent decades.

New conceptions of intelligence and the mind now exist that are more open and dynamic, and that emphasise a multiplicity of capacities and processes, such as the theories of multiple intelligences [28], emotional intelligence [29] and Sternberg’s Triarchic Theory of Intelligence [30]. Of these pluralistic theories, the theory of multiple intelligences (MI), developed by Howard Gardner and his team in the 1980s [28, 31, 32] is of particular interest. This theory revolutionised the definition of intelligence by considering the concept not as something unique, but as a set of skills, talents or abilities, called intelligences, which were independent from each other and potentially present in all people [32]. According to these ideas, Gardner [31] defined intelligence as "a biopsychological potential to process information that can be activated in a cultural framework to solve problems or create products that have value for a culture" (p. 52).

Gardner [28] initially identified seven intelligences: verbal-linguistic, bodily kinaesthetic, logical-mathematical, musical-rhythmic, visual-spatial, interpersonal and intrapersonal. Later, naturalistic intelligence was incorporated into the theory, establishing a total of eight intelligences. Each individual develops and combines them in different ways, shaping a unique profile and, although we are all born with them, none of us has exactly the same intelligences in quite the same combination [31].

This theory has generated great interest in the fields of psychology and education because it constitutes an interesting model of personal development; knowing a person’s profile of intelligences offers the chance to create experiences that will develop strengths, intervene to support the development of weaker areas and activate abilities that may be dormant.

Some tests have evolved to include content that recognises a greater variety of abilities, such as the inclusion of fluid reasoning skills, image concepts, and routine aspects of short-term memory [33] in the latest editions of the Wechsler scales [25]. However, it is still difficult to find instruments that support realistic assessment of performance in different skills, tasks and experiences, while at the same time manage to be intrinsically motivating for the participant, provide feedback on performance, and automatically record responses for further analysis and processing.

Incorporating Gardner’s ideas [31] points towards a change that would allow the development of a different, potentially better way to conceptualise the human intellect. Serious games can offer this, as a different method of evaluation, because game designers can create opportunities for players to acquire knowledge, social support and self-efficacy, by incorporating the various intelligences into clues, puzzles and game challenges [26].

The new, pluralistic views of the mind, combined with the emergence of new digital tools (capable of measuring skills dynamically and differently), has led to the idea that an evaluation of cognitive profiles utilising serious games is possible.

Therefore, aware of the need for evaluation tools that cover an increasing number of skills and the growing interest in such tools in education and psychology, as well as the cultural impact of video games on today's society, this study aims to analyse the possibilities of using a serious game called Cutie Cuis—based on the theory of multiple intelligences—as a tool for assessing cognitive skills and identifying cognitive profiles in a sample of adults. To that end, Latent Profile Analysis (LPA) will be used to determine the classification accuracy of profiles produced by this serious game through analysis and comparison with profiles obtained using a self-report MI questionnaire.

A sample of 209 adults from Northern Spain (Principality of Asturias) aged between 19 and 59 years old (M = 22.83, SD = 6.36) took part in the study. The participants were selected by convenience sampling. Of the total sample, 22.5% were men, with an average age of 24.04 (SD = 7.83), and the remaining 77.5% were women, with an average age of 22.48 (SD = 6.73). There were no statistically significant differences in the distribution of the genders in the sample (p = 0.161). The educational attainment of the sample was as follows: 4 participants (1.9%) had completed compulsory secondary education; 58 (27.8%) had finished post-compulsory secondary education; 20 (9.6%) had done vocational training; 112 (53.6%) studied at university; 7 (3.3%) had done a Master’s degree; while 8 (3.8%) did not specify their educational attainment.

Two evaluation instruments were used to gather information about the sample’s cognitive profiles: a digital tool based on serious games and a self-report MI questionnaire for adults.

This study was conducted in accordance with the Declaration of Helsinki and the World Medical Association [38], which establishes the ethical principles of research with humans. Approval was obtained from the ethics committee of the Principality of Asturias (reference: CPMP/ICH/70/19, code: vRTI_Learning). The participants volunteered for the study and anonymity was ensured.

Before administering the serious game, an initial meeting with possible participants was organised, in which they were informed about the main aim of the project, the description of the games contained in the application, the duration of the tasks and the anonymity of the results. This opportunity was also used to answer any questions they may have had about the project. At the end of the meeting, participants who agreed to participate in the project signed their informed consent.

The administration of the Cutie Cuis application and self-report questionnaire took 2 h and participants used their own mobile phones to play the TOI software video games and complete the self-report. Once the project was completed, all participants received a detailed report containing the results from both the serious game and the self-report. Thus, each participant received feedback informing them of their cognitive profiles, as obtained by both assessment tools (serious game and self-report) and based on the Theory of Multiple Intelligences [31].

To obtain the different cognitive profiles of the participants, a series of latent profile analyses were performed in several phases. First, descriptive statistics corresponding to the serious game and self-report questionnaire were calculated and analysed. Secondly, several Latent Profile Analysis (LPA) was performed [39], both for the data obtained through the questionnaire and for those obtained in the video game, through tablets. The variables used for profile analysis were the multiple intelligences (verbal, logical, visual, kinaesthetic, musical and naturalistic in the case of serious games, and including emotional in the case of self-report). MPlus, version 7.11 [40] was used to determine the best fit to the data from a finite set of models, adding successive latent classes to the target model. Third, the model with the optimal number of classes was determined by considering the Akaike information criterion (AIC), the Schwarz Bayesian information criterion (BIC), the sample size adjusted BIC (SSA-BIC), the formal test of the adjusted maximum likelihood ratio by Lo, Mendell and Rubin [41] (LMRT), and the sample size for each subgroup. The p value associated with the LMRT indicates whether the solution with more or fewer classes is the best fit to the data. The AIC, BIC and SSA-BIC indices are descriptive, with lower values indicating better model fit. Also, classes containing less than 5% of the sample are considered spurious, a condition indicative of excessive profiling [42], although it is possible that a class with less than 5% of the subjects makes theoretical sense. Finally, once the best-fitting model was selected, to determine the ranking accuracy of the selected model, the ex post probabilities and the entropy statistic were taken into account (entropy ranges from zero to one, with values closer to one indicating higher ranking accuracy).

Several latent profile models were analysed for each of the applications: serious game (two- to five-class models) and self-report (two- to seven-class models). All models were specified assuming that variances could differ between indicators within each group, but they were constrained to be equal between groups. Also, independence between indicators, both within and between groups, was imposed as a constraint.

Table 3 shows the results of the model fit for the two applications. For both applications, the process was stopped when indicated by some of the model comparison indices. Specifically, model fit was stopped at five- (serious game) and seven- (self-report) classes, since the BIC values for the five- and seven-class models were higher than for the respective four or six class models. Nonetheless, in both cases, according to the values of the SRML statistician, the five- and seven-class models were not statistically better than the four- and six-class models. In addition, the value of entropy was high in both of these models. Therefore, taking into account the heuristic value of parsimony [44], the four- and six-class models were chosen as the most suitable for the serious game and self-reporting applications, respectively.

The classification accuracy of the two models (serious game = four classes, self-report = six classes) was analysed based on the value of entropy and the probability coefficients a posteriori. Both models exhibited good values for entropy (serious game four classes = 0.85; self-report six classes = 0.84). Table 4 shows the classification accuracy of the four-class (serious game) and six-class (self-report) models, as well as the number of subjects in each class, both in absolute (N) and relative (%) terms. Table 4 shows the posteriori probability coefficient of each participant belonging to a given class. The coefficients associated with the groups to which the participants have been assigned are shown in the main diagonal of the table for both models. The coefficients of the two diagonals (in bold) were all around 0.9, which is indicative of very good classification accuracy and, by extension, a good level of confidence that the participants belong to the assigned profile.

Table 5 shows the average scores of the participants belonging to the multiple-intelligence profiles resulting from the serious game four class model and the self-report six class model (z scores and raw scores). Figures 1 and 2 show graphical representations of these profiles.

The best-fitting model for the profiles for the serious game application (Fig. 1) distinguished four latent classes. The profile with most member was a moderate level multiple intelligence profile (Profile 2 = 58.3%). There was also a high level multiple intelligence profile (Profile 3 = 23.9%), a group of participants with a low level multiple intelligence profile (Profile 1 = 13.9%), and another small group with a very low level multiple intelligence profile (Profile 4 = 3.9%).

The best-fitting model for the self-report application (including EQ) had six classes or highly differentiated profiles (Fig. 2). The most populous (51.7%) was a moderate level multiple intelligence profile (Profile 2). That was followed by a low level multiple intelligence profile (Profile 4 = 15.3%) and a high level multiple intelligence profile (Profile 6 = 14.4%), with musical and logical intelligences standing out in this profile for their comparatively low levels (Fig. 2). There was also a small group of participants with very low profiles in multiple intelligences (Profile 3 = 3.8%), mainly in the kinaesthetic, naturalist, emotional and verbal intelligences. Finally, there were two similar-sized groups with two different profiles, resulting from the combination of different levels of multiple intelligences. Specifically, one profile had low verbal, logical and emotional intelligences, but high visual and kinaesthetic intelligences (Profile 1 = 7.6%), while the other profile had high levels of logical and verbal intelligences, but low levels of visual, kinaesthetic and musical intelligences (Profile 5 = 7.2%).

As Fig. 1 (serious game) shows, Profiles 1 and 4 (Low and Very Low profiles, respectively) indicate the existence of two groups that are very clearly differentiated in terms of which intelligences the participants score low on. Participants in the Low profile group (Profile 1) had notably lower scores in logical, verbal and musical intelligences—areas traditionally related to formal education and academia, while participants in the Very Low profile group (Profile 4) scored lower in kinaesthetic, naturalistic and visual intelligences—areas generally related to informal knowledge or curiosity. This pattern suggests that even when considering participants with low cognitive profiles in MI, there are qualitative differences among them. This means it is important to identify such profiles. With regard to Profiles 2 and 3 (Moderate and High profiles, respectively), it is also worth noting that the kinaesthetic, naturalistic and visual intelligences show the opposite pattern (Moderate but below average in Profile 2, and Very High in Profile 3), meaning that there are again differences between different sub-groups of intelligences.

Figure 2 shows the six profiles obtained from the self-report group; these profiles exhibit more complex qualitative differences. Specifically, Profile 3 shows very low levels and Profile 6 shows very high levels in all of the intelligences evaluated. The lowest scores in Profile 3 are verbal and kinaesthetic intelligences, while in Profile 6, both of those, along with visual intelligence, are at their highest levels. The pattern in Profile 3 (Very Low level) is also found in Profile 4 (Low level), although to a lesser degree (with significantly lower z-scores for Profile 3). In addition, Profiles 1 and 5 (Low/High and High/Low intelligence combinations, respectively) differed substantially between each other. Profile 1 shows high levels in visual and kinaesthetic intelligences and low levels in the others, while Profile 5 shows high levels in verbal and logical intelligences but very low levels in visual and kinaesthetic intelligences and in musical intelligence. Again, visual and kinaesthetic intelligences tend to exhibit similar levels, as do verbal and logical intelligences; sometimes these groups are both high or both low, but in some cases one group is low while the other is high. Additionally, musical intelligence tends to group with the naturalistic and emotional intelligences, with the only exception in Profile 5. Finally, Profile 2 (Moderate level) shows average levels in all of the seven intelligences evaluated.