Entertainment modeling through physiology in physical play

Abstract

This paper is an extension of previous work on capturing and modeling the affective state of entertainment (“fun”) grounded on children's physiological state during physical game play. The goal is to construct, using representative statistics computed from children's physiological signals, an estimator of the degree to which games provided by the playground engage the players. Previous studies have identified the difficulties of isolating elements of physical activity attributed to reported entertainment derived (solely) from heart rate (HR) recordings. In the present article, a survey experiment on a larger scale and a physical activity control experiment for surmounting those difficulties are devised. In these experiments, children's HR, blood volume pulse (BVP) and skin conductance (SC) signals, as well as their expressed preferences of how much “fun” particular game variants are, are obtained using games implemented on the Playware physical interactive playground. Given effective data collection, a set of numerical features is computed from these measurements of the child's physiological state. A comprehensive statistical analysis shows that children's reported entertainment preferences correlate well with specific features of the recorded signals. Preference learning techniques combined with feature set selection methods permit the construction of user models that predict reported entertainment preferences given suitable signal features. The most accurate models are obtained through evolving artificial neural networks and are demonstrated and evaluated on a Playware game and a control task requiring physical activity. The best network is able to correctly match expressed preferences in 69.64% of cases on previously unseen data ($p\text{-value}=0.0022$) and indicates two dissimilar classes of children: those that prefer constantly energetic play of low mental/emotional load; and those that report as fun a dynamic play that involves high mental/emotional load independently of physical effort. The generality of the methodology, its limitations, its usability as a real-time feedback mechanism for entertainment augmentation and as a validation tool are discussed.

Introduction

The principal goal in the reported work is to construct children-user models of a class of game-playing experience during physical play in the Playware playground platform. Specifically, the aim is a model that can predict the children's answers to which variants of a given game are more or less “entertaining” (or “fun,” which is used synonymously in this paper). The word “fun” is used extensively hereafter since it captures best, in our view, children's notion of the term “entertainment” (Read et al., 2002) and is the term used by the children when making their experimental self-reports. This approach is referred to as entertainment modeling. Entertainment generated by a physical game experience is captured through features extracted from the player's physiological state and feature selection is used for choosing appropriate sets of features that successfully predict expressed entertainment preferences. Game-play experiences might very well be identified instead or as well, through features extracted from user-game interaction. Furthermore, game-play behavior could be video recorded and emotions could be recognized by experts or automatically through face gesture detection; however, these approaches are not the focus of this work.

Entertainment is a highly complicated mental state. However, it is correlated with sympathetic arousal (Mandryk et al., 2006) and this can be measured, as reported by researchers in the psychophysiological research field (Zuckerman et al., 2006), using specific physiological signals such as heart rate variability (HRV) and skin conductivity. Although the impact on a subject's physiological state of emotional engagement during computer game playing is well reported in the literature (see Mandryk et al., 2006 among others), there are very few corresponding studies in the physical play domain.

Motivated by the lack of entertainment modeling approaches grounded on a player's physiological state in physical play, the Playware (Lund et al., 2005) physical interactive game platform has been used for recording heart rate (HR) signals of children during play (Yannakakis et al., 2008). In that study the complexity of isolating the HR elements of physical activity from expressed entertainment in physical games were outlined in game experiments with 56 child participants. This complexity was handled, in part, through a carefully designed control experiment of physical activity (Yannakakis et al., 2008).

In the present article entertainment models are constructed using three alternative preference learning techniques (large margin algorithm (LMA), Fiechter and Rogers, 2000, meta-LMA and neuro-evolution) applied to statistical features derived from physiological signals measured during play and children's self-report preference data. The output of the constructed models is a real number y such that more enjoyable games receive higher numerical output and functions as an efficient predictor of reported entertainment preferences given suitable specific physiological signal features. Suitable input feature sets are constructed using two alternative feature selection schemes (n best features selection (nBest) and sequential forward selection (SFS)), the performances of which are compared. This basic approach of entertainment modeling is applicable to a variety of games, both computer (Yannakakis and Hallam, 2006) and physical (Yannakakis et al., 2006a, Yannakakis et al., 2008, Yannakakis and Hallam, 2007e) using features derived from physiological data and/or from the interaction of player and opponent measured through game parameters.

As a sequel to previous work (Yannakakis et al., 2006b, Yannakakis et al., 2008) a new set of experiments for capturing entertainment preferences through physiology in physical play is presented here. This experiment expands the investigation of the physiological state's relation to entertainment preferences from HR to include blood volume pulse (BVP) and skin conductance (SC) signals; employs automatic techniques to identify important features for model construction; and compares preference learning methods as model-building tools. Moreover, the number of child participants is increased to 72 allowing for the creation of more accurate and generic user models. To control for elements of physical activity influencing the physiology of entertainment, an objectively (by human-verification) non-entertaining form of physical activity needs to be tested. For this purpose, a second game experiment, first introduced in Yannakakis et al. (2008), is employed, where a control physical activity task with characteristics similar to game activity is compared with game activity by children.

A statistical analysis reveals that features extracted from HR and BVP that correspond to both physical and mental/emotional effort correlate significantly with expressed preferences. Moreover, preference learning attempts on single features indicate that the energy of the high frequency (HF) band of HRV (derived from power spectral analysis) constitutes the feature that performs best in predicting expressed preferences on unknown data. This feature, which is suppressed during mental or emotional stress (Rowe et al., 1998, Goldberger et al., 2001), is highly anti-correlated to reported entertainment indicating high parasympathetic heart activity on preferred games. This analysis also suggests that collecting physiological signals beyond HR, such as BVP, may provide more meaningful features (e.g. energy of the HF band of HRV) for capturing entertainment preferences of children in physical play.

Comparative studies between the two feature selection methods and the three preference learning approaches reveal that evolving artificial neural network (ANN) models combined with SFS generate the highest accuracy in classifying between preferred and not preferred Playware game variants. These models are trained and validated on game-play data obtained from the first (main) set of experimentation and then are evaluated using unseen data from the second game-play and control experiment set. The results indicate that ANN user models able to predict children's preferred game variants given suitable HR and HRV feature representations can indeed be constructed and that such models not only distinguish game-play from game-like non-entertaining physical activity but also generalize (to some extent) over children's individual preferences.

The paper concludes with a discussion of the limitations of the proposed methodology and of the extent to which it could be applied to other genres of digital entertainment. Its generic use as an efficient baseline for capturing reported entertainment in physical interactive games in real-time is also outlined.