A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies

https://doi.org/10.1016/j.ijhcs.2006.11.011Get rights and content

Abstract

The popularity of computer games has exploded in recent years, yet methods of evaluating user emotional state during play experiences lag far behind. There are few methods of assessing emotional state, and even fewer methods of quantifying emotion during play. This paper presents a novel method for continuously modeling emotion using physiological data. A fuzzy logic model transformed four physiological signals into arousal and valence. A second fuzzy logic model transformed arousal and valence into five emotional states relevant to computer game play: boredom, challenge, excitement, frustration, and fun. Modeled emotions compared favorably with a manual approach, and the means were also evaluated with subjective self-reports, exhibiting the same trends as reported emotions for fun, boredom, and excitement. This approach provides a method for quantifying emotional states continuously during a play experience.

Introduction

Computer games have grown during recent years into a very popular entertainment form with a wide variety of game types and a large consumer group spread across the world. As researchers develop novel play environments, computer and console game markets continue to grow rapidly, outperforming the film industry in terms of total revenues in many regions (Pagulayan et al., 2002). Although gaming technology has continued to evolve, researchers and traditional computer game developers suffer from a lack of effective evaluation methods.

The development of evaluation methodologies in human–computer interaction research (HCI) has been rooted in the cognitive sciences of psychology and human factors, in the applied sciences of engineering, and in computer science (Norman, 2002). Although the study of human cognition has made significant progress in the last decade, the idea of emotion, which is equally important to design (Norman, 2002), is still not well understood, especially when the primary goals are to challenge and entertain the user. Traditional measures for productivity environments, such as task performance, are not applicable to affective environments since we are not interested in performance; we are interested in what kind of emotional experience is provided by the play technology and environment, regardless of performance (Pagulayan et al., 2002). Although traditional usability measures may still be relevant, they are subordinate to the emotional experiences resulting from interaction with the play technology and with other players.

Our research interest is in how to quantify emotional experience when engaged with affective technologies, by developing an evaluation methodology for entertainment environments that is as robust as methods for evaluating productivity. This paper motivates why we need such an approach and describes the process by which we designed a new evaluative methodology for measuring emotional experience with interactive entertainment technologies.

Traditional evaluation methods have been adopted, with some success for evaluating entertainment technologies, and include both subjective and objective techniques. The most common methods of assessing emotion are through subjective self-reports including questionnaires, interviews, and focus groups (Fulton and Medlock, 2003) and through objective reports from observational video analysis (Lazzaro, 2004).

The success of a play environment is determined by the process of playing, not the outcome of playing (Pagulayan et al., 2002). We must consider this when evaluating emotional experience during interaction with play technologies, as current methods suffer from low evaluative bandwidth, providing information on the whole experience, rather than continuously throughout time.

Subjective reporting through questionnaires and interviews is generalizable, and is a good approach to understanding the attitudes of the users, but subjects are bad at self-reporting their behaviors in game situations (Pagulayan et al., 2002). In addition, subjective techniques only generate data when a question is asked, and interrupting game play to ask a question is too disruptive. Desmet (2003) developed a non-verbal questionnaire designed specifically to assess 14 separate emotional responses to products. Although it addresses some of the drawbacks of language scales, the evaluative bandwidth is still low.

Using video to code gestures, body language, facial expressions and verbalizations, is a rich source of data; however, there is an enormous time commitment, which requires between 5 and 100 h of analysis for every hour of video (Fisher and Sanderson, 1996).1 Also, the analysis is generally event-based (user is smiling now), rather than continuous (degree of smile for every point in time), which could be important for exploring the process of play.

There has been some recent research on using inspection methods, such as heuristics (Wiberg, 2003; Desurvire et al., 2004; Sweetsner and Wyeth, 2005) to evaluate the playability of an entertainment technology, but these discount methods do not involve actual users, but are administered by usability specialists. Heuristics also give an overview of the playability, rather than examining a user's change in emotions over time.

Researchers in human factors have used physiological measures as indicators of mental effort and stress (Vicente et al., 1987). See Mandryk and Inkpen (2004) for an overview. Psychologists use physiological measures to differentiate human emotions such as anger, grief, and sadness (Ekman et al., 1983). Recently, physiological measures have been used to assess a user's emotional experience when engaged with computing systems (see Section 2.4); however, physiological data have not been employed to identify a user's emotional state, such as fun or excitement, when engaged with play technologies. Based on previous research on the use of psychophysiological techniques, we believe that capturing, measuring, and analyzing autonomic nervous system (ANS) activity will provide researchers and developers of technological systems with continuous access to the emotional experience of the user. Used in concert with other evaluation methods (e.g. subject reports and video analysis), a complex, detailed account of both conscious and subconscious user experience could be formed.

We designed an experiment to create and evaluate a model of user emotional state when interacting with play technologies. We record users’ physiological, verbal and facial reactions to game technology, and apply post-processing techniques to quantitatively and continuously measure emotional state. We envision that when combined with other evaluative approaches, our technique can help create a rich and robust picture of user experience.

Section snippets

Physiological metrics for evaluation

In this section we briefly introduce the physiological measures used, describe how these measures are collected, and explain their inferred meaning. Based on previous literature, we chose to collect galvanic skin response (GSR), electrocardiography (EKG), and electromyography of the face (EMGsmiling and EMGfrowning). Heart rate (HR) was computed from the EKG signal.

Identifying emotions

There has been a long history of researchers attempting to use physiological data to identify emotional states. William James first speculated that patterns of physiological response could be used to recognize emotion (Cacioppo et al., 2000), and although this viewpoint is too simplistic, recent evidence suggests that physiological data sources can differentiate among some emotions (Ekman et al., 1983; Levenson, 1992). For example, Picard et al. (2001) performed a feature-based recognition of

Experimental details

We conducted a study to investigate whether we could model emotional responses to play technologies. To generate values for user emotion, we modeled the data in two parts using a fuzzy logic approach. First, we computed arousal and valence values from the normalized physiological signals of GSR, HR, EMGsmiling, and EMGfrowning. We then used these arousal and valence values to generate emotion values for boredom, challenge, excitement, frustration, and fun.

The details in this section apply to

Fuzzy logic

We used normalized GSR, HR, EMGsmiling, and EMGfrowning signals as inputs to a fuzzy logic model. To generate values for user emotion, we modeled the data in two parts. First, we computed arousal and valence values from the normalized physiological signals, then used these arousal and valence values to generate emotion values for boredom, challenge, excitement, frustration, and fun.

Fuzzy logic mimics human control logic in that it uses an imprecise but descriptive language to deal with input

Modeling arousal-valence space

The first stage was to transform the physiological signals into AV space (arousal-valence space). To generate the models, we used half of the participants (one for each play condition order), reserving the other six participants for validation of the model. To make use of the continuous nature of physiological data, we used the complete time series for each input. As such, we were able to generate a new time series of the participant's experience in AV space, rather than having only one data

Modeling emotion from AV space

The second phase of the emotion model is to use the arousal and valence information to model different emotions. To make the most of the rich, continuous physiological data, we modeled the entire AV space time series, creating continuous metrics of emotional experience. Five emotions were modeled: boredom, challenge, excitement, frustration, and fun. These are five of the seven emotions that participants rated after each play condition.

The experience states of fun and challenge are not emotions

Using the model of emotion

To analyze the effectiveness of our model, we used data gathered from the six subjects not used in the generation of the model. Obtaining successful results using a clean set of data would show the generalizability of our model across individuals, but not across situations or applications. A complete description of the validation experiment, results, and statistics is presented in (Mandryk et al., 2006a). In addition, information on the applicability of the work to designers and other HCI

Conclusions

We used a fuzzy logic approach to transform GSR, HR, EMGsmiling, and EMGfrowning into arousal and valence. The results from the fuzzy model were comparable to a manual approach. In addition, the results were consistent with predictions based on the results from prior experiments. A second fuzzy model was used to convert arousal and valence into five emotions: fun, challenge, boredom, frustration, and excitement. Modeled emotion was represented both as an average over a condition, and as a time

Acknowledgments

Thanks to NSERC, NECTAR, EA Sports, and SFU Surrey for funding and equipment. Also, thanks to Dr. Kori Inkpen, Dr. Tom Calvert, Dr. Kelly Booth, and Dr. Lyn Bartram.

References (41)

  • P.M.A. Desmet

    Measuring emotion: development and application of an instrument to measure emotional responses to products

  • Desurvire, H., Caplan, M., Toth, J.A., 2004. Using heuristics to evaluate the playability of games. In: the Extended...
  • P. Ekman

    Basic emotions

  • P. Ekman et al.

    Autonomic nervous system activity distinguishes among emotions

    Science

    (1983)
  • C. Fisher et al.

    Exploratory data analysis: exploring continuous observational data

    Interactions

    (1996)
  • A.J. Fridlund et al.

    Guidelines for human electromyographic research

    Psychophysiology

    (1986)
  • Fulton, B., Medlock, M., 2003. Beyond focus groups: Getting more useful feedback from consumers. Paper presented at the...
  • P.J. Lang

    The emotion probe

    American Psychologist

    (1995)
  • P.J. Lang et al.

    Looking at pictures: affective, facial, visceral, and behavioural reactions

    Psychophysiology

    (1993)
  • Lazzaro, N., 2004. Why we play games: 4 keys to more emotion. Paper presented at the Game Developer's Conference, San...
  • Cited by (0)

    View full text