Measuring User Engagement

1. Front Matter

   Pages i-xv

   PDF

2. Introduction and Scope

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 1-10

3. Approaches Based on Self-Report Methods

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 11-29

4. Approaches Based on Physiological Measurements

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 31-46

5. Approaches Based on Web Analytics

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 47-59

6. Beyond Desktop, Single Site, and Single Task

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 61-72

7. Enhancing the Rigor of User Engagement Methods and Measures

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 73-86

8. Conclusions and Future Research Directions

   • Mounia Lalmas, Heather O’Brien, Elad Yom-Tov

   Pages 87-91

9. Back Matter

   Pages 93-116

   PDF

User engagement refers to the quality of the user experience that emphasizes the positive aspects of interacting with an online application and, in particular, the desire to use that application longer and repeatedly. User engagement is a key concept in the design of online applications (whether for desktop, tablet or mobile), motivated by the observation that successful applications are not just used, but are engaged with. Users invest time, attention, and emotion in their use of technology, and seek to satisfy pragmatic and hedonic needs. Measurement is critical for evaluating whether online applications are able to successfully engage users, and may inform the design of and use of applications. User engagement is a multifaceted, complex phenomenon; this gives rise to a number of potential measurement approaches. Common ways to evaluate user engagement include using self-report measures, e.g., questionnaires; observational methods, e.g. facial expression analysis, speech analysis; neuro-physiological signal processing methods, e.g., respiratory and cardiovascular accelerations and decelerations, muscle spasms; and web analytics, e.g., number of site visits, click depth. These methods represent various trade-offs in terms of the setting (laboratory versus ``in the wild''), object of measurement (user behaviour, affect or cognition) and scale of data collected. For instance, small-scale user studies are deep and rich, but limited in terms of generalizability, whereas large-scale web analytic studies are powerful but negate users' motivation and context. The focus of this book is how user engagement is currently being measured and various considerations for its measurement. Our goal is to leave readers with an appreciation of the various ways in which to measure user engagement, and their associated strengths and weaknesses. We emphasize the multifaceted nature of user engagement and the unique contextual constraints that come to bear upon attempts to measure engagement in different settings, and across different user groups and web domains. At the same time, this book advocates for the development of ``good'' measures and good measurement practices that will advance the study of user engagement and improve our understanding of this construct, which has become so vital in our wired world.

• Yahoo Labs, USA

  Mounia Lalmas

• University of British Columbia, Canada

  Heather O’Brien

• Microsoft Research, USA

  Elad Yom-Tov

Mounia Lalmasis a Principal Research Scientist at Yahoo Labs London. Prior to this, she held a Microsoft Research/RAEng Research Chair at University of Glasgow. Before that, she was Professor of Information Retrieval at Queen Mary, University of London. From 2002 until 2007, she co-led the Evaluation Initiative for XML Retrieval (INEX), a large-scale project with over 80 participating organizations worldwide, which was re sponsible for defining the nature of XML retrieval, and how it should be evaluated. Her research is concerned with developing innovative models and metrics of user engagement, through the study of user behavior, web analytics, the analysis of users’ emo tion and attention, and mouse and gaze movement. She is studying user engagement in areas such as advertising, digital media, social media, and search, and across devices (desktop, tablet and mobile). She also pursue research in social media and search Heather O’Brien is an Assistant Professor at the University of British Columbia whose focus is user engagement with tech nology. She has explored the conceptual nature of engagement to isolate key attributes of an engaging experience, developed a survey instrument to evaluate users’ perceptions of their en gagement, and is currently examining the manifestation of en gagement and how to measure it in information rich contexts such as online news and search systems. Her research, which has appeared in the Journal of the American Society of Information Science and Technology, Information Processing and Management, Interacting with Computers and various conference proceedings, is supported by the Social Sciences and Humanities Research Council of Canada (SSHRC) and the Networks of Centers of Excellence Graphics and New Media Project (NCE GRAND) Elad Yom-Tov is a Senior Researcher at Microsoft Research. Before joining Microsoft he was with Yahoo Research, IBM Research, and Rafael. Dr. Yom-Tov studied at Tel-Aviv Uni versity and the Technion, Israel. He haspublished two books, over 60 papers (of which 3 were awarded prizes), and filed more than 30 patents (15 of which have been granted so far). His primary research interests are in large-scale Machine Learn ing, Information Retrieval, and Social Analysis. He is a Senior Member of IEEE and held the title of Master Inventor while at IBM

Policies and ethics