Being there again – Presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task

https://doi.org/10.1016/j.ijhcs.2017.01.004Get rights and content

Highlights

  • virtual environments are an alternative for user experience studies.

  • Ratings for usability are correlated with presence in VR.

  • Ratings for user experience are correlated with presence in VR.

Abstract

The possibility of using virtual environments instead of real field or laboratory environments is a promising research field. However, before virtual environments are able to replace the traditional environments the differences between the methods must be worked out. We take up on previous studies which compared different real and virtual environments concerning presence and usability and expand the research on the factor of user experience. We compared a virtual field environment (CAVE – Cave Automatic Virtual Environment) and a real field environment (city center of Chemnitz, Germany) in a between-subject-design concerning presence, and evaluate its impact on the usability and the user experience of a geocaching game. The data of 60 participants was analyzed and shows significantly higher ecological validity for the real field environment but higher values for engagement and negative effects in the virtual field environment. Concerning usability, significant differences were verified between the two environments. All presence factors correlated significantly with usability in the CAVE, but did not correlate in the real-field environment. Concerning user experience, the CAVE showed significantly higher hedonic quality values, whereas the real field environment had higher pragmatic quality values. In both conditions presence and user experience factors were partly correlated. Our results indicate that virtual environments can be an alternative to real environments for user experience studies, when a high presence is achieved.

Introduction

Technical inventions such as interactive maps are increasingly enriching our lives. These applications do not only show the places of interest, but they also track the user's position, provide the best local traffic route, reveal which restaurant is nearby and how other users rated it. However, comprehensive information is not satisfactory for the user. Two factors – usability and user experience (defined in Table 1) – are essential for interactive products and therefore for mobile navigation applications. Both factors determine the success of a product, because a good usability supports us when using applications and a good user experience leads to using an application with pleasure.

Typically, researchers conduct field or laboratory studies to get products or systems evaluated by users. Both of these settings exhibit different strengths and weaknesses: field studies are more ecologically valid but confounding variables can be less controlled, whereas laboratory studies are characterized by better experimental control, but are less realistic to participants. The use of virtual environments offers the possibility to combine the benefits of both environments (Loomis et al., 1999). Due to advances in the field of virtual reality (VR), a variety of systems are providing a lot of different options for human computer interaction studies. However when using such VR systems for these kinds of studies, they have to meet high requirements concerning realistic visualization and a plausible storytelling. Without a convincing presentation of the setting, the user will not have the feeling of being in the mediated environment and the benefits of the virtual setting compared to laboratory or field studies are nullified. This phenomenon is described by the term “presence”, the participant's feeling of “being there” (Barfield et al., 1995) (see Table 1).

Because of the advantages of virtual environments, it might be reasonable to use them instead of field or laboratory environments for product evaluation studies. However, we must first examine the influence of the environment on the perceived presence. Some studies research the concept of presence in different virtual environments (Gorini et al., 2011, Lorenz et al., 2015; Tang et al., 2004), whereas others compare virtual and real environments concerning perceived presence (Busch et al., 2014, Mania, 2001, Mania and Chalmers, 2004, Nisenfeld, 2003, Usoh et al., 2000). But only one study by Busch et al. (2014) directly addressed influences on presence and usability caused by different environments (virtual versus real laboratory). Unfortunately, this study lacks the comparison of virtual and real laboratory environments in terms of user experience, as well as the evaluation of a real field environment. As described in Table 1 it is insufficient to consider usability alone and it is also important to compare both kinds of environments with virtual environments to make meaningful statements. In addition, in this said study investigating usability, user experience was not considered. Only the fitness of use was assessed but not the acceptance.

Our study tries to approach the fundamental understanding of the relationships between presence, usability and user experience by examining them using a mobile navigation task in a virtual and a real field environment. Our results were in line with previous studies (Busch et al., 2014) and confirmed that a virtual field environment may be used as a substitute for a field environment even if the influences of presence have to be considered. The results showed a positive connection of a virtual field environment and hedonic qualities and confirmed the effect of usability on perceived presence. We offer guidance for usability and user experience researchers and practitioners for performing virtual user experience and usability product evaluations and discover important factors that have to be considered for the interpretation of the results.

The results presented in this paper are part of larger study and relevant for different scientific areas. The parts of the results that are important for an industrial/ergonomic viewpoint were cited in (Brade et al., 2016), where we concentrated on discussing the benefits virtual reality offers for the user. In (Brade et al., 2016) we show how traditional industrial approaches for product development can benefit from using virtual environments in industrial product development to increase productivity. Contrary to this, the results are interpreted from the psychological viewpoint in this paper.

Section snippets

Presence in real and virtual environments

Virtual reality provides the possibility to immerse someone into a computer-generated environment. To understand the differences between the technical aspects necessary to immerse someone, and a person's psychological involvement in a virtual scenario it is important that one understands the differences between the terms immersion and presence and how they are related. Slater and Wilbur (1997) define immersion as “a description of a technology, and describes the extent to which the computer

Research questions and hypotheses

The discussed related studies clearly demonstrate the need to examine presence, usability and user experience in VR in comparison to a real field environment, as this has not been done before. We compared two kinds of field environments (real and virtual) and investigated whether significant effects on reported presence, usability and user experience were evident (H1). We used a mobile navigation application as the evaluation scenario. Table 2 gives a detailed overview of the used environment,

Results

In this section we present all the results from the study to later interpret them from the psychological viewpoint. The parts of the results relevant for the discussion of industrial/ergonomic impact were reported in a condensed form in (Brade et al., 2016).

Summary of the study and important findings

In this study we analyzed the difference in effects of virtual and real field environments on presence, and the possible consequences for the self-reported user experience and usability. Our results show significantly higher values on the engagement and the negative effects scale for the CAVE, in contrast to the real field environment with a higher ecological validity. We are confirming previous work (Busch et al., 2014) that showed the same results or partially similar findings (Mania, 2001,

Conclusion

The current study shows that a virtual field environment is a tangible alternative to a real field environment when performing studies for evaluating products, if the influence of presence is considered, as it affects the user ratings for usability and user experience. A highly immersive virtual field environment with a realistic virtual model and a sound simulation of reality, provide the bases for reliable results of product evaluation. Virtual field environments are a good choice for

Jennifer Brade is a scientific assistant at the Division of Process Informatics and Virtual Product Development, Professorship for Machine Tools and Forming Technology, Technische Universität Chemnitz. Her research is focused on Human-Computer Interaction with the emphasis on Augmented Reality/Virtual Reality applications. In this context she is exploring the differences of virtual and real environments concerning presence, usability and user experience.

References (39)

  • X. Sun et al.

    A comparison of field-based and lab-based experiments to evaluate user experience of personalised mobile devices

    Adv. Hum.-Comput. Interact.

    (2013)
  • S. Sylaiou et al.

    Exploring the relationship between presence and enjoyment in a virtual museum

    Int. J. Hum.-Comput. Stud.

    (2010)
  • A. Toet et al.

    Effects of signals of disorder on fear of crime in real and virtual environments

    J. Environ. Psychol.

    (2012)
  • U.-V. Albrecht et al.

    Effects of mobile augmented reality learning compared to textbook learning on medical students: randomized controlled pilot study

    J. Med. Internet Res.

    (2013)
  • A. Bangor et al.

    Determining What individual SUS scores mean: adding an adjective rating scale

    J. Usability Stud.

    (2009)
  • W. Barfield et al.

    Presence and Performance within Virtual Environments, Virtual Environments and Advanced Interface Design

    (1995)
  • J. Brade et al.

    Virtual reality for industrial application – The influence of presence and its relation to usability and user experience

    Adv. Mater. Res.

    (2016)
  • J. Brooke

    SUS - A Quick and Dirty Usability Scale, Usability Evaluation in Industry

    (1996)
  • M. Busch et al.

    Being There For Real – Presence in Real and Virtual Environments and its Relation to Usability

    (2014)
  • C. Coelho et al.

    Media Presence and Inner Presence: The Sense of Presence in Virtual Reality Technologies, From communication to Presence: Cognition, Emotions and Culture Towards the Ultimate Communicative Experience, Festschrift in honor of Luigi Anolli

    (2006)
  • C. Cruz-Neira et al.

    Surround-Screen Projection-Based Virtual Reality: the Design and Implementation of the CAVE, SIGGRAPH

    (1993)
  • C. Cruz-Neira et al.

    The cave: audio visual experience automatic virtual environment

    Commun. ACM

    (1992)
  • Y.A.W. de Kort et al.

    virtual Laboratories: Comparability of real and virtual environments for environmental psychology

    Presence.: Teleoperators Virtual Environ.

    (2003)
  • H.B.-L. Duh et al.

    Usability evaluation for mobile device: a comparison of laboratory and field tests

    Mobile HCI 2006

    (2006)
  • Fraunhofer-IGD, 2016. Instant...
  • A. Gorini et al.

    The role of immersion and Narrative in mediated presence: the virtual Hospital experience

    Cyber., Behav., Soc. Netw.

    (2011)
  • Guy, E., Punpongsanon, P., Iwai, D., Sato, K., Boubekeur, T., 2015. LazyNav: 3D Ground Navigation with Non-Critical...
  • M. Hassenzahl et al.

    User experience - a research agenda

    Behav. Inf. Technol.

    (2006)
  • A. Kaikkonen et al.

    Usability testing of mobile applications: a Comparison between laboratory and field testing

    J. Usability Stud.

    (2005)
  • Cited by (86)

    • BIM-based immersive collaborative environment for furniture, fixture and equipment design

      2022, Automation in Construction
      Citation Excerpt :

      The System Usability Scale (SUS) questionnaire [9] was used during the quantitative phase of this study to assess the perceived usability and validity of COFFEE. The participants' perceived sense of presence (SoP) whilst using COFFEE was measured, as extant research (e.g., [7,12,33,41]) suggested that the sense of presence in the virtual environment has a stronger connection with the usability of the system and will encourage the acceptance of VR technology. Hence, measuring participants' sense of presence (SoP) while using COFFEE was relevant.

    • User Experience of Digital Voice Assistant: Conceptualization and Measurement

      2023, ACM Transactions on Computer-Human Interaction
    • Factors Influencing Engagement in Hybrid Virtual and Augmented Reality

      2023, ACM Transactions on Computer-Human Interaction
    View all citing articles on Scopus

    Jennifer Brade is a scientific assistant at the Division of Process Informatics and Virtual Product Development, Professorship for Machine Tools and Forming Technology, Technische Universität Chemnitz. Her research is focused on Human-Computer Interaction with the emphasis on Augmented Reality/Virtual Reality applications. In this context she is exploring the differences of virtual and real environments concerning presence, usability and user experience.

    Mario Lorenz is a Research Associate at the Division of Process Informatics and Virtual Product Development, Professorship for Machine Tools and Forming Technology, Technische Universität Chemnitz and a visiting scientist at the Department of Orthopaedic Surgery, Traumatology and Plastic Surgery, University Hospital Leipzig. His research interests are focused on Augmented Reality/Virtual Reality applications for mechanical and medical engineering. Furthermore, he is researching the effects of presence in the context of Human-Machine Interaction.

    Marc Busch is scientist at AIT Austrian Institute of Technology, holds a degree in psychology from the University of Vienna and works on user-centered design of interactive systems and technology acceptance. Furthermore, he is specialized in advanced quantitative and qualitative usability and user experience methodology, research methods and statistics in Human-Computer Interaction.

    Niels Hammer is an Associate Professor at the Department of Anatomy, University of Otago, New Zealand. He is an MD and holds degrees as an Anatomist and Specialist Anatomist, with a background in Orthopedics and Trauma Surgery. His research is focused on the healthy and pathologically altered musculoskeletal system with focus on tissue biomechanics and numerical simulation.

    Manfred Tscheligi is a Professor of Human-Computer Interaction at the University of Salzburg and Head of the Business Unit Technology Experience at AIT. He has been working in the area of Interactive Systems, Human-Computer Interaction, Usability Engineering, User Interface Design and User Experience Research for more than 20 years. He is a pioneer in establishing this field in Austria, an author of several publications and has been a distinguished speaker at conferences.

    Philipp Klimant holds a diploma in Electrical Engineering from the University of Applied Sciences Mittweida. He joined the Technische Universität in 2007. His research focuses are Virtual and Augmented Reality, Machine and Factory Simulation, and Medical Engineering. In 2013 he received his PhD in Mechanical Engineering. Since 2011 he has been the Head of the Division of Process Informatics and since 2014, the Head of the new Division Process Informatics and Virtual Product Development. He is responsible for the VRCP laboratory (Virtual Reality Center Production Engineering). Since 2013 he has also been the Head of the Central Coordination Office of the Saxon Cluster of Excellence eniPROD.

    1

    Equal contributors.

    View full text