Multimodal Interaction in Virtual Reality: Assessing User Experience of Gesture- and Gaze-Based Interaction

Virtual reality (VR) applications are entering people’s everyday lives and being used in professional applications, research, and science. The question arises as to whether there may be interaction modes that are more suitable than the standard controllers with which these devices are typically equipped. To address this question, it is necessary to conduct human-centered studies, which is typically done in the field of human factors research. We present a forthcoming study in which we integrate mid-air gesture-based interaction and gaze-based interaction into a VR setup so that users can interact with the system without touching a button, surface, or device. We then compare these two approaches to the controllers that are provided with the VR device. For gesture recognition, we use a Leap Motion device, which is mounted on the front of the VR device. For gaze-based interaction, we use the eye-tracking system that is integrated into the HCT Vive. We use a basic application, in which users can rotate and move virtual 3D blocks. In the study described, we will focus on user experiences and ask the participants to rate the prototypes according to their subjective impressions. Additionally, we will document errors caused by the system or participants’ incorrect actions to investigate whether perceived errors affect user experiences. Lastly, we will ask for reports of pain or discomfort in the body parts involved and measure the size of the participants’ fingers and hands.