Virtual, Augmented and Mixed Reality: Applications in Health, Cultural Heritage, and Industry

We plan to conduct a longitudinal mixed-methods study to evaluate the impact of home-use Virtual Reality (VR) for chronic pain (CP) management with cancer survivors over a six-week period. The participants will play VR games for a minimum of 30 min per day for at least three days per week. Each week, the participants are required to use randomly assigned VR content from a group of VR titles that ranged from commercially-available VR games to VR meditation designed specifically for people with CP. Well before launching the longitudinal study, we compiled, tested, categorized and eliminated commercial VR titles, and developed a short list that varied by genre and methods of interaction. Therefore, to better understand which are the best VR games to use, how to offer good VR gaming experiences, and validate our choices, we compared CP patients’ experiences of some selected games in VR and non-VR conditions in this research. Seven pain patients took part in the focus group study; initial results suggest participants preferred VR conditions compared to PC games according to both of the quantitative ratings and the qualitative interviews. They also liked puzzle-solving game mechanism comparing to exploration games. Participants also mentioned that the platform (VR vs. PC monitor) is more important than the actual content. This result will be helpful for the researchers to select proper VR games to test in the longitudinal study for pain distraction and management purposes since it has been done with patients.

This paper seeks to determine if multimodal feedback, from auditory and haptic stimuli, can affect a user’s perceived exertion in a virtual reality setting. A simple virtual environment was created in the style of a desert to minimize the amount of visual distractions; a head mounted display was used to display the environment. Users would in the real world drive a Combi Bike and the velocity generated was translated to a vehicle in the virtual environment, moving it forward on a predetermined path. Each user traversed a total of eight hills, two in each of the four conditions. The perceived exertion was measured several times during each condition using the Borg Scale. The results show that there is no significant difference between the four conditions, which had different combinations of auditory and haptic feedback.

Chronic and acute stress are persistent and troubling health concerns for many people and military veterans in particular. Clinicians are increasingly turning to mindfulness techniques to provide people with the skills they need to self-manage that stress. However, training and getting people to adhere to the practice is difficult. In this paper, we talk about a virtual reality based system designed specifically to help veterans learn mindfulness-based stress reduction techniques.

This paper describes a Mixed Reality (MR) based testbed and a framework for studying social interactions using a game theory approach. MR technology enables moving and interacting naturally with the real and virtual world in three dimensions, thereby using affordances for a better sense of presence. That, in turn, provides a unique opportunity to create a realistic but controlled interaction scenario to stimulate the natural response of a subject, making it an efficient platform for studying situated interactions. The users are engaged with digital characters in repetitive social interactions where the behavior of the characters is modeled using game theory. Game theory interaction models stimulate behavioral gestures in subjects that can be studied to explore psycho-pathological traits or other behavioral aspects of subjects. A proof of concept MR application is presented that provides exposure therapy for treatment of individuals with Social Anxiety Disorder. A case study for exposure therapy treatment using the proposed testbed is also presented. The small pilot study demonstrates the feasibility of the developed approach for studying social situations as well as the influence of the facial expressions displayed by 3D avatars on the user’s perception of realism in an MR environment.

In this paper, we introduce a new 3D-sound-based VR game named Escape from Dark Jungle, with the design goal of regulating players’ emotion. Our game design is based on the “Stimulus - Response” theory of behavioral method, implemented using Low-cost, real-time 3D audio technologies. We conducted an extensive user study to evaluate the game’s effectiveness on emotion regulation. The results show that this game achieves this goal by effectively making players more positive or negative, excite or calm.

Depression is a serious public health concern. The problem is further exacerbated due to social stigma and stereotypical attitudes. Thus, many people with depression keep the suffering to themselves and avoid seeking professional psychological help. Virtual reality applications offer a way to provide a virtual counseling experience without being stigmatized. In this study, we produced 360-degree virtual reality videos in which a person is shown sharing stories about his experience of living with depression. The participants (n = 12), with mild and moderate levels of depression, were invited in the lab and watched the 360-degree videos using Oculus Rift. After each video, participants were asked to say-out-loud their personal experiences about living with depression. We hypothesize that such private and confidential experience of talking about one’s illness will help the users to express their feelings without the fear of being stigmatized. Results show that participants expressed more positive emotions, compared with negative emotions, after watching the 360-degree videos. Participants also expressed high positive attitude toward help seeking behavior. However, the study did not find an above average behavioral intention to seek help. Qualitative data gathered from the thought-listing exercise provides further insights about the effectiveness of virtual reality videos to promote help seeking behavior among depressed individuals. The study offers implications for improving mental health help-seeking, specifically on college campuses.

This study investigates the health message effect of Spatial Augmented Reality (SAR) with projection mapping technology while comparing a two-dimensional flat screen and three-dimensional image of a participant’s body. This current study offers insight that the dualprocess model occurs when people experiencing spatial presence, also known as feeling of “being there,” are particularly experiencing affective feeling, not cognitive feeling. In turn, the affective feeling as spatial presence enhances negative attitudes on cigarette smoking and online viral behavior intentions while demonstrating heuristicsystematic process model (HSM).

We present an experimental study evaluating the effectiveness of immersive virtual reality games (combined with stationary cycling) on health-related physical activity in comparison to stationary cycling with the same game in a non-immersive (2D) setting. In the experiment, participants were asked to play a cycling video game with a stationary bike either in the immersive virtual reality (3D VR) environment or in a traditional 2D (Non-VR) display setting. Based on several theories, we anticipate that users of immersive virtual reality will feel more present in the game be induced to physically performed better than when they played the same game with a traditional 2D screen. In addition, we also expect to see people who are eager to accept and use a new technology will show better cycling performance in an immersive virtual environment than people who think accepting the newest technology in a timely manner is not very important for their lives.

Fear is one of the basic emotions of human which are comprehensive representations of human psychology and consciousness. The evocation of emotion has been applied to many fields, such as mental diseases diagnosis, mental health assessment, study of cognitive science and game design. The present methods for affective stimuli mainly include texts, pictures, sounds, odors and computer games. However, they have some limitations in terms of inefficient emotion elicitation and high susceptibility to ambient interferences. Virtual reality technology is a potential technology to solve some of these problems by providing immersive and realistic experience in terms of emotion elicitation. Therefore, in this study we set up a systematic approach for the design of emotion evocative VR scenes, using fear as an example. Firstly, we extracted fear evocative elements from the International Affective Picture System (IAPS), the Chinese Affective Digital Sound System (CADS), the Chinese Affective Video System (CAVS), horror films and video clips. Secondly, we developed three virtual reality fear scenes using the classified evocative elements. At last, we used the Self-Assessment Manikin (SAM) scale to test the effectiveness of fear evocation using these VR scenes. In conclusion we have built up a series of fear evocative VR scenes, which can induce different levels of fear with high effectiveness of fear elicitation in the VR environment.

Depression has caused serious problems and attracted extensive attention in our society nowadays. Traditional depression diagnosis methods mainly include scale examination, blood test and medical imaging. But they did not obtain promising performance because of patient’s stigma, arbitrariness and distraction. The virtual reality technology can solve some of those problems by providing immersive experience and rich interaction in terms of detecting patient’s emotions. In this paper, we propose utilizing VR in the field of depression assessment. Specifically, we have designed a personalized VR depression diagnosis scene and interactive models for depression assessment. To validate the effectiveness of our approach, preliminary experiments were carried out and good results were obtained from survey of user experience. The average score of each ten questions is over 3.2 points with a total score of 5 points. The visibility of doctor’s position obtains the highest score. The lowest is naturalness of the scene, the main factor being the coordination between the human doctor image and the virtual scene. In future work, we are going to further improve the user experience and the efficiency of assessment.

Virtual Reality, (VR), and Augmented Reality, (AR), have been used for business, social life, culture, education and health. As a part of information technology, VR is a tool with the potential to improve the quality of life of people. Currently, it is possible to use tools in combination with IT, such as for situations related to entertainment and education activities, as well as in health treatments. This work is focus on health area mainly on how to prevent the Diabetes effects using applications based on a Virtual Reality.

An analysis of the state of the art in relation to the articles and authors who have worked on the subject of this study is presented, with such analysis we can summarize the actual situation and position this study within the framework of knowledge found.

An integrative review of the areas of knowledge is presented in relation to the objective, which are Virtual Reality in health, Diabetes Studies, in turn this is subdivided into Virtual Reality applied in chronic degenerative diseases treatments and works related to physical and emotional rehabilitation. Regarding the area of studies on diabetes, it is subdivided into the literature focused on the prevention of this condition and in the literature dedicated to the population of young people and adults at México and Spain.

In this paper, we verify the effect of the chair system which we developed as “Breath Chair.” If a person sits in the “Breath Chair,” the subject can be contacted with the movement which was simulated as human’s breathing motion. We verified that the chair system’s breathing movement reduced fear or anxiety of the person who sit in the chair system. Furthermore, the person’s fear or anxiety was reduced even if the chair system’s breathing movement was generated by a human who was not familiar with that person.

This paper examines the impact of an augmented reality (AR) tour guide on users’ art engagement. In a between-subjects experiment in an art museum, users’ art engagement with a novel video see-through augmented reality guide was examined against user behavior with a book guide. The AR users’ liking of art increased more than the book users’, whereas the book users learned more. The AR users enjoyed using the application, and it helped them engage with art; however, they felt physical fatigue from holding the tablet and preferred interacting with a still image over the live augmentations. The users were concerned about the screen time the AR guide required; it distracted them from looking at the art with the naked eye, creating more of an impression of learning rather than actual learning. The findings call for more impactful AR applications with art as the focal point.

In recent years, virtual reality has seen a growth in its investments by the technology industry, despite all the experts’ predictions about the duration of the hype around this technology and also with some problems still unresolved, such as motion sickness. One of the segments that receive most of these investments is the Games Area for VR, supported by technology enthusiasts, wishful for technological innovation. This essay analyzes some studies about the VR gaming industry for the understanding of all its actors: from manufacturers to their consumers, including developers, artists and researchers.

The rapid development of VR technology in the past three years allowed artists, filmmakers and other media producers to create great experiences in this new medium. But filmmakers are, however, facing big challenges, when it comes to cinematic narration in VR. The old, established rules of filmmaking do not apply for VR films and important techniques of cinematography and editing must be completely rethought. Possibly, a new filmic language will be found. But even though filmmakers eagerly experiment with the new medium already, there exist relatively few scientific studies about the differences between classical filmmaking and filmmaking in 360 and VR.

We therefore present this study on cinematic narration in VR. In this we give a comprehensive overview of techniques and concepts that are applied in current VR films and games. We place previous research on narration, film, games and human perception into the context of VR experiences and we deduce consequences for cinematic narration in VR. We base our assumptions on a conducted empirical test with 50 participants and on an additional online survey.

In the empirical study, we selected 360-degree videos and showed them to a test-group, while the viewer’s behavior and attention was observed and documented.

As a result of this paper, we present guidelines which suggest methods of guiding the viewers’ attention as well as approaches to cinematography, staging and editing in VR.

The aim of this article is to present and discuss the methodology and the findings of a qualitative evaluation of a Virtual Reality (VR) game. The object of this article is the player satisfaction and the thematic proficiency of game ‘Angest’, developed by Black River Studio. The definition, particularities and challenges that the platform (Samsung Gear VR) and genre (walking simulators) imposed to this research are presented and discussed in the article, as well how they could benefit one another. The methodology was defined based on these particularities and given research questions. It is explained which references, apparatus and techniques of co-related fields (as HCI and game evaluation) was selected and the reason why. From the evaluation, it was possible to confirm the benefits of qualitative validation for this kind of genre and platform as well what should be used from standard user experience and player experience evaluation, what need to be adapted and what should be avoided.

We propose a novel AI-based VR earthquake simulator that can easily simulate an arbitrary real indoor environment. The user first scans any room using inexpensive RGBD sensors, such as Kinect sensors. The automatic model generator then detects objects in RGBD data using a deep-learning-based classification method and generates 3D models given appropriate physical characteristics, such as centroid and friction. In this manner, users can obtain appropriate content for earthquake simulation simply by scanning the room. The content can provide the user with a real experience in the event of an earthquake in the room.

When working with emergent and appealing technologies as Virtual Reality, Mixed Reality and Augmented Reality, the issue of definitions appear very often. Indeed, our experience with various publics allows us to notice that technology definitions pose ambiguity and representation problems for informed as well as novice users.

In this paper we present Immercity, a content curation system designed in the context of a collaboration between the University of Montpellier and CapGemini, to deliver a technology watch. It is also used as a testbed for our experiences with Virtual, Mixed and Augmented reality to explore new interaction techniques and devices, artificial intelligence integration, visual affordances, performance, etc. But another, very interesting goal appeared: use Immercity to communicate about Virtual, Mixed and Augmented Reality by using them as a support.

VAIR Field is a mobile virtual reality (VR) system that enables multiple players to play a battle game with multiple mobile devices. By tracking the mobile devices’ positions and detecting their rotation and movement, VAIR Field displays images viewed from each player’s position on their displays like a head-mounted display (HMD). VAIR Field enables players to immediately play with the VAIR Gun and VAIR Bow without needing instructions. It also enables children under 13 to safely play VR or augmented reality (AR) games because it does not require a HMD. We compared our new VAIR Gun and BOW with a HTC Vive controller, the initial VAIR Gun, and our previous electric bow interface. The new VAIR Gun and BOW mostly realize VAIR Field’s concepts of user-friendliness for children, sports-like play, and “feel of things.”

A Command and Control (C2) display system using the Microsoft HoloLens and the Intelligent Multi-UxV Planner with Adaptive Collaborative Control Technologies (IMPACT) has been developed as a demonstration of a new advanced user interface. This allows for human-to-human-to-machine collaboration for situational awareness, decision making, and C2 planning and execution of simulated multi-unmanned heterogeneous autonomous vehicles. The advanced user interface allows multiple operators to collaborate across a shared holographic sand table and control multiple vehicles. Multiple networking frameworks were used to offload the computation of vehicle autonomy and planning algorithms to allow the HoloLens to run efficiently for an improved user experience. Additionally, the concept of pseudo-classified information filtering allows for tiers of classification levels for each HoloLens user derived from a ‘need-to-know’ classification basis.

One of the key driving technologies for a better communication, representation, interaction, and visualization of design and engineering data has been Virtual Reality (VR). The idea of Interactive Conceptual Simulations (ICS) combines real-time interaction and visualization in a turn-around loop with the CAE simulation. However, the automation of the processes between the changes of the domain and the resulting simulation requires update rates within min 30 Hz, in order to remain interactive, not blocking the CG engine redrawing the update. Here, the overall simulation requires an advanced CAE process chain from model import, model manipulation, simulation results generation and visual preparation. Although many publications have been addressing the CAD/VR and only few the CAE/VR process chain, one might assume that endeavors for this are regarded as unnecessary, as a tidy showcase with a low-level quality visualization would be sufficient for engineers. This paper will object this hypothesis by presenting the results of a qualitative validation based on industrial use of our established VR environment for ICS. It follows the methodology and complements the quantitative analysis presented in earlier work providing the results of a qualitative assessment of engineers for our established VR based ICS post-processing unit (IDEFix – Immersive Data Explorer for CAx Integration).

This work provides an overview of three prototype augmented reality (AR) applications developed for the Microsoft HoloLens with the intention of exploring the strengths and weaknesses of AR for supporting planning and decision making–specifically, within the domain of US Army Mission Command and Battle Management Command and Control (BMC2) execution. We present each prototype application accompanied by the target audience from whom we sought feedback, key features of the application, and technical goals we hoped to achieve, demonstrate, and evaluate. Findings with respect to AR strengths and weaknesses, framed around the technical goals of each of the AR applications, are then presented to begin to shed light on limitations and opportunities of the state-of-the-art in AR hardware, and the potential to support Mission Command and BMC2 stakeholders. This material is based upon work supported by the Communications-Electronics Research, Development and Engineering Center (CERDEC) under Contract No. W56KGU-18-C-0002. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of CERDEC.

Aerial refueling is one of the most demanding and dangerous activities faced by pilots. To monitor refueling, pilots must focus for long periods of time while looking up and outside the aircraft (“off-axis”), a more difficult task than focusing forward in the direction of flight (“on-axis”). To address these challenges, we designed a set of augmented reality display strategies for head-mounted displays (HMDs) that support pilot spatial orientation during off-axis activities, such as refueling. These display strategies include extending traditional on-axis displays (e.g., pitch ladders) for the off-axis context and designing new displays that convey critical information specifically tailored for the off-axis context. In this paper, we present our overall approach and a subset of concepts to address these needs. We also describe plans for formal evaluations.

In this paper we present and discuss common visual elements in Augmented Reality which create a distinct information context while presenting them in either video- or optical-see-though setups, and which align with the promise of AR being able to bridge the gap between real world objects and the digital information space about them.

Reflecting on nowadays common elements with these premises in mind, we collected and categorized a variety of visual elements, e.g. annotation & labels, visual highlights, assisting visual aids and trans-media elements. Focusing on industrial AR applications, we discuss their suitability in terms of mediation and communication goals, instead of technological and implementation considerations.

In doing so, we seek to identify the currently most relevant visual elements and discuss the deployed meaning that can be created in utilizing these elements for a informed and successful communication. From there we introduce a first framing meta-model that on the one hand helps clarifying the mediation strength of these elements and on the other enables to reflect their suitability on a more strategic level.

There has been a lot of research conducted on human-robot interaction (HRI) with drones as well as human-computer interaction (HCI) with virtual reality (VR). However, little work was done on VR as HRI, even less for VR and drones in particular. In this paper, we evaluate the usability of VR interfaces used to control drones through working with UC Berkeley’s Immersive Semi-Autonomous Aerial Command System (ISAACS) project, which experiments with new ways for humans to interact with drones in a VR environment. Our experiment setting focuses on the drone mission planning phase and on creating an onboarding experience for new users. We develop a usability evaluation framework for the ISAACS VR system, and use this framework to conduct two iterations of user testing and prototyping with a human-centered design process.

Cyber security remains an ominous task for the military. Generating awareness of some cyber challenges is daunting, yet aspects of cyber vulnerabilities are often ambiguous to everyday operators. Many of these challenges have been documented in prior papers. The current effort focused on two studies which describe levels of general awareness of cyber challenges among students (a typical age range for entry-level military operators). The studies asked participants to evaluate aerial images and to determine if the images had been degraded. Following a positive degradation identification, participants were asked to list the reason for the perceived degradation by drawing from a set of four options. The option related to cyber attack was proportionally used much less than the others, even in the context of an explicit experimental prime to motivate cyber awareness and vulnerabilities. The implications for these results for interactions with autonomous systems are discussed.

Human operators will increasingly team with autonomous systems in military and security settings, for example, evaluation and analysis of threats. Determining whether humans are threatening is a particular challenge to which future autonomous systems may contribute. Optimal trust calibration is critical for mission success, but most trust research has addressed conventional automated systems of limited intelligence. This article identifies multiple factors that may influence trust in autonomous systems. Trust may be undermined by various sources of demand and uncertainty. These include the cognitive demands resulting from the complexity and unpredictability of the system, “social” demands resulting from the system’s capacity to function as a team-member, and self-regulative demands associated with perceived threats to personal competence. It is proposed that existing gaps in trust research may be addressed using simulation methodologies. A simulated environment developed by the research team is described. It represents a “town-clearing” task in which the human operator teams with a robot that can be equipped with various sensors, and software for intelligent analysis of sensor data. The functionality of the simulator is illustrated, together with future research directions.

Mixed Reality (MxR) technologies have previously been explored in military applications oriented towards supporting both individual situational awareness and team collaborations. Ongoing technological advances in MxR have expanded its potential usage by military analyst teams to view, digest, and evaluate information from multiple data sources for uncovering adversarial behavior. Towards facilitating improved situational awareness, MxR is a promising medium for explaining patterns in data to uncover vital information. By extension, analyst collaborations conducted using MxR may further enhance collaborative decision making techniques used in military settings. In general, explanations provide summarizations or descriptive information that supports conclusions, depending on the desired level of abstraction. However, explanations that summarize information may not always preserve the underlying uncertainty present in the data. This work proposes a fused reason-based explanation technique for MxR that may help bring clarity to data where patterns may be unexpected, potentially revealing adversarial behavior.

This work provides an overview of a future human-agent collaborative decision-making framework to be developed for naval systems using an augmented reality platform. We present the basic concept behind the framework, key features of the application, and some details about a future proof of concept prototype that will demonstrate and evaluate the concept against a baseline design.

Designing and testing new information and safety features for industrial vehicles do not need to involve the realization of high-fidelity and expensive simulators. We propose a low-cost mixed reality environment which allows for rapid development and rearrangement of a virtual and physical setup of a simulator for industrial vehicles.

Our mixed reality simulator allows for safe testing of controls, information, and safety features to support drivers of industrial vehicles. In this paper, we test the implications of showing extra digital information to excavator drivers through a virtual environment, an external head-up display as well as a head-down display. Through user tests we have seen first indications that projected information through our mixed reality system and content on a head-up display is perceived as more helpful and intuitive than using head-down displays, when controlling our industrial vehicle simulator. Moreover, we have seen that the fear of overseeing an obstacle or other important information is lower when using a head-up display, in comparison to other tested visualization options.