Augmented Reality, Virtual Reality, and Computer Graphics

In this paper, the authors developed a multimedia SCORM courseware player that supports Web AR and Web VR. It was designed based on the Sco units and integrated Web AR and Web VR. This courseware player made up for the flexibility of AR and VR applications in SCORM, and can also expand the possibility of application in teaching practice.

The paper reports ongoing research toward the design of multimodal affective pedagogical agents that are effective for different types of learners and applications. In particular, the work reported in the paper investigated the extent to which the type of character design (realistic versus stylized) affects students’ perception of an animated agent’s facial emotions, and whether the effects are moderated by learner characteristics (e.g. gender). Eighty-two participants viewed 10 animation clips featuring a stylized character exhibiting 5 different emotions, e.g. happiness, sadness, fear, surprise and anger (2 clips per emotion), and 10 clips featuring a realistic character portraying the same emotional states. The participants were asked to name the emotions and rate their sincerity, intensity, and typicality. The results indicated that for recognition, participants were slightly more likely to recognize the emotions displayed by the stylized agent, although the difference was not statistically significant. The stylized agent was on average rated significantly higher for facial emotion intensity, whereas the differences in ratings for typicality and sincerity across all emotions were not statistically significant. A significant difference in ratings was shown in regard to sadness (within typicality), happiness (within sincerity), fear, anger, sadness and happiness (within intensity) with the stylized agent rated higher. Gender was not a significant correlate across all emotions or for individual emotions.

Unboxing a product, i.e. removing a newly acquired product from its packaging, often is a momentary strong emotional experience. Therefore, it is increasingly regarded as a critical aspect of packaging design. This study examines the user interaction when unboxing a product in Virtual Reality (VR) by comparing three plain textured boxes, each with a unique packaging design, and by inquiring the participants’ emotions provoked by each design. The boxes were created based on the following interaction design guidelines: freedom of interaction, interaction pattern, and richness of motor actions. The findings indicate that there is a difference between the strength of certain positive emotions when unboxing the three boxes in VR. This study discusses the relationships of certain interactions in VR, and how they can shape people’s expectations. These findings can serve as guidelines for how designers can improve the virtual experience through different interaction methods, and they provide insights into the differences between the unboxing experience in the real and the virtual world.

In this paper, we present a novel web-based collaborative data visualization approach combining the output of multiple rendering methods into a final seamless merged visualization. We have extended an existing web-based visualization framework Med3D with support for additional visualization plugins and an option for merging the outputs into a seamless final visualization. Such an approach allows users to place the anchors of 3D pinned annotations into volumetric data by ray casting mouse position to selected mesh geometry. The presented web-based collaborative volumetric data visualization framework is to the best of our knowledge the first of it’s kind using state-of-the-art volumetric rendering system.

Virtual reality technologies offer several intuitive and natural interactions that can be used to enhance and increase awareness during the different phases of the product design, such as the virtual prototyping or the project reviews. In this work, we focus on the benefits of virtual reality environments for assembly retrieval applications, which usually are employed to reuse existing CAD models by operating minor modifications. The retrieval system adopted to evaluate the different similarity among CAD assemblies is able to detect three different similarity criteria (i.e. shape, joint and position) at different levels (globally and locally). Considering the complexity of assembly models, conveying these different types of information simultaneously is quite difficult, especially through the traditional desktop systems. Hence, this paper presents a system for the visualization and the inspection of the results returned by a retrieval system with the aim of easing the analysis especially when the retrieved objects to analyze are complex. The proposed tool exploits the 3D space to represent and communicate the different types of similarity and gestures as well as voice commands to interact in the VR environment. Finally, the usability of the system has been tested over a sample of interviewers, who used the system and the different functionalities provided and then expressed their judgment filling out a questionnaire. The results indicate that user are able to interpret the different similarity criteria and that the gestures help acting in a natural manner.

The paper discusses research and development in the field of software visualization based on virtual reality environments. Spatial metaphors play an important role in such systems. A brief overview of the projects of software visualization systems based on vir- tual reality is provided. Among the systems developed over the past decades, one can find systems both for program visualization and for visual programming. Descriptions of prototypes of software visualization systems, software objects visualization and supercomputer performance data visualization, realized by the authors of the paper, are presented. These prototypes, designed for virtual reality environments, were developed with the use of several versions of a cosmic space metaphor and an extended city metaphor. The paper also discusses psychological aspects of the human factor in developing software visualization systems with the use of virtual reality.

One of the all time issues with Virtual Reality systems regardless if they are head-mounted or projection based is that they can only provide perspective correctness to one user. This limitation affects collaborative work which is the standard in any industry. Several approaches have been proposed to generate perspective correct images for different users but not only are they highly complex but also require lots of custom circuitry. On this paper we present the design, architecture and the mathematical background of an affordable optical routing multi user VR system that uses lenticular lenses for separating users and providing perspective correct images.

It is clear that in recent years have emerged countless simulators in all possible work areas for reaching the goals of Industry 4.0. Unity Pro software focuses on studying the options and possibilities offered virtual environments for the teaching in schools and universities. The aim of this paper is intended to create a practical training environment for the simulation and control of a pneumatic process for the students of Industrial Process Engineering of Automation. This virtual environment uses MQTT protocol into a Raspberry Pi to transmit instruction for control a real FESTO pneumatic lab based on what students develop during their virtual classes, granting and facilitating the student and teacher a new method that accelerates the learning process.

The article deals with simulation of fluids in the digital environment. It provides a brief overview of the use and various ways of simulation and displaying results and stresses the possibility of particle representation through particle systems and describes the particle method Smoothed Particle Hydrodynamics simulation. It focusses on description of the implementation of this method using the possibilities of a modern programmable graphics processing unit (GPU) in both the simulation and the visualisation process. For implementation of multi pass GPU algorithms the special data representations of simulation and rendering properties are designed.

It is well understood that multi-sensory stimulation can be used to enhance immersion in virtual environments such as Virtual Reality (VR) and Augmented Reality (AR). State of the art VR technologies have enhanced visual stimulation in aspects such as pixel density in recent years, whilst the area of haptics has remained less developed. The Ultrahaptics Evaluation Kit is a relatively new technology that consists of a \(16\times 16\) array of ultrasound transducers used to create ultrasound haptic sensations. We have developed a proof of concept large scale haptic system by integrating this device with an HTC VIVE, Leap Motion and Rethink Robotics’ Baxter Robot to provide ultrasound haptic feedback in a volume greater than 1.5 m\(^3\) for VR users. The system was evaluated through a user study with 19 participants. The study focused on users’ assessments of the location of the haptics produced by the system. The results of the study offer a means of validating the system, as well as providing comparisons in accuracy for a haptic perception task vs a visual perception task. There have been opportunities recognised for improving accuracy of the system. However, the system has been deemed suitable in creating haptic feedback for low fidelity models within large volumes in VR.

This article describes the development of a virtual reality application that allows the application of psychometric tests to drivers, both professional and non-professional, who obtain their driver’s license for the first time or are in the process of renewing or changing the category of their license; these implemented evaluations are aimed at measuring the reactions of drivers, as various audio-visual stimuli are applied in a more user-friendly manner. These implemented tests were developed in function of the traditional tests, that is to say complying with the objectives to be measured and evaluated; these evaluations have been developed with a Unity 3D graphic engine able to simulate the practical driving tests, to obtain results a numerical calculation software is used that presents graphs and statistics of the results obtained. The application is very attractive and allows users to perform their evaluations in a way that is close to reality, the tasks to be developed are performed with equipment that are very common for the user such as the steering wheel and pedals.

In this paper we suggest to use new gestures in controlling of the objects in virtual physical laboratory. These custom gestures are based on Leap Motion controller SDK and allows creating any of desirable hand motions. Such an approach allows creating the most intuitive and friendly interaction between human and software. In educational applications as in our case, this way of interaction provides the highest level of knowledge adoption because of native way of information purchasing.

In this paper, we describe a new approach to designing architectural spaces, called SADE, in which virtual reality and logic programming techniques are used to support and simplify the architectural design process. In the presented system, users have the possibility to design, configure and visualize architectural spaces within an immersive virtual reality environment, which helps them get better understanding of the created spaces. The system simplifies the design process by taking into account formal design rules, which describe domain knowledge, such as the construction law, technical conditions, design patterns, as well as preferences of a designer. The use of additional knowledge represented in the design rules can significantly shorten and improve the design process.

An adequate warm-up prior to intensive exercise can bring benefits to athletes, these requirements may vary depending on the physical activity and the training needs. The immersive virtual reality could have benefits in the warm-up and it can be determined by physiological data of the athlete. This work presents a mobile virtual reality application to stimulate the warm-up of an athlete using a standard treadmill, where the developed application is composed of a pleasant and stimulating environment. A Smartphone and the Gear VR are used as an HMD device, and wireless headphones are placed in the users, in addition, the heart rate of the athletes is monitored using a Polar H7 sensor. Experimental results are obtained in athletes with similar characteristics and conditions, identified a direct relation of the virtual environment with the pulsations per minute (ppm), denoting pulsations greater than usual in case of the stimulating environment and lower pulsations for the pleasant environment. Finally, a usability test is performed that shows the level of sociability of the system.

This paper presents a transdisciplinary view on virtual space, through a description of how different domains of knowledge inform the concepts of virtuality and space. The aim is to show how these different perspectives come together in the virtual space that facilitates combining science and technology with cultural aspects coming from arts and other domains of knowledge. The argument leads to two models of the understanding of virtual space. The first model is an explanation of virtual space as a hybrid that has emerged from both nature (represented by sciences) and culture (represented by arts). The second model puts the observer in the center, exploring the physical-virtual space through an embodied interaction. The contribution of this paper is twofold. First, it presents virtual space as a platform for transdisciplinary work, exposing its underlying processes from both theoretical and practical point of view. Second, it introduces a model for the way transdisciplinarity can inform the understanding of virtuality that is taking increasing part of our everyday lives as well as variety of knowledge production in form of advanced visualizations, simulations and virtual reality approaches.

3D point clouds represent a structured collection of elementary geometrical primitives. They can characterize size, shape, orientation and position of objects in space. In the field of building modelling and Cultural Heritage documentation and preservation, the classification and segmentation of point clouds result challenging because of the complexity and variety of point clouds due to irregular sampling, varying density, different types of objects. After moving into the era of multimedia big data, machine-learning approaches evolved into deep learning approaches, which are a more powerful and efficient way of dealing with the complexity of semantic object classification. Despite the great benefits that such approaches brought in automation, a great obstacle is to generate enough training data, which are nowadays manually labeled. This task results time-consuming for two reasons: the variety of point density and geometry, which are typical for the Cultural Heritage domain. In order to accelerate the development of powerful algorithms for CH point cloud classification, in this paper, it is presented a novel framework for automatic generation of synthetic dataset of point clouds. This task is performed using Blender, an open source software which permits to access to each point in an object creating one in a new mesh. The algorithms described allow to create a great number of point cloud synthetically, simulating a virtual laser scanner at a variable distance. Furthermore, these two algorithms not only work with a single object, but it is possible to create simultaneously many point clouds from a scene in Blender also with the use of an existing model of ancient architectures.

Virtual stores and showrooms gain increasing attention in e-commerce, marketing and merchandising to investigate customers’ behavior, preferences and the usefulness of shopping and exhibition spaces. Although virtual stores may be designed using numerous available 3D modeling tools and game engines, efficient methods and tools enabling development and personalization of virtual stores are still lacking. In this paper, we propose a novel approach to the development of personalizable contextual virtual stores that can be generated and configured on-demand, using interfaces based on semantic web technologies. A virtual store model is created as a combination of three elements: an exposition model, a collection of product models, and a virtual store configuration. The first element visually reflects an existing or imaginary 3D store layout. The second element contains 3D models of all products that can be presented in the exposition. The third element is an ontology, which connects the two previous elements using domain-specific knowledge and reasoning. Based on a virtual store model, a personalized virtual store is generated in response to a specific user’s request.

Virtual Reality (VR) is a powerful modern medium for immersive data visualization and interaction. However, only a few research efforts targeted the issue of complementing VR applications with features derived from real-time human behavior analysis in virtual environments. This paper addresses an interactive application for analysis of user behavior in a VR environment. In this work, real-time data communication is employed to collect data about the VR user’s location and actions in the virtual environment. To ensure the authenticity of interactions in the virtual environment, the VR application aims at achieving complete immersion. Our findings pertaining to behavioral patterns in immersive environment suggest that there is a potential in applying knowledge of user behavior models to improve the interactivity between the user and the virtual environment. Analysis of VR users’ behavioral models also complements studies typically performed by traditional survey techniques.

In this paper we present Bestbrau, a reactive programming framework for the development of 3d real-time applications. We show its most relevant features, explaining how it differs from the systems used nowadays in commercial game engines and underlining which new functionalities it would provide to a 3D engine integrating it. Our framework is entirely written in C++17 and has been tested on the SAM Engine, a work-in-progress 3D game engine that we have designed and developed in parallel with the reactive framework itself. Bestbrau features are also compared with those offered by Unreal Engine 4 and Unity 2019, two of the most important commercial game engines today available. After a brief introduction, in the second section we briefly introduce the generalities of reactive programming, in the third section we focus, instead, on the FRP (Functional Reactive Programming). In the latest sections we make an accurate and exhaustive description of the peculiarities of such a framework and of the opportunities it provides to 3D and VR programming.

This paper has the purpose to show a stabilization video-streaming methodology feasible to low-power wearable devices. Thanks to an Inertial Measurement Unit (IMU) mounted together with the camera we are ready to stabilize directly on video-stream without the delay and the complexity due to image processing used by classic software stabilization techniques. The IMU gives information about the angle rotation respect to the three main orthogonal axes of the camera; the wearable device transmits the video along with the IMU data synchronized frame per frame then a base station receives and stabilizes while renders the video. The result is that the shaking and the unwanted motions of the human body wearing the system are compensated giving a clear and stable video. Numeric results prove that the video is more stable: we cut-off the half of the motion noise in the scene.

Virtual Reality (VR) applications are getting progressively more popular in many disciplines. This is mainly due to (1) technological advancement in rendering images and sounds, and (2) immersive experience of VR which intensifies users’ engagement leading to improved joy, sense of presence and success. Although VR creates a realistic experience by simulating visual and auditory sense, it ignores other human senses. The absence of haptic feedback in Virtual Environments (VE) can impair users’ engagement. Our objective is to provide haptic feedback in VE using the most affordable method, proxy haptics. Proxies are physical objects which are co-registered with virtual objects in a way that by touching the proxies, users see their virtual hand touching the virtual objects. So, the haptic feedback will be applied to users’ hands by touching the proxies. The important question which arises here is: what are the influences of providing haptic feedback on users’ performance and usability? In this work, we have designed an experiment in Virtual Reality to answer this question. In the virtual environment, we presented a keyboard with touchable keys and asked our participants to type three words using the keyboard. They repeated the experiment twice: once they used their own hands and pressed the keys and once they used VR controllers for typing. Our results show that participants can type words with fewer errors when they are using their own hands while they found VR controllers easier to use.

Augmented Reality (AR) has a high potential to help healthcare professionals in the operating room and researchers have been working on bringing AR to the operating room for many different types of surgeries so far. However, there are thousands of surgery types that might benefit from the AR. Our goal is to identify which surgeries could benefit the most from AR and identify the common functionality desired in an AR app for surgery and then come up with a unified user interface (UI) design for these surgeries. To achieve this goal, we analyzed the historical number of surgeries performed at a teaching and research hospital to find the most frequently performed surgeries. Starting with the most frequently performed surgeries, we collaborated with the experienced surgeons to find out the surgeries that could benefit from AR the most. For each surgery, we identified the functionality and the information that could be presented on an AR app before, during and after surgery. We prepared the list of surgeries that might benefit from AR through a series of thorough interviews with experienced surgeons. We also identified the functionalities desired in AR apps for surgery and prepared a unified UI design for the common functionality.

We believe that the unified UI that we designed for the AR App of surgical procedures is the first step towards an AR app that might be beneficial for many types of surgeries. We believe surgeons equipped with AR apps will achieve better results; the success rates of surgical procedures will increase while the number of errors decreases.

Technological improvements have contributed to the development of new innovative approaches to rehabilitation. The use of virtual reality (VR) in physical and cognitive rehabilitation brings several advantages, such as greater motivation and patient enjoyment, allowing for the performance of everyday life tasks in a safe and controlled environment. This work presents a virtual supermarket to be experienced with the HTC Vive Head Mounted Display, aiming at the training of post-stroke patients. The immersive virtual environment was validated on a sample of healthy subjects; a comparison was made with the previous software version of the virtual supermarket developed to train cognitive capabilities in elderly users. System usability, cyber-sickness and sense of presence have been investigated to highlight desirable modifications before a final assessment on clinical patients.

In this article an application is developed based on 3D environments for the upper limbs rehabilitation, with the aim of performing the measurement of rehabilitation movements that the patient makes. A robotic glove is used for virtualized the movements with the hand. The hand movements are sent to a mathematical processing software which runs an algorithm to determine if the rehabilitation movement is right. Through virtual reality environments, the injured patients see the correct way to perform the movement and also shows the movements that the patient makes with the robotic glove prototype. This system allows to evaluate the protocol of upper limbs rehabilitation, with the continuous use of this system the injured patient can see how his condition evolves after performing several times the proposed virtual tasks.

Shoulder pain is very common in adult population (16%–26%) and it has several impacts on the Activities of Daily Living (ADLs). In this context, an effective rehabilitation is needed to assist the patients to regain autonomy and improve ADLs.

An alternative approach to monotonous traditional rehabilitation can be offered by Virtual Reality (VR) technology and video game console systems.

In this paper, we present a proof of concept of a computer game for shoulder rehabilitation. The system consists in a VR application developed with the Unity game engine, and it is currently controlled via mouse interactions; however, the software application has been designed for the Nintendo Wii Remote MotionPlus as the main game controller, but this tool will be integrated in the next development phase.

Our goal is to develop a rehabilitation game: suitable to be used both at home and at a hospital, highly motivating for the patient, and requiring low-cost technology (e.g. a consumer-grade tracking system).

The game has been tested preliminary with three healthy subjects who agreed that the application is engaging, motivating and intuitive.

The quantification of Bell’s palsy using virtual reality, haptic devices and other technologies has been the subject of numerous investigations in recent years, however those affected with facial palsy have not benefited from efficient diagnoses that would give the physician the confidence to deliver an efficient clinical picture. GYMEDIC in its initial version developed modules of analysis (House-Brackmann System) and rehabilitation (virtual environments), previous research that comes to be the key to implement new Nottingham Scale graduation systems manipulating interactive virtual environments in order to improve figures of the diagnosis of Bell’s palsy. This system develops new features for the estimation of facial paralysis: (a) symmetry at rest, (b) symmetry at adjusted rest, (c) symmetry with voluntary facial movements, which allows the inclusion of new measurement scales to the system. In addition, as an interesting point to research, AI is implemented through a simple neuronal network, which allows to adjust figures in three areas: eyebrows, eyes and mouth and therefore improve the diagnosis of patients.

This article presents an alternative rehabilitation system based on a visual feedback system for people suffering from cerebral palsy disorder. The proposed feedback system handles textures and movements in a 3D graphic environment, specially designed to develop skills that improve patient performance. The interface is developed in the Unity3D software, identifying patterns of the body is done through motion Kinect and validation of the correct execution of the exercises sensor is carried out, using the technique of machine learning for training rehabilitation system. The experimental results show the efficiency of the system that generates an improvement in the motor abilities of the upper and lower extremities of the patient.

VR navigation systems have emerged as a particularly useful tool for the guidance of minimally invasive surgical procedures to restore the 3D perception of the surgical field and augment the visual information available to the surgical team. Over the past decade, X-ray free VR navigation systems based on Electromagnetic (EM) tracking have been proposed to guide catheter-based minimally invasive surgical procedures, including transcatheter valve implantation, with a reduced intra-procedural radiation exposure and contrast medium injections. One of the major limits of current navigation platforms is the lack or the poor reliability/efficiency of the real-time modeling unit to reproduce the deformations of the cardiovascular anatomy and surgical instruments. In this work we propose an EM navigation platform, with software functionalities, developed using the Unity game engine, to mimic physical behaviors of surgical instruments inside the patient vasculature, handle the interaction between the tools and the anatomical model via collision detection, and (in the next version of the software) deform in real-time the anatomy. Additionally, the technology used (Unity) can allow a seamless integration of AR headset to further improve the ergonomics of the 3D scene visualization.

Being able to travel in public transportation autonomously is one daily task that doesn’t imply any special effort or learning process for most of the human population. But, for people with intellectual disabilities, it requires an extensive learning process where many possible situations should be considered before the person can perform successfully this task. Currently these people are taught by specialists following supervised learning methodologies in real environments. But, through the use of immersive virtual reality and interaction within a virtual world, they could be trained in a safe environment, increasing their self-confidence in a saver and quicker way. This paper describes the creation of a customized simulator that prepares subjects to use the bus as a means of transportation in a medium-size city. The design of the virtual reality environment and the interaction mechanics is thought to be highly customizable for each subject with a very realistic environment, thanks to the use of photogrammetric digitalization technologies. The environment has been developed in Unreal engine and uses the head mounted display Oculus rift, as well as Oculus touch for interaction with the different assets.

Attention-deficit hyperactivity disorder (ADHD) is a neurodevelopmental disorder that is expressed through different symptoms belonging to three different dimensions: inattention, impulsivity and motor hyperactivity, each of which contributes to the learning and adaptation problems within the different contexts of life. ADHD children have to focus on three main elements: learn to self-control, make and keep friends and feel good about themselves. The BRAVO (Beyond the tReatment of the Attention deficit hyperactiVity disOrder) project aims to realize an immersive therapeutic game context, based on an innovative ICT system, with which improving the relationship between young patients and therapies (administered by means of serious games and gamification). By using wearable equipment and Virtual and Augmented Reality devices, new personalized processes of therapy will be implemented. Such processes will be able to dynamically change in order to follow the patients evolution and support the therapists in the rehabilitation program management.