Augmented Reality, Virtual Reality, and Computer Graphics

In this paper, we present a web-based system for visualization of flow simulation results in the vascular system for use with consumer-level hardware. The presented tool allows users to design, execute and visualize a flow simulation with a simple workflow on a desktop computer or a mobile device. The web interface allows users to select a vascular model, define the flow simulation parameters, execute the simulation, and interactively visualize the simulation results in real time using multiple visualization techniques. The server-side prepares the model for simulation and performs the simulation using SimVascular. To provide a more efficient transfer of the large amounts of simulation results to the web client, as well as reduce storage requirements on the server, we introduce a novel hybrid lossy compression method. The method uses an octree data subdivision approach combined with an iterative approach that regresses the data points to a B-Spline volume. The evaluation results show that our method achieves compression ratios of up to 5.7 for the tested examples at a given error rate, comparable to other approaches while specifically intended for visualization purposes.

This work presents a system for strengthening upper limbs in children through an interactive videogame system and the use of myoelectric muscle sensors. The system allows the acquisition of myoelectric signals taken by electrodes placed in the muscles of interest so they are sent to the computer to be visualized in the virtual interface. Several virtual interfaces were developed in the Unity 3D graphical engine in which the degree of difficulty of the videogame can be selected as well as the muscle affectation and the duration of the repetitions of each exercise. User personal data is stored in a data sheet. The data transmission is carried out using Bluetooth wireless technology in charge of establishing a reliable and real-time communication. Tests were performed on 5 users (3 boys and 2 girls) with ages between 6 to 12 years, and the SUS usability test was applied with results (84.5 ± 0.62), which allows to determine that the system has a good acceptance to be used in muscle strengthening.

We present a 3D virtual system for children’s fine motor rehabilitation, it is created three environments with the software Unity. These environments generate playful and entertaining backgrounds; several tests for the system operation are performed which include working with children with ages of 5 to 14 years. The results allow to determine the fine motor movements, in addition it is determined that the trajectories made by children’s fingers when performing the games are suitable for fine motor rehabilitation. Finally, the usability test SEQ is also performed, which give us results of (56.4 ± 0.37), this shows the user’s acceptation of the system.

Musculo Skeletal Disorders (MSDs) is the most common disease in the workplaces causing disabilities and excessive costs to industries, particularly in EU countries. Most of MSDs prevention programs have focused on a combination of interventions including training to change individual behaviors (such as awkward postures). However, little evidence proves that current training approach on awkward postures is efficient and can significantly reduce MSDs symptoms. Therefore, dealing with awkward postures and repetitive tasks is the real challenge for practitioners and manufacturers, knowing that the amount of risk exposure varies increasingly among workers depending on their attitude and expertise as well as on their strategy to perform the task. The progress in MSDs prevention might come through developing new tools that inform workers more efficiently on their gestures and postures. This paper proposes a potential Serious Game that immerses industrial workers using Virtual Reality and helps them recognize their strategy while performing tasks and trains them to find the most efficient and least risky tactics.

This work presents the development of an interactive virtual rehabilitation system as an assistant tool to help in the rehabilitation process of lower limbs muscles, specifically focused on children from seven years and older who suffer sicknesses that limit the normal movement of the body. The system is mainly based in algorithms that recognize gestures of the user captured through the Microsoft Kinect 2.0. Furthermore, the experimental results are presented and discussed since the point of view of the usability and the advantages that the proposed system achieves.

This article describes a virtual rehabilitation system with work and entertainment environments to treat fine motor injuries through an active orthosis. The system was developed in the Unity 3D graphic engine, which allows the patient greater immersion in the rehabilitation process through proposed activities; to identify the movement performed, the Myo armband is used, a device capable of receiving and sending the signals obtained to a mathematical algorithm which will classify these signals and activate the physical hand orthosis completing the desired movement. The benefits of the system is the optimization of resources, infrastructure and personnel, since the therapy will be assisted by the same virtual environment, in addition it allows selecting the virtual environment and the activity to be carried out according to the disability present in the patient. The results show the correct functioning of the system performed.

In mini-invasive surgery and in endoscopic procedures, the surgeon operates without a direct visualization of the patient’s anatomy. In image-guided surgery, solutions based on wearable augmented reality (AR) represent the most promising ones. The authors describe the characteristics that an ideal Head Mounted Display (HMD) must have to guarantee safety and accuracy in AR-guided neurosurgical interventions and design the ideal virtual content for guiding crucial task in neuro endoscopic surgery. The selected sequence of AR content to obtain an effective guidance during surgery is tested in a Microsoft Hololens based app.

This research shows an alternative device in the field of fine motor rehabilitation. This device is a flexible orthosis that helps the flexion and contraction of the hand fingers using a virtual interface environment. In addition, it has been designed with the ability to perform different assistive and resistive tasks, allowing its adaptation to the recovery status of the patient. The mechatronic prototype is controlled by algorithms based on fuzzy logic that compares data from the Unity3D graphics engine and flex sensors from the device. For the correct execution of the rehabilitation tasks the proposed fuzzy algorithms have been implemented using a Raspberry Pi. The proposed system is aimed to users with deficits in fine motor skills because of tendon injuries, achieving excellent control results for an efficient execution of tasks.

This paper presents a virtual reality app for pulmonary diseases treatment in children through VR game, which consists of visual and auditory stimulation thus the therapy won’t be annoying to the patient. The procedure consists of a previous stage of respiratory training which the patient is asked for a deep inhalation followed by an exhalation, both within a period of 3 to 6 s and requested 3 times. The app will offer the patient different activities in a virtual environment depending on the inhalation and exhalation exercise required by the stages of their therapy, using HTC VIVE virtual reality glasses. Recognition of inhalation and exhalation exercises is performed using a DTW classifier running in real time in MATLAB and with a direct, two-way connection to Unity. Therefore, the app encourages the patient to perform the exercises therapy in an accurate and fun way and as a result respiratory signal graphs are shown for evaluation by the therapist.

This paper proposes a tool to support the rehabilitation of upper limbs assisted remotely, which makes it possible for the physiotherapist to be able to assist and supervise the therapy to patients who can not go to rehabilitation centers. This virtual system for real-time tele-rehabilitation is non-invasive and focuses on involving the patient with mild or moderate mobility alterations within a dynamic therapy based on virtual games; Haptics Devices are used to reeducate and stimulate the movement of the upper extremities, at the same time that both motor skills and Visual-Motor Integration skills are developed. The system contains a virtual interface that emulates real-world environments and activities. The functionality of the Novint Falcon device is exploited to send a feedback response that corrects and stimulates the patient to perform the therapy session correctly. In addition, the therapy session can vary in intensity through the levels presented by the application, and the amount of time, successes and mistakes made by the patient are registered in a database. The first results show the acceptance of the virtual system designed for real-time tele-rehabilitation.

The present paper proposes a system oriented to the rehabilitation of carpal tunnel syndrome by means of the use of haptic devices with force feedback. The system, based on entertainment, handles different applications, as well as movements in a 3D graphic environment, which is devised with daily tasks that allow to develop skills and abilities to reduce patient´s affection. The system is designed with the integration of Unity3D, as well as Novint Falcon haptic device. In which, the patient interacts with the developed applications while receives force feedback. At the same time, the patient performs physiotherapeutic exercises that attack the affection in a proper manner to improve patient’s health. Experimental results manifest the system’s validity, which generates necessary and efficient exercises for the process of carpal tunnel rehabilitation. In addition, the system deploys a human – machine interaction oriented to the development of physic therapies.

This article proposes an application of assistance for speech and language therapies, aimed at children suffering from this pathology. The system was implemented in a virtual environment developed with the graphics engine Unity 3D, in order to provide an application that optimizes both the use of materials, infrastructure, time resources, among others. The developed system consists of five games based on Speech and Language Therapy, SLT, which encourage the child to perform the different exercises of the five areas of this disorder, hearing, structure and oral function, linguistic formulation, expressive language and Articulation, receptive language. The proposed tool allows the user to select both the work environment and the level of difficulty during the therapy process.

This article describes an analysis of the merchandising techniques used in Neuromarketing through a virtual reality application to inquire the consumer behavior when purchasing mass consumption products. The results of the virtual test will affect the costs optimization, reduction of time, and logistics when studying the effectiveness of each technique. The application consists of a virtual environment replica of a real supermarket, which allows the user to interact with the products according to: the conscious and unconscious perception at a purchase and the influence of merchandising techniques (location of the products, assortment management and accessibility, and so on). The results of the experiment validate the merchandising techniques with product rotation processes and sales increase, without study them in a real supermarket.

We present an overview of IMPACT, a system that uses virtual and augmented reality to enhance the mirror therapy for the treatment of phantom limb pain and stroke rehabilitation. The system tracks users’ movements and the head position using a combination of depth cameras and inertial sensors. This information is then fused together in a 3D full-body human model able to adapt to the patient’s characteristics (e.g., skin color, limb size) and avoid unnatural movements due to possible bad tracking (e.g., a joint bended in an unnatural way). IMPACT includes three serious games that can be played in first-person point of view and that allow different levels of immersion according to the needs of the treatment.

Surgical simulation can offer novice surgeons an opportunity to practice skills outside the operating theatre in a safe controlled environment. According to literature evidence, nowadays there are very few training simulators available for Hip Arthroplasty (HA).In a previous study we have presented a physical simulator based on a lower torso phantom including a patient-specific hemi-pelvis replica embedded in a soft synthetic foam. This work explores the use of Microsoft HoloLens technology to enrich the physical patient-specific simulation with the implementation of wearable mixed reality functionalities. Our HA multimodal simulator based on mixed reality using the HoloLens is described by illustrating the overall system, and by summarizing the main phases of the design and development.Finally, we present a preliminary qualitative study with seven subjects (5 medical students, and 2 orthopedic surgeons) showing encouraging results that suggest the suitability of the HoloLens for the proposed application. However, further studies need to be conducted to perform a quantitative test of the registration accuracy of the virtual content, and to confirm qualitative results in a larger cohort of subjects.

This paper presents a fine motor rehabilitation system for upper limbs by using a virtual reality environment. For this purpose, a glove of stimulating bilateral haptic is built, which allows directly to determine the finger’s position through flexibility sensors. Also stimulate the medium and ulnar nerves of hand’s palm by using vibratory actuators in charge of feedback to contact with virtual surfaces. This system is based on bilateral communication between the virtual environment in the Unity 3D graphics engine and the haptic glove. It is responsible for analyzing the movements used by the patient and interact with the Oculus Rift and Leap Motion for an increased immersion of the patient in the virtual rehabilitation environment. In addition, it generates vibrating feedback submitted to contact with virtual objects. The connection and transmission of data is done through wireless technologies in charge of creating a reliable and real time communication. The patient performs exercises based on fine motor rehabilitation which they are feedback with haptic glove and validated by algorithms based on Euclidean distance. The experimental results show the correct operation of the glove and the virtual environments oriented to virtual rehabilitation systems.

The development of simulation systems from surgical procedures has been a research topic in several areas in medicine and engineer applications because it supposes a novel alternative for medical skills acquisition, surgical planning, guide during surgery and postoperative control. At the same time, simulation systems represent significant challenges, regarding conceptual design, mathematical, numeric and computational modelling, and finally the validation of the system. In this paper, we present the advances and methodologies applied for the development of a virtual reality system for medical training in neurosurgery. As the case of study, we present the simulation of an aneurysm clipping procedure in two of the main stages: brain Sylvian fissure exploration and aneurysm clipping.

This paper describes the development of an interactive and motivational tool, to give immersion in the cognitive therapeutic exercise (Perfetti method). This system is implemented with the use of virtual environments developed in the Unity 3D graphic engine. The environments present friendly, novel designs, which are shown to the user in augmented reality with the help of a high-end smartphone and virtual reality headset. In addition, the system helps in the process of recording activities and collecting important data for the monitoring and evolution of the users.

The Six Steeds of Zhao Mausoleum were world-renowned for its realistic style and typical ancient Chinese carving skills. They were carved in memory of the six horses belong to the Emperor Taizong of Tang Dynasty and placed on the north slope of Jiuzong Mountain where the mausoleum of emperor located in. By thousand years efflorescing and destructing, they were seriously damaged, and the majority of the surface details were lost. It is urgent and significant to reconstruct the missing parts and give the entire embossments an integrated vision. Cause of the important historical values and complex artistic surface shape, it is hard to reliably restore the missing parts only using the technical method or artist’s method. By combining mathematic method and sculpture modeling method, this paper put forward a way to restore the damaged embossments in virtual space. It is the first time to give the Six Steeds a trustworthy full sight both in the point of artistic style and archaeology; what’s more, the output data of the research can convert into real parts via the CNC machining equipment.

The application of the VR to the renderization of ancient and lost buildings is one important topic in the cultural heritage field nowadays. Moreover, the addition of other senses apart from the graphical view, increases the sensation of immersion in a virtual application. The aim of this paper is to show the work on a virtual acoustic system, based on Unity, FMOD and Csound. This system has been implemented to render binaural auralizations and has been applied to the renderization of an old and lost theatre in València (Spain). This theatre was firstly build in the 16th century, and rebuilt several times until the 18th century. It has been modelled from ancient documents. The auralization of several theatrical excerpts of different Spanish authors of that time is also presented.

In many museums it is still common that visitors have to read static texts from boards to gain information about the exhibits. In times where almost every visitor carries a smartphone in their pocket, these devices could be utilized for a more personalized and interactive visitor experience. In this paper we present a design study for a “Bring your own device” setting that combines augmented reality (AR) and navigation in museums. We applied an iterative user centered design approach that included conceptual design, prototyping, user tests, as well as a field test in a large museum in Vienna. One of the main results is that a new and digital form of navigation isn’t as essential as the museum thought it would be. Apart from that the application was well received during the field test.

Virtual museums are becoming increasingly popular, especially thanks to the recent spread of low-cost immersive technologies enabling even small cultural realities to enrich their technology-based offer. The growing availability of computational power and high-quality visual elements enables the use of virtual characters in order to add depth to the virtual visit experience, up to nowadays often limited to the exploration of lifeless environments. This paper presents a pilot study aimed at investigating the positive effects that the use of avatars can provide to a virtual cultural experience, proposing a virtual museum with three different alternatives of storytelling, including one featuring virtual humans, and comparing the results in terms of engagement and understanding of the proposed content.

This article wants to present the Spatial Augmented Reality technique (hereafter S.A.R.), better known as video mapping, used by the Studio Glowarp on scale models of archeological finds and cultural heritage. The use of S.A.R. technique offers new communication ways for archeology and museum fields. It helps to show fragile or destroyed goods, sometimes far away from the exhibition set, for the listed reasons considered as fragmented and hard to be reached by a bigger audience, become interesting case studies of digital anastylosis. The aim of this paper is to show from a different point of view the S.A.R. method that now can be considered not only as an “artistic” technique but also as a technique serving the cultural heritage, and to recommend it as one of the possible means of the future restoration and of the augmented heritage field.

Augmented reality plays an important role in cultural heritage and education. The magic effect of 3D animated objects can enhance students’ interest, attention and engagement in school subjects. Moreover, augmented reality can help understand spatial concepts and the working principles of complex mechanical components. In such context, machines designed by Leonardo Da Vinci are a perfect case study for augmented reality technologies. This paper presents a mobile augmented reality application developed for an exhibit on Leonardo’s inventions: it shows the overall structure and the working principles of some machines recognized on the pages of the Atlantic Codex. We used Blender to design the animated 3D models of the machines and the ARToolkit and Vuforia frameworks to superimpose virtual models over Leonardo’s sketches framed by the camera of mobile devices. Subjective data about user experience collected through some questionnaires proved how AR applications can improve the dominant learning style in accordance with Kolb’s theory.

Technology could sensitize end-users to preservation of cultural heritage while stimulating more interest based on a greater involvement. Our approach to this goal is based on a novel gamification and story-telling based on scientific foundations. At the same time, this application can offer the pro-user an interactive visualization suitable to study and verify new theories about the artifacts both on their current state and on its 3D reconstruction, but, at the same time, design a workflow capable to integrate different optimization techniques to adjust data depending on different scenarios. This leads to a unified system able to adapt itself to external requirements in a semi-automatic fashion. Finally, new ways of playful interaction, based on simple mathematical operations, were investigated.The second-named author is partially supported by MIUR Excellence Department Project funds, awarded MIUR to the Department of Mathematics, University of Rome Tor Vergata, CUP E83C18000100006.

This paper explores the User Experience (UX) of Augmented Reality applications in museums. UX as a concept is vital to effective visual communication and interpretation in museums, and to enhance usability during a museum tour. In the project ‘MuseumEye’, the augmentations generated were localized based on a hybrid system that combines of (SLAM) markerless tracking technology and the indoor Beacons or Bluetooth Low Energy (BLE). These augmentations include a combination of multimedia content and different levels of visual information that required for museum visitors. Using mobile devices to pilot this application, we developed a UX design model that has the ability to evaluate the user experience and usability of the application. This paper focuses on the multidisciplinary outcomes of the project from both a technical and museological perspective based on public responses. A field evaluation of the AR system was conducted after the UX model considered. Twenty-six participants were recruited in Leeds museum and another twenty participants in the Egyptian museum in Cairo. Results showed positive responses on experiencing the system after adopting the UX design model. This study contributes on synthesizing a UX design model for AR applications to reach the optimum levels of user interaction required that reflects ultimately on the entire museum experience.

In this paper is presented an application of the Augmented Reality technology, for the enhancement of some Apulian archaeological areas, regarding the indigenous population of the Apulia region. The aim is to improve visitors’ knowledge about this population, in order to better understand the archaeological areas of the Museo Diffuso “Castello di Alceste”, in San Vito dei Normanni (BR), and the site of Fondo Giuliano, in Vaste (LE). A mobile application (for Android and iOS devices) has been developed for non-specialized audience; by focusing on a planimetric map of the archaeological areas, 3D models are superimposed on, and combined with other kinds of contents (texts, images, video, audio), in order to allow the visitors to fully understand the history and all of the features of the ancient sites.

In this paper we present Puzzle Battle, an arcade VR-based serious game for promoting and reflect on the history of painting art. The idea behind Puzzle Battle is about to see famous artists like Van Gogh or Frida Kahlo fight against each other or against other artists from different eras, in a classic puzzle game. The player will have to impersonate an artist and play in a fun context where he will improve his historical knowledge about the artist and his works by taking advantage of current studies on immersive learning experiences with virtual reality.

The use of immersive Virtual Reality (VR) technology is a relatively recent trend that is spreading in various areas such as, for example, the digital fruition of cultural heritage. The set of appropriately integrated hardware and software technologies are able to combine three-dimensional models of virtual or real objects obtained, for example, from integrated laser scanner and/or photogrammetric survey. If the user observes a computer monitor there is three-dimensional non-immersive VR to which the user accesses through devices with which he can controls what is displayed onto a flat screen. In the VR a true reality can be simulated, you can navigate and move in real-time in photorealistic environs and interact with the objects presents in them. This paper illustrates the Virtual Museum project, a system of digital fruition of cultural heritage with low-cost virtual reality techniques, under development by AESEI s.r.l. - Spin-off of Polytechnic of Bari. This software, entirely written in C++ language, is compatible with the 64-bit Windows operating systems and interfaces with an Oculus DK2 Head Mounted Display (HMD) device for immersive vision and with the Leap Motion device for gestures recognition.

Technology advancement is changing the way industrial factories have to face an increasingly complex and competitive market. The fourth industrial revolution (known as industry 4.0) is also changing how human workers have to carry out tasks and actions. In fact, it is no longer impossible to think of a scenario in which human operators and industrial robots work side-by-side, sharing the same environment and tools. To realize a safe work environment, workers should trust robots as well as they trust human operators. Such goal is indeed complex to achieve, especially when workers are under stress conditions, such as when a fault occurs and the human operators are no longer able to understand what is happening in the industrial manipulator. Indeed, Augmented Reality (AR) can help workers to visualize in real-time robots’ faults. This paper proposes an augmented system that assists human workers to recognize and visualize errors, improving their awareness of the system. The system has been tested using both an AR see-through device and a smartphone.

Cyber Physical System (CPS) together with Internet of Things, Big Data, Cloud Computing and Industrial Wireless Networks are the core technologies allowing the introduction of the fourth industrial revolution, Industry 4.0. Along with the advances in new generation information technologies, smart manufacturing is becoming the focus of global manufacturing transformation. Considering the competitive nature of industry, it requires manufacturers to implement new methodologies. Realistic virtual models mirroring the real world are becoming essential to bridge the gap between design and manufacturing. In this paper model conceptualization, representation, and implementation of the digital twin is presented, on the real use case of manufacturing industry and in the cyber physical environment. A novel CPS architecture for real time visualization of complex industrial process is proposed. It essentially considers the Simulation technological pillar of Industry 4.0. The results from a real industrial environment show good performances in terms of real time behaviour, virtual reality and WebGL CPS visualization features, usability and readability.

Assembly planning and evaluation are important phases of the product development process. The generation of assembly plans is often a lengthy and costly manual process with a direct impact on the time to market. Many attempts to improve this process through the adoption of Virtual Reality have been done in the last twenty years, however the adoption of these approaches in real industrial environments is still immature and not well documented in the available literature. This paper aims at evaluating, through the adoption of a case study research approach, the real benefits coming from the adoption of Virtual Assembly strategies in an industrial company operating in the Oil & Gas sector.

This work presents the development of an interactive virtual environment for training in emergencies that may arise in industrial environments. This environment contains equipment and usual structures in production plants with a high risk index, simulations of accidents and potential hazards that must be overcome by the user. The application has been created using a CAD tool and the Unity3D graphics engine that, combined with peripheral input devices, provide a high level of immersion, all in order to recreate the sensations that the person could feel in case of facing an adverse situation that requires the user knowledge and skills in security issues and protocols.

The need to implement architectures with a high degree of scalability, a low level of error and the preservation of the integrity of human beings in the oil industry, has led to the development of technologies that use tele operation, in addition to graphic interfaces that reach a total immersion of the user. To achieve this, it is necessary to use tools such as augmented reality and virtual reality, which help to make the transparency of any system infinite. This research presents the design of a tele-operation system that allows periodic inspections of equipment, maintenance tasks, or training of new personnel in the Well-Pads located in Petroamazonas EP, Ecuador. The transmission of data has been made through the MQTT protocol in order to use the lowest possible bandwidth and consume few resources. In the local site several environments of augmented reality and virtual reality have been implemented, this allows to transmit the skill of the operator to the slave robot through the senses of kinesthesia, sight and hearing implementing an operation based on the concept of Intelligent Oil Field.

The visualization of tele-operated and autonomous executions in the field becomes difficult if the real environments are located in remote areas or present potential dangers for visualizing clients. This work proposes an application based on virtual reality to recreate in real time the execution tasks of a UAV, which is operated remotely or autonomously on a real environment. To achieve a third level of immersion, the reconstruction of the real environment where the field tests are executed is considered, offering the possibility of knowing the real scenario where the tests are executed. The consideration of using commercial UAV development kits is taken into account to obtain internal information, as well as to control the drone from client devices. The results presented validate the unification of 3D models and the reconstruction of the environment, as well as the consumption of vehicle information and climate parameters.

This paper presents the implementation of a 3D virtual simulator that allows the analysis of the performance of different autonomous and tele-operated control strategies through the execution of service tasks by an aerial manipulator robot. The simulation environment is development through the digitalization of a real environment by means of 3D mapping with Drones that serves as a scenario to execute the tasks with a robot designed in CAD software. For robot-environment interaction, the Unity 3D graphics engine is used, which exchanges information with MATLAB to close the control loop and allow for feedback to compensate for the error. Finally, the results of the simulation, which validate the proposed control strategies, are presented and discussed.

This work presents the implementation of virtual environments oriented to managing pneumatic controls applied to industrial processes in order to strengthen training and teaching-learning processes. The implemented application enables the multi-user immersion and interaction with the aim to accomplish predefined tasks to be developed within lab environments and virtualized sceneries for industrial processes. Obtained results show how easy it is to interact with the proposed multi-user environment.

Currently, virtual reality is presented as a solution to the difficulties for teaching and training in industrial processes in the technical area, so a virtual environment was developed where trainers, users, teachers and students interact as the case may be to carry out processes in the automotive industry such as engine assembly and car body assembly. By carrying out these processes in a virtual environment, the aim is for users to gain skills and become familiar with the activities to be carried out in real life. The easy usability of accessories and the user-friendliness of the immersion in the virtual environment makes the participants have a good experience and in the future show their interest in continuing to use a virtual environment for training or acquiring new knowledge.

This paper is focusing on the development of a system based on computer vision to estimate the movement of an MAV (X, Y, Z and yaw). The system integrates elements such as: a set of cameras, image filtering (physical and digital), and estimation of the position through the calibration of the system and the application of an algorithm based on experimentally found equations. The system represents a low cost alternative, both computational and economic, capable of estimating the position of an MAV with a significantly low error using a scale in millimeters, so that almost any type of camera available in the market can be used. This system was developed in order to offer an affordable form of research and development of new autonomous and intelligent systems for closed environments.

A fundamental element for the determination of the position (pose) of an object is to be able to determine the rotation and translation of the same in space. Visual odometry is the process of determining the location and orientation of a camera by analyzing a sequence of images. The algorithm allowed tracing the trajectory of a body in an open environment by comparing the mapping of points of a sequence of images to determine the variation of translation or rotation. The use of Lane detection is proposed to feed back the Visual Odometry algorithm, allowing more robust results. The algorithm was programmed on OpenCV 3.0 in Python 2.7 and was run on Ubuntu 16.04. The algorithm allowed tracing the trajectory of a body in an open environment by comparing the mapping of points of a sequence of images to determine the variation of translation or rotation. With the satisfactory results obtained, the development of a computational platform capable of determining the position of a vehicle in the space for assistance in parking is projected.

This paper addresses an interactive virtual reality based application of a physical environment. The application presents notable aspects for education, data analysis and industry since the physical building serves as a research and development center. As the project based on physical environment, one of the main target for the work is also concentrated to give high presence feeling for end-users. The developed application is verified in real time. We also introduced our findings in real-time data communication, detection and analysis of human behavior in immersive environment, control systems integration to VR. Data analysis part of the research is linked to human behaviors based on the perception of computational intelligence methods. The activities in immersive environment are engaged to entertaining and joyful learning approaches. Some ideas for further development are also described.

Synthetic 3D content, which is the main element of virtual and augmented reality applications, typically includes objects, which react to interaction with users and other objects. Interaction may result in changes of the objects’ geometry, structure and appearance. Representation of 3D content covering temporal properties can be useful in application domains that benefit from analysis of users’ and objects’ behavior, such as education, training, e-commerce, marketing and merchandising. However, the available approaches do not enable exploration of 3D content with regards to its time-dependent properties. The main contribution of this paper is temporal representation of interaction. The representation is based on semantic web standards and ontologies, which enable use of general as well as domain knowledge for exploration of content. The representation is discussed in the context of an immersive virtual car showroom implemented using headsets and motion tracking devices.

This technical paper describes a software framework that has been conceived in order to develop collaborative, immersive Virtual Reality applications with the distinctive ability to be fed by real-time full-body tracking data of the multiple collaborating participants. This feature has a key role both to improve the mutual awareness in the visually immersive CVE, and to exploit complete body poses for human-centered joint activities and related assessments. By following recent trends of VR research application development, the framework has been based on a popular game engine (Unreal Engine 4) and leverages its powerful built-in facilities for networked, multiplayer applications and for character skinning. The paper illustrates the implemented techniques for real-time size-independent mannequin animation that could be usefully applied for the same task to different game engines or VR development environments, and also describes the data management strategies devised for an efficient implementation of a CVE architecture dealing with full-body tracking data flows.The knowledge gained during this engine-specific work will be useful for the future definition of an engine-independent communication framework for CVEs among single-user VR applications, based on heterogeneous game engine technologies, that is a strongly-felt need for promoting collaboration among VR research groups and in multi-partner industrial projects.

We present an approach based on a natural user interface and virtual reality that allows the user’s body to be visualized and tracked inside a virtual environment. Our aim is to improve the sensation of virtual reality immersion through low-cost technology such as HTC Vive and Microsoft Kinect 2. The system has been developed using the Unity 3D game engine and C# language. Our approach has been validated through the implementation of an application for 3D mesh painting where the user is able to interact through hand gestures to select a color from the 3D color palette, rotate the 3D mesh and paint it.

User navigation in 3D environments through public large-screen installations is mostly supported by mid-air interactions using motion sensors such as Microsoft Kinect. On the other hand, smartphones have been also used as external controllers of large-screen installations, and they might as well be effective in supporting 3D navigations. The aim of this study is to examine whether a smartphone-based control is a reliable alternative to mid-air interaction for 4-DOF fist-person navigation. We setup an experiment, where users had to navigate and complete a given task in two different scenes using three input modalities. The input modalities and interaction techniques were a known Kinect-based navigation method using body tilt and shoulder rotation, a smartphone-based control though tilting and rotating the device, and a traditional keyboard-based input used as a basis of comparison. We measured quantitative data through automated monitoring of users’ actions in the environment (time to finish the task, number of collisions, total collision time and distance travelled), as well as subjective user ratings through questionnaires. The results indicate that smartphone control performs at least as good as mid-air interactions with Kinect, and it was the preferred method of input in the majority of the users.

Collaborative virtual environments can connect people in social virtual spaces even when they are geographically distant from each other. Hand interactions are fundamental to enable natural collaboration and immersive experiences as they are a visually intuitive means of communication. However, scalability is challenging as numerous participants typically produce a large volume of visualisation data that may overload a single node if the management is centralised. In this paper we propose a transmission strategy where the high-throughput visualisation data (e.g. hand joints) is exchanged amongst participants in a distributed fashion. We use a level-of-detail strategy to further reduce the network traffic accounting for spatial distances amongst participants in the virtual space. We design an experiment where we analyse the network traffic in a virtual environment with up to seven participants whose hands are tracked using Leap Motion. We show that the proposed method can effectively reduce the network traffic of visualisation data when compared to a centralised approach.

Full-body motion support has been extensively utilized as a means to an end, rather than an impactful factor of concepts such as Body Ownership Illusion (BOI) and the immersive virtual experience. In addition, technical setups are seldom co-related to a study’s findings, even though system, equipment and implementation quality are fundamental components that may vastly affect user experience. This study presents a Mixed Reality (MR) environment in which 21 participants had to interact with real objects accurately represented in the virtual space, while having full-body motion control of their avatar with the use of Inverse Kinematics. Sense of presence, BOI, perceived realism and equipment invasiveness were examined in regard to having full-body motion control and real-virtual object handling. Preliminary results indicate that full-body motion support increases BOI, however with high levels of BOI, presence and overall engagement are not concomitant to perceived realism of the virtual environment and experience.

The physical support of touch interaction for a 2D interface when wearing a fully immersive head-mounted display (HMD), e.g., by using the kitchen table in a home environment, improves the user’s quality of interaction. To define interface parameters - button size, adaption over time- we conducted a user study. In two experiments with 30 participants in total, we compared the ability of the HMD user’s pointing to targets on a 2D surface without visual feedback, with visual feedback of the touched position and a real-world baseline. As a result, we give estimates for button dimensions, interaction design based on the learning curve of the user and present insights on the tested feedback modalities. We show that providing no feedback has limitations, presenting the touched position helps to increase accuracy and a head-mounted finger tracker has advantages but also comes with restrictions.

Indonesia’s rare animals are animals whose habitats exist only in Indonesia and are endangered by IUCN (International Union for Conservation of Nature and Natural Resources). In this study, we applied Augmented Reality (AR) technology to represent the endangered species in Indonesia combined with image processing techniques as the interaction between users and the virtual objects of endangered animals. The initial stage is the separation of the desired object from the background with the help of HSV colour method, as well as detecting contour with contour detector. As for the calculation of the number of fingers we are using convex hull and convexity defects. The results of this stage is the number of fingers that will be used as a reference selection of endangered animal. These series of processes generates an augmented reality application where users can view and provide instructions with virtual objects of endangered species of Indonesia. This can provide a different experience for users to learn about Indonesia’s endangered species.

This research has the purpose of allowing anyone, with or without experience handling micro aerial vehicles, to operate unmanned aerial vehicles (UAV) in a natural and intuitive way, unlike typical interfaces that need experience and knowledge in piloting to be used. To achieve this, our approach uses gesture recognition, based on machine learning with Support Vector Machine (SVM) for classification and a RGB-D sensor for the feature extraction. Tests for recognition with different Kernel-SVM and for the RGB-D sensor with different levels of light were carried out.