Autonomous Vehicles and Virtual Reality

This chapter introduces the notion of 3D images, which are now well known to the public through the animated films of Pixar, and also 3D vision, for which stereoscopic glasses are required when watching movies, such as James Cameron’s Avatar and its sequels. It also introduces various 3D visual conflicts, such as those between the human eye’s accommodation and binocular vision, and the importance of other cues for the correct visual perception of distances, such as motion parallax. This latter is provided, for example, by head-tracking devices that are used with 3D VR helmets. This chapter further aims to bring to the reader’s attention the combination of these visual cues and their coordination with other vestibular and kinesthetic perceptual cues, which contribute to cybersickness. A short description of driving simulation is provided, which allows 3D immersive technology to be linked to driving and its use for self-driving, an emerging technology that in 2022 led to the advent of commercially available autonomous vehicles. The technologies required for driving simulation, namely virtual reality and augmented reality, are not only used to validate and verify connected and autonomous vehicle (CAV) systems but also integrated as onboard systems for in-vehicle infotainment and driver comfort. They may allow also for the avoidance of self-driving sickness, an already well-known symptom of car sickness experienced by automobile passengers. Finally, the following questions are also briefly examined: Why are autonomous vehicles only being proposed now, yet various large-scale experiments and demonstrations were presented all throughout the second half of the previous century, with hundreds of thousands of tested miles covered by automobile OEMs (e.g., Daimler) more than 30 years ago? Why is self-driving being proposed now? Is it for the sake of enhanced road safety and comfort, or is it a question of the advent of economically and technically mature technologies? Indeed, cameras, motion sensors, display, vision, and control systems, with the corresponding micro-electromechanical (MEM) systems, are also massively used in 3D immersion technology. All of these technologies are expected to foster the general use of autonomous vehicles and metaverses, with an overall social transformation of mobility and immersion, in a world controlled by big data and AI, thereby shaping our incoming future.

This chapter describes some of the VR and AR systems that have been used in the automobile industry for a couple of decades, not only for vehicle engineering design and testing but also connected and autonomous vehicle (CAV) verification and validation. It demonstrates the convergence between VR and AR technology for engineering design and onboard vehicle visualization systems. Initially, VR technology was mostly driven by virtual engineering design needs in the automotive and aircraft industries as well as by the gaming industry. Recently, the strong development of VR helmets and more recently AR glasses, initiated by Facebook’s Oculus industrial development, and also the advent of wide field of view (FOV) head up  display (HUD)s and AR display systems, which provide information to the driver and passengers, have brought significant enhancements in visual quality. The recent development of wearable glasses, on the market since the 2010s, with customer versions of Google Glasses distributed in 2013 and in the following years by Sony and Toshiba, has led to the increasing maturity of AR glasses. More recent AR glasses, providing binocular Micro-LED optical waveguide display, have showcased a technology that may allow users in the near future to experience immersive visual AR technology of quality, similar to what AR helmets already provide. A major advantage of AR glasses, similar to AR helmets, is that users keep the surrounding stable visual references during observation; thus, cybersickness is significantly decreased due to visuo-vestibular incoherencies (see also Chap. 5, Sect. 5.​2 Motion and cybersickness). In addition, AR glasses weigh less and are less intrusive, and observers wear them today almost like daily corrective glasses or sunglasses, thus retaining their natural and ecological use.

Autonomous vehicles are a relatively old dream, born in the early twentieth century. Initially, they were mostly seen as a technological innovation and progress—a part of advanced vehicle control development, such as the automated guided cars of the Futurama exhibition sponsored by General Motors at the 1939 World’s Fair. Several projects allowed for impressive demonstrations, albeit more proof-of-concept projects than commercially available developments, in the last two decades of the twentieth century in the US, Japan, and Europe. The twenty-first century has changed this approach due to several Silicon Valley-based companies, above all Google’s Waymo, from 2015, followed by Chinese software technology-oriented companies, before it was accepted by traditional American and European automotive OEMs. The Silicon Valley approach was probably mainly technology-oriented, with a software-oriented mindset, considering that everything is possible with software and microelectromechanical system (MEMS) control systems. It worked well, however, in identifying new, large-scale business opportunities, capturing idle drivers for entertainment and other software services, cherished not only by GAFAM but also Baidu and other Chinese technology companies. Traditional automobile companies, initially wary of various technological and legal issues, have finally decided to follow, targeting improved road traffic safety, comfort, and freedom for relaxation during self-driving. A major issue that remains is road and vehicle safety. Early on, new technological validation was recognized to be required, with vehicles being run for hundreds of billions of miles in testing facilities. An additional difficulty identified by vehicle control and human factor engineers is the necessity of placing self-driving vehicles in various critical scenarios. This raises further issues concerning proving grounds and the field testing of road hazards that correspond to potential vehicle collisions when testing accident avoidance. A major toolset was provided by massive (including cloud) and driving simulation, already initiated in automotive driving simulation research conducted in both automotive OEMs and road research organizations. Yet, several drawbacks pose challenges, such as the simulation sickness (e.g., car sickness) that will be further experienced when using autonomous vehicles in the future or the representativity and completeness of simulation.

The advent of AR in automobile systems began in the late 1980s with the HUD systems of Nissan and General Motors (https://​nymag.​com/​intelligencer/​2019/​01/​the-past-and-future-of-the-head-up-display.​html), and today, they are largely offered for many premium passenger cars. The emergence of large-FOV windshield AR is more recent. VR and AR systems in the automobile industry are now introduced as new ADASs for enhanced driver safety and comfort, and they are commonly used for the efficient management of self-driving delegation. This chapter presents some of the issues concerning virtual and augmented technology for engineering design and for onboard vehicle visualization systems. Regarding the latter, wide-FOV HUDs and AR display systems have recently been invented to provide information to the driver and vehicle passengers. Nevertheless, both applications are aimed at coping with the same visual perception issues, including distance perception, FOV, accommodation, luminosity, contrast, limited available space for display systems, and severe and weight constraints. The industrial needs for virtual engineering design, supported by the emerging VR gaming and metaverse industries, are meeting those of large-FOV HUD onboard systems. However, future technological developments will indicate the path for the optimal deployment between onboard AR systems and connected AR glasses, synchronized with vehicle data, as necessary for vehicle safety, navigation, and other entertainment applications.

When confronted with conflicting perceptual inputs on self-motion, users may experience motion and cybersickness symptoms, causing visual or other forms of physiological discomfort, such as troubled vision, headaches, or dizziness, or even more severe sickness effects such as nausea, vertigo, or vomiting. Several theories have been proposed to explain their occurrence, especially for motion sickness when displacing at sea, on a train, or in ground vehicles. With the advent of virtual environments and VR or AR systems, additional sickness effects have been observed due to the extended possibilities of virtual motion and visual conflicts or effects inducing impaired visual perception and recurring sickness effects (i.e., cybersickness), also called Virtual reality induced sickness effects (VRISE). Naturally, there are many methods for measuring and predicting motion and cybersickness, various new display system technologies for reducing visual inconsistency effects, and new navigational techniques for avoiding motion sickness. Today, the importance of these motion- and cybersickness-avoidance techniques is being reinforced with the progressive introduction of autonomous and connected vehicles, which might induce more frequent car sickness effects when the vehicle is in self-driving mode. Indeed, automobile motion sickness has been experienced by a large proportion of vehicle passengers, and as more autonomous vehicles are used in self-driving mode, more drivers will become passengers for extended durations. According to the identified sickness effects, various cybersickness avoidance techniques have been proposed. To avoid inconsistencies between visual eye accommodation and binocular cues, new visual display systems are proposed, often with several display screens, corresponding to different virtual object distances. To deal with visuo-vestibular conflicts during virtual navigation, only rendered by the visualization of the perceived virtual world, various software solutions have been proposed. While many are well-known, they reduce natural navigation or the viewed scene; other more recent ones have not yet been industrially deployed. Finally, car sickness is progressively recognized as a major issue for the autonomous vehicle market by many OEMs, suppliers, and academic organizations, which are now increasingly developing new vehicle comfort and/or car sickness-avoidance methods.

Onboard Virtual Reality (VR systems, which are synchronized with vehicle data and produce alternative virtual world images, have already been demonstrated by several car companies; nevertheless, their use is mostly targeted at exceptional in-cabin experiences. By contrast, Augmented Reality (AR) systems, i.e., Head Up Display (HUD) or AR optical systems, that project information onto a large surface of the windshield have already been proposed as options for premium cars. They are becoming an essential part of driver assistance (i.e., ADASs) and self-driving systems whenever the driver requires real-time visual information for handling traffic hazards for enhanced driver and traffic safety. Many emerging technology-oriented companies invest heavily not only in high-performance HUDs but also in AR eyewear system development with similar hardware and software devices. Their ultimate goal is to provide vehicle occupants with in-cabin 3D audio-visual safety and infotainment systems that they can interact with in real time using natural and intuitive user interfaces while keeping their eyes on the road. This will improve safety as well as reduce the effects of car sickness. AR in-cabin systems are progressively becoming a connected part of the overall AR visual space of vehicle occupants, whether they are wearing 3D AR glasses or looking at onboard HUD systems, screens, or any other display devices.

This chapter focuses on the transformation of the society by the evolution of digital world, based on 3D image captures and diffusion all around future users of augmented reality (AR) and automated vehicles (AV), including the advent of metaverse and Web 3.0. It is also shortly introducing the new ethical questions brought by AI generated decisions, information and data, which may generate non-verifiable, non-traceable, potential IP issues and data handling in non-conformity with the various incoming data act laws in Europe and in the United States. Finally, it’s briefly outlining the consequences of the emerging new display technologies and their use for AR eyewear and automobile systems in the everyday life of the AR and AV users.