Springer Handbook of Augmented Reality

This chapter introduces the different “realities”: virtual, augmented, (deliberately) diminished, mixed, mediated, multimediated, and phenomenological. The contributions of this chapter include (1) a taxonomy, framework, conceptualization, etc., of all of the “realities”; (2) a new kind of “reality” that comes from nature itself, which expands our notion beyond synthetic realities to include also phenomenological realities; and (3) methods of using phenomenological reality as a means of visualizing as well as understanding hidden phenomena present in the world around us. VR (virtual reality) replaces the real world with a simulated experience (a “virtual” world). AR (augmented reality) allows the real world to be experienced while at the same time, adding to it, a virtual world. Mixed reality provides blends that interpolate between real and virtual worlds in various proportions, along a “virtuality” axis, and extrapolate to an “X-axis” defined by “X-Reality” (eXtended reality). Mediated reality goes a step further by mixing/blending and also modifying reality, including, for example, deliberate diminishing of reality (e.g., a computerized welding helmet that darkens bright subject matter while lightening dark subject matter). This modifying of reality introduces a second axis called “mediality.” Mediated reality is useful as a seeing aid (e.g., modifying reality to make it easier to understand) and for psychology experiments like George Stratton’s 1896 upside-down eyeglasses experiment. Multimediated reality (“all reality” or “All R”) is a multidimensional multisensory mediated reality that includes not just interactive multimedia-based “reality” for our five senses but also includes additional senses (like sensory sonar, sensory radar, etc.), as well as our human actions/actuators. These extra senses are mapped to our human senses using synthetic synesthesia. This allows us to directly experience real (but otherwise invisible) phenomena, such as wave propagationWave propagation and wave interference patterns, so that we can see radio waves and sound waves and how they interact with objects and each other, i.e., phenomenological reality. Moreover, multimediated reality considers not just multiple axes in addition to the X-Reality axis but also that the origin of the axes exists at zero sensory stimuli. In this way, we can account for various (virtual, augmented, etc.) realities in a sensory deprivation float tank. Consider, for example, wearing an underwater VR headset while floating in a sensory deprivation tank. Then consider an augmented reality headset while floating in an immersive multimedia pool. Consider Internet-connected immersive multimedia water therapy and music therapy pools that allow multiple users to play a hydraulophone (underwater pipe organ) while meditating together in a hydraulically multimediated collective. These are examples of what we call “fluid (user) interfaces,” and they fall outside the range of any previously existing “reality taxonomy” or conceptualization.

Augmented reality (AR) is emerged in the last decade as a potentially disruptive technology capable of superimposing virtual objects generated by a computer onto the real world surrounding the user. By using see-through displays, AR systems make the user perceive both the real surrounding environment and virtual elements in a consistent way with regard to user point of view and virtual content size. The recent developments of low-cost AR technologies and mixed reality (MR) devices such as Google Glass, Microsoft Hololens, Vuzix, and many others are capturing interest of users and researchers, suggesting that AR could be the next springboard for technological innovation as also highlighted but its inclusion as enabling technology of Industry 4.0 paradigm. However, the AR is not as young as it seems. The concept of AR was formulated in the 1960s, and the first commercial AR tools appeared in the late 1980s. This chapter presents an overview of AR, the historical evolution since its first appearance, its usage in the most relevant domains including the emerging instance of Mobile/Wearable AR applications, the technical challenges in implementing AR systems, as well as main technological and applicative trends for the near future.

Tracking is the main enabling technology for Augmented Reality (AR) as it allows realistic placement of virtual content in the real world. In this chapter, we discuss the most important aspects of tracking for AR while reviewing existing systems that shaped the field over the past years. Initially, we provide a notation for the description of 6 Degree of Freedom (6DoF) poses and camera models. Subsequently, we describe fundamental computer vision techniques that tracking systems frequently use such as feature matching and tracking or pose estimation. We divide the description of tracking approaches into model-based approaches and Simultaneous Localization and Mapping (SLAM) approaches. Model-based approaches use a synthetic representation of an object as a template in order to match the real object. This matching can use texture or lines as tracking features in order to establish correspondences from the models to the image, whereas machine learning approaches for direct pose estimation of an object from an input image have also been recently introduced. Currently, an upcoming challenge is the extension of tracking systems for AR from rigid objects to articulated and nonrigid objects. SLAM tracking systems do not require any models as a reference as they can simultaneously track and map their environment. We discuss keypoint-based, direct, and semi-direct purely visual SLAM system approaches. Next, we analyze the use of additional sensors that can support tracking such as visual-inertial sensor fusion techniques or depth sensing. Finally, we also look at the use of machine learning techniques and especially the use of deep neural networks in conjunction with traditional computer vision approaches for SLAM.

3D object and hand pose estimation have huge potentials for Augmented Reality, to enable tangible interfaces, natural interfaces, and blurring the boundaries between the real and virtual worlds. In this chapter, we first motivate the topic and explain the challenges. After a brief review of early work and Deep Learning techniques on which recent works are based, we present the recent developments for 3D object and hand pose estimation using cameras, when considered separately and together. We examine the abilities and limitations of each technique. We conclude by discussing the possible future developments of the field.

This chapter gives an overview of the interaction techniques for mixed reality with its variations of augmented and virtual reality (AR/VR). Various modalities for input and output are discussed. Specifically, techniques for tangible and surface-based interaction, gesture-based, pen-based, gaze-based, keyboard and mouse-based, as well as haptic interaction are discussed. Furthermore, the combinations of multiple modalities in multisensory and multimodal interaction as well as interaction using multiple physical or virtual displays are presented. Finally, interactions with intelligent virtual agents are considered.

With the release of commercial head worn augmented reality (AR) devices, the AR community has grown exponentially. These AR devices come with robust computer vision algorithms to perform adequate object tracking and spatial localization of the user as well as objects in the AR environment. They allow for user input in various ways to interact with the scene. In addition, these devices finally became lightweight and comfortable enough to be used for real jobs in real professional environments. A new era has begun that allows the development of AR applications on a new level. It is no longer necessary to be an expert in computer vision or computer graphics to create meaningful solutions for application ideas that mostly have been proposed already many years ago. The usage of top-level game engines enables app developers to integrate well-designed and textured models, animations, physical effects, and illumination into their solutions, even without programming knowledge. In the past, the big majority of AR applications has focused on placing additional rigid or animated objects into the AR environment. Alternatively, tracked real objects have been superimposed by virtual supplementary information such as virtual structures inside a real object. However, another type of augmented content has almost been neglected so far. This chapter addresses the integration of and interaction with virtual characters in AR environments. It provides an overview of research that has been published by the AR community in the past and classifies different types of characters. In addition, we introduce the reader to Game AI, an R&D subdiscipline of game engineering that focusses on controlling non-player characters (NPCs) in video games. Furthermore, this chapter discusses peculiarities of the interaction with NPCs in AR environments and takes a closer look at the psychological subjects namely perception, communication, and behavioral psychology.

Mixed-reality (MR) technology development is now gaining momentum due to advances in computer vision, sensor fusion, and realistic display technologies. Despite this, most of the research and development has been focused on delivering the promise of MR; concerns on potential security and privacy risks are continuously being pointed out, and only a few are working on the privacy and security implications of the technology. We put into light these risks and look into the latest security and privacy work on MR. In this chapter, we present an exposition and categorization of the latest security and privacy work on MR.

Augmented reality (AR) is the next frontier for visual displays. In the optimal AR display, the mechanics, electronics, and optics must interact seamlessly. In this chapter, optical science concepts are developed to facilitate the reader’s understanding of the optics found within current AR technology. Various optical architectures are being used in the current AR display technology, and those architectures will be dissected and discussed. A combination of physical, electrical, and optical constraints limits the capabilities of recent AR displays. As the details of the optical challenges in designing AR displays are examined, emerging technologies that could facilitate a fundamental change in design processes, such as holographic optics, freeform optics, and metasurfaces, will be introduced.

A single sensor that tracks the user’s pose is usually not sufficient for the creation of compelling augmented reality (AR) owing to system drift, noise, and tracking errors. Recent systems thus combine tracking information from multiple sensors, such as red green blue and infrared cameras, accelerometers, and depth sensors, to improve the illusion of an augmented world. In this chapter, we first introduce calibration methods, tracking sensors, and hardware that can track objects of interest, followed by a discussion of methods that can align the information gathered by these sensors.

Wearable computers have seen a recent resurgence in interest and popularity in which augmented reality (AR) smartglasses are poised to influence the way we complete our work and daily tasks. Nowadays, industrial applications of these smartglasses are focused on interior designs, remote collaborations, as well as e-commerce. Under five key constraints on AR smartglasses such as miniature touch interface, small-screen real estate, user mobility, limited computational resource, and short battery life, existing user interaction paradigms designed for desktop computers and smartphones are obsolete and incompatible with the scenarios of AR smartglasses. The cumbersome and difficult interaction with the AR smartglasses becomes a hurdle to their wider industrial applications. Thus, an unmet demand for designing interaction techniques on AR smartglasses is undoubtedly critical.In this chapter, we present three interaction techniques, namely, TiPoint, HIBEY, and TOFI, in order to enhance object manipulation and text entry in the constrained environment of AR smartglasses. These techniques are devised in a way that leverage on advantageous features of human body and experiences such as the dexterity of fingertip, lexicographical order ingrained in our memory, proprioception, as well as opposable thumbs. We thoroughly address the key constraints on AR smartglasses and explore different modalities with various hardware and peripherals.

Mobile augmented reality (MAR) applications are gaining popularity due to the wide adoption of mobile and especially wearable devices such as smartglasses. These devices often strike a compromise between mobility, energy efficiency, and performance. On the other hand, MAR applications rely on computationally intensive computer vision algorithms with extreme latency requirements. Cyber-foraging allows resource-constrained devices to leverage the computing power of nearby machines, and has often been proposed as a solution for increasing the computing capabilities of constrained devices. However, this process introduces new constraints in the application, especially in terms of latency and bandwidth. MAR applications are so demanding that current network infrastructures are barely ready for such traffic. Such resource-hungry applications may rapidly saturate future wireless networks such as 5G. As such, developing cyber-foraging solutions for MAR applications requires a delicate balance in network resource usage to ensure seamless user experience. In order to identify the opportunities for further improving the Quality of Experience (QoE) for mobile AR, we break down the end-to-end latency of the pipeline for typical MAR applications and pinpoint the dominating components in the critical path, from the physical transmission to the application. We then derive a set of actions to enhance the possibilities of cyber-foraging, taking into account the specificities of future networking technologies.

This chapter focuses attention on the potential of augmented spatial experience technologies in the pedagogical use of art and how these can significantly enhance the role of art education. The reasons why we feel it is necessary to dedicate deeper analysis to the theme of augmented reality in arts education are fueled by the acknowledgement that the theme is developing rapidly but lacks a systematization of the field experiences, coming from different research fields. An assiduous interdisciplinary discussion, with special reference on the one hand by the scholars of the Digital Heritage, is forever committed to the documentation and valorization of the tangible and intangible historical-artistic heritage and, on the other by the scholars of arts pedagogy and educational technology, seems more than ever necessary in order to contribute to minting the same coin, that is, the one relating to the safeguarding of the value of the arts and the heritage for the development of individual and, therefore, a society capable of evolving starting from the memory of its own expressive capacities.

Augmented or mediated reality tools superimpose or composite virtual objects into the real world, supplementing reality rather than replacing it. The educational benefits include assisting learners appreciating a real-world experience more fully and developing independent thinking, creativity, and critical analysis. The intelligence amplification potential of this technology has been recognized for many years, and the development of low-cost augmented reality hardware integrated into mobile devices now means that educational implementations are plausible and can be practically implemented in a wide range of contexts. This chapter surveys the conceptualization of augmented reality and provides a detailed review of its educational applications, analyzing the underlying theoretical foundations that describe the learning and cognitive impact of its use as well as the practical benefits for learners and educators.

Physical education (PE) courses involve the provision of sports knowledge and the engagement in motor skill practices. Instruction based on augmented reality (AR) has been rarely applied in the hands-on teaching of these courses. Currently, PE teaching is mostly aided by video-assisted instructions. However, such instruction does not provide interactive experience in practices and fails to integrate textbook-based static learning with dynamic learning based on motor skill demonstrations. Because AR can overlay virtual information on a real object, this technology enables learners to, for example, manipulate an interactive three-dimensional human character model overlayed on a textbook while reading the textbook. Thus, AR can overcome the disadvantage of video-assisted instruction. To investigate the effects of AR-assisted instruction on learning outcomes, motor skills of various difficulty levels, and learning motivation in students, two experiments were conducted using different teaching materials; specifically, instructions on basic running drills and Mach drills were offered to students. A quasi-experimental design was adopted. The results indicated that AR-based teaching materials outperformed video-based ones, particularly in the learning outcomes of difficult motor skills.

A significant progress in technological support aroused a general interest in areas such as data visualization, augmented and virtual reality, and artificial intelligence. The innovative capabilities of these technologies increased the potential and relevance of data visualization and interaction services. Traditional teaching and learning methods look insufficient in a context where digitalization invades processes and tools. A strong claim is made for aligning those methods with such technological developments and thus allows students to acquire the skills to successfully integrate the emergent information society. Given the widespread of mobile devices, and the increasing role of computer games in education, their combined use has the potential to play a central role in responding to these demands. This chapter aims at exploring the integration of serious games with virtual environments and technologies as a complement to facilitate and enhance learning. More specifically, we present and discuss the motivation, design, and development process of three tools that use augmented reality in combination with a serious game to teach (i) mathematics, (ii) to explore the platonic solids, (iii) and to teach coding. Further, three pilot user studies are described and discussed and confirm the potential of these tools as powerful new teaching/learning tools for education.

Augmented reality applications for Cultural Heritage have been implemented in the last years. The use of AR is currently diffused for many purposes, from technical and managing activities to dissemination. On the communication side, the main potential of such approach is the extension of human sight as to cover simultaneously the current situation of a point of interest (monuments, archaeological sites, artifacts, etc.) and the reconstruction of its ancient condition in different historical periods. Moreover, it allows to compare different possible hypotheses and evaluate the reliability of each of them in the general context of the known data. Despite the current limits of AR (due to the approximation in device positioning), which still influence the use of applications for mobile use, such technology is very promising in the fields of tourism, education, and entertainment; it allows to enrich 3D scenarios with different types of content. In the next future, it is reasonable to imagine a huge amount of information, in different formats, potentially reachable by people simply having a look at the remains of the ancient past and choosing their favorite topics. At the same time, the connection of AR with other emerging technological infrastructures (such as IoT) will allow to collect, view, and manage simultaneously lots of real-time diagnostic data in the same framework. This will foster the research activity toward the definition of new communication metaphors and cognitive solutions.

Augmented reality (AR) is a new medium with the potential to revolutionize education in both schools and museums by offering methods of immersion and engagement that would not be attainable without technology. Utilizing augmented reality, museums have the capability to combine the atmosphere of their buildings and exhibits with interactive applications to create an immersive environment and change the way that audiences experience them and therefore providing the ability to perform additional historical perspective taking. Holocaust museums and memorials are candidates for augmented reality exhibits; however, using this technology for them is not without concerns due to the sensitive nature of the subject. Ethically, should audiences be immersed in a setting like the Holocaust? How is augmented reality currently being used within Holocaust museums and memorials? What measures should be taken to ensure that augmented reality experiences are purely educational and neither disrespectful to the victims nor cause secondary trauma? These are the questions that this chapter will seek to answer in order to further develop the field of augmented reality for Holocaust education. To achieve this, previous AR apps in Holocaust museums and memorials have been reviewed, and a series of studies on the usage of AR for Holocaust education have been examined to identify the ethical considerations that must be made and the ramifications of utilizing AR technology to recreate tragic periods of history.

Live theatrical performance is an ever-evolving art form in which visionary theater makers are incorporating augmented reality (AR) into performances to connect and engage modern audiences. While theater productions are generally limited by the physical environment’s constraints, AR offers a means to significantly expand the types of sets, effects, and stories that can be told. Furthermore, the addition of 3D tracking and interactive projections enables a new performance methodology with exciting new options for theatrical storytelling, educational training, and interactive entertainment. In this chapter, the authors discuss recent inclusions of AR in live performance, present helpful insights for those looking to include AR into productions, and explore the future of AR live theatrical performance.

The increased proliferation of augmented reality (AR) technology has especially impacted the tourism industry, and many players in this sector have made initial attempts to integrate AR into their marketing strategies. Although both the tourism sector and the academic tourism community have made significant contributions to the AR discipline, some gaps remain. While many studies have assessed the drivers behind the acceptance of specific use cases, a comparison of different use cases remains scarce. Likewise, little is known about how evaluations of AR differ in different tourist segments. Therefore, this research aims to examine how tourists evaluate touristic AR use cases and how these evaluations differ between tourist segments. Based on an empirical study among 553 German city tourists, the findings reveal that virtual guides and history AR apps are among the most popular touristic AR use cases and that different tourist segments (explorers, hedonists, and culture gourmets) value the usefulness of tourist AR use cases differently, whereas almost no significant differences can be found when it comes to demographic data such as age and gender. It is thus important to adapt touristic AR use cases to the motivations and needs of tourists – and not so much to the demographic data, as many organizations still do.

Augmented reality’s (AR) first aerospace application came around at the end of the First World War in the form of the aircraft gunsight. Over the last century, this technology evolved to serve different areas of need, including engineering, training, space operations, and flight crew support. The most notable AR applications are the head-up display (HUD) and the helmet-mounted display (HMD), which are used to provide navigation assistance to pilots flying high-profile missions. In recent years, various aerospace corporations have turned to AR technology to enhance their prototyping, manufacturing, and maintenance operations. Similarly, airlines and airports are investigating the use of AR to support daily tasks of operating crew and personnel. These efforts are supported by research groups studying the potential of AR in air traffic control and management, airport operations and security, crew training, aircraft cabin visualization, and In-Flight Entertainment and Communication (IFEC). This chapter presents an overview of AR’s history and applications in the aerospace industry throughout the years.

On average, annual building maintenance and repair cost 1–5% of the initial building cost, which, accumulated over the building life, could even exceed the initial construction cost. New advancements in architecture, engineering, construction, and operation (AECO) are transforming the current practice of building maintenance operations using visualization and sensing technologies. This chapter will describe a use case for the application of augmented reality (AR) visualization in building maintenance and repair. AR enhances user’s perception of the surroundings by overlaying virtual objects on real-world views and can lead to new forms of user interaction. For instance, AR visualization embedded in the building maintenance instruction manual (BMIM) can be used to guide facility managers and repair personnel. We present the design and evaluation of an AR-integrated BMIM to improve the quality of building operation and maintenance. We adopt the design science research (DSR) methodology to carry out a systematic literature review, characterization of AR features that can be applied to BMIM, development of AR artifacts for incorporating into BMIM, and assessment of user performance through experiments with measurements taken with the NASA TLX protocol. Results are implemented in two applications, namely, Living Augmented Reality (LAR) and Manual Augmented Reality (MAR), with different visualization scales. Analysis of users’ workload data indicates that a majority agree that BMIM can be effectively enhanced with a high degree of acceptability using AR on mobile devices. It is also found that this integration will be helpful to future evolution of the BMIM and integration with the Internet of Things paradigm.

Finite element analysis (FEA) is usually carried out off-site and using computer desktops, i.e., computer-generated graphics, which does not promote a user’s perception and interaction and limits its applications. This chapter first gives an overview of related FEA and AR technologies and presents the feasibility of enhancing finite element structural analysis with AR technology. A novel system has been proposed which integrates sensor measurement, FEA simulation, and scientific visualization into an AR-based environment. This system can acquire input data using sensors and visualize FEA results directly on real-world objects. Several intuitive interaction methods have been developed for enhancing structural investigation and data exploration. A prototype system has been built and tested using several case studies to validate the proposed methods and evaluate the system performance.

In maintenance, the use of augmented reality (AR) has often been perceived as a solution in resolving the various problems faced by the industrial sector. For any reliable production line, maintenance is crucial in order to ensure the safe, continued, and productive operation of the entire manufacturing process. However, the difficulty of maintaining modern machinery and products is increasing exponentially such that operators may not be able to cope. As a human-computer interaction (HCI) tool, AR can potentially provide the competitive advantage to the maintenance operators to keep up with this rate of increase. This chapter presents new research and application developments in AR in maintenance. The chapter provides a holistic literature review on the recent developments in the state of the art of AR in maintenance. Current research directions and future developments are discussed.

As an emerging and powerful technology, augmented reality (AR) combines the real world with virtual and computer-generated objects in order to give to users useful data about the task they are performing. Due to its features, AR is a promising way to enhance many activities requiring the intervention of specialized personnel combining, sometimes, the use of other tools such as real-time monitoring, fault diagnosis, and communication systems.This chapter presents the use of augmented reality (AR) methodologies and technologies to improve maintenance, repair, and troubleshooting activities by enhancing the technicians’ performance from the training to the on-site interventions to reduce time and costs. Then, it presents an innovative framework designed to facilitate the development of high-performance 3D real-time solutions for Industry, Marketing, Cultural Heritage, and Healthcare. Finally, the chapter ends with use case, test, and experimentation.

Since the pioneering work of Ivan Sutherland and the success of Tom Furness’ “Super Cockpit,” augmented reality (AR) has been an interest of the US Department of Defense (DoD) for some time. As hardware advances enabled more ambitious applications, the US Navy and US Marine Corps (USMC) have carried this research forward into new application areas and into development programs to explore the practical potential of the technology to assist in accomplishing tasks required in Naval applications (including land-based operations of USMC). We review a selection of AR research and development conducted by the US Navy, ranging from basic research to prototype development. We draw lessons from the progression of projects through the R&D spectrum and explore common threads through varied applications. Challenges that currently limit the application of AR technology to naval domains are construction of environmental models that enable integration of the virtual and real elements, ability to operate in mobile scenarios, and usability concerns such as information overload.

Industry 4.0 is paving the way for the automation of many industrial fields that can improve the efficiency of manufacturing processes. Such an improvement can be remarkable on industries that involve a relevant number of complex processes, which need to be optimized individually in order to increase their overall performance. Shipbuilding is one of such industrial fields that can benefit from the application of the principles of Industry 4.0 and from the use of the latest technologies. Among the different Industry 4.0 technologies, augmented reality (AR) and mixed reality (MR) can be especially helpful for enhancing shipbuilding tasks, since they can be applied to many of them, while providing useful and attractive visual interfaces that enable shipyard operators to receive information on the tasks they are working on and that allow them to interact with physical and virtual elements. This chapter first reviews the state of the art on the application of commercial and academic AR/MR solutions to shipbuilding. Moreover, the most relevant shipbuilding tasks to be enhanced with AR/MR and their challenges are analyzed. Furthermore, this chapter also details the latest and most promising AR and MR hardware, software, and communication architectures aimed at being deployed in shipyard workshops and on ships under construction. As a result, this chapter provides a thorough review of the most recent developments on the application of AR and MR to shipbuilding and includes useful guidelines for future developers.

Augmented Reality (AR) reduces the technicians’ cognitive effort mainly resulting in both time and error rate reductions. Still, its application in remote assistance has not been fully explored yet. This paper focuses on understanding the benefits of providing assistance to a remote technician through AR. Augmented Reality for Remote Assistance (ARRA) has been designed and developed for local novice maintainer to request assistance and communicate with a remote expert. The remote expert can manipulate virtual objects, which are then overlaid on the real environment of the novice maintainer. ARRA has been tested with the help of 60 participants. This involved performing an assembly/disassembly operation on a mock-up of a piping system. The participants were remotely assisted through ARRA or video-call. Quantitative spatial referencing error data has been collected. The results showed a 30% improvement in terms of spatial referencing when utilizing ARRA as remote assistance support as opposed to video-call. Future studies should investigate into quantifying the improvements due to other factors involved in remote assistance, especially language barriers and connectivity issues.

With the development of augmented reality devices, new imaging hardware, and better computer simulations, the field of computer-assisted surgery is rapidly evolving. From an augmented visualization of medical images during an intervention to a fully automatic fusion of the patient’s virtual anatomy with the surgical view, many ways of augmenting a surgery can be envisioned. This chapter provides a description of the main challenges, state of the art, and results in this revolutionary area by looking at the contributions of augmented reality in several medical application fields with an emphasis on deformable techniques. We will address several key questions, such as the creation of patient-specific models of the anatomy, the registration of the virtual model onto the actual view, or the real-time computation of biophysical phenomena on this anatomy.

Augmented reality (AR) is being implemented in the lives of children and young adults and is becoming increasingly relevant in various fields. AR initially influenced popular culture through gaming by encouraging children to go outside in order to play. It then evolved for use in other fields, such as social media. AR is commonly used in social media by youth when they use lenses and filters to augment their pictures. AR is even used in arts and sports to improve children’s understandings and abilities. Additionally, in education, AR can assist teachers by providing engaging opportunities for learning, such as 3D models or field trips without leaving the classroom. Finally, an important application of AR use is for children in medical settings to improve patient experiences and outcomes. The fields of gaming, social media, sports, arts, education, and medicine can increasingly incorporate AR technology as it continues to advance and improve. This chapter will discuss the current and future applications of AR in these fields as it relates to children and young adults, as well as important safety and accessibility considerations.

The number of people with low upper-extremity functions is increasing due to the sedentary lifestyle, muscular disuse, and aging of population. Therefore, healthcare exercising systems that aim to enhance upper-extremity skills are desirable. The improvement of motor functions is an ordered process, and hence, the development of an upper-extremity training plan with stages with respect to the capability of the users is an important issue. Augmented Reality (AR) -assisted motor-skills training applications have been proven to be effective. This chapter discusses the importance of providing AR-assisted healthcare exercises in stages. The chapter reviews the current AR-assisted healthcare exercising systems and makes a comparison with virtual reality-based systems as well as conventional systems. A novel AR-assisted Three-stage Healthcare Exercising system (ARTHE) is presented to demonstrate stage-based AR-assisted systems for training activities of daily living.

Augmented reality (AR) is a rapidly developing technology that has introduced new approaches for the design of human technology interaction in various fields. However, application of AR in the area of assistive systems for cognitively impaired people is not deeply studied yet. In this chapter, we investigate the state of the art of AR in assistive systems for the cognitively impaired. Specifically, we will investigate the role of AR during the design, implementation, and performance assessment of perception and memory augmentation assistive systems. We start with a summary of various technologies utilized for the development of AR systems, including sensor and camera technology, data visualization methods, computing paradigms, and intelligent data processing algorithms. Then, we discuss the fundamental mechanisms behind human memory system and look at the examples of the first technology-based human memory and intelligence augmentation systems. We overview various methods for estimation of the human cognitive state and mental workload utilized during the evaluations of such assistive systems in human involved experimental studies. Our main objective in this work is to find out the current status, challenges, and future perspectives of AR in the research of human memory and perception augmentation systems for the people with cognitive impairments.

Human society is encountering a new wave of advancements related to smart connected technologies with the convergence of different traditionally separate fields, which can be characterized by a fusion of technologies that merge and tightly integrate the physical, digital, and biological spheres. In this new paradigm of convergence, all the physical and digital things will become more and more intelligent and connected to each other through the Internet, and the boundary between them will blur and become seamless. In particular, augmented/mixed reality (AR/MR), which combines virtual content with the real environment, is experiencing an unprecedented golden era along with dramatic technological achievements and increasing public interest. Together with advanced artificial intelligence (AI) and ubiquitous computing empowered by the Internet of Things/Everything (IoT/IoE) systems, AR can be our ultimate interface to interact with both digital (virtual) and physical (real) worlds while pervasively mediating and enriching our lives.In this chapter, we describe the concept of transreality that symbiotically connects the physical and the virtual worlds, incorporating the aforementioned advanced technologies, and illustrate how such transreality environments can transform our activities in it, providing intelligent and intuitive interaction with the environment while exploring prior research literature in this domain. We also present the potential of virtually embodied interactions – e.g., employing virtual avatars and agents – in highly connected transreality spaces for enhancing human abilities and perception. Recent ongoing research focusing on the effects of embodied interaction are described and discussed in different aspects, such as perceptual, cognitive, and social contexts. The chapter will end with discussions of potential research directions in the future and implications related to the user experience in transreality.

This chapter contends that the Internet of Things (IoT), which consists of technologies for networked embedded devices and decentralized software applications, must also have rich interface solutions that both inform and engage its users. This is a new frontier in IoT human-computer interaction (HCI), and meeting these challenges requires interface designs that present information to users in a context-sensitive manner, which can inform users while allowing them to remain engaged in their environmental tasks. Mixed reality, combined with context sensing, also known as X-Reality (XR), allows the display of 3D content in situ, providing novel interface design possibilities for IoT edge devices. These XR-IoT (XRI) hybrid systems can become more personal, immersive, embedded, information-rich, decentralized, multiuser, and agent-driven. This repositions the IoT from the passive background embedded environment into active, engaging, foreground information infrastructures. This chapter highlights the challenges and considerations to meet the convergence of the IoT with mixed reality, toward a transformation of IoT devices into hybrid virtual and physical objects that adapt to their user’s context, and presents information in engaging ways. Such a merger will contribute to the IoT broadly and impact HCI across existing and future IoT deployments.

Nowadays, the digital transformation (driven by the Industry 4.0 (I4.0) paradigm) is becoming one of the most promising and valuable strategies to address business needs of manufacturing players in terms of agility, efficiency, and real-time reactivity. Among available digital (I4.0-based) technologies, Digital Twins (DTs) are described by researchers and practitioners as a key element in terms of smart manufacturing because of their potential to enable the shift from automation to autonomy. To this aim, the present chapter highlights characteristics and benefits of DTs, by proposing a strategic tool to support and guide Small- and Medium-sized Enterprises (SMEs) toward their adoption and exploitation. In addition, potentialities coming from the integration of both DTs and Extended Reality (ER) tools are shown through a laboratory application case.

Augmented reality techniques can be used to support system modeling and industrial operations at different levels, enabling designers and engineers to augment their real environment with relevant virtual content. In aerospace, these techniques are tightly coupled to digital twins. Together, they can enhance scarce operational resources, facilitating skill transfer and knowledge retention. It is possible to define the term digital twin based on conceptual data models as used in model-based system engineering. In this definition, a conceptual data model is used to accompany a product as an unique evolving system model through the whole lifecycle, starting from virtual abstractions and progressing toward a virtual replication of the real-world entity. Throughout the whole lifecycle, digital twins help to analyze or predict system behavior for improving decision-making and avoiding cost-expensive prototyping. In this chapter, we discuss how digital twin representations can leverage on augmented reality approaches and provide an overview of how model-based system engineering can help to maintain information consistency through the different phases of the product lifecycle. In this context, we address different aspects related to the use of augmented reality approaches for digital twins in aerospace, such as overlay precision, interaction, data visualization, and remote collaboration. Our example applications take different phases of the product lifecycle into account, from creation to operation.