The Engineering of Mixed Reality Systems

Human–Computer Interaction (HCI) is no longer restricted to interaction between users and computers via keyboard and screen: Currently one of the most challenging aspects of interactive systems is the integration of the physical and digital aspects of interaction in a smooth and usable way. The design challenge of such mixed reality (MR) systems lies in the fluid and harmonious fusion of the physical and digital worlds. Examples of MR systems include tangible user interfaces, augmented reality, augmented virtuality and embodied interfaces. The diversity of terms highlights the ever growing interest in MR systems and the very dynamic and challenging domain they define.

Technological advances in hardware manufacturing led to an extended range of possibilities for designing physical–digital objects involved in a mixed system. Mixed systems can take various forms and include augmented reality, augmented virtuality, and tangible systems. In this very dynamic context, it is difficult to compare existing mixed systems and to systematically explore the design space. Addressing this design problem, this chapter presents a unified point of view on mixed systems by focusing on mixed objects involved in interaction, i.e., hybrid physical–digital objects straddling physical and digital worlds. Our integrating framework is made of two complementary facets of a mixed object: we define intrinsic as well as extrinsic characteristics of an object by considering its role in the interaction. Such characteristics of an object are useful for comparing existing mixed systems at a fine-grain level. The taxonomic power of these characteristics is discussed in the context of existing mixed systems from the literature. Their generative power is illustrated by considering a system, Roam, which we designed and developed.

This chapter addresses issues related to usability and user experience of mixed reality (MR) systems based on a naturalistic iterative design approach to the development of MR applications. Design and evaluation of MR applications are still mostly based on methods used for development of more traditional desktop graphical user interfaces. MR systems are in many aspects very different from desktop computer applications, so these traditional methods are not sufficient for MR applications. There is a need for new approaches to user-centred design and development of MR systems. One such approach is based on the concepts of cognitive systems engineering (CSE). In this chapter we show how this approach can be applied to the development of MR systems. Two case studies are described, where a holistic CSE approach to design, implementation and evaluation has been used. The results show that allowing real end users (field/domain experts) to interact in a close to naturalistic setting provides insights on how to design MR applications that are difficult to attain otherwise. We also show the importance of iterative design, again involving real end users.

The deployment of mixed reality environments for use by members of the public poses very different challenges to those faced during focused lab studies and in defined engineering settings. This chapter discusses and presents examples from a set of case studies that have implemented and evaluated fully functioning mixed reality environments in three different organisational settings. Based on these case studies, common themes that are critical in the engineering of publicly deployed mixed reality are drawn out. Specifically, it is argued how the creation of a mixed reality interaction space depends on the technology as well as the environment, how asymmetric access provided to different sets of participants can be desirable and how social interaction reflects the particulars of the embedded technology, the length of deployment and the existing social organisation.

This chapter introduces a multidisciplinary holistic approach for the general design of successful bridge experiences as a cross-context human–information interaction model. Nowadays it is common to interact through a number of different domains in order to communicate successfully, complete a task, or elicit a desired response: Users visit a reseller’s web site to find a specific item, book it, then drive to the closest store to complete their purchase. As such, one of the crucial challenges user experience design will face in the near future is how to structure and provide bridge experiences seamlessly spanning multiple communication channels or media formats for a specific purpose.

In this chapter, we discuss the design of tangible interaction techniques for mixed reality environments. We begin by recalling some conceptual models of tangible interaction. Then, we propose an engineering-oriented software/hardware co-design process, based on our experience in developing tangible user interfaces. We present three different tangible user interfaces for real-world applications and analyze the feedback from the user studies that we conducted. In summary, we conclude that since tangible user interfaces are part of the real world and provide a seamless interaction with virtual words, they are well adapted to mix together reality and virtuality. Hence, tangible interaction optimizes users’ virtual tasks, especially in manipulating and controlling 3D digital data in 3D space.

This chapter presents steps for designing an intergenerational mixed reality entertainment system, which focuses on physical and social interactions using a mixed reality floor system. The main design goals include the following: facilitating interactions between users with varied levels of skill in utilizing technology, utilizing the familiar physical motions from other activities to make an intuitive physical interface, and encouraging social interactions among families and friends. Detailed implementation of these steps is presented in the design of our intergenerational entertainment system, Age Invaders. Our design process is based on user-centered design. The results of the study help to focus the refinements of the existing platform from a usability standpoint and also aid in the development of new physical entertainment and interactive applications. This study provides insights into user issues including how users interact in a complex mixed reality experience.

Presence, the “perceptual illusion of non-mediation,” is often a central goal in mediated and mixed environments, and sound is believed to be crucial for inducing high-presence experiences. This chapter provides a review of the state of the art within presence research related to auditory environments. Various sound parameters such as externalization and spaciousness and consistency within and across modalities are discussed in relation to their presence-inducing effects. Moreover, these parameters are related to the use of audio in mixed realities and example applications are discussed. Finally, we give an account of the technological possibilities and challenges within the area of presence-inducing sound rendering and presentation for mixed realities and outline future research aims.

Humans experience their physical and social environment through their bodies and their associated movement actions. However, most mixed reality systems approach the integration of the real with a virtual world from a computational perspective, often neglecting the body’s capabilities by offering only limited interaction possibilities with a few augmented tangible objects. We propose a view on mixed reality systems that focuses on the human body and its movements, because we believe such an approach has the potential to support novel interaction experiences, as explored by a prototypal gaming system that was inspired by exertion actions exhibited in table tennis. “Table Tennis for Three” enables augmented bodily experiences while offering new opportunities for interaction, such as supporting three players simultaneously across geographical distances. This case study offers an exploration of the role of the human body and its associated movement actions in mixed reality systems, aiming to contribute toward an understanding of the use of exertion in such systems. Such an understanding can support leveraging the many benefits of exertion through mixed reality systems and therefore guide future advances in this research field.

Taking advantage of the physical objects surrounding the user and the human ability to manipulate them fosters the development of multiple, new and advanced interaction techniques, called mixed interactive systems (MIS). Much work has been done to address specific aspects of the development of MIS. However, there is still no unifying conceptual framework to link these contributions and that presents a global approach for the development of MIS. In this context, this chapter presents a domain-specific development process that goes beyond ad hoc approaches and attempts to overcome barriers between different types of developer expertise, through a set of connections between steps of the MIS development process. Furthermore, to facilitate iteration in the design, these connections are observable, thus allowing a designer to review their decisions. The development process is illustrated via a concrete museum application.

Developing usable and robust mixed reality systems requires unique human–computer interaction techniques and customized hardware systems. The design of the hardware is directed by the requirements of the rich 3D interactions that can be performed using immersive mobile MR systems. Geometry modeling and capture, navigational annotations, visualizations, and training simulations are all enhanced using augmented computer graphics. We present the design guidelines that have led us through 10 years of evolving mobile outdoor MR hardware systems.

We describe a non-visual interface for displaying data on mobile devices, based around active exploration: devices are shaken, revealing the contents rattling around inside. This combines sample-based contact sonification with event playback vibrotactile feedback for a rich and compelling display which produces an illusion much like balls rattling inside a box. Motion is sensed from accelerometers, directly linking the motions of the user to the feedback they receive in a tightly closed loop. The resulting interface requires no visual attention and can be operated blindly with a single hand: it is reactive rather than disruptive. This interaction style is applied to the display of an SMS inbox. We use language models to extract salient features from text messages automatically. The output of this classification process controls the timbre and physical dynamics of the simulated objects. The interface gives a rapid semantic overview of the contents of an inbox, without compromising privacy or interrupting the user.

Position and orientation tracking is a major challenge for mixed/augmented reality applications, especially in heterogeneous and wide-area sensor setups. In this chapter, we describe trackman, a planning and analysis tool which supports the AR-engineer in setup and maintenance of the tracking infrastructure. A new graphical modeling approach based on spatial relationship graphs (SRGs) eases the specification of known as well as the deduction of new relationships between entities in the scene. Modeling is based on reusable patterns representing the underlying sensor drivers or algorithms. Recurring constellations in the scene can be condensed into reusable meta patterns. The process is further simplified by semi-automatic modeling techniques which automize trivial steps. Data flow networks can be generated automatically from the SRG and are guaranteed to be semantically correct. Furthermore, generic tools are described that allow for the calibration/registration of static spatial transformations as well as for the live surveillance of tracking accuracy. In summary, this approach reduces tremendously the amount of expert knowledge needed for the administration of tracking setups.

Creating a mixed reality experience is a complicated endeavour. From our practice as a media lab in the artistic domain we found that engineering is “only” a first step in creating a mixed reality experience. Designing the appearance and directing the user experience are equally important for creating an engaging, immersive experience. We found that mixed reality artworks provide a very good test bed for studying these topics. This chapter details three steps required for authoring mixed reality experiences: engineering, designing and directing. We will describe a platform (VGE) for creating mixed reality environments that incorporates these steps. A case study (EI4) is presented in which this platform was used to not only engineer the system, but in which an artist was given the freedom to explore the artistic merits of mixed reality as an artistic medium, which involved areas such as the look and feel, multimodal experience and interaction, immersion as a subjective emotion and game play scenarios.

Augmented reality systems often involve collaboration among groups of people. While there are numerous toolkits that aid the development of such augmented reality groupware systems (e.g., ARToolkit and Groupkit), there remains an enormous gap between the specification of an AR groupware application and its implementation. In this chapter, we present Fiia, a toolkit which simplifies the development of collaborative AR applications. Developers specify the structure of their applications using the Fiia modeling language, which abstracts details of networking and provides high-level support for specifying adapters between the physical and virtual world. The Fiia.Net runtime system then maps this conceptual model to a runtime implementation. We illustrate Fiia via Raptor, an augmented reality application used to help small groups collaboratively prototype video games.

The domain of mixed reality systems is currently making decisive advances on a daily basis. However, the knowledge and know-how of HCI scientists and interaction engineers, used in the design of such systems, are not well understood. This chapter addresses this issue by proposing a software engineering method that couples a process for designing mixed reality interaction with a process for developing the functional core. Our development method features a Y-shaped development cycle that separates the description of functional requirements and their analysis from the study of technical requirements of the application. These sub-processes produce Business Objects and Interactional Objects, which are connected to produce a complete mixed reality system. The whole process is presented via a case study, with a particular emphasis on the design of the interactive solution.

This work presents a development approach for mixed reality systems in health care. Although health-care service costs account for 5–15% of GDP in developed countries the sector has been remarkably resistant to the introduction of technology-supported optimizations. Digitalization of data storing and processing in the form of electronic patient records (EPR) and hospital information systems (HIS) is a first necessary step. Contrary to typical business functions (e.g., accounting or CRM) a health-care service is characterized by a knowledge intensive decision process and usage of specialized devices ranging from stethoscopes to complex surgical systems. Mixed reality systems can help fill the gap between highly patient-specific health-care services that need a variety of technical resources on the one side and the streamlined process flow that typical process supporting information systems expect on the other side. To achieve this task, we present a development approach that includes an evaluation of existing tasks and processes within the health-care service and the information systems that currently support the service, as well as identification of decision paths and actions that can benefit from mixed reality systems. The result is a mixed reality system that allows a clinician to monitor the elements of the physical world and to blend them with virtual information provided by the systems. He or she can also plan and schedule treatments and operations in the digital world depending on status information from this mixed reality.

The eXperience Induction Machine (XIM) is one of the most advanced mixed-reality spaces available today. XIM is an immersive space that consists of physical sensors and effectors and which is conceptualized as a general-purpose infrastructure for research in the field of psychology and human–artifact interaction. In this chapter, we set out the epistemological rational behind XIM by putting the installation in the context of psychological research. The design and implementation of XIM are based on principles and technologies of neuromorphic control. We give a detailed description of the hardware infrastructure and software architecture, including the logic of the overall behavioral control. To illustrate the approach toward psychological experimentation, we discuss a number of practical applications of XIM. These include the so-called, persistent virtual community, the application in the research of the relationship between human experience and multi-modal stimulation, and an investigation of a mixed-reality social interaction paradigm.

Running is an enjoyable exercise for many people today. Trainers help people to reach running goals. However, today’s busy and nomadic people are not always able to attend running classes. A combination of a virtual and physical coach should be useful. A virtual coach (MyCoach) was designed to provide this support. MyCoach consists of a mobile phone (real time) and a web application, with a focus on improving health and well-being. A randomised controlled trial was performed to evaluate MyCoach. The results indicate that the runners value the tangible aspects on monitoring and capturing their exercise and analysing progress. The system could be improved by incorporating running schedules provided by the physical trainer and by improving its usability. Extensions of the system should focus on the real-time aspects of information sharing and “physical” coaching at a distance.

In typical mixed reality systems there is only a one-way interaction from real to virtual. A human user or the physics of a real object may influence the behavior of virtual objects, but real objects usually cannot be influenced by the virtual world. By introducing real robots into the mixed reality system, we allow a true two-way interaction between virtual and real worlds. Our system has been used since 2007 to implement the RoboCup mixed reality soccer games and other applications for research and edutainment. Our framework system is freely programmable to generate any virtual environment, which may then be further supplemented with virtual and real objects. The system allows for control of any real object based on differential drive robots. The robots may be adapted for different applications, e.g., with markers for identification or with covers to change shape and appearance. They may also be “equipped” with virtual tools. In this chapter we present the hardware and software architecture of our system and some applications. The authors believe this can be seen as a first implementation of Ivan Sutherland’s 1965 idea of the ultimate display: “The ultimate display would, of course, be a room within which the computer can control the existence of matter …” (Sutherland, 1965, Proceedings of IFIPS Congress 2:506–508).

The domain we address is creative design, mainly architecture. Rooted in a multidisciplinary approach as well as a deep understanding of architecture and design, our method aims at proposing adapted mixed-reality solutions to support two crucial activities: sketch-based preliminary design and distant synchronous collaboration in design. This chapter provides a summary of our work on a mixed-reality device, based on a drawing table (the Virtual Desktop), designed specifically to address real-life/business-focused issues. We explain our methodology, describe the two supported activities and the related users’ needs, detail the technological solution we have developed, and present the main results of multiple evaluation sessions. We conclude with a discussion of the usefulness of a profession-centered methodology and the relevance of mixed reality to support creative design activities.