Collaboration in Augmented Reality

In the Otherland saga, Tad Williams (1996, 1998, 1999, 2001) describes a future world with a widespread availability of full-immersion Virtual Reality (VR) installations which allow people to access an online world, called simply ‘the Net’. Within the Net, a group of people aim to achieve immortality. In his novel Rainbows End (Vinge 2007) Vernor Vinge describes how the main character, Robert Gu, is slowly recovering from Alzheimer’s disease and adapts to a changed world in which almost every object is networked and the use of Augmented Reality (AR) is normal. Humans experience AR by wearing smart clothes and contact lenses that can overlay the physical environment with computer graphics. In this future world, AR technology is used for various purposes, such as large-scale commercial gaming, supporting maintenance workers with blueprints of machines or buildings, communication with virtual avatars, and medical diagnosis.

AR allow users to see the real world, with virtual objects superimposed upon or composited with their real environment (Azuma 1997). Here the virtual objects are computer graphics that exist in essence or effect, but not formally or actually (Milgram and Kishino 1994). AR systems are not limited to use of head-mounted devices (HMD), but AR technology mainly has to combine real and virtual objects, to be interactive in real-time, and to register virtual objects within the real 3D environment (Azuma 1997).

Williams and Vinge forecast a vision for the future that current research on collaboration in AR is addressing. Several years from now, technology will provide an infrastructure for physical and virtual connectivity just as described in Rainbows End. Everyday objects will be connected and be able to provide and exchange information. Instead of only overlaying the physical environment with computer graphics, future AR systems will move closer to the ultimate display that Ivan Sutherland envisioned as “a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.” (Sutherland 1965). Such a display would allow for holistic embodied experiences addressing all of the human senses, such as sound, smell, taste and touch, as envisioned in Otherland. Early science fiction motion pictures, e.g., Star Wars (Lucas 1977), used holistic experiences to support the interaction and collaboration of geographically distributed persons. When this fiction comes true, the use of and interaction within AR environments will be as natural as in the real world.

First steps towards a combined vision of Williams and Vinge have already been taken. There has been research on introducing smell (Kim et al. 2011) and touch (Samur 2012) into movie theaters and television. However, one of the most difficult aspects to reproduce is realistic interaction with other (real or virtual) humans. Olson and Olson (2000, 2014) analysed the efforts of geographically distributed teams to use technology to create sense of being in one place and work together. They came to the conclusion that distance matters and that current technology is not mature enough to enable virtual co-presence, and even future technology will struggle to enable this. In their opinion, providing awareness among co-workers and enabling co-reference as well as spatial referencing will remain a challenge. Gaver (1991) stresses the importance of supporting awareness information to help actors shifting from working alone to working together.

Considering current groupware technology, this forecast is still correct. Complex problem solving still requires a team of experts to physically meet and interact with each other. The identification of the problem and the creation of a shared understanding are major challenges for efficiently solving a problem (Piirainen et al. 2012). Unfortunately, due to experts’ availability, critical timing issues or accessibility of a location, it is not always possible to bring a team together to handle a complex situation. While in the novel Rainbows End, such situations are supported with AR technology, current AR experiences are not there yet.

This special issue includes four articles that focus on remote collaboration using different types of AR systems. They advance the state of the art and provide insights about how AR can be used for remote guidance, what are critical design factors, and what is the impact on awareness. Most importantly, they show how close current AR systems are to the future Science Fiction visions described above. After reviewing the state of the art of collaboration using AR systems, this introduction builds upon the articles in this special issue and outlines a research agenda for future work on remote collaboration supported by AR systems.

Several studies have explored the effectiveness of using AR for complex tasks. For individual users, Baird and Barfield (1999) showed that AR can improve the effectiveness of assembly tasks. Similarly, Tang et al. (2003) showed that the use of AR improves performance and reduces mental effort in assembly tasks. Bauer et al. (1999) show that in a distributed setting a remote user can effectively guide and direct a local user’s activities using an AR telepointer. Wang and Dunston (2011) showed that AR systems can improve performance time and mental effort in collaborative design tasks. Dong et al. (2013) found that AR facilitates communication and discussion of engineering processes. Alem and Li (2011) showed that AR can improve the satisfaction with remote collaboration on physical tasks. Recent results on using AR for collaboration among crime scene investigators, indicates that it supports mutual understanding, leads to consensus and supports hypothesis testing (Poelman et al. 2012).

There are several examples of using AR technology for supporting face-to-face collaboration. The Transvision system (Rekimoto 1996) allows multiple users to share computer-generated graphics on a table. The Collaborative Web Space (Billinghurst and Kato 1999, 2002) allows users in the same location to collaboratively browse the web while seeing the real world and communicate naturally about the visited pages. The Studierstube system (Schmalstieg et al. 2002; Szalavári et al. 1998) targets presentations and education and allows users to walk around shared virtual 3D scientific data superimposed over the real world. EMMIE (Butz et al. 1999) uses AR to connect people and devices by visualizing cross-device interactions, search, and privacy status during a meeting. VITA (Benko et al. 2004) supports multiple users in exploring scaled and full-size representations of an archaeological dig. More recent systems considered the use of AR in dynamic emergency response tasks (Nilsson et al. 2009) or within a board game, in which users play social games with physical game pieces using a handheld on a tabletop (Huynh et al. 2009). Similarly, ARVita (Dong et al. 2013) enables multiple users around a table wearing HMDs to observe and interact with dynamic visual simulations of engineering processes. Some of the key lessons learned from these systems are that users can interact with shared AR content as naturally as with physical objects, AR reduces separation between task space and communication space, and that AR enhances natural face-to-face communication cues.

Examples for AR systems that support remote collaboration can be found in Billinghurst and Kato (1999), Höllerer et al. (1999), Stafford et al. (2006), Minatani et al. (2007), Poelman et al. (2012), Datcu et al. (2014a), or Billinghurst and Thomas (2011). For example, WearCom (Billinghurst and Kato 1999) enables users to see remote collaborators as virtual avatars in real space. Höllerer et al. (1999) allow indoor AR users to visualize the locations and paths of outdoor AR users, and create shared annotations. Stafford et al. (2006) introduce a new interaction metaphor called “god-like interaction” which supports multi-scale interaction between outdoor wearable AR users and indoor users using a tabletop projector, and aims at improving sharing of situational and navigational information between remote users. Minatani et al. (2007) describe a system that enables two distributed users to a play a tabletop game in an AR conferencing scenario. Poelman et al. (2012) introduce and evaluate an AR system that allows remote experts to collaborate with local investigators on a crime scene investigation in order to secure evidence. Along the same line Datcu et al. (2014a), present a system that supports situational awareness of cross-organisational teams in the security domain. Some of the key lessons learned from these systems are that AR technology can reproduce some of the spatial cues used in face-to-face collaboration that are normally lost in remote conferencing systems, AR can increase social presence compared to other technologies, and AR also allows remote collaborators to interact naturally in the local user’s real environment (Billinghurst and Thomas 2011).

In addition to the above systems, collaborative AR has also been used in several industry scenarios such as product design, maintenance or factory planning. The ReMote system (Alem et al. 2011; Huang et al. 2013) is a remote assistance system that allows a remote expert to assist and guide in real time a local maintenance operator in a mine site. The remote expert uses their own hands (hand gestures over a live video of the workspace) as well as annotations on still images to explain how the repair /maintenance task needs to be performed. The MagicMeeting system (Regenbrecht et al. 2002) is a multi-user AR system that allows up to four co-located users to have a design zone review meeting. Fata Morgana (Klinker et al. 2002) supports five different scenarios for car designers where they can rotate a car, perform an overview evaluation, focus on details, discuss with colleagues, and compare designs. Roivis (Pentenrieder et al. 2007) is an AR application that supports factory design and planning between co-located users that share a view. DARCC (Hammad et al. 2009) is a distributed AR application that allows multiple users to simulate collaborative construction tasks, e.g., coordinating crane operations. Gauglitz et al. (2014) introduce a tablet-based system that incorporates a touch screen interface through which a remote user can navigate a physical environment and create world-aligned annotations for supporting maintenance. Oda et al. (2013) present a system for remote equipment maintenance. These systems show that collaborative AR technology could have significant impact in design and industrial applications.

In summary, AR technology is becoming mature enough to support a variety of complex collaboration scenarios (Carmigniani et al. 2011; Piekarski and Thomas 2009). However, there are still a number of open issues that need further research. One major issue with regard to AR collaboration is in relation to the presence of remote users (Billinghurst and Thomas 2011). Local users may feel remote controlled by the remote experts, while the experts may feel that they miss something when not physically being present at the scene (Poelman et al. 2012). It is unknown if the proven concepts of awareness support from traditional desktop collaboration situations (Schümmer and Lukosch 2007) can be transferred to AR interaction spaces, or whether completely new approaches are necessary. Billinghurst and Thomas (2011) also point out that further research on how users can interact within collaborative AR systems is necessary. For example, investigating whether bare hand interaction or interaction with a physical object is more effective in AR (Datcu et al. 2015).

Recent research has shown that virtual co-location is in fact possible. Poelman et al. (2012) allows experts to spatially collaborate with others at any other place in the world without traveling and thereby creating the experience of being virtually co-located. In the field of crime scene investigation, a remote expert guides and collaborates with a colleague in a crime scene to collect evidence. In this setting, the remote colleague wears a head-mounted display with a camera and shares the local view with the remote colleague, while both can annotate what they see with virtual objects. Datcu et al. (2014a, b) use AR to establish virtual co-location and improve the telepresence of a remote colleague for police investigations, fire fighting or reconnaissance. In 3DReMoTe (Huang and Alem 2013; Huang et al. 2013), the expert sees a real time a reconstruction & display of a remote workspace via a HMD. The hands of the expert are captured in 3D and co-located in the 3D remote workspace, allowing the expert to assist in spatial tasks. These projects show that AR can enable virtual co-location and allow experts at a distance to interact with local users to perform collaborative tasks.