User-Centered Requirements for Augmented Reality as a Cognitive Assistant for Safety-Critical Services

Over the years, human-centered service systems have evolved, and technology has become smarter over time, necessitating the development of improved systems to leverage new technologies (Peters et al. 2016). As these new technologies are becoming increasingly important for service development and improvement in various industries, the use of these technologies has gained attention in both service research and practice (Ostrom et al. 2015; Wirtz et al. 2018). When adopting new technologies, IT resources either complement or improve the impact of non-IT resources on the performance of a process or replace them (Jeffers et al. 2008). Well-known examples of services enhanced by technology are electronic check-in and check-out systems in hotels, electronic money transfer in financial services, e-tickets in the airline or event industry, etc. (Collier 1994; Böhmann et al. 2014). Focusing on the human factor and its capabilities when improving services, so-called cognitive assistance systems, which explicitly aim at enhancing human capabilities instead of replacing them, play an important role (Engelbart 1962; Peters et al. 2016). In this context, cognitive assistants offer the possibility of significantly improving the functioning of complex service systems (Peters et al. 2016).

An example of a cognitive assistant that has been studied in various research areas is augmented reality (AR) applications. With the support of AR head-mounted displays (HMDs), for instance, current challenges, such as in technical customer services, can be overcome (Niemöller et al. 2019). HMDs can be used to restructure services by enabling information to be displayed directly into the user’s field of view under hands-free navigation without media breaks and limited mobility (Niemöller et al. 2019). As a result, AR cognitive assistants, like HMDs, have a high potential to support and improve (Elder and Vakaloudis 2015) services such as health care (Klinker et al. 2017), technical customer service (Niemöller et al. 2019), and logistics services (Rauschnabel and Ro 2016).

While previous studies have primarily focused on services in which one task is performed, it is not well understood how AR cognitive assistants can support services where two separate tasks are frequently switched. This task switch is the situation in our use case of water depth management at a large European maritime logistics hub. As part of the water depth management, deviations in water depths of the port area must be continuously monitored, and if necessary, harbor areas must be dredged to maintain the waterway infrastructure (Osterbrink et al. 2021; Bräker et al. 2022b). The measurement of water depths is called soil sounding and is provided by special vessels equipped with echo-sounding systems. Floating drones can be used individually or to support vessels to collect supplementary measurement data by being more agile and flexible. Skippers and drone operators are responsible for two parallel tasks during service delivery. First, they must navigate their vessels safely through the harbor area during regular maritime traffic, i.e., taking into account other vessels, water levels, tides, currents and obstacles in the water. Second, the measurement is a task that needs high accuracy and is a collaboration with a measurement engineer. This involves continuously coordinating measurement areas and routes with the measurement engineer, operating the echo sounder system, and checking the data quality of the measured water depths in real time. The measurement engineer, who is likewise on board the vessel, guides the measuring route and controls the quality of the measured real-time data. Since mistakes or inaccuracies in the performance of the skippers and drone operators can have serious consequences, directly and indirectly, the use case is a safety-critical service. These can generally be characterized by the fact that a “failure might endanger human life, lead to substantial economic loss, or cause extensive environmental change” (Knight 2002, p. 547). Especially for this reason, effective support and improvement of the soil-sounding process are very important, as the resulting information is crucial for vessels to navigate harbor areas. Consequently, soil sounding is a key service to ensure safety for all actors that use the harbor infrastructure, as this infrastructure is partially dynamic due to tides and currents. Besides the two parallel tasks, another distinctive feature of our use case is the setting on the water. In contrast to the already difficult implementation of AR in navigation contexts such as cycling or driving a car, navigation on the water brings further challenges for developing an AR application. The instability of the water surface brings an additional dimension of motion that leads to, among others, much more complex tracking conditions. However, AR applications with an additional dimension of motion have not yet been investigated, so this represents another gap in previous research. Against this background, our research is guided by the questions (RQ1) “How amenable is the service process of soil sounding to be supported by AR?” and (RQ2) “What are the requirements for an AR cognitive assistant to improve the service process of soil sounding?”. The results demonstrate how AR can improve operations in an edge case in the context of safety-critical maritime navigation with high requirements. We derived five requirements for such applications that inform future AR solutions in a wide application area. Additionally, we deal with unique tracking challenges that increase the understanding of AR and support systems when dealing with applications for use on the water.

The remainder of the paper is structured as follows: First, we give an overview of the application of AR solutions in different service processes. We distinguish between static settings and those where the environmental setting adapts to the user’s motion, such as when driving a car and describe their technological challenges. In Sect. 3, we introduce the methodology. We conducted a case study with workshop data, expert interviews, and process tracing using video observations and verbal protocols in the form of think-aloud sessions. In Sect. 4, we present the use case of water depth measurement and analyze it firstly with regard to its augmentability (RQ1). In the second part of the analysis, we use the conducted think-aloud sessions to derive the user-centered requirements for an AR cognitive assistant for the service process of soil sounding (RQ2). The results and their limitations are discussed in Sect. 5, and in Sect. 6, we summarize the main findings and give an outlook on future research issues.

Innovations in the field of service continue to emerge, with many of the innovations today being digital by integrating resources throughout service systems (Lusch and Nambisan 2015). In order to apply service innovations and related innovative technologies that can be used to support services, it is important to understand the innovations within the service for which they are to be used, their potential, and the requirements involved (Matijacic et al. 2013; Jessen et al. 2020). One opportunity to support service is cognitive assistants, enabled or represented by rapidly evolving technologies and innovations. These assistants are primarily characterized by the aim to enhance human capabilities rather than replace them (Engelbart 1962; Peters et al. 2016).

One promising technology that falls into this category of cognitive assistants is AR. As previous research shows, this technology can be used in many different environments. In the context of maintenance and repair, increased efficiency was observed for using AR-applied devices (Henderson and Feiner 2011). Another AR application that was investigated is the use of smart glasses to support the runtime modeling of services, allowing the process to be documented on-site by the service provider during the execution of his activity (Niemöller et al. 2019). Further applications of AR do exist in the construction industry (Etzold et al. 2014), education (Kaufmann and Schmalstieg 2002), as well as healthcare (Klinker et al. 2020). These use cases tend to occur in static environments, i.e., the user does not move significantly, and the environment remains mostly constant.

On the other hand, AR use cases in mobile settings have rarely been the subject of research. These environments can be characterized by the fact that they are not limited in space and that objects – in contrast to those in static environments – do not necessarily have fixed positions, but the positions of these objects can change due to movements so that a motion emanates from the user’s environment. Examples of mobile environments include navigation contexts, such as driving a car, where the motion of other traffic participants influences their own process of driving, e.g., when it is necessary to brake because of the car in front. With regard to AR applications in the context of moving environments, the study by Berkemeier et al. (2018), in which requirements for an acceptable smart glasses-based information system to support cyclists in cycling training are identified, can be mentioned. The study aims to augment information such as speed or route details, which would otherwise have to be displayed by other devices, into the cyclist’s field of view to promote road safety. There are also studies on cars in which head-up displays or HMDs display relevant information in the driver’s field of view to reduce road traffic risks (Heymann and Degani 2016).

There are additional challenges when using AR in a maritime context compared to use cases in the domain of cycling or driving a car. One of the biggest technical challenges relates to tracking: “A ship is not fixed but will roll, pitch and yaw. So – depending on the use case – we have to determine the movement of the ship relative to the earth and the movement of a person (or device) relative to the ship” (von Lukas et al. 2014, p. 466). This means the vessel can move on the water in six degrees of freedom (DOF) (see Fig. 1). Mainly rotation motions are caused by the movement of the water on which the vessel is sailing, e.g., by currents or waves. In addition, the user can move on the ship in six DOF when wearing an HMD. The combination of the two makes tracking complex and challenging. In addition, harsh weather conditions, reflections from the water, and lack of GPS reception make accurate tracking difficult. For this reason, our use case presents a unique characteristic that raises a research gap that needs to be investigated.

Moreover, in other safety-critical services – as our use case – there are already some investigations on the support potential of AR and how it can be used. For example, the user requirements for an AR-based refinery training tool were determined for an oil refinery to develop usable and safe AR applications (Träskbäack and Haller 2004). Another example is the application of AR in an innovative airport control power, where the aim was to provide the air traffic control operators in the airport control tower with complete head-up information (Balci and Rosenkranz 2014; Bagassi et al. 2020). All safety-critical services require situational awareness of involved actors. Situational awareness in this regard is defined as “[…] the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley 1995, p. 36). The phenomenon is crucial in many applications and a lever for improved safety (Carroll et al. 2006; Jones et al. 2018; Reuter et al. 2019). Especially, AR bears the potential to improve situational awareness, as some recent findings conclude (Coleman and Thirtyacre 2021). In a user study, Lee et al. (2016) investigated how multitasking while driving distracts users and causes misbehavior. Study participants had to solve tasks on a classic infotainment system while driving a car. In particular, the risk of accidents increased when participants made mistakes in the parallel task and were distracted by view shifts and long glances at the infotainment system. Our research addresses a similar situation, albeit not in road traffic but maritime traffic.

Illing et al. (2021) examined in a study to what extent AR can help to support multitasking assembly exercises under time pressure. Especially when two or more tasks are performed in parallel, mental stress and reduced attention can occur. The authors were able to show that AR assistance systems can help to reduce the error rate and the cognitive load and, at the same time, increase the quality and motivation of the actors. The possibility of performing the task hands-free by using AR improves safety and helps users to comprehend the actual task faster. Hands-free working helps to support collaboration with other actors. Especially in mobile settings, where users move around the environment and have to collaborate and communicate at the same time, HMDs bring an advantage (Johnson et al. 2015). Nevertheless, in processes that require situational awareness and hands-free working while collaborating at the same time, AR support brings great benefits. An example of this is the security industry, where security personnel needs to be aware of the environment and simultaneously collaborate with, for example, the police while performing other activities that require hands-free working (Lukosch et al. 2015).

Theoretical accounts have developed concepts to capture how technologies affect service processes. A well-known theory is the process virtualization theory, which is concerned with explaining and predicting whether a process can be performed virtually (Overby 2008). Examples of virtualized processes include e-commerce, online distance learning, and online banking (Overby 2008; Balci and Rosenkranz 2014). The theory has been applied and adapted in many different contexts (Osterbrink et al. 2021; Bose and Luo 2011; Balci and Rosenkranz 2014) and was also examined with regard to augmentation, which led to the theory of process augmentability (Yeo 2017), which provides the basis for the analysis of the augmentability of our use case of the soil-sounding service. In contrast to the definition of virtualization, augmentation can be defined as the supplement of a synthetic or physical interaction. Consequently, the main constructs of the process virtualization theory concerning the potential removal of physical interactions from the process cannot be applied to augmentation. Therefore, a new main construct – the authenticity requirement – is proposed to affect the dependent variable, namely process augmentability, positively. Process augmentability is described as “how amenable a process is to being conducted in AR environments” (Yeo 2017, p. 5). The need for authenticity influences process augmentability, meaning that AR offers the possibility to provide authentic experiences, although a process is virtualized. An authentic experience is given if sensory genuineness is needed. Because AR can simulate sensory experiences, processes that require this authentic feeling are amenable to being augmented. Besides, three moderating constructs developed from the definition of AR are proposed to create an authentic experience. By definition, AR must meet three criteria: combine physical and virtual, be interactive in real-time, and be registered in the real world (Azuma 1997). Accordingly, Yeo (2017) proposes 3D visualization, spatial association, and synchronization as moderating constructs affecting authenticity, hereafter referred to as characteristics. The requirement for 3D visualizations means that three-dimensional objects and overlays enable a more authentic experience. For the characteristic of spatial association, examples were given by Yeo (2017) of how AR uses the geospatial environments to enrich process experiences (Henderson and Feiner 2011; Hertel et al. 2021; Bräker et al. 2022a). The spatial registration of objects in space influences augmentability, and if objects must be spatially anchored for an authentic experience, the process is suitable for AR. In terms of the synchronization characteristic, most physical processes conducted in the real world tend to be synchronous. Therefore, physical movement needs to be connected to maintain the sense of immersion (Overby 2008). This means that synchronization ensures an authentic experience, i.e., the process is more authentic if no delays can be tolerated.

Based on existing research, our case delineates as follows: we are in a mobile wide-area environment that is constantly changing. At the same time, we are inside a vessel whose interior initially remains static. Due to the vessel’s movements on the water, compared to conventional navigation settings, there is an added challenge that the water surface is unstable, and thus the vessel can move in 6 DOF. In addition, our users, the skippers, are performing a safety-critical process where errors can have drastic effects on themselves and others. They must be able to multitask, meaning they perform two parallel tasks and simultaneously collaborate and communicate with other involved parties. The combination of all these characteristics makes our use case a technological edge case, as it is particularly challenging, complex, and thus interesting for research. Likewise, aspects of it are found in loads of other use cases.

We conducted a case study (Yin 2003) to explore the requirements for a user-centered AR cognitive assistant for safety-critical services in the maritime sector. The use case we investigated was the process of soil sounding, i.e., water depth measurement in a harbor environment carried out during navigation on a vessel or with the help of a soil-sounding drone.

In order to analyze how amenable the considered use case of the soil-sounding process is to be supported by AR (RQ1), we evaluated to what extent the aspects of the theory of process augmentability (Yeo 2017) are fulfilled. To assess the degree of fulfillment of the process augmentability criteria, we first studied a use case video of a European harbor operator that gave us a contextual understanding of the soil-sounding process as a foundation for further analysis. In the second step, we collected data by conducting two semi-structured interviews and two workshops with business and IT experts of the same harbor operator (see Table 1). The first interview was held with the port operator’s Head of IT Innovation to get an initial overview of the use case. The second interview occurred with the Deputy Port Hydrographer of the same port operator. It gave us a deeper understanding of the use case and the context in which the soil-sounding process is performed. Both the interviews and the workshops were documented by video recording. In addition, the participants provided more detailed documents concerning the use case, such as the depth measurement management cycle (see Fig. 3) and information about and pictures of the soil-sounding vessel and drone (see Figs. 4 and 5). Two workshops further contributed to the understanding of the use case. In the first workshop, with the Deputy Port Hydrographer, researchers, and port operator personnel, we briefly discussed the first ideas for initial requirements for an AR cognitive assistant for the service process of soil sounding (RQ2). In the second workshop, the Project Manager R&D, researchers and further port operator personnel participated. During this workshop, we defined the single process steps of the soil-sounding service process and discussed which initial requirements might apply to each process step.

Since we wanted to identify user-centered requirements, we collected additional data with three skippers and one drone operator who performed the process of soil sounding. For simplicity, the common term “actor” is used hereafter for skippers and drone operators when both are meant. We were particularly interested in safety aspects, environmental conditions of the vessel, the cognitive and physical demands of the skipper, the type and intensity of collaboration between the actors, and general challenges and problems encountered by the actors in their daily work. In order to gain this central data source, we used the process-tracing method (Todd and Benbasat 1987) of thinking aloud (Van Someren et al. 1994), where the actors were asked to “think aloud” and to explain everything they do while simultaneously engaging in the soil-sounding service. We used this method to analyze which user-centered requirements an AR cognitive assistant must meet to support the soil-sounding process. As the situational features of the service are crucial, we extended the traditional implementation of the method by recording videos from different perspectives. For this purpose, we attached a camera to the actors’ forehead to retrace their field of view and placed another camera on the monitor to observe the actors directly (see Fig. 2). An exact tracking of the eye movements of the actors was not necessary since only the direction of the actors’ view was relevant. With the help of the two recording perspectives and the recorded audio track, we were able to trace the process of soil sounding and derive user-centered requirements for an AR cognitive assistant.

As the service is critical to maintaining harbor operation, only experienced actors are considered for soil sounding. The fourteen think-aloud sessions differ in the type of soil-sounding job and its focus as well as the difficulty of the soil-sounding, which depends on factors such as the soil-sounding environment or traffic volume (see Table 1). For each soil sounding, we added anonymized codes for the skipper and the measurement engineer. During the think-aloud sessions, three different skippers (S_A–S_C), one drone operator (D_A) and four measurement engineers (M_A–M_D) were involved.

For analyzing the video recordings and verbal protocols of the think-aloud sessions, we used the scanning method, which is one of the four major categories of protocol analysis and the most straightforward one (Bouwman 1983). We did not perform a verbatim transcription of what was said since the observation of the actors was the primary object of investigation that we used for our analysis. Instead, the actor’s statements helped to supplement and explain what was observed. For the identified video sequences in which observed behaviors indicated challenges, we transcribed what was said, which often helped clarify and support the observation. We analyzed the video and audio material with three independent researchers and initially focused on existing challenges and problems in the process of soil sounding. After identifying all difficulties in the process, we derived problem categories by grouping duplicates and similar ones. Based on these problem categories, we derived contextual requirements for each problem. Afterward, we abstracted from the contextual requirement and derived five general requirements for augmenting the soil-sounding service. The detailed results are described in Sect. 4.2.

For our analysis, we chose the service process of water depth management in a European maritime logistics hub. Harbor personnel must continuously monitor water depth change due to sedimentation and erosion. To ensure the safety of vessel traffic and maintain the infrastructure, water depth management must be ensured by permanent soil sounding as well as finding and recognizing nautically critical obstacles on the water ground, e.g., bikes, cars, or shopping carts. Special soil-sounding vessels and drones are used that have the technical equipment to monitor water depth and generate a digital landscape model of the water ground live on board. Figure 3 illustrates the depth measurement management cycle to give an overview of the use case context. This states that measuring water depths generates knowledge, which in turn causes action, such as deepening a certain area. Since this action causes a change, the changing area must be controlled by measurement.

The most frequently used technique for measuring water depth is the use of soil-sounding vessels (see Fig. 4). These are special vessels equipped with an echo sounder that enables real-time measurement of water depths under the vessel. Different soil-sounding vessels operate daily in the harbor area, which an autonomous floating drone can optionally support. Each vessel needs to sound different defined areas in the port daily and investigate them for changes and irregularities to ensure safe operations within the harbor. Soil-sounding assignments and allocations are managed centrally by a hydrographic office and prioritized as needed and based on tides. The assignments are usually processed by two people on board each vessel: a skipper and a measurement engineer.

The task of measuring water depth is particularly challenging: The skippers of soil-sounding vessels use echo sounders to measure the water depths in the harbor while simultaneously paying attention to the vessel’s traffic and keeping an eye on a variety of information for measurement on monitors so that they have to permanently switch between the monitors and the view out of the window. This leads to limited safety in vessel traffic, extreme exhaustion due to the constant change of view and context, and several health issues. These side effects of the constant shift of perspective can also be observed in other safety-critical services, such as air traffic control in an airport control tower (Bagassi et al. 2020). In addition to simultaneous navigation of the vessel and measurement of data, the skipper needs to interact with the measurement engineer, who is accountable for controlling the quality of the measured data. If the measurement engineer cannot be on the vessel in person, it is possible to work from the home office.

To efficiently perform the measuring tasks assigned by hydrography, the skipper and the measurement engineer need to determine the best possible order of the tasks at the beginning of the workday. This depends on external factors, such as tide, currents or other vessels blocking measurement areas. In addition, the internal prioritization of the hydrographic office is considered. If the order of the assignments has been coordinated, the skipper navigates to the next measuring area. In consultation with the measurement engineer, the skipper either navigates the area at his own discretion and expertise or the measurement engineer places a profile with an outline on the skipper’s map view so that the skipper has a point of reference. In addition, the measurement engineer can draw a so-called bearing line for the skipper, which specifies the optimal route through the measurement area. Once the soil-sounding data is available in the appropriate quality, the next measurement area is approached.

In addition to the described approach using special soil-sounding vessels, floating drones can be used for certain use cases (see Fig. 5). The soil-sounding drone is approximately 1.65 m long and is controlled either from land or the soil-sounding vessel. It can be operated manually by a drone operator via remote control or autonomously along a predefined route given by the measurement engineer, which he defines on his monitor. On board the drone – analogous to the soil-sounding vessel – an echo sounder is installed and, in addition, a camera whose image shows the environment from the drone’s point of view and is streamed to the monitors of the drone operator and measurement engineer. Compared to the conventional approach of measuring water depth with a soil-sounding vessel, the drone is used for tasks where it would otherwise be too dangerous to use a soil-sounding vessel, for example, because the water is too shallow or maneuvering is difficult. In addition, the drone can be used as an escort vessel to assist in sounding large areas.

The main difference compared to the soil-sounding vessel is the vehicle size and the actor’s perspective. In contrast to the skipper, the drone operator can take two perspectives: the perspective from outside the drone and the drone’s perspective via the installed camera. Even though the perspective is slightly different from that of the skipper, the same negative side effects occur when controlling and measuring with the drone as when working with a soil-sounding vessel since the drone operator is also subject to a constant shift between different views. Furthermore, at long ranges, the distances between the drone and other objects in the water and the shoreline are more difficult for the drone operator to estimate. While the drone must always remain in view for safety and regulatory reasons, this can nonetheless be hundreds of meters to several kilometers away from the drone operator in broad harbor basins. Even if the drone operator additionally gets the camera image of the drone streamed to a display, this does not provide a substitute for the outside view of the drone since the image can be disturbed by splash water, for example, and it only visualizes a limited camera image and not a complete 360-degree view at once. Thus, the drone operator also has parallel tasks – first, the safe and precise control of the drone using two different perspectives, and second, the control of the measurement data in collaboration with the measurement engineer. Even in cases where the drone is operating autonomously, the drone operator must maintain full attention, as he or she must always be able to intervene in an emergency. For simplicity, the analysis of the use case uses the common term “vessel” for the soil-sounding vessel and the soil-sounding drone when both are intended.

Before investigating what requirements an AR cognitive assistant must meet to support the service process of soil sounding, we examined the augmentability of the process using the theory of process augmentability (Yeo 2017). To answer the first research question (RQ1), we could determine that all four characteristics – which according to the theory, must be present in a process – are existent in the process of soil sounding so that augmentation of the process is sensible and could help to facilitate and improve the process.

Knowing the augmentation potential, we derived five generalized user-centered requirements for the soil-sounding process, using the results of the thinking aloud sessions as a foundation: (1) real-time overlay, (2) variety in displaying information, (3) multi-dimensional tracking, (4) collaboration, and (5) interaction requirement. This answers our second research question (RQ2). The requirements for an AR solution as a cognitive assistant, which we have determined with regard to the navigation task of the actors, correspond to the results of previous research on AR applications in road traffic. However, prior research did not investigate such a complex process in the maritime industry, so we are contributing to the research at this point. On the one hand, the moving vessel, in combination with the moving user inside the vessel, poses a great challenge in terms of multi-dimensional tracking possibilities. On the other hand, it is a safety-critical process that requires multitasking and situational awareness, i.e., a constant shift between the navigation of the vessel and the measurement of water depth, as well as collaboration with the measurement engineer. With the help of these five requirements, we provide practitioners and scholars with a foundation to assist in the development of AR applications in the above environment.