Production logistics and human-computer interaction—state-of-the-art, challenges and requirements for the future

Automation and application of artificial intelligence (AI) are a rife topic in production, transportation, and logistics—from autonomous cars and trucks to ergonomic enhancements of workers in production process handling and picking for example [16‐20]. Manufacturing and production management changed over the last decades, e.g., with the advent of cheap sensors and actuators communicating within the Internet, enabling the real-time connection between systems, materials, machines, tools, workers, customers, and products as the IoT [3, 21, 22]. The resulting data volume (Big Data) generated by all connected objects represents the new raw material of our time, changing many business models and environments. IoT allows for a production paradigm with digital customer involvement from the development and design phase [7, 23‐25]. At the same time, market demand requires high volumes of individual products, bearing on I40 with a transformation of industrial production by merging the Internet and information and communication technologies with traditional physical manufacturing processes [20, 26, 27].

With I40, new smart factories grounded on the manufacturing and assembly process digitalization are expected. Such factories are characterized by a high production flexibility through the use of reconfigurable machines that enable a personalized production in batches as small as lot size one [28, 29]. A remarkable opportunity to target these goals is the development of a brand-new generation of manufacturing and assembly systems implementing the I40 principles to production processes (Fig. 1).

Besides that, complex production systems and strategies like, e.g., make-to-order, require advanced supply and logistics concepts as high-volume stock keeping is not feasible for such production strategies and customer demands regarding quality and delivery time have to be met nevertheless [20, 30, 31]. Especially the automotive industry has been a strong leader and innovator in applying new automation and enhancement technologies. This has been true for logistics concepts like just-in-time or just-in-sequence in the past and is still true today for innovations such as I40 and CPS [25‐27, 32, 33].

In order to shed light on the actual business practice developments in production logistics as well as challenges and hurdles in automation as requested by I40 concepts, the following section is outlining a current process study of a mid-size firm. The inputs are also essential in terms of the derived HCI optimization model in the later part of the paper.

The example case provided here is addressing a mid-size logistics company. The company has about 200 employees and is strongly innovating company processes, especially in the areas of order picking and logistics. The company is involved in many production logistics settings for larger companies in retail and manufacturing and has a focus on automation as workers still handle heavy materials and products. On a site area of 40,000 m², the workers perform typical production preparation processes like re- and co-packing. Usually, a wide variety of specific product and process requirements are demanding high qualification levels of the included workers. For example, they have to keep meticulously the required quantity and quality as well as scheduling and sequence profiles. In order to allow for comprehensive tracking and tracing of all activities, they use support systems to document all movements and actions, e.g., with barcode scanning or voice approval of used parts. Moreover, they rely on fully automated processes, e.g., for securing transport items within the site and towards other production areas. Furthermore, the company applies automation in order to safeguard workers regarding hard working conditions, with high ergonomic challenges especially for the back during lifting activities as well as during repeated monotonous handling activities performed many thousand times a day. Automation and robot support in this area is also helping the company to enlarge the potential worker pool as human resources are scarce and there are not many people willing to work in production logistics. Therefore, this study reports and discusses the hurdles and experiences with additional automation steps and concepts, also in order to derive further insights for a generalized HCI model in production logistics.

Altogether, the example process described shows that including workers in the change management process, allowing for an extended test and implementation period as well as top management commitment is crucial for successful automation. This is in line with traditional change management concepts. In addition to these internal factors, external factors such as customer requirements and technical or market influences are important for automation changes and have to be managed and included sensibly too. For example, the range of required variants and sequences from the demand side may well restrict the technically feasible automation options available, which is a crucial result before entering expensive change processes towards automation.

In most production settings, solutions are connected to the state-of-the-art in computer science and robot technology as well as artificial intelligence applications, which are outlined in the following section in order to complement this first perspective.

The ongoing development in recent years brought a larger paradigm shift to HCI: new sensors allow for new interaction modalities.¹ While mobile devices recently introduced the most notable change (touch and gesture-based interaction evolved into the main interaction modality), the advances in machine learning, cognitive computing and sensor technology will have a larger impact on how we interact with machines. We also see constant improvement in the field of language and speech processing. We will now review recent advances in HCI with regard to autonomous systems and then focus on how to create the interaction with such systems. It should be noted that we use the terms autonomous systems and robots synonymously.

Sheridan [35] reviews the status of Human-Robot-Interaction (HRI) in recent times and proposes a set of different challenges that need to be solved for proper interaction with autonomous systems. For the sake of brevity, we will focus on three topics that Sheridan [35] identifies: (i) tasks must be clearly divided between humans and robots, i.e., when developing the system, we must explicitly define which tasks should be carried out by humans and/or autonomous systems. (ii) A direct consequence of a robot’s tasks is its physical form, whose definition itself is a major challenge. Furthermore, possible consequences of robot actions must be determined to avoid negative outcomes. Finally, (iii) humans need a mental model of the robot’s capabilities and vice versa—also robot’s need to understand human actions and reactions.

Furthermore, research suggests that humans feel better when autonomous machines explain their actions [36] and communicate their intents [65]. For example, different modalities can be used to inform pedestrians about an autonomous vehicle’s planned actions [38], which is important because pedestrians tend to rely on direct communication with drivers. Autonomous systems should also incorporate methods for signaling problems to humans, especially due to the increasing reliance on a robot’s action. Users seem to perceive robots communicating failures and problems as more trustworthy [39]. Also, Honig, Oron-Gilad [40] point out that recent research tends focus on technical reliability of the autonomous system and do not take problems resulting from error-prone interaction between human and machine into account. The authors suggest a taxonomy specifically distinguishing between technical failures and interaction failures; the latter including interaction between autonomous systems and humans as well as between different autonomous systems.

Alongside the new HCI capabilities, a new quality aspect for applications emerged: the User Experience (UX). Users today expect applications not only to function well, but to be fun and enjoyable, too. While no widely accepted definition of UX exists, most acknowledge it to be a highly subjective matter and that interaction with the application and context of use play a dominant role [41]. Hassenzahl, Tractinsky [42] describe three UX aspects derived from scientific literature: technology is more than a mere tool; the user’s context and internal state are important; and each user has an individual, subjective sense of good UX. Their result is that these three cannot be separated, but overlap, and they conclude that HCI surpasses the pure usefulness of applications and lies at the core of a good UX. Also, Hinckley, Wigdor [43] and Watzman, Re [44] define UX with a special focus on HCI. Furthermore, the authors emphasize that constant testing and evaluation with users are crucial to achieve a good UX.

However, creating a suitable UX is a challenging task [45]. While their work not directly relates to autonomous systems, the underlying assumptions are as well valid: designing UX implies to specifically define UX goals. In their example, they define two seemingly contradicting UX goals: using the system feels like magic and sense of control over the system. While the former should surprise users, the latter implies that the system should react as expected. Both UX goals can be easily mapped to environments with autonomous systems: while they most certainly feel like magic—some mechanical thing doing something on its own—humans may still need to know why systems do things to not lose the feeling of being in control.

Tonkin et al. [46] describe a method to develop UX in HRI-application based on LeanUX [47] and Agile Science [48]. While the authors’ focus is on creating applications using humanoid robots, their approach can be translated to any application involving autonomous systems. Essentially, they extend classic UX design with two additional steps: personality design and interaction design. Personality design involves giving robots a demeanor depending on the desired UX. In simple terms, designers must define how autonomous machines react to humans. Are they submissive to humans or competitive (which could be used for, e.g., serious gaming in working environments)? Humans tend to assign personalities to humanoid robots [49], thus specifically integrating personality design into the UX design process supports creating an appropriate UX. In any case, the designed personality must match the desired task, context, and working environment. Besides personality, Tonkin et al. [46] suggest to specifically design interaction with the robot. As described before, interaction design is an important part of UX design and thus must be specifically tailored to the task at hand. In the approach of Tonkin et al. [46], UX designers should specifically model the robot’s behavior for each activity, location, and task.

The UX always depends on technical constraints defined by the used hard- and software, essentially creating a natural threshold for possible interaction. While we should of course aim for the best UX, we need to settle for the optimal UX. UX development thus requires a cross-functional team having a task-oriented design and a technology-oriented engineering perspective to determine the best suitable compromise between desirable UX and possible technical capabilities. Designers and engineers, however, tend to develop different solutions [50] and have different views on desired tasks and systems: while designers focus on the developed task and the user’s perspective—a top-down view—engineers tend to take technical capabilities more into account—a bottom-up view [51]. Engineers thus tend to create systems based on existing possibilities (essentially adapting the user to the system), while designers tend to generate alternative ideas (adapting the system to the user’s needs). Both approaches have their advantages and disadvantages, but in an optimal development they cannot exist without each other and both perspectives must be considered to find the optimal solution—as Tonkin et al. [46] suggest, UX development requires cross-functional teams.

Unfortunately, UX cannot be measured, thus iterative development steps and constant testing are necessary [52]. It is furthermore decisive to identify possible sources of critical UX and optimize the affected components. Steinfeld et al. [53] describe a set of metrics for HRI that are still used to evaluate autonomous systems [54] of which three biasing effects might influence the interaction’s effectiveness and performance: communications, robot response, and user. Especially the latter emphasizes that besides technical capabilities a deeper understanding of the human’s role within the system is necessary to create an acceptable UX. Steinfeld et al. [53] point to Scholtz [55], who differentiates between five possible roles for humans: supervisor, operator, mechanic, teammate, and bystander. While each of them might interact with the system, their tasks and requirements differ, thus their respective UX and forms of interaction must be considered.

In our current work, we are focusing on the teammate and bystander roles. Both require humans and autonomous machines to consider each other and maybe—depending on the underlying approach to cooperation—to find consensus for actions. We developed a consensus algorithm for purely autonomous systems that we will subsequently extend to integrate humans.

Autonomous systems make decisions and act independently in their environment. However, this does not exclude them from communication with other systems operating in the same environment. In the most challenging case, this can range up to cooperation in order to achieve overall goals, for example in a warehouse setting [e.g., 56] or to investigate serious accidents [57]. However, the need for communication and negotiation begins as soon as there is potential access to shared and limited resources in the environment. Communication can be solved differently, either centralized [e.g., 58, 59] or distributed [e.g., 60]. The term resource is not limited to concrete objects, but is broadly defined in this context and includes everything whose allocation can lead to a potential conflict, such as locations on which a system moves or wants to store something. Especially in logistics, this results in a multitude of situations in which autonomous systems must encounter each other and find joint solutions. In a work environment in which humans and robots increasingly work together, this also affects humans as soon as they act in areas that are also used by autonomous systems. In the following, we demonstrate an example application to avoid collisions on intersecting paths to show how a system for decentralized consensus finding can look from a technical point of view and discuss the challenges that exist to integrate people into such a system. We also structure the solution space and discuss possible solutions.

Drivers are an important group of workers in logistics and transportation. Automation and AI applications such as cruise control systems [10, 68, 69] are introduced in their work environment to improve efficiency and competitiveness, safety of drivers and other traffic participants as well as working conditions [70‐73]. The outstanding problem is the often insufficient connection between humans and automation resulting in critical issues [74]. Cummings, Bruni [75] emphasize the necessity to promote the HCI as a collaboration especially in highly complex decision-making situations since not all factors can be included in the algorithms. They describe functions that can be assigned to the actors (either human or AI) in the decision-making process differentiated in (i) the moderator to ensure the progress of the decision-making process; (ii) the generator who searches, identifies and develops decisions; and (iii) the decision-maker who finally decides or has a veto.

However, the use and outcome of such automated concepts in logistics depend on the collaboration with humans and their acceptance of such systems [75]. This is true also for the simulated traffic control situation on shopfloors, e.g., by forklifts as these vehicles driven by human workers or by AI as automated systems have the joint task to avoid accidents and share the same traffic space efficiently. Thus, the potential for substantially improving performance often is obstructed by users’ unwillingness to accept and use of technology [74, 76, 77]. Certain models and theories aim to explain the acceptance of new technologies like the UX models mentioned in Section 3. For example, the Technology Acceptance Model (TAM) measures the effects of certain functionalities (design features) of computer-based information systems in organizational contexts on user acceptance with the help of intervening variables. Davis [78] assumes that concrete functionalities, both the perceived usefulness as well as the perceived user-friendliness determine acceptance. Perceived usefulness is defined as “the degree to which an individual believes that using a particular system would enhance his or her job performance”; perceived ease of use is defined as “the degree to which an individual believes that using a particular system would be free of physical or mental effort” [78]. So far, numerous empirical studies have supported the TAM [79, 80]. Furthermore, models theorize the adoption of an innovation by an individual [81] which can be transferred to dealing with new technologies and AI applications in digitalized work settings. The innovation-decision process starts from first knowledge of an innovation to forming an attitude towards it, to a decision to adopt or reject, to an AI or robotics implementation and use, and to confirmation of this decision [82]. Consequently, the process typically consists of the following phases:

The rate of adoption is the relative measure with which members of a social system adopt an innovation operationalized as the number of individuals adopting new technologies in a certain time. Most of the variance in the rate of adoption of an innovation is explained by the attributes introduced above: relative advantage, compatibility, trialability, observability, and complexity [82]. Reactance of people can be expected when they perceive restrictions given that they value freedom and autonomy [83, 84]. Moreover, there might be a gap between the individual’s views, attitudes, intentions, and the actual behavior as the Theory of Reasoned Action (TRA) proposes. This theory is based on the assumption that the behavior of individuals results from certain intentions which depends on the attitude of a person as well as on social influences [85]. The TRA is refined by the Theory of Planned Behavior (TPB) focusing on situations in which individuals do not have complete control over their behavior proposing that the individual’s self-efficacy is decisive [86].

In a temporal perspective, human interaction with AI applications and automation [14, 17] can be characterized by three hurdles or areas of resistance (see Fig. 5). Once an area is overcome, usually acceptance settles in [87, 88].

The three depicted hurdles (“increased resistance areas” or “waves”) are connected to three AI functional areas and represent an increasing, but temporary, level of resistance (y axis) throughout this development in line with an increasing level of personal intrusion (x axis):

Connected to the experimental setting before, these levels or hurdles can be seen throughout a sequential level of personal intrusion, arriving at a completely new situation after the three hurdle areas: The situation of trust with respect to an AI application, where humans are inclined to trustfully and intuitively cooperate with such applications [92]. This is closely related to the Turing test, where passing the test implies that humans are not able to distinguish between human or artificial for their communication with another unknown entity [93]. The stage of AI trust is a special form of passing the Turing test as it can be assumed that a human being may only be able to develop trust towards an AI application if an interaction-based evaluation can judge the collaboration partner to behave like a human being. This is a crucial and business-relevant form of trust between human beings and AI applications in logistics for a successful partnership, taking into account realities in a world of human cooperation as well as HCI. In addition, this is extending the traditional view of technology acceptance in the past [80], where application-specific trust and acknowledgement of human workers was tested and analyzed. Now, long-term work situations like driving and machine handling with possible life-threatening situations are concerned with a required trust towards AI applications. Trust in turn can be seen as a major prerequisite for workers to develop intuition as this can only be trained and grown by trial and error experiences in practice, not in theory. Therefore, the next section is discussing the question and growth circle of intuition and self-efficacy along such experience processes for workers.

Recently, it has been argued that intuition might be able to complement rationality as an effective decision-making approach in organizations [94‐96]. Intuition helps to cope with a wide range of critical decisions and is integral to successfully completing tasks that involve high complexity and short time horizons [97]. However, conceptualizations of intuition lack clarity so far. In general, intuition is differentiated in reliance on gut feelings (creative intuition) or in reliance on past experiences (justified intuition) [98]. Adding to this discussion, Carter et al. [99] consider intuition as a major human driving force in decision-making for logistics management. On the basis of a qualitative content analysis of academic literature and interviews with supply chain management experts and quantitative testing with experts in supplier selection, they conceptualize intuition as a multidimensional construct consisting of the following dimensions: (i) Experience-based intuition implies that persons recognize parallels to past decisions in making the current decision and, thus, refer to knowledge that builds over time, (ii) emotional processing means that affect or (positive and negative) feelings guide decisions and actions, and (iii) automatic processing implies that persons quickly and almost instantly decide without awareness or knowledge of specific decision rules. Based on their analyses, they argue that intuition is rather experience-based in situations with high time pressure, while emotional processing can be found in contexts with both, high time pressure and high information uncertainty. Against this background, obviously dual-processing theories (intuition vs. rationality) seem be too simplistic since different dimensions of intuition occur likewise.

Any approach to intuition has in common that intuitive judgments occur beneath the level of conscious awareness (i.e., they are tacit). The importance of intuition in working behavior has already been emphasized for top executives revealing that intuition was one of the skills used to guide their most important decisions [95, 100]. Khatri, Ng [101] surveyed senior managers of companies representing computer, banking, and utility industries in the United States and highlight that intuitive processes are in fact used in organizational decision making conceptualizing intuition as a form of expertise or distilled experience based on deep knowledge. Also Hogarth [102] argues that intuition is effective when a person is knowledgeable and experienced within a certain domain. The effective use of intuition has even been seen as critical in differentiating the more from less successful workers [97]. Especially for workers in digitalized logistics settings, intuition seems to be decisive in view of the requirement to cope with a wide range of critical decisions, highly complex tasks and short time horizons. This can interestingly underline and explain the results in the experimental setting from Section 3 with the hybrid decision model as the most efficient one.

In search for approaches to enhance intuition, Agor [100] reveals that physical and/or emotional tension and anxiety or fear are conceived as factors that impede the use of intuition, and in turn, positive feelings such as excitement lead to an enhanced sense of confidence in own judgments. Burke, Miller [103] argue that mentors, role models, and supervisors as well as working with diverse groups of people and learning about their decision making styles helps to develop intuition. In this sense, intuition is based on implicit learning [104] and automaticity [105]. Interestingly, these key determinants for using intuition tie in perfectly with the core assumptions of Social Cognitive Theory [SCT; 106]. SCT accounts for the development and maintenance of self-efficacy across a broad range of knowledge, values, and associated behavior patterns [107]. An efficacy expectation is the conviction that a person can successfully execute the behavior required to produce certain outcomes. In this sense, self-efficacy means that coping behavior will be initiated, how much effort will be expended, and how long it will be sustained in the face of obstacles and aversive experiences [108]. Thus, self-efficacy regulates behavior, effort, and persistence over extended periods of time and is concerned with individual beliefs about ones capability to cope with certain and new situations, to use ones abilities to solve problems and tasks, e.g., whether they feel competent in using and learning how to use new technologies. In this respect, it can be expected that individuals with high self-efficacy are more proactive and confident in their decision-making, dealing with digital devices and show greater intuition. Moreover, self-efficacy is similar to the concept of perceived ease of use as defined above [76]. In work contexts, especially following leverages are identified to support the worker’s self-efficacy: (i) Performance accomplishments are based on personal mastery experiences in a sense that repeated success enhances self-efficacy, (ii) vicarious experience means that seeing others performing activities and its consequences generate expectations that persons will improve and intensify their efforts as well, (iii) verbal persuasion is quite common to influence human behavior, i.e., people are led through suggestions, and, finally, (iv) emotional arousal affects perceived self-efficacy in coping with certain situations, i.e., removing dysfunctional fears.

In our context of logistics workers, the concept of self-efficacy is decisive for the use of digital devices and the perceived ease of use. Also, the role of intuition has already been emphasized for successful workers in other fields guiding important decisions in working life. However, more research is required to understand the role of human intuition within an IoT and AI application environment in logistics and supply chain processes. Especially in these fields, we find new technologies and digitalized working contexts requiring the intuition of the workers, affecting their behavior, and the reaction speed. Moreover, the link between intuition and self-efficacy has not been discussed although these two concepts show striking commonalities and provides approaches to further develop human intuition as basis for effective decisions in digitalized work contexts in logistics. As outlined, AI is developing fast within the supply chain and logistics management domain and there is a considerable body of literature concerning intuition and self-efficacy. However, both subjects are brought forward mainly independent of each other. This constitutes a major research gap, as obviously HCI concepts and existing business experience are testimony to the fact that the human factor is largely influencing technology implementation and AI efficiency, especially in logistics as seen in Sections 2 and 3. Thus, research concepts combining these perspectives are needed urgently as mid- to long-term lead times can be expected until results are obtained and transferred into applicable concepts for increasing AI application effectiveness in logistics.

The three different discipline parts and perspectives have established distinctive findings for the use in a HCI efficiency description in production logistics:

As identified with the preceding analysis steps, the question of HCI in production logistics is—among other factors—influenced by the question of acceptance, adoption, intuition and reaction speed of the human as well as the computer actor(s). The following figure is outlining the presumed connection between HCI and decision efficiency as a possible success factor, indicating that there might be an optimum feasible regarding the adjusted learning and reaction rates of humans as well as automated actors (see Fig. 6). On the x-axis, we determine the “reaction mode” of the actors in the system: whereas on the left side, only human actors are reacting to the computer actors (e.g., letting automated transport vehicles pass when a crash seems likely, therefore “taking a step back”, representing a Robot First approach), on the right side only the autonomous computer actors are reacting to the (fixed) actions of human actors, e.g., stopping when human actor transport vehicles are approaching in order to determine right of way and avoid accidents (Human First approach). The y-axis is representing the level of decision efficiency, e.g., measured in the number of passes realized at an intersection—or alternatively by the number of successful passes in relation to accidents occurring.

The interesting area is the development of actions in-between, when both sets of actors increasingly (and v/v decreasingly) react to the actions of the other actor group. It is proposed therein that the actual decision improvement (measured, e.g., in the number of vehicles passes handled at intersections) is strongly depending on the mutual learning/adjustment/reaction rates of human and computer actors. With low and high adjustment rates, it can be presumed that decision efficiency will decline as actors either commence into a “deadlock” (e.g., human and computer vehicles both stopping at intersections, waiting for each other) or in too much mutual activity (e.g., vehicles starting and stopping too often in reaction to each other). Whereas it can be conceptualized that a medium rate of adjustment and reaction on both sides might be able to actual improve decision efficiency as depicted, forming an “optimum lens” of HCI interaction.

It has emerged in the interdisciplinary analysis approach of this paper, that the most single important factor for successful automation concepts is embedded in the question of implementing new technologies and how to integrate human workers into the pathway towards automated systems [74]. It is potentially less crucial how the final “steady state” automated production system is looking like—given that new systems will emerge very fast, too—but more how the way getting there is structured, how framing and motivation are incorporated to optimize HCI [69, 109]. Consequently, the successful AI and robotics implementation in HCI contexts requires [110].

Whereas many ideas and experiences might be transferable from existing approaches to improve the efficiency and success rate of automation concepts and HCI in production logistics settings, also new insights are required as for example how automation and robotics concepts have to be designed in order to allow for a smooth change management in this specific application field of production logistics and to develop the worker’s self-efficacy as outlined in Section 4 [11, 17, 19, 111, 112].

This is highly relevant for management practice as there are many projects for further automation steps at the doorsteps in most companies. Especially the HCI efficiency description as highlighted in Fig. 5 might be an important practical contribution in this context, as especially for operations people like engineers the suggested best solution with a medium adjustment rate or speed for human and AI actors in production logistics settings might be counter-intuitive.