Information presentation method for teleoperated robots to support the multifaceted understanding of fire sites

In recent years, natural disasters that have unexpected scales have frequently occurred. Natural disasters can be primary and secondary disasters; sometimes, secondary disasters are larger than primary disasters. For example, in the Great East Japan Earthquake, when the buildings collapsed, this was a primary disaster, and the large tsunami that caused significant damage was considered to be a secondary disaster [1]. Fire is one of the most common secondary disasters caused by natural disasters. Hence, firefighting activities are important, and continuous improvements are required. Although the overall number of fires is decreasing, the number of special fires and fires and spills in hazardous materials facilities such as petroleum complexes is increasing due to the increasing size, height, and depth of buildings, and the safety of firefighters has become a problem. The use of teleoperated firefighting robots or autonomous firefighting robots is expected to be a solution to this problem. However, it is difficult to predict the environment of a disaster site beforehand, and recovery work at disaster sites consists of activities that are conducted in unknown environments. Therefore, if autonomous robots are applied to disaster sites, they must be able to flexibly determine the appropriate actions in unknown environments. Teleoperated robots, by contrast, are operated by humans; hence, they do not require the same decision-making capabilities as autonomous robots, but they must be used by humans without feeling the workload. Autonomous robots are ideal because they do not need to be operated by humans; therefore, they do not require any operation workload. However, there are many problems that need to be solved before these autonomous robots can be developed [2‐4], and this will take time. Natural disasters can occur at any time, and one of the rational measures to minimize the damage is to apply teleoperated robots with excellent usability to disaster sites. There have been many studies on the usability of teleoperated robots [5‐7]. Casper has shown that the human machine interface (HMI) should be a priority in the design of teleoperated rescue robots [8]. Much of the research on HMI for teleoperated robots is concerned with how to operate the robot and how to present the information that is needed for operation [9‐11]. However, in a disaster recovery, it is important to be able to operate the robot well, recognize the environment, and make decisions on what actions to take to perform the recovery work more effectively. These perceptions and judgments vary depending on the type of work, and they are considered to be cognitive skills that are specific to that work. In addition, as the overall number of fires is decreasing, young firefighters' (novices’) lack of experience in firefighting has become a problem. Therefore, it is necessary for the HMI of teleoperated firefighting robots to include information presentation that supports firefighters' environmental cognition and decision making in firefighting.

Firefighting activities can be divided into two main categories: reconnaissance activities and attenuation (water discharge) activities. Reconnaissance is the collection of information, and Attenuation is the analysis of the collected information and the decisions based on that analysis. For each of the two types of activities using teleoperated robots, Tamura et al. clarified that experienced firefighters (experts) developed their understanding of the fire sites from multifaceted perspectives [12, 13]. The multifaceted perspective of experts is based on the basics of firefighting tactics, and even novices are aware of these. According to the data information knowledge wisdom (DIKW) model [14], novices are familiar with items such as knowledge, but they have not yet sublimated them to wisdom like the experts. As a result, novices can only recognize certain aspects of the fire sites. The gap between the novices and experts is usually bridged by years of firefighting experience. In addition, both activities involve considerable cognitive load because they deal with a considerable amount of information. However, we believe that if we can classify the perspectives that are difficult for novices to recognize by type and clarify their characteristics, an HMI can be designed that presents them in a way that requires less cognitive load. In this way, even novices will be able to perform firefighting activities similar to those of experts, which will contribute to the prevention of fire damage.

The fire site is a dynamic environment, and firefighters are required to act according to the situation at the time. Therefore, this study designed an information presentation method that is based on the contextual control model (COCOM) [15] instead of simply presenting information. The COCOM is a theory that states that human behavior is not selected according to a predetermined set of rules but is influenced by the context, and it is carried out by a control mode according to that context. The information in this study indicates the messages that support the recognition of the items required for firefighting activities (evaluation items in the chapter, “Approach”). The effectiveness of the information presentation method was confirmed through evaluation by 12 firefighters.

The remainder of this paper is organized as follows. The “Problem” and “Approach” chapters present the proposed method, the “Experiment” describes the experiments, the “Results” presents the results, the “Discussion” provides a discussion of the study, and the “Conclusion” presents the conclusions.

Although the situations at fire sites change dynamically, if there are leaks or incorrect information in the collected information, there is a possibility that the appropriate firefighting activities cannot be carried out. The control of a nuclear power plant or the operation of an aircraft are examples of such situations outside of firefighting. However, firefighting is not automated, whereas nuclear power plant control and aircraft cockpits are. Therefore, in these systems, the discussion has focused on how to present the necessary information and not so much on the amount of information [16, 17]. For information management in non-automated activities such as firefighting, not only the form but also the quantity of information to be presented is considered to be important [18]. For example, if there is a large amount of information that is displayed at once, it may be necessary to divide the information by using certain criteria.

It is important to design interfaces that are based on realistic problem recognition to verify whether or not to display the information at once. Kitamura stated that it is necessary to clarify the system user's requirements so that they do not deviate from the real problems, to embody a normatively rational and purposeful design, and the information provision should be consistent with the user’s control modes in COCOM [19]. The control modes in COCOM are divided into four control modes: scrambled control mode, opportunistic control mode, tactical control mode, and the strategic control mode. In COCOM, these control modes are divided based on the orderliness and the regularity of the performance. The scrambled control mode is a control mode in which the choice of action is random. The opportunistic control mode is a control mode in which the next action is chosen based on the current situation. The tactical control mode is a control mode in which the next action is chosen according to some rules in advance. Finally, the strategic control mode is a control mode in which the overall situation is taken into account. The situation in the strategic control mode indicates that the information needed to achieve the desired action is clear, but this information is the information needed to execute reconnaissance and attenuation activities, which is different from the evaluation items in the chapter, “Approach”. Previous studies on the COCOM include controlling the simulated energy distribution system and airplane scheduling [20, 21]. Both studies confirmed that these activities can be divided into control modes, and their transitions are also defined by the COCOM. However, as mentioned above, these activities are different from firefighting activities in terms of automation. Therefore, the effectiveness of the COCOM-based information presentation in firefighting is unclear. Especially, it is important to classify the information that is necessary for firefighting activities into the COCOM control modes, and how to present them to firefighters.

In this study, two types of firefighting activities are targeted: reconnaissance and attenuation activities. This is because these activities require different information that need to be managed, and both activities are performed during firefighting.

As mentioned earlier, the closer the control mode in the COCOM is to the strategic control mode, the more the orderliness and the regularity of the performance is, the lower the cognitive load becomes, which means that the firefighters can operate with more margin. In order to respond appropriately to changing circumstances in dynamically changing situations, it is ideal to always be able to operate in a strategic control mode with a margin of cognitive capacity. Therefore, evaluation items for reconnaissance and attenuation activities are classified into four control modes in COCOM based on its cognitive difficulty and importance (weight), and the characteristics of items classified into the strategic control mode are clarified. We believe that it is possible to construct an interface that can evaluation items classified as other than the strategic control mode in the same way as the strategic control mode by presenting information with the added function of evaluation items classified as strategic control mode. The reason why the classification into each control mode is based on the cognitive difficulty and importance is that both the cognitive difficulty and the importance have effects that changes the degree of understanding of the situation depending on their degree. For example, when the importance is large and the cognitive difficulty is low, firefighters will act based on models and theories because failure is not allowed. On the other hand, when the importance is small and the cognitive difficulty is high, the firefighters will act based on habit or association, because a certain amount of uncertainty is allowed. We believe that the relationship in firefighting activities between each control mode, cognitive difficulty, and importance is shown in Table 1. The reason why scrambled control mode is excluded here is that firefighters, even novices, are professionals in firefighting and will never fall into scrambled control mode, which is a completely random behavior (firefighters train hard every day to avoid falling into scrambled control mode).

In this study, the degree of cognitive difficulty was defined on two axes: whether the item was visible or not, and whether it had a clear characteristic or not. The more visible the item was and the clearer the characteristic was, the lower the cognitive difficulty of the item.

The importance of each evaluation item was conducted by the Analytic Hierarchy Process (AHP), which is an analytical method that allows weighting of multiple factors by quantifying their evaluations through pairwise comparisons [22]. The AHP is a method used in decision making problems and has been used in many papers because it can handle the qualitative evaluation criteria of evaluators quantitatively [23]. In this study, the AHP was conducted based on the evaluation item ratings by 5 experts different from the experiment participants, and the mean and standard deviation of the weights of each evaluation item were calculated. The results of the weight discrimination of the evaluation items in the reconnaissance activity are shown in Fig. 1, and it was found that the weight of the evaluation items in the reconnaissance activity increased as the information became more visible, such as the location of the fire and the water utilization. On the other hand, Fig. 2 shows the results of the weight discrimination of the evaluation items in the attenuation activity. Although the order of the weights could not be determined clearly due to the large standard deviation, it was found that the weights tended to be higher for visible information.

Based on the above, for both reconnaissance and attenuation activities, the more visible and the more distinct the characteristics, the lower the cognitive difficulty and the larger the importance, so the correspondence between the control mode and the characteristics of the evaluation items is as shown in Table 2. From Table 2, The evaluation items for the reconnaissance and attenuation activities are classified as shown in Tables 3 and 4.

Since the goal of firefighting is to extinguish the fire as soon as possible, it is important to take immediate action without procrastination. Therefore, it is considered that firefighters generally act in the order of strategic control mode, tactical control mode, and opportunistic control mode. In this study, we hypothesize that presenting information to the user in this order of control modes will reduce the cognitive load. To verify this hypothesis, this study examines two axes of effective information presentation in firefighting activities: with and without the control mode in COCOM, and whether the information is presented at once or not. Of these four types, the type that does not take into account the control mode in the COCOM and presents all the information at once will have the worst results. This has been demonstrated from the cognitive load theory that there is a limit to human working memory [24]; hence, this experiment will be omitted.

According to this classification, messages that are related to the evaluation items are created as work support messages, which are presented on the operation screen. Based on the characteristics of the evaluation items categorized into strategic control mode, more detailed work support messages will be created as the control mode becomes tactical control mode and opportunistic control mode.

The specific method for creating the message is as follows.

Strategic control mode: Display the name of the evaluation item.

Tactical control mode: Display the name of the item and the characteristics of the item.

Opportunistic control mode: Display the name of the evaluation item, the characteristics of the evaluated item, and its specific checking example.

The information for this experiment is described as follows.

The purpose of this study was to design an information presentation method with low cognitive load that would enable novices to conduct firefighting activities from multiple perspectives in dynamically changing fire situations. In this study, two axes of evaluation were prepared: with or without the control theory of COCOM, and whether the information was presented at once or in separate views. In this chapter, we will discuss the results of the experiment through a comparison of three information presentation methods that have been created based on these two axes. The difference between reconnaissance and attenuation is that reconnaissance is an information-gathering activity, and attenuation is a decision-making activity based on the gathered information. Since most of the participants in this experiment have plenty of experience in firefighting or defense, reconnaissance activities in residential areas were familiar to them, and they have a clear idea of what to check. In this experiment, they can be considered experts in reconnaissance activities. Since the difference in the mental workload between types I and III was not significant, the idea of what to check during reconnaissance for the experts is considered to be consistent with the control mode of the COCOM. The difference in the mental workload between types I and II indicates that the mental workload increases when the order of the messages is not in accordance with the control mode, even if the number of messages that are displayed is small.

By contrast, because the attenuation activity was an activity that not all of the participants had experienced before, differences in the message presentation type were observed. The judgment work consists of interpreting the gathered information, and plenty of experience is required for this task. Hence, in this experiment, the participants with no experience in oil refining facilities could be considered as novices in attenuation activities. The determination of the water discharge position during the attenuation activities is more difficult than during the reconnaissance activities. This is because there are trade-off problems between the evaluation items that are used for this judgment, and it is necessary to decide the order of priority. In this experiment, there were trade-off problems between the evaluation items that are listed in Table 18.

Therefore, the participants need to prioritize the evaluation items that have conflicting advantages and disadvantages. This trade-off problem is particularly difficult for novices because, unlike experts, they have no experience to refer to. This may be the reason why the objective evaluation of type III is lower than that of type I. As a result, the cognitive load increased and the attention to the other evaluation items that was presented was neglected. This may have resulted in the objective evaluation of type III, which has more information, since it is lower than the objective evaluation of type I. Since it has been known in cognitive science for a long time that the effective field of the view shrinks as the cognitive load increases [30, 31], it is believed that the same thing is happening. There was no significant difference between types II and III in terms of the attenuation check score. This indicates that there is no difference between giving information all at once and giving information randomly to the novice in that activity.

From the above, it is considered that the information presentation method that enables even novices to perform firefighting activities like experts is a method that can present information sequentially based on the control mode of the COCOM as in type I. However, a limitation of this study is that this result is obtained using a single robot, and in the case of multiple robots, the results may change due to the division of roles. In addition, the number of cameras mounted on the robot was set to one, but it is thought that the mounting of multiple cameras or presentation of information about the environment, such as current temperature and carbon monoxide concentration are effective means to improve the environmental cognition. In such a case, it is necessary to study how to arrange the images of multiple cameras or information about the environment on the operation screen in combination with the results of previous studies [32, 33]. These are issues for further study.

In this study, an information presentation method was developed that is based on the control modes in the COCOM to ensure that even novices can have a multifaceted perspective in firefighting, which is the key difference between experts and novices. The effectiveness of the information presentation method was verified by evaluating firefighters. This method was evaluated with 12 firefighters that performed two different types of firefighting activities: reconnaissance activities and attenuation activities.

The experimental results showed that types I and were the easiest information presentation methods to use in the reconnaissance activity, and the type of methods to use depends on the preference. By contrast, in the attenuation activity, which is more difficult for the participants, the order of the checks in type I was the most similar to the control mode, and the score was higher. These results verified the validity of the hypothesis that firefighters generally act in the order of the strategic, tactics, and opportunistic control modes when estimating the situation in firefighting. This information presentation method will enable firefighters to support their environmental cognition and decision making in firefighting activities, and young firefighters to perform the same firefighting activities as experienced firefighters, thus contributing to the prevention of fire damage.

Estimating the situation within a limited time frame are necessary activities not only in fires but also in many other disaster scenes. Therefore, it is believed that the information presentation method in this study will be useful for teleoperated robots that are operating in disaster sites. In that case, it is necessary to compare the work of novices and experts, extract the characteristics of experts' work, and then apply the information presentation method in this study. In addition, this study organized the details of information and the order of its presentation that are necessary for estimate work of the situation in firefighting. Therefore, the results of this study can be useful in terms of training firefighters. The results of this study were obtained using a single robot and camera, and questions; the results may differ for multiple robots and cameras, and additional information about the environment. Therefore, the information presentation method for multiple robots and cameras, and additional information about the environment will be a future study topic.