Skip to main content

Information presentation method for teleoperated robots to support the multifaceted understanding of fire sites

Abstract

Although the overall number of fires is decreasing, the number of special fires as buildings become larger, taller, and deeper and fires and spills in hazardous materials facilities such as petroleum complexes is on the rise and ensuring the safety of firefighters has become a problem. The use of teleoperated firefighting robots is expected to be a solution to this problem. It is important to be able to operate the robot well, recognize the environment, and make decisions on what actions to take to perform the firefighting more effectively. This is closely related to the design of human machine interface (HMI) for teleoperated robots, but much of the research on HMI for teleoperated robots is concerned with how to operate the robot and how to present the information that is needed for operation. In addition, as the overall number of fires is decreasing, young firefighters' (novices’) lack of experience in firefighting has become a problem. Therefore, teleoperated firefighting robots are required to have the HMI that enables supports to recognize the environment and make decisions in firefighting. This study presents an information presentation method that is based on the control mode in the contextual control model (COCOM) to ensure that novices can have multifaceted perspectives during reconnaissance and attenuation like experienced firefighters (experts). The effectiveness of this method is confirmed by 12 firefighters. As a result, this study confirms that the information presentation is designed considering the control mode in the COCOM, and it improves the score from multiple perspectives. The findings of this study will contribute to the future development of teleoperated firefighting robots.

Introduction

In recent years, natural disasters that have unexpected scales have frequently occurred. Natural disasters can be primary and secondary disasters; sometimes, secondary disasters are larger than primary disasters. For example, in the Great East Japan Earthquake, when the buildings collapsed, this was a primary disaster, and the large tsunami that caused significant damage was considered to be a secondary disaster [1]. Fire is one of the most common secondary disasters caused by natural disasters. Hence, firefighting activities are important, and continuous improvements are required. Although the overall number of fires is decreasing, the number of special fires and fires and spills in hazardous materials facilities such as petroleum complexes is increasing due to the increasing size, height, and depth of buildings, and the safety of firefighters has become a problem. The use of teleoperated firefighting robots or autonomous firefighting robots is expected to be a solution to this problem. However, it is difficult to predict the environment of a disaster site beforehand, and recovery work at disaster sites consists of activities that are conducted in unknown environments. Therefore, if autonomous robots are applied to disaster sites, they must be able to flexibly determine the appropriate actions in unknown environments. Teleoperated robots, by contrast, are operated by humans; hence, they do not require the same decision-making capabilities as autonomous robots, but they must be used by humans without feeling the workload. Autonomous robots are ideal because they do not need to be operated by humans; therefore, they do not require any operation workload. However, there are many problems that need to be solved before these autonomous robots can be developed [2,3,4], and this will take time. Natural disasters can occur at any time, and one of the rational measures to minimize the damage is to apply teleoperated robots with excellent usability to disaster sites. There have been many studies on the usability of teleoperated robots [5,6,7]. Casper has shown that the human machine interface (HMI) should be a priority in the design of teleoperated rescue robots [8]. Much of the research on HMI for teleoperated robots is concerned with how to operate the robot and how to present the information that is needed for operation [9,10,11]. However, in a disaster recovery, it is important to be able to operate the robot well, recognize the environment, and make decisions on what actions to take to perform the recovery work more effectively. These perceptions and judgments vary depending on the type of work, and they are considered to be cognitive skills that are specific to that work. In addition, as the overall number of fires is decreasing, young firefighters' (novices’) lack of experience in firefighting has become a problem. Therefore, it is necessary for the HMI of teleoperated firefighting robots to include information presentation that supports firefighters' environmental cognition and decision making in firefighting.

Firefighting activities can be divided into two main categories: reconnaissance activities and attenuation (water discharge) activities. Reconnaissance is the collection of information, and Attenuation is the analysis of the collected information and the decisions based on that analysis. For each of the two types of activities using teleoperated robots, Tamura et al. clarified that experienced firefighters (experts) developed their understanding of the fire sites from multifaceted perspectives [12, 13]. The multifaceted perspective of experts is based on the basics of firefighting tactics, and even novices are aware of these. According to the data information knowledge wisdom (DIKW) model [14], novices are familiar with items such as knowledge, but they have not yet sublimated them to wisdom like the experts. As a result, novices can only recognize certain aspects of the fire sites. The gap between the novices and experts is usually bridged by years of firefighting experience. In addition, both activities involve considerable cognitive load because they deal with a considerable amount of information. However, we believe that if we can classify the perspectives that are difficult for novices to recognize by type and clarify their characteristics, an HMI can be designed that presents them in a way that requires less cognitive load. In this way, even novices will be able to perform firefighting activities similar to those of experts, which will contribute to the prevention of fire damage.

The fire site is a dynamic environment, and firefighters are required to act according to the situation at the time. Therefore, this study designed an information presentation method that is based on the contextual control model (COCOM) [15] instead of simply presenting information. The COCOM is a theory that states that human behavior is not selected according to a predetermined set of rules but is influenced by the context, and it is carried out by a control mode according to that context. The information in this study indicates the messages that support the recognition of the items required for firefighting activities (evaluation items in the chapter, “Approach”). The effectiveness of the information presentation method was confirmed through evaluation by 12 firefighters.

The remainder of this paper is organized as follows. The “Problem” and “Approach” chapters present the proposed method, the “Experiment” describes the experiments, the “Results” presents the results, the “Discussion” provides a discussion of the study, and the “Conclusion” presents the conclusions.

Problem

Although the situations at fire sites change dynamically, if there are leaks or incorrect information in the collected information, there is a possibility that the appropriate firefighting activities cannot be carried out. The control of a nuclear power plant or the operation of an aircraft are examples of such situations outside of firefighting. However, firefighting is not automated, whereas nuclear power plant control and aircraft cockpits are. Therefore, in these systems, the discussion has focused on how to present the necessary information and not so much on the amount of information [16, 17]. For information management in non-automated activities such as firefighting, not only the form but also the quantity of information to be presented is considered to be important [18]. For example, if there is a large amount of information that is displayed at once, it may be necessary to divide the information by using certain criteria.

It is important to design interfaces that are based on realistic problem recognition to verify whether or not to display the information at once. Kitamura stated that it is necessary to clarify the system user's requirements so that they do not deviate from the real problems, to embody a normatively rational and purposeful design, and the information provision should be consistent with the user’s control modes in COCOM [19]. The control modes in COCOM are divided into four control modes: scrambled control mode, opportunistic control mode, tactical control mode, and the strategic control mode. In COCOM, these control modes are divided based on the orderliness and the regularity of the performance. The scrambled control mode is a control mode in which the choice of action is random. The opportunistic control mode is a control mode in which the next action is chosen based on the current situation. The tactical control mode is a control mode in which the next action is chosen according to some rules in advance. Finally, the strategic control mode is a control mode in which the overall situation is taken into account. The situation in the strategic control mode indicates that the information needed to achieve the desired action is clear, but this information is the information needed to execute reconnaissance and attenuation activities, which is different from the evaluation items in the chapter, “Approach”. Previous studies on the COCOM include controlling the simulated energy distribution system and airplane scheduling [20, 21]. Both studies confirmed that these activities can be divided into control modes, and their transitions are also defined by the COCOM. However, as mentioned above, these activities are different from firefighting activities in terms of automation. Therefore, the effectiveness of the COCOM-based information presentation in firefighting is unclear. Especially, it is important to classify the information that is necessary for firefighting activities into the COCOM control modes, and how to present them to firefighters.

Approach

In this study, two types of firefighting activities are targeted: reconnaissance and attenuation activities. This is because these activities require different information that need to be managed, and both activities are performed during firefighting.

As mentioned earlier, the closer the control mode in the COCOM is to the strategic control mode, the more the orderliness and the regularity of the performance is, the lower the cognitive load becomes, which means that the firefighters can operate with more margin. In order to respond appropriately to changing circumstances in dynamically changing situations, it is ideal to always be able to operate in a strategic control mode with a margin of cognitive capacity. Therefore, evaluation items for reconnaissance and attenuation activities are classified into four control modes in COCOM based on its cognitive difficulty and importance (weight), and the characteristics of items classified into the strategic control mode are clarified. We believe that it is possible to construct an interface that can evaluation items classified as other than the strategic control mode in the same way as the strategic control mode by presenting information with the added function of evaluation items classified as strategic control mode. The reason why the classification into each control mode is based on the cognitive difficulty and importance is that both the cognitive difficulty and the importance have effects that changes the degree of understanding of the situation depending on their degree. For example, when the importance is large and the cognitive difficulty is low, firefighters will act based on models and theories because failure is not allowed. On the other hand, when the importance is small and the cognitive difficulty is high, the firefighters will act based on habit or association, because a certain amount of uncertainty is allowed. We believe that the relationship in firefighting activities between each control mode, cognitive difficulty, and importance is shown in Table 1. The reason why scrambled control mode is excluded here is that firefighters, even novices, are professionals in firefighting and will never fall into scrambled control mode, which is a completely random behavior (firefighters train hard every day to avoid falling into scrambled control mode).

Table 1 The relationship in firefighting activities between each control mode, cognitive difficulty, and importance

In this study, the degree of cognitive difficulty was defined on two axes: whether the item was visible or not, and whether it had a clear characteristic or not. The more visible the item was and the clearer the characteristic was, the lower the cognitive difficulty of the item.

The importance of each evaluation item was conducted by the Analytic Hierarchy Process (AHP), which is an analytical method that allows weighting of multiple factors by quantifying their evaluations through pairwise comparisons [22]. The AHP is a method used in decision making problems and has been used in many papers because it can handle the qualitative evaluation criteria of evaluators quantitatively [23]. In this study, the AHP was conducted based on the evaluation item ratings by 5 experts different from the experiment participants, and the mean and standard deviation of the weights of each evaluation item were calculated. The results of the weight discrimination of the evaluation items in the reconnaissance activity are shown in Fig. 1, and it was found that the weight of the evaluation items in the reconnaissance activity increased as the information became more visible, such as the location of the fire and the water utilization. On the other hand, Fig. 2 shows the results of the weight discrimination of the evaluation items in the attenuation activity. Although the order of the weights could not be determined clearly due to the large standard deviation, it was found that the weights tended to be higher for visible information.

Fig.1
figure 1

The weight discrimination of the evaluation items in the reconnaissance activity

Fig.2
figure 2

The weight discrimination of the evaluation items in the attenuation activity

Based on the above, for both reconnaissance and attenuation activities, the more visible and the more distinct the characteristics, the lower the cognitive difficulty and the larger the importance, so the correspondence between the control mode and the characteristics of the evaluation items is as shown in Table 2. From Table 2, The evaluation items for the reconnaissance and attenuation activities are classified as shown in Tables 3 and 4.

Table 2 The correspondence between control mode and characteristic of the evaluation items
Table 3 Classification of the evaluation items for a reconnaissance activity
Table 4 Classification of the evaluation items for an attenuation activity

Since the goal of firefighting is to extinguish the fire as soon as possible, it is important to take immediate action without procrastination. Therefore, it is considered that firefighters generally act in the order of strategic control mode, tactical control mode, and opportunistic control mode. In this study, we hypothesize that presenting information to the user in this order of control modes will reduce the cognitive load. To verify this hypothesis, this study examines two axes of effective information presentation in firefighting activities: with and without the control mode in COCOM, and whether the information is presented at once or not. Of these four types, the type that does not take into account the control mode in the COCOM and presents all the information at once will have the worst results. This has been demonstrated from the cognitive load theory that there is a limit to human working memory [24]; hence, this experiment will be omitted.

According to this classification, messages that are related to the evaluation items are created as work support messages, which are presented on the operation screen. Based on the characteristics of the evaluation items categorized into strategic control mode, more detailed work support messages will be created as the control mode becomes tactical control mode and opportunistic control mode.

The specific method for creating the message is as follows.

Strategic control mode: Display the name of the evaluation item.

Tactical control mode: Display the name of the item and the characteristics of the item.

Opportunistic control mode: Display the name of the evaluation item, the characteristics of the evaluated item, and its specific checking example.

Experiment

The information for this experiment is described as follows.

Participants

The test was conducted with 12 firefighters that were dispatched to the National Research Institute of Fire and Disaster. Their backgrounds and headquarters were scattered. All participants were informed about the handling of data and personal information obtained in the study, and consent was obtained from all participants. Table 5 shows the participants' experience in firefighting, defense, prevention, and oil refining facilities. Defense is an activity in which the fire department directly exercises its firefighting power. Prevention is an activity based on the Fire Service Act that supports and ensures proactive efforts in private enterprises.

Table 5 Participants' experience in firefighting, defense, prevention, and oil refining facilities

Types of information presentation

  1. (1)

    Type I: Display the checklist information in an order that takes the COCOM into account. The information is displayed in the order of the strategic control mode, tactical control mode, and the opportunistic control mode. The participants could switch between the control modes at their own discretion. When a participant switches a message, he/she cannot see the previous message. This type is the proposed method in this study.

  2. (2)

    Type II: Display the checklist information in an order without considering the COCOM.Regardless of the control mode, the evaluation items were displayed in a discrete order. The participants could switch between control modes at their own discretion. When a participant switches a message, he/she cannot see the previous message. This type improves the attention to the checklist information due to its prominence from the sense of discomfort [25].

  3. (3)

    Type III: Display all the checklist information at once. The information is displayed in the order of the strategic control mode, tactical control mode, and opportunistic control mode. This type reduces the possibility of missing the checklist information due to increased multitasking [26].

In this experiment, the participants are multitasking to view the checklist information while operating the robot. In Type I, there is a possibility of missing or forgetting checklist information because of the switching of checklist information. Type II and Type III scores may be better than Type I scores because Type II and Type III are less prone to missing or forgetting checklist information.

To eliminate the order effect, this study varied the order of the types for each participant, such as type I → type II → type III for participant A and type I → type III → type II for participant B.

Targeted firefighting activities and their objectives.

Reconnaissance activity: Gather information on the fire site.

Attenuation activity: Determine the location of the water discharge.

Configuration

  1. (1)

    Reconnaissance activity

    Disaster situation: An epicentral earthquake has caused fires to break out simultaneously in many places in the prefecture. Citizens reported that there were fires in a residential area.

    Purpose of the reconnaissance: To begin the water discharge as soon as the fire engine arrives in this area, the firefighters reconnoiter with the teleoperated robot and decide on the positions for the water discharge.

    Activity timing: For the sake of simplicity, the participants started reconnaissance after the evacuation of the citizens was completed.

    Weather conditions: Table 6 summarizes the weather conditions that were considered in this experiment. It was assumed that the weather conditions resemble a summer day in the capital region in Japan. This is similar to the seasons in which this experiment was performed.

  2. (2)

    Attenuation activity

    Disaster situation: An epicentral earthquake has caused fires to break out simultaneously in many places in the prefecture. The valve of the oil refinery was loosened by an earthquake, and hydrogen gas leaked.

    Purpose of the attenuation: To allow office staff and fire brigade members to enter the leak site and perform the work to stop the leak.

    Activity timing: To make the experiment easier, the participants started the attenuation of hydrogen gas after the citizens or office staff evacuated. Only hydrogen gas leaked, no fire had occurred, and all the facilities were operating.

    Leak area: The participants were informed that the leaking area of the hydrogen gas was inside the first floor of the central facility, which is preliminary information. The participants were also notified about the percentage lower explosive limit (% LEL) data of the leak area. The % LEL is the degree to which the explosive limit is reached, and when this percentage reaches 100%, it indicates an environment in which an explosion can occur. Hydrogen gas concentrations can be obtained from gas detection sensors that are installed at various locations in the facility.

    Weather conditions: The weather conditions in this experiment are listed in Table 7. It was assumed that the weather conditions resemble a typical summer day in the capital region in Japan, which is similar to the season when the experiment was performed.

Table 6 Weather conditions
Table 7 Weather conditions

Environment

The experiment was conducted by using computer simulations that were constructed with ROS, Gazebo, and Rviz. The reconnaissance and attenuation were conducted in the environments that are shown in Figs. 3 and 4, respectively. Figure 3 shows a dense area of four apartment buildings. Figure 4 shows the area of the oil refining facilities in a petroleum complex.

Fig. 3
figure 3

Experimental field of reconnaissance

Fig. 4
figure 4

Experimental field of attenuation

For the fire, this study used the image of the flag that is shown in Fig. 5, which is used during fire training. The yellow smoke flag used in this experiment indicates that it is not simply burning, but a growing flame. Since the firefighters use this flag in their daily training, the difference in cognition of flame is expected to be smaller than the computer graphics flame. The emergency entrance in the Building Standard Act is represented by an equilateral triangle, as in the real world, and the water source is represented by a fire hydrant in the simulator.

Fig. 5
figure 5

Equipment in the experimental field

Robot

The robot that was used in this simulation was developed based on Husky [27] and it was equipped with a pan-tilt camera (white part) to allow a visual confirmation of the surrounding environment (Fig. 6). In this experiment, a single camera system was used to eliminate the effect of multiple camera windows layout In addition, a water cannon with the following performance was set to be in the same location as the pan-tilt camera. Movement of the robot after arrival at the water discharge position was not allowed. The robot was assumed to be explosion-proof.

  • ・Water discharge amount: 4,000 L/min.

  • ・Type of water discharge.

Fig. 6
figure 6

Appearance of Husky

Straight (width: approximately 2.5 m, range: approximately 75 m, height: approximately 30 m)

Wide-range (width: approximately 7 m, range: approximately 30 m, height: approximately 15 m)

The robot can be controlled by a gamepad or mouse, and the roles of each are as follows:

  • ・Moving the robot: a gamepad.

  • ・Control of the camera: a mouse.

To prevent the participants from malfunctioning, not all the operations can be performed with the gamepad, but they are deliberately separated. The messages to be displayed were switched by the signal of the participants. Specifically, the participants were asked to switch messages when the current information became unnecessary. However, once the participants switch messages, they cannot return to the previous message.

The operation screen of the robot is illustrated in Fig. 7. The operation screen has two windows: one for the camera image and the other for the work support messages.

Fig. 7
figure 7

Operation screen of the robot

As an example, the switching of work support messages in the case of type I and the reconnaissance activity is shown in Fig. 8. In type I, these are switched in the COCOM order, and in type II, they are switched in a random order. In type III, all of this information is displayed at once. Figure 9 shows the operating stations that are used in this experiment.

Fig. 8
figure 8

Switching the work support messages for type I and the reconnaissance activity

Fig. 9
figure 9

Operator station for this experiment

Presentation message

Table 8 shows the messages that are presented during the reconnaissance activities, and Table 9 shows the messages that are presented during the attenuation activities. These messages were presented on the display, as shown in Fig. 8.

Table 8 Presentation messages during the reconnaissance activity
Table 9 Presentation messages during the attenuation activity

Procedure

  1. (1)

    Experimental explanation and simulator operation

  2. (2)

    Have the participants reconnoiter the disaster site (dense area of an apartment building) by using the robot.

  3. (3)

    Have the participants fill out the questionnaire after completing one trial.

  4. (4)

    Have the participants attenuate the hydrogen gas at another disaster site (the area of the oil refining facilities). This study did not cover the water discharge, but only the determination of the water discharge position.

  5. (5)

    Interviews were conducted with each participant based on the results of the questionnaire.

In the questionnaire, the following items were confirmed for the reconnaissance and attenuation.

  • What are the items you checked during the reconnaissance/attenuation activities?

  • What is the order in which you checked the items that are listed in the previous question?

  • Which items would you not have checked during the reconnaissance/attenuation if you had not received the work support message?

  • Which item is the most difficult to check?

  • Which of the three work support messages is easier to use?

Evaluation

This study conducted two types of evaluations for each information presentation method: a subjective evaluation and an objective evaluation. The subjective evaluation was based on the user's ranking of easy-to-use methods and the mental workload measurement by NASA-TLX [28]. Here, the mental workload refers to the workload that was subjectively felt by the participants, which is different from the cognitive load (the amount of working memory resources that are used). The objective evaluation was based on the number of evaluation items and the Levenshtein distance. The basis sequence for calculating the Levenshtein distances is the strategic, the tactical, and the opportunistic control mode. However, since this sequence is based on the hypothesis in “Approach” chapter, the validity of the hypothesis is checked by the number of evaluation items, and then the evaluation by the Levenshtein distance is conducted.

Results

Scores of the types that were easy to use

Table 10 shows the scores of the types that were easy to use in a ranked order (1st: 5 points, 2nd: 3 points, 3rd: 1 point).

Table 10 Scores of the types that were easy to use in a ranked order

The scores were tested by applying Wilcoxon’s signed-rank sum test because the comparison was correspondent and without normality. Since multiple comparisons were performed between the three groups, the p-values were corrected by using Shaffer's method.

These p-values are summarized in Table 11. When the p-values are less than 0.05, the values of effect size are also shown. If the p-value is less than 0.05 and the effect size is higher than 0.5, a substantial difference is confirmed [29]. Table 11 showed that there were significant differences between types I and II, and types II and III for both reconnaissance and attenuation activities.

Table 11 P-values based on a comparison for each type

Figures 10 and 11 illustrate the means of the scores for each type during the reconnaissance and attenuation activities. The error bars indicate the standard error of the mean (SEM) for each type. * represents p < 0.05.

Fig. 10
figure 10

Means of the score for each type during the reconnaissance activity (*p < 0.05)

Fig. 11
figure 11

Means of the score for each type during the attenuation activity (*p < 0.05)

NASA-TLX scores of the types

Table 12 summarizes the NASA-TLX scores for each type of participant during the reconnaissance and attenuation activities.

Table 12 Scores of NASA-TLX for each type

The scores were tested with a t-test because the comparison was correspondent and with normality. Since this is a multiple comparison between the three groups, the p-values were corrected by using Shaffer's method. The p-values are summarized in Table 13. Table 13 showed that it was a significant difference during the reconnaissance between types I and II.

Table 13 p-values based on a comparison for each type

Figures 12 and 13 display the means of the scores for each type during the reconnaissance and attenuation activities. The error bars indicate the SEM for each type. * represents p < 0.05.

Fig. 12
figure 12

Means of the score for each type during the reconnaissance activity (*p < 0.05)

Fig. 13
figure 13

Means of the score for each type during the attenuation activity

Total scores of the types for evaluation items

Table 14 summarizes the total scores of each type for the items that are checked during the reconnaissance and attenuation activities. The score of each item was two if there was an intense answer, one if there was a non-conscious answer, and zero if there was no answer. In this study, “the conscious answer” means that the information is used consciously, “the non-conscious answer” means that the information is used non-consciously, and “no answer” means that the information is not used.

Table 14 Total scores of each type for the items that are checked during the reconnaissance and attenuation activities

The scores were tested by applying Wilcoxon’s signed-rank sum test because the comparison was correspondent and without normality. Since this is a multiple comparison between the three groups, the p-values were corrected by using Shaffer's method. The p-values are summarized in Table 15. Table 15 showed that there were significant differences during the attenuation between types I and II, and types I and III.

Table 15 p-values based on a comparison for each type

Figures 14 and 15 show the means of the scores for each type during the reconnaissance and attenuation activities. The error bars indicate the SEM for each type. * represents p < 0.05.

Fig. 14
figure 14

Means of the score for each type during the reconnaissance activity

Fig. 15
figure 15

Means of the score for each type during the attenuation activity (*p < 0.05)

Levenshtein distance

Since the validity of the hypothesis in “Approach” chapter was checked by the number of evaluation items (Figs. 14 and 15), the evaluation by the Levenshtein distance was conducted. Table 16 shows that the order of the evaluation items during the reconnaissance and attenuation activities was evaluated by the Levenshtein distance to determine if the order of the evaluation items followed the grouping order based on the COCOM.

Table 16 Order of the evaluation items during the reconnaissance and attenuation activities was evaluated by the Levenshtein distance

In this study, the Levenshtein distance was assigned one point if it replaced a character (0.5 points when replacing a character with the same control mode) and three points to insert or delete a character.

The scores were tested by applying Wilcoxon’s signed-rank sum test because the comparison was correspondent and without normality. Since this is a multiple comparison between the three groups, the p-values were corrected by using Shaffer's method. The p-values are summarized in Table 17. Table 17 showed that there were significant differences during the attenuation between types I and II, and types I and III.

Table 17 p-values based on a comparison for each type

Figures 16 and 17 show the means of the scores for each type during the reconnaissance and attenuation activities. The error bars indicate the SEM for each type. * represents p < 0.05.

Fig. 16
figure 16

Means of the score for each type during the reconnaissance activity

Fig. 17
figure 17

Means of the score for each type during the attenuation activity (*p < 0.05)

Discussion

The purpose of this study was to design an information presentation method with low cognitive load that would enable novices to conduct firefighting activities from multiple perspectives in dynamically changing fire situations. In this study, two axes of evaluation were prepared: with or without the control theory of COCOM, and whether the information was presented at once or in separate views. In this chapter, we will discuss the results of the experiment through a comparison of three information presentation methods that have been created based on these two axes. The difference between reconnaissance and attenuation is that reconnaissance is an information-gathering activity, and attenuation is a decision-making activity based on the gathered information. Since most of the participants in this experiment have plenty of experience in firefighting or defense, reconnaissance activities in residential areas were familiar to them, and they have a clear idea of what to check. In this experiment, they can be considered experts in reconnaissance activities. Since the difference in the mental workload between types I and III was not significant, the idea of what to check during reconnaissance for the experts is considered to be consistent with the control mode of the COCOM. The difference in the mental workload between types I and II indicates that the mental workload increases when the order of the messages is not in accordance with the control mode, even if the number of messages that are displayed is small.

By contrast, because the attenuation activity was an activity that not all of the participants had experienced before, differences in the message presentation type were observed. The judgment work consists of interpreting the gathered information, and plenty of experience is required for this task. Hence, in this experiment, the participants with no experience in oil refining facilities could be considered as novices in attenuation activities. The determination of the water discharge position during the attenuation activities is more difficult than during the reconnaissance activities. This is because there are trade-off problems between the evaluation items that are used for this judgment, and it is necessary to decide the order of priority. In this experiment, there were trade-off problems between the evaluation items that are listed in Table 18.

Table 18 Evaluated items with a trade-off relationship during the attenuation activities

Therefore, the participants need to prioritize the evaluation items that have conflicting advantages and disadvantages. This trade-off problem is particularly difficult for novices because, unlike experts, they have no experience to refer to. This may be the reason why the objective evaluation of type III is lower than that of type I. As a result, the cognitive load increased and the attention to the other evaluation items that was presented was neglected. This may have resulted in the objective evaluation of type III, which has more information, since it is lower than the objective evaluation of type I. Since it has been known in cognitive science for a long time that the effective field of the view shrinks as the cognitive load increases [30, 31], it is believed that the same thing is happening. There was no significant difference between types II and III in terms of the attenuation check score. This indicates that there is no difference between giving information all at once and giving information randomly to the novice in that activity.

From the above, it is considered that the information presentation method that enables even novices to perform firefighting activities like experts is a method that can present information sequentially based on the control mode of the COCOM as in type I. However, a limitation of this study is that this result is obtained using a single robot, and in the case of multiple robots, the results may change due to the division of roles. In addition, the number of cameras mounted on the robot was set to one, but it is thought that the mounting of multiple cameras or presentation of information about the environment, such as current temperature and carbon monoxide concentration are effective means to improve the environmental cognition. In such a case, it is necessary to study how to arrange the images of multiple cameras or information about the environment on the operation screen in combination with the results of previous studies [32, 33]. These are issues for further study.

Conclusion

In this study, an information presentation method was developed that is based on the control modes in the COCOM to ensure that even novices can have a multifaceted perspective in firefighting, which is the key difference between experts and novices. The effectiveness of the information presentation method was verified by evaluating firefighters. This method was evaluated with 12 firefighters that performed two different types of firefighting activities: reconnaissance activities and attenuation activities.

The experimental results showed that types I and were the easiest information presentation methods to use in the reconnaissance activity, and the type of methods to use depends on the preference. By contrast, in the attenuation activity, which is more difficult for the participants, the order of the checks in type I was the most similar to the control mode, and the score was higher. These results verified the validity of the hypothesis that firefighters generally act in the order of the strategic, tactics, and opportunistic control modes when estimating the situation in firefighting. This information presentation method will enable firefighters to support their environmental cognition and decision making in firefighting activities, and young firefighters to perform the same firefighting activities as experienced firefighters, thus contributing to the prevention of fire damage.

Estimating the situation within a limited time frame are necessary activities not only in fires but also in many other disaster scenes. Therefore, it is believed that the information presentation method in this study will be useful for teleoperated robots that are operating in disaster sites. In that case, it is necessary to compare the work of novices and experts, extract the characteristics of experts' work, and then apply the information presentation method in this study. In addition, this study organized the details of information and the order of its presentation that are necessary for estimate work of the situation in firefighting. Therefore, the results of this study can be useful in terms of training firefighters. The results of this study were obtained using a single robot and camera, and questions; the results may differ for multiple robots and cameras, and additional information about the environment. Therefore, the information presentation method for multiple robots and cameras, and additional information about the environment will be a future study topic.

Availability of data and materials

Not applicable.

Abbreviations

HMI:

Human machine interface

DIKW:

Data information knowledge wisdom

AHP:

Analytic hierarchy process

% LEL:

% Lower explosive limit

COCOM:

Contextual control model

SEM:

Standard error of the mean

ROS:

Robot operation system

References

  1. Fire and Disaster Management Agency (2011) FY2011 White Paper on Firefighting. https://www.fdma.go.jp/publication/hakusho/h23/. Accessed 16 May 2021

  2. New Energy and Industrial Technology Development Organization (NEDO) (2014) White Paper on Robot 2014. Chapter 5. https://www.nedo.go.jp/content/100563899.pdf. Accessed 16 May 2021

  3. Krotkov E, Hackett D, Jackel L, Perschbacher M, Pippine J, Strauss J, Pratt G, Orlowski C (2017) The DARPA robotics challenge finals: results and perspectives. J Field Robot 34(2):229–240

    Article  Google Scholar 

  4. Asada M (2020) Rethinking autonomy of humans and robots. J Rob Soc Jpn 38(1):7–12 (Japanese)

    Article  Google Scholar 

  5. Tadokoro S (2019) Disaster robotics result from the ImPACT tough robotics challenge. Springer, Berlin

    Book  Google Scholar 

  6. Delmerico J, Mintchev S, Giusti A, Gromov B, Melo K, Horvat T, Cadena C, Hutter M, Ijspeert A, Floreano D, Gambardella LM, Siegwart R, Scaramuzza D (2019) The current state and future outlook of rescue robotics. J Field Robot 36(7):1171–1191

    Article  Google Scholar 

  7. Gancet J, Motard E, Naghsh A, Roast C, Arancon MM, and Marques L (2010) User interfaces for human robot interactions with a swarm of robots in support to firefighters. In Proceedings of IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, May, 2846–2851

  8. Casper J, Murphy RR (2003) Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Trans Syst Man Cybern Part B Cybern 33(3):367–375

    Article  Google Scholar 

  9. Kadous MW, Sheh RK and Sammut C (2006) Effective User Interface Design for Rescue Robotics, Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, 250–257

  10. Fuchida M, Chikushi S, Moro A, Yamashita A, Asama H (2019) Arbitrary viewpoint visualization for teleoperation of disaster response robots. J Adv Simul Sci Eng 6(1):249–259

    Google Scholar 

  11. Olivares R, Zhou C, Bobby B, Adams JA (2003) Interface Evaluation for Mobile Robot Teleoperation, ACM

  12. Tamura Y, Amano H, Ota J (2020) Analysis of firefighting skill with a teleoperated robot. Robomec J. https://doi.org/10.1186/s40648-020-00177-y

    Article  Google Scholar 

  13. Tamura Y, Amano H, Ota J (2021) Analysis of cognitive skill in a water discharge activity for firefighting robots. Robomec J. https://doi.org/10.1186/s40648-021-00201-9

    Article  Google Scholar 

  14. Rowley J (2007) The wisdom hierarchy: representations of the DIKW hierarchy. Int J Inf Commun Sci 33(2):163–180

    Google Scholar 

  15. Hollnagel E, Woods DD (2005) Joint cognitive systems: foundations of cognitive systems engineering. CRC Press, Boca Raton

    Book  Google Scholar 

  16. Burns CM, Skraaning G, Jamieson GA, Lau N, Kwok J, Welch R, Andresen G (2008) Evaluation of ecological interface design for nuclear process control: situation awareness effects. Hum Factors 50(4):663–679

    Article  Google Scholar 

  17. Van Dam SBJ, Mulder M, Paassen MM (2008) Ecological interface design of airborne conflict support in flexible use airspace. AIAA Guidance, Navigation, and Control Conference and Exhibit. https://doi.org/10.2514/6.2005-6471

    Article  Google Scholar 

  18. Japan Association for Fire Science and Engineering (2018) Kasai binran 4th edition. [Handbook for fire science and engineering 4th edition]. Japanese

  19. Kitamura M (2000) Human factors in nuclear power plant with focus on application of man-machine interface and advanced data processing technologies to nuclear power industry. INSS J 7:303–326

    Google Scholar 

  20. Stanton NA, Ashleigh MJ, Roberts AD, Xu F (2001) Testing Hollnagel’s Contextual Control Model: Assessing team behaviour in a human supervisory control task. Int J Cogn Ergon 5:111–123

    Article  Google Scholar 

  21. Feigh K, Pritchett A, Denq T, Jacko J (2006) Contextual control modes during an airline rescheduling task. J Cogn Eng ecis Mak 1(2):169–185

    Article  Google Scholar 

  22. Saaty TL (1980) The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. McGraw Hill, New York

    MATH  Google Scholar 

  23. Yu D, Kou G, Xu Z, Shi S (2021) Analysis of collaboration evolution in AHP research: 1982–2018. Int J Inf Technol Decis Mak 20(1):7–36

    Article  Google Scholar 

  24. Sweller J (1988) Cognitive load during problem solving: effects on learning. Cogn Sci 12(2):257–285

    Article  Google Scholar 

  25. Miller EK, Buschman TJ (2013) Cortical circuits for the control of attention. Curr Opin Neurobiol 23:216–222

    Article  Google Scholar 

  26. Simons DJ, Jensen MS (2009) The effects of individual difference and task difficulty on inattentional blindness. Psychon Bull Rev 16:398–403

    Article  Google Scholar 

  27. Clearpath Robotics (2015) Husky. [Online]. https://clearpathrobotics.com/husky-unmanned-ground-vehicle-robot. Accessed 18 May 2021

  28. Sandra G, Staveland HE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol 52:139–183

    Article  Google Scholar 

  29. Field A (2017) Discovering statistics using IBM SPSS statistics, 5th edn. SAGE, Thousand Oaks

    Google Scholar 

  30. Williams LJ (1982) Cognitive load and the functional filed of view. Hum Factors 24(6):683–692

    Article  Google Scholar 

  31. Maples WC, DeRosier W, Honens R, Bendure R, Moore (2008) The effects of cell phone use on peripheral vision. Am Optom Assoc 79:36–42

    Google Scholar 

  32. Yokokohji Y, Tubouchi T, Tanaka A, Yoshida T, Koyanagi E, Matsuno F, Hirose S, Kuwahara H, Takemura F, Ino T, Takita K, Shiroma N, Kamegawa T, Hada Y, Osuka K, Watasue T, KimuraT, Nakanishi H, Horiguch Y, Tadokoro S, and Ohno K (2006) Guidelines for human interface design of rescue robots. In: Proceedings of SICE-ICSE International Joint Conference. Busan, Korea, p 3455–3460

  33. Nikolov, S. Tsubota, T. Watanabe, C. Ohta and K. Uehara (2003) Rescue System Based on Location Aware Information Exchange in Mobile AdHoc Network, Proc. of the 2003 International Workshop on Mobile Systems, E-Commerce and Agent Technology, 325–330

Download references

Acknowledgements

This work was partially supported by JSPS KAKENHI (Grant Number 20H04261).

Funding

Not applicable.

Author information

Authors and Affiliations

Authors

Contributions

YT conducted all of the research and experiments. HA and JO developed the research concept. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yoshihiro Tamura.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tamura, Y., Amano, H. & Ota, J. Information presentation method for teleoperated robots to support the multifaceted understanding of fire sites. Robomech J 9, 10 (2022). https://doi.org/10.1186/s40648-022-00224-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40648-022-00224-w

Keywords

  • Contextual control model
  • Human machine interface
  • Firefighting
  • Teleoperated robots
  • Fire
  • Reconnaissance
  • Attenuation