Analysis of firefighting skill with a teleoperated robot

Teleoperated robots are expected to perform tasks in extreme environments such as disaster sites. However, only few teleoperated robots are used for this purpose because their usability is poor compared to human skill in such tasks. Further, the use of human–machine interfaces used by teleoperated robots is in itself a human skill. In this study, we compare the skills of firefighters (experts) when using teleoperated robots and that of firefighters (novice) and analyze the results. Firefighters gain technical skills required for fighting fires from practical experiences; in comparison, novice firefighters have trouble acquiring these skills. We conducted a comparative experiment of the reconnaissance skills of firefighters in which ten firefighters (five novices and five experts) reconnoiter two simulated fire sites (one in a residential area and the other in a commercial facility area) with a teleoperated robot. The experimental results indicated that experts were more multifaceted in their evaluation of reconnaissance areas as compared to novices.


Introduction
In recent years, large-scale disasters have occurred frequently globally. Various teleoperated robots have been developed to quickly overcome problems, such as conducting search and rescue for humans and reducing or recovering damage to structures, and improve operational efficiency in disaster areas and dangerous work sites; however, these robots are rarely used. Few of these robots exhibit superior usability in terms of movement and camera operations. In emergencies, it is important to provide the operators of teleoperated robots with a human-machine interface (HMI) that filters input information effectively [1][2][3][4][5]. However, there are few reports of teleoperated robots that reflect this knowledge.
Despite their infrequent current use and poor HMI, the application of teleoperated robots is anticipated for use at the sites of fires because fires are one of the most common disasters that put firefighters at risk. Several teleoperated robots have been developed for firefighting to reduce the risk to firefighters [6][7][8][9][10].
In addition, because teleoperated robots are work tools, we believe that the effectiveness of the work using a firefighting teleoperated robot depends on the firefighting skills of the operator. Yokokohji et al. [11] summarized various knowledge about the information display method in the rescue robots, but did not mention what kind of information display method is used to assist the skill shortages of operator. To fill the skill gap between novices and experts or to improve the effectiveness of teleoperated robots, several prior studies have evaluated the differences in skill between novices and experts [12][13][14][15][16][17][18][19][20][21][22]. For example, in an investigation of robotic surgery training, Kumar et al. [12] designed evaluation criteria for surgical skills by quantifying objective skill differences between novices and experts. Similarly, Guru et al. [13] evaluated differences in surgical performance with the da Vinci Surgical System for surgeons with different skill levels, ranging from novice to expert. Morioka et al. [14] presented a work information database for cell production assembly created by analyzing the working motions of experts and extracting their skills to improve work efficiency. Roy et al. [15] suggested the usefulness of using human-accessible hierarchical semantic labels to teach novices skills acquired by robots from experts.
Other prior studies have examined surgical operations, assembly work in factories, grasping and transportation work in residencies, creative activities with LEGO, communication, and nursing; however, differences in firefighting skill between novices and experts with teleoperated robots are yet to be clarified.
The mission of firefighting is to protect the lives of citizens and properties from fires or disasters. At disaster sites, mistakes by firefighters may result in immediate loss of the lives of many citizens and damages to property. Therefore, firefighting skills are indispensable for maintaining a safe and secure society. However, it is difficult to acquire firefighting skills because fire sites are diverse and most skills of a firefighter are derived from experience. Furthermore, along with the diversification of disasters and the evolution of firefighting equipment, the required firefighting skills have become more complex and advanced. Developing methods to teach firefighting skills is a significant problem in various Japanese fire departments [23]. In one recent countermeasure against this problem, the Tokyo Fire Department proposed methods to support knowledge succession with social networking services [24].
We believe that there are additional opportunities for improving the utility-especially in terms of HMI-of teleoperated robots for firefighting by evaluating the differences in firefighting skills between novices and experts. Although approaches with augmented reality (AR) or virtual reality (VR) could be considered as methods for analyzing firefighting skills, firefighting activities with AR or VR are less realistic than using teleoperated robots such as unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV). Therefore, in this paper, we analyze the firefighting skills of experts by comparing the behavior of novices and experts in a firefighting activity using a firefighting teleoperated robot deployed at several fire departments. Human skills develop through five stages of skill levels: novice, advanced beginner, competent, proficient, and expert [25]. In this paper, "novices" corresponds to the "competent" skill level and "experts" corresponds to the "expert" skill level. As a first step toward improving the HMI of teleoperated robots for firefighting, we select a reconnaissance activity as the analysis target; this is because reconnaissance at the site of a fire is one of the most important firefighting activities. We create evaluation items to measure reconnaissance activity according to the basic tactics of firefighting and compare the behavior of novices and experts based on these evaluation items. This analysis result is useful for improving the HMI of firefighting teleoperated robots for reconnaissance, such as those used in [19][20][21]. Since the experiment in this paper uses the firefighting robot, the results show the difference between novice and experts firefighting skills using the robot. Therefore, we think that researchers of disaster response robots can smoothly use the results of this paper in the design of HMI.
The remainder of this paper is structured as follows: "Evaluation checklist items for reconnaissance" details the evaluation checklist items to be compared between novices and experts for reconnaissance. "Experiment" section describes the experimental setup. "Result" section outlines the results, and "Discussion" section discusses the results in detail. Finally, "Conclusion" section provides the conclusions.

Evaluation checklist items for reconnaissance
Firefighting activities allow organized firefighting units to regain control of disaster sites and return them to a safe condition using firefighting skills and technologies. Reconnaissance is a firefighting activity; its purpose is to understand the disaster situation (structure damage, the spread of fire, injured persons, and the operating conditions of firefighting equipment) and develop a fire extinguishing and rescue method. In this paper, we define the reconnaissance activity as the reconnaissance required to find a water discharge point, and we confirm whether there is a difference in the approach between novices and experts in this activity. Therefore, we create two sets of evaluation items based on the fundamental tactics of firefighting: one evaluating the reconnaissance activity and the other determining the water discharge point.

Basic firefighting tactics
All firefighting activities, including reconnaissance, are based on common basic tactics. Basic firefighting tactics for general buildings (for example, wooden and fireprotected buildings) follow the ten principles provided Table 1 [26]. All firefighters learn and practice these tactics at the fire academy over half a year; therefore, even novice firefighters have knowledge of the basic principles of firefighting.

Evaluation items
We created evaluation items for the reconnaissance activity and for determining the water discharge point using the basic tactics of firefighting. The evaluation items for the reconnaissance activity are listed in Table 1. In this study, reconnaissance is considered to start after people are evacuated from the area, and therefore, searching for injured persons is not included in Table 2. Next, the evaluation items for selecting a water discharge point are listed in Table 3. We used the Wilcoxon rank sum test to test Tables 2 and 3. This is because we thought that the result of evaluation items did not follow a normal distribution. Figure 1 shows a diagram and photograph of the experimental field, which measured 24 m (breadth) × 36 m (length). Two fire areas-one simulating a residential area and the other a commercial facility area-were prepared in this field. The residential area, representing crowed houses and condominiums, comprised one building and three passenger cars (red vehicles in Fig. 1b) in the experimental field. The commercial facility was meant to simulate a large shopping mall (a fireproof building) surrounded by a parking lot. The experimental field was prepared by reusing a firefighter training facility.

Experiment location
Each area was divided by road cones and poles to prevent easy movement between the two areas. The reconnaissance starting point, from which the robot moves, is the midpoint of these two areas. Nine marker cones ( Fig. 2) represented fire hydrants (yellow) and fire cisterns (red) in the field. Fire points were indicated by flags called "genjibata" (Fig. 3). We used three types of flags in this experiment: white, yellow and red, which respectively indicate the early, growth, and Perform firefighting activities that prevent the spread of fire to surrounding buildings 3 The first team locates a water discharge point near the fire for utilization and stops its firefighting vehicles so that they will not be an obstacle to teams arriving later 4 For enlarged fires, point a hose at the back or side of the burning building and enter the building from the top floors 5 In early small-scale fires, point a hose at the front of the burning building to extinguish the fire in one push 6 The fire spread prevention points are the inner walls, attic spaces, corridors, and stairway entrances 7 For enlarged fires in a city, point a hose in the leeward direction to mitigate the risk of the fire spreading   high-temperature phases of fire smoke. In this experiment, we did not simulate the progression of fire over time.
Because the building and the flag in this experiment were used in actual firefighter trainings, and because the firefighting robot in this experiment has already been equipped in several firefighting headquarters, we thought that this experimental environment would be familiar to firefighters, would be less likely to cause misidentification of the this environment than a computer simulation, and would reduce the variation in the level of understanding of this experiment. Therefore, we did not use a computer simulation but a real robot and environment in this experiment. From the perspective of safety and feasibility, we simulated fire hydrants and fire cisterns with marker cones in consultation with firefighters.

Experimental configuration
To develop a practical situation, we assumed the following conditions that were decided upon consultation with experienced firefighters. Indeed, these firefighters were not part of the experimental participant group.
Fire situation: An epicentral earthquake has caused fires to break out simultaneously in many places in the prefecture. Participants are firefighters in the A City Fire Department (CFD:A). When all firefighting vehicles in the CFD:A go out for firefighting activities in A City, a request for support is sent from the B City Fire Department (CFD:B) to the CFD:A. The CFD:B has received information about the locations of the fires in B City, but has not yet received water utilization information about B City. All firefighting vehicles in B City are going out for firefighting activities in B City and A City.
• Purpose of reconnaissance: To begin water discharge as soon as the fire engine arrives at this area, the firefighters in the CFD:A reconnoiter with the teleoperated robot and decide on positions for water discharge. • Reconnaissance timing: For the sake of simplicity, participants start reconnaissance after the evacuation of citizens was completed. • Teleoperated robot: Fig. 4 shows the reconnaissance robot for firefighting, "FRIGO-M" [27], which was developed by the Mitsubishi Electric TOKKI System Corporation.
This robot is a nonignitable and explosion-proof and has a crawler mechanism and two cameras (one for moving and the other for reconnaissance). Table 4 lists FRIGO-M's basic specifications.
Because operating this robot with an optical cable is difficult, we set it up wirelessly in this experiment.   Figure 5 shows the work station of the operator: the laptop is used to operate FRIGO-M and the PC monitor is used to measure the eye trajectories of participants. Participants operated this robot using a PC game controller installed on the laptop.
• Experimental participants: Ten firefighters (five novices and five experts) participated in this experiment. It is difficult for young firefighters to acquire firefighting skills because many depend on practical experience; therefore, many years of firefighting are required to become an expert. In this experiment, we define the threshold for an expert as 10 years of firefighting experiences, because most firefighters become sergeants, who focus on protection and rescue, after 10 years of service.
Since our aim was not to compare the skill of the firefighter in operating FRIGO-M, all participants were provided 2 weeks of practice in FRIGO-M's operation before the experiment.
• Force of the fires: Participants were informed that the force of fire in the commercial facility was larger than that in the residential area as preliminary information. • Firefighting vehicle: Two firefighting vehicles were scheduled to arrive at the fire site after reconnaissance: fire engine A (discharge amount: 1500 L/min) and fire engine B (discharge amount: 2000 L/min). In this experiment, we assume that only one firefighting vehicle can be located in each area. • Weather conditions: Table 5 summarizes the weather condition considered in this experiment. We assume that the weather condition is that of a day in summer. This is similar to the seasons when this experiment is carried out. • Data acquisition: To compare the reconnaissance behaviors of novices and experts, we obtained several types of data in this experiment, which are listed in Table 6.

Experimental procedures
The experiment procedures are detailed below.
(1) Explain the reconnaissance area and purpose to the firefighter, after providing them with a recording sheet (Fig. 6), which is typical of a Japanese recording sheet.    (2) The firefighter operates the teleoperated robot, beginning from the midpoint between the residential area and commercial facility area, for reconnaissance. (3) After reconnoitering the fire site, the firefighter describes two discharge points on the recording sheet. (4) The firefighter completes a questionnaire and interview regarding the reconnaissance checklist items.
In order to conduct interviews based on the questionnaire results, the questionnaire is composed of open questions.
• How did you decide which of the commercial facility and residential areas to reconnoiter first?
• How did you select the water utilization points at each area? • How did you decide on the location of the fire engine at each area? • What did you pay attention to during reconnaissance? • What additional functions do you want for this reconnaissance robot? Please provide reasons for each. Table 7 shows the responses from the recording sheets of the participants. Figure 7 shows the operating monitor during reconnaissance. Participants can check the direction of movement of the robot, the pan or tilt angle Please describe the water discharge position for each area as Χ, the used water utilization as ○, the positions of two fire engines (Fire Engine A : 1500 L/min; Fire Engine B : 2000 L/min) as □, and the state of laying the hose as a straight line or a curve. Fig. 6 Recording sheet of the camera, and pitch or roll angle of the robot on the operating monitor. The participants operate the robot and camera with a PC game controller to check areas of concern. Figure 8 shows an example movie measured by the eye tracker for one expert. The green circle surrounds the viewing position of the participant in the eye tracker in Fig. 8. Table 8 summarizes the scene description of Fig. 8. The operator is checking one side of the commercial facility thoroughly. Other experts and novices checked this side of the commercial facility with a similar method. We confirmed from interviews that even if the teleoperated robot is used in this experiment, the path followed by the camera to look around the object is the same as the eye movement in normal reconnaissance activities without the teleoperated robot.

Wind Direction
Next, we compared the time taken for reconnaissance by novices and experts. Table 9 summarizes the reconnaissance times of novices and experts for thoroughly checking one side of the commercial facility, their average, and unbiased standard deviation. Although the number of samples (ten) is not large, we assume that the reconnaissance time follows a normal distribution because it occurs under the same conditions. Further, for a large number of samples, it would still follow a normal distribution. We used a F-test to confirm the Since the t value is smaller than t 0.05 , there was no significant difference between the population means of novices and experts. We compared the camera operations for reconnaissance by novices and experts. Table 10 summarizes the camera operations of novices and experts for thoroughly checking one side of the commercial facility. In Table 10, the numbers of camera operation show the area that is divided into four sections on one side of the building (Fig. 9). We specified how the camera operation was performed with these sequences of numbers and used levenshtein distance to compare the differences in camera operations between novices and experts. The basis sequence for calculating the levenshtein distances is I, II, III, and IV, the most popular sequence in participants' results.
We used the Wilcoxon rank sum test to confirm the presence or absence of a significant difference between the levenshtein distance of novices and experts listed in Table 10, because we thought that the result of levenshtein distance did not follow a normal distribution. From Table 10, the p value was 0.90, and it was found that there was no significant difference between the two groups.
Therefore, in terms of reconnaissance time and checking order, there is no significant difference between novice and expert populations.     -I  II  III  IV  -Novices   A  IV  III  I  II  4   B  IV  III  II  I  4   C  I  II  III  IV  0   D  I  II  III  IV  0   E  IV  III  I  II  4   Experts   F  I  II  III  IV  0   G  IV  III  II  I  4   H  I  II  III  IV  0   I  III  IV  I  II  4   J  IV  III  I  II  4  Tables 11 and 12 show the results of the evaluation  items from Tables 2 and 3, respectively. These tables were drawn by integrating the interview and questionnaire results. The total score in Tables 11 and 12 is the total value with a score of 1 if an evaluation item has been checked and 0 if it has not been checked. We used the Wilcoxon rank sum test to confirm the presence or  Table 11 Result of the evaluation items for reconnaissance activity

Novices Experts
Total score 2 2 2 4 2 4 6 6 5 5 Table 12 Result of the evaluation items for deciding on the water discharge point absence of a significant difference between the total scores of novices and experts listed in Tables 11 and 12.  From Table 11, the p value was 0.012 (effect size: r = 0.76); it was found that there was a significant difference between the groups of novices and experts. Because the effect size is higher than 0.50, a substantial difference was confirmed [28].

Novices Experts
From Table 12, the p value was 0.19 (effect size: r = 0.40), and it was found that there was no significant difference between the two groups.
According to these results, experts evaluate reconnaissance areas with a more multifaceted perspective than novices, even when reconnoitering the same area. Figures 10 and 11 show the graphs of the evaluation items from Tables 11 and 12, respectively. In these figures, the axis value of each item shows the number of participants who fit each item. Table 13 show the result of the novices' interviews on the evaluation items. From Table 13, all novices can check more evaluation items if others point out.

Discussion
This study aimed to investigate the differences between novices and experts for both a reconnaissance activity and for deciding on a water discharge point. We assumed that there would be significant differences for both tasks; however, no significant difference was observed in the task of deciding the water discharge point. We believe that the various shapes and types of building are reasons why there is a significant difference between novices and experts in the evaluation items for the reconnaissance activity. Novices have insufficient and biased experiences against the fire types on buildings, and therefore, it is difficult for them to organize what to check for in a reconnaissance activity. However, experts organize the items to check for reconnaissance activity from their various firefighting experiences.
There was no significant difference in the tends of novices and experts to operate monitor system. We believe that this indicates that both novices and experts are aware of the importance of looking the entirety of the objects without omission to gather information. However, unlike experts, many novices only look at the current situation. It is thought that this is because novices have little firefighting experience and their imagination does not work well for what happens next.
The interviews confirmed that if novices are directed to checklist items by others, they can perform reconnaissance in as multifaceted a manner as experts (Table 13). This may present a method of filling in the gap between the firefighting skill and reconnaissance  abilities of novices and experts, such as in determining the water discharge point. For determining the water discharge point, theories of water discharge (e.g., to discharge water from the windward direction) are common for firefighters independent of the fire types, and we think that there is no significant difference between novices and experts.
From these results, we consider that the evaluation items for deciding on the water discharge point are items that are not affected by differences in experience.
Because the human cognitive resource is limited, the more information the HMI presents, the more ineffective it is; we believe that information suitable for the purpose should be extracted and presented appropriately.
The result of the recording sheets of the participants (Table 7) and individual evaluation items (Figs. 12 and 13) are discussed in the supplementary information. However, these discussion needs to cover the qualitative differences between novices and experts. To confirm the significant difference in the results for individual evaluation items, the sample size used in this study is small, and therefore, a more detailed experiment is necessary for future work. Table 7 indicates the differences between novices and experts as seen in the reconnaissance order and selection of firefighting vehicle. Novices tended to check commercial facility area first and to select the fire engine B for this area, whereas experts tended to check the residential area first and selected diverse firefighting vehicles. From the results of interviews and questionnaires, we confirmed that the preliminary information given influenced this decision, as the force of fire in the commercial facility area was greater than in the residential area. Almost all novices were sensitive to this information, they checked the commercial facility area first without considering the spread of fire in the residential area, and they then selected the fire engine B for its discharge amount. Although all firefighters learn the principle of focusing on preventing the spread of fire to surrounding buildings, almost all novices were distracted by the preliminary information. In contrast, all experts acted on this principle without being distracted by the preliminary information and considered more situations than novices. Even though experts H and I checked the commercial facility area first, this reason was to confirm that there was no danger of the fire spreading outside the commercial facility area, which they were not familiar with. The above results indicate that novices tend to be anchored in preliminary information. Figure 12 indicates the differences between novices and experts in their reconnaissance of the influence of fire on the environment, the building entrance, rapid Fig. 12 Comparison of checklist items evaluated by novices and experts during reconnaissance combustion, and the influence of elapsed time. These items are difficult to assess and the assessment depends on many years of firefighting experience. Figure 13 indicates the difference between novices and experts in checking the source of fire. Fire extinguishing methods change according to the source of fire. Since experts understand that the cause of fire is important for effective firefighting, they tend to check not only the location of the fire but also its source.

Conclusion
In this study, we analyzed the firefighting skills of experts by comparing the behavior of novices and experts in a firefighting activity with teleoperated robots. By comparing the behaviors of novices and experts during reconnaissance, we confirmed that the difference between novices and experts is the presence or absence of multifaceted evaluation. Experts evaluate the reconnaissance areas in a multifaceted way and act based on that result. In contrast, novices evaluate a limited range of the reconnaissance areas. As stated in "Introduction" section, "novices" corresponds to the "competent" skill level and "experts" corresponds to the "expert" skill level in this paper. The "competent" skill level indicates the ability to identify important items within a limited range. Therefore, many novices cannot evaluate reconnaissance areas in as multifaceted a manner as experts can.
Interviews confirmed that if checklist items are pointed out to novices by others, novices can perform as multifaceted an evaluation as experts. Based on the results of this experiment, we believe that by developing the HMI of firefighting teleoperated robots that presents evaluation items on the operation screen, reconnaissance from a multifaceted perspective similar to that of an experienced firefighter is possible, even for novice operators. Casper et al. [29] reported that more research on perceptual user interfaces was needed for future development of HMI. We believe that such a HMI with the function to present information checked by experts is one of the good perceptual user interfaces. Since the experiment in this paper uses the firefighting robot, the analysis results of this paper extract the firefighting skills using the robot. Therefore, we think that the extracted skill, the multifaceted perspective, can be directly used as necessary elements in the design of HMI for disaster response robots.