- Research Article
- Open Access
Research on effective teleoperation of construction machinery fusing manual and automatic operation
ROBOMECH Journal volume 4, Article number: 14 (2017)
It is known that the work efficiency of teleoperated construction machinery is lower than that of directly operated machinery. Assistance via automatic control is expected to improve the work efficiency. However, this assistance might break the feeling of control and prohibit control adjustments by the operator.
We propose a semiautomatic system that fuses manual and automatic operation while maintaining the same feeling of control as manual operation. The assistance approach to the working trajectory is based on the assumption of the existence of an ideal trajectory for the hoist swing that is a major component of an excavator. We evaluate the feeling of control using a sense of agency.
The results of an examination using a miniature excavator show that the assistance improved the perspective error from manual with a high sense of agency.
Therefore proposed assistance was effective during teleoperation without a sense of perspective.
The development of unmanned construction using teleoperation has been accelerated for disaster sites or mines. However, it is known that the work efficiency of teleoperated construction machinery is lower than that of directly operated machinery . We need to improve the work efficiency. Unmanned construction work for practical use may be realized by teleoperation and autonomous operation. However, both approaches have some known weaknesses.
Autonomous operation plans and controls the construction machine with measured around information. Yamamoto proposed autonomous excavating operation using a hydraulic excavator . The complete calculation of the behaviors of stone and sand is difficult, as these behaviors may change owing to the machine control, the weight, the ratio of water content, the ratio of composition, etc. It is not easy to measure the ratio of water content and the ratio of composition. Even if this information can be measured, the large number of particles of sand will make real-time calculation difficult. Therefore, current autonomous operation cannot control a construction machine in all situations. Of course, expert operators can manipulate the sand as they imagined as long as they are on board the machine.
A teleoperation system controls a construction machine from a remote place using images captured by a camera attached to the construction machine. During teleoperation, operators lack a visual field, the acceleration of the machine, the sound of an engine, and a sense of perspective. In particular, a two-dimensional (2D) image makes it difficult to obtain a sense of perspective. A lack of perspective increases the working time because it is difficult for operators to adjust the position . Past studies supplied a sense of perspective with three-dimensional images , superimposed computer graphics (CG) , etc. However, three-dimensional images tire the operators because the working distance of a construction machine is different from the screen distance. Superimposed CG provides distance information to the operators. The tip of a construction machine was superimposed on the ground with CG. However, the CG information is in a different format from the normal perspective. Therefore, operators must think and convert the CG information into the perspective information in their mind. In contrast, autonomous operation can control the position accurately and quickly by measuring the distance to the target.
Therefore, we need both autonomous operation and teleoperation for unmanned construction. However, if the unmanned construction system switches between the autonomous and manual modes, the work cycle is separated into parts. The work efficiency is not improved, and the operator is frustrated when switching modes. In contrast, a semiautomatic system assists with manual operation according to a designed plan combining automatic and manual operation. Researchers have previously proposed assistance systems [6, 7]. Shimano proposed assistance that limited control in the case of an out-of-design plan . Kubo adjusted a manipulability measure when controlling the manipulator of a leg-type robot . However, these researchers evaluated the accuracy or performance of the system, but they did not evaluate the feeling of control. In situations that are difficult to calculate for the assistance system, misdirected assistance causes a sense of discomfort during control and a low work efficiency.
Ideally, the assistance would not cause discomfort and result in a high work efficiency. One strategy for achieving this is for the operator to remain unaware of the assistance. If operators remain unaware, they obtain a feeling of control that is the same as manual control. Igarashi proposed assistance without human awareness . Igarashi et al. modified the dynamical parameters of a robot to approach an internal model to operate a robot with complete control by an operator . However, they failed because of discomfort felt by the operators when changing the actual robot dynamics. Therefore, a limit on the rate of change in the dynamics was proposed, and the assistance worked without human awareness and improved performance .
In an experiment, operators controlled a mobile robot by steering to trace reference lines. However, in our construction machine, operators control on the basis of a work plan without a detailed reference trajectory. Operators have common actions, e.g., excavation and manipulation of a hoist arm. Operators change these actions according to the work target, habitual actions, and work progress. In particular, in teleoperation, it is difficult for operators to control the tip of a construction machine in three-dimensional space without perspective.
Therefore, in this paper, a semiautomatic system that fuses control by manual and automatic operation is proposed, which achieves a high work efficiency and a feeling of control that is the same as manual control, by the teleoperation of a construction machine. The target of this work is the hoist swing that is typically used during the operation of a construction machine.
The first experiment involves a 2D CG situation. It is difficult for a skillful operator to control the construction machine. However, operators could obtain information for control as long as the 2D CG situation. In this experiment, awareness of assistance was evaluated automatically and manually along with two assistance methods by a sense of agency .
The second experiment involves teleoperation of miniature excavator. Teleoperation does not provide the operators with a sense of perspective, which is different from the 2D CG situation. The assistance parameter of the 2D CG situation was reflected in the teleoperation. The work accuracy and the sense of agency were evaluated without a sense of perspective.
Target work and evaluation list
The target of this work is a hoist swing. Figure 1 shows the digging cycle. Figure 1(2) shows an image of a hoist swing. Operators use an excavator for multiple types of work in the field, e.g., excavating, carrying, loading, dozing, etc. The movement of the hoist swing is the most typical and repeated work. The other side hoist swing is most possibility work that fuses manual and automatic operation. Figure 2 shows an image demonstrating assistance for the hoist swing.
In teleoperation work, the semiautomatic system should assist the manual control. The lower-right image in Fig. 2 shows a camera image during teleoperation. The bucket tip, i.e., the tip of the construction machine, moves over the ideal position because of the lack of a sense of perspective. Operators might adjust the position by using the changes in the picture relative to the surroundings and dumping sand. However, the work time may be increased. In order to improve the work efficiency, it is necessary to adjust the position at the same time as boarding. Therefore, operators need assistance.
A hoist swing may be suitably carried out with an automatic function. The work trajectory of a hoist swing during repeated work by an expert operator is almost constant. This trajectory can be approximated by a cubic interpolation relatively easily. Therefore, it is thought that a trajectory close to the trajectory of the operators could be automatically generated, and the operators’ trajectory was assumed to be the ideal work trajectory.
Although the dumping position is influenced by work progress, an automatic function can create a hoist swing trajectory. Sometimes, it is necessary for the operators to finely adjust the position near the dumping point because the automatic system cannot fully calculate the behavior of the sand. If the operator does not notice the assistance, they can continue to control the machine. Therefore, in this study, an assistance method was evaluated, assuming an ideal work trajectory for the hoist swing.
The assistance method was evaluated by the following criteria:
Do not let the operator notice the assistance.
Guide the bucket tip to an ideal position.
Guide the bucket tip along an ideal work trajectory.
The first point evaluates whether the feeling of control is the same as manual operation. If the semiautomatic system assists the control of the operators, operators can control same as in manual operation without awareness of the assistance. The second point evaluates the accuracy of the end point. Operators control the tip of a construction machine to a target point. Closer is better, but operators cannot achieve good control using teleoperation without assistance. The third point evaluates whether the actual trajectory is close to the ideal trajectory. If the end point is evaluated, the ideal trajectory does not contribute to the work efficiency. However, the assistance is stronger at the end point if the actual trajectory is farther away from the ideal trajectory. Moreover, assisted trajectory will be away from the natural trajectory of the operators, the movement efficiency will decrease, and the risk of conflict surrounding the object will increase.
In this paper, the authors propose trajectory assistance and compare it to other two assistance methods. Trajectory assistance provides assistance to achieve the ideal trajectory, as in Lane Keeping Assist System. Since the position is close to the ideal trajectory, it is estimated that the power of the assistance is lower, and the feeling of control is better. One of the other assistance methods is goal assistance. Previous research described a method for assistance to a target position. If a control error accumulated, a large amount of power is provided during assistance to achieve the target position accuracy, which might disturb the feeling of control. The other assistance method is automatic assistance. The automatic method forcibly moves the tip along the ideal trajectory. However, operators are sensitive to the feeling of the control speed. Therefore, the control speed was adjusted to the input speed.
Figure 3 shows trajectory assistance and goal assistance. v i is the input speed vector estimated by the lever input. v a is the assistance speed vector. v m is the mixed speed vector, which was the tip speed after assistance. The trajectory method provided assistance to achieve the ideal trajectory. An estimate of the next tip position was calculated by the tip position and input speed. l t is the minimum distance between the estimated tip position and the ideal trajectory. v a is a function of l t . Two types of functions of l t were examined: linear and quadratic. The coefficient l t of was defined for each examination.
The sum of the speed vectors v i and v a indicated the direction of the mixed speed vector. The sum of the speed vector was higher than the input speed and increased the feeling of uncomfortableness. Therefore, the speed of the mixed speed vector was adjust to the input speed. The mixed speed vector is expressed as follows:
The goal method provided assistance to the target position directory without the ideal trajectory. From Fig. 3, l g is the distance between the tip position and the target position, and v a is a function of l g . The mixed speed vector was calculated using formula 1, as in the trajectory method.
The automatic method provided forcible assistance to achieve the ideal trajectory. The mixed speed vector was calculated using the trajectory method. To maintain a position on the ideal trajectory, v a was same as l t .
CG situation experiment
In the first experiment, the assistance methods were evaluated with a 2D CG simulation. The test operators could not control a real excavator without a license. It was difficult to obtain many expert operators. Moreover, amateur operators could not control the excavator well in the real world. However, it is easy to control a 2D CG simulation. Operators became familiar with the controls immediately and could obtain all of the control information without removing the perspective. Therefore, the assistance methods were evaluated using a simulation, and the results were reflected in the teleoperation experiment.
Operators controlled a marker instead of the tip of an excavator in this experiment while looking at a monitor (23 inch Mitsubishi Electric RDT2324WLM monitor) placed 700 mm away. Operators controlled the speed of the marker with a joystick (Saitek Corp. SN-PS41) located 450 mm away from the monitor. The image resolution of the monitor is 1920 × 1080 pixels (width × height).
Figure 4 shows the task of this experiment. This task was defined to trace the ideal trajectory imitating a hoist swing from the start point to the goal point. The ideal trajectory was showed to the operators for 5 s before starting and was hidden during each trial. During actual work, operators cannot see the ideal trajectory. However, as previously described, operators almost control the hoist arm along a fixed trajectory. It is estimated that the operators are imagining a shared trajectory. Therefore, this experiment showed the ideal trajectory to imagine the shared trajectory.
All operators could not match the reference trajectory precisely. However, there was no problem if the operators imagined a slightly misaligned trajectory because each operator in a real situation has an image of the trajectory, and the automatically generated trajectory did not match the ideal trajectory perfectly.
The ideal trajectory was generated by a third-order spline curve through three points. The coordinates of these points are as follows:
Start point: [−600, 200]
Relay point: [0, 200]
Target point: [600, 300]
The condition of the spline curve was designed that it is not necessary to invert the joystick input along the Y axis. The background image was set to many randomly located white dots. An image of the dump was placed near the target point. A white square imitated a barrier object. These conditions helped the operators to imagine the speed of the marker.
The input direction of the lever and the movement direction of the marker correspond to each other. The maximum speed was defined 300 pixel/s along the x and y axes. The maximum speed at 45° was 424 [pixel/s]. Some operators conducted preliminary tests to determine the right speed. To improve the fine control, three graded linear control maps were used.
Six types of assistance methods were compared as the experimental level:
Manual: v a = 0
Trajectory1: v a = 3l t
Trajectory2: v a = 0.036l2l t 2
Goal1: v a = 360/((l g /100)2 + 1)
Goal2: v a = 360e −lg/400
Auto: v a = l t
Two functions were utilized for trajectory assistance and goal assistance. In addition, the manual and automatic methods were compared. The process of (a) was the same as trajectory assistance, and (a) has the same output as manual by setting v a to zero. The coefficients for (b) and (c) were designed that the marker did not to go out of the image of the dump when moving at the maximum speed. The coefficients for (d) and (e) were set heuristically. v a for (d) increased sharply as it approached the target position. v a for (e) gently increased as it approached the target position. The automatic assistance traced the ideal trajectory using the trajectory method.
Twelve operators participated. They consisted of 10 adult males and 2 females aged 22–63. Ten operators had never controlled a real excavator. Two operators had a license, but they are not professionals. They practiced the trajectory 50 times using manual control as imagined. The total number of trials of the experiment was 100, and each assistance method was randomly chosen. To make it difficult to determine whether assistance is applied, the number of manual trials (a) was 50, and the numbers of trials for the other assistance types (b–f) were 10 each.
Before the experiment, operators were showed the assistance types and informed that the system might provide assistance at the time of operation. However, operators did not know the existence and type of assistance for each trial. The operators were instructed not to modify their control many times near the target point.
The evaluation metrics were the cycle time, the accuracy of the distance from the target point, the accuracy of the distance from the ideal trajectory, and the feeling of control. The feeling of control was evaluated by the sense of agency . The sense of agency was defined by Gallagher as “The sense that I am the one who is causing or generating an action.” If operators feel strangeness compared with manual operation, the operators may notice the assistance and lose the sense of agency. Previous researchers evaluated the awareness with the sense of agency [11, 12]. Operators provided three different answers to the question “Did you feel the sense that you are the one who is moving marker?”: “I felt assistance,” “I did not know,” and “I did not feel assistance.” Score were set 1, .5, and 0.
Results and discussion
Figure 5 shows the results for the sense of agency. The horizontal axis shows the assistance type, and the vertical axis shows the average number of points for the sense of agency. The error bars indicate the standard deviation.
The effect of the assistance method was significant according to an analysis of variance [F(5;1194) = 70.26; p < .01]. There was a significant difference according to a Bonferroni multiple comparison, except for (a–c), (b–c), (b–d), (b–e), and (d–e).
It was important that operators often felt assistance during the manual trials. Moreover, operators often did not notice the assistance during the auto trials. Thus, if an operator has an unsure feeling, the assistance provides a feeling of reassurance outside of recognition. Considering this result, the operators’ sense is ambiguous, and it can be considered that there is possibility that assistance can be used without awareness.
However, the sense of agency for the trajectory assistance was better than that for goal assistance. It seems that there is a small difference between trajectory assistance and goal assistance regarding the sense of agency. Figure 6 shows the average number of points for operators that felt assistance. When operators felt assistance with trajectory or goal assistance, the points were set to 1. It was also possible to select duplicates. Focusing on the rate of noticed points where operators felt assistance during the manual trials, the operators felt the assistance more in the middle of the trajectory than near the goal. The trend of the noticed points for trajectory assistance was the same as that for manual assistance. The cases where operators judged that there was assistance without awareness might be included for trajectory assistance. However, operators felt assistance near the goal for goal assistance. Therefore, it was concluded that for goal assistance, operators really noticed the assistance, which is not a misunderstanding.
In this experiment, when the assistance power was higher, the assistance was less likely to be noticed. Whether or not the assistance is noticed is thought to be influenced by a combination of the assistance force, the assistance timing, the duration of assistance, the assistance ratio to the input, etc. Further consideration will needed to yield any findings about these factors.
There was not a significant difference between each assistance type with regard to the cycle time. Figure 7 shows results for the error in the position, and Fig. 8 shows the results for the error in the trajectory. The horizontal axis shows the assistance type, and the vertical axis shows the error in the distance from the target point or the ideal trajectory. The error values in Figs. 7 and 8 were expressed as the ratio of the error of manual (a) since the degree of mastery of operation was different for each individual. The error bars indicate the standard deviation. The trend for the error values shows that the assistance was as designed. Therefore, the position error of the each assistance method was better than manual assistance, and goal assistance was better than trajectory assistance with regard to the error in the position. Moreover, trajectory assistance was better than manual assistance and goal assistance with regard to the error in the trajectory. Trajectory assistance was a little effective considering the results for the sense of agency, but there was not much difference for this experiment.
In the 2D experiment, it was a relatively easy to notice the assistance because the operators easily controlled the marker and obtained all information. It is predicted that it will become more difficult to notice the assistance during teleoperation, where a perspective is lacking. Therefore, the second experiment was evaluated by teleoperation of a miniature excavator.
Experiment with teleoperation of a miniature excavator
The task of this experiment was a hoist swing using a teleoperation platform for a miniature excavator . The teleoperation platform was configured to work with the miniature excavator and an interface for control and display. The miniature excavator was a one-twelfth-scale hydraulic excavator (PC-200, manufactured by Komatsu Ltd.). Figure 9 shows the miniature excavator, and Fig. 10 shows the movable range. A camera (Camera: FL3-U3-88S2C-C, Point Grey Research, Inc.; Lens: LM5JC10M, Kowa Company, Ltd.) was set on the operator seat of the miniature excavator for capturing images.
The operator interface was configured with a display (REGZA55EX3, Toshiba Corp.) and joystick. The display showed an image captured by the camera to operators. Operators controlled the miniature excavator with the joystick, which was the same as a real one. Figure 11 shows the experimental setup.
Operators controlled the tip of the excavator to trace the ideal trajectory from the start point to the target point in an ISO pattern. The trajectory was determined by a heuristic reference to the real trajectory of the hoist swing. The bucket of the excavator was kept almost flat during a real hoist swing. To provide easier operation during this experiment, the operators did not control the bucket, which was always controlled downward. It was more difficult to control the excavator in this experiment than in the CG experiment, and operators sufficiently practiced before the experiment. Therefore, the number of the operators was reduced to 3, and operators had 5 days to practice for a total of 3 h. During training, display system superimposed an ideal trajectory on the camera image. Regarding the recognition of the depth, when the bucket is before the ideal trajectory, we hides the ideal trajectory of CG and taught it. Training was conducted in three cases superimposing the ideal trajectory with occlusion, superimposing the ideal trajectory without occlusion, and displaying only the camera image without the ideal trajectory. In experiment, the screen displayed only the camera image without the ideal trajectory.
The miniature excavator has different dynamic characteristics and control performance than a real excavator. Thus, the authors obtained a log for a real excavator when professional operators controlled a hoist swing during actual working conditions. Ideally, it is better to configure a controller that reproduces the transfer function of a real excavator. However, the transfer function changes depending on the load and engine throttle situation. Therefore, the authors reproduced the delay time and acceleration of a real machine relative to the lever input for the miniature excavator and approximated a real excavator.
The assistance types were (a) manual, (c) trajectory2, and (e) goal1 from the CG experiment. Assistance (c) provided the best sense of agency in the CG experiment. There were no differences between assistance (e) and assistance (d). The coefficients of v a was determined by scale from the CG experiment.
Manual: v a = 0
Trajectory: v a = 0.1785l2l t 2
Goal: v a = 104e −lg/100
Note that the power of assistance (C) was very strong near the target position. Therefore, the power of this assistance became 0 within 5 mm.
The evaluation metrics were the same as those for the CG experiment. Three operators conducted the experiment. The total number of trials for the experiment was 40, and each assistance method was randomly chosen. To make it difficult to determine whether there is assistance, the number of manual trials (A) was 20, and the numbers of trials for (B) and (C) were 10 each.
Results and discussion
Figure 12 shows the working trajectories of an operator from the experiment in each assistance method. The Y axis is the depth direction. It was confirmed that the accuracy in the depth direction decreases without a sense of perspective with manual control.
Figure 13 shows the results for the error in the trajectory, and Fig. 14 shows the results for the error in the position. Figure 15 shows the stop position of an operator in the depth direction that is prone to having errors during teleoperation. The error values in Figs. 13 and 14 were expressed as the ratio of the error of manual (A).
The error along the Y axis was larger than that along X axis for manual assistance (A). However, the error along the Y axis was decreased for trajectory assistance (B), the same as goal assistance (C). Each assistance method supported and improved the accuracy by teleoperation. Trajectory assistance (B) was better than other the assistance methods with regard to the accuracy of the trajectory, as designed. Trajectory assistance (B) and goal assistance (C) were better than manual assistance (A) with regard to the accuracy of the position. There was not a significant difference between each assistance type with regard to the cycle time.
Figure 16 shows the results for the sense of agency, and Fig. 17 shows the noticed points. The effect of the assistance method was significant according to an analysis of variance [F(2;117) = 14.93; p < .01]. There was an insignificant difference between trajectory assistance (B) and manual assistance (A). However, there was a significant difference between goal assistance (C) and manual assistance (A) according to a Bonferroni multiple comparison. The results show that trajectory assistance fills three evaluation metrics, possibly improves the accuracy, and maintains a feeling of control that is the same as manual assistance during teleoperation.
In this paper, the target points were not changed. However, it is conceivable that the target point of the automatic system may be different from that of the operator. Further consideration will be needed to yield any findings about the modified target point and trajectory according to the intention of the operator.
The teleoperation of a construction machine has a low work efficiency due to the lack of perspective. If an automatic system can provide assistance with the operator’s control, operators may feel uncomfortable. Therefore, the purpose of this study was to develop a semiautomatic system with a high work efficiency and achieve a feeling of control that is the same as manual control using teleoperation of a construction machine.
In this paper, the target work was the hoist swing, and an experiment was conducted assuming an ideal trajectory. A hoist swing is the typical and repeated work of a construction machine and was generally a fixed trajectory. The previous assistance method becomes too strong near the target point and disturbs the operator’s feeling of control. Therefore, the semiautomatic system supported the ideal trajectory with little assistance and maintained the operator’s feeling of control to be the same as manual control.
The first experiment was carried out in a 2D CG environment that does not require perspective, in which the target is easy to control. The second experiment was carried out using the teleoperation of a miniature excavator that is difficult to control in a 3D environment without a sense of perspective. In these experiments, the awareness of assistance was evaluated for automatic, manual, trajectory, and goal assistance by the sense of agency. Operators often felt assistance during the manual trials and often did not notice the assistance during the automatic trials. Considering the results, the operators sense is ambiguous, and it can be considered that there is possibility that assistance can be used without awareness. When operators controlled the semiautomatic system with assistance to ensure the ideal trajectory, the sense of agency was better than that for goal assistance. Moreover, trajectory assistance decreased the error in the depth direction, which was higher for manual control. The results show that trajectory assistance possibly improves the accuracy and maintains a feeling of control that is the same as manual control during teleoperation. In the future, the feeling of control when operators modify the target position and a trajectory that is different from the automatic target will be evaluated.
Hiramatsu Y, Aono T, Nishio M (2002) Disaster restoration work for the eruption of Mt Usuzan using an unmanned construction system. Adv Robot 16(6):505–508
Yamamoto H, Moteki M, Shao H, Ootuki T, Kanazawa H, Tanaka Y (2009) “Basic technology toward autonomous hydraulic excavator”. 26th international symposium on automation and robotics in construction (ISARC), pp 288–295
Moteki M, Fujino K, Nishiyama A (2013) Research on operator’s mastery of unmanned construction. Proceedings of the 30th international symposium on automation and robotics in construction and mining (ISARC), pp 540–547
Furuya H, Kuriu N, Shimizu C (2012) Development of next generation remote-controlled machinery system—Remote operation using the apparatus and experience 3D images—(in Japanese). Proceedings of the 13th symposium on construction robotics in Japan, pp 109–116
Tanimoto T, Fukano R, Shinohara K, Kurashiki K, Kondo D, Yoshinada H (2016) Research on superimposed terrain model for teleoperation work efficiency. J Robot Mechatron 28(2):173–184
Shimano Y, Kami Y, Shimokaze K (2014) Development of PC210LCi-10/PC200i-10 machine control hydraulic excavator (in Japanese). Komatsu Tech Rep 60(167):2–7
Kubo Y, Igarashi H, Suzuki S, Kakikura M (2006) Tele-operation support for legged robot equipped with manipulator (in Japanese). The robotics and mechatronics conference 2006, 2P1–D28
Igarashi H (2008) Human adaptive calibration for machine operation without awareness. J Robot Mechatron 20(4):595–601
Igarashi H, Takeya A, Harashima E, Kakikura M (2006) Human adaptive assist planning for teleoperation. The 32nd annual conference of the IEEE industrial electronics society, pp 4522–4527
Gallagher S (2000) Philosophical conceptions of the self: implications for cognitive sciences. Trends Cognitive Sci 4(1):14–21
Kawabe T (2013) Inferring sense of agency from the quantitative aspect of action outcome. Conscious Cogn 22(2):407–412
Maeda T, Takahata K, Muramatsu T, Okimura T, Koreki A, Imashita S, Mimura M, Kato M (2013) Reduced sense of agency in chronic schizophrenia with predominant negative symptoms. Psychiatry Res 209(3):386–392
TT carried out the design, constructed the system, and drafted the manuscript. KS constructed the system, carried out the evaluation experiment, and analyzed the results. HY conceived the study, participated in its design, and helped to resolve errors. All authors read and approved the final manuscript.
The authors wish to thank Dr. S. Nakamura, Dr. K. Kurashiki, and Dr. D. Kondo for their advice regarding the construction the system.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.