Development of test method for evaluation of UAS mobility capability in GNSS-denied environment

Abstract

This paper introduces the development of test methods for capability evaluation of Unmanned Aircraft Systems (UASs) in Global Navigation Satellite System (GNSS)-denied environments. The purpose of this development is to facilitate growth in the UAS industry. We discuss the test method’s approach and what UAS’s capability evaluation is essential for UAS use in GNSS-denied environments. As a result, we decided to adopt an approach that a test method evaluates a capability to perform a single simple task. In addition, these test methods should be along with the requirement of UAS manufacturers and users. Thus, first, we develop test methods to assess mobility in narrow spaces with obstacles shielding GNSS radio. We repeatedly have demonstrations and discussions of our test method with UAS manufacturers and users from the early stage of this development and collect their opinions for improvement to proceed with the development while building consensus with them. In addition, we evaluate several UASs by the test methods and examine whether our test method makes it possible to show the difference in UAS capability. This paper describes the approach of the test methods, the test methods for mobility in a GNSS-denied environment, the demonstrations and discussions with UAS manufacturers and users, and the results of the UAS’s performance evaluation by the test methods.

Introduction

This paper describes the research for UAS evaluation in GNSS-denied environments. The purpose of this research is to boost UAS development in industry in Japan, and it is a part of the Drones and Robots for Ecologically Sustainable Societies project by the New Energy and Industrial Technology Development Organization [1]. In this research, a GNSS-denied environment is defined as places where UASs cannot obtain positioning data from external systems.

Recently, the UAS industry has been remarkably developing fields such as surveying, mapping, and inspection. In the future, UASs are expected to expand in various places and situations and provide many solutions. Such a place and situation includes GNSS-denied environments. For example, a UAS inspection of plant equipment, such as a boiler, reduces inspection costs and increases the plant’s operating ratio. Also, UASs expect the same effect for infrastructure inspections in high places, such as bridges or tunnels. In disaster response, UAS building surveying help for the safety of human responders. To use UAS for such services, the UAS user should understand the UAS performance in the place and situation. However, it is not easy to discuss UAS performance, much less in environments that deny some UAS functions. In addition, it is not necessarily that UAS users are technical experts in UAS. Therefore, for industrial boosting, it is important to develop methods to evaluate UAS performance objectively, in parallel with UAS technological development. Thus, we develop test methods to evaluate the UAS capabilities with the restraint of a GNSS-denied environment. These test methods aim to describe the UAS’s capability in an easy-to-understand form for even people without technical knowledge of UASs. The evaluation results of the test methods make it possible to compare UASs and help discuss which UAS is suitable for their requirements.

We discuss the approach for UAS evaluation in this project and what performance the UAS users require for UASs in a GNSS-denied environment. As the result of this discussion, we focus on evaluating mobility in narrow spaces with obstacles that shield GNSS radio. Specifically, we develop 2 test methods to evaluate the capability for horizontal or vertical control to pass a narrow space; test fields and test procedures. In addition, we conduct demonstrations of our test method with UAS manufacturers and users for opinion exchange and to examine whether our test method makes it possible to show the difference in UAS performance.

Approach for development of UAS evaluation in GNSS-denied environment

In this section, we discuss matters that the test method in this project should be, but first, this section describes a UAS as the object of the test. In this research, a UAS is defined as a system that consists of 2 parts; an issuing command agent and a vehicle to act; fly, move, or capture images, following the issued commands. Such systems can be classified as remotely controlled or autonomous based on the form of command-issuing agents. The evaluation result of the test method should be independent of whether the command-issuing agent is remotely controlled or autonomous because the evaluation for remotely controlled or autonomous should defer to users’ requirements.

In this research, a single test method evaluates not the total of a UAS system but the capability for performing a single simple task. Because a total evaluation of a system varies depending on what the UAS user requires to do and where he/she uses it, a method for total evaluation lacks focus for users. In contrast, evaluation based on the actual performance of a single task is more accessible for even people without technical knowledge to understand how much the UAS can do the task. The evaluation result in this research is described by quantitative metric, for example, the time for the UAS to accomplish the test task or the number of task sub-items achieved, for easy-to-understand which UAS is better for the task. Thus, the test method provides valuable information for users to select a UAS suitable for their requirements.

This approach applies the concept of the Standard Test Method (STM) by the National Institute of Standards and Technology (NIST) [2, 3]. NIST is working to develop a comprehensive set of test methods, such as a test method for mobility or manipulation, to evaluate quantitatively key capabilities of emergency response robots. NIST test methods are developed based on the requirement of emergency responders. In contrast, our project aims to develop a set of test methods for UAS flight in a GNSS-denied environment, according to the requirement of UAS users. In particular, environmental applicability evaluation, such as how small space the UAS can fly in, has a significant value for the UAS industry to provide various services; on the other hand, NIST test methods mainly focus on rescue missions and training for these missions. Therefore, in this research, environmental factors that are dominant for the performance, for example, the space size for movement performance, should be variable in the test method since such factors vary according to UAS technological progress or the actual place and situation. In addition, our test methods should describe its dominant environmental factor in a form that is easy to compare to a specific environment.

Moreover, the test methods should be easy to conduct. The test procedure is simple, and anyone understands it without technical knowledge. In addition, the apparatus used for the test method is easily obtainable, widely used, and low-cost.

In such a development, it is essential to discuss with test method developers, UAS manufacturers, and UAS users and to proceed while building consensus. Therefore, adopting Agile Development [4], we have demonstrations and discussions of our test method under development with UAS manufacturers and users from earlier in this development and repeatedly improve along with the opinions at these discussions.

UAS mobility required in GNSS-denied environment

In general, UASs should go to a destination point and return. Thus, one kind of the essential capabilities of UASs is mobility. For UAS mobility, it is supposed that GNSS-denied environments have something to shield GNSS radio, which becomes an obstacle. In such an environment, the UAS should observe and avoid or withstand contact with the obstacle. This hypothesis is consistent with uses for inspection of infrastructures at high places, factory facilities, and inside of destroyed buildings in disaster for rescue and search, in terms of the scenario to use UAS in GNSS-denied environments. Moreover, some technical research for such a scenario indicates the expectation for a UAS’s industrial expansions. Nikolic et al. [5] develop an autonomous flight system to inspect industrial facilities and conduct flight experiments in a boiler of a thermal power plant. Petrlik et al. [6] is also aimed to develop an autonomous flight system in constrained environments, and they are focusing on use for inspection and assistance in search and rescue operations. Okada et al. [7] focus on the visual bridge inspection that requires seeing a target object close, and they develop a vehicle with a passive rotating spherical shell. Finally, they have actual tests of bridge inspection and achieve to find some cracks at a high place. Tan et al. [8] develop a lightweight and high-resolution imaging system with a rotating camera system for autonomous inspection in a deep tunnel by UAS.

Thus, we develop test methods to evaluate how narrow an area the UAS can pass with obstacles. Specifically, in this work, we develop 2 test methods to evaluate the horizontal and vertical control capability to pass a narrow space. The following section describes the test fields and procedures for these test methods.

Test methods for UAS mobility in narrow space

In this development, first, we discuss the basic structure of the test field and, next, improve it according to the opinions of manufacturers and users. This section describes the test field’s basic structure and the test procedure, and the next section explains the improvement with the demonstration and the discussion with manufacturers and users.

The test method should be conducted under indistinguishable conditions to compare the UAS’s capability. Therefore the reproducibility of a test field is important. Thus, a simple structure is better for a test field. Moreover, a simple test field is also along the approach that test methods are easy to conduct. On the other hand, according to the approach for our development, the test field for narrow space should be able to vary its narrowness as a dominant environmental factor for the performance, for UAS technological progress, or various user requirements. Figure 1 shows the structure and a photo of the test field to evaluate the capability for horizontal control to pass a narrow space. This test field consists of a frame and obstacle boards, and the size of the frame and the board obstacles define the narrowness of the space. In addition, L, M, and N, shown in Fig. 1, specifically describe the narrowness. Furthermore, to pass the test field, the UAS needs to avoid obstacles horizontally due to the board obstacles. Similarly, in the test field of vertical control evaluation, the size of the frame and board obstacles indicate the narrowness of the space. In contrast, to pass this field, the UAS needs to avoid vertically (see Fig. 2).

Fig. 1

Test field for horizontal control. a Shows the structure of the test field. b Shows a picture of an actual test field. c Shows the route to pass the test field

Fig. 2

Test field for vertical control. a Shows the structure of the test field. b Shows a picture of an actual test field. c Shows the route to pass the test field

The test-taker is the operator of the remotely controlled UAS or the administrator of the autonomous UAS and performs the test task under the examiner’s direction. The test task in these test fields is to fly the route, go to the halfway point and return to the starting point shown in Figs. 1c or 2c, 5 times. In addition, each flight in the test has a time limit of 5 min. This time limit is long enough to accomplish the test task. The procedure of one flight for the test examiner is as followings.

• 1. Receives notification from the test-taker that the UAS is ready for the test and confirms that the vehicle is waiting at the starting point or vertically above the starting point.

• 2. Notifies the test-taker of the test start and starts measuring the time.

• 3. Confirms that the vehicle passes the inside of the test field and that the entire vehicle reaches the halfway point.

• 4. Confirms that the vehicle returns at the starting point through the inside of the test field, and the entire vehicle reaches the starting point.

• 5. Records the flight’s success or failure and time.

Finally, these test methods measure the following as the quantitative metric that indicates the UAS’s capability.

• Number of successes for flights

• Time for flight

Improvements according to discussions with UAS manufacturers and users

Table 1 Implementation of demonstration and discussion with manufacturers and users

Fig. 3

Improvement of the test field based on the opinions of manufacturers and users; (a) The primary structure of the test field, (b) Putting colored tape at random on its obstacle wall to make it easy to observe the vehicle movement from camera images, (c) Attaching plastic curtains to prevent autonomous UASs from planning a route where bypass the test fields to outside, (d) Changing plastic curtains to the net due to the unstable influence of the wind caused by the vehicle

The implementation of the demonstrations and discussions is shown in Table 1. The participants are UAS users, manufacturers, and test developers to discuss the test method. In addition, some in other fields interested in test method development, such as marine robot developers, join the discussions. For these demonstrations, to conduct and record a test easily, we prepare a scoring sheet; the example of the vertical control case is shown in the Appendix. The scoring sheet has the entry column of the evaluation metric and the test field configuration because of a description of applicability for an environment. In addition, the scoring sheet also has an illustration and text of the test procedure to make it easy to conduct the test. Then, UAS manufacturers, users, and we, the test method developers, conduct the test method. These demonstrations are conducted in the test building at the Naraha Center for Remote Control Technology Development (Naraha center) of JAEA. After the demonstration each day, we discuss whether the evaluating capability of the test methods is appropriate for UAS flight in a GNSS-denied environment and whether the test method is appropriate for evaluating the capability.

In this way, we discuss based on an actual working test with face-to-face communication with manufacturers and users for efficient collecting opinions. In addition, to effectively take the opinions, this research proceeds by repeating to improve the test methods according to the opinions in each discussion and demonstrate the improved test method in the next. The following of this section introduces the proceedings of improvement with the opinions.

First, there is no opinion that evaluating the capabilities by the test methods is inappropriate or that the basic structure of the test field or the test procedure is inappropriate for the evaluation. In contrast, several opinions indicate that some improvement in the test field implementation is needed. Figure 3 shows the outline of the improvement proceedings, and Fig. 3a is the primary structure.

For the boards of the test fields, some opinions indicate that a uniform color is inappropriate. Uniform color makes it difficult for the command-issuing agent to observe the vehicle’s movement due to the lack of change in camera images when the vehicle is moving. Thus, some kind of print that makes it possible for the command-issuing agent to observe the vehicle’s movement is required. An example of the solution is to paste colored tape shown in Fig. 3b. These colored tapes are set at random because random patterns are easy to observe 3-D information by image processing for autonomous command-issuing agents, and it is easy to apply.

Some manufacturers developing autonomous-type UASs point out that something that obstructs the vehicle at the side of the test fields is needed because autonomous-type UASs can plan a route that bypasses the test field to outside. On the other hand, some manufacturers also say that it is helpful to see the inside of the test field when the vehicle is flying to discuss UAS development. Thus, we tried to set a transparent plastic curtain at the side of the test field, as shown in Fig. 3c. However, a participant in the demonstration and discussion points out that the plastic curtain’s flutter makes the wind’s influence caused by the vehicle unstable. Finally, to meet these opinions, the net is attached at the side of the test field, as shown in Fig. 3d.

Fig. 4

The field for the experiment; This test field is to evaluate the capability for horizontal control and provides the route of 2 m width and 2 m height for the UAS

Table 2 Results of UAS evaluation by the test method for horizontal control

Experiment

This section describes an experimental setup for UAS evaluation by the test methods. This experiment is conducted at the demonstration on Nov.17 and 18, 2021, with the participants as test-taker. This experiment aims to verify 2 things. One is that the test methods can evaluate UAS’s capability to pass a narrow space. The other is that the test result makes it possible to show the difference between UASs on the evaluation metrics. In this experiment, 2 groups, a novice group and an expert group, conduct the test method by remotely controlled type UAS. The test-taker IDs of the operators in the novice group are 1, 2, or 3 in Table 2, and they use a commercial vehicle (vehicle ID is A or B in Table 2). The expert group’s operator (test-taker ID is 4 or 5 in Table 2) has the experience to be engaged in operating a UAS for some works, such as equipment inspections. The vehicle of the expert group is the same type of vehicle that the operator of the expert group uses in his works (vehicle ID is C or D in Table 2). Therefore, the evaluation result is expected to indicate that the expert group is superior to the novice group. This experiment is conducted in the test field to evaluate the capability for horizontal control shown in Fig. 4. In the demonstrations on May 26–27 and Aug. 25–27, some UAS can not go into the 1 m width or height route because their auxiliary control function for obstacle avoidance steers around the route. Thus, this field configuration provides the route of 2 m width and 2 m height to facilitate obtaining experimental subjects for comparing the evaluation metrics. Table 2 shows the result of the UAS evaluation in this experiment.

Result and discussion

This section discusses whether the test methods can describe a difference in UAS capabilities based on the experimental result. The improvement in the test methods is also discussed, according to the obtained opinions through the experiment.

All UAS of the expert group (C-4 and D-5) success all 5 flights. In contrast, A-1 and B-3 in the novice group have 1 failure in 5 times flights. In addition, there is a clear difference between the novice and expert groups in both average and fastest times. The average times of the expert group are 40 s range. These are faster than the fastest time of B-3 of 55 s, the fastest time of the novice group. Thus, the failure of the novice group and the difference in time between the novice group and the expert group indicate that this test method provides information to help to compare UASs.

However, several participants of the demonstration require the discussion of only the vehicle’s capability evaluation, which means the capability excluding the operator’s skill, for comparison remote controlled UASs. On the other hand, excluding the operator, remotely controlled UASs cannot perform a task, and an evaluation is difficult to describe the capability in an easy-to-understand form without actual performance. This problem resolves by noting information on the operator’s skills, for example, the total training time for the last half year. That is, the note of the operator’s skill information makes it possible to compare UASs with similar skill level operators and to discuss the difference of only vehicle capabilities. In addition, the note on the operator’s skill is helpful for UAS users to discuss the practical use that includes operator training required to use the UAS.

Conclusions

This work is the development of test methods to evaluate the capabilities of UASs with the restraint of GNSS-denied environments. The purpose of this development is to facilitate development in the UAS industry. The test methods aim to describe the capability of a UAS with a form that is easy to understand without technical knowledge and to make it possible to compare the capability of UASs. Therefore, one test method evaluates the capability of UASs for a single simple task with actual performance, and an evaluation result is described quantitatively.

We discuss the capability that is required for UASs in GNSS-denied environments. As a result, we decide to develop test methods to evaluate mobility in environments with obstacles because GNSS-denied environments generally have something that shields GNSS radio and becomes an obstacle for the UAS flight. Specifically, we develop 2 test methods (the test fields and the test procedures) to evaluate the horizontal and vertical control capability to pass a narrow space.

These test methods improve with the opinions obtained through the demonstrations and discussions with UAS manufacturers and users. The participants of the demonstrations conduct practically the test method. This result shows that the test method quantitatively describes the UAS’s capability and the difference between UASs.

In future works, we should have more demonstrations for manufacturers and users and verify the effectiveness of the test methods based on more extensive test results. Moreover, this test method needs to be capable of adapting a UAS user-required narrowness. Therefore, we should discuss realistic environments to use UASs with UAS users and improve the test fields to simulate the environments in terms of narrowness.

Appendix

The score sheet that used in the demonstration with manufacturers and users is shown in Fig. 5.

Fig. 5

Score sheet