Skip to content

Advertisement

  • Research Article
  • Open Access

Multiple player detection and tracking method using a laser range finder for a robot that plays with human

ROBOMECH Journal20185:25

https://doi.org/10.1186/s40648-018-0122-x

  • Received: 31 March 2018
  • Accepted: 12 September 2018
  • Published:

Abstract

We are developing a robot that can play an outdoor game with children. In realizing such a robot, the person detection and tracking methods play an important role. In this paper, we propose methods for improving person detection and tracking using a laser range finder (LRF). A problem facing player detection using an LRF during play is that detection of a player’s body sometimes fails because of the posture of the player. Thus, we propose two improvements of the conventional person detection algorithm. One is to determine the body boundary when an arm overlaps the body, and the other is to choose the most probable object as the body among the detected objects. Another problem is how to track players who are hidden by other players. We introduce a new algorithm to robustly track a player even when the player is behind other players. We conducted three experiments. In the first experiment, we investigate the performance improvement by the proposed player detection algorithm. The second experiment proves that the proposed player-tracking algorithm can robustly track a player even after the LRF has lost that player. In the last experiment, we conduct the game Darumasan ga Koronda using the developed system and confirm that the system can perform the game without large problems.

Keywords

  • Mobile robot
  • Laser range finder
  • Darumasan ga koronda

Introduction

The research and development of robots that play with humans have been conducted since the late 1980s [13]. Soccer is a popular target of the game for robots, because of the RoboCup soccer project [4]. Since the beginning of the project, many robots that play soccer have been developed [58].

The robots that play with children, especially the ones that play physical games such as sports, need to move around humans to achieve specific tasks. Developing such robots is a large challenge and includes ensuring safety for both human and robot, human-robot communication, understanding of human behavior, etc.

We have developed a robot that plays the Darumasan ga koronda game with children [9, 10]. Darumasan ga koronda is a Japanese children’s traditional game. Several research works have treated this game as a subject of human-computer interaction [11, 12] or health care [1316]. There are two roles in this game: the players and “it”, where the players try to tag “it” while “it” is counting to ten.

Our final goal is to develop a robot that plays Darumasan ga koronda, acting in the two roles mentioned above, with children. In the previous work, we have developed methods for realizing a robot that plays the game as “it”, tracking the players’ movements and determining whether a player is moving when the player should freeze [9]. We applied a human detection based on a laser range finder (LRF) for a human-following method [17] to track the players [10]. This method is robust when a person to be tracked is walking normally. However, in the game, a player sometimes walks in a distinct posture, which results in a failure to detect the human bodies. Moreover, from the viewpoint of “it”, the players often overlap, which makes it difficult to sense the positions and movements of the individual players.

In this paper, we propose two new methods to detect and track players robustly using an LRF. In “The game and the previous works” section, we briefly describe the rules of the game, and we also review the previous works on person detection and tracking. In “Analysis and improvement of person detection” section, we propose an improvement over the person detection method using an LRF. In “Person tracking” section, we propose a multiple-person tracking method that can track players even when the robot loses some of the players’ positions. “Experiment” section describes two experiments to confirm the effectiveness of the proposed player detection and tracking methods. In “Evaluation of the total system” section, we describe the total system that can actually play with players and the results of the experimental games are reported. Finally, we conclude the paper in “Conclusion” section.

The game and the previous works

The “Darumasan ga koronda” game

First, we briefly introduce the game and define the rules of the game [9, 18]. Darumasan ga koronda is a Japanese game that is similar to Red light, green light or Statues as played by children in the United States or the United Kingdom. The game is played by several players and one “it”. First, “it” stands in a field, and the players stand at the far end. Then, “it” turns to the other end (with its back facing the players) and counts to ten. While counting, the players get nearer to “it” to tag. After counting, “it” turns around and looks at the players; at that time, each player must freeze. Otherwise, the player is caught by “it” and makes a chain with “it” by holding hands. When a player tags “it”, the player wins, all the caught players are released, and the game starts again from the beginning. If all players are caught, “it” wins, and another player becomes the next “it.”

Figure 1 shows illustrations of each state of the game.
  1. 1.

    When the game starts, “it” (the red person in Fig. 1) and the other players stand in the distance.

     
  2. 2.

    “It” faces the back and counts to ten by saying “da-ru-ma-sa-n-ga-ko-ro-n-da” (those words have ten syllables). When “it” has its back to the players and is counting, the players approach “it” to tag.

     
  3. 3.

    After counting, “it” faces the players. At this time all of the players must freeze. When “it” finds a player moving, “it” calls that player “out.”

     
  4. 4.

    The player who is called “out” is caught by “it” and makes a chain by holding hands with “it.” If all players are caught, “it” wins, and the game ends.

     
  5. 5.

    If a player tags “it”, the caught players are released.

     
  6. 6.

    When the players are released, “it” says “stop” to stop the players.

     
  7. 7.

    After stopping the players, “it” moves three steps to the nearest player. If “it” tags the player, the player is called “out” and caught by “it.” Then, the game starts again from the beginning.

     
We have implemented and evaluated the basic framework for a robot to play Darumasan ga koronda. Considering safety and ease of evaluation, we have slightly changed the rules of the game. Here are the differences from the ordinary rules:
  • The robot’s role is fixed as “it”, and the robot does not become a player even when it catches all of the players.

  • The robot does not turn around when counting.

  • The players are not caught by the robot even when they are called “out.”

  • After the robot is tagged, the robot calls “stop,” and then moves to the nearest player by a distance of three steps of a typical human.

  • When moving to the nearest player, the robot does not tag the player and instead calls “out” if it approaches within 0.5 m of the player.

The flowchart of the behavior of the robot is shown in Fig. 2.
Fig. 1
Fig. 1

State of the game

Fig. 2
Fig. 2

Flowchart of the robot behavior in the game

To realize a robot that plays the game, we need to develop methods that achieve the following six tasks.
  1. (A)

    Detection of persons.

     
  2. (B)

    Tracking of multiple persons.

     
  3. (C)

    Calculation of the motion of a person.

     
  4. (D)

    Detection and tracking of the nearest person.

     
  5. (E)

    Moving to the nearest person.

     
  6. (F)

    Speech synthesis.

     
Among the six tasks, person detection and tracking play a central role. Thus, we review previous works on person detection, and describe a basic method of person detection for which we propose improvements in this paper.

Conventional person detection methods and their problems

A number of methods have been proposed for the detection and tracking of persons. The sensors are installed either on the robot [19, 20] or in the environment [21]; cameras [19, 21], laser range finders (LRF) [17, 19] or radars [22] are used as sensors.

Considering the content of the game, it is difficult to install the sensors in the environment since the game is played outdoors. Thus, the sensors need to be attached to the robot. Next, there are several choices of sensors, such as cameras, LRFs, rader or ultrasound sensors. The situation of the game requires that (1) the sensor should be robust against lighting conditions because the game is played outdoors, (2) the sensor should be able to measure the distance to players several meters away, and (3) the frame rate of the measurement should be short (less than 0.1 s/frame) so that moving players can be captured. Considering requirement (1), a camera is not suitable for this purpose under various weather conditions. Requirement (2) excludes the ultrasound sensor because it cannot measure an object several meters away. Finally, requirement (3) can be satisfied by any of the camera, LRF and radar; however, camera-based methods, such as stereo camera, require computationally expensive processing. Based on these considerations, we chose an LRF as the sensor.

A number of works have been proposed that use LRFs for person detection. Horiuchi et al. proposed a method to detect human legs using a single LRF for detection and tracking of persons by a small robot [23]. Similar methods that observes a person’s legs were also proposed by Sung and Chung [24], Chung et al. [25], Aguirre et al. [26], and Leigh et al. [27]. Carballo et al. developed a method that uses two LRFs to measure the shoulders and legs of a person [28]. The method proposed by Luo et al. [29] combined measurements from a single camera and an LRF that observes the legs of persons. Hoshino and Morioka [30] proposed a method that combines measurements from an LRF and a Kinect sensor.

These works can be classified into two types: methods that uses a single LRF and measures a person’s legs [2327] and methods that combines measurements from multiple sensors, including multiple LRFs [2830].

Considering that most robots are much smaller than humans, it is reasonable to capture a human’s legs to estimate his/her position. When estimating a person’s position from the positions of his/her legs, a model of legs is assumed. For example, [27] assumes that two legs are always separately observed by the LRF and that each leg can be tracked using a Kalman filter. Other methods, such as [24], exploit a model of the walking pattern of humans to estimate a human’s position robustly. However, as described in the next section, players of the game have a wide variety of postures compared with walking persons, as shown in Fig. 5. Thus, it is difficult to precisely estimate humans from the positions of their legs. Thus, observing players’ waists seems to be more robust than observing their legs. Moreover, we can estimate the direction of the body by observing the waist. Using only one LRF is advantageous over the methods that use multiple sensors from the perspective of a robot’s cost.

Conventional person tracking methods and their problems

Multiple-person tracking is another important task for realizing the playing robot. Here, we review person tracking methods based on observation of persons using an LRF. Most works that detect persons using an LRF also perform person tracking [23, 2527]. These works perform either simply observe people’s positions contiguously [23, 24] or track their positions using a statistical method, such as a Kalman filter [27] or particle filter [26]. These works assume that the persons to be tracked can be continuously observed. However, when playing the game, the players overlap each other from the viewpoint of “it.” This problem can be solved if we install multiple sensors in the environment [21]; however, this solution is difficult to use for our purposes because we want to play the game in any playground. Thus, we need to develop a multiple-person tracking method that can cope with overlapping players.

Person detection using a laser range finder

We chose a person detection method that uses an LRF installed on the robot to observe the waist positions of the persons [17]. The advantage of using an LRF is that it is robust against the lighting condition [31], which is crucial for a robot that moves outdoors. Moreover, observing players’ waists is more robust than observing their legs because it is not affected by unusual gaits and postures.

We have developed a system to achieve the abovementioned tasks. The details of the calculation of motion (item C, for “out” judgment) were described in our previous paper [9]. We first describe the algorithm briefly.

The LRF installed on the robot measures the distance from the LRF to the obstacle at a specific angle counterclockwise. The LRF observes the human body at the waist position (1.0 m higher than the floor level). Compared with methods that observe a persons at the shoulders [19] or legs [24], observing the waist can detect the person more robustly. Figure 3 shows the measurement of distances using an LRF. The triangle in the figure indicates the LRF, and the circles represent the measured points. The LRF measures the rightmost point first and then measures the distance from right to left step by step. Let \(D(\theta )\) be the distance from the LRF to the obstacle at the angle step \(\theta \). Then we calculate the difference in distances at every angle.
$$\begin{aligned} \Delta D(\theta ) = D\left( \theta \right) - D\left( \theta - 1 \right) \end{aligned}$$
(1)
As shown in Fig. 3, \(|\Delta D( \theta )|\) is large at the boundary of the object (red points) and small within the object (blue points). Therefore, we can determine the boundary of the object by choosing the angle \(\theta \) where \(|\Delta D( \theta )|\) exceeds a pre-defined threshold \(D_{\text {th}}\). An observed point at angle \(\theta \) is determined as the rightmost point of an object when
$$\begin{aligned} \Delta D\left( \theta \right) < - D_{\text {th}}, \end{aligned}$$
(2)
and as the leftmost point when
$$\begin{aligned} \Delta D\left( \theta + 1 \right) > D_{\text {th}}. \end{aligned}$$
(3)
After detecting the objects, we classify the objects into persons and non-persons based on their widths. Figure 4 shows the classification. The center point (red point) is the middle point of the leftmost and rightmost points. If the width (distance between the leftmost and rightmost points) of an object is similar to the typical width of a human body, that object is classified as a human. When two or more human-like objects are found in neighboring areas, the leftmost object is chosen as the candidate for the human body.
Fig. 3
Fig. 3

Segmentation of the measurement data [17]

Fig. 4
Fig. 4

Detection of human body

Many human detection methods utilize machine-learning-based classifiers for the final decision of human detection [32, 33]. Compared with those methods, the decision in our method is quite simple and deterministic. Although our method is not optimal from a statistical point of view, the advantage of our system is that it does not require any classifier optimization when installed in a new environment.

Analysis and improvement of person detection

The person detection algorithm explained in the previous section worked robustly when the persons to be tracked were walking normally. However, when we applied this method to the players when playing the game, the algorithm often failed to detect the players or misdetected a player’s arm as the human body. The biggest reason seemed to be that the posture of the persons was different from that of persons when walking normally.

Observation of players’ postures in human–human games

To investigate the postures of the players, we conducted an experiment to observe the actual postures of players. We asked five participants (4 players and one “it”) to play Darumasan ga koronda in a large area (24 m by 12 m, indoors), and recorded the game using three cameras. The game was played six times.

As a result of observing the recorded video, we found that there were four patterns of player posture while moving. Examples of the postures are shown in Fig. 5. The posture in Fig. 5a is a moving form with arms swinging at 30°, which was observed during the usual walk. Figure 5b is a moving form with bending of the elbows at 90°, observed when a player was trotting. Figure 5c shows that a player swings his/her widely apart from his/her body. This posture was observed when the player was running. Figure 5d shows that a player puts his/her hands down, swinging his/her entire body, while moving. When moving with this form, a player often puts his/her hands on his/her thighs.
Fig. 5
Fig. 5

Observed postures of players when moving

We found two major patterns where detection of the human body fails. When a player moves with a posture such as that shown in Fig. 5c, the left and right arms are detected as individual objects, as shown in Fig. 6a. In this case, when the arms are recognized as large objects, they are sometimes misclassified as human bodies. Because the leftmost object is chosen as the candidate for the body, the right arm is recognized as the body, which results in the error of the position of the person’s center point. Another case occurs when the player’s posture is similar to that shown in Fig. 5a. In this case, one or two arms overlap with the body, which causes a failure in the detection of the body boundary. As shown in Fig. 6a, when the right arm overlaps the body, the leftmost point of the body (at angle \(\theta \), shown in the red line) is not detected, because \(\Delta D(\theta +1)<0\) (the point at \(\theta +1\) is nearer than that at \(\theta \)); thus, as shown in Fig. 6b, when the right arm overlaps the body, \(\Delta D(\theta +1)\) becomes negative (the point at \(\theta +1\) is nearer than that at \(\theta \)), and the condition shown in formula (3) is not satisfied. Thus, the leftmost point of the body (at angle \(\theta \), shown in the red line) is not detected. If \(\Delta D(\theta +1)\) is small enough to satisfy formula (2), the rightmost point of the arm (the green point in Fig. 6b) can be detected.

Thus, we need to develop a player detection method to detect the bodies of the players when the arms are separately detected as objects, as well as a method to detect the body boundary when the arms overlap the body.

Robust detection of the boundary points of the body

The observation of human-human play revealed that we need to focus on improving the processing methods of (A) and (B) described in “The “Darumasan ga koronda” game” section. Specifically, we need to develop the following two methods: detection of a player’s body when the arms are separately detected as objects, and determination of the player’s body when the arms overlap the body.

First, we solve the problem shown in Fig. 6b. As shown in Fig. 7a, when the right arm overlaps the body, detection of the body boundary points fails. Thus, we apply the rules to determine the “temporal endpoint.”
Fig. 6
Fig. 6

Problems of person detection

Fig. 7
Fig. 7

Determination of temporary endpoint

Let us define \(P\left( \theta \right) \) as the surface point of angle step \(\theta \), and let the predicates RM(P) and LM(P) denote that a point P is the rightmost or leftmost point of a certain object, respectively. Let a predicate OBJ(P) denote that P is a point on the surface of an object. Then, the rules are described as follows.
  1. 1.

    If \(RM\left( P\left( \theta \right) \right) \) and \(OBJ(P\left( \theta - 1 \right) )\), then \(P(\theta - 1)\) becomes the temporal endpoint (the leftmost point of the object).

     
  2. 2.

    If \(LM(P\left( \theta \right) )\) and \(OBJ(P\left( \theta + 1 \right) )\), then \(P(\theta + 1)\) becomes the temporal endpoint (the rightmost point of the object).

     
An example of applying these rules is shown in Fig. 7b. Here, the point just right of the rightmost point of the right arm is determined as the temporal endpoint. By determining the temporal endpoint, we can calculate the width and center of the body.

Determination of the best candidate for the human body

The problem shown in Fig. 6a occurs when the left and right arms are determined to be bodies. We can avoid this problem using the fact that the body is usually wider than the arms. Therefore, when two or more candidates of human bodies are detected, we compare the widths of the objects and choose the widest object as the final candidate for the body. Figure 8 shows an example of the method. In this example, three objects are detected, and their widths are \(W_1, W_2\) and \(W_3\). Then, we compare the widths of the objects, and the widest one (in this example, the center object with width \(W_2\)) is chosen.
Fig. 8
Fig. 8

Determination of the body

Person tracking

In this section, we propose a multiple-player tracking method. As explained in section 2.3, the conventional person tracking methods do not cope with the occlusion of players, which is inevitable for the playing robot. Thus, we developed a simple method for tracking the players who have become lost by being hidden by other players. In this section, we first describe the basic person detection method and then describe a method to track the lost players again.

The basic person tracking method

As explained in the previous section, the person detection is performed for each LRF scan. After detecting the human bodies, we need to track the players and ignore those who are not the players. To do this, we employ a simple method for multiple-person tracking.

At first, we assume that there are only players in the pre-defined playing area. Figure 9 shows the initial positions of the players. We define a rectangular area (shown as red rectangles in the figure) around the position of a player. Let the position of a player p at time t be \((x_p(t),y_p(t))\). Then the area of the player p is defined as
$$\begin{aligned} S_p(t) =\; & \{(x,y): x_p(t)-T_{x1} \le x \le x_p(t)+T_{x2} \text { and }\nonumber \\& y_p(t)-T_{y1} \le y \le y_p(t)+T_{y2}\} \end{aligned}$$
(4)
Here, \(T_{x1}, T_{x2}, T_{y1}\) and \(T_{y2}\) are the thresholds. When the position of a player q at time \(t+1\) is obtained as \((x_q(t+1),y_q(t+1)) \in S_p(t)\), we regard player q as identical to player p in the previous time t. In the experiment described later, we set \(T_{x1}=T_{x2}=300\) mm, \(T_{y1}=600\) mm and \(T_{y2}=300\) mm. \(T_{ y1}\) was set to be larger than \(T_{y2}\) considering that a player is moving toward the robot.
Fig. 9
Fig. 9

Tracking three persons

Tracking of lost players

This tracking algorithm works when the robot does not lose the person to be tracked. However, it fails when a player is hidden by another player. In a case shown in Fig. 10, player A goes behind player B, and thus player A is not detected (and not tracked).
Fig. 10
Fig. 10

An example where tracking fails when a player goes behind another player

In this case, the previous position of player A should be kept in the tracking system. When the system loses player A at time t, the position is assumed to be the same as the position at time \(t-1\).
$$\begin{aligned} \left(x_A(t),y_A(t)\right) = \left(x_A(t-1),y_A(t-1)\right) \end{aligned}$$
(5)
If a person is detected at time \(t^\prime >t\) and enters into \(S_A(t^\prime -1)\), then the person is regarded as player A and tracked again. Figure 11 shows the tracking algorithm. Here, d(pq) is the distance between the two persons, i.e.,
$$\begin{aligned} d(p,q)=\sqrt{(x_p - x_q)^2+(y_p-y_q)^2} \end{aligned}$$
(6)
Figure 12 is another example for which the tracking fails. In this case, the lost player A becomes visible again, but player A’s new position is outside of \(S_A(t-1)\), and thus player A is not identified as the same player as the one who was observed before. To address this case, we introduce the following rule:

If a player is known to be lost and another person exists near the last position where the lost person was observed, the newly observed person is regarded as the lost person.

When a player p is lost, his/her position does not move at all,
$$\begin{aligned} (x_p(t-1),y_p(t-1))=(x_p(t),y_p(t)). \end{aligned}$$
(7)
Thus, we use this condition to determine whether the player is lost or not. Figure 13 shows the revised algorithm. In this algorithm, the detected persons are associated with the players based on the threshold, and the lost players are searched for among those who are not associated with any players.
Fig. 11
Fig. 11

The person tracking algorithm 1

Fig. 12
Fig. 12

An example of tracking failure after the re-detection of the player A

Fig. 13
Fig. 13

The person tracking algorithm 2

Note that this algorithm works when only the players are in the area observed by the LRF; otherwise, this algorithm incorrectly identifies non-players as the players.

Experiment

We conducted three experiments. The first experiment is to examine if the proposed human detection method can address the various postures shown in Fig. 5. The second experiment is to test the tracking performance when players overlap each other. The last one is the total experiment of playing Darumasan ga Koronda.

The robots used in the experiment

Figure 14 shows the robot system used in the experiment. The mobile robot is the same as that used in the previous work [9]. Table 1 shows the specifications of the mobile robot, and Table 2 shows those of the LRF installed on the robot. The LRF is installed at a height of 1000 mm on the robot. To play Darumasan ga koronda, the robot (playing the role of “it”) first needs to face the players and then turn around. However, it is not desirable for a large robot to turn around from a safety point of view. Therefore, we installed a robot avatar [34] on the mobile robot base. The robot avatar is a small robot for communication that can make gestures. By letting the robot avatar turn and make gestures, we can make the robot safer.
Fig. 14
Fig. 14

The mobile robot (Carry PM3)

Table 1

Specifications of the mobile robot

Size (WDH) mm

395 × 510 × 1375

Weight (kg)

8.2

Movement speed (mm/s)

800

Table 2

Specifications of the LRF(URG-04LX-UG01)

Measurement distance (m)

5.6

Measurement range (°)

240

Scan period (s/scan)

0.1

Angular resolution (°)

Approximately 0.36

“out”, the robot avatar points to the player and utters “out” using a speech synthesizer. Figure 15 shows the robot avatar. Figure 15a is the front view of the avatar, and Fig. 15b shows the degrees of freedom of the avatar. When the robot calls “out”, the robot avatar points to the player and utters “out” using a speech synthesizer.
Fig. 15
Fig. 15

The robot avatar

Experiment I

We first investigated the effect of the proposed methods for person detection. Here, we denote the method proposed in “Robust detection of the boundary points of the body” section as “method A” (determination of the endpoint when an arm overlaps the body) and that in “Determination of the best candidate for the human body” section as “method B” (selection of the best candidate for the human body).

Figure 16 shows the environment of the experiment. We prepared three walking paths (A-B, C-D, E-F). We marked every 900 mm on the path so that the participant stepped on the marks when walking. The participant was recorded using two video cameras from different angles, as shown in Fig. 16.
Fig. 16
Fig. 16

Experimental environment

A participant walked each of the three paths ten times, five times of which were analyzed using method A, and the other five times were not (thus 30 times in total). The walking speed was 1.8 m/s. We calculated the body center coordinates with and without method B for both data.

Figure 17 shows examples of the experimental results. We tested four conditions (with/without method A and with/without method B). Figure 17a shows the results without method A, and Fig. 17b shows those with method A. The tick marks of “\(\times \)” show the results without method B, and the “\(+\)” marks show those with method B. At most points, those marks overlap and look like “✳”. In the results of Fig. 17a, the target person was lost at approximately \(Y=2000\) mm with and without method B. It could be improved using method A (Fig. 17b). Moreover, looking at Fig. 17b, method A without method B (“\(\times \)” marks) misdetected the arm as a body (“Moved point” in the figure), which was improved by applying method B (“\(+\)” marks).
Fig. 17
Fig. 17

Trajectory of the center points of the target person

The misdetection rates are shown in Table 3. These values are the ratios of misdetected or lost points of the body center. In this experiment, the participant moved with a fixed velocity (1.8 m/s), and the LRF measured the position of the participant ten times a second. Thus, we assumed that the velocity of the participant was exactly constant and compared the assumed center point and the measured point. When the assumed and measured points differed by more than 120 mm, we regarded the point as misdetected. Figure 18 shows the measurement and decision of misdetection. From Table 3, it is obvious that the misdetection rate can be reduced drastically by applying methods A and B. For the three paths (A-B, C-D, E-F) we did not observe any significant difference in the misdetection rates.
Table 3

Misdetection rates of person detection for different paths (%)

Method A

Method B

A-B

C-D

E-F

Total

No

No

1.1

0.70

0.71

0.84

No

Yes

0.0

0.70

0.71

0.47

Yes

No

1.1

3.2

0.69

1.7

Yes

Yes

0.14

0.14

0.0

0.09

Fig. 18
Fig. 18

Decision of misdetection of a measured person’s position

In addition, we investigated the effect of walking speed. In this experiment, only the A-B path was used, and the participant walked at three speeds (0.7 m/s, 1.4 m/s, and 1.8 m/s), three times for each speed condition. In this experiment, method A was always applied, and we compared the results with and without method B. The result is shown in Table 4. Before applying method B, detection errors were observed at rates of 1.4 to 4.8 %, while no misdetection occurred when using method B.
Table 4

Misdetection rates of person detection for different speeds (%)

Speed (m/s)

Without B

With B

0.7

4.8

0.0

1.4

4.2

0.0

1.8

1.4

0.0

Next, we investigated the effect of walking form. As shown in Fig. 5, there are four types of walking forms when playing the game. Thus, we asked a participant to walk in the four walking forms shown in Fig. 19.

One participant walked in three forms (as shown in Fig. 19a–c) at 1.8 m/s and in the form of Fig. 19d at 0.7 m/s. The reason why the velocity of form (d) was different is that it was difficult to move quickly in form (d). The trial for each form was iterated three times.
Fig. 19
Fig. 19

Four forms examined in the experiment

In this experiment, the proposed method A was always applied, and we compared the results with and without method B. Examples of the experimental results are shown in Fig. 20. We can see that applying method B improved the estimated trajectory for forms (a), (b) and (c). For form (d), because the result without method B was good enough, applying method B did not improve the result.
Fig. 20
Fig. 20

Estimated trajectories for different walking forms

Table 5 shows the total results. Although we observed 0.69% detection errors for Form (c) even when using method B, we could reduce the misdetection of the players using method B. We did not observe any detection errors for form (d).
Table 5

Misdetection rates of person detection for different forms (%)

Form

Without B

With B

(a)

1.4

0.0

(b)

1.2

0.0

(c)

2.1

0.69

(d)

0.0

0.0

Experiment II

In the next experiment, three players moved according to various patterns, and we tested whether the system could track all of the players even when a player was hidden by other players. Figure 21 shows the environment in which the experiment was carried out. As shown in the figure, there were twenty-one position marks in the playing field. Three players moved on the marks synchronously. We prepared twenty motion patterns, as shown in Table 6. At first, players A, B and C stood at the positions 1, 3 and 5, respectively. Then the players moved according to the patterns. The patterns shown in Table 6 were designed so that at least one player was hidden by another player or two or more players stood at the same position to cause misdetection of the players. In Table 6, the bold letters indicate that the players at that position were hidden by other players, and the italic letters indicates that two or more players stood at the same position.
Fig. 21
Fig. 21

Experimental environment

Table 6

Motion patterns

Pattern

Player

Position

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  1

1

B

3\(\rightarrow \)  13\(\rightarrow \)  13

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  6

2

B

3\(\rightarrow \)  13\(\rightarrow \)  13\(\rightarrow \)  13

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  13\(\rightarrow \)  11

3

B

3\(\rightarrow \)  3 \(\rightarrow \)  3

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  13\(\rightarrow \)  13\(\rightarrow \)  11

4

B

3\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1

5

B

3\(\rightarrow \)  3 \(\rightarrow \)  3

 

C

5\(\rightarrow \)  13\(\rightarrow \)  15

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1\(\rightarrow \)  1

6

B

3\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  13\(\rightarrow \)  13\(\rightarrow \)  15

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1

7

B

3\(\rightarrow \)  13\(\rightarrow \)  13

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  5

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1\(\rightarrow \)  1

8

B

3\(\rightarrow \)  13\(\rightarrow \)  13\(\rightarrow \)  13

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  10

 

A

1\(\rightarrow \)  2 \(\rightarrow \)  1

9

B

3\(\rightarrow \)  2 \(\rightarrow \)  3

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  2 \(\rightarrow \)  7 \(\rightarrow \)  6

10

B

3\(\rightarrow \)  2 \(\rightarrow \)  7 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  5\(\rightarrow \)  5\(\rightarrow \)  5

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1

11

B

3\(\rightarrow \)  4 \(\rightarrow \)  3

 

C

5\(\rightarrow \)  4 \(\rightarrow \)  5

 

A

1\(\rightarrow \)  1\(\rightarrow \)  1\(\rightarrow \)  1

12

B

3\(\rightarrow \)  4 \(\rightarrow \)  9 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  4 \(\rightarrow \)  9 \(\rightarrow \)  10

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  1

13

B

3\(\rightarrow \)  8 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  17\(\rightarrow \)  18

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  20 \(\rightarrow \)  19

14

B

3\(\rightarrow \)  8 \(\rightarrow \)  13 \(\rightarrow \)  13

 

C

5\(\rightarrow \)  17\(\rightarrow \)  17\(\rightarrow \)  18

 

A

1\(\rightarrow \)  17\(\rightarrow \)  16

15

B

3\(\rightarrow \)  8 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  5

 

A

1\(\rightarrow \)  17\(\rightarrow \)  17\(\rightarrow \)  16

16

B

3\(\rightarrow \)  8 \(\rightarrow \)  13 \(\rightarrow \)  13

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  20 \(\rightarrow \)  21

 

A

1\(\rightarrow \)  8 \(\rightarrow \)  6

17

B

3\(\rightarrow \)  17\(\rightarrow \)  17

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  5

 

A

1\(\rightarrow \)  8 \(\rightarrow \)  13 \(\rightarrow \)  11

18

B

3\(\rightarrow \)  17\(\rightarrow \)  17\(\rightarrow \)  17

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  20 \(\rightarrow \)  21

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  1

19

B

3\(\rightarrow \)  3 \(\rightarrow \)  3

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  5

 

A

1\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  6

20

B

3\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  8

 

C

5\(\rightarrow \)  3 \(\rightarrow \)  8 \(\rightarrow \)  10

As a result, all players were correctly detected and tracked at the final position of all patterns. As examples, images of the players, as well as screenshots of the player detection system for patterns 8 and 20 are shown in Figs. 22 and 23, respectively. The video clips of those two motions are provided as Additional files 1 and 2.
Fig. 22
Fig. 22

Result of Pattern 8

Fig. 23
Fig. 23

Result of Pattern 20

In Fig. 22, player B moved forward, and player C moved behind player B (Fig. 22b, c). At this time, the system lost player C, which can be confirmed in Fig. 22f in which only the center point (the pink circle) is displayed in the green rectangle and the body was not detected. As player C stepped aside (Fig. 22h, i), the system found player C and tracked him again (Fig. 22l).

In Fig. 23, the three players stood at the same position (Fig. 23c) and moved forward (Fig. 23h). At this time, the system could not recognize the players as individual persons (Fig. 23k). When players A and C stepped aside (Fig. 23i), the system detected all of the players again (Fig. 23l).

Evaluation of the total system

Finally, we investigated the total system by actually playing the game (see the attached video). Two players participated in the game together. The environment of the game was the same as that shown in Fig. 16. The game was recorded by two video cameras (Video 1 and Video 3). The two players started from the positions A and E, respectively. Five sets of play were examined, with each set ending when either a player tagged the robot or the two players were called “out”. When a player tagged the robot, the robot called “stop” and tried to get nearer to the nearest player. If the robot could arrive within 4.095 m of the nearest player (the typical length of three steps by a human player [9]), “it” won; otherwise, the players won.

As a result, “da-ru-ma-sa-n-ga-ko-ro-n-da” was called 18 times (3.6 times/set). Both players were not called “out” in all sets. In the first and second sets, “it” won by moving to the nearest player. In the third and fourth sets, the nearest player moved outside of the observation range of the LRF (5.6 m), and the players won. In the fifth set, the players won by moving outside of the robot’s moving area (4.095 m).

We confirmed that the robot avatar properly pointed to the appropriate player when calling “out”. In the experiment, the robot could only judge those players who were within the observation range of the LRF. Further judgment will be made available by switching the LRF with one with a longer measurement range (such as the UTM-30LX).

Figure 24 shows pictures of the playing experiment recorded by two videos for the fourth set. The new system could detect and track the players even when the players raised their arms, a situation in which the previous system failed to track the players’ bodies.
Fig. 24
Fig. 24

Evaluation experiment of the total system

Conclusion

In this paper, we proposed methods to improve the player detection and tracking based on an LRF to realize a robot that plays a game with humans. For the player detection, the previous method had limitations in that the target player was lost when his/her arms were detected as a body or when his/her arms overlapped his/her body. The proposed method first determines the leftmost and the rightmost points, even when the arms overlap with the body, and then the body is determined based on the width of the object within the observation area. Moreover, we proposed an improved player-tracking algorithm that can robustly discover and track the players who were once hidden by other players.

We conducted experiments to confirm the proposed method’s ability to detect the target persons robustly. Based on the four postures of the players, we confirmed that the proposed method hardly lost or misdetected the players regardless of the posture. We also conducted an evaluation experiment of the proposed tracking method that showed that the proposed method worked under various motion patterns.

Finally, we conducted an actual game with two human players in which the robot successfully tracked the players and conducted its role throughout the entire game.

The person detection and tracking method developed in this paper can contribute to not only the Darumasan ga Koronda game but also other applications. For example, the robust person tracking method that can re-discover the target person will contribute to the realization of a robot that can follow a person [35], which can be applied to a robotic hand cart [36] or a smart wheelchair [27]. Moreover, this technology can also be applied to a guidance robot [37, 38] that guides guests to a destination. When the robot is guiding the guest, the robot may lose the guest when the guest is hidden by other people. Using the proposed method, the guidance robot can track a guest even when the robot loses the guest. Because the LRF is not affected by lighting conditions, the proposed method is also effective for realizing a patrol robot [3941] that works at night. Finally, while this work aims to realize a robot that plays Darumasan ga Koronda, the proposed technology of person detection and tracking can be used for other games, such as the Tag [42, 43].

In future works, we will extend the robot system so that the robot can act as a player (rather than “it”), changing the role of the player and “it.” The experiment of the game was conducted indoors, but the game should be played outdoors.

Additionally, we need to consider the safety of the players (since the players may be children), as well as the safety of the robot (since children sometimes tend to harm robots) [44].

Declarations

Authors' contributions

YN prepared the research and carried out all experiments. YH initiate the research, designed the experiment, and wrote the draft paper. AI advised the design of the experiment and analysis, and finish the final paper. All authors read and approved the final manuscript.

Acknowledgements

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Ethics approval and consent to participate

Not applicable.

Funding

Part of this work was supported by JSPS Kakenhi JP16K00363

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
Osaka Institute of Technology, 1-45, Chayamachi, Kita-ku, Osaka 530-8568, Japan
(2)
Tohoku University, 6-6-5 Aramaki aza Aoba, Aoba-ku, Sendai 980-8579, Japan

References

  1. Faessler H, Buffinton K, Nielsen E (1988) Design for high speed robot. Skilled in the play of ping-pong. In: Proceedings of international symposium on industrial robots. pp 221–232Google Scholar
  2. Hirota K, Arai Y, Hachisu S (1989) Fuzzy controlled robot arm playing two-dimensional ping-pong game. Fuzzy Sets Syst 32(2):149–159View ArticleGoogle Scholar
  3. Andersson RL (1989) Dynamic sensing in a ping-pong playing robot. IEEE Trans Robot Autom 5(6):728–739View ArticleGoogle Scholar
  4. Kitano H, Asada M, Kuniyoshi Y, Noda I, Osawa E (1997) RoboCup: the Robot World Cup Initiative. In: Proc. 1st International Conference on Autonomous Agents. pp 340–347Google Scholar
  5. Park SW, Kim JH, Kim EH, Oh JH (1997) Development of a multi-agent system for robot soccer game. In: Proceedings of International Conference on Robotics and Automation, vol. 1. pp 626–631Google Scholar
  6. Karol A, Nebel B, Stanton C, Williams MA (2003) Case based game play in the robocup four-legged league part I the theoretical model. In: Robot Soccer World Cup. Springer. pp 739–747Google Scholar
  7. Behnke S, Schreiber M, Stuckler J, Renner R, Strasdat H (2006) See, walk, and kick: humanoid robots start to play soccer. In: Proceedings of IEEE-RAS international conference on humanoid robots. pp 497–503Google Scholar
  8. Duan Y, Liu Q, Xu X (2007) Application of reinforcement learning in robot soccer. Eng Appl Artif Intell 20(7):936–950View ArticleGoogle Scholar
  9. Sakai K, Hiroi Y, Ito A (2015) Playing with a robot: realization of “Red Light, Green Light” using a laser range finder. In: Proceedings of international conference on robot, vision and signal processing (RVSP)Google Scholar
  10. Nakamori Y, Hiroi Y, Ito A (2017) Enhancement of person detection and tracking for a robot that plays with human. In: Proceedings of IEEE/SICE international symposium on system integration. pp 494–499Google Scholar
  11. Baba K, Oga M, Nodera Y, Tanaka K, Imai H, Kitazawa J, et al (2004) OHAYO players. In: Proceedings of ACM SIGGRAPHGoogle Scholar
  12. Terada K, Ito A (2010) Can a robot deceive humans? In: Proceedings of 5th ACM/IEEE international conference on human–robot interaction (HRI), Osaka. pp 191–192Google Scholar
  13. Okano T, Kitakoshi D, Suzuki M (2003) A preliminary study on preventive care system based on game playing with communication robots. In: Proceedings of conference on technologies and applications of artificial intelligence (TAAI), Taipei. pp 84–89Google Scholar
  14. Nakamura T, Munekata N, Nakamura F, Ono T, Matsubara H (2011) Universal game based on traditional children’s outdoor games. In: Proceedings of 10th international conference on entertainment computing. pp 59–64Google Scholar
  15. Kitakoshi D, Okano T, Suzuki M (2017) An empirical study on evaluating basic characteristics and adaptability to users of a preventive care system with learning communication robots. Soft Comput 21(2):331–351View ArticleGoogle Scholar
  16. Kitakoshi D, Suzuki M, Noguchi S, Suzuki K (2017) A study on fall-prevention system for older adults with communication robot. In: Proceedings of IEEE international conference on biometrics and kansei engineering (ICBAKE). pp 153–157Google Scholar
  17. Hiroi Y, Matsunaka S, Ito A (2012) A mobile robot system with semi-autonomous navigation using simple and robust person following behavior. J Man Mach Technol 1(1):44–62View ArticleGoogle Scholar
  18. Nakabayashi H (1996) Idea Book 2—teaching Japanese for juniors. KYOZAI-L.O.T.E, Teaching Aids, HealesvilleGoogle Scholar
  19. Kobayashi Y, Kuno Y (2010) People tracking using integrated sensors for human robot interaction. In: Proceedings of IEEE international conference on industrial technology. pp 1597–1602Google Scholar
  20. Ardiyanto I, Miura J (2014) Partial least squares-based human upper body orientation estimation with combined detection and tracking. Image Vision Comput 32(11):904–915View ArticleGoogle Scholar
  21. Brščić D, Kanda T, Ikeda T, Miyashita T (2014) Person tracking in large public spaces using 3D range sensors. IEEE Trans Hum Mach Syst 43(6):522–534View ArticleGoogle Scholar
  22. Misu K, Miura J (2015) Specific person tracking using 3D LIDAR and ESPAR antenna for mobile service robots. Adv Robot 29(22):1483–1495View ArticleGoogle Scholar
  23. Horiuchi T, Thompson S, Kagami S, Ehara Y (2007) Pedestrian tracking from a mobile robot using a laser range finder. In: Proceedings of IEEE international conference on systems, man and cybernetics. pp 931–936Google Scholar
  24. Sung Y, Chung W (2011) Human tracking of a mobile robot with an onboard LRF (Laser Range Finder) using human walking motion analysis. In: Proceedings of 8th international conference on ubiquitous robots and ambient intelligence (URAI). pp 366–370Google Scholar
  25. Chung W, Kim H, Yoo Y, Moon CB, Park J (2012) The detection and following of human legs through inductive approaches for a mobile robot with a single laser range finder. IEEE Trans Ind Electron 59(8):3156–3166View ArticleGoogle Scholar
  26. Aguirre E, Garcia-Silvente M, Plata J (2014) Leg detection and tracking for a mobile robot and based on a laser device, supervised learning and particle filtering. In: Armada MA, Sanfeliu A, Ferre M (eds) Proceedings of first iberian robotics conference: advances in robotics (ROBOT2013), vol 1. Springer, Berlin, pp 433–440View ArticleGoogle Scholar
  27. Leigh A, Pineau J, Olmedo N, Zhang H (2015) Person tracking and following with 2D laser scanners. In: Proceedings of IEEE international conference on robotics and automation (ICRA). pp 726–733Google Scholar
  28. Carballo A, Ohya A, Yuta S (2009) Multiple people detection from a mobile robot using double layered laser range finders. In: Proceedings of IEEE ICRA 2009 workshop on people detection and tracking. pp 94–100Google Scholar
  29. Luo RC, Chang NW, Lin SC, Wu SC (2009) Human tracking and following using sensor fusion approach for mobile assistive companion robot. In: Proceedings of 35th annual conference of IEEE industrial electronics. pp 2235–2240Google Scholar
  30. Hoshino F, Morioka K (2011) Human following robot based on control of particle distribution with integrated range sensors. In: Proceedings of IEEE/SICE international symposium on system integration (SII). pp 212–217Google Scholar
  31. Atray PK, Hossain MA (2010) Multimodal fusion for multimedia analysis: a survey. Multimed Syst 16:345–379View ArticleGoogle Scholar
  32. Mozos OM, Kurazume R, Hasegawa T (2010) Multi-part people detection using 2D range data. Int J Soc Robot 2(1):31–40View ArticleGoogle Scholar
  33. Munaro M, Menegatti E (2014) Fast RGB-D people tracking for service robots. Auton Robots 37(3):227–242View ArticleGoogle Scholar
  34. Hiroi Y, Ito A, Nakano E (2009) Evaluation of robot-avatar-based user-familiarity improvement for elderly people. KANSEI Eng Int 8(1):59–66View ArticleGoogle Scholar
  35. Hiroi Y, Ito A (2013) ASAHI: OK for failure: a robot for supporting daily life, equipped with a robot avatar. In: Proceedings of ACM/IEEE international conference on human–robot interaction (HRI). pp 141–142Google Scholar
  36. Tsuda N, Harimoto S, Saitoh T, Konishi R (2009) Mobile robot with following function and autonomous return function. In: Proceedings of ICCAS-SICE IEEE. pp 635–640Google Scholar
  37. Fujiwara Y, Hiroi Y, Tanaka Y, Ito A (2015) Development of a mobile robot moving on a handrail—control for preceding a person keeping a distance. In: Proceedings of 24th IEEE international symposium on robot and human interactive communication (RO-MAN). IEEE. pp 413–418Google Scholar
  38. Triebel R, Arras K, Alami R, Beyer L, Breuers S, Chatila R et al (2016) Spencer: a socially aware service robot for passenger guidance and help in busy airports. Field and service robotics, Springer, Berlin, pp 607–622Google Scholar
  39. Kim K, Bae S, Huh K (2010) Intelligent surveillance and security robot systems. In: Proceedings of IEEE workshop on advanced robotics and its social impacts. pp 70–73Google Scholar
  40. Fan SH, Xia DX (2011) Study on the navigation of patrol robot of transformer substation based on color vision and RFID. Adv Mater Res 268:1259–1264View ArticleGoogle Scholar
  41. Tseng CC, Lin CL, Shih BY, Chen CY (2013) SIP-enabled surveillance patrol robot. Robot Comput Integr Manuf 29(2):394–399View ArticleGoogle Scholar
  42. Moreno A, van Delden R, Poppe R, Reidsma D, Heylen D (2016) Augmenting playspaces to enhance the game experience: a tag game case study. Entertain Comput 16:67–79View ArticleGoogle Scholar
  43. Agrawal G, Karlapalem K (2016) Wheeled robots playing chain catch: strategies and evaluation. In: Proceedings of international conference on autonomous agents and multiagent systems. pp 1283–1284Google Scholar
  44. Brščić D, Kidokoro H, Suehiro Y, Kanda T (2015) Escaping from children’s abuse of social robots. In: Proceedings of ACM/IEEE international conference on human–robot interaction (HRI). pp 59–66Google Scholar

Copyright

© The Author(s) 2018

Advertisement