A measurement system toward to skill analysis of wall painting using a roller brush

Abstract

In this study, we describe a measurement system aiming to skill analysis of wall painting work using a roller brush. Our proposed measurement system mainly comprises an RGB-D sensor and a roller brush with sensors attached. To achieve our requirements in understanding roller operation, we developed an algorithm that is suitable to estimate the roller part pose with high accuracy. We also show a method to generate a swept map that can be used for both visualization and evaluation. In the proof experiment, a dataset for actual painting work was collected using the proposed measurement system. Then, the dataset was analyzed, and the quality of the work was quantitatively evaluated by comparing skilled and unskilled persons.

Introduction

The number of construction workers is decreasing, and human resource development, labor-saving, and automation are being promoted in each stage of construction work; however, such often requires high skill. If we aim to teach one construction work to others or replace part of the work with automated machines, it is first required to clarify the essence of the skill. One of the methods to achieve this is as follows. First, how skilled people move, etc. is measured by sensors. Next, the data obtained as such are analyzed to extract relevant essences that should be called skills. In this series of flows, various issues occur depending on the work content, such as what to measure, how to measure, how to analyze, and how to evaluate.

In this study, we consider finding the skills in the painting of wall surfaces. In particular, we focus on the painting method using a roller brush. Currently, such work is done by craftsmen. However, owing to the aging of craftsmen and decline in the number of applicants for work, there is a concern that there will be a shortage of workers in near future. Skill inheritance is also an issue. Therefore, we aim to extract skills for successful painting from the data obtained by measuring how the painting technician moves. It is hoped that this will improve the efficiency of skill transfer from person to person and lead to painting by automated machines.

In the considered painting work, how a worker operates a roller brush well is one key to achieving proper painting. The reason the roller brush has the main role is that even if the human body movement (HBM) is analyzed, the movement may differ for each person. Meanwhile, the operation of the roller brush has less variation than the HBM, and there is a possibility of finding immutable skills.

The contributions of this study are as follows.

• We organize the matters that should be measured in painting work and points to be noted when making measurements, and construct a system that readily available to operators.

• We develop and verify a method that can stably estimate the pose of a roller brush during painting work. We also show a method for generating a swept map, which is useful for evaluation and visualization after painting.

• We performed experiments to measure actual painting work. The obtained data were analyzed, and the quality of work was quantitatively evaluated by comparing the skilled people’s data with unskilled people’s data.

The remainder of this article is organized as follows. In “Related work” section, we present related work. “Issues and approach” section shows our issues and approach. “Motion and force measurement from the roller part” section explains a method for roller pose estimation, and describes force sensing by an improved roller brush. “Generating swept map” section describes a method for generating a swept map. “Fundamental experiments” and “Experiments and discussion” sections report fundamental experiments, data collection, data analysis, and evaluation. Finally, we conclude this study in “Conclusion” section.

Related work

Studies on skill extraction can be broadly divided into methods that target the physical movements of humans performing work and methods that target the operation of tools used for work. First, we introduce some studies that measure HBMs. Yokoyama et al. [1] extracted skills in painting with a roller brush. They used optical motion capture (OMC) to measure HBMs. Then, velocity and acceleration were calculated from the movement data of the arm holding the roller brush, and these were used as features to compare skilled and unskilled people. Ikeda et al. [2] extracted skills in grooving work with a milling machine and chamfering work with a file. OMC was also employed. Ikemoto et al. [3] extracted skills in car painting work using a spray gun. They measured how the spray gun was moved and clarified that the positional relationship between the spray gun and object to be painted and the speed at which the spray gun is moved differed between skilled and unskilled personnel. As such, OMC has often been used to measure HBMs.

Motion sensors, such as IMU, have also been used for measuring HBMs. Ahmadi et al. [4] extracted the characteristics of a skilled tennis player for tennis serve. They measured HBMs using a gyro sensor. Lai et al. [5] and Okuda et al. [6] found the difference between skilled and unskilled golfers in golf swings. Lai et al. [5] attached IMUs to the arms, pelvis, and upper trunk, whereas Okuda et al. [6] measured reaction force from the floor in addition to OMC. From the measured data, they found the difference between skilled and unskilled golfers in trunk rotation and weight transfer. Ghasemzadeh et al. [7] evaluated the skill level of baseball swing movements from measurement data of multiple IMUs worn on the body. Enokibori et al. [8] found a relationship between the extraction of the sanding skill of metal ingots and the verbalized skill evaluation. IMUs were attached to several body parts, and acceleration and angular velocity were measured to determine the speed of the head and hips. Notably, IMU techniques are simpler than those of OMC; however, high measurement accuracy cannot be guaranteed.

Thus, OMC and IMU have been used in many tasks to measure HBMs. However, HBMs may vary among workers; in that case, it is better to focus on tools to be operated rather than humans. In the abovementioned studies [2, 3], not only HBMs but also tool operations were measured by OMC. Williams et al. [9] also extracted skills for tennis serves, similar to Ahmadi et al. [4]. However, their measurement target was not the HBM but the tennis racket movement. Hayashi et al. [10] analyzed the skill of kitchen knife operation. Markers were attached to the knife, and three-dimensional (3D) movement was measured. Then, a comparison was made between skilled and unskilled people.

When focusing on the operation of tools, it may be difficult to attach artificial markers for OMC due to restrictions such as tool size and work content. In such a case, it is necessary to consider a dedicated measurement method for the tool operation. Irie [11] proposed a method of table tennis racket posture estimation. The estimation was achieved by combining a monocular camera and an IMU. Zhang et al. [12] used a monocular camera and a special cylindrical marker and measured the posture of a small surgical instrument. For our method, artificial markers cannot be used because paint may adhere to the roller. Therefore, we employ another approach to measure the roller posture.

Issues and approach

Problem setting and skills to be extracted

Figure 1 shows a work of painting a vertical wall surface using a roller brush. In this work, after soaking an appropriate amount of paint into the roller brush, the roller is moved up, down, left, and right for applying the paint to the wall surface. By repeating this, the painting work will proceed. We observed how a skilled person paints as a preliminary study. Based on that experience, the skills to be extracted and their reasons are summarized as follows. In this paper, pose means a combined expression of position and orientation.

• Skills to arrange the roller part trajectory, control the roller brush pose, and apply force to the roller part during the movement of reciprocating the roller brush up and down or left and right once. This operation is the basic unit of painting work, and it should be performed in a certain direction and force. However, as serial link manipulators such as the human arm are inherently unsuitable for linear tracking, some pose errors may occur in the human hand. In addition, the amount of paint soaking in the roller part decrease as the painting progresses. There seem to be skills to tolerate these uncertainties. Therefore, it is desirable to know the quality of skilled movement with respect to the aspects mentioned above.

• Skills to decide when and where to paint with the roller brush in the entire work. This is necessary for the paint to be evenly distributed on the wall surface. In the painting work, the roller part is rolled on the wall surface, but the movement of the roller brush at that time has a certain rhythm and does not move completely irregularly. This means that the movement has a certain degree of repetitive nature. Therefore, it is desirable to clarify how the roller brush moves during the entire work.

Fig. 1

Wall painting using a roller brush

Besides, from the fact in the second item, we found that if the work is measured from an appropriate fixed viewpoint, it is unlikely that the roller will be hidden by the worker’s body. Based on the above, the functions required for measurement systems and data analysis are summarized below.

• 1. The pose and movement of the roller on the wall surface can be estimated accurately.

• 2. The force applied to the wall surface from the roller can be estimated.

• 3. It is possible to estimate which part on the wall surface was painted how many times.

Based on the results described above, we develop a measurement system toward the evaluation of the quality of painting work.

Issues

Issues on implementing the abovementioned functions mentioned are as follows.

• 1. The presence of painting material cannot be ignored when estimating the roller pose. For example, to achieve pose estimation, a method for attaching optical markers to the roller part can be considered. However, in painting work, there is a possibility that paint will adhere to the marker. In that case, the tracking process will fail. Also, attaching the markers is time consuming. Therefore, it is necessary to formulate a method that does not require optical markers.

• 2. How to measure the force that a person applies to a roller. Considering that data must be collected from multiple workers, it is a burden to attach a device to the workers, so it should be avoided.

• 3. The degree of painting work should be judged not only as a sequence of roller movements but also as a result of the work on the painted surface. In other words, it is necessary to record and visualize which part of the surface the roller part swept and how many times.

Approach

To solve the abovementioned problems, we propose the system shown in Fig. 2. First, an operator stands in front of the painting target. The operator holds a roller brush equipped with a force sensor and an IMU (at least it can measure three-axis acceleration and angular velocity) and paints the target surface. The goal is to achieve an even and homogeneous coating. This process is measured by a fixed RGB-D sensor, which is placed at a position where both the worker and the painting target can be observed.

Fig. 2

Overall picture of our approach. The interior of the blue dotted line indicates the proposed system. Roller poses, force applied to painted surface, and sweeping history are extracted and used for skill analysis

Of the sensor data obtained as such, we use depth image data to extract the roller and estimate its pose ($x, y, z, \theta$ in Fig. 2). Thus, we can obtain the timeseries pose change of the roller. The angle between the handle of the roller brush and painted surface ($\phi$ in Fig. 2) is measured by the IMU. To emphasize the estimation accuracy rather than the processing speed, these processes are assumed to be performed offline.

Using this measurement system, the issues in the previous subsection are resolved as follows. The list numbers correspond to the numbers in the above subsections.

• Pose estimation is based on 3D point clouds from RGB-D sensors, which eliminates the need for artificial markers. Moreover, because the pose estimation accuracy is inevitably lower than that of OMC, it is necessary to devise a method to maintain the accuracy. For this reason, we first roughly identify the roller pose from the 3D point cloud and then perform an additional alignment process. The detail is explained in the next section.

• A six-axis force sensor is inserted in the middle of the handle of the roller brush to indirectly measure the force applied by the operator. In other words, the operator does not need to wear anything for force measurement. This is described in “Generating swept map” section.

• Using the method described in item 1, the timeseries 3D poses of the roller can be estimated. Using this result, we make a swept map that can quantitatively clarify the sweeping part by the roller. We also visualize the painting results so that the user can understand them intuitively. The details are given in “Fundamental experiments” section.

Motion and force measurement from the roller part

Roller pose estimation and tracking

The pose of the roller brush comprises the pose $(x,y,z,\theta )$ of the roller part and the angle $\phi$ between the painting surface and handle (Fig. 2). These were determined based on interviews with skilled painters. Figure 3 shows the general flow of the roller pose estimation. The details of each procedure are described below.

Fig. 3

Flow of pose estimation of roller part

First, we describe the pose estimation and tracking of the roller part using a particle filter (Fig. 3(A)). To track using the particle filter, it is necessary to determine the state, system model, and observation model. In this study, the state of the roller part is defined as the coordinates $u,v$ and their velocities $\dot{u},\dot{v}$ in an image, and the system model is a linear prediction model. Therefore, the state vector is ${\varvec{u}}={\left[u,v,\dot{u},\dot{v}\right]}^{T}$, and the system model at time $t$ can be expressed as follows:

$${{\varvec{u}}}_{t}=\left[\begin{array}{cccc}1& 0& 1& 0\\ 0& 1& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]{{\varvec{u}}}_{t-1}+{{\varvec{w}}}_{t}^{(i)},$$

(1)

where ${{\varvec{w}}}_{t}^{(i)}$ indicates system noise, given by ${{\varvec{w}}}_{t}^{(i)}=\mathcal{N}\left(0,\boldsymbol{ }{{\varvec{\Sigma}}}_{4}\right).$

This means that ${{\varvec{w}}}_{t}$ is randomly generated according to a four-dimensional normal distribution with mean $0$ and covariance matrix ${{\varvec{\Sigma}}}_{4}$, where $i$ is the identity of a particle and $N$ is the number of particles, $i=1,\cdots ,N$.

The observation model is designed to have a higher likelihood when the observed particle is regarded as on the roller surface. For this purpose, the radius is calculated as follows. The observed data are depth images. However, because a human arm can be misdetected as a roller, a region derived from the human body is detected as a preprocess. Position and normal maps are then generated, excluding the region. Here the position maps are image-sized data that each pixel has 3D position coordinates calculated by perspective projection using depth value. Normal maps are image-sized data that each pixel has a 3D unit normal vector calculated from the depth data.

After generating the position and normal maps, the following process is performed on the $w\times w$ ($w$ is an odd number) neighbor pixels centered on the coordinates of each particle. First, the inner product of the normal vector ${{\varvec{n}}}_{p}$ at the position of the particle and the neighboring normal vector ${{\varvec{n}}}_{n}$ is calculated. The angle between the normal is calculated from the inner product; if the angle is larger than a predefined threshold, the particle is assumed to be on the roller part, and the radius $R$ is calculated as follows.

$$R=\frac{\Vert {{\varvec{l}}}_{d}\times ({c}_{1}+1){{\varvec{n}}}_{p}\Vert }{||{{\varvec{l}}}_{d}||},$$

(2)

where ${{\varvec{l}}}_{d}$ is given by

$${{\varvec{l}}}_{d}={{\varvec{p}}}_{n}-{{\varvec{p}}}_{p}+{c}_{2}{{\varvec{n}}}_{n}-\left({c}_{1}+1\right){{\varvec{n}}}_{p}.$$

${{\varvec{p}}}_{n}$ is a value of neighboring pixels at position map and ${{\varvec{p}}}_{p}$ is a value in the pixel of the particle. Figure 4 shows these definition and geometric relationships. ${c}_{1}\mathrm{ and }{c}_{2}$ are calculated as follows [13]:

Fig. 4

Definitions and geometric relationships of position and normal

$${c}_{1}=\frac{\left({{\varvec{n}}}_{p}\cdot {{\varvec{n}}}_{n}\right)\left({{\varvec{p}}}_{p}-{{\varvec{p}}}_{n}+{{\varvec{n}}}_{p}\right)\cdot {{\varvec{n}}}_{n}-{||{{\varvec{n}}}_{n}||}^{2}\left({{\varvec{p}}}_{p}-{{\varvec{p}}}_{n}+{{\varvec{n}}}_{p}\right)\cdot {{\varvec{n}}}_{p}}{{||{{\varvec{n}}}_{p}||}^{2}{||{{\varvec{n}}}_{n}||}^{2}-{\left({{\varvec{n}}}_{p}\cdot {{\varvec{n}}}_{n}\right)}^{2}},$$

$${c}_{2}=\frac{{||{{\varvec{n}}}_{p}||}^{2}\left({{\varvec{p}}}_{p}-{{\varvec{p}}}_{n}+{{\varvec{n}}}_{p}\right)\cdot {{\varvec{n}}}_{n}-\left({{\varvec{n}}}_{p}\cdot {{\varvec{n}}}_{n}\right)\left({{\varvec{p}}}_{p}-{{\varvec{p}}}_{n}+{{\varvec{n}}}_{p}\right)\cdot {{\varvec{n}}}_{p}}{{||{{\varvec{n}}}_{p}||}^{2}{||{{\varvec{n}}}_{n}||}^{2}-{\left({{\varvec{n}}}_{p}\cdot {{\varvec{n}}}_{n}\right)}^{2}}.$$

Among the radius $R$ obtained for all neighboring pixels, the one closest to the known roller radius ${R}_{true}$ is defined as ${R}_{near}$. The difference ${R}_{diff}$ between ${R}_{true}$ and ${R}_{near}$ will be smaller if the particle is located on the roller surface. Therefore, the likelihood function $L$ is assumed to be a normal distribution with the variance of ${R}_{diff}$ as ${\sigma }_{diff}^{2}$. $L$ is expressed as follows:

$$L=\frac{1}{\sqrt{2\pi {{\sigma }_{diff}}^{2}}}{ e}^{\left\{-\frac{{\left({R}_{diff}\right)}^{2}}{2{\sigma }_{diff}^{2}}\right\}}.$$

(3)

We now describe Fig. 3B. First, calculate the covariance matrix of coordinates $u$ and $v$ for the distribution of particles predicted by the system model. Then, ellipsoid region is created using the eigenvalues and eigenvectors of the covariance matrix. The ellipse is used as the region of interest, and the points that exist inside the region are extracted.

Finally, we explain Fig. 3C. The point cloud extracted by the above process includes the roller part and painted surface. Therefore, a cylindrical shape parameter is estimated for the point cloud using RANSAC [14], and points close to the known shape parameter of the roller part are extracted from the results. Then, point cloud registration is applied. The reference point cloud is made in advance with the same dimensions as the roller part. The Iterative Closest Point (ICP) algorithm [15] is employed for this process. The reason for extracting only the point cloud of the cylindrical shape is to prevent convergence to local minima.

Estimation of the angle between painted surface and handle

To estimate the angle $\phi$ between the painted surface and handle, we use the normal vector of the painted surface and the orientation of the IMU in the coordinate system with respect to the RGB-D sensor. The normal vector of the painted surface (${{\varvec{n}}}_{PS}$ shown in Fig. 5) is the average normal vector obtained from the normal map used in the likelihood calculation.

Fig. 5

Relationship between surface normal and IMU axis

The orientation of the IMU in the coordinate system of the RGB-D sensor (${{\varvec{v}}}_{IMU}$ shown in Fig. 5) is obtained by the following procedure. First, as a preliminary procedure, place the actual RGB-D sensor and IMU in the same orientation so that the coordinate system of each sensor faces the same orientation. From this result, the orientation of the RGB-D sensor coordinate system observed from the IMU world coordinate system, which is based on geomagnetism sensing, is obtained. Then, attach the IMU to the roller brush and start the measurement. Afterward, the measured IMU orientation is converted to the orientation observed from the RGB-D sensor coordinate system. Then, the angle $\phi$ between the painted surface and handle on the RGB-D sensor coordinates is calculated using ${{\varvec{n}}}_{PS}$ and ${{\varvec{v}}}_{IMU}$ shown in Fig. 5.

Force measurement

To measure the force while painting, a six-axis force sensor is embedded into the roller brush. Figure 6 shows the roller brush modified for our purpose. The roller part and handle part remain commercially available. However, the abovementioned force sensor is inserted and attached using screws. The joint parts at both ends of the force sensor are made of ABS resin manufactured by a 3D printer. Because a part of the iron rod is cut off and the force sensor is inserted instead, the length from the base of the handle to the tip of the brush is the same as the ready-made product.

Fig. 6

Structure of the roller brush device

Generating swept map

Based on the estimated pose of the roller part using the method described in “Motion and force measurement from the roller part” section, a “swept map” representing which part was swept how many times can be generated. First, the painting target surface is represented by a 3D point cloud. Then, to stabilize the contact judgment described later, the entire point cloud is translated by several tens of millimeters in the positive direction of the normal to the surface. The coordinate system of this point cloud is the same as the RGB-D sensor-based coordinate system. In the swept map generation, the coordinates of points are converted to the roller-based coordinate system. The origin is placed on the intersection point between the center core of the roller part and one side of the bottom circle. Then, the $z$-axis is defined on that point (Fig. 7). The $x$- and $y$-axis are also temporarily defined but not used in the following process. Next, vectors ${{\varvec{a}}}_{R}$ and ${{\varvec{v}}}_{R}$ are defined. ${{\varvec{a}}}_{R}$ is a vector directed from origin to the center of the other bottom circle, and ${{\varvec{v}}}_{R}$ is the position vector of ${}^{Roller}{{\varvec{p}}}^{(j)}$, where $j$ indicates identity of the point derived from the painting target surface. $M$ is the number of points, where $j=1,\cdots ,M$.

Fig. 7

Definition of vectors and roller coordinates. x, y, and z axes are shown by red, green and blue arrows, respectively

Next, following the process flow shown in Fig. 8, contacting condition between the roller model and point cloud of painted surface is investigated. Equation (4) in this flow judges whether ${}^{Roller}{{\varvec{p}}}^{(j)}$ exists between the roller’s two end faces. Equation (5) judges if ${}^{Roller}{{\varvec{p}}}^{(j)}$ is included in the roller model with radius $r$. Every time a point is judged to be included, the number registered to the point increases by 1. As a result, we obtain a map to know which part was swept how many times.

Fig. 8

The process flow for contacting detection

Fundamental experiments

In this section, we present fundamental proof experiments about the method described in “Motion and force measurement from the roller part”, “Generating swept map”, and “Fundamental experiments” sections. The RGB-D sensor used was Xtion Pro Live manufactured by ASUS Inc [16]. As a force sensor, CFS 034CA 301U manufactured by Reptrino Inc. [17] was selected based on compactness and the requirement that the sensor can measure the possible force on painting work. The rated load of this sensor is 150 N for translational forces Fx and Fy, 300 N for Fz, and 4.0 Nm for couples Mx, My, and Mz. The resolution is 1/4000 of the rated value. The employed IMU was a nine-axis IMU sensor module BWT901CL manufactured by WitMotion Inc [18].

To confirm estimation accuracy, OptiTrack V120: Trio [19] was employed to create ground truth. OptiTrack is an optical motion tracking system comprising an infrared light-emitting diode camera and artificial reflection markers. The pixel size of the image sensor used in OptiTrack is 6 × 6 µm, and the standard accuracy is 0.50 mm when tracking a 14 mm marker in a 9 m × 9 m area. The arrangement of the equipment in the experiment is shown in Fig. 9. The roller brush pose was recorded as a value in the coordinate system shown in this figure. Four artificial optical markers were attached to the roller part (Fig. 10). The calculation method of the roller pose is as follows:

Fig. 9

Experimental environment

Fig. 10

Roller brushes and marker placement to be used

Center position of the roller part: $\frac{{{\varvec{p}}}_{1}+{{\varvec{p}}}_{2}}{2},$

Inclination angle $\theta$: ${\mathrm{cos}}^{-1}\left(\frac{({{\varvec{p}}}_{1}-{{\varvec{p}}}_{2})\bullet {{\varvec{n}}}_{x}}{\Vert {{\varvec{p}}}_{1}-{{\varvec{p}}}_{2}\Vert \Vert {{\varvec{n}}}_{x}\Vert }\right),$

Angle between the painted surface and handle $\phi$: $\frac{\pi }{2}-{\mathrm{cos}}^{-1}\left(\frac{\left({{\varvec{p}}}_{4}-{{\varvec{p}}}_{3}\right)\bullet {{\varvec{n}}}_{y}}{\Vert {{\varvec{p}}}_{4}-{{\varvec{p}}}_{3}\Vert \Vert {{\varvec{n}}}_{y}\Vert }\right),$ where ${{\varvec{p}}}_{k}$ is the 3D position of $k$ th marker, ${{\varvec{n}}}_{x}$ and ${{\varvec{n}}}_{y}$ are, respectively, the x- and y-axis shown in Fig. 8. Comparing this result, the pose estimation result by the proposed method was evaluated.

As mentioned in Sect.  4.1, removing a region derived from a human worker in advance is needed for stable roller part tracking. For this, Nuitrack [20] was employed. In this experiment, the operator moved the roller at a maximum speed of about 1000 mm/s. The number of particles was experimentally set to 1500. Local window in the calculation of position and normal maps were set to 5 $\times$ 5. For noise countermeasures, the interval with neighboring pixels was increased to 2. The roller part tracking was performed on data replayed at one-tenth the speed of actual work. This is because accuracy was more important than online performance.

As a fundamental evaluation, five motion patterns were attempted:

• Case 1: five round trips with a 300-mm width in the x-direction,

• Case 2: five round trips with a 600-mm width in the x-direction,

• Case 3: five round trips with a 300-mm width in the z-direction,

• Case 4: five round trips with a 600-mm width in the z-direction,

• Case 5: making nine round trips with a 600-mm width in the z-direction while making one round trip with a 300-mm width in the x-direction.

Table 1 shows the comparison between the result from the proposed method and OptiTrack, showing the mean and standard deviation of absolute error. The position errors in the x- and y-direction were less than 5 mm. However, the error was larger when the roller moved in the x-direction than when it was moved in the z-direction. When the roller was moved in the z-direction, the error was about 15 mm in the z-direction. One characteristic is that the error tended to increase in the moving direction.

Table 1 Result of the accuracy of roller pose estimation

Possible causes of the error are that the RGB-D data obtained from the moving object are inaccurate, or that the timestamps of the sensor data are misaligned. In general, we found that the roller pose can almost be estimated with an accuracy that meets our purpose. From Table 1, the angle error did not exceed 5°. The left panel of Fig. 11 shows an example of particle distribution and the ellipsoidal region generated from the distribution. The right panel shows the final result. A cylindrical shape (blue) was superimposed to the original image as the estimated result.

Fig. 11

Particles and ellipses (left); Estimated pose (right)

Next, with respect to the five cases, visualization by swept map was performed. Figure 12 shows the results. The number of sweeps is shown here in colors, with the number of sweeps approaching green from blue, indicating that the number of sweeps was higher in that area. Because it was reciprocated several times in a specific direction, the part had the same color, meaning that an appropriate map can be generated.

Fig. 12

Generated swept maps. The number next to the color bar indicates the number of times the area was swept

Experiments and discussion

Method

In order to confirm the degree to which the proposed system can make meaningful measurements, actual painting operations were measured, and skill analysis was conducted to the extent conceivable. We conducted an interview survey with lecturers at the Tokyo Metropolitan College of Painting Technology. From there, we had the impression that expert skills would appear in how to handle the roller brush, such as how to move the roller during painting work and how to apply force. It was also expected that other skills could be confirmed by observing the history of the painting actions of the workers after the painting work was completed.

The experimental settings for painting work were as follows. A painting target, which is 900 × 600-mm² MDF plywood, was fixed to an aluminum frame and erected vertically. A relatively small roller brush with a width of 6 inches and a hair length of 13 mm was selected because it is more difficult to apply force to the roller part when the brush is smaller, so differences in skill level would appear more.

Then, the skilled and unskilled persons were asked to perform the painting work, and their work was measured using the proposed system (details are given in the next subsection)^┼. Further, features were extracted from the data and evaluated through statistical tests (see details in “Features for skill evaluation” section).

Data collection

There were nine measurement participants, five (41.6 average age) who serve as instructors or assistants at the painter’s school comprised the skilled group, and four (41.5 average age) who have little or no experience in painting with a roller brush comprised the unskilled group. All participants were right-handed men. Participants in the skilled group performed three to six trials each and recorded a total of 21 trials. In addition, the unskilled group recorded 10 trials, two to three times each.

In the painting work, a sufficient amount of paint was prepared in a special bucket. Participants were merely instructed to paint one side of the painting target to be painted evenly, and the method of painting, pose, and time spent were left to each person.

Features for skill evaluation

To investigate the stability of the roller brush operation and difference between the two groups, we divided the obtained data in each trial according to a predetermined standard and calculated the mean and standard deviation of each data. The mean represents the main behavioral characteristics of the worker in a trial, and the standard deviation is an indicator of its stability.

In the related work [1], which has a similar purpose to this study, the authors focused on the difference in arm movement between skilled and unskilled people, with the reciprocating motion of moving the roller brush up and down as the basic unit operation. In this experiment, different from [1], the procedure or motion of subjects for painting the target were not restricted to collecting the whole motion data during the task. Because of this, there is a large difference of the average task finish time between skilled and unskilled groups (Skilled:84.53 s, Unskilled: 153.71 s).

However, especially for unskilled subjects, this experiment was almost the first time that they painted with a roller brush and a special bucket. Then, the measured time included idle time to think or wonder about the usage of tools and the procedure of paint. Therefore, to reveal the meaningful difference between groups, the same policy with [1] was adopted, thus, the extraction of unit motions, the calculation of various features by these motions and the conduction of the statistical test to them. Specifically, the movement from the initial position of a brush to the movement upward or downward and returning to the initial position is defined as the basic unit operation. Notably, multiple basic unit operations can be extracted from each trial. For this, the frame where the roller brush velocity becomes zero at the top end was vertically detected and was regarded as a breaking point. Due to the characteristic of the roller painting operation, such a point clearly appears. According to this criterion, it is possible to align the data quality even if it is done manually. Then, because the time taken for each basic unit operation differed, it was normalized by the time taken.

For every basic unit operation data, the following process was applied. First, the data were divided into $K$ bins according to the time axis; then, the mean value ${\overline{x}}_{i}^{j}$ was calculated from data included in each bin, where $i$ and $j$ indicates the number of the present bin and basic unit operation, respectively.

Next, standard deviation ${s}_{i}$ is calculated as follows:

$${y}_{i}=\frac{1}{J}{\sum }_{j = 1}^{J}{\overline{x}}_{i}^{j},$$

(6)

$${s}_{i}=\sqrt{\frac{1}{J}{\sum }_{j = 1}^{J}{\left( {\overline{x}}_{i}^{j} - {y}_{i}\right)}^{2}},$$

(7)

where ${y}_{i}$ is a representative value for the $i$ th bin, and $J$ is the number of basic units used. For example, ${y}_{1}$ presents a mean value in the beginning part of basic unit operation, calculated from several data included in one painting trial by one subject. ${s}_{1}$ indicates how much it varies between the basic unit operations. In this experiment, the number of bins was $K$= 30. This number was consciously determined so that each bin contained data. As the RGBD camera was 30 FPS and an up-and-down roller motion took approximately 2 s. Therefore, by setting $K$= 30, it was confirmed that no empty bins were created and that there was little loss of information. Two trials by one skilled person and three trials by two unskilled persons who did not perform appropriate basic unit operations were excluded from the subsequent analysis. That is, 19 trials by five skilled persons and 7 trials for four unskilled persons were used for analysis. Figure 13 shows examples of the obtained data. Here shows four examples of graphs created from data obtained when one person performs a single painting work. The numbers at the top mean that, for example, in the leftmost example, there were 14 up-and-down roller movements in one painting work. The plots show the average and the variation of each up-and-down movement in one trial for one subject. The center line and the filled area indicates mean and standard deviation, respectively.

Fig. 13

Examples of organized data

Based on the above preprocess, four types of features, S, Vphi, F, and VF, were defined to compare the differences in painting skills among skilled and unskilled persons.

• Roller part velocity (S): Roller part velocity (S) is an index of the quickness of the painting operation. In addition, tips for painting by skilled people may be reflected, such as the need to change the speed depending on the paint’s viscosity. S is determined by selecting the largest absolute value from the 30 representative values derived from Eq. (6) in each trial. The velocity of the roller part is applied as $x$.

• The amount of variation in angle ${\varvec{\phi}}$ (Vphi): The variation in the angle between the painted surface and handle indicates whether the roller brush is always operated in a stable posture in the basic unit operation. Thus, Vphi is derived from the average of 30 representative values calculated from Eq. (7) in each trial. The angle $\phi$ is applied as $x$.

• The amount of contact force (F) and variation (VF): In the painting work by skilled people, less and more stable force might be applied to perform the coat evenly with less paint consumption. Therefore, F and VF are defined as indexes of the amount of the contact force, which is the force component perpendicular to the painting target derived from the force sensor and roller posture and the force variation. In individual trials, F and VF are derived from the average of the 30 representative values obtained from Eqs. (6) and (7) each. The contact force between the roller brush and paint target is applied as $x$.

In addition, the coefficient of variation of the number of sweeps by the roller, which is denoted as CV, was used as an evaluation index of the painting result. The evaluation index is a quantity calculated throughout one painting trial, unlike the above four features. For this calculation, swept maps were used. Figure 14 shows examples of swept maps.

• The coefficient of variation of the number of sweeps (CV): This indicates whether the roller has swept the painting area evenly and is calculated from the swept map described in “Fundamental experiments” section. It is calculated by dividing the standard deviation of the number of swept times for each point on the painted surface by its mean value. When the number of swept times of each point is uniform, the value decreases.

Fig. 14

Examples of swept maps

As for the coefficient of variation, the amount of data for the skilled people was 18. This is because, in one trial by one skilled person, the number of sweeps was not calculated correctly due to the problem of the installation position of the RGB-D camera.

Analysis result

Tables 2 and 3 shows the mean values and the results of two-way ANOVA performed to analyze the effect of Groups (skilled and unskilled) and Trials. In Table 3, symbol * indicates items with significant differences because p < 0.05. From this table, there was no statistically significant interaction effect of Group and Trial on all features. On the other hand, regarding the main effect of Group, there is a significant difference in the maximum speed of the roller part (S), F (1, 17) = 34.28, p < 0.001. This result suggests that skilled people move the roller brush faster than unskilled people.

Table 2 Mean values of features

Table 3 Analysis result of two-way ANOVA

There were also significant differences in the magnitude of contact force (F) and that of its variation (VF) (F (1,17) = 22.87, p < 0.001, F (1,17) = 14.82, p = 0.001, respectively). From these results, we confirmed that the force applied by the skilled people was smaller than that of unskilled people, and the painting was performed with a stable force. Moreover, in the angle variation between the painted surface and handle (Vphi), no statistically significant differences were found between the groups. In addition, regarding the coefficient of variation of the number of sweeps (CV), there was a significant difference between the groups (F (1,17) = 21.37, p < 0.001). This suggests that the sweeps by skilled people were more uniform than unskilled people.

Discussion

Based on the experimental results, we first discuss the measurement data obtained by the proposed system. In this experiment, we tried to evaluate whether each individual’s “skill” in painting operation was archived and could be extracted as meaningful data. If this result makes it possible to clarify important movements and tips in painting possessed by experts in the future, it will be a stepping stone to establish an objective evaluation index for painting technology and to apply it for automation, etc.

As we have seen, significant differences between skilled and unskilled people were recorded in terms of the velocity at which the rollers were moved, the magnitude of the contact force, and their variations. This means that unskilled people struggled with the unfamiliar operation of the roller brush, and the painting was tedious and unstable. Such characteristics were appropriately recorded by the proposed system.

Besides, there was no significant difference between the two groups in the angle $\phi$ between the painted surface and handle. Before the experiment, it was expected that the roller pose would be unstable for unskilled people. However, as shown in Table 2, the difference of Vphi between groups was only 0.56 deg, which is smaller than the measurement error value evaluated in “Fundamental experiments” section (shown in Table 1). It is believed that, in the movement of reciprocating up and down with a roller, the strategy of holding and moving was fixed to some extent by the range of motion of the wrist, and it seems that the pose was similar regardless of skill level. It is indicated that to distinguish the difference, it is essential to employ a more high-precision roller posture estimation method. In addition, a common brush instead of a roller brush can be used to paint. Painting with such a brush will require more skill to control the posture of the tool than painting with a roller brush. In the future, we would like to extend and apply the proposed system to various painting tools.

The coefficient of variation of the number of sweeps (CV), which was used as an index of painting stability, showed a significant difference between skilled and unskilled people. As summarized in Table 2, the average difference of CV is 0.06. Depending on a simple assessment with numerical simulation performed to estimate the effect of the measurement errors on the CV value, the amount of influence by the error is estimated at 0.01. Therefore, it shows that this value could represent the difference between groups. Thus, the skilled people swept the entire surface relatively evenly compared with the unskilled people.

This index was set as a guideline for evaluating the painted surface itself, but it is difficult to conclude from this result that the expert can paint evenly. This is because uneven coating does not necessarily depend only on the number of sweeps. For example, if the roller is pressed strongly against the painted surface, a lot of paint will adhere in one sweep. However, the amount of the paint transferred to a target surface depends on the amount of paint contained in the roller brush at that time. Therefore, to evaluate whether the target surface is evenly painted, it is necessary to comprehensively estimate the amount of paint contained in the roller brush, magnitude of the force applied in the sweep, and amount of paint consumed.

Conclusion

In this study, we constructed a measurement system for properly recording painting work using roller brush. We focused on the operation of the roller brush and developed an algorithm that can estimate the roller pose with high accuracy. Our method allows us to estimate the roller pose with an error of approximately 5 mm and 2 degrees. We also presented a method to generate a swept map after painting. To verify the usefulness of the proposed system, we planned experiments which skilled people participate and the dataset for actual painting work was collected. Then, the dataset was analyzed by five types of features. The results show some obvious differences between skilled and unskilled people. Although the results were in line with intuition, we believe that this quantitative confirmed results demonstrates the usefulness of our system.

Based on the current measurement system, we will increase the quality and variety of data that can be acquired. For instance, it would be good to know the amount of paint contained in the roller while painting. Another important remaining work is to construct a system that can measure and analyze the state of the painted surface after painting. This will clarify the relationship between the painting operation and painted result. Furthermore, the proposed system will be used in conjunction with other skill analysis methods based on human motion measurements to clarify the effects of different human actions on roller motion. By doing so, we aim to deepen the means of skill analysis in tool manipulation.