Skip to main content

Pose estimation by extended Kalman filter using noise covariance matrices based on sensor output

Abstract

This paper presents an extended Kalman filter for pose estimation using noise covariance matrices based on sensor output. Compact and lightweight nine-axis motion sensors are used for motion analysis in widely various fields such as medical welfare and sports. A nine-axis motion sensor includes a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer. Information obtained from the three sensors is useful for estimating joint angles using the Kalman filter. The extended Kalman filter is used widely for state estimation because it can estimate the status with a small computational load. However, determining the process and observation noise covariance matrices in the extended Kalman filter is complicated. The noise covariance matrices in the extended Kalman filter were found for this study based on the sensor output. Postural change appears in the gyroscope output because the rotational motion of the joints produces human movement. Therefore, the process noise covariance matrix was determined based on the gyroscope output. An observation noise covariance matrix was determined based on the accelerometer and magnetometer output because the two sensors’ outputs were used as observation values. During a laboratory experiment, the lower limb joint angles of three participants were measured using an optical 3D motion analysis system and nine-axis motion sensors while participants were walking. The lower limb joint angles estimated using the extended Kalman filter with noise covariance matrices based on sensor output were generally consistent with results obtained from the optical 3D motion analysis system. Furthermore, the lower limb joint angles were measured using nine-axis motion sensors while participants were running in place for about 100 s. The experiment results demonstrated the effectiveness of the proposed method for human pose estimation.

Introduction

Compact and lightweight nine-axis motion sensors have been developed through advances in micro-electromechanical systems technology; they have come to be used for motion analysis in widely various fields [1,2,3,4,5,6,7,8]. The nine-axis motion sensors are applicable both indoors and outdoors because of their portability. Several experiments have been conducted to measure the motion of a skier gliding down a slope and jumping off a hill using motion sensors [9, 10]. The nine-axis motion sensors include a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer. Using information obtained from the motion sensors, several sensor fusion algorithms have been proposed for pose estimation: as one example, a sensor fusion algorithm that can correct gyroscope drift using information obtained from the other two sensors has been used for human pose estimation during daily activities and exercise [11,12,13]. Furthermore, a sensor fusion algorithm able to correct the magnetometer output using information obtained from a gyroscope has been used for pose estimation in a variable magnetic field [14, 15]. The Kalman filter [16,17,18,19,20] and the complementary filter [21,22,23,24,25] are some pose estimation methods using sensor fusion.

The Kalman filter estimates the system state with a small computational load. Nevertheless, determining the process and observation noise covariance matrices in the Kalman filter is complicated. For a case in which the process and observation noise covariance matrices are time-invariant, the estimation accuracy might decrease if the sensor output noise increases. Moreover, the noise of the sensor output might vary because of long-term measurements. For that reason, adjusting the noise covariance matrices based on sensor output is important.

To estimate the lower limb joint angles for this study, a method was devised to determine the process and observation noise covariance matrices in the extended Kalman filter based on sensor output. The postural change appears in the gyroscope output because the rotational motion of the joints produces human movement. Therefore, the process noise covariance matrix was set based on the gyroscope output. When the accelerometer output increased, the observation noise covariance matrix was set to increase. The observation noise covariance matrix was also set to increase when the magnetometer output drastically changed. During a laboratory experiment, the lower limb joint angles of three participants were measured using an optical 3D motion analysis system and nine-axis motion sensors while the participants were walking. Several studies have demonstrated that an optical 3D motion analysis system measured human movement with high accuracy. Therefore, the system is used for verifying the pose estimation accuracy in widely diverse fields [26,27,28,29]. We verified the accuracy of the proposed method by comparing its results to those of an optical 3D motion analysis system. Furthermore, the lower limb joint angles were measured using nine-axis motion sensors while the participants were running in place. Finally, the effectiveness of the proposed method was verified using experiment results.

Measurement method

Definition of roll-pitch-yaw

The 3D posture of the sensor is represented by the roll angle (\(\phi\)) around the x-axis, the pitch angle (\(\theta\)) around the y-axis, and the yaw angle (\(\psi\)) around the z-axis. The reference coordinate system is a right-handed system with a vertical z-axis. The counterclockwise rotation is defined as positive. The reference coordinate system and the definition of the joint angles are presented in Fig. 1.

Fig. 1
figure1

Definition of the lower limb joint angles and the reference coordinate system

Roll-pitch-yaw calculation

For this study, Euler angles (roll, pitch, and yaw) were calculated using nine-axis motion sensors. The nine-axis motion sensor (SS-WS1792; Sports Sensing Co., Ltd.) used for this study includes a three-axis gyroscope (± 1500 dps), a three-axis accelerometer (± 16 G), and a three-axis magnetometer (± 10 Gauss). The 38 × 53 × 11 mm sensor weighs 30 g.

The initial roll and pitch angles were calculated using the accelerometer output at rest [30, 31]. The relation between the acceleration sensor output and the gravitational acceleration in the reference coordinate system is expressed using Eq. (1) because the accelerometer measures only the.

gravitational acceleration while at rest:

$${}^{i}A = ({}^{O}R_{i} )^{T} {}^{O}A,\,(i = 1,2,3,4)$$
(1)

where.

$${}^{i}A = \left[ {\begin{array}{*{20}c} {{}^{i}A_{x} } \\ {{}^{i}A_{y} } \\ {{}^{i}A_{z} } \\ \end{array} } \right],\,{}^{o}A = \left[ {\begin{array}{*{20}c} 0 \\ 0 \\ g \\ \end{array} } \right],$$

Therein, iA denotes the accelerometer output, oA represents the acceleration in the reference coordinate system, and g stands for gravitational acceleration. For the experiment, sensors 1, 2, 3, and 4 were placed respectively on the waist, left thigh, left shank, and left foot. In addition, the rotational matrix from the sensor coordinate system to the reference system oRi is the following:

$${}^{o}R_{i} = \left[ {\begin{array}{*{20}c} {\cos {}^{i}\psi } & { - \sin {}^{i}\psi } & 0 \\ {\sin {}^{i}\psi } & {\cos {}^{i}\psi } & 0 \\ 0 & 0 & 1 \\ \end{array} } \right]\,\left[ {\begin{array}{*{20}c} {\cos {}^{i}\theta } & 0 & {\sin {}^{i}\theta } \\ 0 & 1 & 0 \\ { - \sin {}^{i}\theta } & 0 & {\cos {}^{i}\theta } \\ \end{array} } \right]\,\left[ {\begin{array}{*{20}c} 1 & 0 & 0 \\ 0 & {\cos {}^{i}\phi } & { - \sin {}^{i}\phi } \\ 0 & {\sin {}^{i}\phi } & {\cos {}^{i}\phi } \\ \end{array} } \right]$$
(2)

Then, the accelerometer output iA is represented by substituting Eq. (2) into Eq. (1) as shown below:

$$\left[ {\begin{array}{*{20}c} {{}^{i}A_{x} } \\ {{}^{i}A_{y} } \\ {{}^{i}A_{z} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} { - \sin {}^{i}\theta \cdot g} \\ {\cos {}^{i}\theta \sin {}^{i}\phi \cdot g} \\ {\cos {}^{i}\theta \cos {}^{i}\phi \cdot g} \\ \end{array} } \right]$$
(3)

The initial roll and pitch angles using Eq. (3) are:

$${}^{i}\phi_{A} = atan2\frac{{iA_{y} }}{{{}^{i}A_{z} }}\,\,( - \pi < {}^{i}\phi_{A} < \pi )$$
(4)
$${}^{i}\theta_{A} = atan2\frac{{ - {}^{i}A_{x} }}{{\sqrt {{}^{i}A_{y}^{2} + {}^{i}A_{z}^{2} } }}( - \pi < {}^{i}\theta_{A} < \pi )$$
(5)

where iAx, iAy, and iAz respectively denote the accelerometer output for the x, y, and z axes. Therein, iφA and iθA respectively denote the initial roll and pitch.

To correct the yaw angle, calculations require the roll iφA, pitch iθA, and magnetometer output as:

$$\left[ {\begin{array}{*{20}c} {^{c,i} m_{x} } \\ {^{c,i} m_{y} } \\ {^{c,i} m_{z} } \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {\cos {}^{i}\theta_{A} } & {\sin {}^{i}\phi_{A} \sin {}^{i}\theta_{A} } & {\cos {}^{i}\phi_{A} \sin {}^{i}\theta_{A} } \\ 0 & {\cos {}^{i}\phi_{A} } & { - \sin {}^{i}\phi_{A} } \\ { - \sin {}^{i}\theta_{A} } & {\sin {}_{{}}^{i} \phi_{A} \cos {}^{i}\theta_{A} } & {\cos {}^{i}\phi_{A} \cos {}^{i}\theta_{A} } \\ \end{array} } \right] \cdot \left[ {\begin{array}{*{20}c} {{}^{i}m_{x} } \\ {{}^{i}m_{y} } \\ {{}^{i}m_{z} } \\ \end{array} } \right]$$
(6)

where imx, imy, and imz respectively denote the magnetometer outputs for the x, y, and z axes. Therein, c,imx, c,imy, and c,imz respectively represent the corrected magnetic field data for the x, y, and z axes. The positive directions of c,imx, c,imy, and c,imz coincide with those of each axis of the reference coordinate system in Fig. 1. The x-axis pointed in the direction of the azimuth at 112.5 degrees (east-southeast). The y-axis pointed in the direction at the azimuth of 202.5 degrees (south-southwest) in the reference coordinates.

The following equation is used for calculating yaw:

$${}^{i}\psi_{m} = atan2\frac{{ -^{c,i} m_{y} }}{{^{c,i} m_{x} }} ( - \pi < {}^{i}\psi_{m} < \pi )$$
(7)

where \({}^{i}\psi_{m}\) denotes the azimuth on the xy plane of the reference coordinates in Fig. 1.

The differential Euler angles in the reference coordinate system are the following:

$$\left[ {\begin{array}{*{20}c} {{}^{i}\dot{\psi }} \\ {{}^{i}\dot{\theta }} \\ {{}^{i}\dot{\phi }} \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} 0 & {\sin {}^{i}\phi \sec {}^{i}\theta } & {\cos {}^{i}\phi \sec {}^{i}\theta } \\ 0 & {\cos {}^{i}\phi } & { - \sin {}^{i}\phi } \\ 1 & {\sin {}^{i}\phi \tan {}^{i}\theta } & {\cos {}^{i}\phi \tan {}^{i}\theta } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {{}^{i}\omega_{x} } \\ {{}^{i}\omega_{y} } \\ {{}^{i}\omega_{z} } \\ \end{array} } \right]$$
(8)

Therein, \({}^{i}\dot{\phi }\), \({}^{i}\dot{\theta }\), and \({}^{i}\dot{\psi }\) respectively represent the differential roll, pitch, and yaw angles, \({}^{i}\omega_{y}\) and \({}^{i}\omega_{z}\) respectively stand for the gyroscope output for the x, y, and z axes. Then the roll, pitch, and yaw angles are calculated by substituting Eq. (8) into Eq. (9):

$$\left[ {\begin{array}{*{20}c} {{}^{i}\psi } \\ {{}^{i}\theta } \\ {{}^{i}\phi } \\ \end{array} } \right]_{t + 1} = \smallint \left[ {\begin{array}{*{20}c} {{}^{i}\dot{\psi }} \\ {{}^{i}\dot{\theta }} \\ {{}^{i}\dot{\phi }} \\ \end{array} } \right]dt + \left[ {\begin{array}{*{20}c} {{}^{i}\psi } \\ {{}^{i}\theta } \\ {{}^{i}\phi } \\ \end{array} } \right]_{t}$$
(9)

Extended Kalman filter

State-space model

The roll, pitch, and yaw angles of each sensor placed on the lower limb are estimated by the sensor fusion using the extended Kalman filter. The nonlinear state equation was developed using Eq. (9). The nonlinear observation equation was developed using Eq. (7) and the acceleration sensor output. The nonlinear state and observation equations are shown respectively in Eqs. 10 and 11:

$${}^{i}x_{t + 1} = {}^{i}F\left( {{}^{i}x_{t} , {}^{i}\omega_{t} } \right) + {}^{i}w_{t} ,$$
(10)
$${}^{i}y_{t} = {}^{i}H\left( {{}^{i}x_{t} } \right) + {}^{i}v_{t} ,$$
(11)

where,

$${}^{i}x_{t} = \left[ {\begin{array}{*{20}c} {{}^{i}\psi } \\ {{}^{i}\theta } \\ {{}^{i}\phi } \\ \end{array} } \right]_{t}$$
$${}^{i}F\left( {x_{t} , \omega_{t} } \right) = \left[ \begin{gathered} {}^{i}\psi_{t} + \sin {}^{i}\phi_{t} \sec {}^{i}\theta_{t} {}^{i}\omega_{y,t} \cdot Ts + \cos {}^{i}\phi_{t} \sec {}^{i}\theta_{t} {}^{i}\omega_{z,t} \cdot Ts \hfill \\ {}^{i}\theta_{t} + \cos {}^{i}\phi_{t} {}^{i}\omega_{y,t} \cdot Ts - \sin {}^{i}\phi_{t} {}^{i}\omega_{z,t} \cdot Ts \hfill \\ {}^{i}\phi_{t} + {}^{i}\omega_{x,t} \cdot Ts + \sin {}^{i}\phi_{t} \tan {}^{i}\theta_{t} {}^{i}\omega_{y,t} \cdot Ts + \cos {}^{i}\phi_{t} \tan {}^{i}\theta_{t} {}^{i}\omega_{z,t} \cdot Ts \hfill \\ \end{gathered} \right]$$
$${}^{i}y_{t} = \left[ {\begin{array}{*{20}c} {{}^{i}\psi_{m} } \\ {{}^{i}A_{{S_{x} }} } \\ {\begin{array}{*{20}c} {{}^{i}A_{{S_{y} }} } \\ {{}^{i}A_{{S_{z} }} } \\ \end{array} } \\ \end{array} } \right]_{t} ,{}^{i}H\left( {x_{t} } \right) = \left[ {\begin{array}{*{20}c} {{}^{i}\psi_{t} } \\ {\left( {{}^{0}R_{i} } \right)_{t}^{T} \left[ {\begin{array}{*{20}c} 0 \\ 0 \\ g \\ \end{array} } \right]} \\ \end{array} } \right]$$

In those equations, \({}_{t}\), \({}^{i}\theta_{t}\), and \({}^{i}\psi_{t}\) respectively denote Euler angles of the sensor placed on each lower limb segment, as estimated using the extended Kalman filter. Ts stands for the sampling time. In addition, \({}^{i}\omega_{x,t}\), \({}^{i}\omega_{v,t}\), and \({}^{i}\omega_{z,t}\) respectively denote the gyroscope outputs for the x, y, and z axes. Also, \({}^{i}A_{{S_{x} }}\), \({}^{i}A_{{S_{y} }}\), and \({}^{i}A_{{S_{z} }}\) respectively express the accelerometer output for the x, y, and z axes. Therein, \({}^{i}w_{t}\)and \({}^{i}v_{t}\) denote white noise.

Yaw angle \({}^{i}\psi_{m}\), which was calculated using the magnetometer output, and the accelerometer output were used as the observation values in Eq. (11). Equation (1) represents the relation between the accelerometer output and gravitational acceleration. Consequently, the right side of Eq. (1) was used in \({}^{i}H\left( {{}^{i}x_{t} } \right)\) of the observation equation. Although the proportion of the centrifugal acceleration and the tangential acceleration in the accelerometer output would have increased during exercise, these acceleration components were processed as observation noise. In addition, \({}^{i}\psi_{m} = {}^{i}\psi_{t}\)is a simple equation representing the relation between the magnetometer output and the yaw angle of the state values. Therefore, the yaw angle \({}^{i}\psi_{t}\) of the state value was used in \({}^{i}H\left( {{}^{i}x_{t} } \right)\) of the observation equation.

The partial derivatives of \({}^{i}F\left( {{}^{i}x_{t} , {}^{i}\omega_{t} } \right)\) and \({}^{i}H\left( {{}^{i}x_{t} } \right)\) are shown below:

$${}^{i}f\left( {{}^{i}x_{t} , {}^{i}\omega_{t} } \right) = \frac{{\partial {}^{i}F\left( {{}^{i}x_{t} , {}^{i}\omega_{t} } \right)}}{{\partial {}^{i}x_{t} }},$$
(12)
$${}^{i}h\left( {{}^{i}x_{t} } \right) = \frac{{\partial {}^{i}H\left( {{}^{i}x_{t} } \right)}}{{\partial {}^{i}x_{t} }},$$
(13)

Then, the prediction step [Eqs. (14) and (15)] and the filtering step [Eqs. (18), (19), (20)] are calculated using the nonlinear discrete-time system represented by Eqs. (10) and (11)] . Here, Eq. (16) and Eq. (17) are used for calculating the likelihood of the state-space model:

$${}^{i}x_{t + 1}^{ - } = {}^{i}F\left( {{}^{i}x_{t} , {}^{i}\omega_{t} } \right)$$
(14)
$${}^{i}P_{t + 1}^{ - } = {}^{i}f_{t} {}^{i}P_{t} {}^{i}f_{t}^{T} + {}^{i}Q_{t} ,$$
(15)
$${}^{i}V_{t + 1} = {}^{i}y_{t + 1} - {}^{i}H\left( {{}^{i}x_{t + 1}^{ - } } \right),$$
(16)
$${}^{i}B_{t + 1} = {}^{i}h_{t + 1} {}^{i}P_{t + 1}^{ - } {}^{i}h_{t + 1}^{T} + {}^{i}R_{t} ,$$
(17)
$${}^{i}K_{t + 1} = {}^{i}P_{t + 1}^{ - } {}^{i}h_{t + 1}^{T} \left( {{}^{i}h_{t + 1} {}^{i}P_{t + 1}^{ - } {}^{i}h_{t + 1}^{T} + {}^{i}R_{t} } \right)^{ - 1} ,$$
(18)
$${}^{i}x_{t + 1} = {}^{i}x_{t + 1}^{ - } + {}^{i}K_{t + 1} \left( {{}^{i}y_{t + 1} - {}^{i}H\left( {x_{t + 1}^{ - } } \right)} \right)$$
(19)
$${}^{i}P_{t + 1} = \left( {I - {}^{i}K_{t + 1} {}^{i}h_{t + 1} } \right){}^{i}P_{t + 1}^{ - }$$
(20)

In those equations, \({}^{i}P\) represents the error covariance matrix, \({}^{i}V\) denotes the prediction error matrix, \({}^{i}B\) stands for the prediction error variance matrix, and \({}^{i}K\) denotes the Kalman gain. Therein, \({}^{i}Q\) and \({}^{i}R\) respectively denote the covariance matrices of process noise \({}^{i}{w}_{t}\) in the nonlinear state equation and observation noise \({}^{i}{v}_{t}\) in the nonlinear observation equation.

Noise covariance matrices based on sensor output

The process and observation noise covariance matrices in the extended Kalman filter were determined based on the state-space model dynamics and the sensor noise. The postural change appears in the gyroscope output because the rotational motion of the joints produces human movement. Consequently, the process noise covariance matrix was determined based on the gyroscope output as presented below:

$${}_{{}}^{i} Q_{t} = \left[ {\begin{array}{*{20}c} {{}_{{}}^{i} \Omega_{\omega ,t} } & 0 & 0 \\ 0 & {{}_{{}}^{i} \Omega_{\omega ,t} } & 0 \\ 0 & 0 & {{}_{{}}^{i} \Omega_{\omega ,t} } \\ \end{array} } \right]$$
(21)

where,

$${}^{i}\Omega_{\omega ,t} = a\sqrt {{}^{i}\omega^{2}_{x,t} + {}^{i}\omega^{2}_{y,t} + {}^{i}\omega^{2}_{z,t} } + b$$

In those expressions, \({}^{i}\omega_{x,t}\), \({}^{i}\omega_{y,t}\), and \({}^{i}\omega_{z,t}\) respectively stand for the gyroscope output for x, y, and z axes. Also a and b are adjusting parameters. For this study, a and b were determined to maximize the log-likelihood (\({}_{{}}^{i} LL\)) shown in Eq. (22):

$${}^{i}LL = - \frac{N}{2}ln\left( {2\pi } \right) - \frac{1}{2}\mathop \sum \limits_{j = 1}^{N} \left( {ln\left( {{}^{i}B_{j} } \right) + \frac{{{}^{i}V_{j}^{2} }}{{{}^{i}B_{j} }}} \right)$$
(22)

In that equation, N stands for the number of time-series data; j represents time-series. In addition, \({}^{i}{B}_{j}\) expresses the prediction error variance; \({}^{i}{V}_{j}\) signifies the prediction error.

The observation noise covariance matrices must be set at a high value when the sensor noise increases [32]. Therefore, the observation noise covariance matrix was determined based on the accelerometer and magnetometer output because the two sensor outputs were used as observation values [33]. The observation noise covariance matrix is presented below:

$${}^{i}R_{t} = \left[ {\begin{array}{*{20}c} {{}^{i}\Omega_{m,t} } & 0 & 0 & 0 \\ 0 & {{}^{i}\Omega_{a,t} } & 0 & 0 \\ 0 & 0 & {{}^{i}\Omega_{a,t} } & 0 \\ 0 & 0 & 0 & {{}^{i}\Omega_{a,t} } \\ \end{array} } \right]$$
(23)

where,

$${}^{i}\Omega_{m,t} = c\left( {\sqrt {{}^{c,i}m_{x,t}^{2} + {}^{c,i}m_{y,t}^{2} + {}^{c,i}m_{z,t}^{2} } - \overline{m}} \right) + d$$
$${}^{i}\Omega_{a,t} = e\left( {\sqrt {{}^{i}A_{x,t}^{2} + {}^{i}A_{y,t}^{2} + ({}^{i}A_{z,t} - g)^{2} } } \right) + f$$

In the matrix, \({}_{{}}^{c, i} m_{x,t}\), \({}_{{}}^{c, i} m_{y,t}\), and \({}_{{}}^{c, i} m_{z,t}\) respectively denote corrected magnetic field data for the x, y, and z axes. In addition, \(\overline{m}\) represents the average value of the magnetometer output over the entire measurement time. Furthermore, iAx,t, iAy,t, and iAz,t respectively express the accelerometer outputs for the x, y, and z axes. Therein, c, d, e, and f are adjusting parameters. In addition, c, d, e, and f were determined to maximize the log-likelihood (LL) shown in Eq. (22). Several studies have proposed process and observation noise covariance matrices based on sensor output [31, 33]. In those earlier studies, noise covariance matrices were produced after calculating the deviation of each sensor using many equations. The adjusting parameters a to f in the noise covariance matrices are determined simply using only log-likelihood calculations presented in Eq. (22). In addition, interaction between the process noise covariance and observation noise covariance is considered in log-likelihood calculations.

The roll, pitch, and yaw angles in the sensor i coordinate system obtained from the sensor fusion are converted into the rotational matrix of the reference coordinate system using Eq. (2). The lower limb joint angles are calculated by substituting Eq. (2) into Eq. (24) as shown below:

$${}^{i - 1}R_{i} = \left( {{}^{0}R_{i - 1} } \right)^{T} \cdot \left( {{}^{0}R_{i} } \right)$$
(24)

In that equation, i−1Ri denotes the rotational matrix from the sensor i coordinate system to the sensor i-1 system. The hip joint angle is estimated using the output of the two sensors placed on the waist and thigh. The knee joint angle is estimated using the output of the two sensors placed on the thigh and shank. The ankle joint angle is estimated using the output of the two sensors positioned on the shank and foot.

Verification experiment

Participants and experiment conditions

Three healthy participants (A, B, and C) were examined during the experiment. Anthropometric data are shown in Table 1. After maintaining the upright posture for about 5 s, the first step that a participant took was with the left foot. They were instructed to walk using a natural stride in time with a metronome (70 bpm). Measurement started simultaneously when a participant started to maintain the upright posture. The measurements finished when the participant placed the right foot flat on the floor during the sixth step. Following an explanation of the purpose and requirements of the study, the participants gave their written informed consent to participate in the study. Study approval was obtained from the Research Ethics Board, Kogakuin University, and National Institute of Technology, Akita College.

Table 1 Anthropometric data

During the experiment, kinematic data were collected using an optical 3D motion analysis system (Bonita 10; Vicon Motion Systems Ltd.), two force plates (9286; Kistler Japan Co. Ltd.), and four nine-axis motion sensors in synchronization. The heel strike and toe off were ascertained from force plate data. The sensors were placed on the waist, left thigh, left shank, and left foot using double-sided tape and elastic straps. The sensor positions are presented in Fig. 2. Definitions of the length of the thigh, shank, and foot were referred from reports of earlier research [34]. Positions of reflective markers for the optical 3D motion analysis system were found by reference to the Vicon Plug-in Gait model. The sampling frequencies of the nine-axis motion sensors, the optical 3D motion analysis system, and the force plates were 100 Hz.

Fig. 2
figure2

Sensor positions and sensor coordinate system

Results

Table 2 shows parameters a to f in the walking experiment, which were found to maximize the log-likelihood in Eq. (22). In the walking experiment, more or less equivalent parameters were obtained for all participants.

Table 2 Adjusting parameters of NBS in the walking experiment

The joint angles of participant A are depicted in Fig. 3. Black solid curves represent results obtained from the optical 3D motion analysis system. Red solid curves represent results obtained from the extended Kalman filter using the noise covariance matrices based on sensor output, hereinafter designated as NBS. Orange solid curves represent results obtained from NBS, which used gyroscope output for the process noise covariance matrix and which used a constant value for the observation noise covariance matrix, hereinafter designated as NBS (Only process noise). Green solid curves represent results obtained from NBS, which used the constant value for the process noise covariance matrix and which used accelerometer and magnetometer output for the observation noise covariance matrix, hereinafter designated as NBS (Only observation noise). Blue solid curves present results obtained from the extended Kalman filter using the constant noise covariance, hereinafter designated as CNC. In addition, Ωω (= 0.0005), Ωm (= 1500), and Ωa (= 1500) of CNC were determined to maximize the log likelihood (LL) shown in Eq. (22). The horizontal axis shows the normalized time, where one gait cycle is 100%. One gait cycle in Fig. 3 extends from the beginning of the stance phase of the left leg (the third step) until the end of the swing phase. Table 3 shows root mean square errors (RMSE) between the estimated results and the results obtained from the 3D motion analysis system.

Fig. 3
figure3

Left lower limb joint angles during walking obtained using optical 3D motion analysis system, the extended Kalman filter using NBS, NBS (Only observation noise), NBS (Only process noise), and the extended Kalman filter using CNC (participant A). a The ankle joint angle. b The knee joint angle. c The hip joint angle.

Table 3 Root mean square errors for results obtained from two extended Kalman filters and 3D motion analysis system results (participant A)

Joint angles obtained from the three NBSs and CNC are generally consistent with the results obtained using the optical 3D motion analysis system. Results show the same tendency for joint angle variation as that found in an earlier study [35].

The ankle joint angle obtained from NBS (Only process noise) is generally consistent with results obtained using the optical 3D motion analysis system at the dorsiflexion peak after toe-off. The ankle joint angle obtained from NBS (Only observation noise) is much smaller than the result obtained from the optical 3D motion analysis system. The results indicate that the process noise covariance matrix based on the gyroscope output contributed to increased accuracy of the dorsiflexion peak during the swing phase.

In the early stance phase and the end of the swing phase, the ankle joint angle obtained from NBS (Only process noise) is much smaller than the result obtained from the optical 3D motion analysis system, whereas the ankle joint angle obtained from NBS (Only observation noise) is generally consistent with the result obtained using the optical 3D motion analysis system. The results indicate that the observation noise covariance matrix based on the accelerometer and magnetometer output contributed to increased accuracy at the early stance phase and at the end of the swing phase. Therefore, the process noise covariance matrix based on the gyroscope output and the observed noise covariance matrix based on the accelerometer and magnetometer output might have contributed to the increased accuracy at different phases.

For knee and hip joint angles, all results show the same tendency. However, NBS (red line) has the smallest RMSE in all results of all three joints. The results show that using both processes of noise covariance matrix based on the gyroscope output and the observed noise covariance matrix based on the accelerometer and magnetometer output might have contributed to increased accuracy. The two noise covariance matrices seem to have influenced one another.

Running experiment

Participants and experiment conditions

The nine-axis motion sensors measured lower limb joint angles of the same participants while they were running in place to verify the effectiveness of NBS when continuously capturing data of fast-moving participants. The nine-axis motion sensors were placed in the same positions as those used for the verification experiment. The measurement time was about 100 s. During the experiment, kinematic data were collected using an optical 3D motion analysis system with four nine-axis motion sensors in synchronization. Participants were instructed to run in place in time with a metronome (150 bpm) after maintaining the upright posture for about 5 s. The sampling frequencies of the nine-axis motion sensors and the optical 3D motion analysis system were 100 Hz.

Results

Table 4 shows parameters a to f for the running.

Table 4 Adjusting parameters of NBS in the running experiment

experiment, which were determined to maximize the log-likelihood in Eq. (22). From the running experiment, different parameters were obtained among the joints. In addition, parameters a, c, and e for running measurements tended to be larger than those in the walking measurement. The results indicate that the noise covariance matrices for the running experiment might have had larger values because the process and observation noise can increase if the motion velocity increases.

The estimated joint angles of participant A are presented in Figs. 4, 5, and 6. In each of Figs. 4, 5, and 6, panels (a) present results obtained over the entire measurement time. Panels (b) present results obtained between 33 s and 35.5 s from the start of measurements. In each of Figs. 4, 5, and 6, panels (b) are used for a detailed examination of the results. Black solid curves present results obtained from the optical 3D motion analysis system. Red solid curves present results obtained from NBS. Blue solid curves present results obtained from CNC.

Fig. 4
figure4

Results obtained for ankle joint angles (Subject A). a The results over the entire measurement time. b The results between 33 and 35.5 s.

Fig. 5
figure5

Results obtained for knee joint angles (Subject A). a The results over the entire measurement time. b The results between 33 and 35.5 s.

Fig. 6
figure6

Results of hip joint angles (Subject A). a The results over the entire measurement time. b The results between 33 and 35.5 s.

The estimated ankle joint angle using NBS.

in Fig. 4(a) changes periodically between − 25° and 25° over the entire measurement time, which is generally consistent with results obtained using the optical 3D motion analysis system. The estimated ankle joint angle using CNC in Fig. 4(a) changes periodically between − 70° and 0° over the entire measurement time. Although the waveform of the result obtained using CNC in Fig. 4(b) is similar to the result obtained using NBS, the result obtained using CNC is much smaller than that obtained using NBS. Additionally, the waveform of the result obtained using CNC has a larger dorsiflexion peak than that obtained using NBS at about 33.7, 34.4, and 35.2 s.

The estimated knee joint angle obtained using NBS in Fig. 5(a) changes periodically between 20° and 110° over the entire measurement time, which are generally consistent with the results obtained using the optical 3D motion analysis system. Whereas the estimated knee joint angle using CNC in Fig. 5(b) changes periodically between − 60° and 0° over the entire measurement time. Although the waveform of the result obtained using CNC in Fig. 5(b) is similar to the result obtained using NBS, the result obtained using CNC is much smaller than that obtained using NBS. Additionally, the waveform of the result obtained using CNC has a smaller flexion peak than that obtained using NBS at about 33.6, 34.35, and 35.2 s.

The hip joint angle estimated using NBS in Fig. 6(a) changes periodically between 10° and 35° over the entire measurement time, which are generally consistent with results obtained using the optical 3D motion analysis system. The estimated knee joint angle using CNC in Fig. 6(a) changes periodically between − 35° and − 15° over the entire measurement time. Although the waveform of the result obtained using CNC in Fig. 6(b) is similar to the result obtained using NBS, the result obtained using CNC is much smaller than that obtained using NBS. All results obtained for the other two participants showed similar tendencies. The results demonstrated the effectiveness of the extended Kalman filter using NBS.

Conclusions

For this study, a method for ascertaining the process and observation noise covariance matrices in the extended Kalman filter based on sensor output was constructed to estimate the lower limb joint angles. The lower limb joint angles of the three healthy participants during walking and running were estimated using the method. Results yielded the following conclusions.

  1. 1.

    The joint angles obtained from the extended Kalman filter using the process and observation noise covariance matrices based on sensor output were generally consistent with results obtained using the optical 3D motion analysis system in the verification experiment.

  2. 2.

    In the running motion analysis, the results obtained using noise covariance matrices based on sensor output indicated that the estimated joint angles changed periodically within an appropriate range. The results obtained using the constant noise matrices covariance indicated that the estimated joint angles changed abnormally.

Noise covariance matrices based on sensor output can be effective for accurate pose estimation because noise covariance matrices can be time-variable when continuously capturing human motion with long-term measurements. The proposed method is expected to be useful for estimating motion in sports and healthcare applications.

References

  1. 1.

    King K, Yoon SW, Perkins NC, Najaf K (2008) Wireless MEMS inertial sensor system for golf swing dynamics. Sens Actuators, A 141(2):619–630

    Article  Google Scholar 

  2. 2.

    Favre J, Aissaoui R, Jolles BM, Guise JA, Aminian K (2009) Functional calibration procedure for 3D knee joint angle description using inertial sensors. J Biomech 42(14):2330–2335

    Article  Google Scholar 

  3. 3.

    Ameli S, Naghdy F, Stirling D, Naghdy G, Aghmesheh M (2017) Objective clinical gait analysis using inertial sensors and six minute walking test. Pattern Recogn 63:246–257

    Article  Google Scholar 

  4. 4.

    Nüesch C, Roos E, Pagenstert G, Mündermann A (2017) Measuring joint kinematics of treadmill walking and running: Comparison between an inertial sensor based system and a camera-based system. J Biomech 57:32–38

    Article  Google Scholar 

  5. 5.

    Kruk E, Schwab AL, Helm FCT, Veeger HEJ (2018) Getting in shape: Reconstructing three-dimensional long-track speed skating kinematics by comparing several body pose reconstruction techniques. J Biomech 69:103–112

    Article  Google Scholar 

  6. 6.

    Bavan L, Surmacz K, Beard D, Mellon S, Rees J (2019) Adherence monitoring of rehabilitation exercise with inertial sensors: a clinical validation study. Gait Posture 70:211–217

    Article  Google Scholar 

  7. 7.

    Hullfish TJ, Qu F, Stoeckl BD, Gebhard PM, Mauck RL, Baxter JR (2019) Measuring clinically relevant knee motion with a self-calibrated wearable sensor. J Biomech 89(24):105–109

    Article  Google Scholar 

  8. 8.

    Nazarahari M, Noamani A, Ahmadian N, Rouhani H (2019) Sensor-to-body calibration procedure for clinical motion analysis of lower limb using magnetic and inertial measurement units. J Biomech 85(6):224–229

    Article  Google Scholar 

  9. 9.

    Hirose K, Doki H, Koda S, Nagasaku K (2011) Studies of the dynamic analysis and motion measurement of skiing turn using extended Kalman filter. Trans Jpn Soc Mech Eng, Series C 77(774):470–480 ((in Japanese))

    Article  Google Scholar 

  10. 10.

    Chardonnens J, Favre J, Cuendet F, Gremion G, Aminian K (2013) A system to measure the kinematics during the entire ski jump sequence using inertial sensors. J Biomech 46(1):56–62

    Article  Google Scholar 

  11. 11.

    Hirose K, Doki H, Kondo A (2012) Studies of orientation measurement in sports using inertial and magnetic field sensors. Jpn Soc Sports Ind 22(2):255–262 ((in Japanese))

    Article  Google Scholar 

  12. 12.

    Saito A, Miyawaki K, Kizawa S, Kobayashi Y (2018) A study of estimating the knee joint angle during walking using the motion sensors (Focusing on the effect of centrifugal acceleration and tangential acceleration). Trans JSME. https://doi.org/10.1299/transjsme.17-00488 (in Japanese)

    Article  Google Scholar 

  13. 13.

    Saito A, Miyawaki K, Komatsu A, Iwami T (2018) A study of sensor position for thigh and lower leg motion sensors during walking (Focusing on the knee sagittal plane angle). Trans JSME. https://doi.org/10.1299/transjsme.18-00263 (in Japanese)

    Article  Google Scholar 

  14. 14.

    Kondo A, Doki H, Hirose K (2013) A study of calibration method of magnetic field sensor for body motion measurement using inertial sensor. Proc Sports Eng Human Dyn. https://doi.org/10.1299/jsmeshd.2013._212-1 (in Japanese)

    Article  Google Scholar 

  15. 15.

    Saito A, Nara Y, Miyawaki K (2019) A study of estimating knee joint angle using motion sensors under conditions of magnetic field variation. Trans JSME. https://doi.org/10.1299/transjsme.19-00061 (in Japanese)

    Article  Google Scholar 

  16. 16.

    Rigatos GG (2010) Extended Kalman and Particle Filtering for sensor fusion in motion control of mobile robots. Math Comput Simul 81(3):590–607

    MathSciNet  MATH  Article  Google Scholar 

  17. 17.

    Adachi S, Maruta I (2012) Fundamentals of Kalman filter. Tokyo Denki University Press, Tokyo, pp 95–111

    Google Scholar 

  18. 18.

    Ran C, Deng Z (2012) Self-tuning weighted measurement fusion Kalman filtering algorithm. Comput Stat Data Anal 56(6):2112–2128

    MathSciNet  MATH  Article  Google Scholar 

  19. 19.

    Kamil M, Chobtrong T, Günes E, Haid M (2014) Low-cost object tracking with MEMS sensors, Kalman filtering and simplified two-filter-smoothing. Appl Math Comput 235:323–331

    MATH  Google Scholar 

  20. 20.

    Zheng Z, Qiu H, Wang Z, Luo S, Lei Y (2019) Data fusion based multi-rate Kalman filtering with unknown input for on-line estimation of dynamic displacements. Measurement 131:211–218

    Article  Google Scholar 

  21. 21.

    Baerveldt AJ, Klang R (1997) A low-cost and low-weight attitude estimation system for an autonomous helicopter. In: Proceedings of IEEE International Conference on Intelligent Engineering Systems. https://doi.org/10.1109/INES.1997.632450.

  22. 22.

    Mahony RE, Hamel T, Pflimlin JM, (2005) Complementary filter design on the special orthogonal group SO(3). In: Proceedings of the 44th IEEE Conference on Decision and Control, and the European Control Conference 2005. https://doi.org/10.1109/CDC.2005.1582367

  23. 23.

    Euston M, Coote P, Mahony R, Kim J, Hamel T (2008) A complementary filter for attitude estimation of a fixed-wing UAV. In: Proceedings of 2008 IEEE/RSJ International Conference of Intelligent Robots and Systems. https://doi.org/10.1109/IROS.2008.4650766

  24. 24.

    Mahony R, Hamel T, Pflimlin JM (2008) Nonlinear complementary filters on the special orthogonal group. IEEE Trans Autom Control 53(5):1203–1218

    MathSciNet  MATH  Article  Google Scholar 

  25. 25.

    Sugihara T, Masuya K, Yamamoto M (2013) A complementary filter for high-fidelity attitude estimation based on decoupled linear/nonlinear properties of inertial sensors. J Robot Soc Jpn 31(3):251–262 ((in Japanese))

    Article  Google Scholar 

  26. 26.

    Aoki K (2008) The brand-new technology: Motion analysis system. J Soc Biomech 32(3):167–172 ((in Japanese))

    Article  Google Scholar 

  27. 27.

    Ehara Y (2008) The VICON. J Soc Biomech 32(2):107–111 ((in Japanese))

    Article  Google Scholar 

  28. 28.

    Syam PN, Sheila G, Graham A, Rami A, Weijie W (2010) A method to calculate the centre of the ankle joint: a comparison with the Vicon® Plug-in-Gait model. Clin Biomech 25(6):582–587

    Article  Google Scholar 

  29. 29.

    Stief F, Böhm H, Michel K, Schwirtz A, Döderlein L (2013) Reliability and accuracy in three-dimensional gait analysis: a comparison of two lower body protocols. J Appl Biomech 29:105–111

    Article  Google Scholar 

  30. 30.

    Vaganay J, Aldon MJ, Fournier A (1993) Mobile robot attitude estimation by fusion of inertial data.In: Proceedings of the IEEE International Conference on Robotics and Automation. https://doi.org/10.1109/ROBOT.1993.291995

  31. 31.

    Jurman D, Jankovec M, Kamnik R, Topic M (2007) Calibration and data fusion solution for the miniature attitude and heading reference system. Sens Actuators, A 138(2):411–420

    Article  Google Scholar 

  32. 32.

    Hirose K, Kondo A (2014) Special issues no. 3: measurement technique for ergonomics. Jpn J Ergon 50(4):182–190 ((in Japanese))

    Google Scholar 

  33. 33.

    Miyamoto G, Sonobe M, Shibata K, Hirose K (2019) Development of gait measurement system using IMU sensors. Proc Chugoku and Shikoku Reg Conf . https://doi.org/10.1299/jsmecs.2019.57.1001 ((in Japanese))

    Article  Google Scholar 

  34. 34.

    Ae M, Tang H, Yokoi T (1992) Estimation of inertia properties of the body segments in Japanese athletes. Biomechanism 11:23–33 ((in Japanese))

    Article  Google Scholar 

  35. 35.

    Yamamoto H, Yanagida Y (2011) The various patterns of knee angle in the stance phase. Soc Phys Ther Sci 26(2):269–273 ((in Japanese))

    Google Scholar 

Download references

Acknowledgements

Not applicable

Funding

This work was supported by JSPS KAKENHI Grant Number JP 20K19588.

Author information

Affiliations

Authors

Contributions

AS conceived the study and drafted the manuscript. AS and KM carried out all experiments and analyzed the data. YK and SK participated in the research design and sequence alignment. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ayuko Saito.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Saito, A., Kizawa, S., Kobayashi, Y. et al. Pose estimation by extended Kalman filter using noise covariance matrices based on sensor output. Robomech J 7, 36 (2020). https://doi.org/10.1186/s40648-020-00185-y

Download citation

Keywords

  • Kalman filter
  • Motion sensor
  • Noise covariance matrix
  • Pose estimation
  • Sensor fusion