Modification of body schema by use of extra robotic thumb

Abstract

In recent years, there has been great interest in the possibility of using artificial limbs as an extension of the human body as well as replacement of lost limbs. In this paper, we develop a sixth finger system as an extension of the human body. We then investigate how an extra robotic thumb, that works as a sixth finger and gives somatosensory feedback to the user, modifies the body schema, and also affecting the self-perception of existing limbs. The sixth robotic finger is controlled with the thumb of the opposite hand, and contact information is conveyed via electrostimulation to the tip of the thumb controlling the movement. We conducted reaching task experiments with and without visual information to evaluate the level of embodiment of the sixth robotic finger and the modification of the self-perception of the finger controlling the system. The experimental results indicate that not only the sixth finger is incorporated into the body schema of the user, but also the body schema of the controlling finger is modified; ability of the brain to adapt to different scenarios and geometries of the body is also implied.

Introduction

Artificial limbs had been used for long time as prostheses, and it has been studied how much are these artificial limbs embodied to the users self-perception of the body [1]. It has been shown that an increase in embodiment improves the performance and comfortability of these artificial limbs [2]. In recent years, great interest has arisen on the possible use of artificial limbs not just as replacement of lost limbs, but also as an extension of the human body. Some research groups have studied the use of artificial limbs as extra limbs, and their possible applications [3, 4].

Extra limbs offer the possibility to increase workspace, dexterity, strength and reduce fatigue of the user [5]. However, extra limbs require new control strategies because extra limbs perform actions that the users have not experienced. Control strategies of extra limbs proposed previously can be divided into two approaches. First approach is the indirect control from the user by moving the artificial limbs by synergy with other movements from the user [6] or predicting the intended movement [7]. Second approach is the direct control from the user mimicking the motion from a similar limb [8].

It has been shown that artificial limbs can be incorporated in the body schema of users when replacing lost limbs as prostheses [9], or as supernumerary limbs [10]. Specially, the use of somatosensory feedback increases the sense of embodiment [11] and performance on grasping motions [12]. This embodiment means that the user can have an accurate idea of the position of the artificial limb without visual information.

One example of these possible extra limbs are supernumerary fingers [8]. That have been studied to increase grasping range and ability to perform activities with one hand [13]. They have been also studied to help the recovery of stroke patients with somatosensory feedback [14] and without it [15].

Although there are several approaches to control devices of extra limbs, it has not been elucidated how these different control strategies affect the embodiment of the artificial limbs, or modify the body schema of the users [2]. In this paper, we focus on the effect of controlling a supernumerary limb with direct control from the motion of a similar limb with somatosensory feedback, and its effect on the body schema of the users.

We develop a sixth finger device system as an extension of the human body, a thumb motion capture device to control movements of the device, and a electrical stimulation device to convey contact information. We then perform experiments using the sixth finger device with somatosensory feedback during 1 month with varying rest time between sessions, with and without visual feedback. In addition, the sixth finger device is controlled by mirroring the movement of the thumb from the opposite hand, while giving somatosensory feedback from contact information via electrostimulations. From the experimental results, the performance on a reaching task and the modification of self-perception of the controlling finger after each session are evaluated. Finally, we discuss about ability of the brain to adapt to different scenarios and geometries of the body though the evaluations.

Body schema

Body schema was firstly put forward by Holmes and Head in 1911 [4]. Humans can perceive intuitively the position and movement of our own body without looking because experience about our own body has been accumulated as a mental model over time. The body schema is the map of our own body generated based on the previous experience. When we move or perceive our posture, the body schema is referenced unconsciously. The body schema has various properties [16]. Adaptability is one of these properties of the body schema. For example, when we use tools, the body schema will be changed and take the tools into consideration [17]. We can use tools dexterously owing to this property. When we stop the use of these tools, the body schema will be restored to its initial state. We call embodiment when the prolonged use of these tools writes them strongly in the body schema.

Supramodal representation is also one of the important properties of the body schema. The body schema integrates a variety of sensory feedbacks, usually called afferent input, such as position sense, tactile sense obtained from peripheral nerves. This property makes us possible to identify the position stimulated without looking at it directly. Moreover, the body schema generates an efferent copy when we perform an activity [18]. Efferent copy is an expected image generated before we move. We can expect the result of the movement before we actually move. The body schema associates afferent copy with efferent copy and the difference is used to update the body schema. The difference between the performed motion and the expected motion is inversely proportional to the accuracy of the efferent copy and body schema.

Somatosensory feedback has been shown to be an important factor in the performance of different interfaces [19] and artificial limbs [20]. In this study, we provide pseudo-tactile feedback using electrical stimulation. The use of electrical stimulation has been shown to simulate contact and tactile feedback [21]. This pseudo afferent input increases the extra robotic thumb embodiment. As a result, a more accurate efferent copy will be generated, and operability of the extra robotic thumb will be improved.

Extra robotic thumb system

Extra robotic thumb (ERT)

Figure 1 depicts a new developed extra robotic thumb (ERT) that is a super numerary limb device used to simulate an extra finger on the hand. The ERT attaches on the ulnar side of the palm to produce the feeling of having a second thumb on the hand. The ERT is located at the point so that the robotic thumb mirrors the position of the real thumb in the opposite side of the hand.

The ERT is designed to resemble a thumb, in its size, movement and degrees of freedom. It consists of three links and an attachment to the hand. The robotic thumb is made of ABS resin and total weight of the device is 61 g. The movable range of the robotic thumb is enough to touch five fingertips of the hand. Denavit–Hartemberg parameters of the kinematic chain are shown in Table 1. The base reference frame is located at skin level on the ulnar side of the palm. Three servomotors (JR Propo, DS318) are attached at each joint. The maximum torque of the servo motor is 1.8 kgfcm, and it is controlled via an Mbed microcomputer. The ERT is capable of reaching 3.2 N of force for grasping actions by using these servomotors. A pressure sensor (Optoforce, OMD-20-SE-40N) is installed at the tip of the finger in order to feedback contact force information to the users.

Fig. 1

Extra robotic thumb

Table 1 Denavit–Hartemberg parameters of extra robotic thumb [22]

Thumb motion capture

The TMC is a device attached to the posterior side of the right hand and the right wrist, and it is used to capture the movements of the right thumb. Figure 2 depicts the overview of the device. This device is designed to follow the movements of the thumb, and record the position of the tip of the thumb with respect to its base. It is designed in such a way to not interfere with the movements of the thumb.

The TMC consists of a base attached to the back of the hand and a 4-link mechanism to follow the movement of the thumb without perturbing its movement. The Denavit–Hartemberg parameters of the kinematic chain are shown in Table 2. A base reference frame is located on the carpometacarpal joint of the thumb. Each joint of the TMC device has a potentiometer to measure the angle of each joint to calculate the position of the tip of the thumb with respect to the hand. The position of the tip of the thumb is then send to the Mbed microcomputer that controls the movement of the ERT.

Fig. 2

Thumb motion capture

Table 2 Denavit–Hartemberg parameters of thumb motion Capture [22]

Electrical stimulation device

Surface electrical stimulation has been shown to be a reliable method to feedback tactile information to users [21]. Electrodes can be very light and easily attached to the location where the stimulation is desired without interfering with the activity of the user. In this study, we attached the electrodes to the tip of the right thumb. Electrical stimulation activates several skin mechanoreceptors depending on the frequency and intensity of the stimulation. With the right parameters, a user can easily and comfortably discriminate between presence and absence, or strength of the stimulation. The developed electrical stimulation system consists of an electric stimulator, a current amplifier, an I/O board, a signal multiplexor, a switching circuit, and electrodes as shown in Fig. 3.

Human skin has four main types of mechanoreceptors under the skin: Pacinian corpuscles, Meissners corpuscles, Merkels discs, and Ruffini endings. This electrical stimulation device system stimulates mainly Meissners corpuscles, which have a role in perceiving tactile information. The frequency of the stimulation is 50 Hz to effectively stimulates Meissners corpuscles [23]. Maximum applied current is 10 mA, and stimulation is provided in pulses of 1% duty cycle.

Fig. 3

Electrical stimulation device

Control method

Figure 4 shows the whole control architecture of the system. Users attach the thumb motion capture and the electrical stimulation electrodes to the right thumb to control the system.

Fig. 4

Control diagram of the sixth finger system

The angle information of each joint is used to estimate the position (x,y,z) of the thumb tip with respect to its base using Eq. 1, with the use of the Denavit–Hartemberg parameters of the TMC. This position is then used to move the extra robotic thumb in a similar fashion by calculating the angles ($\theta _1, \theta _2, \theta _3$) that would solve Eq. 2 with the ERT parameters. Thus, the tip of the robotic thumb is the same position relative to its base as the right-hand thumb is. In addition, contact force information is conveyed to the fingertip of the right thumb via electrical stimulation. Thus, the user is aware of when the robotic thumb tip makes contact with something and the force of this contact.

$$\begin{aligned} \begin{bmatrix} x \\ y \\ z \\ 1 \\ \end{bmatrix}= & {} \prod _{n=1}^{5} \begin{bmatrix} {\text{cos}}(\theta _n)&-{\text{sin}}(\theta _n){\text{cos}}(\alpha _n)&{\text{sin}}(\theta _n){\text{sin}}(\alpha _n)&r_n{\text{cos}}(\theta _n) \\ {\text{sin}}(\theta _n)&{\text{cos}}(\theta _n){\text{cos}}(\alpha _n)&-{\text{cos}}(\theta _n){\text{sin}}(\alpha _n)&r_n{\text{sin}}(\theta _n) \\ 0&{\text{sin}}(\alpha _n)&{\text{cos}}(\alpha _n)&d_n \\ 0&0&0&1 \\ \end{bmatrix}\end{aligned}$$

(1)

$$\begin{aligned} \begin{bmatrix} x \\ y \\ z \\ 1 \\ \end{bmatrix}= & {} \prod _{n=1}^{3} \begin{bmatrix} {\text{cos}}(\theta _n)&-{\text{sin}}(\theta _n){\text{cos}}(\alpha _n)&{\text{sin}}(\theta _n){\text{sin}}(\alpha _n)&r_n{\text{cos}}(\theta _n) \\ {\text{sin}}(\theta _n)&{\text{cos}}(\theta _n){\text{cos}}(\alpha _n)&-{\text{cos}}(\theta _n){\text{sin}}(\alpha _n)&r_n{\text{sin}}(\theta _n)\\ 0&{\text{sin}}(\alpha _n)&{\text{cos}}(\alpha _n)&d_n \\ 0&0&0&1\\ \end{bmatrix} \end{aligned}$$

(2)

Experiment

Experimental setup

We conducted experiments of a finger reaching task with and without visual feedback from the movement of the ERT in order to evaluate the level of embodiment of ERT.

The task consists on asking the user to move the ERT to touch the tip of one of four fingers of the left hand (index, middle, ring or pinkie). The finger to touch is randomly selected and presented to the user in a screen located in front of the user Fig. 5. Once the user moves the ERT to touch a finger, a new target is presented to the user, and a hit is recorded if the user touched the right finger, or a miss if any of the other three fingers are touched. One round lasts 30 s, and users were asked to try to obtain as many hits as comfortably possible during this time. Four male subjects (average age 24 years, variance 1.6), were divided into two groups randomly. First group can see their left hand and the ERT during the experiment. Second group had its vision obstructed on the left side, and the subjects of the group were not able to see their left hand during the experiment. Both groups had its vision obstructed on the right side, and both group were not able to see their right hand during the experiment as shown in Figs. 6 and 7. Experiments were performed in sessions of 10 rounds during a period of one moth with varying times between sessions. Since subjects were not able to see their right hand during the experiment, we measured any change in perception of location of the right thumb by asking the subjects to try to point the right thumb tip with the left index finger while their eyes were closed, before and after each session.

Fig. 5

Screen shot of the interface that the subjects see during experiment

Fig. 6

First group condition, possible to see left hand

Fig. 7

Second group condition, cannot see left hand

Experimental results of reaching task

The experiment was performed five times with rest times of 1 day between sessions, then four times with rest time of 1 week, and finally one last time, after 1 month had passed since last session, in order to see if progress is affected, or performance ability forgotten over time.

Figure 8 shows the number of hits and misses of both groups. It is possible to see that performance of both groups increases with practice and reaches a steady performance after a couple of weeks. The performance of the group that was not allowed to see their left hand is slightly better by the end of the experiment. Reach count on the last session is 4% higher for the obstructed vision group. Which indicate a stronger sense of embodiment of the device.

Fig. 8

Number of correct and incorrect fingers touched in each of the 10 rounds per session for both groups. In gray is marked the rest time between sessions

Experimental results of proprioceptive drift

The subjects were asked to point, as closely as possible, the position of the right thumb tip with the left index finger while having the eyes closed. The subjects were asked to perform this activity before and after each session to see if the use of the sixth finger device had any effect in the self-perception of the subject.

Figure 9 shows the proprioceptive drift of the distance from the pointed location to the tip of the thumb after each session to the distance of the pointed location to the tip of the thumb before each session. The proprioceptive drift is always positive meaning that the pointed location is always further from the tip of the thumb after each session. We believe the results caused by modification of the subjects body schema by relating the action of the right thumb to the movement of the sixth finger in the left hand. This may indicate a transfer of self-perception from the right thumb to the extra finger analogous to the rubber hand illusion, but actively created by the users actions.

Fig. 9

Proprioceptive drift given by the difference between the distance of the pointed location where the subject believe their right thumb tip is located before and after each session

Discussion

The experimental results of reaching task indicate that the number of channels and type of feedback is an important factor on the level of embodiment that a supernumerary limb device can achieve in a set amount of time. From previous results [8] we know that the presence of somatosensory feedback is an important factor for the performance of this kind of devices. We assume that the level of embodiment is directly related to the performance in reaching tasks, via the accuracy of the efferent copy of the body schema of the system user-device. If the movements predicted by the feed forward model of the user match the real actions of the sixth finger, then the user will be able to reach more targets in a given time. In our experiments, the group without visual feedback from the sixth finger had an slightly better performance, pointing that they had to rely less on the visual information, and more in the tactile information, which would have helped in creating a better efferent copy of the sixth finger system.

The experimental results of the proprioceptive drift indicate that the use of direct control to drive a supernumerary limb affects not only the creation of a new body representation of the supernumerary limb, but also modifies the existing body representation of the controlling limb. In this case, the self-image of the position of the fingertip is modified during the course of the experiment, pointing to the plasticity of the brain to modify the existing body schema.

There is a light correlation between performance and proprioceptive drift, which could indicate that the process of creating the new body representation of the sixth finger to the movement of the right thumb is related to the accuracy of the right thumb body representation. This could be caused by the brain using the efferent information of the right thumb in creating the efferent copy of the sixth finger, and not in its movement, degrading the efferent copy of the real thumb.

The decrease in accuracy in the body representation of the right thumb is lower, than the increase of performance of the sixth finger. Meaning that the brain is able to adapt to a more complex body schema, introducing new limbs without seriously sacrificing existing parts of the body representation. The limit of the complexity of the body schema achievable is still an open question to be researched.

It has been studied that patients recovering from cervical injuries [24] or stroke [25] can regain part of the body schema lost due to injury or disease [26]. It has also been studied the use of extra robotic fingers to help patients recovering from stroke [27]. Our results indicate that it might be possible to help these kinds of patients to achieve a new body schema by sacrificing other part of the body schema, specifically, part that is no longer in use.

Conclusions

We presented a sixth finger system composed of an extra robotic thumb attached to the left hand, that simulates a second thumb in that hand and has a force sensor in its tip; a thumb motion capture, that attaches to the right hand, and captures the movements and position of the tip of the thumb in that hand, to mirror its movements into the extra robotic thumb. And a electrostimulation device that provides pseudo tactile feedback from the force sensor on the tip of the robotic thumb to the tip of the right thumb.

We showed the performance of this sixth finger device with somatosensory feedback measured via a pointing task by comparing two groups of subjects; the subjects of the first group were allowed to see the sixth finger device during the test, and the subjects of second group were not allowed to see the sixth finger device during the test. The movement of the device was controlled by the opposite hand thumb and the contact information was conveyed via electric stimulation from the tip of the extra robotic thumb to the tip of the finger controlling the movement. The experimental results show the group without visual feedback had a better performance than the group with visual feedback; this indicates that subjects of this group achieved better embodiment of the system in the user’s body schema.

In addition, we measured the proprioceptive drift of the controlling thumb by asking the user to point the location of the tip of the right thumb, before and after the experiment. The results show that the proprioceptive drift is always positive meaning that the accuracy is always lower after the experiment. This results also indicate that the sixth finger is not just embodied, but also controlling the device in a direct manner modifies the controlling thumb schema; there is a light correlation between performance and proprioceptive drift. It is also suggested that the ability of the brain to adapt to different scenarios and geometries of the body even those with supernumerary limbs that were not present before, and the body schema of existing limbs can be modified via somatosensory feedback. The body schema will be modified in a way that resembles transfer of embodiment from the controlling limb to the new supernumerary one by using the efferent information from one movement to control a new limb, and providing afferent information from the actions of the new limb.

Proper embodiment of artificial limbs means that users can control them without visual feedback, and embodiment of extra limbs increase its usefulness and usability as extensions of the human body. This technique can be used to extend the body schema of the user with the addition of supernumerary limbs or help recovery patients to regain lost body schema.

As a future work the effect and comparison of different kind of interfaces on the embodiment of these kind of systems should be tested. Furthermore, usability cases of these kind of devices in daily life, work or medical treatments could be studied. Also the level of embodiment achievable by using extra robotic limbs that do not have a geometry similar to human body remains to be researched.