Origami manipulation by robot hand utilizing electroadhesion

Abstract

This study presents strategies for the three fundamental origami operations of grasping, bending, and folding using a novel robot hand and simple motions. These operations are executed using a simple geometric model and without any visual feedback or physical modeling not to restrict the motions. With a few applications in the field of paper manipulation, the electroadhesion technology is employed to perform single hand grasping. Bending is realized by a single hand utilizing the elasticity of origami and friction. Folding is performed by holding an origami with more than two points to fix it at any moment for preciseness. In addition to the design of hardware and motions, operations are demonstrated with general criteria for the crease precision evaluation.

Introduction

Unlike rigid objects flexible ones are difficult to handle for robots because they change shape, making it difficult to estimate state of the object. In particular, robotic paper folding is expected to be applied for wrapping [1] or building origami cartons [2] in addition to just performing origami as an entertainment. However, compared to cloth folding [3,4,5,6,7,8], the task is difficult because paper is lighter and thinner than fabric and having bending stiffness. Due to the stiffness, operation for making a crease, which is not seen in cloth manipulation, is needed to set a permanent deformation. Considering the situation, we decided to design a novel robot hand and its use for origami operations and set scope to grasping, bending, and folding because most of creases can be made by combining them. Indeed, five basic shapes of origami shown by Zhu [9] can be made by combining these manipulations.

The existing works in the field of origami manipulation are either too specific or have restrictions in terms of motion. Some works employed a mechanism or a motion that is highly customized for a specific task, which limit the kinds of origami work it produces or make it inefficient in adapting to new work. Others frequently used bimanual operations, which need a careful consideration to avoid robot arm interference. Some reports in the literature used visual feedback or a physical model with state estimation by computer vision, which continuously restricts the motion of the robot hand not to hide the origami. Considering these issues of existing robot systems, we design a widely usable robot hand and realize the origami operations of grasping, bending, and folding withdrawing the limitations of motion.

The main study contributions are proposal of a novel robot hand and the presentation of strategies for realizing the origami tasks of grasping, bending, and folding using this robot hand. As an important feature, our grasping and bending motions are completely performed by a single hand. The presented folding motion stably holds the origami, such that a crease can be made precisely. These motions are executed with minimum recognition steps; the motions are not greatly restricted. Our robot hand serve as a valuable application case of electroadhesion for dexterous paper manipulation. A novel method for installing the device to the rigid fingertip keeping its flexibility was taken in fabrication process of the hand. In addition, we propose herein the general criteria for evaluating the crease precision, regardless of the origami size or the sort of crease, to serve as the shared benchmarks for origami-folding robots.

Related works

There are several works involving origami manipulation using robots.

Grasping

Some works utilized two robot hands for grasping. Some commonly used one hand to shift a part of the origami in the air and another for grasping. In the work of Elbrechter et al. [10], a robot hand slid the paper to the end of the table until a corner protruded from it and the other hand grasp the corner. Namiki et al. [11] used a robot hand to hold a part of the origami and other to hold and slide another part of the origami, narrowing the gap between the hands. The free part of the origami between the hold area made an arc, in which a finger of the robot hand can go under for grasping. Suzuki et al. [12] realized grasping using supplemental robot arm to press the center of the origami on a flexible workbench, make a moderate volley in the origami, and lift the corners enough to be grasped. The abovementhioned methods commonly encounter the risks of interference, especially for a small piece of paper. They also have the position limitation for grasping the paper.

Balkcom et al. [13] employed a suction cup for grasping. However, suction needed a tube to control the pressure in the cup, making the fingertip larger and potentially disturbing dexterous origami operations. Suction also seemed to fail in sticking a sheet of paper from the layered stack.

Another possible way for grasping is use of electroadhesion. Electroadhesion is a phenomenon where electric field made by voltage gap between a pair of electrode generate force to attract object nearby. Since electroadhesion devices themselves cannot move, the devices are typically combined with soft actuators such as pneumatic actuators [14,15,16] or dielectric elastomer actuators [17, 18]. Though demonstration of paper sticking is popular among studies of electroadhesion [15,16,17, 19], only a few devices are specialized for paper operations and such devices are too large to be installed to a fingertip of the gripper. For example, electrode for flipping pages of books developed in Lee et al. [20] have almost the same width to longitudinal side of the page. Also, document-sorting robot of Itoh et al. [21] have an electroadhesion pad which cover A4 paper. In addition, the device can expand function other than sticking objects. For instance, pair of electrode in the device can also be used for capacitive sensor [19] or materials of the device would be chosen to have desired plasticity [22].

Grasping approach similar to ours is employed in Digumarti et al. [23] using electroadhesion pad with rigid gripper for fabric manipulation where the electroadhesion device on a finger stuck and lift a part of fabric and another finger clamped it to make a grasp. However, the gripper needed room for avoiding interference with the workbench and some of its parts going below the pad during the adhesion.

Bending

Tanaka et al. [24] proposed a manual teaching method for the successful bending of origami using a hidden Markov model. However, a human has to teach the bending motion for a few dozen of times for each condition; hence, the method was considered time consuming. Namiki et al. [11] and Suzuki et al. [12] employed visual feedback to adjust corners of origami to be placed in the same position. This visual feedback restricted the position or posture of the robot hands to put the origami in sight, especially when used with a small piece of paper. Learning-based approach utilizing physical simulation for training is employed in Tong et al. [25] for paper bending with a single gripper restraining slippage of the paper. The trained model is robust against change in paper size or stiffness. However, the model cannot deal with creases not perpendicular to a symmetric axis.

Folding

Balkcom [13] proposed a simple tool for robots, which can fold paper using gulf and blade. However, the tool failed to perform complex folding tasks, such as making a partial crease for a designated line. Sakata et al. [26] introduced customized tools to cope with each step of making an origami crane. The hardware seemed too specific for adapting to other sorts of origami work. Namiki et al. [11] and Elbrechter et al. [10] presented a similar method of folding, in which a robot hand sweeps the paper on the desired crease. However, this method needed consideration and control on the holding force of the finger and the friction between the finger and the paper, which can make motion designing difficult.

Recognition and estimation of the paper state

An appropriate origami operation necessitates the recognition, estimation, or simulation of the origami state. Elbrechter et al. [10] utilized a physical model for estimation, which was synchronized to the real origami through the position of points captured by a fiducial marker. Similarly, Namiki et al. [11] employed a physical model updated by the actual position of the four corners of the origami. These estimation methods needed continuous updates by sight, thereby potentially limiting the robot hand motion.

Hardware

Fig. 1

System overview and elements: a overall setting, wherein robot arms are arranged to be mirror-symmetric; b robot hand by a single servo motor; c fingertip structure (i.e, aside from being fixed, the electroadhesion device can flexibly deform); d electroadhesion device layers

Requirements for the robot system

As mentioned in Sect. 2, the existing robot systems performing origami works do not achieve simple motions. The use of tools for each specific task or the limits concerning the interference or interaction with the camera is not desirable considering the many kinds of origami work. Therefore, we tackled the problem from the hardware design perspective.

Next, we determined the three requirements that robot system should satisfy for these operations; (I) realizing the three fundamental origami operations with the least restriction; (II) having the least amount of potential interference cases; and (III) motion of fingers that can easily be imitated by a human hand. Requirement (I) was for origami operations to be performed in various of conditions. Requirement (II) was set for making motion designing processes simple by withdrawing consideration on interference. Requirement (III) was added for the instant validation of a new motion before implementation, which accelerate the motion designing process. For example, Tanaka et al. [27] checked that the designed shape of parts and the motion strategy correctly worked using a mock-up before making the parts. Although the validation was conducted after the production of the hand in our case, we utilized the idea to examine the robot hand motion. Requirement (I) and (II) will also be considered in motion designing process described in Sect. 4.1.

Overview of the robot system

Our robot system with features listed in Sect. 3.1 is shown in Fig. 1. Fig. 1a illustrates that the robot system has a pair of 7-DOF, a Mitsubishi PA10 robot arm, wherein the robot hand we made was set to the wrist, the workbench was placed between the arms, and a 3D camera was installed to recognize the paper above them. The arms were arranged to be mirror-symmetric. Fig. 1b depicts a robot hand on the arm’s wrist. Fig. 1c shows a fingertip of the robot hand. Fig. 1d explains the structure of the electroadhesion device installed on the end of the fingertip. The arms, robot hands, electroadhesion devices, and camera were controlled by Python2.7 and OpenRTM-aist. This robot system had the following features; (i) a pair of robot arms have sufficient degrees of freedom (DOF) and size; (ii) a 1-DOF robot hand that can perform origami grasping, bending, and folding; (iii) a fingertip equipped with an electroadhesion device; and (iv) almost 5 cm long fingers of the robot hand that can open and close. Feature (i) was derived not to restrict the origami operations by the range of motion and to fill requirement (I). Feature (ii) was directly corresponding to requirement (I) by abolishing customized tools for a specific task, while the least DOF led to the compact hardware for requirement (II). Feature (iii) realizes a grasping motion with a single hand and subsequently related to (II). Feature (iv) is for requirement (III).

Robot hand

An approximately 25 cm long end-effector was placed at the end of PA10. The effector comprised a medium-density fiberboard (MDF) board and a pair of pliers that open and close, driven by a DINAMIXEL MX-106 servo motor, and acted as a 5 cm long finger with a rotary joint. One plier handle was actuated through a smaller gear connected to the motor and a larger gear fixed to the handle. The other handle was rigidly fixed to the robot hand. The pliers had an angled edge; hence, the hand had to be tilted to make the edge horizontal while performing the folding motion described in Fig. 10. This was advantageous in avoiding the interference because the hand posture kept the wrists of the two robot arms apart. A high-voltage source for electroadhesion (Sect. 3.4) was inside the hand. This source was insulated by a housing composed of MDF and an acrylic plate to prevent electricity from leaking and harming the robot arm or the servo motor.

We also designed a fingertip shape suitable for grasping and folding paper (Fig. 2).

Fig. 2

Fingertip shape. Grooves were designed for installation to the robot hand. The fingertip had surfaces for grasping and electroadhesion

The most important role the fingertip plays is the realization of electroadhesion and grasping. The tilted surface for electroadhesion device in Fig. 2 enabled the hand to conduct the grasping motion without interference to the workbench, which will be described in Sect. 4.2. This is a distinct advantage to the gripper of Digumarti et al. [23] where room for interference avoidance is needed due to the electroadhesion pad on the clamping surface. As for grasping, we designed the antislip surface shown in Fig. 3a. However, we realized that the surface cannot stop the grasped paper from rotating. Therefore, the shape shown in Fig. 3b was adopted, successfully preventing the paper from rotating.

Fig. 3

Comparison of the fingertip shapes: a initial shape, wherein the surface for grasping is like a square, and the paper can easily rotate; and (b) revised shape, wherein the surface for grasping is elongated to be tolerant of the torque, such that the paper cannot rotate. The demonstrations were conducted with the robot hand grasping origami and the author trying to rotate the paper by the hand

Electroadhesion device

The fingertip was equipped with the electroadhesion device with structure shown in Fig. 1d. This device was made by combining the four layers: base, electrode, insulator, and antislip. The base layer was acrylic foam, VHB 4905, cut by laser and used to fix the base of the device to the fingertip. This layer also worked as an insulator between the electrode and the fingertip. The electrode layer comprised two electrode sheet pieces of ARcare (Adhesive Research, Inc.) laser-cut into a comb shape. When a high-voltage is applied to the electrodes, they store electric charge like a capacitor. The insulator was made of polyimide, Kapton, that worked as an insulator between the electrode and the object to stick. The antislip was employed considering poor friction of Kapton. The antislip was composed of silicon EcoFlex 00–30 (SMOOTH-ON) and silicon adhesive TSE387-C at a ratio of 1:1 by weight. Isooctane was added to the mixture to make this layer thin and smooth. Although the layer structure was inspired by that in the work of Shintake et al. [17], we changed some layers and materials. They proposed an electroadhesion device that can also act as an actuator. We did not expect the device to actuate; therefore, we removed the layers involving the actuation. As for the materials, we had to use high-voltage electricity to obtain sufficient adhesive force. Thus, we could not use silicon for the insulator and substituted it with VHB and Kapton. The electrode was changed to ARcare considering the combination to the insulators. We added an antislip layer made of silicon to cover the lower friction of Kapton.

We had to design the electrode carefully for stable sticking of the paper. According to Shintake et al. [17], the longer the edge length of the electrode facing the counterpart, the stronger the device attract the object. Therefore, we increased the edge length by employing comb-shaped electrodes for the device. The electrode width and the gap between them was set to 0.5 mm each. The former was determined by the limitation of the laser cutting process, where a narrower electrode was dramatically deformed due to heat. The latter was experimentally determined considering that a narrower gap produces a stronger electric field for a more limited depth, which was pointed out by Shintake et al. [17].

We also tried to make the device wider to increase the edge length by expanding it and substituting it with the rubber in Fig. 1c. In that plan, the protruded area of the device will be pinched by the fingertips and act as an antislip of the gripper. However, we found that the device cannot endure the grasping pressure applied by the fingertips, resulting in short-circuited electrodes. Therefore, we amended the device not to be pinched by the fingertip to avoid a short circuit.

Fig. 4

Procedure for setting cables to the electroadhesion device. On the electrode of the fabricated device (a), the cables were installed (b) and wrapped by the base layer (c). Each joint was then quickly enveloped by a heat-shrinkable tube

Fig. 5

Installation of the electroadhesion device to the fingertip. The small piece between the device and the fingertip plays an important role in maintaining the adhesion force while the grasping motion is being executed

The device was applied with up to 5 kV voltage to obtain a sufficient adhesion force. Therefore, when installed to the robot hand, the device must be insulated appropriately to protect the robot hand and the arm while keeping the flexibility. Considering this situation, we adopted the procedure shown in Fig. 4 to attach cables to the device, where each cable was wrapped with an electrode sheet by a base layer(VHB). After the wrapping, a heat-shrinkable tube was set to stabilize the joint.

The fabricated devices were installed to the finger with a narrow rectangular piece of VHB between them (Fig. 5). With some trials in the initial designing process, we found that fixing all the base layer surfaces of the device to the fingertip would impede the adhesion. The device adhered to the paper with electrostatic force; thus, when the gap between the electrode and paper increases, the adhesion force rapidly decreases. This would be caused by the direct decrease of the Coulomb force and the decrease in the dielectric polarization charge in the paper. To address this issue, we employed the small piece to allow the device to follow the paper deformation and prevent the paper from moving during the grasping motion.

The installed electroadhesion device can effectively peel only a sheet of the layered paper as shown in Fig. 6. The below layer is stuck weakly by electrostatic force compared to the top one due to increased distance to the device. This characteristic is advantageous for origami operations such as opening folded paper.

Fig. 6

Electroadhesion device that only takes the surface layer of the folded paper

A control circuit was adopted to control the output voltage of the DC/DC converter Q50–5(XP Power) applied to the device. The device was so fragile that it can suffer from a short circuit when applied with 5 kV of voltage, which was the maximum voltage, in a moment. The voltage must be gradually increased. Considering that the output voltage is proportional to the input, we employed PWM control with a transistor for the input. We experimentally found that the voltage can be safely increased at a 0.5 kV rate per half second step by step.

Other elements

A 3D camera, called Ensenso N45-602-16-IR(optonic), was used for the paper recognition. Aside from returning the point cloud of objects, the camera can produce IR gray-scale images. As an interesting feature, some colors of the origami seem brighter than that of the gray-scale from the RGB camera, which was advantageous for a uniform origami distinguishing by brightness binarization.

The camera uses IR flash to project specific shadow pattern for executing stereo matching to get point cloud. Therefore, we set the camera in such angle that the flash would not return to it directly. Otherwise, the reflected flash hides the shadow patterns and interferes with the stereo matching.

Workbench of the robot system was made of a 30 mm-thick sponge with a sheet of antislip on it. Thanks to the workbench flexibility, the robot hand can keep contact with the origami by designating the height objective slightly below the top of the workbench without a careful adjustment of the height or force feedback.

Origami operations

Strategies for motion design

Using the hardware introduced in Fig. 1, we implemented the origami grasping, bending, and folding operations. For the implementation, we considered the requirements listed in Sect. 3.1, which request to achieve origami operations with the least restrictions and potential interference. For requirement (I), we decided to avoid using any visual feedback loop or physical model for the recognition or estimation of the origami state because they potentially restrict motion, as pointed out in Sect. 2. We instead employed a geometry-based approach for the motion planning. Under this condition, the robot system will be able to recognize the paper state at a limited number of steps and will execute the operations with the step state. Therefore, the precision of the outcome must not displace the paper from a recognition step to the next. As for requirement (II), we decided to conduct grasping and bending motions only by a single hand so that no consideration on interference between the robot arms is needed.

We also attempted to realize the fundamental operations with respective objective. Grasping is completed by only a single hand utilizing electroadhesion. Bending is completed by only a single hand by considering paper elasticity. Folding is executed by holding the paper at least two points in any moment of motion.

Grasping

The grasping motion using an electroadhesion device was executed using the following procedure (Fig. 7):

• 1 Move the robot arm above the desired point for grasping.

• 2 Open the end-effector. Make the electroadhesion device parallel to the paper (Fig. 7a).

• 3 Softly touch the paper with the electroadhesion device (Fig. 7b).

• 4 Rotating the end-effector, lift the paper (Fig. 7c).

• 5 Close the end-effector (Fig. 7d).

Fig. 7

Flow of the grasping motion

The advantage of this method was efficiency because the motion was conducted with a single robot arm, where no consideration on interference between the robot arms is needed. The tilted surface on the fingertip for installation of electroadhesion device realizes posture shown in Fig. 7b, which is advantageous to avoid interference with the workbench during the adhesion. No strong pressure that can make an unexpected crease or curve was applied to the paper during the motion. Thus, this method was not only efficient, but also good for the quality of the origami. Electroadhesion uses electrostatic force; hence, when the paper is peeled even slightly, sticking force will rapidly decrease. Therefore, it is essential to reduce peeling force, which is perpendicular to the device. From this perspective, robot hand is tilted in step 4 beside going up to reduce element of gravity, which acts as the peeling force. The motion bends the paper naturally compared to that without rotation; hence, the ripping force by the origami’s elasticity also seems to decrease. Thanks to this motion, we realized a more stable sticking.

Bending

Fig. 8

Flow of the bending motion

When the grasping motion in Sect. 4.2 is completed, the bending motion starts as follows (Fig. 8):

• 1 Bend the end of the paper (Fig. 8b).

• 2 Position the hand vertically (Fig. 8c).

• 3 Detect the paper corners according to the procedure shown in Sect. 4.5 and calculate the destination.

• 4 Move the robot hand 1 cm above the destination (Fig. 8d, e).

• 5 Move the robot hand to the destination (Fig. 8f).

This motion utilizes the friction between the paper and the workbench, with the top covered by a soft nonslip. The initial bending in Step 1 causes the paper to generate an elastic force that cancels the deformation and consequently increases the pressure on the workbench. Aside from making the elastic force stronger, Step 2 eliminates the interference of sight from the camera (Fig. 9). These steps sufficiently increases the friction between the paper and the workbench, allowing the paper to be fixed in a position that does not anymore require more recognition after Step 3. The destination calculation in Step 3 is based on the paper’s position and the relational position of the robot hand grasping the paper. The line symmetrical point of the grasping position for the desired fold line is calculated as a destination. The destination computation for the experiment performed in Sect. 5.4 will be an instance of the procedure. After the grasping motion, the bending motion also completed only by a single hand, which withdraws the necessity of consideration on interference between the robot arms. In addition, this bending motion is not based on any physical model; therefore, unlike Tong et al. [25], the use is not restricted by modeling conditions.

Fig. 9

Sight of the camera with a different robot arm posture. a In Step 1 of bending, a large portion of the paper is hidden by the arm. b In Step 2, three corners of the paper are seen in the image

Folding

The following algorithm was made for origami folding using our robot hand (Fig. 10a–f):

• 1 Bend the paper following the procedure in Sect. 4.3 with a robot arm and hold the lower side of the paper with a finger.

• 2 Put the fingers of another robot arm on the paper. This time, the pulling motion from the gripper’s direction creates tension in the paper (Fig. 10a).

• 3 Lift a finger in a rotational motion, keeping another finger on the paper (Fig. 10b).

• 4 Open the robot hand. Put both fingers on the paper again (Fig. 10c, d).

• 5 Lift another finger, similar to Step 3 (Fig. 10e).

• 6 Close the robot hand. Return to Step 2 (Fig. 10f).

A crease was made by iterating steps 2 to 6. In contrast to the sweeping motion for folding in Namiki et al. [11] or Elbrechter et al. [10], our motion consisted of a sequence of paper pressing by fingers so that no consideration on friction would be needed. The paper was held at least by two points (i.e., one point by the hand grasping the paper and another one or more points by the fingers of the other hand) throughout the operation, avoiding paper displacement as mentioned in Sect. 4.1. Another point of this motion was tensing the paper at the beginning (i.e., the first time to execute Step 2), preventing the paper from curving and allowing us to make a crease at the desired position.

Fig. 10

Folding algorithm

Fig. 11

Variables and procedure of the folding motion calculation: a relationship between the dimension of the robot hand and the trackr radius, b calculated overlays tracked to the desired crease, and (c) overall flow of the calculation

To perform the folding motion above with our robot hand, consideration on structure of the hand was essential. The robot hand was designed to have a rotary joint and to be tilted in the origami operations as shown in Fig. 11a to reduce risk of interference. Under these conditions, the fingertips of the robot hand making a crease followed a circular track (Fig. 11b). We prepared the following procedure to adjust the track to the desired crease (Fig. 11c). First, parameters $d_{max}$ and $a_{max}$ were given by a human to control the track characteristics, where the details of each parameter will be explained later. Second, corresponding to Fig. 11c, step 2, the r shown in Fig. 11a, b, the track radius, gained by following equation:

$$\begin{aligned} r&= \frac{l}{\cos \theta } \end{aligned}$$

(1)

where l is the length of a finger, and $\theta$ is the tilted hand angle. Third, the crease is split into N fragments (Fig. 11c, step 3(i)-3(iii)). N is initially set to 1 and increases if necessarily considering feasibility and requirements of the motion. If the crease L length is greater than the track 2r diameter, the fingertip cannot follow it; therefore, N is increased until the split crease length L/N is less than 2r. The track is a circular arc; hence, the fingertip will go far from the crease at the middle of it. The maximum displacement of the fingertip can be reduced by increasing the crease split. Thus, the crease is split such that the maximum fingertip displacement d is smaller than the given limit $d_{max}$. d is gained as follows:

$$\begin{aligned} d&= r\left( 1-\cos \left( \frac{\phi }{2}\right) \right) \end{aligned}$$

(2)

$$\begin{aligned} \phi&= \arccos \left( \frac{2r^2-\left( \frac{L}{N}\right) ^2}{2r^2}\right) \end{aligned}$$

(3)

Where $\phi$ is the angle corresponding to the circular track. Then, each of N fragments is divided into n segments (Fig. 11c, step 4(i) and 4(ii)). The folding motion makes the crease by pressing discrete points; hence, an excessively large distance between the points is not appropriate for making a firm crease. The maximum step size for each iteration $a_{max}$ is specified by a human. n is set to minimum integer which makes a, interval of pressing points, smaller than the given $a_{max}$. a is computed as follows:

$$\begin{aligned} a&= \sqrt{2r^2 \left( 1-\cos \frac{\phi }{n} \right) } \end{aligned}$$

(4)

Finally, as step 5 of Fig. 11c, a is converted into the angular displacement of the joint of the robot hand $\lambda$ given as follows:

$$\begin{aligned} \lambda&= \arccos \left( \frac{2l^2-a^2}{2l^2}\right) \end{aligned}$$

(5)

N, n, $\lambda$, and $\phi$ are returned as a result of the calculation.

In the procedure above, Eqs.(3), (4), and (5) were driven by the cosine theorem. Fig. 11b shows an example of the produced track, where N was set to 2, and n was 4. Note that calculation above is only for determining the horizontal track of the fingers and vertical positions are determined based on the given height of the workbench to make proper contact with the paper.

Giving $d_{max}$ and $a_{max}$ in advance allowed the robot system to automatically compute the track with an ordered quality. Using the calculation process output, the robot hand performed the folding for each of the N segments of the crease starting from the end of a segment with the fingertip heading exactly outward the circular track ($\phi /2$ apart from the perpendicular of the crease). For a single segment, the folding motion was iterated for n times, and the robot hand opened a finger to $\lambda$ for each folding.

Origami recognition

This section discusses the procedure for recognizing the origami for executing the operations presented in Sects. 4.2 to 4.4. The integrated flow of the origami operations and recognition was detailed in addition to the processes of extracting the spatial information of origami corners from the captured point cloud and converting it to the robot coordinate. Note the point cloud precision is – 1.61$-$3.39 mm for the z-axis of the camera according to the manufacturer [28]. Considering the typical paper thickness below 0.1 mm, the camera used was not competent enough to navigate the robot in the paper height direction. Therefore, the information from the following recognition steps was utilized only to designate the horizontal position of the origami. The vertical position was based on the workbench height measured carefully in advance so that fingers can stably keep contact without pressing the paper excessively; therefore, precision of the crease would not be affected significantly by setting the height to the fixed value. Recognition is conducted by following procedure. First, obtain the paper corners using an IR grayscale image (Fig. 12).

• (a) The workbench position is previously given.

• (b) Capture the IR image of the origami.

• (c) Trim the IR image in the workbench.

• (d) Binarize the image by brightness. Apply erosion with a 3 $\times$ 3 kernel. Erosion excludes the pixels of noise and background around the origami.

• (e) Detect polygons in the binarized image.

• (f) Specify a polygon having a maximum area with more than three corners as origami. The corners of that polygon will be used in the subsequent steps.

Then, determine the three-dimensional (3D) position of the paper in the camera coordinate using the pixel positions of the detected corners. Finally, convert the 3D position from the camera coordinate to the robot arm coordinate with the parameters gained in advance from the calibration. We transmitted the positions of the paper corners to the robot arms following the abovementioned steps. This spatial information was used in the origami operations based on the paper’s position.

Fig. 12

Origami recognition flow

The following flow depicts the overall procedure of the origami-folding task, where recognition is integrated to grasping, bending, and folding. First of all, positions of origami corners are detected by recognition procedure above. Secondly, based on the corners’ positions, the desired crease and the point to grasp are calculated in the robot coordinate. After executing grasping and three steps of bending, positions of the corners are updated by recognition and compute the line symmetrical position of the grasped point as the bending motion destination. Finally, the remaining of bending steps and folding are completed. We inserted the recognition during the bending operation because we cannot prevent the origami from slipping on the workbench during the initial bending steps while the remaining operations can be conducted without slippage. An example use of this procedure for diagonal folding is introduced in Sect. 5.4.

Demonstrations

We conducted demonstrations with the perspective to show the validity of our origami operations and determine the dexterity of our robot system. The outcome of the operations was then compared to that of another robot system. Though a precision criterion was presented in a previous work [11], that was not general and did not contain enough information concerning error of the crease; therefore, we started from creating the criteria.

Precision criteria

Along with the robot systems and their motions, the general criteria for comparing the precisions of outcomes must also be developed. Namiki et al. [11] employed the distance between two right-angled corners on top of a triangle as a precision benchmark of diagonal folding. However, the criteria were not general because they were heavily dependent on the work scale. In other words, we cannot compare the distances directly if the experimental condition (e.g., paper size or sort of crease) is different. Thus, we decided to present general criteria that can directly be compared, regardless of these conditions.

Evaluating the precision requires the consideration of errors between the desired and actual creases. Two kinds of errors can be observed: positional and directional. In this work, we employed the two criteria of accuracy, namely average displacement S/w and signed angle $\alpha$ gained by following calculation procedure. First of all, measure $y(x_1), y(x_2)$, and w in Fig. 13. Then, calculate the area S representing the positional error as follows:

$$\begin{aligned} S = \int _{x_1}^{x_2} y(x) \textrm{d}\text{ x } \end{aligned}$$

(6)

After that, S is normalized to be compared with other creases as follows:

$$\begin{aligned} \frac{S}{\text {{ w}}}&= \frac{1}{\text{ w }}\int _{x_1}^{x_2} y(x) \textrm{d}\text{ x }\nonumber \\ \quad&= \frac{1}{\text {{ w}}}\left\{ \lim _{n\rightarrow \infty }\frac{\text {{ w}}}{n}\sum _{k=0}^{n} y\left( x_1+\frac{k}{n}\text {{ w}}\right) \right\} \nonumber \\ \quad&= \lim _{n\rightarrow \infty }\frac{1}{n}\sum _{k=0}^{n} y\left( x_1+\frac{k}{n}\text {{ w}}\right) \nonumber \\ \quad&= y_c \end{aligned}$$

(7)

where $y_c$ is the crease center position on the y-axis, and $y_c$ is a conceptually unified error with the crease length logically symbolizing the average displacement of the actual crease from the desired one (This calculation is valid whether desired and actual creases cross on the origami or not as shown in Fig. 13a, b). Beside S/w, angle $\alpha$ representing the directional error is calculated as follows:

$$\begin{aligned} \alpha = \arctan \left( \frac{y(x_2)-y(x_1)}{\text {{ w}}}\right) \end{aligned}$$

(8)

The calculations above conform to the following definitions of $x_1, x_2, y(x)$, and w. The desired crease of the origami is treated as x-axis and y-axis is set to be perpendicular to it. Actual crease is considered as a function y(x). $x_1, x_2$ represent x coordinates corresponding to two edges of actual crease, respectively. w is defined as the length of the actual crease measured in x-axis.

Fig. 13

Precision evaluation. Calculation of the positional error between the desired and actual creases represented by area S depending on whether they are (a) separated or (b) crossing

This evaluation method can provide general representation of the positional error and directional error regardless of size or sort of crease. In addition to the magnitude, the criteria can show tendency of these errors with the sign. The criteria is calculated with only three measurements of $y(x_1)$, $y(x_2)$, and w; hence they are more sensitive to a slight errors compared to the direct measure by a ruler or a protractor. Note that seeing a tendency with a sign is valid only when the paper is similarly aligned. For example, rules like the “upper layer of origami in folding always comes to the y-positive side” are available.

Validity of electroadhesion

We determined the electroadhesion device validity by performing an experiment. In the experiment, our robot hand tried to grasp origami put randomly on the workbench. The robot hand was navigated based on the positions of the corner gained by recognition to grasp 2 cm apart from a corner of the origami. We attempted grasping with and without electroadhesion for five times each to ensure that the grasping success is due to electroadhesion.

The result of the experiment was shown in Table 1.

Table 1 Grasping success rate

Table 1 shows that a successful grasping was made only when electroadhesion was applied. In other words, electroadhesion was necessary in realizing our grasping motion. A possible reason for the failure cases with electroadhesion would be inappropriate contact of the paper and the device because even a slight displacement from the device leads to great reduction of the sticking force. This would happen when the device is not parallel to the paper or surface of the device is formed to be uneven in the fabrication process.

Folding motion precision

We also validated our folding motion alone by performing an experiment. First of all, a robot hand moved 1 cm above the point for holding the paper. After the author handed over the paper to the gripper, the gripper moved 1 cm below to hold the paper, and the other robot hand started making crease. We intended to see the folding error caused by the folding motion only. Hence, we fully conducted the experiment by preparing the previously tuned positional parameters instead of using a camera. We manually aligned the paper using the mark on the origami shown in Fig. 14a, b and line on the workbench shown in Fig. 14c. 3 cm-long creases on both ends of the line to fold were adjusted to the line on the workbench as shown in Fig. 14d, which realizes stable adjustment in the minimum length. The line on the workbench was flexible, thin, and smooth and did not interfere with the folding result.

Under the abovementioned condition, we ordered our robot system to make a 150 mm-long partial crease on center of the diagonal for 10 times using a 61 $\upmu$m-thick origami.

Fig. 14

Experimental setup for the folding precision measurement. The origami was marked (a) point to grasp, (b) desired crease, and (c) line on the workbench that was used for (d) the paper alignment

From the experiment, we acquired a reslut shown in Table 2. The variables in the tables were similar to those described in Sect. 5.1. We also measured the distance for the two corners of origami x presented in the study of Namiki et al. [11]. Fig. 15 displays the pictures of the folded paper.

Table 2 Folding error of our motion

Fig. 15

Folding motion outcome: a folded state of the origami and (b) unfolded one with a desired crease

As demonstrated in Fig. 15, the folding was precise because the actual crease seemed similar to the desired one. Considering that all $y(x_1)$, $y(x_2)$ were less than 1 mm on average, and the minimum scale of the ruler we used was 0.5 mm, the folding precision seemed to be beyond the measurement scale.

As an interesting tendency, all the calculated S in this experiment were positive. Thus, the crease center was always on the y-positive side and biased to the grasped point. This can be attributed to the gripper holding the paper with excessive force. When tuning the parameter for this experiment, we sometimes ordered the gripper to hold the paper in very low positions. The paper’s lower layer became strongly deformed as the gripper went down, making a sharp corner at the held point. This time, the paper was pulled for the lower layer, resulting in such an error. The holding force of the gripper seemed too strong to some degree. Integrating a force sensor to prevent the excessive force with keeping proper contact to the paper is a possible solution. Also, during multiple folding trials, modifying horizontal position of the holding gripper according to S of the latest trial like feedback control would be a feasible counterplan.

Namiki et al. [11] reported that their folding error in x for the ”Shawl fold,” which had the same objective shape to our experiment and represented a 150 mm square origami with the same size as ours, was 2.70 mm on average (Table 3). Therefore, when properly set, our folding strategy is potentially better in terms of precision compared to existing works.

Table 3 Distance between the corners of the origami measured in the work of Namiki et al. [11]

Table 2 shows that the x of both trials 9 and 10 were 1.2 mm, while S/w and $\alpha$ differed. Therefore, the proposed criteria can distinguish the actual error state more in detail than x. Considering that the S/w of Trial 10 was greater than that of Trial 9, and that $\alpha$ was calculated as 0, the translational displacement S/w of Trial 10 contributed to a x greater than that in Trial 9.

Precision throughout the whole motion

We performed a sequence of these operations and evaluated the crease error to show that our strategies of origami grasping, bending, folding, and recognition function as a whole and determine the operation precision. Each motion followed the procedure explained in Fig. 16. Recognition was integrated to the operations as follows (Fig. 17):

• (a) Initial recognition was conducted to determine the position of the four origami corners.

• (b) The position of the point for grasping and the crease to be made were calculated using the position of the four corners. For this experiment, the point for grasping was 3 cm away from a corner on a diagonal line. The crease was set to the other diagonal line of the origami.

• (c) After completing the bending motion by Step 2 in Sect. 4.3, recognition was again executed to update the position of the three corners other than the grasped one.

• (d) The motion objective of the gripper and the updated position of the desired crease were computed using the information obtained from Step (c).

• (e) The remaining bending motion was executed according to the (d) result.

The demonstration was iterated for five times using a 61$\upmu$m-thick paper.

Fig. 16

Flow of the integrated origami operations: a grasping, b, c bending, and (d–f) folding executed in order

Fig. 17

Recognition and calculation steps for the origami operations conducted in the experiment. After (a) the initial recognition, b the calculation of point for grasping and the desirable crease was performed. After grasping and initial bending, c the system again conducted recognition and (d) computed the position objective for (e) the remaining bending motion

Our method successfully made creases in all attempts (Fig. 16). Table 4 presents the results of the experiment involving the error. Fig. 18 displays the folded papers.

Table 4 Folding errors in a whole motion

Fig. 18

Folded papers in the entire experiment: a papers sharing a common error tendency and (b) crease error

Comparing the results in Table 4 to those in Table 2, the precision worsened. The crease error can clearly be seen in Fig. 18b in contrast to Fig. 15b, where it is difficult to tell the desired and actual creases apart. The recognition, grasping, and bending steps were added to the folding procedure in Section 5.3, which increased the error. We infer that major factors of the increased error derived from paper posture estimation, calibration between the camera and the robot hand, and positional information process in grasping. As for paper posture estimation, we found our robot system recognized 150 mm-long edges of a square origami as 148.2 mm on average with 0.9 mm standard deviation and all the edges were recognized slightly smaller than their real size. We suspect erosion for removing noise of binarized image shown in Fig. 12d is the cause of this mismatch. Furthermore, to determine the accuracy of calibration between the camera and the robot hand, we attached a thin needle at the center of the electroadhesion device and ordered the robot system to conduct grasping motion as shown in Fig. 19a. Distance between desired contact point and actual one indicated by a small hole made by the needle was 5.0 mm on the paper; hence coordinate transformation from camera to robot arm or shape modeling of the robot hand would also be a factor. The way to deal with the positional information in grasping and bending phases was also a great error factor.

Fig. 19

Analysis of error in recognition and grasping step

In the grasping motion, the robot hand was navigated according to the position of the electroadhesion device. By contrast, in the bending motion, the point of rubber on the fingertip gripping the paper was used to determine the destination. Although the electroadhesion device and the rubber were on a separate plane on the fingertip (Fig. 5), we used the same positional information for these operations this time. Assuming that paper tightly fit to the finger without any space in the grasping phase, this error can be measured using developed view. As shown in Fig. 19b, distance between geometric centers of the two planes is 6.94 mm, which serve as a typical error for regarding position of these planes as the same. These errors of paper posture estimation, calibration, and positional information process would result in the common tendency of the error in each trial of the experiment. Similar shape of each result in Fig. 18a and tendency that S was always positive while $\alpha$ was always negative support this view.

Compared to the existing works in Table 3, our method was inferior in terms of precision. However, considering only the folding precision in Sect. 5.3, our method potentially exceeded those in existing works by improving the problem involving the positional information of the paper.

On the other hand, the sticking force of the electroadhesion device suddenly decreased greatly during the experiment. This led the robot hand to fail in grasping the paper. This decrease occurred when the fixture of the device by adhesive partly broke and was fixed again. The similar phenomena were observed when the device made a curve due to the inappropriate production process or when it was permanently deformed by force from the confront fingertip or workbench. These phenomena were attributed to the roughness on the device surface. A slight hill on the device already prevented it from fitting closely to the paper, which led to the direct decrease of the sticking force along with less friction and adsorption made by vacuum.

Conclusion

This study presents a novel robot hand used for the three fundamental origami operations of grasping, bending, and folding. The robot hand tackled two issues in existing works; too customized robot system for a specific task and motions restricted by interference or recognition. As an advantage, the robot hand can conduct these three operations with a single sort of hardware. Grasping motion utilizing electroadhesion and bending motion harnessing elasticity of paper and friction are completed by a single hand, where consideration on interference of robot arms is not needed. The folding motion makes a precise crease when correctly prepared (Sect. 5.3). The sequence of operations is conducted by only two recognition steps at the beginning and during the bending, which is less restrictive to the motions compared to visual feedback. The usage of the electroadhesion device is also valuable as an early example of application to the dexterous task of paper manipulation. Installation of the device to the rigid gripper keeping its flexibility is an important feature for enhancing stability of the adhesion. Our criteria can widely be used to fairly evaluate the precision of creases made under different conditions.

The future work must address some points related to the usage of the robot hand employed in this study and its motions. For example, in terms of durability and strength of the sticking force, the electroadhesion device can be improved by revising its design or changing some materials. The electroadhesion device can change sticking force according to the applied voltage; thus, the next step is to design the device which can generate strong sticking force and determine proper voltage to stick only one paper. On the other hand, if sufficient sticking force cannot be gained by redesigning of the device, another strategy should be employed to grasp layered papers simultaneously. The displacement expected to be caused by the positional gap of surfaces for electroadhesion and grasping can also be compensated.