Automatic grasp planning for self-adjustable gripper frames

Abstract

In high-mix, low-volume manufacturing, end-of-arm tooling (EOAT) must be able to handle various products. Conventional EOATs usually comprise fixed or limited-adjustable frames (i.e. manually or using short-stroke pistons) over-equipped with multiple grippers, which are selectively activated based on the product to be grasped. This paper presents a smart gripper frame equipped with only four grippers capable of automatically adjusting to a product’s unique geometry. To this end, a two-dimensional grasp planner has been developed that is supplied with product contours from depth images. The proposed approach has been successfully validated in multiple industrial use cases involving objects with different dimensions and materials, and applying various grippers.

Introduction

Nowadays, the manufacturing industry is increasingly dominated by mass customisation with the production of goods in large variations but low quantities. These products often need to be manipulated multiple times, e.g. during machining, inspection, and packaging [1]. More specifically, 15% to 70% of the product cost is spent on handling [2] with the price of end-of-arm tooling (EOAT) often reaching 20% of the robot cost [3]. These EOATs constitute multiple grippers mounted on a gripper frame: a solid structure carried by a handling device such as a robot or overhead crane. In practice, two main types of gripper frames can be distinguished: fixed frames with an immutable gripper placement and adjustable frames allowing variable gripper positioning through manual adjustments or by employing short-stroke pistons. To be able to handle a wide variety of products and reduce manual interventions, EOATs often end up being over-equipped with multiple grippers selectively controlled based on the dimensions of the product at hand; or being fitted out with large-area vacuum grippers that can handle vacuum leakages to a limited extent when not completely being covered by the product.

Fig. 1

3D-rendering of the tetrapodal smart-adjustable gripper frame used to elucidate the presented grasp planner (left) and the kinematic tree superimposed on the top-down view of this gripper frame (right). In addition, different gripper geometries and arrangements, i.e. offsets and angles are illustrated

Besides existing model-based and model-free grasp planners tuned to a particular vacuum gripper’s characteristics (e.g. [4,5,6,7,8]), this work discusses the development and validation of a two-dimensional grasp planner for controlling a smart-adjustable gripper frame composed of long-stroke linear guideways (Fig. 1). Provided with only four grippers, this EOAT can cover a wide range of products between 580 × 275 [mm] and 1100 × 1050 [mm] with a maximum weight of 300 kg, therewith outstripping conventional fixed and short-stroke gripper frames [9]. In addition, the grippers mounted on the discussed EOAT can be modularly exchanged depending on the characteristics of the products to be lifted and the task to be executed [10]. Where relevant, these grippers can be selected automatically by using, for instance, the web application presented in [11]. Due to the EOAT’s scalable design, both industrial robots as well as manually operated hoist systems can act as potential handling devices.

This paper is organised as follows. Section 2 describes the modelling of gripper frame configurations and the proposed grasp planner. Whether the term “grasp” is applicable to every grasping principle is the cause of some debate. This terminology semantically stems from the act of holding onto something by enclosing or pinching it between one’s fingers or hands. Likewise, when an impactive gripper (e.g. jaw gripper) firmly secures an object by means of a force- or form-closure, it is referred to as “grasping”. The stability of such grasps depends not only on finding appropriate contact points but also on the impactive forces applied in the three-dimensional space. Therefore, one may question the use of this terminology for astrictive grippers (e.g. suction cups or magnets), which generate a force field and solely rely on the contact area in the two-dimensional plane. However, to preserve consistency with similar publications, this paper also applies the term “grasp planning” to astrictive grippers, although wherever appropriate, “grasp” and the more general term “pick” are used interchangeably. In Section 3, the mechanical setup of the self-adjustable EOAT and the developed product contour extractor from depth images are presented. Section 4 discusses their experimental validation in industrial scenarios, addresses current challenges, and points out future research directions. Finally, Section 5 concludes with a summary of this work.

Two-dimensional grasp planning

Modelling gripper frame configurations

The presented approach formalises gripper frames in a similar fashion as the Unified Robot Description Format (URDF) [12]. Conceptually, this well-established format among roboticists composes robot models of both links and joints. Links are rigid entities used for connecting joints. Joints, on the other hand, are entities responsible for the dynamics of the robot model by allowing different types of relative motions between neighbouring links.

Fig. 2

Excerpt from the URDF-like definition of a gripper frame’s links, fixed and linear joints, and grippers using the JSON-format. The coordinate frames in all three links are visualised to reveal the underlying transformations

Programmatically, a gripper frame configuration is formatted in the JavaScript Object Notation (JSON) [13] as displayed in Fig. 2. The highest level specifies the gripper frame’s links, joints, and grippers, each holding an array of their respective sub-objects. Besides a unique identifier that characterises the link, joint, and gripper objects, a joint is composed of four additional properties: the type of motion it exhibits, its origin, the adjacent parent and child links, and optionally the identifiers of its interdependent joints (i.e. joints driven by the same motor). In turn, a gripper is composed of four additional properties: its shape, angle, dimensions, and offsets from the gripper’s centroid to the link it is attached to.

At this point, two types of joint motions have been considered: fixed joints that retain their size, and linear joints that can contract or extend along their axes of orientation. The origin property consists of an object representing the 2D-coordinate transformation from the joint’s parent to child link. As a result, similar to URDF, the origin of a joint aligns with that of its child link. The translational property corresponds to the displacement vector between both reference frames, whereas the rotational property equals the rotation around the z-axis in the counterclockwise direction. In accordance with best practices, all values are specified in SI-units and all coordinate systems are right-handed. Finally, the link property contains an object holding the identifiers of the joint’s neighbouring parent and child links. It is defined that a linear joint enables its child link to move along the x-axis of its parent link’s reference frame within the range dictated by the x-coordinate of the origin’s translational property. Consider for instance the example shown in Fig. 2. Due to linear joint j2, child link l3 is able to translate within a range of 0.0 to 0.5 metres with respect to the local reference frame of l2 and this in the direction of l2’s x-axis. Furthermore, child link l3 rotates clockwise at an angle of $\pi /2$ with respect to l2’s reference frame.

Fig. 3

Top-down view of some tripod (left) and tetrapod (right) gripper frame configurations. Red lines visualise fixed joints; dark blue lines represent joints that exhibit a linear motion

Figure 3 depicts two gripper frame configurations that will be used throughout this work to elucidate the presented grasp planning approach. The first gripper frame, hereinafter referred to as the tripod configuration, is composed of a star-shaped arrangement of six fixed and six linear joints equipped with three gripper mounts at their endings. The second gripper frame, hereinafter referred to as the tetrapod configuration, consists of an orthogonal arrangement of six fixed and six linear joints provided with four gripper mounts at the linear joints’ ends.

After specifying the gripper frame’s configuration in the described JSON-format, it is converted into the kinematic tree representation as rendered in Fig. 4 for the tripod and tetrapod configurations. A kinematic tree can formally be described as a directed rooted tree: a directed acyclic graph $T = (V, E)$ whose vertices $V_{T} = L = \{l_{1}, l_{2}, \ldots , l_{f}\}$ correspond to the set of links (i.e. coordinate systems) with length $f$ that make up the entire gripper frame. The directed edges $E_{T} \subset \{(l_{u}, l_{v}) \in L^2|l_{u} \ne l_{v} \wedge u, v = 1, 2, \ldots , f\}$ represent the connections between neighbouring links as defined by the parent–child relations following from the gripper frame’s description in JSON. The mapping from these edges to the EOAT’s joints, $m: E_{T} \mapsto J$ with $J = \{j_{1}, j_{2}, \ldots , j_{h}\}$ and length $h$, is stored in a hash table. The kinematic tree originates from a common root vertex $v^{src}_{T} \in V_{T}$ corresponding to the world link $l_{W} = v^{src}_{T}$ (i.e. the world’s coordinate system) and terminates in one or more leaf vertices $V^{snk}_{T} \subset V_{T}$ representing the gripper links $L_{G} = V^{snk}_{T}$ (i.e. the grippers’ coordinate systems) with $G$ being the set of gripper indices. Following the definition of a directed rooted tree, each gripper link $l_{g} \in L_{G}$ in $T$ is connected to the world link $l_{W}$ by exactly one directed path $P_{g} = ((p_{r}, p_{r+1}) \in E_{T}| p_{0} = v^{src}_{T} \wedge p_{z} \in V^{snk}_{T} \wedge r = 0, 1, \ldots , z-1)$ with length $z$. Subsequently, every directed path $P_{g}$ can be translated into its corresponding joint chain $C_{g} = (j_{1}, j_{2}, \ldots , j_{z})$ by iteratively applying map function $m$ to $P_{g}$’s elements. Note that since a rooted tree only allows at most one directed path to each leaf vertex, the existence of redundant joint chains—which would simultaneously be responsible for the same gripper’s motion—is ruled out.

Fig. 4

Kinematic tree representations of the tripod (left) and tetrapod (right) gripper frame configurations. Red lines visualise fixed joints; dark blue lines represent joints that exhibit a linear motion

Aligning the gripper frame to the product’s pose

To lift the object as stably as possible, the EOAT is initially positioned in the centre of area of the product and rotated according to its principal axes. To this end, the centroid of the product’s polygon in the two-dimensional Cartesian plane is calculated—assuming a uniform mass distribution—using the area of this polygon, while considering its shell and every cut-out. Subsequently, the moments and product of inertia relative to the object’s centroid are obtained by applying the parallel axis theorem. Note that the inertia quantities of all cut-outs are subtracted from those of the product’s shell. Finally, the orientations of the principal axes are determined based on these inertia quantities. Oftentimes, these geometrical polygon properties can simply be calculated using off-the-shelf functions from software libraries available in any contemporary programming language.

Fig. 5

Composing the gripper zones for the tripod (left) and tetrapod (right) configurations. The gripper frame is automatically aligned with the product’s centre of area and principal axes of inertia. Legend: grey shape: product polygon; black dots: links; red lines: fixed joints; dark blue lines: linear joints; blue rectangles: gripper polygons; black dashed lines: gripper zones

Composing the gripper zones

The EOAT’s gripper zones delineate the regions described by all of its grippers when every linear joint is operated at its upper and lower bounds (Fig. 5, dashed lines). Algorithm 1 describes the process of composing zone $Q_{g}$ of gripper index $g \in G$. For each joint $j$ of this gripper’s joint chain $C_{g}$, lines 1 to 15 determine the outer motion limits $M$ considering the type of this joint. These motion ranges are then grouped into sequence $\mathbb {M}$ in reverse order as opposed to $C_{g}$. Line 16 subsequently calculates the Cartesian product of $\mathbb {M}$ resulting in all possible outer configurations $\mathbb {S}$, i.e. sequences of joint sizes, in which the joints of $C_{g}$ can be adjusted simultaneously. Indeed, joints that operate interdependently are not considered thus far. These constraints will be addressed in Section 2.5. Subsequently, lines 17 to 29 apply forward kinematics to determine the cluster of corners that makes up gripper $g$’s zone. To this end, every chain of transformations $S \in \mathbb {S}$ is traversed from the respective gripper link to the world link for all corners of polygon $Y_{g}$ that describe $g$’s geometry. These transformed corners are finally clustered together in set $Q_{c}$ of which the enclosing convex hull represents gripper $g$’s zone polygon $Q_{g}$ (see Algorithm 1, line 30).¹

Algorithm 1

Composing gripper $g$’s zone

Determining the product-zone overlaps

However, the previously constructed gripper zones $Q_{g} \in \mathbb {Q}$ for every gripper index $g \in G$ can still contain sub-regions in which the EOAT’s grippers will not be able to establish a steady grasp due to the absence of contact with the product. To discard these regions, the overlapping areas between the product polygon $\mathcal {P}$ and every gripper zone $Q_{g}$ are determined using set-theoretic intersection operations: $\mathbb {Q}' = \{\mathcal {P} \cap Q_{g}|Q_{g} \in \mathbb {Q}\}$. These result in the light orange polygons $Q'_{g} \in \mathbb {Q}'$, for every gripper index $g \in G$, pictured in Fig. 6 that visualise the common regions between the product to be grasped and the formerly calculated gripper zones. Note that some grasping devices, including large-area vacuum grippers and magnetic grippers, do not require full coverage of the grasping surface but instead can handle vacuum and field leakages to a limited extent. This degree can be tuned by the overlapping factor $\varPsi$ described in the following section.

Fig. 6

Determining the product-zone overlaps for the tripod (left) and tetrapod (right) configurations. Legend: grey shape: product polygon; black dots: links; red lines: fixed joints; dark blue lines: linear joints; blue rectangles: gripper polygons; black dashed lines: gripper zones; light orange areas: product-zone overlaps

Computing and refining the gripper centroid areas

The gripper centroid areas represent the regions in which the centroid of every gripper can be located (Fig. 7, dark orange areas) considering the gripper frame’s kinematics and a pre-specified minimum degree of contact between the product and the grippers’ surfaces. The latter is controlled by the overlapping factor $\varPsi \in \mathbb {R}^{[0,1]}$, which could be useful for grippers that do not require full coverage of the grasping area such as large-area vacuum grippers and magnetic grippers. Lowering $\varPsi$ will relieve the constraints set on the grasp planning process by enlarging the gripper centroid areas resulting in more suitable picking points. Algorithm 2 describes the process of computing centroid area $A_{g}$ of gripper index $g \in G$. From lines 1 to 10, each corner $y$ of this gripper’s polygon $Y_{g}$ in origin $O$ is rotated according to the reverse order of orientations of each joint $j$ in this gripper’s joint chain $C_{g}$. For each line segment of this gripper’s product-zone overlap $Q'_{g}$, lines 11 to 30 determine the inward offset $d_{g}$ by which each line segment should be displaced. This shift is determined by the overlapping factor $\varPsi$ and the maximum of distances $D_{g}$ between the gripper’s centroid and its corners along the normal direction of that line segment. The intersections between these newly created line segments delineate the gripper centroid areas (see Algorithm 2, lines 31 to 37).

Algorithm 2

Computing gripper $g$’s centroid area

Fig. 7

Computing the gripper centroid areas for the tripod (left) and tetrapod (right) configurations. Legend: grey shape: product polygon; black dots: links; red lines: fixed joints; dark blue lines: linear joints; blue rectangles: gripper polygons; black dashed lines: gripper zones; light orange areas: product-zone overlaps; dark orange areas: gripper centroid areas for the individual product

In general, outward or inward offset curves (i.e. gripper centroid areas) are determined by assuming that the offset curve’s points are at a certain distance in respectively the outward or inward normal direction of the original curve at each point. Besides this definition, implemented in Algorithm 2, the offset curves could also be considered as the (outward or inward) enclosure of copied addition polygons, with their centroids at each point of the original curve. This resembles the Minkowski sum $\mathcal {A} \bigoplus \mathcal {B} = \bigcup _{i}(a_{i} \bigoplus \mathcal {B}) = \{a + b| a \in \mathcal {A}, b \in \mathcal {B}\}$ of the original curve’s polygon $\mathcal {A}$ and the addition polygon $\mathcal {B}$. To determine the outward offset curve, both polygons have to be oriented counterclockwise; to determine the inward offset curve, the addition polygon has to be oriented clockwise as opposed to the original curve’s polygon. This approach is mainly employed when both polygons are convex. However, if at least one polygon is non-convex, the following is required: the decomposition of the non-convex polygon (e.g. $\mathcal {A}$) into convex sub-polygons $\mathcal {A} \bigoplus \mathcal {B} = \bigcup _{i}(\mathcal {A}_{i} \bigoplus \mathcal {B})$; or the convolution of the (non-)convex polygons $\mathcal {A} * \mathcal {B} = \{a + b| a \in \mathcal {A}, b \in \mathcal {B}, t_{A} = t_{B}\}$ with $t_{A}$ and $t_{B}$ the tangent values of both curves. For the latter approach, the number of counterclockwise turns the original curve makes around any point $p$ (i.e. winding number $\texttt {Ind}$) must be positive to determine the outward offset curve $\mathcal {A} \bigoplus \mathcal {B} = \{p \in \mathbb {R}^2| \texttt {Ind}_{\delta \mathcal {A} * \delta \mathcal {B}}(p) > 0\}$, and equal to one to determine the inward offset curve $\mathcal {A} \bigoplus \mathcal {B} = \{p \in \mathbb {R}^2| \texttt {Ind}_{\delta \mathcal {A} * \delta \mathcal {B}}(p) = 1\}$ [14].

When grippers share linear joints in their kinematic chains, their gripper centroid areas are influenced by each other. Given each gripper’s kinematic chain $C_{g}$, the motion limits of each joint $j \in J$ is determined by the inverse kinematics of every corner of the centroid areas $A_{g} \in \mathbb {A}$, as demonstrated by lines 1 to 16 of Algorithm 3. Subsequently, lines 17 to 22 supplement the motion limits of interdependent joints, sharing the same motor, with each other’s motion limits. Finally, for each joint $j \in J$, the minimum and maximum motion limits $(m_{\text {min}}, m_{\text {max}})$ respectively correspond to the maximum lower limit and minimum upper limit of the joint’s set of motion limits (see Algorithm 3, lines 23 to 27).

Algorithm 3

Determining shared and/or interdependent motion limits for all joints

Algorithm 4 describes the process of refining centroid area $A_{g}$ of gripper index $g \in G$ due to the presence of shared and/or interdependent joints. For each joint $j$ of this gripper’s joint chain $C_{g}$, lines 1 to 9 determine the joint’s sequence of motion limits $\mathbb {M}$ and orientations $\Omega$ in reverse order as opposed to $C_{g}$. Line 10 then calculates the Cartesian product of $\mathbb {M}$ resulting in all possible outer configurations $\mathbb {S}$, i.e. sequences of joint sizes. Subsequently, lines 11 to 21 apply forward kinematics to determine the cluster of corners that makes up gripper $g$’s bounding box. To this end, every chain of transformations $S \in \mathbb {S}$ is traversed from the respective gripper link to the world link for gripper $g$’s centroid $c'$ at the origin $O$. These transformed centroids are finally clustered together in set $C_{c}$ of which the enclosing convex hull represents bounding box $B$. The intersection of this bounding box $B$ with gripper $g$’s centroid area $A_{g}$ results in the refined version $A'_{g}$ (see Algorithm 4, lines 22 to 23). Figure 8 depicts these refined areas in which the gripper centroids can be placed (i.e. dark orange areas), taking into account the motion restrictions imposed by shared and interdependent joints. The three central linear joints of the tripod configuration (i.e. $j_{1}$, $j_{5}$ and $j_{9}$ in Fig. 4) and the two central linear joints of the tetrapod configuration (i.e. $j_{1}$ and $j_{7}$ in Fig. 4) are driven by the same motor and therefore interdependent.

Algorithm 4

Refining gripper $g$’s centroid area

Fig. 8

Refining the gripper centroid areas for the tripod (left) and tetrapod (right) configurations. Grey shape: product polygon; black dots: links; red lines: fixed joints; dark blue lines: linear joints; blue rectangles: gripper polygons; black dashed lines: gripper zones; light orange areas: product-zone overlaps; dark orange areas: gripper centroid areas for the individual product, which are refined based on shared and interdependent joints

Deriving feasible picking points on individual products

After executing the previously discussed steps, Fig. 9 visualises the resulting areas on four manually composed polygons in which the gripper centroids can be positioned securely. In order to obtain the most optimal set of grasping points within these gripper centroid areas (Fig. 9, dark orange areas), several metrics can be devised among which the smallest or largest Euclidean distance from the gripper frame’s centre, respectively aiming at the most compact configuration or the least deflection of the product during lifting. The remainder of this paper will employ this second metric.

Fig. 9

Deriving feasible picking points on individual products for the tetrapod gripper frame configuration. Legend: grey shape: product polygon; light orange areas: product-zone overlaps; dark orange areas: gripper centroid areas for the individual product, which are refined based on shared and interdependent joints

Extra: deriving feasible picking points on product families

When provided with a (limited) set of product polygons, the presented approach could additionally be adopted for dimensioning a single rigid gripper frame capable of handling every product in that family $\mathcal {P} \in \mathbb {P}$. After calculating the refined centroid area $A'_{g} \in \mathbb {A}'$ for every gripper index $g \in G$, the common gripper centroid areas of the static frame are found by computing the intersection of the set of corresponding gripper centroid areas across the range of products: $\mathbb {F} = \big (\bigcap _{j=0}^{\Vert \mathbb {P} \Vert -1} \mathbb {A}'(i)| \mathbb {A}'(i) \sim \mathbb {P}(j)\big )_{i=0}^{\Vert G \Vert -1}$. Figure 10 visualises the resulting common areas for four manually composed polygons in which the gripper centroids can be positioned securely using a static gripper frame. In order to derive the most optimal set of picking points, the metrics mentioned in Section 2.6 apply here as well.

Fig. 10

Deriving feasible picking points on product families for the tetrapod gripper frame configuration. Legend: grey shape: product polygon; light orange areas: product-zone overlaps; black dashed lines: gripper centroid areas for the individual product, which are refined based on shared and interdependent joints; green areas: gripper centroid areas optimised for handling all four products using a single adjustment of the gripper frame

Self-adjustable gripper frame

Mechanical setup

The grasp planning method described in Section 2 has been deployed on a tetrapod gripper frame configuration (Fig. 11) in the context of the Flanders Make² SmartHandler project [9]. This gripper frame is horizontally extendable and foldable due to six linear guideways (i.e. rack and pinion drives) powered by servo motors. The two lengthwise guideways at the frame’s centre change position symmetrically due to a flip-flop arrangement of two opposite racks moved by one joint pinion. The remaining four guideways along the widthway of the gripper frame displace asymmetrically by four independent servo motors. The four gripper mounts can be modularly equipped with any type of astrictive (e.g. suction cups and magnets) or even ingressive (i.e. needles) grippers. For this work, the gripper frame was mounted on an overhead crane (i.e. hoist) and guided by an operator provided with the following controls: a switch for activating the grippers, and three push buttons for (1) homing the incrementally controlled guideways, (2) capturing a depth image of the current scene containing the product to be manipulated, and (3) automatically adjusting the linear guideways to the optimal picking positions dictated by the grasp planner.

Fig. 11

SmartHandler tetrapodal gripper frame operating on (left) a nearly cylindrical aluminium fuel tank for trucks, and (right) a wooden EPAL-pallet. The intuitive GUI designed for assisting the operator can be viewed in the left image

Software setup

A user-friendly graphical user interface (GUI) has been designed for supporting the operator in running the grasp planner and subsequently positioning the gripper frame with respect to the product. In addition to the outcome of the grasp planner (Figs. 12c, d), this GUI displays the live depth stream registered by the LiDAR-camera below the gripper frame. These depth images are supplied to the planning algorithm for determining appropriate picking positions. As depicted in Fig. 12a, the depth data from the Intel$^\circledR$ RealSense$^{\textrm{TM}}$ L515 LiDAR-camera are converted to greyscale images in which darker colours correspond to increased distances with respect to the camera. Using a Gaussian mixture model, every pixel of these depth images is segmented into different clusters (Fig. 12b) according to the layer they belong to, i.e. background, middle ground, and foreground. When there are multiple objects visible in the scene, the product intended by the operator for lifting is estimated based on two metrics: the products’ heights and distances to the image centre, allowing to select the product closest to the camera. Finally, the product contours are extracted using Suzuki’s topological structural analysis [15] in OpenCV. Additional contour smoothening is achieved by means of the Ramer-Douglas-Peucker’s algorithm [16, 17] for straight edges with sharp corners or the Savitzky-Golay’s algorithm [18] for curved edges (Fig. 12c).

Fig. 12

Collection of images showing the grasp planning process for a wooden EPAL-pallet: a grey-scaled depth image acquisition, b product-background segmentation ($n = 3$), c world coordinate conversion, contour smoothing and grasp planning ($\varPsi = 0.4$), and d the resulting view presented to the user via the GUI

Results and discussion

Experimental validation

The previous sections offered an insight into the presented two-dimensional grasp planning approach and its application on an in-house developed self-adjustable gripper frame. To validate the robustness of the developed method, five industrial scenarios have been established involving products with various dimensions and materials, different gripper geometries and changing lighting conditions.

These experiments consist of handling of a wooden EPAL-pallet (Fig. 12d), several cardboard sheets with three distinct cut-out patterns (Figs. 13a–c), and a truck’s nearly cylindrical aluminium fuel tank (Fig. 13d) using the tetrapodal SmartHandler gripper frame. To this end, four sets of commercially available grippers have been adopted among which Piab’s square Kenos$^\circledR$ KCS (Fig. 13a) and rectangular Kenos$^\circledR$ KVG (Fig. 13d) large-area vacuum grippers, Goudsmit’s circular MagVacu$^\circledR$ combi gripper (Fig. 13c), and FormHand’s circular FH-R150 gripper combining vacuum technology with granulate-filled gripping pads (Fig. 13b). Thanks to the URDF-like modelling of the gripper frame’s kinematics, various frame layouts as well as gripper geometries can be configured easily.

Fig. 13

Depth images displaying the extracted product contours and picking positions for cardboard sheets with three different cut-out patterns (a–c), and a nearly cylindrical aluminium fuel tank for trucks provided with a protruding refuelling opening and a hole for installing a level sensor (d). Four commercially available grippers with various geometries were applied during grasp planning: a Piab Kenos KCS, $\varPsi = 1.0$; b FormHand FH-R150, $\varPsi = 0.6$; c Goudsmit HGC-RO-100-VR-M-G-F, $\varPsi = 1.0$; d Piab Kenos KVG 480 $\times$ 122 [mm], $\varPsi = 0.2$

Figures 12d and 13 picture the products’ inner and outer contours extracted from the depth images in yellow whereas the light-blue polygons represent the most optimal picking positions calculated by the two-dimensional grasp planner. During operation, these images are displayed via the intuitive GUI to guide the operator in positioning the SmartHandler frame before activating its grippers. Currently, the four positions with the largest Euclidean distance from the gripper frame’s centre are selected to guarantee a stable manipulation. A future extension might entail implementing different metrics that could be selected automatically or manually during product handling.

Table 1 Computing times (mean ± SD, $n = 5$) for finding the contours and picking positions for the five products displayed in Figs. 12 and 13

Table 1 lists the online computing times for the five experiments. The total time needed for extracting the product’s contours and calculating the optimal locations for picking range on average between 5.99 and 8.37 seconds depending on the use case. The largest share is dedicated to the former with the depth image segmentation and contour smoothing being the most computationally demanding sub-operations. However, this is still faster than manually reconfiguring the gripper frame and/or changing grippers. Moreover, if the products are known in advance, the picking positions could be determined offline based on the products’ CAD models and uploaded to the self-adjusting gripper frame. On average, grasp planning amounted to 36 percent of the overall computing time using the rather restricted single-board computer installed on the SmartHandler gripper frame. These experiments were performed on an Odroid XU4 with an octa-core Samsung Exynos 5422 processor (2.1 GHz + 1.4 GHz) and 2 GB RAM.

Fig. 14

Grasp planning results for the self-adjustable gripper frame applied on four polygons that idealise the actual products from Figs. 12d and 13a–c. In addition, the same grippers were applied on these manually engineered polygons and positioned by the metric with the largest Euclidean distance from the gripper frame’s centre. Note that due to the selected overlapping percentage, part of the gripper centroid area calculated for product b partially covers one of its holes. Legend: grey shape: product polygon; blue rectangles: gripper polygons; light orange areas: product-zone overlaps; dark orange areas: gripper centroid areas for the individual product, which are refined based on shared and interdependent joints

Similar tests on a PC with an i7–7820HQ Intel processor (2.9 GHz) and 32 GB RAM, involving the idealised polygons in Fig. 14 of the actual products, show significant speed gains with the two-dimensional grasp planning requiring on average between 0.028 and 0.11 seconds to complete (see Table 2). In addition to the improved hardware, this is also likely due to the rather limited number of points that describe these ideal polygons. However, simply increasing the smoothing parameters with the aim of reducing the complexity of real product contours will adversely affect the time needed by this pre-processing step. Future research could therefore focus on optimising or implementing faster smoothing algorithms.

Moreover, it has been demonstrated that the developed grasp planning algorithm is also useful for dimensioning a rigid gripper frame based on the geometries of a set of products. In this study, this functionality was extensively tested on multiple families of ideal product polygons. Acquiring this information from a company’s CAD-database could also have been among the possibilities. Going forward, determining the common gripper areas of product families will additionally be validated on real product contours obtained by the developed contour extractor from depth images.

Besides a hoist system, the SmartHandler gripper frame has been designed to be additionally mounted on an industrial robot manipulator. Since the open-source Robot Operating System (ROS) supports most robots nowadays, the developed software is currently being incorporated in a ROS2-driver.

Table 2 Computing times (mean ± SD, $n = 5$) for calculating the picking positions for the four hand-crafted polygons displayed in Fig. 14

Challenges

Despite their promising results with regard to performance and generality, the presented two-dimensional grasp planner and product contour extractor currently face some challenges. This section addresses them point-by-point and outlines potential refinements.

During the experiments conducted in Section 4.1, it is observed that when working with fairly reflective or metallic products, the infrared (IR) laser beam emitted by the LiDAR depth camera underneath the gripper frame could be occasionally (and rather locally) disturbed by direct incident sunlight. In addition, scattering materials or curved surfaces that reflect the emitted IR light in odd ways tend to challenge this camera’s underlying depth technology. In those cases, the contour extractor developed in this study might encounter difficulties resulting in rough or partly missing edges. For this reason, the top surface of the fuel tank mentioned in Section 4.1 was covered with masking tape in preparation for the experiments. Note that this concerns a hardware limitation of the selected depth camera rather than a constraint of the developed grasp planning algorithm. Therefore, in environments suffering from bright ambient light, stereo vision cameras equipped with optical filters have shown to yield superior performance. For dealing with featureless two-dimensional surfaces (e.g. paper sheets, evenly-painted panels) or objects with small repetitive patterns, one can rely on active IR stereo vision cameras. A software solution could entail applying more sophisticated refinement techniques to the extracted contours for improving adversely affected sections.

Since the SmartHandler gripper frame approaches the product along its top view, it was initially decided to create a two-dimensional grasp planner. As demonstrated, this resulted in a performant and versatile algorithm enabling the control of adjustable handling devices, with a preference for products consisting of flat top surfaces. Nonetheless, the application in Fig. 13d proved that slightly curved surfaces can be considered as well, provided that the selected grippers are suited. As such, large-area vacuum grippers with thick compressible foam pads at their gripping surfaces were adopted for picking the near-cylindrical fuel tank. Another interesting application might be the simultaneous lifting of individual objects, e.g. each gripper of the frame handling a cardboard box of a different height. For this, the contours of every product in the depth frame must be combined into a single disjoint polygon for the grasp planner to work on. While the proposed algorithm is capable of dealing with multiple products at once, the gripper frame used in this study is restricted by its kinematics and therefore not able to vertically move each gripper independently. To mitigate this hardware restriction, the product contour extractor was instructed to output only one contour, the closest to the camera (see Section 3.2), for each depth frame. Again, since this is rather a hardware limitation, connecting each gripper via a vertical compensator to the gripper frame—whether actively or passively controlled—could potentially open up a novel field of application.

With the aim of achieving a stable pick, the gripper frame is aligned with the product’s centroid and principal axes at the start of the planning process (see Section 2.2). For this, it is assumed that the product possesses a uniform thickness and homogeneous mass distribution. This is true for two-dimensional products such as sheet metal, plasterboards, and plywood panels. However, in case of three-dimensional objects and non-homogeneous materials, this initial alignment of the gripper frame might turn out to be sub-optimal. In addition, it may happen that when aligning the gripper frame along its centroid and principle axes, an overlap could not be found for every gripper if the product features large cut-outs. Due to its actual mass distribution and weight, lifting the object in such case might result in an unsatisfactory operation. Also gravitational and dynamic loads due to accelerations are presently not factored in by the grasp planner. A potential solution could be allowing the algorithm to optimise the orientation of the gripper frame with respect to the product’s pose prior to grasp planning. Moreover, the user can always set the minimum number of grasping points s/he considers necessary for the intended application, the available gripper frame and its grippers. Ultimately, it remains the responsibility of the operator to assess whether the calculated gripper positioning is capable of providing the necessary lifting force and stability to manipulate the product correctly and safely. If necessary, the relatively fast computing times reported in Section 4.1 allows the grasp planner to be run more than once until a satisfactory gripper positioning is found.

Finally, besides the object’s contour, no other product properties are provided as input to the planner. As a result, the operator’s ability is called upon to verify that the grippers installed on the gripper frame match the material properties of the product to be lifted, e.g. magnetic grippers for ferromagnetic materials and vacuum grippers for non-porous materials. For adequate gripper selection, one could rely on existing web-based selection tools as presented in [11].

Conclusion

This work presents a two-dimensional grasp planner and depth-based contour extractor for the robust handling of diverse products using a self-adjustable gripper frame. The dimensioning of rigid gripper frames based on product families is additionally discussed. The overall approach has been successfully validated in five industrial use cases involving products with different dimensions and materials, and various astrictive grippers. Future work will target software and hardware related improvements to further enhance the robustness and speed of the proposed method, in particular (1) devising additional metrics that could be selected automatically or manually during product handling, e.g. objectives that favour picking locations with higher contact rates, and (2) implementing faster contour refinement algorithms.