Robust proprioceptive grasping with a soft robot hand

The fabrication of a single finger is based on a lost-wax casting process (Katzschmann et al. 2015; Marchese et al. 2015). As described in Homberg et al. (2015) and Fig. 2, the process has an added step where the bend and force sensors are added to the stiff constraint layer.

Figure 3 shows an image of the inside of the finished finger; the constraint layer and the sensors are visible.

The updated DRL finger is streamlined at 1.8 cm wide by 2.7 cm tall by 10 cm long, contains both bend and force sensors, and is not prone to popping or leaks. Various views of a completed finger can be seen in the left column of Fig. 4. The new version benefits from shaping of the internal air channels and eternal finger shape to avoid all sharp corners which can be places of stress on the rubber. While the old version of the finger often broke intermittently, sometimes after only light use, the new version of the finger lasts several months and many hundreds of grasps before succumbing to rubber fatigue.

Each finger is connected via a tube attached along the arm to a pneumatic piston. The actuation system is described in Katzschmann et al. (2015); Marchese et al. (2015).

There are two sensors in each finger: the Flexi-force force sensor at the tip of the finger and the Bendshort-2.0 flex sensor from iCubeX. Both sensors are resistive sensors: as the sensor is pressed or bent, the resistance of the sensor changes.

Due to the construction of the sensor, the relative change in resistance increases as the curvature of the sensor increases. Thus, the sensor has better accuracy and resolution as its diameter decreases. The diameter we refer to is the diameter of a circle tangent to the bend sensor at every point, for some constant curvature bend of the sensor. This relation between diameter of the finger and sensor value is shown in Fig. 5, where sensor values versus finger curvatures are plotted for the unloaded case.

Due to the inherent changes in variance for the sensor values, we are able to distinguish objects more accurately for objects with a smaller diameter.

For the hand-specific control, there are three sets of components: the physical fingers, the electronics, and the control software. The fingers are actuated via pneumatic tubing. The pneumatic tubing is pressurized by a piston, and the piston is driven by a linear actuator. Motor controllers, one per linear actuator, set the position of the linear actuators, setting the volume of the air in each finger. Additionally, each finger has a bend and a force sensor. Each of the sensors are connected to filtering and buffering electronics and then measured using an Arduino board.

On the software side for the hand, there is a middle-level controller enabling us to command the hand using primitive actions such as “close the hand” or “open the hand”. This middle-level controller communicates with the low-level motor controllers via serial communication. It also receives sensor values from the Arduino board on the hand through rosserial.¹

On the robot side, the two key pieces of hardware are the hand cameras and the robot arm, to which the hand is attached via 3D printed interface parts. For the robot software, we implemented the grasping and object recognition pipeline using a set of ROS nodes (Quigley et al. 2009). One main ROS node coordinates the overall behavior. One ROS node reads the camera input streams and performs object detection using basic image processing in OpenCV (Bradski 2000). One strength of the DRL soft hand is its ability to grasp unknown objects with uncertain pose estimation. This vision system serves to detect approximate poses of objects even if they are completely unknown to the robot. A suite of ROS nodes run for the MoveIt planner Sucan and Chitta 2018. One object in the codebase interfaces with the MoveIt planner to coordinate calls to plan motions to different locations. For side grasps, the motion planning node finds a grasp plan given a potential object location using an intermediate way point. The planner considers 16 potential directions by which to approach the object. Along the direction, it first considers an offset pre-grasp location which is offset far enough to be simple to plan to without getting too close to the object. For top grasps, the motion planner is called to find a plan to a pose where the hand is vertically above the object. For top grasps of small objects, first the fingers are half closed to allow the hand to approach closer to the table without the fingers hitting the table and being unable to bend to grasp due to excessive friction. Another object handles the control of the soft hand, opening, closing, and grasping. A separate node sends specific commands via serial to the motor controllers.

The value measured from the force sensor is an approximate force. Due to noise after the hardware low pass filter, we buffer the output in software and consider an average of the past five data samples. If the average of the data samples crosses a certain threshold, we consider this to be a contact between the fingertip and an external object.

In grasping an object, we keep increasing the volume of air in each finger until we detect a point of contact. Since the grasp criterion for each finger is independent, it does not matter if an object is oddly shaped; the fingers will each close the correct amount. If no contact is detected, the fingers simply keep closing until their maximum closure (Algorithm 1).

We incorporated the DRL soft hand into a complete, end-to-end grasping system. This grasping system demonstrates the versatility of the soft hand by showing its robustness to uncertainty in the location of the object and the minimal need for grasp planning.

The grasping system pipeline consists of three phases: location, approach, and grasp (Algorithm 2). A successful execution of a grasp means that all steps of Algorithm 2 are executed without failure. To be more specific, this entails that the arm motion is successfully planned and executed, and the object is then successfully grasped, correctly identified, dropped into the bin.

Once trained, the DRL soft hand is able to identify the grasped objects based on a single grasp. We first characterize the relation between hand configurations and sensor readings. Then, we present a data-driven approach to identify an object based on sensor readings.

(1) Modeling the sensor noise The DRL hand has different configurations as it interacts with the environment and grasps objects. We define a configuration of the DRL hand as a vector \(\mathbf {q} = [ q_1, q_2, q_3, q_4]\), where each \(q_i \in \mathbb {Q}\) represents the way finger i is bent. \(\mathbb {Q}\) is the configuration space of a finger: that is, the space of all different shapes our soft finger can achieve. For a given configuration of the hand, we get bend sensor readings \(\mathbf {s} = [ s_1, s_2, s_3, s_4 ]\), where each \(s_i\) represents the bend sensor reading for finger i and a force value \(\mathbf {f} = [f_1, f_2, f_3, f_4]\), where each \(f_i\) represents the force sensor reading for finger i.

The sensor readings are noisy. We represent the sensor reading given a hand configuration as a probability distribution, \(p(\mathbf {s}, \mathbf {f} \, | \, \mathbf {q})\). Given the configuration of a finger, the sensor values of that finger is independent of the configuration of the other fingers. Therefore, the sensor model of the whole hand can be expressed in terms of the sensor model for each finger:We model \(p(s_i \,|\,q_i)\), the bend sensor noise for a finger, in a data-driven way by placing the finger at different configurations and collecting the sensor value data. In Sect. 3.4 we present experiments for such a characterization, where we use constant curvature configurations of the unloaded finger. The force value will depend not just on the configuration of the finger but also on the interface of the finger with the environment, as the sensor values differ based on where exactly the sensor is pressed. In order to model the desired probability \(p(s_i, f_i \,| \,q_i)\), we also need to take into account the interaction with the environment. In the absence of any external interaction, the finger is constructed such that \(f_i\) will always be equal to 0.

Note that when the finger is loaded in a grasp, the resulting finger configurations and the corresponding sensor readings have significant variation due to the highly compliant nature of the fingers. Therefore, to identify objects during grasping, we use data collected under the grasp load.

The grasping configuration for an object can be different for different types of grasps. In this work we focus on two types of grasps: enveloping grasps and pinch grasps. For a given object, o, we represent the configuration at the end of an enveloping grasp as \(\mathbf {q}_o^{envel}\); and we represent the configuration at the end of a pinch grasp as \(\mathbf {q}_o^{pinch}\).

For given sensor readings \(\mathbf {s}\) and \(\mathbf {f}\) and a grasp type \(g \in \{envel, pinch\}\), we define the object identification problem as finding the object with the maximum likelihood:where \(\mathbb {O}\) is the set of known objects and \(o^*\) is the predicted object. Applying Bayes’ rule, we get:Since the denominator is a constant over all objects o, we see:Assuming a uniform prior over finger configurations, the above formulation becomes:In our experiments we use a trained dataset to build an empirical model of \(p(\mathbf {s}, \mathbf {f} \,|\, \mathbf {q}_o^g)\) for different objects and grasp types. Then, we identify the object for a new grasp (Eq. 5) using a k-nearest neighbor algorithm.

3) Trained Object Identification: Algorithm 3 uses an initial trained dataset. We train using a dataset of sensor values for repeated grasps of known objects. We use the same dataset as for clustering, but with the originally known identities of each of the objects. We use this training set to identify objects as they are grasped in a separate testing phase. After each new grasp, the five nearest neighbors of the new point in the original training data are determined. We calculate the distance via the Euclidean metric on the 4-dimensional point comprised of the four sensor values, one per finger. The object is identified based on the most common identity of the five nearest neighbors, using the KNeighborsClassifier from scikit-learn (Pedregosa et al. 2011). The identification algorithm runs in less than 0.01 seconds. This algorithm is flexible: it was used on three-fingered and four-fingered versions of the hand without modification. Given a number of fingers, D, and a number of objects to identify, N, the running time of the algorithm grows as \(O(D \log (N))\). The number of classes we are able to successfully distinguish is limited by the sensor resolution: with noisy sensors, object clusters must be relatively far apart in order to be distinguishable. Higher fidelity sensors, additional sensors in each finger, or the use of additional sensing sources (e.g., vision) would enable this technique to work with more classes of objects.

Essentially, when grasping an object, the algorithm considers the distance between the values of the sensors for that object and all of the other objects currently in the dataset. Based on the average distance to all data points from a label and the number of data points with that label, a score is calculated for each label. If the label with the highest score has a score higher than a fixed cutoff score, then the grasped object is labelled with that label. This cutoff score was empirically determined based on the sensor variability across identical grasps. The score for a label is equal to \(1 / ( avg_{dist} \cdot n)\) where \(avg_{dist}\) is the average distance from the current sensor values to all data points (in the 4D space of the sensor values for the four fingers) with the given label and n is the number of data points with that label (Algorithm 4). Given a number of fingers, D, and a number of past grasps M, the running time of the algorithm grows as \(O(D \cdot M)\). Further work should consider an adaptive cutoff score and post-processing to re-balance learned classes. Such methods could allow the same algorithm to adapt to sensors with different levels of noise.

To first evaluate the general grasping capability of the soft hand, we tested it with a wide variety of objects. The full set of 100 objects can be seen in Fig. 6. Some grasps of these objects can be seen in Fig. 7.

During the experiments, each object was placed at a known pose and a grasp was attempted by the Baxter robot using the DRL soft hand. For this set of experiments, our goal was to focus on the grasping capability of the DRL soft hand, and therefore we took the liberty to implement different grasping strategies for different objects. (Section 5.3 presents our set of experiments where we evaluated the soft hand within an end-to-end system using autonomous perception and planning.) Some objects were grasped via enveloping grasps. Others were picked up via a top grasp with two or three fingers in a pinch grasp. The flat objects, e.g. the CD and the piece of paper, were grasped off of the table as was shown in Fig. 8b. All objects were positioned in the orientations as they are in Fig. 6. The DRL soft hand was able to successfully grasp and pick up 94 of 100 objects.

We made three key observations during these experiments.

First, the DRL soft hand was capable of grasping a wide variety of objects with different sizes and shapes. The objects in Fig. 6 were chosen to explore the extents of the grasping capability of the soft hand and it, thanks to its soft compliance, easily adapted to different shapes and sizes.

Second, the DRL soft hand was capable of grasping objects that require simultaneous compliant interaction with the environment. Specifically, we tested grasping a CD and a piece of paper off of a table, again using both the DRL soft hand and the default rigid gripper. The default gripper was unable to pick up a CD or piece of paper. Our soft hand was reliably able to pick up the CD and the piece of paper. Figure 8b shows how the soft gripper smoothly interacts with the environment to pick up the CD.

Third, the DRL soft hand was qualitatively better at grasping a compliant object when compared with the rigid gripper. Specifically, we tested grasping a soft paper cup using the DRL soft hand and the default rigid parallel grippers of the Baxter robot. When the default gripper picked up the cup (Fig. 8a), it crushed it; the soft gripper was able to pick it up repeatedly without crushing.

The DRL soft hand was not able to pick up six of the 100 objects. These objects can be seen in Fig. 6b. The hand was not able to pick them up primarily because they were too heavy or too flat for the finger to gain any traction on them. The gripper had trouble picking up a spoon, a pair of scissors, and a propeller because they were not high enough – the fingers were unable to make a good connection. The gripper was unable to pick up an amorphous bag of small objects because of the combination of its odd shape and heaviness, the fingers did not get a solid grasp below the center of mass and the bag deformed to slip out of the fingers. The fish tail could be grasped, but slipped due to its weight. The screw was simply too small to be reliably grasped.

We performed our second set of experiments to evaluate the proprioceptive capability of the DRL soft hand. First, we performed experiments to identify objects based on finger curvature after grasping. Second, we performed experiments to perform force-controlled grasping of objects.

1) Object identification using finger curvature: We first tested the trained object identification algorithm described in Algorithm 3. To characterize the hand’s capabilities in different grasping modes, we performed experiments both for enveloping grasps and pinch grasps. For enveloping grasps we used ten objects and the empty grasp (Table 1, Fig. 10), and for pinch grasps we used seven objects and the empty grasp (Table 1, Fig. 10). For each grasp type, we first performed ten grasps of an object and labelled it with its object id. Then we performed additional unlabeled grasps (55 for enveloping grasps and 40 for pinch grasps) and used Algorithm 3 to identify the objects based on proprioception. In Fig. 9, we present the distribution of the 4-dimensional proprioceptive data and the labels Algorithm 3 assigned to each grasped object. 94.5% of tests (52/55 trials) identified the objects correctly for enveloping grasps; the breakdown per object is shown in Table 1. For pinch grasps, 87.5% of tests (35/40 trials) identified the objects correctly; again, the breakdown per object can be seen in Table 1. This includes correctly identifying the empty grasp when the robot did not actually pick up an object.

We also tested the online object identification algorithm outlined in Algorithm 4 by grasping the same objects used in the pinch grasp tests. We started by training the empty grasp, then picked up the wood block, plastic lemonade bottle, goggles, hedgehog, bin, tennis ball, and eggbeater in that order. We trained the empty grasp with three iterations and then picked up each of the other objects three times. Except for the hedgehog, for which all grasps were identified as the previously-grasped lemonade bottle, the algorithm correctly identified each object as a new object the first time and as itself for subsequent grasps, for an identification success rate of 85.7%, on par with the trained results for pinch grasps. Notably, once the system identified an object correctly as a distinct object, it successfully matched future grasps of the object correctly in all cases. 6/7 objects were identified correctly as distinct objects. The plot of the identified data points can be seen in Fig. 11.

(2) Proprioceptive force-controlled grasping We also performed experiments to test the accuracy of the proprioceptive force-controlled grasping algorithm and the force sensors. First, we calibrated the force sensors and identified a threshold that was high enough so that we did not get any false positive contact signals. Afterwards, three grasps each of seven objects were tested. Table 2 shows how many fingers stopped grasping due to force contact before the grasp was completed.

We made two key observations during the proprioceptive grasping experiments using the finger curvature sensors and the force sensors.

First, during these experiments, we had to place a stiff pad behind the rubber bands to provide a backdrop (Fig. 12) and allow the primary conforming of the hand to come from the fingers rather than the rubber bands. This observation will play an important role in designing future versions of the hand.

Second, while the finger curvature sensors provided valuable data which resulted in an impressive object identification performance, the data from force sensors resulted in mixed performance as shown in Table 2. Many objects were simply too small so maximum closure was required before the force sensors were activated. For others, the fingers grasped them at an angle that did not activate the sensor.

In our third set of experiments we evaluated the grasping performance of the DRL soft hand under object pose uncertainty within an end-to-end system.

We performed this evaluation in three steps of increasing system complexity:For all three test types, we tested both side grasps and top grasps with an appropriate object for each. We present the details of these three sets of experiments below.

1) Grasping under Object Pose Uncertainty: The first tests considered the range over which top and side grasps would successfully grasp the object. The hand was centered over a 10x10 cm grid and performed repeated grasps of an object placed at the center. Over multiple trials, the object was moved to different positions in the grid and we recorded whether or not the object was successfully grasped. For each grasp, the hand moves to the same location; the purpose of this test is to see how much uncertainty in object location the gripper can handle while still successfully grasping the object.

For side grasps, the object used was the lemonade bottle. See Fig. 13a to see the configuration of the test setup and the approach angle of the hand. Figure 14a shows the grid used in the test with dots at points which were tested. For each location, two trials were performed. For the 41 locations shown in the grid, 82 grasps were attempted, 55 of which were successful.

For top grasps, the object used was a foam block covered with black electrical tape. The test setup can be seen in Fig. 13b. The object was placed with its longer dimension along the hand’s opening as seen in the figure. The hand descended on a centered position and closed three fingers in one of the types of top grasp. Figure 14b shows the grid used in the test with dots at points which were tested. For each location, two trials were performed. For the 121 locations shown on the grid, 242 grasps were attempted of which 145 were successful. Again, we observed that the soft hand was able to grasp the object reliably even when the object was significantly away from the center, the exact value changing between 3cm and 5 cm depending on the axis, since the object is asymmetric. Other asymmetries in the plot are due to different dynamic interactions between the fingers and the block during grasping.

In Table 3, we present the capture areas of the DRL Soft Hand as a general measure of how robust it is to object pose uncertainty during grasping. In the total area of \(100 \, cm^2\) within which the object position was varied, the capture area shows the size of the region for which the grasps were robustly successful. The values in the table are found by measuring the areas spanned by the green dots in Fig. 14. If a data point is missing on a particular grid point, we assumed that the capture region is convex (Dogar and Srinivasa 2010). We are only using this assumption for calculating the capture region as a means of representing the raw data in Fig. 14 with a summarizing metric. As Table 3 shows, the DRL Soft Hand was robust to an uncertainty region of \(53 \, cm^2\) during side grasps and \(66 \, cm^2\) during top grasps.

For side grasps, we again used a lemonade bottle to ensure a fair comparison between this test and the previous test. Figure 14c shows the grid used in the test with dots at points which were tested. For each location, two trials were performed. For the 33 locations shown on the grid, 66 grasps were attempted, 52 of which were successful. The difference in offset in the NW/SE axis offset versus the pre-programmed scenario is most likely due to an offset in the vision system, sending the hand to a different location, on average, than the correct location. The average location that the hand went to is approximately (.56m, .28m) while the ground truth location for a centered grasping pose is approximately (.55m, .26m), measured in the robot coordinate system. Specifically, this means that the robot was aiming more to the right and bottom of the image, shifting the pattern of successful grasps to the top left compared to the pre-programmed position, as expected. The size of the capture area, shown in Table 3, reduces to \(50 \, cm^2\) for side grasps with the end-to-end system. This is expected as the uncertainty of the vision system and the motion planner also affects grasping performance during these tests.

For top grasps, we again used the foam block with tape as a test object for grasping. Figure 14d shows the grid used in the test with dots at points which were tested. For each location, two trials were performed, except for the orange dots, where after one success and one failure an additional trial was run. For the 25 locations shown on the grid, 52 grasps were attempted, 34 of which were successful. Again, there is a slight offset, in the same direction, between the system with vision tests and the preset grasping location, again due to an offset in the vision system object detection. The extent of the grasp area is roughly similar to the prior test, though at the edges some more objects were knocked away during the trajectory than with the perfectly straight down trajectory in the previous trials. The size of the capture area, shown in Table 3, reduces to \(41 \, cm^2\) for top grasps with the end-to-end system. Again, this is expected due to the uncertainty of the vision system and the motion planner.

For side grasps, we used a dark cylindrical object. The results can be seen in Fig. 15a. Again, we use dots to show the data at each point; the locations represented by the dots were spread uniformly 10 cm apart in both dimensions. Two grasps were attempted at each grid point. For the 55 grid points, 110 grasps were attempted, 40 of which were successful.

Here, there are more potential causes of failure: sometimes, the vision system did not identify the object. Often, errors came from the vision system reporting an inaccurate location for the object – often due to objects far away from the camera being elongated due to perspective – so the arm knocked over the object on its trajectory over or failed to grasp it since the object was outside the successful uncertainty range. Asymmetries in the vision spotting are due to the fact that the right hand had a two-finger gripper while the left hand had a three-finger gripper which blocked more area.

For top grasps, we again used the same foam block as in the previous top grasp tests. Since the block was shorter than the cylindrical object used in the side grasp tests, the locations determined for grasps were much more accurate throughout the range of the table. This led to the increased success rate for top grasps versus side grasps. The results can be seen in Fig. 15b. Two grasps were attempted at each grid point. For the 55 grid points, 110 grasps were attempted, 51 of which were successful. Again, we use dots to show the data at each point; the locations represented by the dots were spread uniformly 10 cm apart in both dimensions.

We made two key observations during these final set of experiments.

Some potential methods to further improve the grasp success rate are to