Stiffness or elasticity estimation of everyday objects using robot grippers is highly desired for object recognition or classification in application areas like food handling and single-stream object sorting. However, standard robot grippers are not designed for material recognition. We experimentally evaluated the accuracy with which material properties can be estimated through object compression by two standard parallel jaw grippers and a force/torque sensor mounted at the robot wrist, with a professional biaxial compression device used as reference. Gripper effort versus position curves were obtained and transformed into stress/strain curves. The modulus of elasticity was estimated at different strain points and the effect of multiple compression cycles (precycling), compression speed, and the gripper surface area on estimation was studied. Viscoelasticity was estimated using the energy absorbed in a compression/decompression cycle, the Kelvin-Voigt, and Hunt-Crossley models. We found that (1) slower compression speeds improved elasticity estimation, while precycling or surface area did not; (2) the robot grippers, even after calibration, were found to have a limited capability of delivering accurate estimates of absolute values of Young’s modulus and viscoelasticity; (3) relative ordering of material characteristics was largely consistent across different grippers; (4) despite the nonlinear characteristics of deformable objects, fitting linear stress/strain approximations led to more stable results than local estimates of Young’s modulus; and (5) the Hunt-Crossley model worked best to estimate viscoelasticity, from a single object compression. A two-dimensional space formed by elasticity and viscoelasticity estimates obtained from a single grasp is advantageous for the discrimination of the object material properties. We demonstrated the applicability of our findings in a mock single-stream recycling scenario, where plastic, paper, and metal objects were correctly separated from a single grasp, even when compressed at different locations on the object. The data and code are publicly available.
Notes
Shubhan P. Patni and Pavel Stoudek contributed equally to this work.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1 Introduction
Although visual object recognition has seen tremendous progress in recent years, not all object properties can be perceived using distal sensing. In particular, physical object properties like stiffness, roughness, or mass are better perceived through manipulation. Object recognition from haptic exploration can be more robust as it is insensitive to lighting conditions and object attributes that may be irrelevant (e.g. color).
Sanchez et al. [1] provide a survey of robotic manipulation and sensing of deformable objects. Objects are considered deformable if they have no compression strength (ropes and clothes) or have a large strain.1 Additionally, classification based on geometry is presented. In this work, we leave objects of types I–III (linear, planar, cloth-like) aside and focus on type IV: triparametric objects—solid objects such as sponges or plush toys, which are also the least researched object type [1]. Here we extend the elasticity range and test much less elastic but still deformable materials like cardboard, plastic, and metal.
Advertisement
In this work, we specifically focus on material elasticity and viscoelasticity, and to what extent it can be estimated online using common robot equipment like two-finger grippers. Grippers are designed for grasping rather than stiffness estimation, and often lack tactile sensors at the fingertips, hence their ability to provide accurate estimates of physical quantities like the modulus of elasticity is limited. With the help of a professional biaxial compression testing device, we systematically assess the performance of these grippers and a force sensor at the robot wrist, under different settings.
The primary application areas are in fruit ripeness estimation and single-stream recycling. Research on automatic fruit ripeness estimation (tomatoes, nectarines [2], apples, strawberries [3]) or delicate edible manipulation [4] featured special-purpose soft or variable stiffness grippers and tactile sensors. Automatic waste separation often relies on distal sensing ([5] discuss near infrared (NIR), X-ray fluorescence (XRF), and vision sensing for plastic separation); Chin et al. [6] presented a special-purpose gripper prototype for haptic discrimination of plastic, paper, and metal. Additional application areas where deformable objects need to be handled include object picking for manufacture [7, 8], virtual/augmented reality [9], and medical diagnosis through palpation of subcutaneous tissue [10]. Our work differs in that we evaluate the potential of the use of standard industrial robot grippers for online elasticity estimation and material recognition from a single grasp.
This article is structured as follows. After reviewing related work, the essentials about the physics of deformation are explained (Section 3). Sample objects and the measuring devices are described in the Experimental setup section, followed by Experiments and results. The final sections of the paper provide a brief conclusion of all the work, followed by discussion and future work.
The dataset including raw as well as processed data collected during the experiments in this work is publicly available [11, 12]. Two accompanying videos are available: Part 1, on the results of using different models to estimate elasticity and viscoelasticity https://youtu.be/8DMMdEezC1M; Part 2, showing the application of stiffness estimation to sort materials in a single-stream recycling scenario: https://youtu.be/XHiNKFZ158o.
Advertisement
2 Related work
Recent surveys of haptic or tactile robot perception are provided by [13, 14]. Li et al. [13] list the object and material properties that can be extracted via tactile perception: stiffness, friction, surface texture, thermal conductivity, and adhesion. However, Luo et al. [14] point out a gap between rapid development of tactile sensing hardware and insufficient knowledge about how to interpret the sensory information obtained. In this paper, we try to reduce this gap by exploring the data streamed from the sensors and extracting material properties that may improve the interactions between robots and objects. We focus on stiffness/elasticity and how it can be perceived through a pinch grasp (squeezing). For deformable objects, the same action also allows us to perceive another material property: viscoelasticity. The next subsections provide an overview of the latest research in estimating stiffness and viscoelasticity in a robotic setup, i.e. without the use of industrial precision measurement instruments but rather with sensors commonly integrated with robots—RGB-D cameras and tactile sensors that measure reaction force.
2.1 Object discrimination from tactile feedback
Spiers et al. [15] used a two-finger compliant gripper with force sensors to classify objects of various shapes, sizes, and stiffness. Random forests were used for classification and the importance of different extracted features was evaluated. Delgado et al. [16] used the Shadow Hand to grasp deformable objects and computed the deformability ratio to infer the maximum force allowed to be exerted on the object. Hosoda and Iwase [17] used a human-like robot hand with artificial muscles and soft skin with tactile receptors as input to a recurrent neural network with context nodes for object classification. Industrial research in the area is focused towards recognizing objects for improved grasping and manipulation. Gemici and Saxena [18] learned haptic representations of food items. Four actions were used to explore six object characteristics (hardness, plasticity, elasticity, tensile strength, adhesiveness, brittleness). Scimeca et al. [19] used a two-finger gripper for mango ripeness estimation. Chin et al. [6] implemented a custom-made soft two-finger gripper with pressure and strain sensors to automated recycling separation (paper, plastic, metal). Finally, most related to this work, Wang et al. [20] used the deformation cues from pinch grasps of different soft objects to classify them using functional principal component analysis (FPCA features).
2.2 Estimating elasticity
Smardzewski et al. [21] employed a professional axial compression device for estimating the compressive modulus of elasticity, in line with the corresponding standards [22]. Such measurement is very different from practical robotics as grippers are much less accurate and the shape of real-world objects complicates measurements of pure material elasticity.
Some works employ the finite element method (FEM) to model the object based on deformation data. Frank et al. [23] estimated the stiffness by capturing the object via RGB-D camera, creating a three-dimensional finite element mesh in simulation software, and then numerically optimizing the stiffness parameter of the mesh by minimizing the error between the simulated and observed deformations. Visual and haptic information was also combined by Longhini et al. [24, 25]. Marker nodes placed on the surface of the deforming object were tracked visually to learn deformation models of various textiles for control and manipulation. Stress–strain models were used to gain information about deformable materials without explicitly estimating extracting stiffness/elasticity values. Narang et al. [26] used feedback from the BioTac tactile sensor as input to a multi-layer perceptron network to predict the deformation of the target object in simulation. The network was trained to convert feedback to deformation and vice versa via a latent space representation. This solution had been previously applied to a medical scenario in [27] for estimating the variable stiffness of soft tissue. However, the finite element method comprises optimizing the target parameter for a large number of nodes. It is computationally time consuming and cannot be employed for quick, online estimation. Further, these works press the object against a barrier with a fingertip, while in this work, we employ “squeezing” or pinching between the gripper jaws.
Zaidi et al. [28] employed a fingertip contact model simulating the deformation of the target object. The stiffness and damping were computed at the contact point and used as parameters in the deformation simulation to estimate grasp stability. Haddadi et al. [29] used the Hunt-Crossley dynamic model for real-time identification of contact environments. This method is quicker as compared to finite element methods because it regards the object as a whole rather than as a mesh of a large number of nodes. Only three parameters have to be computed from collected force vs. deformation data. Further, the feedback is only from the gripper and attached force sensors (no visual input). Bednarek et al. [30] did not use a robotic hand or gripper, but instead use a spherical tip with the OptoForce three-axis optical force sensor. The time series from this sensor was processed using an LSTM neural network employed for material classification. Yao et al. [31] used the compression action to create three-dimensional stiffness maps of household objects.
2.3 Our contribution
1.
We use three different robotic devices (two parallel jaw grippers and one force/torque sensor at the robot flange) and systematically study the effect of the characteristics of the device itself and different parameters of the object compression process on the quality of the mechanical response curves.
2.
We test a large set of deformable objects and employ a professional biaxial compression device to provide reference values.
3.
We assess the potential and limitations of robot grippers to gauge material elasticity—absolute values of Young’s modulus as well as relative ordering of objects by their elasticity.
4.
We assess the accuracy of different methods for estimating viscoleasticity.
5.
We consolidate our findings into a mock waste sorting scenario where a single grasp by the robot gripper suffices to separate recyclable materials.
6.
The collected dataset including the code for data processing is publicly available at https://osf.io/gec6s/.
The unique contribution of this work lies in a systematic experimental assessment of the potential and limitations of using standard robot grippers for elasticity estimation and object discrimination based on material composition. Related research on haptic elasticity estimation or material recognition has relied on custom-built complex grippers with dedicated sensors [2‐4, 6]. We compare several methods for gauging elasticity and viscoelasticity from the experimentally obtained stress/strain response curves and conclude with practical guidelines on how the material properties can be best estimated from a single compression cycle (grasp), culminating in a demonstrator resembling a waste sorting scenario.
3 Physics of deformation
3.1 Elasticity
If a force is exerted on an object, the object deforms and its dimensions change.2 Depending on the direction of the force, the object either compresses or elongates. In this work, we focus on axial compression as this mode of testing objects/materials applies to robot grippers. If the amount of compression, \(\Delta L\), is small compared to the dimension of the object, \(\Delta L\) is proportional to the force exerted. This is sometimes referred to as Hooke’s law and can be written as
$$\begin{aligned} F = k \Delta L, \end{aligned}$$
(1)
where F represents the force applied on the object, \(\Delta L\) is the change in length, and k is a proportionality constant. This linear relationship is a good approximation for many common materials, but applies only up to a certain \(\Delta L\) called the proportional limit. With additional compression, the linear relationship does not hold anymore, but up to the elastic limit, if the force is released, the object still fully returns to its original dimensions. The strain is the ratio of the change in length \(\Delta L\) to the original length \(L_0\). It is proportional to the force applied and the original length, but also inversely proportional to the cross-sectional area. To eliminate the object’s size or shape, a constant of proportionality that depends only on the object material is desired. This is the elastic modulus or Young’s modulus, E:
$$\begin{aligned} E = \frac{stress}{strain} = \frac{F/A}{\Delta L / L_0}, \end{aligned}$$
(2)
where stress is the force (F) per unit area (A) and has thus units \(Nm^{-2}\), or Pascals. For the soft materials in this work, we will use kiloPascals (kPa). Strain is a dimensionless quantity (no units).
3.2 Viscoelasticity
Soft objects like the polyurethane foams, for example, have an elastic component and a viscoelastic component. Due to molecular rearrangement, energy as heat is dissipated when a load is applied and then removed, i.e. during a compression and decompression cycle. The strain rate is also time-dependent. These properties can be characterized by the hysteresis in the stress–strain curve or by models taking into account both elasticity and viscosity. These will be detailed below.
3.2.1 Calculating viscoelasticity from loop enclosed area
The “paths” for compression and decompression on the stress/strain plot are different for deformable objects. There is an area enclosed by these paths—the hysteresis loop—which corresponds to the amount of energy lost (as heat) as shown in Fig. 1. By plotting the energy lost versus the speed of compression/release, we can obtain the viscoelastic coefficient \(\eta \).
Fig. 1
Sample stress–strain curve with hysteresis loop charactersistic of viscoelasticity
×
3.2.2 Kelvin-Voigt model
The Kelvin-Voigt model is a simple model that views every object as a spring and damper system arranged in parallel. The equation used to describe objects is a linear second-order differential equation:
where F is the applied force, K is the stiffness/elasticity parameter, \(\eta \) is the damping/hysteresis parameter, and x(t) and \(\dot{x}\)(t) are the deformation and rate of deformation respectively. Once a cycle of compression and release data has been collected in the form of deformation and force feedback, this data can be fit into Eq. 3 using multi-variable linear regression to obtain values of K and \(\eta \).
3.2.3 Hunt-Crossley model
The Hunt-Crossley model expands upon the simple Kelvin-Voigt model by adding nonlinear power terms to the stiffness and damping components. Equation 3 is modified as
The added power terms \(x^n(t)\) accommodate the nonlinearities of the material. The term n is a new parameter obtained as a result of the model. Since it is an exponential, instead of fitting data on this equation, we fit data on the logarithmic form:
The set of object samples is described, followed by the robot grippers and a professional measuring device. Additional details about the experimental procedure and how the robot grippers were commanded can be found in a student thesis [33]. The code used to command the grippers and preprocess data is available at [34].
4.1 Objects set
We considered four sets of deformable objects. The first set, cubes and dice, consisted of seven cuboid objects with a different size and degree of softness. The second set, polyurethane foams set, consisted of 20 polyurethane foam blocks of similar size, along with reference values for elasticity and density provided by the manufacturer. A third, mixed, set was formed from these two. For these object sets, the deformation can be regarded as elastic—objects returned to their undeformed shapes once the external force is removed. For some objects, this recovery was slow and they can be regarded as viscoelastic. A fourth set was formed from groceries for the final waste sorting demonstrator. Several of these objects were compressed beyond their elastic limit.
Cubes and dice set This set of seven soft objects is visualized in Fig. 2(a) together with their names. Their dimensions are listed in Table 1. The “blue cube” is composed of the same material like the “blue die”—it has been cut out from another exemplar of the same object. “White die”, “Kinova cube”, “Yellow cube”, and “Blue cube” have roughly same dimensions but different material composition. The dataset was deliberately designed in this way such that the effects of material elasticity and object shape and size would be apparent in the results.
Polyurethane foams set To complement the “ordinary objects set”, we tested on another set of objects: 20 polyurethane foams (Fig. 2(b)). These were samples provided by a manufacturer of mattresses and other foam products. The set includes hard insulation foams, memory foams, mattress foams, and soundproof foams. Compared to the ordinary soft objects, “cubes and dice”, this dataset is much more uniform—both in terms of object dimensions and elasticity. The first two letters code for the standardized material type. For example, “RP” stands for Richfoam Polyether (STANDARD 100 by OEKO-TEX); the “T” series are polyurethane foams used to make furniture (hyper elastic polyurethane furniture foams [21]). Additionally, the complete code also contains density and one measure of the material’s elasticity: the compression stress value at 40% compression (\(CV_{40}\) [22])—see Table 2. However, we could not directly use these elasticity values as reference because the testing conditions prescribed by the ISO standard [22] cannot be met with robot grippers; moreover, these values have been obtained from samples tested within 72 h after manufacture.
Fig. 2
a Cubes and dice set. b Polyurethane foams set. c Mixed set
Table 1
Cubes and dice set—dimensions
Description
Dimensions (mm)
Kinova cube
56 \(\times \) 56 \(\times \) 56
Blue cube
56 \(\times \) 56 \(\times \) 56
Yellow cube
56 \(\times \) 56 \(\times \) 56
Blue die
90 \(\times \) 90 \(\times \) 90
White die
59 \(\times \) 59 \(\times \) 59
Pink die
75 \(\times \) 75 \(\times \) 75
Darkblue die
43 \(\times \) 43 \(\times \) 43
×
Mixed set (Cubes, dice, foams) An additional object set was formed from the previous sets, taking two “cubes and dice” (“Blue die” and “Kinova cube”) and six polyurethane foams—Fig. 2(c). This set of eight objects was tested on a professional biaxial compression setup (Section 4.4) to provide a reference for the measurements using robot grippers.
Waste sorting set The set shown in Fig. 3 was acquired in the local grocery store to provide a simple test-bed resembling a single-stream recycling plant where waste should be sorted according to the material composition into plastic, paper, metal, and other materials.
Table 2
Polyurethane foams set
Label
Dimensions (mm)
Density
\(CV_{40}\)
V4515
118 \(\times \) 120 \(\times \) 40
45
1.5
V5015
119 \(\times \) 120 \(\times \) 42
50
1.5
GV5030
118 \(\times \) 119 \(\times \) 40
50
3.0
GV5040
118 \(\times \) 118 \(\times \) 39
50
4.0
N4072
118 \(\times \) 117 \(\times \) 37
40
7.2
NF2140
105 \(\times \) 100 \(\times \) 50
21
4.0
T1820
125 \(\times \) 125 \(\times \) 50
18
2.0
T2030
125 \(\times \) 120 \(\times \) 40
20
3.0
T3240
123 \(\times \) 123 \(\times \) 50
32
4.0
T2545
125 \(\times \) 125 \(\times \) 50
25
4.5
RL3529
119 \(\times \) 118 \(\times \) 40
35
2.9
RL4040
117 \(\times \) 120 \(\times \) 40
40
4.0
RL5045
118 \(\times \) 118 \(\times \) 39
50
4.5
RP1725
118 \(\times \) 120 \(\times \) 41
17
2.5
RP2440
118 \(\times \) 120 \(\times \) 38
24
4.0
RP27045
117 \(\times \) 119 \(\times \) 39
270
4.5
RP30048
123 \(\times \) 121 \(\times \) 39
300
4.8
RP3555
117 \(\times \) 119 \(\times \) 39
35
5.5
RP2865
118 \(\times \) 118 \(\times \) 38
28
6.5
RP50080
121 \(\times \) 118 \(\times \) 39
500
8.0
Label—code supplied by manufacturer. Density is expressed in \({kg \cdot m^{-3}}\). Elasticity is the \(CV_{40}\) in kPa—compression stress value at 40% strain [22]
Fig. 3
Waste sorting objects set
Fig. 4
Robot devices. a Robotiq 2F-85 gripper. b OnRobot RG6 gripper. c Robotiq FT300 force/torque sensor
×
×
4.2 Robot setups
For experimentation, the following robotic setups were used: the Universal Robot UR10 with the OnRobot RG6 gripper and the Kinova Gen3 robotic arm with the Robotiq 2F-85 gripper. In addition, a force/torque sensor Robotiq FT300 was appended to the UR10 robot as a makeshift third setup.
4.2.1 Robotiq 2F-85 gripper
This gripper (Fig. 4(a)) has two fingers with 85-mm stroke and offers a grip force from 20 to 235 N. The fingertips dimensions can be approximated to a rectangle of \(37.5\times 22\,\textrm{mm}\). Velocity control was used during object compression. This is quoted in % of the maximum closing speed.
Action parameters: Four different nominal squeezing speeds were executed: 0.68% (approx. 1.6 mm/s), 14.45% (30 mm/s), 50.85% (80 mm/s), and 100% (131.33 mm/s). These speeds gave rise to approximately 500, 300, 100, and 50 samples (points on the stress/strain curve) per compression cycle, respectively.
Sensory channels: two channels: gripper position (gap between jaws 0–85 mm), motor current (A).
4.2.2 OnRobot RG6 gripper
The gripper (Fig. 4(b)) has a 160-mm stroke and adjustable force and gripper status feedback (digital and analog). Gripper fingertips’ surface area is \( 866\,\textrm{mm}^{2} \). Continuous closing while recording force feedback was not possible in our setup; it was approximated by incrementally increasing the force threshold.
Action parameters: None. The gripper was commanded to close with 1-mm increments until a certain force threshold was reached. Then, the threshold was incrementally increased. The force range was 25–120 N, with 1-N increments. Approximately 100 data points were obtained during compression of a typical object.
Sensory channels: gripper position (gap between jaws 0–160 mm) and flag for force threshold reached, plus the actual threshold (N). Data was collected only at the occasions when force threshold was reached.
4.2.3 Robotiq FT300 force/torque sensor
The UR10e manipulator is equipped with a Robotiq FT300 force/torque sensor (Fig. 4(c)) that has a measuring range \(\pm 300\,\textrm{N}\) and \(\pm 30\,\textrm{N}\cdot \textrm{m}\) for every axis in Cartesian coordinate system. The sensor noise for the force measurements should be 0.033 %.
Action parameters: None. We pressed the objects with the robot flange against the table with a speed of 30 mm/s, which gave rise to approximately 30 samples on the stress/strain curve.
Sensory channels: position in z-axis (mm) and force in the downward direction (N).
4.3 Gripper calibration
We used a S9M/500-N force transducer (strain gauge) along with a ClipX amplifier system (both from HBM) to calibrate the force feedback from the robotic grippers.
The Robotiq 2F-85 gripper outputs motor current in the range 0–1 A. Measurements were collected by manually operating the gripper as well as using velocity control with the range of speeds used later in the experiments. The relationship of force and current was found to be nonlinear and the best fit obtained was:
where i was the current reading (in Amperes) obtained from the gripper. Additionally, the closing speeds in % had to be calibrated and converted to mm/s (using the gripper position signal).
For the OnRobot RG6 gripper, feedback in Newtons is available. However, we have performed calibration as well and an approximately linear fit was obtained.
Raw gripper effort vs. position plots. a–cCubes and dice set.a Robotiq 2F-85 at 80 mm/s. b OnRobot RG6. c Robotiq FT300 at 30 mm/s. d–fPolyurethane foams set.d Robotiq 2F-85 at 80 mm/s. e OnRobot RG6. f Robotiq FT300 at 30 mm/s
×
×
4.4 Professional biaxial compression setup
This setup was deployed to provide reference or control measurements for the “mixed set” of objects (Section 4.1). A biaxial experimental system Zwick/Roell was used (see Fig. 5). Samples were cyclically compressed between two antagonistic actuators, with a position resolution of \(1\mu \cdot m\). The actuator axes were connected to force sensors U9B from HBM with a 0.5-kN range, set to \(\pm 250\) N, suitable for static and dynamic force measurements (accuracy class 0.2; IP67). On every side, a transparent pad of plexiglass was attached, with dimensions \(59\times 59\times 10\) mm and mass of 60 g. The sample was placed between the pads and cyclically compressed and decompressed from both sides. Six cycles of compression and decompression were performed for every sample.
Action parameters: Compression speeds: One sample (RP1725) was tested with 1, 1.6, 10 30, 50, and 80 mm/s. Remaining samples from the mixed set were tested with 1.6, 30, 50, and 80 mm/s. Decompression speed was the same as compression speed with the exception of sample V4515 which took long to restore its volume—1.6 mm/s decompression was used. On average, we acquired 1200 points for the compression portion of each cycle. This is significantly higher than the data acquired from the robotic grippers.
Sensory channels: Force (N) and displacement (mm) at every actuator.
5 Experiments and results
The presentation of experiments starts with plots of mechanical response curves from the different devices in their standard units, followed by a transformation into stress/strain curves. The effect of compression speed and precycling is then analyzed. Several methods to estimate the Young’s modulus of elasticity from the empirical data are compared in Section 5.4, followed by estimations of viscoelasticity. Finally, Section 5.7 presents a practical demonstration of the findings in a mock waste sorting scenario. Illustration of some of the experiments is available in the accompanying videos: Part 1, on the results of using different models to estimate elasticity and viscoelasticity https://youtu.be/8DMMdEezC1M; Part 2, showing the application of stiffness estimation to sort materials in a single-stream recycling scenario: https://youtu.be/XHiNKFZ158o.
5.1 Gripper effort and position
We compare the raw plots that can be obtained in the standard operation mode of every robotic device, as shown in Fig. 6 for the cubes and dice. No preprocessing or filtering of the data is applied. On the x-axes, object compression proceeds from right to left, as the gripper width decreases. Contact with the object is not detected and as the objects have different width, the curves corresponding to gripper effort start rising at different locations.
For the Robotiq 2F-85 gripper, 50% compression speed (approx. 79 mm/s) was used. Motor current is plotted against the gripper width. For the OnRobot RG6 gripper, force feedback was retrieved only during discontinuous object compression with incremental adjustment of the force threshold. During measurements with the FT300 sensor, the robot arm was commanded to press downward against the object with a speed of 30 mm/s.
Fig. 7
Mixed set—stress/strain. a Robotiq 2F-85 gripper. Compression speed 80 mm/s (50%). b OnRobot RG6. Discontinuous compression. c Robotiq FT300 force/torque sensor. Compression speed 30 mm/s. d Professional setup. Compression speed 50 mm/s
×
From Fig. 6, it is apparent that the mechanical response curves from all three devices have similar characteristics. The force/torque sensor (Fig. 6(c),(f)) delivers the most clean data. The lines on the OnRobot RG6 plot (Fig. 6(b),(e)) are noisy and not monotonic. This is because they were not collected during continuous compression of the objects, but through incremental increasing of the force threshold. The least accurate response curves come from the Robotiq 2F-85 (Fig. 6(a),(d)), mainly because there are too few data points (combination of limited sampling rate and high speed of compression).
Fig. 8
Compression and decompression cycles with different speeds on professional device—mixed set. a–c Visualizations from biaxial compression device—force at one sensor over displacement of one jaw. Six compression and decompression cycles for every speed and sample. d Variation of estimated moduli with cycle number and speed, for the foam RP50080
×
The 20 polyurethane foams constitute a more homogeneous object set, with similar dimensions and a narrower range of elasticity. Similar observations about the precision and continuity of data can be made for the deformable objects set and the polyurethane foams set.
5.2 Stress/strain response curves
To estimate material elasticity, the stress/strain relationship during compression of the material in question needs to be obtained (see Section 3 and Eq. 2). The key requirement is to detect the moment of contact of the gripper with the object, which marks the beginning of the stress/strain curve; the gripper width corresponds to \(L_0\). To recognize the moment of contact, the force/effort needs to be measured, which is necessary at any rate to obtain stress. The accuracy of contact detection affects the results and is gripper dependent. Position can typically be measured accurately and thus obtaining strain as \(\Delta L / L_0\) is straightforward. To obtain stress as force/area, the surface area will be the smaller from the gripper jaws and the side of the object. The crucial part, however, is the (compression) force. If force is not directly available from the gripper, its calibration is necessary (Section 4.3).
After collecting raw compression data on the robotic setups from both the objects set and the polyurethane foams set, the mixed set of foams and cubes was created (Section 4.1) and tested on a professional biaxial measuring device (Section 4.4). The stress/strain curves are shown in Fig. 7. The curves from the professional setup, Fig. 7(d), will serve as reference. We observe that the samples are clustered with the “Kinova cube” being the least elastic material, followed by RL5045 and RP50080 foams. The rest of the samples occupy the bottom of the plot. This clustering is apparent also on the data from the RG6 gripper, Fig. 7(b), and the FT300, Fig. 7(c); it is less pronounced for the 2F-85 in Fig. 7(a). The nonlinear response visible in the feedback recorded by the professional setup is partially preserved in the plots from the other devices. An important observation is the range on the y-axis. The absolute values of stress differ significantly, which limits the possibilities of estimation of the real value of the modulus of elasticity. The 2F-85 values seem most in line with the professional setup; the FT300 is underestimating the stress by approximately a factor of 2; the RG6, on the other hand, is overestimating the stress by the same factor. For estimation of real values of elastic moduli, such discrepancy is problematic.
5.3 Effect of compression speed and precycling
The ISO standard for low-density polymeric materials [22] prescribes that samples be compressed at a speed of 1.67 mm/s and that the stress/strain characteristics should be evaluated during the fourth compression cycle. To assess the effect of these parameters, we first used the biaxial professional setup. The compression and, in this case, also decompression curves for the objects of the mixed set are shown in Fig. 8(a)–(c). Note that the upper part of every curve corresponds to compression. With higher speeds, the difference between the compression and decompression curve (hysteresis) is bigger. Individual compression cycles (1–6) did not have a significant effect on the overall shape. Figure 8(d) shows how the estimated modulus varies at different values of strain and with compression speed and cycle.
We have further studied the effect of speed and precycling on estimating Young’s modulus (Section 5.4.3 and Fig. 10) across the different devices. Precycling indeed did not significantly affect the results. Slower compression speeds tend to have more samples per compression cycle, which is advantageous when using regression models to estimate material properties from feedback.
Fig. 9
Young’s modulus estimation—OnRobot RG6 gripper. a Variation of \(R^{2}\) scores with window size. b The best fit line plotted for different window sizes. Four cycle 5 of compression of the foam RL5045. Similar curves were obtained for all foams
×
5.4 Young’s modulus of elasticity
For the materials we tested, the stress/strain curves are nonlinear. Therefore, the modulus of elasticity will be different at different points on the curve. Thus, we estimate at different values of strain (0%, 5%, 40%, and 70%) by locally estimating the gradient, and a “linear” modulus that assumes the whole curve is linear.
5.4.1 Positions on stress/strain curve and window size
The response curves obtained from grippers are not densely sampled and are noisy. Hence, prior to modulus estimation, the feedback is filtered using the Savitzky-Golay filter which smooths data using least-squares polynomial fitting on smaller window sizes. For the analysis of the data from the OnRobot RG6 and the Robotiq 2F-85 in this section, calibrated force values (Section 4.3) are used.
To find the modulus at any chosen strain percentage, we create a neighborhood around the strain value and find the slope of the best fit line in that neighborhood using scikit-learn’s LinearRegression module. To ensure that the best fit line obtained is actually representative of the slope at the chosen point, we vary window sizes to obtain the highest \(R^{2}\) correlation as a measure. It is observed that this \(R^{2}\) vs. window size variation is consistent in shape for the modulus values obtained at 40% and 70% strains. As shown in Fig. 9(a), it can be seen that smaller window sizes have the highest \(R^{2}\) scores at 40% and 70% strain. Alternatively, the \(R^{2}\) scores for the best fit lines near the 0% and 5% strain values are very poor.
The window size for our best fit lines was fixed at ± 10% strain on either side. All results obtained from the robotic grippers are based on this window size.
5.4.2 Estimating Young’s modulus from professional setup
We calculate the modulus values from the data from the professional setup. Modulus at 0%, 5%, 40%, and 70% was estimated using linear regression. Also, a “linear” modulus was calculated for the entire curve. For one sample, RP50080, statistical tests (t-test) were conducted to assess whether the compression cycle or the speed affect the modulus estimation. For the moduli at 0% (\(E_0\)) and 5% (\(E_{05}\)), significant differences between cycle 1 and cycle 5 (\(p=0.01\)) were found. For \(E_{40}\) and \(E_{70}\), no significant differences were found. Statistical tests comparing moduli estimation between slow speeds (1–10 mm/s) and fast speeds (30–50 mm/s) were also conducted and no significant differences were found. See also Fig. 8.
The moduli \(E_{40}\) and \(E_{70}\) moduli for the mixed set of samples are shown in Table 3 (mean over compression speeds, first compression cycle only). The error is presented as a ratio between the standard deviation and the mean. Since the sampling was dense, approximately 2% window on both sides of the desired strain value was sufficient.
Table 3
Mean values of 40% modulus (\(E_{40}\)) and 70% modulus (\(E_{70}\)) for mixed object set on the professional setup; first compression cycle, all speeds
Foam
\(E_{40}\) (kPa) (Err)
\(E_{70}\) (kPa) (Err)
Blue die
6.783 (10.0%)
45.609 (15.2%)
Kinova cube
82.638 (17.9%)
82.638 (17.9%)
NF2140
9.756 (10.2%)
46.047 (18.6%)
RL5045
26.323 (4.5%)
116.240 (14.5%)
RP1725
6.391 (8.6%)
20.557 (29.0%)
RP30048
12.439 (5.7%)
50.479 (20.6%)
RP50080
20.044 (3.53%)
75.912 (9.9%)
V4515
9.952 (27.32%)
42.400 (30.68%)
Fig. 10
Mixed set—Young’s modulus at 40% strain. Estimation using different devices and settings. a Compression at 1.6mm/s, no precycling effect. b Compression at 80mm/s, no precycling effect. c Compression at 1.6mm/s, with precycling. d Legend
Fig. 11
Linear modulus obtained by different grippers (compression speed = 1.6mm/s); mixed set. Please note the different scale on the y-axes
×
×
5.4.3 Young’s modulus—all devices
Data from all the different devices and different settings were used for Young’s modulus estimation on the mixed set sample. Estimations at 0% and 5% strain are not reported here as these were hard to obtain in all but the professional setup due to sparse sampling and presence of noise. At least five compression and decompression cycles were performed with every setting and the moduli were estimated at cycle 1 and cycle 5. Speed was varied for the professional setup and for the Robotiq 2F-85 gripper. In addition, to account for the effect of the surface area of the gripper jaws, the same pads (59 \(\times \) 59 mm) used in the professional setup were mounted onto the OnRobot RG6 and Robotiq 2F-85 jaws as an additional setting.
The results of Young’s modulus estimation at 40% strain are shown in Fig. 10. The green bars in every subplot are from the professional setup and serve as reference. The following observations can be made: (1) no robotic device provides an accurate and reliable estimate of Young’s moduli corresponding to the values obtained from the professional setup; (2) adding the pads and thus increasing the surface area for the two-finger grippers brings the absolute values closer to the reference, but adds noise to the estimation (disrupting the relative ordering of samples by elasticity); (3) the FT300 force sensor provides the most stable estimation, albeit the absolute values of Young’s modulus are significantly lower compared to the reference. This is consistent with the raw strain/stress curves in Fig. 7; (4) there is no clear effect of precycling (5th versus 1st compression cycle); (5) bigger compression speed degrades the estimation for the Robotiq 2F-85 gripper.
Fig. 12
Viscoelasticity—mixed set, on the professional setup. Variation of the amount of energy lost per compression and release cycle with different speeds
×
5.4.4 Linear modulus and relative elasticity estimation
The previous sections have shown that the capability of robotic grippers to deliver accurate estimates of Young’s modulus at different strain values is limited. Due to noise and limited resolution of the devices, local windows for estimation at e.g. 40% and 70% are unreliable. From a practical point of view, relative discrimination of objects based on their elasticity will often suffice. After experimentation, we propose to measure the stress/strain ratio over the whole compression area. This assumption is not correct since the mechanical response curves are clearly nonlinear, but it provides stable results in terms of ordering the samples by their elasticity, largely consistent across the different devices—see Fig. 11 and below:
$$\begin{aligned}&Kinova Cube> RP50080, RL5045> V4515,\nonumber \\&RP30048, NF2140 > RP1725, Blue Die \end{aligned}$$
(8)
Table 4
Viscoelastic coefficient obtained by calculating the hysteresis energy loss for multiple speeds on the professional setup
Comparing viscoelasticity (\(\eta \)) values obtained using the Hunt-Crossley model from the professional setup (left) and the Robotiq 2F-85 gripper (right), averaged over all testing speeds: 1.6, 30, 50, and 80 mm/s
×
5.5 Viscoelasticity
5.5.1 Viscoelasticity and the hysteresis loop
All the materials composing the mixed set have hysteresis loops (explained in Section 3.2.1) as can be seen in Fig. 8. The amount of energy lost is shown to be linearly proportional the compression speed in Fig. 12. The linear coefficient of the energy lost vs. speed graph is a characteristic of the material, and we label it \(\eta \). The values of this energy loss coefficient obtained from the professional setup are listed for all materials in Table 4. The materials fall approximately into two groups based on the values of \(\eta \). The viscous component is pronounced for the Blue die, Kinova cube, and the V4515 polyurethane foam. The Blue die and Kinova cube differ from the foams in that they are not completely homogeneous—they have a smooth outer shell that encloses the inside of the material and changes the properties. The difference in viscosity of the polyurethane foams may have to do with their porosity. Viscosity can provide and additional dimension next to elasticity to improve discrimination of soft materials.
5.5.2 Elasticity and viscoelasticity—Kelvin-Voigt and Hunt-Crossley models
All the models to characterize the viscous characteristics of the samples introduced in Section 3.2—area inside the hysteresis loop, Kelvin-Voigt model, and Hunt-Crossley model—were applied to the mixed set samples as measured by the professional setup and the Robotiq 2F-85 gripper.3 Both devices were used to compress and release the samples with the following speeds: 1.6, 30, 50, 80 mm/s. To find parameters of the Kelvin-Voigt model and the Hunt-Crossley model, data from the first and fifth cycles for different speeds was fit into Eqs. 3 and 4 individually and then averaged.
An overview of the results of estimating viscoelasticity from the Hunt-Crossley model is in Fig. 13. The correspondence between the viscosity/damping parameter obtained from the professional setup and from the Robotiq 2F-85 gripper is not satisfactory for the area inside the hysteresis loop (\(R^{2}=0.46\)) and the Kelvin-Voigt model (\(R^{2}=0.35\)) is not satisfactory. Only the Hunt-Crossley model (\(R^{2}=0.81\)) provides a reasonable correlation—we will thus use this model in what follows.
5.6 Material discrimination from elasticity and viscoelasticity
Given the limited capabilities of robotic grippers to gauge material elasticity (Section 5.4) and viscosity (Section 5.5.2) alone, if the goal is online discrimination of deformable objects, it is advantageous to use both these quantities simultaneously. This is visualized in Fig. 14 with the values obtained from the professional setup and the Robotiq 2F-85 gripper side by side. One can observe that there is a good qualitative match between the two measuring devices and that the second dimension importantly aids the possibility of discriminating the objects.
Fig. 14
2D object discrimination—elasticity using the linear assumption (Section 5.4.4) on x-axis and viscosity as obtained from the Hunt-Crossley model (\(\eta \) Eq. 4) on y-axis. (Left) Professional setup. (Right) Robotiq 2F-85. Clusters formed are marked by hand
×
5.7 Online waste sorting demonstrator
Finally, we demonstrate the feasibility of object classification via physical property estimation in a mock waste sorting scenario. A robotic arm was presented with a sequence of objects that it had to grasp and lift, classify into one of four classes of recyclable materials, and deposit them in their respective bins. We demonstrated this on the Kinova Gen3 robot with the Robotiq 2F-85 gripper. A video is available at https://youtu.be/XHiNKFZ158o.
A new dataset with the objects shown in Fig. 3 was collected. These objects were typical and potentially recyclable grocery items. The aim was to sort these objects into different classes of recyclable materials, utilizing the Hunt-Crossley model whose performance we found to be the best. The gripper was provided with the grasping position and initial thickness of the object as input, and estimated the stiffness and damping values of the object from the feedback from single squeezing action. Stiffness and damping are properties that represent the elasticity and viscoelasticity of objects, while taking into account not only the material of the object, but also the shape and geometry. Our four material categories were Paper and Cardboard, PET and Plastic, Sheet Metal Containers, and Too Stiff. First, sample data was collected from the set of recyclable objects on the professional setup and on the Kinova Gen3 robot, and stiffness ranges were derived for each of these categories. Then, we set up a routine that repetitively explored objects and sorted them based on estimated stiffness.
Raw stress–strain plots for the recycling objects set. a Using the professional setup, compression at 1.6 mm/s. b Using the Kinova Gen3 arm with the Robotiq 2F-85 gripper, with different objects, compression at 1.6 mm/s. c Using the robot with different grasps on the same objects, compression at 1.6 mm/s
×
The data presented in Table 5 shows how the properties estimated by the Hunt-Crossley model in Eq. 4 varied for irregular objects of different materials. Notably, objects made from recyclable plastics seem to have very low \(\eta \) and very high exponent n based on the shape of their stress–strain curves. Further, our material categories could be separated based on ascending order of the stiffness value, which allowed us to make a practical demonstration of the usefulness of stiffness estimation in robotic systems. The sample data from the test objects are shown in Fig. 15. The unprocessed data from the Robotiq 2F-85 gripper was converted from current-position to stress–strain via Eq. 6, and with knowledge of the gripper fingers’ surface area and the object’s thickness. The objects’ estimated properties are presented in Table 5, and the model fits are displayed graphically in Fig. 16. Figure 17(a) shows the data collected from the professional setup and the estimated stiffness and damping values of different objects. Figure 17(b) shows different test objects being categorized via stiffness estimation. Figure 17(c) shows the effects of different grasp positions on the classification task—even though the estimated values are different when objects are grasped at different locations, the values are still within range of their material categories. From Fig. 17, we also observed that although the absolute values estimated by the robot are erroneous, the relative ordering of the stiffness of different materials remains consistent across both setups.
Fig. 16
Model estimation performance. The solid lines represent actual data from the objects on the Robotiq gripper compressing the objects at 1.6 mm/s. The dashed lines are recreated by the Hunt-Crossley model
×
6 Conclusion
We have systematically experimentally evaluated the capacity of standard robot grippers to gauge material elasticity and object stiffness and the possibility of discriminating the sample objects based on their material composition. A professional biaxial compression device served as a reference. We have adapted and tested several methods to process the mechanical response curves obtained by squeezing the objects. Based on our results, we provide specific practical recommendations. The data and code are available at https://osf.io/zetg3.
Gauging elasticity/stiffness:
We found (i) the robot grippers, even after calibration of the force feedback, were found to have a limited capability of delivering accurate estimates of absolute values of the modulus of elasticity (Young’s modulus) at different compression levels; (ii) relative ordering of the objects based on their stiffness is largely consistent across the devices; (iii) precycling—measuring elasticity not on the first compression cycle but on the fourth or fifth as recommended by standards [22]—did not have a significant effect and hence data from a single compression cycle can be directly used; (iv) slower compression speeds are advantageous for estimation. Although the mechanical response stress/strain curves of the soft objects are nonlinear, in practice, it proved more accurate not to attempt fitting gradients and estimating Young’s modulus at different compression levels. Instead, fitting a single straight line over the whole stress/strain curve gave the most stable results.
Gauging viscoelasticity:
Deformable objects typically have viscoelastic properties characterized by (i) a hysteresis curve corresponding to energy dissipation during a compression and decompression cycle and (ii) a damping term that adds the compression velocity to the stress/stain relation (see Section 3.2). We found (i) the area under the hysteresis curve is hard to use in practice as it requires continuous compression and decompression, which is possible only for specific objects like foams, and accurate measuring apparatus; (ii) the damping term in the Hunt-Crossley model can be fitted online from a single compression and estimation from gripper data (Robotiq 2F-85) was sufficiently similar to the estimation from the reference setup in terms of the relative ordering of the samples’ damping term. The viscoelasticity parameter can thus also be estimated from a single grasp and complement the elasticity estimation, allowing better material discrimination thanks to the second dimension.
Online waste sorting:
We developed a mock single-stream recycling scenario that sorted objects regularly found in dry waste into their material classes by estimating their material properties. We collected data on four different material classes Paper and Cardboard, PET and Plastic, Sheet Metal Containers, and Too Stiff on with the Robotiq 2F-85 gripper. By using our best performing model, the Hunt-Crossley model, we estimated stiffness and damping for all objects and material classes, and used these estimates to successfully categorize all objects in real time.
Fig. 17
2D stiffness/damping classification with all the recycling objects. a Using the professional setup, compression at 1.6 mm/s. b Using the Kinova Gen3 arm with the Robotiq 2F-85 gripper, with different objects, compression at 1.6 mm/s. c Using the robot with different grasps on the same objects, compression at 1.6 mm/s
×
Practical recommendations for object discrimination based on material elasticity:
From the experiments and analyses performed, we derive the following guidelines: (i) a single grasp (compression cycle) with a two-finger robot gripper gives rise to stress/strain curves that separate even highly similar samples by their elasticity; (ii) two values (elasticity and viscoelasticity) can be estimated online from a single compression cycle facilitating object discrimination using even simple linear classifiers; (iii) robust estimates of elasticity and viscoelasticity were relative—preserving the same ordering of objects as estimated from the professional setup; absolute values of elasticity and viscoelasticity differ dramatically between setups; (iv) elasticity and viscoelasticity values of objects can be estimated online and simultaneously from a single compression cycle using the Hunt-Crossley model and allow for discriminating grocery items by their material composition; (v) slower compression speeds yielded better estimates and discrimination; (vi) precycling or compression followed by decompression did not improve estimates, suggesting that a single grasp suffices and objects can be transferred to a corresponding container immediately after or even while they are picked.
7 Discussion and future work
Overall, this work provided a systematic investigation of the feasibility of online haptic discrimination of objects based on their viscoelastic properties using standard robot grippers. All the processing we used—detecting contact, obtaining the stress/strain curve, estimating elasticity, and fitting a Hunt-Crossley model—is white-box and simple (compared to finite element method) and can work online. For discrimination, we used simple linear classifiers. Employing state-of-the-art classifiers will certainly increase performance (see [35] where SVMs and LSTMs were used on partly the same data). Visual-based methods can complement haptic data. The viscoelastic properties perceived can be further used to devise appropriate manipulation policies [18, 36].
In terms of the robot devices we tested, the Robotiq 2F-85 gripper is limited in its sampling rate, but it has the important capability to be able continuously measure the effort during a compression and release cycle. The effort is in motor current (A) and calibration is necessary to obtain force (N) to estimate the elastic modulus in real physical units. However, even after calibration, such estimation was not accurate. On the other hand, if only relative ordering or discrimination of viscoelastic objects is desired, this device will suffice. The OnRobot RG6 in our settings could only measure force when a set threshold was exceeded. The threshold then had to be reset to continue compression. Such measurement is slow but sufficient to coarsely estimate material elasticity. However, viscoleasticity or damping cannot be estimated as the rate of compression (or decompression) is not available. For comparison, we also show data from pressing an object with the robot flange and measuring the resistance with a force sensor at the manipulator’s wrist. This proved accurate but may not be applicable in many situations when grasping an object is needed simultaneously with elasticity measurements.
Acknowledgements
We would like to thank Karla Stepanova for her inputs, Bedrich Himmel for assistance with gripper calibration, and Jiri Hartvich for early code for online material classification.
Declarations
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Note that the OnRobot RG6 gripper cannot be used since continuous object compression with different speeds while recording force feedback is not possible.