iCLAP: shape recognition by combining proprioception and touch sensing

The sense of touch plays an important role in robot perception and many tactile sensors have been developed in the last few decades (Zhang and So 2002; Xie et al. 2013; Dahiya et al. 2013). In addition to obvious applications of collision detection and avoidance, tactile sensors can be applied in multiple tasks such as object recognition (Luo et al. 2015; Pezzementi et al. 2011; Luo et al. 2018), dexterous manipulation (Kappassov et al. 2015; Bimbo et al. 2016) and localization (Luo et al. 2015; Pezzementi et al. 2011). The haptic object recognition can be considered at two scales, i.e., local and global shapes (Lederman and Klatzky 2009; Torres-Mendez et al. 2000). The local object shape, e.g., shapes that can fit into fingertips, can be recovered or recognized by a single touch, analogous to human cutaneous sense. The global shapes, e.g., contours that extend beyond the fingertip scale, usually require the contribution of both cutaneous and kinesthetic inputs. In such case, mechanoreceptors in joints are also utilized to acquire the movement of the fingers/end-effectors in space, with the assistance of local tactile features, to recognize the object identity. The kinesthetic inputs here are similar to human proprioception that refers to the awareness of positions and movements of body parts.

In this paper, an algorithm named Iterative Closest Labeled Point (iCLAP) and its extensions are presented to incorporate tactile and kinesthetic cues for haptic shape recognition. With only tactile readings, a dictionary of tactile features can be first formed. By searching for the nearest word in the dictionary, each tactile feature is assigned a label number. A four-dimensional data point is then obtained by concatenating the 3D position of the object-tactile sensor interaction point and the word label number. In this manner, 4D point clouds of objects can be obtained from training and test data. The partial 4D point cloud obtained from test set iteratively matches with all the reference point clouds in the training set and the identity of the best-fit reference model is assigned to the test object. Furthermore, weighted sum based and product based fusion strategies have also been developed for haptic shape recognition.

The contributions of this paper are:This paper extends our previous work (Luo et al. 2016) by including extension methods of the iCLAP algorithm and introducing more thorough experiments. The remainder of this paper is organized as follows. The literature in the haptic object shape processing is reviewed in Sect. 2. The proposed tactile-kinesthetic shape recognition system is described in Sect. 3. A series of the fusion approaches, both product-based and weighted sum based, are then introduced in Sect. 4. In Sect. 5, the experimental setup and the data acquisition are described. The experiment results are then provided and analyzed in Sect. 6. Finally, in Sect. 7 the conclusions are drawn and possible applications and future research directions are presented.

Thanks to the development of tactile sensor technologies (Khan et al. 2015; Dahiya et al. 2011; Li et al. 2013), haptic object shape processing has received increasing attention in the recent years (Luo et al. 2017). Some research produced point clouds by collecting contact points to constrain the geometry of the object (Casselli et al. 1995; Allen and Roberts 1989) whereas some others relied on tactile appearances by extracting features from tactile readings (Schneider et al. 2009; Pezzementi et al. 2011; Luo et al. 2015). In contrast, for humans, the sense of touch consists of both kinesthetic (position and movement) and cutaneous (tactile) sensing (Lederman and Klatzky 2009). Therefore, the fusion of spatial information of sensor movements and features of tactile appearance could be beneficial for object recognition tasks.

The methods based on contact points often employ techniques from computer graphics to fit the obtained cloud of contact points to a geometric model and outline the object contour. These methods were widely used by early researchers due to the low resolution of tactile sensors and prevalence of single-contact force sensors (Allen and Roberts 1989; Charlebois et al. 1999; Okamura and Cutkosky 2001). The resultant points from tactile readings can be fit to either super-quadric surfaces (Allen and Roberts 1989) or a a polyhedral model (Casselli et al. 1995) in order to reconstruct unknown shapes. Different from the point cloud based approaches, a non-linear model-based inversion is proposed in Fearing and Binford (1991) to recover surface curvatures by using a cylindrical tactile sensor. In more recent works (Ibrayev and Jia 2005; Jia et al. 2006; Jia and Tian 2010), the curvatures at curve intersection points are analyzed and thus a patch is described through polynomial fitting. Tactile array sensors have also been utilized to obtain the spatial distribution of the object in space. In Pezzementi et al. (2011) an object representation is constructed based on mosaics of tactile measurements, in which the objects are a set of raised letter shapes. Kalman filters are applied in Meier et al. (2011) to generate 3D representations of objects from contact point clouds collected by tactile array sensors, and the objects are then classified with the Iterative Closest Point (ICP) algorithm. Through utilizing these methods, arbitrary contact shapes can be retrieved, however, it can be time consuming when exploring a large object surface as excessive contacts are required to recognize the global object shape.

Another approach is to recognize the contact shapes by extracting shape features from pressure distributions within tactile images. The image descriptors from computer vision have been applied to represent the local contact patterns, e.g., image moments (Pezzementi et al. 2011; Corradi et al. 2015; Drimus et al. 2014), SIFT based features (Pezzementi et al. 2011; Luo et al. 2015) and raw readings (Schneider et al. 2009; Liu et al. 2012). However, there is only a limited number of approaches available for recovering the global object shape by analyzing pressure distributions in tactile images collected at different contact locations. One popular method is to generate a codebook of tactile features and use it to classify objects (Schneider et al. 2009; Luo et al. 2014, 2015; Madry et al. 2014); a particular paradigm is the bag-of-words model. In this framework, only local contact features are taken to generate a fixed length feature occurrence vector to represent the object whereas the three-dimensional distribution information is not incorporated.

For humans, haptic perception makes use of both kinesthetic and cutaneous sensing (Lederman and Klatzky 2009), which can also be beneficial to robots. In McMath et al. (1991) and Petriu et al. (1992), a series of local “tactile probe images” is assembled and concatenated together to obtain a “global tactile image” using 2D correlation techniques with the assistance of kinesthetic data. However, in this work tactile features are not extracted whereas raw tactile readings are utilized instead, which would bring high computational cost when investigating large object surfaces. In Johnsson and Balkenius (2007), the tensor product operation is applied to code proprioceptive and cutaneous information, using Self-Organizing Maps (SOMs) and Neural Networks. In Gorges et al. (2010), the tactile and kinesthetic data are integrated by decision fusion and description fusion methods. In the former, classification is done with two sensing modalities independently and recognition results are combined into one decision afterwards. In the latter, the descriptors of kinesthetic data (finger configurations/positions) and tactile features for a palpation sequence are concatenated into one vector for classification. In other words, the information of the positions where specific tactile features are collected is lost. In both methods, the tactile and kinesthetic information is not fundamentally linked. In a similar manner, in Navarro et al. (2012) the tactile and kinesthetic modalities are fused in a decision fusion fashion. Both tactile features and joint configurations are clustered by SOMs and classified by ANNs separately and the classification results are merged to achieve a final decision. In a more recent work (Spiers et al. 2016), the actuator positions of robot fingers and tactile sensor values form the feature space to classify object classes using random forests but there are no exploratory motions involved, with data acquired during a single and unplanned grasp.

In our paper, a novel iCLAP algorithm is proposed to incorporate tactile feature information into the location description of the sensor-object contact in a four dimensional space. In this way, the kinesthetic cues and tactile patterns are fundamentally linked. The experiments of classifying 20 real objects show that the classification performance is improved by up to 14.76% by using iCLAP compared to methods based on single sensing sources and a high recognition rate up to 95.52% can be achieved when a hybrid fusion is applied.

In the haptic exploration, at each data collection point, both the tactile reading and the 3D sensor location can be recorded simultaneously. An illustration of data extracted from a pair of scissors is depicted in Fig. 1. Our proposed Iterative Closest Labeled Point (iCLAP) algorithm is based on two sensing sources, i.e., appearance features from obtained tactile images and spatial distributions of objects in space.

To create a label for each tactile feature, a dictionary of tactile features is first formed from training tactile readings and each tactile feature is then assigned a label by “indexing" the dictionary. The dictionary is formed by clustering the features extracted from tactile images using k-means clustering, where k is the dictionary size, namely, the number of clusters. The features of training/test objects are then assigned to their nearest clusters in Euclidean distance and are therefore labeled with the cluster numbers \({\mu } \in \{1, 2,\ldots , k\}\).

With the feature labels created from tactile readings and the sensor locations in 3D space, a single 4D point p can be represented by a tuple \( p = (p_{x}, p_{y}, p_{z}, {p_{\mu }})\), where \( p_{x} \), \( p_{y} \), \( p_{z} \) are the x, y, z coordinates of the tactile sensor in 3D space and \( {p_{\mu }} \) is the word label assigned to this location respectively. In this manner, the object can be represented in a four-dimensional space. To calculate the mutual distance between 4D sparse data point cloud P in the test set and the model (reference) point clouds Q in the training set, the Iterative Closest Point (ICP) algorithm (Besl and McKay 1992) is extended to 4D space. Let data point \( p_{i} \) and model data point \( q_{i} \) be an associated set of the N matched point pairs. With the \(4 \times 4\) rotation matrix R and \(4 \times 1\) translation vector \( \vec {t} \), \( p_{i} \) can be transformed into the coordinate system of model point cloud: \( \overline{p_{i}}=\textit{R} p_{i}+\vec {t} \). To find the closest point in the model point cloud to each transformed test data point \( \overline{p_{i}} \), a k-d tree of the model point cloud is constructed (Zhang 1994; Bentley 1975). An error metric \( E_{iCLAP} \) is defined to evaluate the mean square root distance of the associated point pairs and it is minimized with the optimal rigid rotation matrix R and translation vector \( \vec {t} \). In physical terms, the error metric \( E_{iCLAP} \) can be visualized as the total potential energy of springs attached between matched point pairs. \( E_{iCLAP} \) of point clouds P and Q to be matched is \(E_{iCLAP}(P,Q)=\sum _{i=1}^{N}\Vert \overline{p_{i}}-q_{i}\Vert ^2\).

The centroids of P and Q are \(c_{p}=\frac{1}{n}\sum _{i=1}^{n}p_{i} \) and \( c_{q}=\frac{1}{m}\sum _{i=1}^{m}q_{i} \), where n and m are the number of test and model data points respectively. In general case, n is equal to the number of the matched point pairs N. The point deviations from the centroids of P and Q can be obtained as \( {p_{i}}^{\prime }=p_i-c_{p} \) and \( {p_{i}}^{\prime }=q_i-c_{q} \). \( E_{iCLAP} \) can be rewritten asTo minimize the error metric, \( \vec {t} \) is set as \(\vec {t}=c_{q}-Rc_{p}\). Therefore, the error metric is simplified aswhere R is orthogonal for orthogonal transformation, therefore, \(RR^T=\textit{I}\). Now let \( H=\sum _{i=1}^{N}{p_{i}}^{\prime }{{q_{i}}^{\prime }}^T \). In an expanded form,where \( M_{ab} = \sum _{i=1}^{N} {p_{ia}}^{\prime }{q_{ib}}^{\prime }\), and \( \textit{a}, \textit{b} \in \{ x,y,z,{\mu }\} \). To minimize \( E_{iCLAP} \) the trace tr(RH) has to be maximized. Let the columns of H and the rows of R be \( h_{j} \) and \( r_{j} \) respectively, where \( \textit{j}\in \{1,2,3,4\} \). The trace of RH can be expanded aswhere the inequality is just a reformulation of the Cauchy–Schwarz inequality. Since the rotation matrix R is orthogonal, its row vectors all have unit length. This implieswhere the square root is taken in the operator sense and \( tr(\sqrt{{H^T}H}) \) is the trace norm of H that is the sum of singular values. Consider the singular value decomposition of \( H=U\Sigma V^T \). If the rotation vector is set as \( R=VU^T\), the trace of RH becomeswhich is the maximum according to Eq. 5. It means that \( E_{iCLAP} \) is minimized with the resulting optimal rotation matrix \( R=VU^T \) and translation vector \( \vec {t}=c_{q}-Rc_{p} \).

The iCLAP is iterated until any of the termination conditions is reached: error metric \( E_{iCLAP} > \) preset tolerance; number of iterations > preset maximum number of iterations \( n_{max} \); the relative change in the error metrics of two consecutive iterations falls below a predefined threshold. The obtained distances between the test and reference point clouds are then normalized by L2 norm. A reference point cloud with the minimum \( E_{iCLAP} \) can be found and its identity is assigned to the test object by comparing \( E_{iCLAP} \).

In general, the fusion of multiple modalities can provide complementary knowledge and improve the decision-making performance; it can be performed at either the feature level or the decision level (Atrey et al. 2010; Liu et al. 2017). In feature fusion, the features extracted from different modalities are combined into high dimensional feature vectors prior to the classification, which are fed into a single classifier. In essence, our iCLAP algorithm is performed at the feature fusion level. However, due to the distinct representations of information sources, i.e., sensor locations and tactile labels in our case, normalization is hard to be performed and it needs to find the best normalization parameters by trial and error. Instead, decision fusion combines the decisions made based on individual modalities and makes a final decision in the semantic space where individual decisions usually have the same representation. Therefore, the decision level strategy is also adopted. Two different decision fusion methods are developed, i.e., weighted sum-based and product-based. To take advantage of both feature and decision fusion methods, hybrid fusion approaches are developed to combine the decisions of iCLAP and methods based on single sensing modalities. In total, nine synthesis methods, including one at feature level (iCLAP), two at decision level and six at hybrid level, are developed. There are two recognition pipelines, i.e., tactile based and kinesthetics based.

As illustrated in Fig. 2, the experimental setup consists of a tactile sensor and a positioning device. A Weiss tactile sensor WTS 0614-34A¹ was used that has an array of \(14 \times 6\) sensing elements. It is of 51 mm \(\times \) 24 mm and has a spatial resolution of 3.4 mm \( \times \) 3.4 mm for each element. The sensing array is covered by elastic rubber foam to conduct the exerted pressures. A rate of 5 fps was used as we used in our initial studies (Luo et al. 2014, 2015). A Phantom Omni device² with six degrees of freedom was used for positioning the tactile sensor and its stylus served as a robotic manipulator. The stylus end-effector position can be obtained that has a nominal position resolution of around 0.055 mm. The tactile sensor is attached to the stylus and its surface center is aligned with the end point of the stylus, therefore the end-effector position can be taken as tactile sensor position in the world coordinates.

During the data collection, each object was explored five times. Each exploration procedure was initialized without object-sensor interaction. The stylus was controlled with a speed of around 5 mm/s to explore the object while keeping sensor plane normal to the object surface; in this manner, the object surface was covered while a number of tactile observations and movement data of the tactile sensor were collected. Following Luo et al. (2015) and Schneider et al. (2009), an uninformed exploration strategy was employed. In total, 8492 tactile images with corresponding contact locations for 20 objects were collected, as shown in Fig. 3. It can be found that some objects are of similar appearances or spatial distributions. For example, pliers 1 and 2 are of similar size and have a similar frame, whereas they have different local appearances, i.e., the shape of jaws. On the other hand, some objects have similar local appearances but have different spatial distributions, for instance, fixed wrenches 1 and 2.

To evaluate the performance of the proposed iCLAP algorithm and extended methods, they are utilized to classify the 20 objects in the experiments. A leave-one-out cross validation is taken and averages of cross validation results are used. The general objective is to achieve a high recognition rate while minimizing the amount of samples needed. Following Luo et al. (2016), the dictionary size k is set to 50 through the experiments. In Luo et al. (2016), we also compared different features and Tactile-SIFT features (Luo et al. 2015) were found to perform best that are therefore also used in this paper.

In this paper we propose a novel algorithm named Iterative Closest Labeled Point to integrate tactile patterns with kinesthetic cues applied in the object recognition task. A bag-of-words framework is first applied to exploit the collected training tactile readings and a dictionary of tactile features is formed by clustering tactile features. By indexing the dictionary, each tactile feature is assigned a word label. The numerical label is then appended to the 3D location of tactile sensor where it is obtained, to compose a 4D descriptive point cloud of the objects. During the object recognition, the test object point cloud is transformed iteratively to find the optimal reference model in the training set and the distance between them serves as the similarity criterion. The experimental results of classifying 20 objects show that iCLAP can improve the recognition performance (up to 14.76%) compared to the methods using only one sensing channel.

We also extend iCLAP to a series of approaches by employing hybrid fusion strategies. The recognition performance has been further improved when the decisions of iCLAP and BoW framework (or also with classic ICP) are integrated using either product based or weighted sum based fusion strategy. Besides, it can be observed that the weighted sum based fusion method outstands when limited number of contacts is available whereas both strategies perform quite well when considerable readings are collected (greater than 10 touches). A satisfactory recognition rate of 95.52% can be achieved when 20 touches are used and a weighted sum based hybrid fusion strategy is employed to integrate decisions of iCLAP, BoW and classic ICP.

The proposed iCLAP and its extended approaches can be applied to several other fields such as computer vision related applications. In the view of scene classification (Cordts et al. 2016), as the landscape observations are correlated with the locations where they are collected, the proposed iCLAP combining the two sensing modalities is expected to enhance the classification performance. It can also be applied to medical applications such as Minimally Invasive Surgeries (MIS). If the local tactile patterns or visual appearances are merged with the spatial distributions of these observations by iCLAP, it is estimated to better recognize the interacted workspace within the body.

There are several directions to extend our work. As only the word label is utilized in the iCLAP to represent the tactile data, there is information loss to certain extent. Therefore, it is planned to include more clues of tactile patterns in the future designed algorithm. In addition, it will also be studied to recognize objects with multiple tactile sensing pads. It is proposed to implement our proposed algorithms on an instrumented robotic hand with multiple tactile array sensors on the fingers and the palm. And it will be explored how to minimize the number of touches to recognize objects in such cases.