Improving robot manipulation with data-driven object-centric models of everyday forces

Section 2 discusses related work and contrasts it with our approach. The next three sections present our contributions to three areas. First, Sect. 3 describes our methods of collecting force data from robots and humans, a quasi-static model for doors, and the object-centric representation that we use.

Second, Sect. 4 shows that a standard supervised learning classifier can recognize the class of a door (e.g., refrigerator or kitchen cabinet) and the specific instance of a door based on the forces during opening. Third, Sect. 5 presents a method for detecting anomalous forces using a probabilistic model of the forces captured while successfully opening different doors.

Section 6 presents a comparison of our anomalous force detection method with two baseline methods. It shows that using previously captured haptic data enables the detection of locked doors and contact between the door and an idealized rigid obstacle more quickly and with lower applied forces. Section 7 reports results of anomalous force detection from trials on two distinct robots and discusses the implications of online state estimation on the performance of our anomalous force detection method. Finally, Sect. 8 discusses limitations of our work and directions for future research, after which the paper concludes with Sect. 9.

Although vast quantities of everyday visual and auditory data can be easily accessed on the web, very little haptic data from everyday activities can be found. Researchers have looked at capturing haptic interactions in order to synthesize realistic haptic sensations for human users, for example (Pai et al. 2000; Dupont et al. 1999; MacLean 1996; Angerilli et al. 2001; Weir et al. 2004; Romano and Kuchenbecker 2011). Recent work has modeled the forces while opening a refrigerator door for realistic haptic feedback in a virtual environment (Shin et al. 2012). To date, however, this body of work has emphasized high-fidelity models of objects to convey realistic haptic sensations to people, rather than capturing haptic datasets to inform robots.

Recently, researchers have begun to collect datasets of physical properties of objects (Matheus and Dollar 2010) and of forces during some everyday activities, such as tooth brushing (Redmond et al. 2010), with the motivation of informing the design of robot hardware and software. In previous research, we captured the forces applied to door handles and the trajectories of the handles while pulling open doors with the same motivation (Jain et al. 2010). However, these previous works have not used the datasets to inform robot manipulation.

There is a large body of research on using realtime force and tactile signals to improve manipulation. Recent research in this domain includes work by Romano et al. (2011), Hsiao et al. (2010), Dollar et al. (2010), Prats et al. (2009), Platt Jr et al. (2011), Chitta et al. (2011). Similar in spirit to our approach of using previously captured forces, Chan et al. (2012) have characterized the forces involved in human-to-human object transfer and then presented guidelines for robot controllers for the same task. Wiste et al. (2011) have used force data from everyday activities from the literature on rehabilitation research as a guideline for the design of prosthetic hands.

In contrast to our use of data-driven object-centric models, previous research on haptic recognition and anomaly detection for robot manipulation has often used data-driven models that are robot-centric or models of the dynamics of the robot arm.

Recently, researchers have developed a number of robotic systems to operate doors between rooms (e.g., Klingbeil et al. 2008; Jain and Kemp 2009a; Meeussen et al. 2010; Chitta et al. 2010; Kalakrishnan et al. 2011; Kormushev et al. 2011), and open cabinets, drawers, and appliances (e.g., Jain and Kemp 2010; Wieland et al. 2009; Diankov et al. 2008; Rühr et al. 2012; Becker et al. 2011). These efforts have focused on controllers and planners that enable a robot to open doors. They have not addressed haptically detecting contact between the door and the environment (e.g., collisions), or haptically recognizing the door identity or class.

We have previously developed methods that enable a robot to autonomously open doors and drawers without prior knowledge of the mechanism kinematics (Jain and Kemp 2009b, 2010). We use these methods in this paper to enable the robots Cody and a Willow Garage PR2 (Personal Robot 2) to open doors Fig. (1).

In collaboration with Sturm, Stachniss, and Burgard, we have shown that a robot can use a database of kinematic trajectories of mechanism handles to increase the online prediction accuracy of the kinematic state of a mechanism that it is currently opening (Sturm et al. 2010). We do not use our research from Sturm et al. (2010) in this paper. However, it is complementary to the current paper as it looks at kinematic data, and this paper investigates haptic data.

Lastly, as introduced in Sect. 2.1, we captured the applied forces and the trajectories of the points of contact while people pulled open doors and drawers in our previous work (Jain et al. 2010). We showed that the forces can be transformed to an object-centric representation, and we discussed the implications of such a database of forces and kinematic trajectories for assistive robot design. In the current paper, we demonstrate how robots can use such a database in a variety of ways.

Our methods for haptic identification and anomalous force detection rely on modeling the relationship between the relevant forces, \(f\), applied to a mechanism, \(m\), and the mechanism’s state, \(\theta \). As such, we need to make measurements to obtain estimates of the relevant applied forces, \(\hat{f}\), and the relevant state of the mechanism, \(\hat{\theta }\).

In this work, our kinematic model of doors is a single degree of freedom (DoF) rotary joint whose axis of rotation is parallel to gravity, as shown in Fig. 2. We have found that at relatively low opening speeds, the configuration-dependent forces dominate the haptic interactions (Jain et al. 2010). So, our model assumes that the relevant state consists solely of the angle of the door.

Our model also assumes that the relevant force is the component of the total force applied to the handle that is tangential to the trajectory of the point of contact. From Fig. 2, the tangential component of the applied force will open the mechanism, while other components will result in constraint forces at the hinges. In this paper, we will refer to this tangential component as the opening force.

To measure the applied force, we used a rigid hook that we 3D-printed with ABS plastic and instrumented with a six-axis force-torque sensor (ATI Nano25 with a calibration of SI-125-3) (Fig. 3).

To estimate the mechanism state, we first measured the trajectory of the door handle as a human operator (Sect. 3.3) or a robot (Sect. 3.4) opened the door, using a motion capture system or forward kinematics, respectively. We then fit a circle to this trajectory to estimate the radius and the location of the axis of rotation of the door. This procedure enabled us to estimate the angle of the door for each point of the trajectory.

We also used the estimated angle of the door to compute the component of the force measured by the force-torque sensor that was tangential to the trajectory of the point of contact on the handle. We have previously described our method for estimating the relevant state and relevant force in Jain et al. (2010).

In previous work, we captured the forces and kinematic trajectories as human operators opened 29 doors and 15 drawers in 6 homes and one office. We created a database of estimates of the opening force applied to the handle, \(\hat{f}\), as a function of the estimated opening angle of the door, \(\hat{\theta }\). For this paper, we used this database for haptic identification (Sect. 4) and anomalous force detection (Sect. 5). Figure 1a shows part of our force and motion capture system, described in detail in Jain et al. (2010).

As in our previous work, we filtered the captured data and used trials with a low per-trial average velocity of the door handle. The mean of this average velocity across all trials with all doors was 18.2 cm/s and the median was 14.4 cm/s. For this paper, we removed from consideration all doors that had fewer than two trials in the database.

We use a feedback controller that we developed in Jain and Kemp (2010) to enable two robots to autonomously open a door without prior knowledge of the kinematics. The input to this controller is a 3D location and orientation of the door handle. Additionally, for this paper, we positioned each robot such that the handle was in its workspace and it was facing the surface of the door before running the feedback controller to autonomously open the door.

Both robots use the same feedback controller to autonomously open doors, but they have different low-level control. We use joint space impedance control on Cody (Jain and Kemp 2010) and a Cartesian space stiffness controller on the PR2 (Glaser 2010). Section 7.1 describes the two robots.

While each robot was opening a door, we recorded the trajectory of the tip of the hook (using joint encoders and forward kinematics), and the force measured by the force-torque sensor that we attached to the hook, as shown in Fig. 3. We then used the method described in Sect. 3.2 to estimate the opening force applied to the handle, \(\hat{f}\), and the angle of the door, \(\hat{\theta }\). We use haptic data from trials with the robots along with the database of haptic interactions from humans to report results of anomalous force detection in Sect. 7.

We represent a haptic interaction as a sequence of tuples with the estimates of the applied opening force and the mechanism’s state, \(\{ (\hat{f}_{1}, \hat{\theta }_{1}),\) \((\hat{f}_{2}, \hat{\theta }_{2}),\) \(\ldots (\hat{f}_{N}, \hat{\theta }_{N})\}\). Figure 4 shows examples of raw haptic interactions captured when people opened the pictured mechanisms. For opening doors, the handle defines the location at which the instrumented hook applies forces to the mechanism. For other mechanisms, the haptic interaction would need to explicitly include the point of application of the force relative to the manipulated object.

In this paper, we further process this raw haptic interaction into a more compact and uniform representation. We first quantize the opening angle of the door into 1\(^\circ \) intervals. We then represent each haptic interaction by a fixed length vector, where each element in the vector is set to the mean opening force applied in the corresponding 1\(^\circ \) interval, or set to \(NaN\) if the interval was not encountered. Within this paper, we will refer to this vector as the haptic interaction vector.

We now illustrate that haptic interactions from opening doors can be insensitive to some forms of task variation. First, Fig. 5 illustrates the small variation in the haptic interaction resulting from changes in the PR2’s position relative to the handle of a cabinet. For this test, we positioned the PR2 at three different positions 10 cm apart along a line parallel to the surface of the door. The relative height of the handle and its distance from the robot normal to the surface of the door were the same.

Second, Fig. 1 shows the mean and standard deviation (over multiple trials) of data from four sets of trials in which two humans and two robots opened the same mechanism. For our choice of the relevant component of the force, the configuration-dependent force due to the mechanism dominates the variation due to the operator. This observation and the results in Sect. 7 demonstrate that robots and humans can share haptic data through a common database of haptic interactions.

To reduce the influence of noise and overfitting, we first computed a low-dimensional representation of the haptic interaction vectors with principal component analysis (PCA) using singular value decomposition. This section is an extension of the analysis from our previous publication (Jain et al. 2010). Figure 6 shows the first two principal components and a scatter plot for 148 vectors from 26 doors with the points colored by mechanism class. The first three principal components account for 99.4 % of the variance over these 148 vectors.

The scatter plot of Fig. 6 shows that even after projecting to two dimensions, there tends to be separation between different classes of doors. We use these same classes in Sect. 4.3 to present results on haptic recognition. The points corresponding to the office cabinet class that are close to the freezer class are from a cabinet which has a magnet in the door, instead of a spring in the hinge. Qualitatively, the haptic interaction vectors for this cabinet have a similar shape but lower magnitude compared to freezers, potentially because freezers are also held shut using magnets.

We now present results on haptically recognizing a specific door after opening it. We assume that the database includes haptic interactions from previously opening the same specific mechanism.

Figure 7 shows the leave-one-out cross-validation error for a k-nearest neighbor classifier (\(k=1\) and \(k=3\)) and a support vector machine on our dataset for subspaces of different dimensionality. We used the multiclass C-SVM with a polynomial kernel of degree three as implemented in PyMVPA (Hanke et al. 2009).

The cross-validation errors with the kNN classifier (\(k=1\)) and the SVM were similar. The error was nearly constant for subspaces of dimensionality \(\ge \) 5. Figure 8 shows the confusion matrix for the kNN classifier after projecting the data onto the first five principal components. The leave-one-out cross-validation accuracy for identifying the mechanism was 89.7 %. Confusion occurred between mechanisms for which the opening forces were similar. Scaling this database up to a large size might reveal further mechanism subclasses, such as doors made by various manufacturers.

In this section we look at the problem of identifying the class of a mechanism after a human or a robot opens it. We assume that our database includes haptic interactions from opening mechanisms in the same class, but does not include the specific mechanism.

We assigned the class labels of “freezer”, “refrigerator”, “kitchen cabinet”, “office cabinet”, and “spring loaded door” to each haptic interaction vector in our database.

We used a kNN classifier (\(k=1\)) and a 5 dimensional linear subspace (PCA) for class recognition. For a selected mechanism, we generated a training set by removing all the vectors from that mechanism from our dataset to simulate opening the mechanism for the first time. We then tested the kNN classifier for each of the vectors from the selected mechanism, and repeated this procedure for all 26 mechanisms.

Figure 9 shows the confusion matrix for our dataset. The cross-validation accuracy for identifying the class for the 26 mechanisms, given that the specific mechanism had not been encountered before, was 86.4 %. Most of the classification errors were between the refrigerator and freezer classes. Both of these classes have similar opening forces as a function of the configuration, with the main difference being the initial force required to open them.

Haptic recognition can serve as a way for a robot to check that it is correctly performing a manipulation task. For example, by recognizing a specific mechanism, a robot could confirm that it opened the door it intended to open. Through recognition of a specific door, it might also infer its location in an environment. Recognizing the category of a mechanism could help a robot infer other properties. For example, the category of a door relates to its use, name, appearance, location, the objects found behind it and the category of the room. Haptic information in conjunction with other perceptual modalities could help make mechanism categorization more robust or support multi-modal category learning. For example, a robot could use haptic and visual features together to recognize a door while opening it, which could then be used to better detect anomalous forces, as described in the next section.

An example of the potential benefit of haptic data can be observed from Fig. 10, which shows the mean and standard deviation of the maximum opening force encountered over the first 10\(^\circ \) of opening. The large variation in this initial opening force for different mechanism classes illustrates that knowledge of a mechanism’s class can enable a robot to better select a maximum force to apply to a door before deciding that it is locked or malfunctioning. The applied opening force at which a robot would decide to stop pulling on a door’s handle could vary from less than 10 N for kitchen cabinets to greater than 60 N for springloaded doors. Without using this type of information, a robot risks damaging a locked cabinet or giving up prematurely on a functioning springloaded door.

We now present a probabilistic model of the relevant force, \(f\), applied while successfully operating a mechanism, \(m\), conditioned on the relevant mechanism state, \(\theta \), its class, \(C\), and any previous data from the specific mechanism, \(D_{\theta }\). \(D_{\theta }\) is a vector of any forces previously measured at state \(\theta \) while operating mechanism \(m\).

We model the relevant applied force at a particular mechanism state \(\theta \) as being normally distributed and conditionally independent of other forces, given \(\theta ,\,C\), and \(D_{\theta }\). So,where \(\mu \) is the mean and \(\sigma ^2\) is the variance of our Gaussian model for the relevant force at mechanism state \(\theta \). Our goal is to estimate \(\mu \) and \(\sigma ^2\) given \(\theta ,\,C\), and \(D_{\theta }\).

We detect an anomaly if the force measured at the current state of the mechanism, \(\theta \), equals or exceeds a threshold force, i.e.We do not investigate the potential for a lower bound on the force, although a low force could also be indicative of an anomaly. Additionally, our detector only uses the opening force at the current angle of the door. As discussed in Sect. 8.2.4, features that use information over time or across multiple configurations of the mechanism could also be beneficial for anomaly detection.

We present three methods of determining \(f_\theta ^{thresh}\). The first method uses our probabilistic model of expected forces (Sect. 5.2) to detect when forces are unlikely given the mechanism’s state \(\theta \), the mechanism’s class \(C\), and any previous operation of the mechanism \(D_{\theta }\). For this method,where the parameter \(n\) serves as a detector threshold that adjusts the sensitivity and specificity of the anomaly detector.

The other two detectors are baseline methods that use no prior information about the mechanism or its class. The first baseline detector sets \(f_\theta ^{thresh}\) equal to a constant \(c\). The second baseline detector sets \(f_\theta ^{thresh}= r \cdot \hat{f}_{initial}\), where \(r\) defines a fixed ratio of the initial opening force. We have used this method in our previous work (Jain and Kemp 2010). Like \(n,\,c\) and \(r\) serve as detector thresholds that adjust the sensitivity and specificity of these anomaly detectors.

We evaluated the performance of anomalous force detection methods using: (1) the increase in the magnitude of the force from the onset of contact between the door and an obstacle and the detection of anomalous force; (2) how much time passes between the manually labeled onset of contact and the detection of anomalous force; and (3) the false positive rate (i.e. the frequency with which the detector incorrectly reports a force as being anomalous). Lower force implies less risk of damage to the robot and the environment, and less risk of injury to people. Faster detection implies that the robot can more efficiently respond to the event that caused an anomalous force, such as by trying a new strategy or stopping and asking for assistance. A lower false positive rate implies that the robot will be less likely to falsely detect an anomalous force and thereby unnecessarily change its approach or give up.

Our database of haptic interactions consists of collision-free trials only. In order to compare the performance of these three anomaly detection methods using this database, we simulated locked doors, and contact between the door and an idealized fixed rigid obstacle.

We modeled these situations as a force that increases monotonically with time while the configuration of the mechanism remains constant. The actual rate of increase of the force over time would depend on the robot control algorithm. For example, with an impedance controlled robot, we would expect the force to increase at a rate that depends on the stiffness at the end effector.

This model enabled us to simulate contact at any configuration of the mechanism and investigate how well our anomaly detection methods could detect lower magnitude anomalous forces while avoiding false alarms.

For this comparison, we simulated contact at discrete angles (1\(^\circ \) intervals) for all operations of all doors. Given our model of contact with an idealized rigid obstacle, each of the three detectors will eventually detect the contact, since the magnitude of the applied force will continue to increase over time. As such, we focus on the increase in the magnitude of the force from the onset of contact until its detection, and the false positive rate. This increase in the magnitude of the force is the excess force applied to the mechanism before the detector reports contact with an obstacle. False positives correspond to the detector incorrectly reporting contact for a collision-free trial.

Let us assume that the door is at a configuration \(\theta \) when it makes contact. For a given value of \(f_\theta ^{thresh}\), the excess force before contact is detected will be \(f_\theta ^{thresh}-\hat{f}\), where \(\hat{f}\) is the estimated opening force at configuration \(\theta \) for one collision-free trial of pulling open the mechanism door.

We compute the average value of this excess force over all the configurations of all the trials of all the mechanisms to obtain the mean excess force before an anomalous force is detected. This average value gives us the y coordinate of a point in the plot of Fig. 11. The x axis is the percentage of the configurations for which \(f_\theta ^{thresh}< \hat{f}\). Since our database contains only collision-free trials, \(f_\theta ^{thresh}< \hat{f}\) is a false positive for anomaly detection.

Figure 11 shows the performance of the three detectors for different values of \(n,\,r\), and \(c\) on 148 trials from 26 different rotary mechanisms belonging to 5 different classes. Each point in the plot represents a different value of \(f_\theta ^{thresh}\), obtained by changing the value of the parameters \(n,\,r\), and \(c\) for the different methods of detecting anomalous forces. For all detectors, increasing the free parameter (\(n,\,r\), or \(c\)) decreases the false positive rate, but increases the unnecessary force applied prior to detection.

The blue curve in Fig. 11 shows the performance of our data-driven detector when opening a particular mechanism of a known class for the first time. For it, we computed \(f_\theta ^{thresh}\) for each trial in our dataset by ignoring all data from that specific door and computing the sample mean and variance over all the other mechanisms in the same class, as described in Sect. 5.2.1. The red curve shows the performance of our data-driven detector when opening a particular mechanism of a known class for the second time. For it, we used the data from all the other mechanisms in the same class, along with a single collision-free trial from the mechanism, as described in Sect. 5.2.2 with \(n=1\). We computed our results using all possible initial trials for a mechanism. For example, 5 total trials for a particular mechanism in our dataset would result in 20 tests, due to there being 4 possible initial trials for each of the 5 trials.

Figure 11 shows that using captured haptic data can reduce the excess force for a given false positive rate. It shows that knowledge of the mechanism class enables the detection of an anomalous force with a lower excess force, since for any false positive rate, the blue curve is below the green and yellow curves. Additionally, it shows that even a single trial with the specific mechanism decreases the excess force further, as evidenced by the fact that the red curve is below the blue curve for all false positive rates.

We used two different mobile manipulators for this paper. Figure 1 shows the two mobile manipulators, and Fig. 3 shows how we mounted the hook end effector to them. One mobile manipulator is a PR2 robot from Willow Garage, which has two 7 DoF compliant arms with low gear ratios and current control for the motors at the joints (Wyrobek et al. 2008). The second robot, Cody, has two compliant 7 DoF arms from Meka Robotics with series elastic actuators for torque control at each DoF.

For low-level 1 kHz control on the PR2, we use low stiffness gains with an open source controller (Glaser 2010) that is similar to Cartesian stiffness control (Salisbury 1980). On Cody, we use joint-space impedance control running at 1 kHz (Jain and Kemp 2009b). Although the two robots have different lower-level control and actuation, we use equilibrium point control as described in Jain and Kemp (2010) on both robots to autonomously open doors with a hook end effector.

To detect anomalous forces using the methods of Sect. 5.3, a robot needs to generate online estimates of the state of the mechanism and the opening force while operating the mechanism. We estimate the radius of the trajectory of the handle, \(r\), and the location of the axis of rotation, (\(c_x, c_y\)), which we then use to compute an estimate of the mechanism state and opening force. We denote \((r, c_x, c_y)\) with \(\beta \).

We assume that a perception algorithm gives the robot an initial estimate of the radius of the trajectory of the handle, \(r_p\). As an example, the perception algorithm could compute \(r_p\) based on the estimated width of the door and the location of the handle (Rusu et al. 2008). In addition, the robot estimates the pose of the tip of the hook using forward kinematics while operating the mechanism, giving it an estimate of the trajectory of the handle, \(((x_1, y_1), (x_2, y_2),... (x_n, y_n))\) which we denote as \(T_n\). The number of points in the trajectory of the handle, \(n\), increases with time.

Given \(r_p\) and \(T_n\) we compute a maximum likelihood estimate (Bishop 2006) of \(\beta \) asWe assume that the observed trajectory, \(T_n\), and the perception algorithm’s estimate of the radius, \(r_p\), are conditionally independent given \(\beta \). So,Next, we assume that the perception algorithm’s estimate of the radius is normally distributed around the true radius of the mechanism, with a variance of \(\sigma ^2_r\). So,We compute \(P(T_n| \beta )\) using the assumptions detailed in Sturm et al. (2009), which include assuming that the measurements of the points along the trajectory of the handle, \((x_i, y_i)\), are conditionally independent given \(\beta \) and have a Gaussian error with a variance of \(\sigma ^2_{pos}\). These assumptions result inEquation 11 then simplifies towhich we optimize using the implementation of the BFGS algorithm from SciPy (Jones et al. 2001). We then use \(\hat{\beta }\) to compute the current state of the mechanism. For our tests, we set \(\sigma _r\) = 10 cm, and \(\sigma _{pos}\) = 1 cm. We set these values conservatively to reflect uncertainty due to a variety of factors. For example, perceptual uncertainty when estimating the width of the door would influence \(\sigma _r\), and uncertainty due to joint encoder resolution and our use of a hook with a layer of compliant rubber would influence \(\sigma _{pos}\).

For our results with robots in Sects. 7.4 and 7.5, we used the entire dataset of human trials. This included data from humans opening the same refrigerator and cabinets that the robots opened. As a result, although we investigated cases where the robot opens a door for the first or second time, the database used to model the prior for the class of mechanism (\(P(\mu , \sigma ^2 | \theta , C)\) from Sect. 5.2.2) included haptic interactions from humans opening the same door.

We investigated how noise in online estimates can impact the performance of our data-driven anomalous force detector. For this evaluation, we recorded the trajectory of the hook and the applied force while robots opened doors. We collected a total of fifteen collision-free trials consisting of the PR2 opening an office cabinet five times, Cody opening a different office cabinet five times, and Cody opening a refrigerator five times.

We then generated multiple haptic interaction vectors, simulating online estimation of the mechanism state and a noisy initial estimate of the radius of the door (\(r_p\)), as follows: for each of the 15 trials, we computed multiple values of \(r_p\) in Eq. 15 by sampling from a Gaussian with mean equal to the true radius of the mechanism and a standard deviation of 10 cm. We then used \(\hat{\beta }\) from Eq. 15 to generate simulated online estimates of the state of the mechanism and the opening force, which is the component of the force tangential to the trajectory of the point of contact on the handle. The online estimates of the state of the mechanism improved as the robot opened the door through a larger angle.

Figure 12 shows the trade-off between the mean excess force and the false positive rate, analogous to the results from Sect. 6.2. Errors in the estimates of the mechanism configuration (due to uncertainty about the radius and location of the axis of rotation) resulted in poorer performance when the robot operates a mechanism for the first time (higher force on average before detection of contact with an idealized rigid obstacle). When the robot operates a mechanism with a known identity for the second time, it has the opportunity to use its previous estimates of the mechanism’s radius. We would also expect an improvement in performance if a robot were to use methods for kinematic estimation that yield more accurate initial estimates of the radius of the door and the state of the mechanism, such as in Rusu et al. (2008), Sturm et al. (2009, 2010).

We also investigated the performance of our data-driven anomalous force detector with real collisions and a real locked door. We performed six trials with two robots, Cody and the PR2, shown in Fig. 13. We either placed an obstacle in front of the door or locked the door. We processed data collected from these trials off-line for anomalous force detection using the detector based on our probabilistic model of applied forces (Eq. 10). We set \(n\) in Eq. 10 as the least value that resulted in zero false positives over our entire dataset of human trials. Further, we assumed that the robot has an accurate estimate of the radius of the mechanism (as opposed to being drawn from a Gaussian around the true radius as described in Sect. 7.2). Table 1 shows the time (detection time) and additional force (excess force) between the onset of contact with the obstacle and detection of an anomalous force, or between the force on a locked door first exceeding the expected force and detection of an anomalous force.

Figure 13 illustrates the performance of our data-driven anomalous force detector. The yellow curve is the opening force applied by the robot during the trial. The dashed green and blue lines are the minimum forces at which our methods would report an anomalous force if the robot were operating the mechanism for the first time (with knowledge of the mechanism class), or the second time, respectively. The three numbered circles (1–3) in Fig. 13 represent the following: (1) the manually labeled onset of contact with an obstacle or when the force on a locked door handle exceeds the expected force; (2) the point at which the robot would have detected an anomalous force if it were operating the mechanism for the second time; and (3) the point at which it would have detected an anomalous force if it were operating the mechanism for the first time.

For trials 3 and 4, the robot has a slightly lower force threshold for detecting anomalous conditions when it is operating the refrigerator for the first time than when it is operating it the second time. This also shows up as similar or better performance while opening the first time compared to the second time for the results in rows 3 and 4 in Table 1. We believe that a lower threshold when opening it for the first time is due to the database of human trials having data from only four refrigerators, resulting in the mean and variance not being representative of the mechanism class.

Machine intelligence has benefitted greatly from large collections of sensory data. Web-based databases of user-generated content, such as videos from YouTube, images from Flickr, and 3D models from Google 3D Warehouse, have begun to support the development of robots and technology relevant to robots (Klank et al. 2009; Lai and Fox 2009; Kollar and Roy 2009; Kuffner 2010; Waibel et al. 2011). More generally, research has shown that large datasets can lead to performance gains and make computationally simple methods effective (Torralba et al. 2008; Halevy et al. 2009).

Our results in this paper suggest that humans and robots have the potential to share haptic data to improve robot manipulation. Additionally, by storing associated contextual information, such as where the interactions occurred and the appearance of manipulated objects, robots could anticipate haptic interactions.

In general, this type of common sense knowledge would help robots behave more intelligently. In the future, robots might use these data in numerous ways, including selecting better postures prior to manipulation, detecting when mechanisms are in need of repair, and anticipating when a human will require assistance. Moreover, these data could be used by humans to rationally design robots with the kinematic and force capabilities necessary to perform real-world tasks.

Enabling humans to easily capture the haptic interactions, such as with a wearable system, could potentially accelerate the accumulation of this type of data. With motion capture capabilities and sensors continuing to improve in quality and decrease in cost, there is the potential for the robotics community to accumulate large datasets in a practical manner. Likewise, robots in the field could potentially record their haptic interactions and upload them to an online database in order to produce a continuously evolving source of haptic knowledge for various manipulation tasks.