Evaluation of haptic guidance virtual fixtures and 3D visualization methods in telemanipulation—a user study

Binocular vision has been shown to improve teleoperator speed and precision [11], but requires bulky and expensive 3D capable displays. In contrast, inexpensive commodity RGB-D cameras have provided non-contact means of collecting geometrical information and have grown in popularity in teleoperation applications [10, 33, 37, 51]. Stereo vision is another widely used alternative, and in robot-assisted minimally invasive surgery, stereo endoscopes provide real-time 3D geometries often represented as point clouds [24, 46‐48]. It has been shown that providing the teleoperator with depth information in the form of a real-time point cloud for certain navigation tasks can improve performance when compared to monocular RGB streams [32]. Another efficient method of displaying 3D surface geometries is through voxel occupancy grids [15, 53], which can preserve previously observed occluded geometries.

However, with depth information, several factors can deteriorate the quality of feedback and increase confusion to the viewer. For example, sensor resolution may be an issue when resolving smaller manipulation targets—the visual data may only provide a general localization of the object. Furthermore, depending on material properties, depth information may be noisy or arbitrary (e.g., transparent materials, glancing angles or light absorbant materials) [30, 35]. Other variables such as measurement distance have also been shown to affect measurement noise and density [22]. Because of this variability, it is not certain that 3D-mapping techniques can improve operator performance in telemanipulation tasks.

Mast et al. [32] determined that the usefulness of voxel-based 3D-mapping in navigation varied between environments. In this work, the utility of such 3D representations when applied to a manipulation task is explored. In such a task, 3D information may not be conveyed in a useful way and could in fact be detrimental and confusing when dealing with movement and small objects such as a valve handle. Additionally, occlusions could result in lack of information and heightened interpretation effort, while a monocular RGB stream is intuitive and familiar to most users.

Forbidden region haptic virtual fixtures help to prevent the operator from entering an undesired configuration. In [17, 18], forbidden region virtual fixtures in combination with pretouching sensing were used to help prevent unwanted contacts during exploration of an unknown, potentially delicate object. Similar types of forbidden region fixtures have found their use in surgical contexts [27, 39]. In contrast to forbidden region schemes, haptic guidance virtual fixtures push, prod or otherwise guide the operator’s hand in a desired direction or trajectory [1, 3, 31]. This is useful for maintaining a predefined trajectory [2, 38, 43, 44] as well as adaptive constraints [40]. In the case where depth perception is difficult, avoiding contacts and maintaining a safe, desired path can be assisted with such virtual fixtures. Vision and haptic virtual fixtures have been used in tandem for novel clinical applications as well [7, 8].

Moreover, since the virtual environment and force feedback are calculated in software, the actual robot end effector can be locked out of deviating from the desired path, while a guiding force is applied to the user. In [19], a flexible guidance fixture was demonstrated where computer vision was used to identify obstacles obstructing a predefined 2D virtual guidance trajectory. A modified trajectory was calculated that avoided obstacles. The above types of guidance fixtures deal with fixed, predefined paths. In the case of telemanipulation in an unknown environment, while the task may be predefined, its ideal configuration in the remote location is difficult to determine. However, when enough information about the physical task space is obtained, it is feasible and desirable for the teleoperator to place the desired trajectory [52].

Goodrich et al. [12] outlined several different schemes and levels for providing autonomous assistance in teleoperation. Peon et al. [41] explored the effect of different haptic modalities in combination with audio feedback on subject response time to violating a spatiotemporal constraint. Wang et al. [52] showed an increase in user execution time and accuracy from the combination of both the visual and haptic display of a guidance virtual fixture in a computer simulation, i.e., the user could see the desired trajectory as well as feel guidance forces. The guidance virtual fixture was generated and placed by the user using a computer mouse on a virtual surface [52]. While this method limited the user-defined virtual guidance fixture to the face of an object, it is extendable to trajectories in three-dimensional space. In a similar study, Kuiper et al. [25] examined haptic and visual feedback modes and their effects for nonholonomic steering, whereby a user controlled nonholonomic vehicle in a simulated steering task with different levels of constraints. The virtual fixtures were used to guide users along predicted or suggested vehicle paths. It was found that visual feedback is needed for improvements when providing only predicted trajectories. In [5], various haptic feedback modes for visuo-manual tracking for learning predefined trajectories, namely writing different Arabic and Japanese characters, were evaluated. It was suggested that haptic feedback can assist in learning to write 2D characters. In a similar study, it was shown that haptic information from handwriting can be compared and classified based on the users kinematic variations [54]. Different haptic assistance levels were assessed in completing a 2DOF maze navigation task in [40].

It has been established that haptic virtual fixtures can be useful for object avoidance and following a predefined path based on a priori information [3, 5], and several works explore 2D effects [5, 40]. In this work, the user is given the ability to manually set the virtual fixture for a useful path with predefined geometry to complete a 3D telemanipulation task—however, this path is not required to complete the task. For properly and efficiently placing this fixture, it is imperative that the user be provided with the real-time 3D-mapping representation—the negative effects of inaccuracies in shared haptic guidance feedback are described in [6]. In this study, depending on several spatial, temporal and sensor-limited factors (e.g., the user may have a difficult time placing the virtual fixture), it is not clear whether such a feedback option is beneficial.

The setup for this project includes a bilateral teleoperation arrangement. On the master console station, the teleoperator manipulates a haptic device, the Sensable PHANToM^™ Omni. This device sends 3DOF position commands to control the end effector location of the youBot, and it also receives and displays 3DOF haptic force feedback commands from the virtual fixture software. In addition, the user is presented with visual feedback on a LCD monitor. The teleoperator’s goal is to manipulate the robot arm to turn a gas valve. The master console setup is illustrated in Fig. 1.

The remote robot proxy includes a Kuka youBot robot with a Primsense Carmine RGB-D camera. The youBot has an omnidirectional base and a 5 degree-of-freedom (DOF) manipulator. In this implementation, these joints are controlled by National Instruments’ Compact RIO real-time controller, and commands to the master console are transmitted via an Asus AC router. These features are shown in Fig. 2.

To evaluate the effectiveness of the described feedback modes and user-placed guidance fixtures, teleoperator performances during the valve turn task are compared under the following different user feedback conditions:

Scenario 2 (V) provides a baseline for 3D visual representation. The user is able to rotate and translate his or her view within the 3D representation as well. This representation includes a 3D voxel map of a volume enclosing the youBot’s task space, which is updated with RGB-D sensor information based on a Bayesian statistical model to determine binary occupancy state. Similar methods were explored in [15, 35]; however in this study, the voxel allocation and update is hardware accelerated to ensure real-time acquisition and fast response to motion.

Scenario 3 (VF) provides the operator with a guidance virtual fixture of proper shape for the task and visualization from Scenario 2. This fixture will prevent the operator from deviating from a path known to successfully complete the valve turn and avoid undesired contacts, and overcome confusion caused by occlusion of the valve itself due to the manipulator itself. The trajectory is a series of finely sampled, ordered points. Because the environment is rendered as a 3D voxel grid, it is simple and quick for the operator to place the visualized desired trajectory properly. The three feedback modes are described in detail in Sect. 4.5.

It is of interest to determine whether or not, in this telemanipulation task, 3D-mapping techniques will improve operator performance, even if displayed to the user with 2D visual display. Furthermore, the efficacy of a user-placed guidance trajectory is explored. A comparison is sought between 2D monocular RGB stream (R), 3D voxel mapping techniques (V), and visual + haptic feedback (VF). Three questions are being explored:Because of the variability of the above factors, the nature of this experiment is exploratory and investigates a broad range of metrics with a simple, generalizable task. The results of this work will provide insight into the suitability of 3D-mapping methods and manually placed virtual fixtures for feedback in telemanipulation tasks, and inform metrics for future studies.

The operator is asked to complete a valve turn task. Such a task is motivated from a disaster recovery perspective. In the case of a gas leak during a natural disaster, teleoperation is attractive because it reduces risk to human responders. Moreover, a teleoperated device may be better designed to reach constrained physical scenarios than a human being. In this study, the valve to be turned consists of a ball valve structure with \(90^\circ \) dynamic range. The task can be broken down into two subtasks and turns: The task is depicted in Fig. 3, and it is with this setup that the user study for the project was conducted.

The slave robotic device, as described in Sect. 3.1, consists of a modified Kuka youBot platform and has been assumed to have reached the task location. The user is not required to navigate the robot base.

In this study, recruitment was performed on campus and subjects consisted solely of undergraduate and graduate students. As described previously, a total of three test conditions exist. In this project, a between-user study was employed, in which 21 male subjects participated (seven in each test group). Their age ranged from 18 to 35 years of age (mean age group R: 25.143; V: 23.000; VF: 26.143). Participants were chosen to be male to avoid any effects due to possible differences between males and females in spatial problem solving as described by [20].

Each of the participants used computers at least 10 h per week. In each group (seven participants total), six of the participants played less than 2 h per week of video games, while exactly one participant played more than 10 h per week (mean videogame usage per week R: 2.214; V: 1.857; VF: 2.000). None of the participants had prior experience using the Sensable PHANToM^™ Omni.

In this project, both objective and subjective metrics were employed for comparison. In particular, objective performance metrics included:Each trial was video recorded, and post-experiment was manually labeled for undesired collision count. After the completion of the task, subjective measures were assessed via post-task questionnaires evaluating:Perceived workload was measured using the unweighted NASA Task Load Index (TLX) [13], and situational awareness the three-dimensional Situation Awareness Rating Technique (SART) [50] scaled to (0, 120). Situational awareness is critical for producing effectual robot behavior [12].

The experiments were conducted in an office and the hallway corridor outside. The participant teleoperated from the master console within the office, while the simulated remote environment was in the hallway out of view from the subject. In the hallway, the youBot and the valve structure were placed in the same location for each experiment. Prior to the experiment, the users were allowed to see the valve and robot position, and were further allowed to turn the valve manually to obtain a sense of the range of motion as well as the torque needed to turn the valve.

After viewing the valve structure and youBot, the subjects underwent a training period which lasted for 20 min or when the user was satisfied, whichever happened first. (In all cases in this study, the user was satisfied with the training prior to the 20 min.) The training session occurred with the youBot in the office space within view of the operator. During the training session, the user was only allowed to teleoperate with their given feedback mode only. For the monocular RGB mode (R), the user was presented with a \(640\times 480\) video stream of the manipulator in well lit conditions, for (V) a voxelized representation and for (VF) the operator received the voxel map visual feedback and could furthermore place a haptic guidance fixture. Data was acquired at 50 Hz and visual feedback updated at 30 Hz, the data acquisition rate of the Primesense Carmine camera. The haptic update rate was set at 1200 Hz to maintain realistic force feedback. (A minimum rate of 1 KHz is needed for realistic interaction [28].)

Time to completion was measured from initial movement from the pre-calibrated home position to when the valve turn was completed. Unwanted collisions were manually labeled post-experiment. For this, physical contact of the robot manipulator with any object not the valve handle was considered unwanted. Distinct contacts required lift-off between the contacts; i.e., a dragged contact was counted only once. Path length was calculated via forward calculated end effector trajectory from recorded joint angles. Finally, differentials of the sampled position were used to approximate jerk. A low-pass filter removed high-frequency components introduced through discrete differentiation. These metrics measure operator performance in the valve-turning task.

Figure 8 shows graphically the results across the various feedback modes (R: monocular video stream, V: 3D-voxel mapping, VF: 3D-voxel mapping and guidance virtual fixture) along the four different quantitative metrics. The boxplots show mean values with standard deviation as error bars.

The four quantitative metrics, completion time, number of unwanted collisions, path length and jerk, were measured for each trial using repeatable and consistent methods. The results were grouped by feedback mode, and then, the mean for each category was calculated. The results are shown in Table 1.

Two post-experiment questionnaires were administered, the NASA TLX and SART, to evaluate task load and situational awareness respectively. Figure 9 shows these results. The user responses were compiled, and the scores were grouped by feedback mode. The mean results are shown in Table 2.

Several statistical approaches were employed to analyze the experiments, including a multivariate comparison as well as two different post hoc comparisons. Multivariate analysis of variance (MANOVA) was used as an omnibus measure to determine at least how many dependent variables (i.e., time to completion, number of collisions, path length, jerk, TLX and SART) may significantly differentiate the three groups (i.e., operating modes: R, V, and VF), while simultaneously controlling for multiple comparisons.

The MANOVA resulted in at least two degrees of freedom between groups. Further statistical analyses are thus warranted. The entire data is separated along the two identified degrees of freedom as shown in Fig. 10.

Figure 10 indicates that the second ranked feature deviates greatly from the top ranked feature. Post hoc approaches investigate pairwise comparisons to identify precisely which measures differentiate the three feedback modes, and future comparisons for study. First consider pairwise two-sample t tests between the three feedback groups within the quantitative metrics, as shown in Table 3. Consider next the pairwise two-sample t tests between the three feedback groups within the qualitative metrics, as shown in Table 4. The statistical significance metrics are explained in detail in the following text.

Recall that three separate experimental groups and six different metrics were examined. The result is a total of 18 different hypotheses considered from the data set. Thus, a multiplicity problem arises, and statistical analysis must account for this in order to avoid Type I errors, i.e., falsely rejecting a null hypothesis. Two different post hoc measures were employed:The former probes the propensity of making at least one false discovery, whereas the latter characterizes the probability of false positives.

With Holm–Bonferroni corrections, five total pairwise comparisons are shown to be significant despite the exploratory nature of this work. The significance of these comparisons for FEWR is shown in Table 7.

Holm–Bonferroni is a conservative measure of significance. The exploratory nature and motivation of this work lends itself to a more forgiving correction that focuses instead on FDR, which controls the proportion of discoveries that are false, and can indicate potential comparisons for further in-depth study. Thus, Benjamini–Hochberg corrections are more consistent with the stated contributions and goals of this work, and result in ten total pairwise comparisons of interest. These are shown in Table 8.

The data suggests that it is beneficial to use RGB-D data over a RGB video stream alone. The teleoperator performance was shown only to improve with this modification. Improvements in collision avoidance, overall path length, jerk and task load can expand accessibility and safe operation of limited and highly trained teleoperated tasks—the novice users in this study realized appreciable gains in these metrics from 3D visualization. One highly skilled teleoperated task is underwater telemanipulation: users operate tools, manipulate valves or match cables underwater. This is accomplished with macro-scale imaging [16, 23, 49] and is challenging and expensive due to the high skill level required [45]. Designing systems with small-scale depth imaging and voxel representation, as demonstrated in this work, can enhance the telemanipulation performance of less trained individuals, thus reducing the skill and subsequent operating costs to execute such operations.

Furthermore, when delicate or critical structures are involved, a collision could be disastrous. For such scenarios, this study shows that the addition of haptic virtual fixtures can further enhance telemanipulation collision avoidance. Search and rescue (SAR) robotics is one such application area, a case where safe control is essential in a hazardous, unstructured environment. Robots in this application, also known as response robots, can be used for incident prevention and support, and can be a useful tool for saving human lives and accelerating the search and rescue process. From data collected by search and rescue initiative ICARUS end users, it was a general consensus that in practical SAR applications, ”the robots will always need to be teleoperated [sic] for safety and legal reasons” [9]. These types of robots have been used in trials-by-fire, including response to the Chernobyl meltdown, the Fukushima Daiichi meltdown, the Sago mine disaster, terrorist attacks of September 11th, Hurricane Katrina and La Conchita Mudslide [4, 21].

Current response robots for SAR applications, however, operate under a minimal amount of autonomy and assistance modes. This may be due to the delicate nature of incorporating novel technology into high-risk operations, where safety and legal issues may arise. The complexity and unstructured context of many crisis and SAR missions also make such missions extremely technology-unfriendly [9]. In terms of current utilization, remote robot responders are used merely to obtain geographical information and to assist human responders—human responders are able to remotely navigate and access robot sensor data [9]. In this way, most current practical applications of rescue robots have been passively assisting human responders in situation assessment.

This user study helps to expand the scope and utility of telerobots in rescue situations. The data provides encouraging results toward integrating the use of 3D visualization and haptic assistance in telemanipulation. Of particular note is the significant improvement in collisions avoidance by adding a guidance virtual fixture. Further gains in jerk indicate that these telemanipulation interface improvements offer safer, more direct and smoother operation, all of which are encouraging for designing SAR application telerobots. This contribution provides results, limitations and future implications of the described user interface features. Such information is needed in order to intelligently investigate performance effects in focused studies, with ultimate goals to adopt new technologies that eliminate the need for and risk of in-the-field human responders, replaced instead with intelligently controlled telerobots with capabilities to safely and efficiently execute sensitive tasks.

This user study provides a baseline assessment of the effects of 3D voxel representations and user-placed haptic guidance virtual fixtures on operator performance in telemanipulation. Results provide a comparative baseline for evaluating the effects of additional augmentations, while the exploratory nature of this work involved numerous performance evaluation metrics. Encouragingly, the quantitative metrics resulted in comparison directions consistent with feedback augmentations, i.e., VF performed better than V, which was superior to R. While testing many metrics reduced statistical power, this experiment gives way and direction to focused studies into feedback modes for telemanipulation. In particular, in light of the false discovery rate corrections (compare Table 7 with Table 8), it would be of interest to investigate effects of guidance haptic fixtures and visual feedback onFurthermore, comparing soley VF and V with more complicated telemanipulation tasks (e.g., increased clutter, nonlinear subtasks etc.) and fewer metrics may increase statistical power. This is consistent with the quantitative comparison directions between VF and V that this study revealed, despite a relatively simplistic task and multiple performance metrics.