Solving visual object ambiguities when pointing: an unsupervised learning approach

The development of natural interaction systems is important to foster communicative strategies between humans and robots and to increase their social acceptance [23]. Deictic gestures allow the establishment of joint attention in a shared human–robot workspace for cooperative tasks, and they complement or even substitute verbal descriptions in spatial guidance or when referring to specific objects. The following studies were conducted on distinct robot platforms and employ different vision capture systems.

In [21], the authors evaluated an HRI scenario between the humanoid robot ‘Robovie’ and a human subject was presented . A participant was asked to guide the robot through a package stacking task. The authors put emphasis on attention synchronization when initializing the interaction, the context and the ‘believability’ of the whole scenario, subsumed under the term facilitation process. The underlying hypothesis was that the implementation of this process supports the naturalness of the interaction between the human and the robot. Hence, the whole experiment was driven by two conditions, namely the instruction method using deictics or symbolic expressions (i.e., asking for a box identification number) and whether or not there would be some facilitation factor during the guidance process, as the robot was able to ask for some error correction. Subjects were equipped with a motion capturing system to record the pointing gestures, and an additional speech interface was provided for verbal instruction. In total, 30 persons participated in the study and their responses to the overall scenario regarding six options like ‘quickness’ and ‘naturalness’ were recorded. Overall, the ANOVA test statistics revealed positive enhancement of the interaction when the HRI scenario included the facilitation process, which might be explained by the fact that gestures support verbal communication. Also, the additional feedback made by the robot may strengthen the cooperative nature of the task.

Although deictic gestures are primarily used to represent pointing gestures, Sauppé and Mutlu [24] defined a set of six deictic gestures types, each being used to refer either to a single object or object clusters (specific area of multiple objects). Specifically, the authors distinguished between ‘pointing,’ ‘presenting,’ ‘touching,’ ‘exhibiting,’ ‘grouping,’ and ‘sweeping’ gestures. The whole experiment setup comprised speech with two conditions (full and minimal description), gaze and the named gestures. The humanoid NAO robot was selected as the interaction partner, trained to produce each gesture by a human teacher beforehand (i.e., the motor behavior was executed by a human on the robot). The on-board RGB camera was used as the vision sensor capturing the scene. The objects were differently colored wooden toys with rectangle, square and triangular shape. Experiments with 24 subjects were conducted employing different conditions like noise and ambiguity as well as the neutral condition providing the simplest and clearest setup. An ANOVA evaluation from questionnaires was driven by the question of communicative expressiveness when including gestures and their effectiveness. In essence, the study revealed significant effects of gestures, where ‘exhibiting’ and ‘touching’ stood out in both expressiveness and effectiveness. Both gesture types involve acting on the object, which directly solves ambiguities and visibility issues, especially when lifting a desired object. This result is conclusive when considering a scenario, e.g., in a shop, when both the customer and the vendor do not speak the same language and hence any verbal descriptions become obsolete. However, in a concrete robotic implementation this extra step can be critical in real-time applications or when the computational capacity is low, as also pointed out by the authors of the study. In addition, larger distances between a robot arm and the requested object hinder a meaningful execution of deictic gestures. The ‘pointing’ gestures performed worse than the other gestures except when the distance between the subject and the robot was large and humans presumably use pointing in those cases to refer to a specific object area.

To investigate object ambiguities resulting from different pointing directions, Cosgun et al. [7] implemented both a simulation and a real experiment using an industrial robot arm. The Kinect camera was employed as the necessary vision modality, delivering object point clouds which were used to determine the degree of intersection whenever a pointing gesture with one arm was performed. The latter was represented as a set of joint angles between head–hand and head–elbow computed from the OpenNI skeleton tracker. Six subjects performed pointing in seven target directions, which yielded the horizontal and vertical angles. In a subsequent error analysis between the intended and the ground truth pointing target, the Mahalanobis distance to the objects was computed. For the real experiments, two cans were positioned central in the sensor plane and a subject new to the experiment performed pointing accordingly. The distance between the two objects was varied where 2 cm was the minimum. The authors reported a good separation score for object distances larger than 3 cm, while for lower values the system detected ambiguity. Since the approach uses simple methods to represent pointing gestures, the approach is limited in the flexibility of pointing positions as the targets on the table were defined beforehand. Also, the concrete evaluation falls short because only two objects were arranged in one row without any ambiguities.

A similar scenario using also a robot arm and the Kinect but utilizing multiple objects was presented in [26]. First, synthetic images of hand poses from varying viewpoints were used to extract important features used to train a probabilistic model. This model was used to obtain training and test distributions, where only for the training set the viewpoint is known. Second, a similarity score based on cross-correlation between the two distributions was calculated between train and test images after correcting the latter for any transformations due to the distinct viewpoints. All poses were represented in 6D including the direction from which a distance to objects was estimated. All possible distances were calculated for all objects and evaluated using a confidence measure. The object associated with the highest value was then identified as the pointing target. In total, ten objects from the YCB dataset were used, known to be appropriate objects for robot manipulation tasks. The authors [26] created a dataset comprising 39 training images from object configurations with distinct horizontal and vertical angles. Totally, 180 images showing different gesture scenarios were used for evaluation, 12 for tuning the incorporated kernel in the model and the remaining 168 for testing with nine subjects. The pointing was further split into hand poses from the subject and using an additional tool to enhance the pointing direction. Both the mean and the standard deviation for the best pointing direction estimate were computed as well as the best nearest estimate. The approach showed good qualitative performance for both pointing conditions, which means that there might be no differences between using only the hand or performing the pointing with a tool. Also, the model demonstrated robustness for the differently shaped hands by the participants. However, when the objects were less than \(10^{\circ }\) apart (the proximity values used for training) the analysis revealed object ambiguities, leading eventually to incorrect associations between pointing direction and desired object. In a quantitative evaluation, the model achieved best performance of 99.40% of accuracy when the acceptable error range was \([-15^{\circ }; 15^{\circ }]\) and an additionally introduced factor \(t=1.0\), \(t \in [0;1]\), which weights the hypothesis scores returned by the estimation procedure. For lower error ranges, the accuracy dropped significantly, e.g., to about 55% for \(5^{\circ }\) and 20% for about \(3^{\circ }\).

A more recent study presented by Showers et al. [25] demonstrated a pointing scenario with three objects. The object recognition was facilitated by using AR markers on the objects and providing bounding boxes beforehand for localization. All data were captured with a Kinect device, where the pointing direction was estimated using a finger tracking library. The experimental layouts comprised different constellations of the objects creating ambiguities when pointing. To solve this problem, additional verbal cues like ‘bottom left’ were incorporated. The different features obtained by the visual and speech information were quantified by a so-called confidence heuristic and the resultant locations called points of interest. The whole setup was trained using support vector machine (SVM) and a decision tree. The evaluation of the study revealed worse performance for the pointing information only (between 10 and 27%) for both learning models and best performance using the SVM when combining the verbal input for both naming the object and the respective location (between 68 and 89% when using the confidence heuristic). However, the effect of the introduced heuristic was only marginal.

A study related to explain developmental stages in infants understanding deictics for joint attention was presented by Nagai [19]. An HRI scenario using a humanoid robot called ‘Infanoid’ was established integrating pointing gestures with an object which was pointed to: pointing with the index finger or using the whole hand (referred to as reaching) and tapping. The implementation was based on capturing images and extracting significant edge features of the pointing hand, as well as optical flow [20]. Those features were learned with a neural network architecture, predicting the corresponding motor commands to guide the robots’ gaze direction. The evaluation in a one-person setup showed superior learning performance for reaching compared to pointing for the edge images. Using optical flow, learning the correct gaze direction was superior for tapping, followed by pointing with movement and, last, pointing without movement. These results underpin the hypothesis that deictic comprehension in infants is facilitated by moving gestures [1, 18].

Canal et al. [4] integrated pointing gestures in different settings outside the laboratory (high school, community center and elderly association) and evaluated their system including participants of broad age ranges (9–86). Again, the Kinect device was used to compute significant body joints representing the pointing and waving gesture from the delivered skeleton model. The dynamic nature of the gesture was captured by dynamic time warping (DTW), aligning two time series to determine their degree of similarity. When a pointing was detected, the PCL library was used for ground plane extraction and the corresponding direction was estimated using the hand–elbow coordinates. The authors emphasized a limitation for the concrete pointing due to sensor limitation of the Kinect itself, which made it necessary to point with the arm in front of the body. Also, only an object detection routine was implemented to check the presence of an object. Three objects were used, two milk bottles and a cookie box. Possible object ambiguities were resolved by a robots’ reply to the pointing gesture, requesting the users’ help for disambiguation, e.g., asking whether the left or right object was desired. Sixty-seven subjects unfamiliar with robots tested the HRI scenario and documented their experience in a survey. The questionnaire asked for the response time of the gestures, the accuracy of the pointing and the naturalness of the whole interaction, evaluated on a scale 1–5 (worst–best). While the participants rated the response time relatively high across the three age-groups, the concrete object detection with the pointing was more nuanced, showing a decrease in approval in the groups 35–60 and 61–86. This result is in agreement with the evaluation of the object detection alone, which yielded a recognition rate of 63.33%. This is comparatively low when taking into account a recognition rate of 96.67% for the pointing gesture. Interestingly, the naturalness of the interaction was rated the highest in the group of the elderly, which are a target audience when it comes to service robots and assisting systems in health care.

The results obtained by the studies described above are important for the development of robotic systems in domestic and health care systems, set out for human–robot interaction (HRI). However, most approaches rely on skeleton models obtained by depth sensors, which need an additional hardware setup. This, in turn, can introduce more noise and stability problems when the hardware is, for example, equipped on the robot head. Also, until today only a few studies address object ambiguities in interaction scenarios but concentrate on the different performance of pointing gesture or deictics in general.

In light of successfully benchmarking gesture and object recognition, predominantly using supervised deep learning implementations, using an unsupervised learning strategy does not seem to be intuitive in the first place. However, learning gesture–object associations includes different modeling stages where deep learning, due to its dependence on large training data, has no or only minimal benefit and, thus, does not always qualify to be the potentially best approach in robotics [12]. Hence, we propose our approach using both traditional computer vision and, complementary, introduce the growing-when-required network (GWR, [17]) as an unsupervised approach supporting pointing gesture recognition even in ambiguous object arrangements.

We recorded our own dataset as explained in the paragraph below due to the sparsity of pointing datasets or even benchmarks. Images from specific hand poses are available (the IMHG dataset [27], Marcel Sebastian hand dataset¹), but our focus is on the pointing gestures targeting specific objects, which are further separated into different levels of ambiguities, and not on the evaluation of specific hand poses. Further, the modeling and recognition of pointing gestures is a nontrivial task. In contrast to benchmarks for object recognition or gestures (e.g., ChaLearn [8]) which provide labeled data for categorization, the creation of a ground truth for pointing depends heavily on the concrete view and reference point (here: a robot) as well as on the objects in the scene. Therefore, the reproduction of pointing experiments (note: not only the hand pose) and their evaluation is difficult because of the specific experimental settings. This may explain also the lack of comparison in corresponding research paper as reviewed above. Therefore, we explain our experimental layout in detail below and also how our approach can be generalized to other settings.

The NICO robot is a humanoid robot which serves as a fully controllable robot for multimodal human–robot interaction scenarios. NICO is an abbreviation for ‘Neurally-Inspired Companion,’ signifying its integration into assisting tasks like object grasping [11] and serving as a reliable platform for HRI studies including emotion recognition [6] and human–robot dialogue understanding [28] as well as for the assessment of critical design factors for robots, e.g., ‘likeability’ or safety issues [13], which are crucially important aspects for social robot research. The robot architecture and API are configured in a modular fashion to facilitate the integration, extension and combination of task-specific algorithms. Our implementation follows this specification and is either ready for use as a stand-alone software package also for other robot platforms or complements more complex HRI scenarios. A full description of the robot hardware and details on the software are available in [13].

For our pointing scenario, we defined a controlled experimental setup to reduce noise introduced by environmental variability. In particular, we constrained the interaction to happen only between the NICO robot and one participant in the scene facing each other. We fixed both the illumination conditions in the room and the background, the latter restricted to white curtains. The pointing videos were recorded always from the same perspective using the camera available in the NICO head. As we focus on the pointing gesture, we chose three differently colored cubes on a blue desk to facilitate the object detection and recognition process in our system. We employ a maximum of three objects to keep the number of object permutations and the ambiguity classes manageable. The participant performs the pointing with the right hand, and we only recorded the upper body and arms as we did not take any facial expressions into consideration. Due to a fixed standing position, the pointing hand enters NICO’s vision center during the gesture performance. The viewpoints in the scenario are demonstrated in Fig. 1.

Having explained the general setting, we will now proceed with an outline of the specific table arrangements. The fixed positions of both, the NICO robot and the human subject, allowed us to measure some distances which become important in the subsequent computations of the hand and pointing detection as well as the object arrangements. A complete picture of all distance measurements is provided in Fig. 2.

First, we measured the distance between the NICO robot and the human participant, which we call \(d_1=100\) cm. The distance between the robot and the table is 0cm (the robot is directly sitting at the table) while the experimenter leaves a gap of \(d_2=20\), which is a trade-off between the human anatomy and the natural distance in an interaction. The table area \(A=40 \times 40\) cm is the space for possible object arrangements. The three objects used in our experiments are uniformly sized cubes with an edge length of 5.8 cm. We defined row1 and row2, from where the objects can be placed in various positions. This setup also limits the search space for pointing gestures, which is necessary to evaluate our system.

The row 1 is 28 cm away from the NICO robot, denoted as \(d_3\), and 65 cm away from the experimenter, labeled as \(d_4\). To define row 2, we measured the distance \(d_5=21\) cm from the robot and \(d_6=72\) cm from the human subject. From the robot perspective, row1 is located above row2.

We collected image data from the right NICO RGB camera (initial resolution: \(4096\times 2160\) pixels). To erase effects introduced by the fish-eye lens, we scaled the recorded images down to \(2048\times 1080\) and cropped them to \(700\times 900\) to exclude unnecessary background information. Figure 3 demonstrates the effect of the procedure.

To evaluate the correctness of pointing gestures with an associated object for different levels of ambiguity, we introduce four ambiguity classes \(a_1{-}a_4\). Specifically, we consider the row definitions shown in Fig. 2 and group the objects either into one row (row1 or row2) or arrange them into a V-shape, as demonstrated in Fig. 4. For increasing class index \(a_i\) we reduced the distances between the objects:In total, we recorded 95 object configurations which are summarized in Table 1. Note that one object in a scene is unambiguous and thus not taken into consideration. The different numbers per ambiguity class \(a_i\) result from the constraints of the distances. For instance, for ambiguity class \(a_1\) the distance between two objects is 5 cm, which results in six possible object arrangements on the two rows (shown in Fig. 5). Correspondingly, three objects grouped on the table yield four possible configurations.

For the ambiguity class \(a_4\), the objects are always distributed between row1 and row2. From the robot’s perspective, this arrangement results in an object overlap of 2cm; thus, \(a_4\) can be considered the class with the highest ambiguity.

Additional 23 object combinations were recorded to include the so-called edge cases for a fair evaluation of the overall quality of our interface. The deictic gestures were recorded correspondingly.

We designed our system for HRI scenarios employing deictic gestures; thus, the most important requirement for the interface implementation is to provide real-time processing of the hand detection and, subsequently, the hand segmentation. To achieve this, we follow some commonly known approaches in computer vision: First, we consider skin-color-based detection; hence, we convert images from the RGB to YCbCr color space, keeping only the color distribution in the Cb and Cr components. Second, we choose 15 sample images from our recordings reflecting different hand shapes during pointing to objects, which provide different skin color distributions. To allow more flexibility in the hand detection, we decided to set up a skin color classifier which outputs either skin or no-skin color regions.

For this purpose, we calculated the maximum likelihood estimates for a Cb, Cr pair to belong to skin color (P(CbCr|skin)) and not to belong to skin color (\(P(CbCr|\sim skin)\)). We created 2D histograms for both classes using the Cb and Cr components using pixel bins of ranges [0; 255]. As a result, each histogram cell represents the pixel count of both components. We then convert each histogram into the respective conditional probability:where s[CbCr] denotes the count of (CbCr) pairs in the skin histogram divided by the total count of histograms entries \(T_s\), analogous to the non-skin part.

We can then label a pixel as skin when:where \(\Theta\) is a threshold determining the ratio of true and false positives. We empirically determined \(\Theta =5\) and set all histogram cells values above this threshold to 255 (white) and values below \(\Theta\) to 0 (black), resulting in a binary image with segmented hands. Figure 6 demonstrates the result of the segmentation.

Our work focuses on the detection and correct recognition of pointing gestures even in cases of object ambiguities; therefore, we simplified the objects in the scene to be easily distinguishable by their color. To exploit this object feature, we mapped the RGB images onto the HSV color space (hue, saturation, value), which allows a clear color separation using the hue channel. Again, we need to obtain binary masks segmenting the objects from the rest of the image scene; hence, after determining the color intervals in the HSV color space, pixels belonging to the color interval are assigned the value 255, else 0. The result of the procedure is shown in Fig. 7.

A major challenge in every gesture scenario is to distinguish a meaningful gesture from arbitrary hand movements. In our scenario, we defined the pointing gesture to be performed with the index finger pointing to an object, which we identified as the most natural behavior. We use this constraint on the hand pose to model a pointing intention of the human subject using a convex hull approach. In brief, a hand can be represented by a finite set of points characterizing the respective hand contour. Computing the convex hull of this contour allows a compact hand representation, where further significant hand features can be obtained. In this work, we used the convex hull algorithm introduced by Jack Sklansky [29], also available in the OpenCV library.

In addition to the computation of the convex hull, we also consider the so-called convexity defects helping us to determine whether or not a pointing is intended. Based on our pointing gesture definition, we are specifically looking for the fingertips of extended fingers. Thus, the following process is divided into fingertip detection and the verification that exactly one finger is stretched out. Obviously, the fingertip is always part of the convex hull since the robot and the human subject are facing each other, and the camera angle is in a fixed position.

After the calculation of the convex hull, respectively, the convexity defects, we filtered the latter to avoid disguised local small contour segments. To compute the correct finger points, we have to take into consideration that the convex hull is implemented as a set of indices mapped to their corresponding points in a contour array. Therefore, for each point \(P_0\), two neighboring points \(P_1\) and \(P_2\) can be computed, resulting in a triangle as depicted in Fig. 8. From the triangle, an angle \(\delta\) can be easily obtained. A fingertip is detected, whenever \(\delta\) is an acute angle.

However, the method produces multiple hull points and thus we need to find a mechanism to cluster those points for the final fingertip model. We decided to use hierarchical clustering with single linkage as it creates clusters in a bottom-up fashion without requiring the exact number of clusters. After applying the procedure, we calculate the cluster centroids denoted by \(\Psi _1\) and \(\Psi _2\) for the corresponding points \(P_1\) and \(P_2\) and the final fingertip \(\rho\), all shown in Fig. 9.

After the hand detection, segmentation and the recognition of a pointing intention, we now have to couple the computational steps with the according objects in the scene. The task is whether or not a human subject points to an object, whereby the level of the correct recognition is made more difficult by increasing levels of object ambiguities. We first explain the computation of a pointing array for a simple pointing-object association, then we motivate and introduce the usage of the growing-when-required network (GWR).

In the previous section, we described how to detect the pointing direction with the index finger stretched out. A simple, yet intuitive approach built on top of the previously described method is to extrapolate the direction of the fingertip and calculate whether this pointing array hits an object. Based on the final points obtained from the pointing intention scheme, we calculate another point \(\Psi _{13}\) which lies between the two outer finger points \(\Psi _{11}\) and \(\Psi _{12}\). We connect \(\Psi _{13}\) with the fingertip \(\rho _1\), as shown in Fig. 10a, and extrapolate this line segment called \(\lambda\) as demonstrated in Fig. 10b. The resultant pointing array hits an object as demonstrated in Fig. 10c.

We use this concept to finally assess the pointing-object association in our scenario. We start by first detecting all objects present in the scene. Then, we determine which of the objects are hit by the pointing array, i.e., if the array intersects with the bounding box of an object. If an intersection occurred, we measure how accurately the pointing to an object was. For this purpose, we define a quality measure \(\phi \in [0;1]\) illustrated in Fig. 11. The object bounding box is divided into \(B_1\) and \(B_2\), and both areas are calculated. Clearly, if the pointing array hits the bounding box almost centrally, the ratio of both areas is almost identical and thus close to 1. In contrast, if the pointing is rather lateral, the ratio between the smaller and the bigger areas, as shown in the lower picture of Fig. 11, yields a quality value close to 0. In this case, the pointing can be assumed to be arbitrary or incorrect. Therefore, we cut off a hit at \(\phi <0.2\). In case multiple objects are present, the object yielding the highest value for \(\phi\) is considered as an intended hit.

Although the approach is easily implemented, it also has some serious drawbacks negatively impacting the gesture scenario. One issue relates to the extrapolation itself, where confusion of the correct hit is high when more objects are put on the table, e.g., when introducing, from the robot’s perspective, an additional row above the existing ones. In line with this, increasing the number of objects also means that more objects have to be detected and tracked, which increases the computation time in cluttered or uncontrolled environments. Finally, our system has so far always assumed an actual pointing by computing an array, even in cases of unintentional gestures.

The input to the GWR should contain unique and meaningful hand properties; thus, we decided to use the already identified hand centroid and fingertip positions from the preprocessing procedures explained above. In addition, we integrate the angle of the hand pointing direction. During the performance of different pointing gestures to an object, this angle continuously changes and is, therefore, a complementing feature which might help to increase support for a correct correspondence between the gesture and the object.

To obtain the angle, we considered the whole contour as the three points extracted for the pointing array are error prone to noise and, consequently, may introduce prediction errors. To approximate the pointing direction given the entire contour, we employed the weighted-least-squared error between the pointing line and the contour points. We selected the line with the lowest error and refer to it as pd, i.e., pointing direction (Fig. 12a). We drew a horizontal line h at the level of the fingertip along the whole image width and calculate the intersection with pd, which results in two line segments \(h_1\) and \(pd_1\) (Fig. 12). The specific point is simply called i, as shown in Fig. 12c. It is then easy to obtain the angle \(\alpha\).

For each frame, the following vector is then established which characterizes the hand shape:where \(\alpha _i\) is the described pointing direction, \(c_{*}\) the centroids in the x- and y-plane, and \(\rho _*\) the respective fingertip coordinates. Also, a bounding box representing the target of each individual gesture is computed as:where \(x_{1i}, y_{1i}\) is the top-left coordinate and \(x_{2i}, y_{2i}\) the bottom right corner.

The GWR training is performed in an unsupervised manner, and to evaluate the correspondence between the original data and the learnt representation is, in essence, a comparison between vector distances. However, for realistic implementations, Parisi et al. [22] showed that GWRs equipped with a label function allow the use of unsupervised learning also for typical classification tasks. In summary, the authors [22] introduced a hierarchical processing pipeline of feature vectors for action recognition, where one labeling function was used in the training and a second for the testing phase, i.e., labeling an unseen action. The fundamental difference between this work and our approach is that we operate in a continuous space (i.e., the table locations) in contrast to discrete action classes. Therefore, we will explain how we defined our label function.

For the concrete implementation of this procedure, we distinguish between the node creation and the node training. Whenever a node is created, we calculate both the mean average between an observation and the BMU weight and, additionally, the mean average of both labels. The resulting value is then assigned to the node. Following the representation of the label, i.e., the position of a bounding box, we define the mean of two of those bounding boxes:where \(lc_r\) is the centroid of the new node label, \(lc_{b}\) denotes the centroid of the BMU label and \(lc_{o_i}\) is the centroid obtained from the current observation. Given this measure, a new bounding box is created with the corresponding width and height.

We also update the labels according to:where \(lc_{b}\) is the BMU centroid label and \(lc_n\) the labels of the neighbors as well as the label of the current observation \(lc_{o_i}\). If a new node is added, the mean height and width of a bounding box are determined. The update strategy follows the same intuition as the ones for the weights, as sufficiently trained nodes will not significantly change their associated labels.

For easy prediction tasks with no or only a low ambiguity level as implemented for classes \(a_1\) and \(a_2\), the GWR trained on our recordings returns the label of the BMU. Along with the result, we compare the activation level with a noise threshold to exclude faulty pointings, i.e., unintentional gestures and those diverging significantly from the scenario. We configured this threshold to 0.5, because testing incorrect gestures yielded activations around 0.1–0.35, and we want to provide a certain buffer to remain flexible capturing other improper gestures.

In case ambiguity is present, also more than one label is returned. From the set of returned BMU labels, we compute the union which we call A representing the positions seen to be most similar to the target direction. From the positions united in A, we calculate the corner points as demonstrated in Fig. 13. We set up a function \(F_A\) to detect the correct number of labels:

where \(\#nodes\) are the number of nodes in the GWR and the values empirically determined constants. The reason behind is to keep the size of A relatively constant, as the network growth corresponds to node generation; thus, the network becomes dense and positions tend to overlap or, at least, show less variance capturing pointing positions in the scene. Some results of pointing in ambiguity cases are shown in Fig. 14, where the numeric values denote the IoU.

Table 2 lists the metrics used to evaluate the success of our system in terms of true positives (TP), false positives (FP) and false negatives (FN). A TP translates to a correctly returned object the subject was targeting at, an FP denotes that an object was returned which was not pointed to, and an FN means that no object was returned. From those measures, we also compute precision, recall and the F1 score, which are shown in Table 3. We count a pointing gesture to an object as correct or a hit if the actual and predicted object positions share an \(IoU\ge 0.5\). Otherwise, we declare it as a miss. Our results demonstrate that our system achieved high precision with an overall value of 96.8%. This means that it is likely to get a correct target object, especially underpinned by the 100% precision value for ambiguity classes \(a_1\) to \(a_3\). Similarly, both the recall and the F1 score show high values, ranging from 95.87 to 99.21% for these classes.

A different picture becomes evident for class \(a_4\): The number of misses increases from 7.43 to 22.96%, which consequently affects the recall expressed in a decrease to 71.04%. Also, the number of FPs drastically increased from 6 FPs in class \(a_3\) to 213, with a direct negative effect on the precision, which is with 92.24% the lowest value among the classes.

We identified two factors to explain our obtained results. First, the proximity of objects is very high, causing variations in the pointing array to be classified as a hit and thus increasing the number of FPs accordingly. The second factor can be explained by a slight yellow cast in the data recordings we detected in the postprocessing of the data for class \(a_4\), which negatively affects the hand detection as it relies on a learnt color distribution. Thus, the segmentation is more likely to fail for those frames and, additionally, influence the pointing array to deviate strongly from the intended pointing direction. Therefore, the intersection of the pointing array and the target object is less likely to occur.

To summarize, our baseline approach based on computer vision techniques shows astonishingly good performance for the first three ambiguity classes \(a_1\) to \(a_3\). We conclude that for these classes, it is worth to look into procedures which are easily implemented and, as we demonstrate, robust in terms of the mapping between the pointing gesture and the targeted object. However, increasing the object ambiguity as shown for class \(a_4\) rapidly deteriorates the performance of our interface. Therefore, we propose a flexible modeling approach using unsupervised learning, specifically, involving a specific implementation of a growing-when-required network (GWR), which we explain in the next section.

For training the GWR, we used threefold cross-validation varying the parameters \(a_T\in \{0.85, 0.90, 0.95\}\) and the number of epochs: 30, 50, 100. Table 4 depicts the GWR network parameters which are fixed throughout the training.

Here, \(e_b\) is the learning rate for the best matching unit (BMU), \(e_n\) is the learning rate for the BMU neighbors, \(h_T\) is the firing threshold, and \(\tau _b\), \(\tau _n\), \(a_b\), \(a_n\), \(h_0\) are the habituation parameter for the BMU and its neighbors. The parameters \(age_{\max}\) and \(nb_{\max}\) specify the maximum age of an edge before deletion and the maximum number of neighbors a node can have.

Table 5 shows the averaged result from the cross-validation in terms of the network size and the error, the latter denoting the overall distances between the BMU returned by the network and the actual target position. As expected, the higher the value of \(a_T\) is, the more the nodes and respective edges are created. Consequently, we obtained the lowest error for \(a_T=0.95\), but nevertheless we cannot be sure that nodes are redundant and, more importantly, we may trap into overfitting as the GWR covers the learnt input space but may fail to generalize to unseen positions. Therefore, we conducted a detailed evaluation on (1) the performance of a GWR w.r.t. pointing positions, unveiling the potentially best candidate network balancing network growth and performance, denoted as ‘Individual GWR’ and (2) the integration of the object(s) and their respective ambiguity, denoted as ‘GWR Performance with Objects.’

The results reported in Tables 6, 7 and 8 are the mean and standard deviation averaged over the threefold using the test set. We report precision, recall, the F1 score, as well as the number of misses in %.

A first result to be highlighted for the prediction performance is the high precision over all numbers of epochs and different values for the parameter \(a_T\), accompanied by a standard deviation of nearly 0. This means that, overall, the GWR is almost perfectly able to represent a desired pointing gesture. A more nuanced picture becomes evident for the recall, which differs greatly over the numbers of epochs, best demonstrated for \(a_T=0.85\). Also, the standard deviations show many variations, regulated only by higher numbers of epochs and for \(a_T=0.95\). As a trade-off between both measures, the F1 score depicts reasonable performance for 30 epochs and superior performance for 50 and 100 epochs over all ambiguity classes for \(a_T=0.85\). As an example, the F1 score for class \(a_1\) is 85.32% for 30 epochs and increases to 91.88% for 100 epochs. Similarly, the recall improves from 74.43 to 85.12%, which can be explained when considering also the relatively high number of misses, affecting the recall. Although the training epochs improve on the number, presumably by the higher amount of nodes produced by the GWR, increasing the value for \(a_T\) seems to better capture the negative cases. Finally, for the remaining classes and considering also the mean values among epochs when \(a_T=0.85\), the differences are less pronounced. Interestingly, also the IoU does not significantly change. Thus, the number of training epochs may be lowered when computational resources are limited.

When \(a_T=0.9\), improvements are clearly visible for the recall and the number of misses, yielding superior performance expressed in the F1 score when compared to \(a_T=0.85\). Here, the difference between epochs is negligible both for the individual evaluation of the ambiguity classes and for the mean values. Similar results are obtained for \(a_T=0.95\), which, finally, demonstrates the lowest number of misses and highest IoU with small standard deviation. As shown in Table 5, increasing \(a_T\) leads to a stronger growth of the GWR, but the relationship between network size and performance is not linear. Thirty epochs and \(a_T=0.85\), i.e., 263 nodes have more impact on the performance compared to \(a_T=0.9\) with 541 nodes. However, the influence of the network size w.r.t. the performance vanishes for parameter \(a_T=0.9\) and \(a_T=0.95\), and there might be a saturation point where only redundant nodes are created. We, therefore, conclude that reasonable performance can already be achieved with a lower number of epochs and smaller network size.

The GWR networks have so far shown good performance and robust classification between the actual and predicted pointing direction to an object. However, we cannot make any statements about correctly returned objects, especially when applying the networks to ambiguous object arrangements. To conclude on the final capabilities of GWR networks in our pointing gesture scenario, we additionally performed experiments integrating the object(s). We used the 40th frame of each individual pointing video, because this frame delivered the information we needed. We extracted the color information from the test frame and compared it with the actual color. In case the colors matched, we counted this result as a TP, otherwise we declared it both a FN and FP. In case no object was detected at all, we counted this as a FN and a miss. This procedure can be applied to the ambiguity class \(a_1\), but it did not cover any ambiguous object configurations. Therefore, we also implemented an ambiguity detection by checking the activation level of the BMU returned by the network: If it was below a predetermined noise threshold \(n_T\), we considered the observation as noise, otherwise we approved an existing ambiguity in the current scenario. We then checked the IoU between the predicted and all object positions, followed by the classification into the metrics as just described. Again, we calculated precision, recall, the F1 score and the number of misses, averaged over the threefold cross-validation including the standard deviation. Figure 15 shows exemplary a pointing scenario, demonstrating the activation level for different pointing gestures. The concrete results are listed in Tables 9, 10, and 11.

The evaluation of the prediction with GWRs coupled with ambiguity and noise detection shows considerable improvements for class \(a_1\), where both precision and recall achieve 100% across all networks and parameter values \(a_T\). Due to the ambiguity detection, expressed as correctly detected ambiguity (CDA) in Tables 12, 13 and 14, we can show that this extra step caused a decrease in misses, a crucially negative factor limiting the recall abilities as demonstrated above. Also, the number of misses could be improved from 12.77% to 5.4% for 30 epochs and \(a_T=0.85\) and even to 1.67% for \(a_T=0.90\), positively influencing the recall and F1 score.

Regarding the CDA, it becomes evident, overall, that the performance is reasonably good except for class \(a_2\). We explain this by the fact that objects are closer together and thus can be captured more easily. Noticeably, no noise (FDN) is detected across all networks.

However, our results also reveal a decrease in performance for the ambiguity classes \(a_3\) and \(a_4\). While the differences of the IoU values are rather small (e.g., a drop from 0.725 to 0.713 for 30 epochs and \(a_T=0.90\)), the misses increased considerably. For instance, for 50 epochs and \(a_T=0.90\) in \(a_3\) this number more than doubled from 4.08 to 10.37%, and similarly for \(a_4\) from 2.47 to 6.81%. This effect is reduced when considering \(a_t=0.95\). Our findings show the sensitivity of an integrated object detection for pointing gestures, rendering their recognition non-trivial. When testing our system to new object arrangements, the divergence between the manually annotated bounding boxes and the predictions returned by the GWR is even more noticeable with increasing levels of ambiguities. However, we want to emphasize that our object detection implementation is very simple, yet the GWR predictions yield good results for our gesture scenario.

Our evaluation results for both the computer vision approach and our novel model proposal using GWRs show good performance for ambiguity classes \(a_1\) and \(a_2\), achieving even 100% precision. With an increasing level of ambiguity, the computer vision approach fails to capture the pointing-object associations, expressed in a high value of misses of 22.96% compared to 0.62–9.31% when evaluated individually, and 1.84–14.56% for the GWR prediction. As a result, the GWRs also show better performance in terms of the recall. From our experiments, especially regarding ambiguity class \(a_4\), we can conclude that the computer vision approach is highly sensitive to the environmental laboratory conditions, suffering from image artifacts due to lighting conditions or other confounding factors. In addition, our implemented ambiguity detection shows support for the prediction process, yielding 100% precision among all networks. Finally, our proposed approach modeling pointing gestures with GWR provides a more controlled way for recognizing a pointing gesture: The computer vision approach always extrapolates a certain pointing position and would always target to match a corresponding object in the scene. Thus, in an extreme case, pointing to the ceiling or to another wrong direction would always result in a pointing intention and, therefore, the next object would be aimed for to establish the pointing-object association. In contrast, the GWR allows us to model a set of pointing directions beforehand, independent of any markers on the table. The flexibility of designing a certain topology or range for target directions yields a more accurate prediction of the pointing and excludes any hand movements, may they be random or simply not in the required scenario range.