A probabilistic evaluation of human activity space for proactive approach behavior of a social robot

3.1 Task vs space

Activities involve various movements in body parts. Of them, hands are the most frequently used in most of the tasks. Hand movements will also vary depending on the handedness of a person who performs the task. Figure 1 illustrates four occasions encountered while making a call. Positions of the two wrist joints in space were different in each occasion. Similar to this example, positioning of major body parts during various scenarios will be different depending on the activity the user engages in. Therefore, spatial behavior of certain body joints can be used as a mediator to monitor human behavior. Among these, wrist plays a dynamic role in most of the activities of humans and is a great cue for understanding nonverbal communication [30,31].

Figure 1

An example situation in which an individual is making a call. The points in space to which the right and left wrists move within a period are marked in (a) to (d). Variation in the positions of two joints; right wrist ( H R ) and left wrist ( H L ) are joined for the ease of comparison of variation. Shown in yellow and blue are the positions of right and left wrist, respectively.

In our approach, wrist movements are observed for an adequate period of time, and the behavior is evaluated as a whole. To include a set of points into a single space for the ease of analysis, the space around the user is divided into smaller regions and the observed points are analyzed in clusters. The concept of clustered points within the activity space was developed in a previous stage of this research [32].

During this approach, front and side personal spaces of the human are divided into subspaces called “zones,” which is shown in Figure 2. To increase the accuracy of evaluation, 50 zones were identified. On the one hand, when the number of zones is high, the capability of tracking even smaller movements increases. On the other hand, handling a large number of variables (zones) for decision-making is cumbersome. Therefore, an average of 50 zones was selected for the study. The length of the personal space is divided into six regions, including “far right,” “right,” “mid,” “left,” and “far left.” Areas that do not fall under these five labels belong to zone “51,” which will be defined later in this paragraph. Similarly, height is divided into levels from “level 1” to “level 5” and the width is divided into “front” and “far front.” These are shown in Figure 2(a) and (b). These divisions make 50 zones around the activity space and the space excluding all the 50 zones belongs to zone 51. For example, the label (level 1, far right, and front) describes zone 1 and (level 5, far left, and far front ) gives zone 50.

Figure 2

Division of activity space into regions and levels. (a) Front view of the activity space and (b) side view of the front space.

Hands are the most utilized body parts during many day-to-day activities. Handedness of a particular individual determines the extent to which he/she uses each zone during a certain task. Therefore, activity zones occupied by right and left hands during a certain period were monitored through this study. In the original research, this tracking process of zones occupied by both hands during the period of observation, t, was called “Activity Space Analyzer (ASA).” In ASA, all the regions occupied by a specific body joint within a period of observation are recorded. For instance, consider a walking person. Zones recorded during the walk according to the ASA were as follows.

• Right wrist = {11, 51}

• Left wrist = {14, 7}

This idea is illustrated in Figure 3(a). In this example, the zones recorded for a single hand were 20 and 51. These zones were visited at least once during the period of observation. Rest of the zones was not occupied during the period of observation.

Figure 3

How (a) ASA and (b) QASA records data for analysis is shown. Here in (a), 1 and 0 represent whether the corresponding zone is visited or not visited, respectively.

3.2 Quantitative ASA (QASA)

Sometimes, the same zone will be occupied for a number of times depending on the type of activity. This is common in repetitive tasks. In QASA, the frequency of visits recorded for each zone is considered in contrast to ASA, where only the visited zones are considered irrespective of how many times a particular zone is occupied. This is illustrated in Figure 3(b). The frequency of visits to each zone was counted at the end of the period and the percentage frequency is calculated accordingly. Of all the visits, zone 20 was visited 48% of the time and 51 was occupied 52% of the time. The frequency of obtaining data from a particular user is given under Section 5. The frequency of visits could be used to differentiate the behaviors in tasks that occupy the same set of zones but at different frequencies. This approach further allows the robot to give less priority to random movements, which may record farther zones for fewer times.

During the period of observation, t, a person might move both hands into the similar zones repeatedly. Sometimes, he/she might not move the hands for the entire time. The zone that included the subject’s wrist was recorded in each time unit. Likewise a record of the wrist positions by means of zones was kept throughout t. In both the scenarios, a single zone is considered to be visited more than once. All the zones visited throughout the period of observation were considered in ASA. At the end of t seconds, the number of visits to each zone by the two wrist joints (frequencies) is calculated separately in QASA as in Figure 3(b). Dimensions of zones were chosen by trial and error to increase the accuracy of identifying distinct movements. These dimensions do not depend on the whole-body movement of the person, as zones are positioned relative to the lengths of specific body joints such as the right and left shoulder joints. The considered body joints and dimensions used are illustrated in Figure 4. Front and f a r f r o n t regions are each 0.2 m wide and are measured toward the front of the individual. Other fixed lengths were chosen, so that even movements made over a small range could be tracked.

Figure 4

Dimensions and joints used to separate zones are shown in the skeletal diagram. The distance between right and left shoulder joints is denoted as l and the height from the floor to face joint is denoted as h. Values used for length and width are marked.

3.3 Probabilistic analysis

After the frequency of visits to each zone is calculated at the end of t s, the probability of each zone to be visited during the same activity under same conditions – probability of occupancy – is calculated. Equation (1) is used for this evaluation.

where i = 1, 2,…,51. Here P zone _ i is the probability of occupancy of zone i at the end of t s. f zone _ i and f zone _ j denote the frequency of visits to zone i and j, respectively, during t. The values obtained for P 1 to P 51 are compared before making interaction decisions. Often, only a few zones are visited during a certain task. Therefore, the probability of the rest of the zones will be zero. Figure 5 shows the probabilities of occupancy for each zone visited while the user was “standing.” In this scenario, zones 11, 36, and 51 have been occupied for 56, 23, and 21% of the time, respectively. Hence, P zone _11 = 0.52, P zone _31 = 0.23, and P zone _51 = 0.21 according to (1). This probability provides the tendency of the user to initialize the same zone again during the task.

Figure 5

Approach of QASA. It records the zones each joint occupies and finally calculates the percentage visit to each zone. The observation was made upon an individual who was standing, relaxed.

5.1 Research platform

Experiments were conducted with the participation of 21 individuals of 25–50 years (mean of 26.3 and standard deviation of 8.96). A service robot platform called MIRob was used for the implementation of a proposed system. MIRob includes a Pioneer 3DX Mobile Robot platform attached with a Cyton Gamma 300 Manipulator for assistive tasks and a kinect sensor for vision. Skeletal representation of the tracked body was extracted as 3D coordinates of feature points in Kinect. The experiment was carried out in an artificially created domestic environment.

5.2 Previous experiments

We conducted a human study on the nature of robot-initiated HRI in ref. [34]. We considered a robot could comprehend observable, nonverbal human cues prior to a virtual social behavior of a robot in a human–robot encounter. During the study, we found how important is the robot’s perception of spatial constraints of an encounter, for an interaction to be less or not disturbing to users. During this study, three human cues, namely, speed and positioning of selected body joints and frequently occupied areas of space by these body joints, were considered to generate appropriate responses toward a particular reason. This work considered the movement of elbow and ankle joints in addition to the wrist joint considered in our current work. For simplicity, the space around a user was divided into three parts as right, center, and left. A decision grid was created in order to define which spatial and verbal responses of robot match the observations well. The robot reaches its user from the least obstructed side (out of right, center, and left regions) keeping a calculated interpersonal distance according to (5).

The term refers to the distance from the center line of the body to the considered joint toward one coordinate system. Here it is measured along the z direction, which extends from the center line toward the front region of the user. In addition to proxemics, the robot chose a conversational preference such as no interaction at all, greeting, delivering a service, and having a small talk.

These responses were generated by means of a wizard-of-oz experiment, considering how dynamic (inclined toward a busy state) or static (inclined toward a relaxed state) a person is. The same social robot we use in this work was also used to conduct the study and the platform was visually capable and equipped with a microphone and a speaker to listen and respond to users. A set of tasks were selected for users, since their responses depend on the priority given by him/her to the task. Robot was remotely navigated toward a user during observation and interaction.

Results of the experiment confirmed that the spatial and verbal behaviors of a robot were more socially acceptable through situation awareness gained by observation of human cues. In our set of experiments, situation awareness is confined to the behavior of the user, robot, and spatial constraints. It was observed that proxemic decisions were taken, so that highly engaged areas were least obstructed, received higher feedback scores from users, upon a static approach behavior.

Furthermore, it could be seen that the decisions made by such adaptive systems substantially agree with those of the user. The experiment considered physical, social, and emotional aspects of human behavior by considering a limited number of cues. We extended our contribution to implement such a robotic system to successfully engage with users without violating their expectations.

Furthermore, we tried to embed nonverbal behavior-based situation awareness in approach behavior of a robot. Therefore, proxemic-cautious lively behavior of the robot was replicated during our current set of experiments. For that we selected only the critical joints of human body that contribute significantly to their nonverbal behavior, and the verbal responses generated by the robot were omitted as we consider only the robot’s approach behavior prior to interaction. Age and gender of users were not considered within the scope of this study.

5.3 Experiment

Tasks encountered in a typical domestic and lab environments were selected for the study. The list of selected tasks is shown below. The task names are shortened for the ease of future reference.

1. DES – Engaged in desk activity

2. CLN – Cleaning floor

3. EXR – Exercising

4. WTV – Watching television

5. SEA – Seated, relaxing

6. LAB – Engaged in lab work

7. PHN – Making a phone call

8. LAP – Working on laptop

9. CK – Cooking

10. LEA – Leaning to a wall

Each participant was asked to perform the ten tasks mentioned above. Hence, 210 different scenarios were selected during the experiment. First, MIRob was allowed to approach each user in direction A in Figure 6 where the orientation is 0 ° and d = 1.2 m. Then MIRob was allowed to observe each individual once the activity was started. After the evaluation of the user behavior according to the model, robot navigated toward the user keeping an orientation, a mutual distance, and an approach direction as determined by the proposed model. This approach behavior of the robot was rated by the participant by comparing this approach behavior with that in direction A in Figure 6. Hence, approaching the user from front as in A is used as the ground truth during this experiment. User rated the robot’s behavior based on the convenience or the discomfort he/she felt as the robot approached. Here the period of observation or t was taken as 10 s. This value was determined, so that an adequate amount of data was obtained for the analysis. Visual information was extracted at a rate of five sets of data per second. This data included positions of user’s body joints. The face orientation was obtained at the first instance only in order to avoid the algorithm becoming computer intensive. Proxemic decisions were taken, so that highly engaged areas were least obstructed. To evaluate the performance of the system, we conducted the same experiment two more times with a gap of 7 and 14 days. The feedback scores received at each stage were analyzed. Robot was allowed to wander around in a given map and it stopped and started observation once a human is tracked.

5.4 Observations and discussion

Zones recorded for a single occasion and the calculated probabilities for each zone during each task are shown in Table 1. Only the zones with a nonzero frequency of visits are given. The zones with the highest probability of occupancy were chosen and the distance from the user was calculated according to the decision-making criteria. Figure 7 shows the distances calculated for the ten tasks in Table 1, considering the highest probability of occupancy. Except for the zones given in Table 1, all the other zones received a zero occupancy probability since these zones were never occupied during the activity. After selecting P i _ max for each task, the approach direction, orientation, and the distance between user and robot were calculated. Results obtained for each occasion are given in Table 2. The mean feedback score received from the users at their first encounter with the robot is given here. The position and orientation of the robot with respect to its user after implementing the model are marked in Figure 7. Shown in dotted and dashed lines are restricted areas including the region with the maximum occupancy. Figure 8 shows the implementation of the model during PHN, as given in Table 2.

Figure 7

Distances considered maximum occupancy regions of the selected set of tasks are marked here. The red triangle denotes the position of the user and the other colored triangles denote the position and orientation of the robot during each task. The size of the triangle is not scaled with the size of the user or robot but the orientation is marked by measuring θ . Colored contours represent the restricted area for the robot to invade while approaching its user.

Figure 8

Two occasions encountered during PHN in Table 2. (a) Positions of the robot and the user initially as the robot observes the user. (b) Positions of the robot and user after approaching the user. The orientation and the interpersonal distances during these two occasions are marked.

According to Tables 1 and 2, while the user was performing a desk activity: DES, zone 15, which is on the f a r l e f t region, recorded the maximum probability of occupancy. Therefore, after implementing Algorithm 1, the approach direction was obtained as Right. According to (1), d was calculated as 1.0 + 0.76 × 1.0 m. Similarly, the orientation was 0.76 × 80 ° . Hence, the figures for d and θ were 1.76 m and 61 ° , respectively. Few occasions encountered during this task is illustrated in Figure 8. Robot behavior in this situation received an average feedback score of 8.17 of 10. The reason for the reduction of 2 marks was that the distance robot kept was too high if the user wants to ask for a service from the robot or talk to him. During all the other instances, except WTV and SEA, the behavior of the robot received feedback scores above 8. In WTV, the distance kept by the robot was too large, according to the user. While watching TV, the user preferred to speak only a few words to the robot or ask for some service. The distance was too large for both the preferences. During SEA, the user was relaxed; therefore, he liked to interact with the robot for a longer duration. However, as the robot was nearly 2 m away, the occasion reduced the user’s trend toward a conversation.

The feedback scores received when the experiment was repeated after 7 and 14 days of the first experiment were given in a box and whisker plot in Figure 9. We intended to analyze how user acceptance regarding the behavior of the robot evolve with experience. The mean values have shown an increase with time as the users get the experience of this user-aware behavior of the robot. This could clearly be seen in the tasks: DES, WTV, SEA, LAB, and LAP. CLN, EXR, PHN, and CK recorded higher feedback scores from the beginning; and therefore, only a slight difference was observed between the scores received for these tasks afterward.

Figure 9

This graph shows the mean values of feedback scores received from users for the acceptance of robot’s behavior in the three occasions: initial stage, after 7 days of initial stage, and after 14 days of initial stage. The error bars represent the minimum and maximum scores received.

In DES, the feedback scores increased from 8.17 to 8.36 and then to 8.50 with an overall gap of 0.37 points in between. The pattern continued in a similar manner in LAB and LEA with the overall increments of 0.45 and 0.38. CLN, EXR, LAP, and CK recorded slight improvements in the overall scores but still managed to be the tasks that received the highest scores for robot’s behavior. These four tasks improved the feedback score from points 0.19, 0.14, 0.21, and 0.07, respectively. Even though the PHN recorded an overall decrement in scores by 0.20 points, it still managed to obtain feedback scores above 9 in all three occasions. The scores were 9.5, 9.26, and 9.30, respectively, for three occasions. WTV and SEA recorded the highest improvement in user feedback scores which were 1.02 and 2.07 points. WTV improved the score from 5.83 to 6.90 and then lastly to 6.85. SEA improved its initial score of 6.07 to 7.31 and then to 8.17. A probable reason for this improvement is that the users first disliked being disturbed by a robot in their relaxing times. Later on, they were delighted by the fact that robots learn not to invade their personal space in such relaxing situations.

Table 3 represents the results of a t test performed to analyze the differences between the feedback scores received by the robot for its approach behavior in initial and final stages. Initially, it is assumed by the null hypothesis that no significant difference was observed between the compared groups. From these results, it can be seen that p < 0.05 in DES, WTV, SEA, LAB, LAP, and LEA. Hence, it can be deduced that the null hypothesis cannot be accepted. It shows a significant improvement in the feedback scores in these sets of tasks. In contrast, for the tasks p > 0.05 in CLN, EXR, PHN, and CK, the null hypothesis is accepted. Hence, no significant improvement in the feedback scores. But it can already be seen that the feedback scores received for CLN, EXR, PHN, and CK are much higher. In all the cases considered here, dof and t critical were 20 and 1.724, respectively.

In general, the minimum feedback scores have been increasing with time, with an exception in PHN and CK. Maximum feedback scores either stayed constant or increased with time. From these trends, it can be concluded that a significant improvement was observed in determining appropriate approach behavior of a robot, and the performance of the system could make a positive effect upon the user’s acceptance of robots as well.

6.1 Implications for theory

Findings of this study were mostly based on the fact that people prefer similar proxemic rules when interacting with robots as they do while interacting with other humans. Factors that influence personal spaces such as the nature of gaze, gender, and familiarity with the robot were not considered within this context. Therefore, this method does not replicate all parts of human-human interaction into the HRI scenario. However, we could identify several relationships between preferred proxemic behavior and user experience with a robot’s cognition that hold true for HRI as well. In this study, we assumed that an individual’s state is affected by his/her task. However, numerous other factors are not consider during the study. Social norms in addition to proxemics, cultural values, personality traits and principles, health condition are a few such social and psychophysiological conditions. In addition, no theory combines all these aspects to model a certain situation. Therefore, development of such instinct in robots lacks conceptual and technological basis.

6.2 Implications for design

Based on the findings, we can state that this model can offer better means of determining a robot’s proxemic behavior based on nonverbal human cues such as movements. As humans prefer not getting interrupted by the behavior of the robot, the first design guideline proposed by the study is engraving a sense of respecting personal space based on physical and emotional aspects. It is important to design social robots that can maintain human trust. Otherwise the designer has to stress out the user with a set of rules to follow when using the robot. The feature of “maintaining appropriate proxemic behavior” can be used as an “etiquette” for future robots used in human domains. Even though people have been accustomed to robots in their environment, and will tolerate behaviors with limited perception capability, such behavior is not acceptable in a dynamic environment such as shopping malls, hotels, relaxation environments, etc. Therefore, maintaining etiquettes in behavior can be stated as the second design guideline for robots. The third guideline is to consider human cues as much as possible. During this approach, only the usage of activity space was considered. Although activity space is a clear representation of task, often there are occasions where it is a vague representation of the setting. Therefore, considering factors in the environment can be a plus for better perception of the surrounding. An important fact observed during the experiment was that some people were restricted to a few zones while some had extended movements covering a number of zones. Therefore, a zone-based assessment is advantageous in monitoring human behavior especially to determine proxemics. This zone-based assessment is the forth design guideline that can be highlighted from the results. Body proportions differ from human to human. This is the reason for determining dimensions of zones as well as the whole activity space grid as shown in Figure 4, based on individual body parameters. This can be considered as the fifth guideline to design a proxemic-cautious robot companion.

6.3 Limitations

Involuntary responses of humans before the presence of the robot such as changes in pose and voluntary responses such as verbal cues were not evaluated during this study. However, these parameters may have an effect upon the emotional state of the situation which requires adjustments in proxemic behavior of the robot accordingly. Factors such as age, gender, and the race of the participants were not considered within the scope of this study. Therefore, in the future, it is expected to expand the system to make an assessment of a situation in various aspects except for physical behavior of a human. Other than that, cultural concerns and norms confined to certain communities can be considered in future improvements.