1 Introduction

Teaching of socially relevant behavior to children through games and robotic toys is an area of emerging interest [1–6]. Autonomous robots are of special interest in this area since they provide an opportunity to use the robot as a replacement of a caregiver, as a partner, or a mediator of play [1–9]. A number of purely autonomous robots, both with and without anthropomorphic or zoomorphic shape have been used in games with children [5–13]. Robots in these studies, however, perform simple behavior (as for instance lifting of a hand) that aims to provoke reciprocal human reaction [4, 5, 12] or predict the position or the direction of the movement which is referred to as predicting motor intentions [4, 5, 10, 44], instead of being a tool for more general social learning. Robotic studies on social learning that have the potential to be extended to human–robot interaction scenarios are reported in [10, 14–16, 44].

In this work we aim to go beyond sensory–motor interaction in robotic models of embodied cognition by also taking into account the interplay between constitutive and interactive aspects of autonomy. This implies that sensory–motor interaction has to be enriched with intentional, emotional and reward aspects of interaction. Specifically, we focus on the emotions that are conveyed by movement behavior.

Keltner and Kring [17] point out a highly dependent link between emotion and social meaning. They argue that emotions serve a set of functions that are critical for coordinating social interactions. These functions are:

  • to provide information to the conspecifics about the surrounding environment (e.g., fear may indicate the presence of a predator);

  • to elicit both complementary and similar emotions in others, depending on the context;

  • to be an incentive that promotes social relationships.

This motivates us to pursue a more general social learning framework with robots that include emotional facial expressions, bodily posture and actions of others, and triggers appropriate emotional responses.

Although it is not shown that emotional resonance (response to perceived emotion) and emotion understanding and recognition are related components of the emotional system [7], teaching to just recognize emotions will be beneficial for instance to persons (especially children) with autism which have atypical social and emotional development [18]. Benefits in using robots for behavioral training of autistic children have been investigated in [6, 8, 19, 42].

Teaching emotional and social movement patterns can take many forms. Since we target children, we choose to design game scenarios that will include recognition of emotional movements. Most obvious value of games is enjoyment and sharing of social experiences with others [20]. In addition, play, is widely used as a preferred educational activity for young children [21, 22]. Salen and Zimmerman [23] point out the difference between games and play. They pose that a game is a subset of play, a formalized and focused interaction that occurs when players follow the rules of a game in order to play it. Since the ultimate goal of this research is enhancing the social skills of children with autism who feel comfortable with structure and clearly defined rules, using games to stimulate social interaction between them is a good way to approach this user group.

Physical play [11] is based on motor activities of the players. Using robots for social training raises the interest for nonverbal social communication, expressed via movement and postures. Present day robots express movement as a response to their sensing and decision making, while natural language understanding and synthesis is far beyond the reach of the contemporary science. That is the reason why speaking robots can be used for social tasks only in a tele-operated mode, see for instance [24].

The nonverbal ways for conveying socially relevant information include facial expressions and body movements [25, 26]. Neurological studies suggest that understanding nonverbal communication of facial expressions and body movements involves the mirror neuron system [26, 27]. However, Montgomery and Haxby [27] showed that these two forms of nonverbal social communication have distinct representations within that system, so it is plausible to study them separately. We investigate a specific aspect of social interaction behaviors, namely the perception and expression of emotions and emotional body language, as conveyed by body movements, which is based on our understanding of the common coding principle in the mirror neuron system [14]. However, the construction of a bottom up model of the mirror neuron system on neuronal or functional level is not plausible for this study. Instead, we use Laban movement analysis [28] to embody the common coding principle on behavioral level. Laban movement analysis (LMA) it is a systematic framework for describing all forms of human movement and emphasizes how internal feelings and intentions govern the patterning of movement throughout the whole body. It provides a complex understanding of intention. Similar to common coding/mirroring paradigm LMA is useful to describe the interaction in the physical world, which is caused by physical robots that move or perceive movements of humans and other agents (robots). Therefore using Laban movement analysis for social interaction through physical objects gives many possibilities : to design a robot (intelligent object) that understands the emotional state of a human player and responds in an adequate manner; to design robot behaviors that imitate, enhance or counteract an emotional state of a person; to design socially believable robotic (embodied) characters that provoke social interaction; to create constantly adapting interaction based on movement/emotional understanding of a robot.

In this particular study we develop a framework for expressing and interpreting emotional movements. Based on this framework we recorded and analyzed human movement patterns and designed and implemented emotional movements on robots. User testing of these movements was performed with typical children to validate the plausibility of this design. Also we developed games that include the emotional robot behaviors for training autistic children.

This paper is organized as follows. In Sect. 2 will be discussed how the stages of social development can be advanced by games with robotic toys. Section 3 features the framework for recording and interpreting emotional movements. The common elements of the framework that make it able to analyze and model human movements are outlined. Several applications for modeling and analyzing emotional movements on and by robots or intelligent objects using this framework are described in Sect. 4. Also a platform and a game scenario that involves recognition and control over emotional behaviors is shown in this Section. Section 5 provides a discussion.

2 Stages of social development matched by games with robotic toys

Play behaviour is an expression of physical, social and emotional development of children. There are several stages of play corresponding to the level of social development.

Early researchers categorized the play they observed in young children into age related stages [29]. Later theorists described the stages of play development in specific domains, such as social, motor, and socio-cultural development [30–32].

These stages correspond to the stages of motor, social and the emotional development of the children. Exploratory and sensorimotor types of play are primary during the early infancy period. Symbolic play starts in the first years of child development and it is characterized with reciprocal interaction between the children and the emergence of social games. Wachs [33] distinguishes between object play (exploratory, manipulative, or functional play) as well as social play, related to peer and adult–child interactions. We have argued [34] that interaction with objects is a developmental stage of social behavior, and that shortcomings in the motor level of interaction can result in impaired social behavior [34].This argument is based on our understanding on the role mirror neuron system in social learning and interaction, that has emerged as an additional functionality of the neural structures that have primarily developed for gross and fine motor skills, such as grasping and precision grip. Therefore, difficulties in planning and executing simple discrete movements, for instance, can lead to problems in learning to coordinate diverse muscle groups into a unitary movement pattern. Moreover, when a person is unable to respond to another’s action in a timely fashion they will miss the positive reinforcement associated with interpersonal interaction.

Lack of motor imitation, as for instance observed by children with autism, results in difficulties to understand each other’s behavior. The complementary reciprocal play in early toddlerhood, such as run and chaise [35] may be prerequisite for cooperative social pretend play later on.

Considering each stage of social development, we design games that promote a wider spectrum of social and emotional experiences. We focus on physical play, i.e. play with physical objects that aims to enhance the interaction between the children [1–3]. To trigger more advanced forms of social play, we aim at developing physical objects that have own means to stimulate social interaction. These include: sensors so that the objects can record the changes in the surrounding world, some learning or adaptation mechanism that will facilitate decision making and insure a level of autonomy, and actuators so the objects can express behaviour. A physical object with such features is in a broad sense a robot independently of its shape or means of behaviour. As mentioned in the introduction of this paper, we aim at higher level of autonomy than the largely available robots have at present. The difference by this new level of autonomy is in the robot's ability to interpret human movement behaviour and to behave in such a way that humans understand. Specifically, we aim at designing robotic toys that can themselves express emotional behaviours, as well as understand emotional expressions in other agents. We consider 3 different levels of awareness for robots that can be used in game-planning scenarios:

  • Robots that understand the emotional state of a human player(s) and respond in an adequate manner;

  • Robots that imitate, enhance or counteract an emotional state of a human;

  • Socially believable robotic (embodied) characters that provoke social interaction;

  • Emergence of constantly adapting interaction based on movement/emotional understanding of a robot.

3 Framework for emulating and interpreting emotional movements

This section proposes a framework that will make it possible to express and interpret emotional movement patterns. The framework is based on the understanding that there is a close relation between expressed and perceived emotion which will make it possible for the robot to (1) model or synthesize emotion (e.g. on robot or another object with sensory-motor features) in a way that it can be understood by a human observer and (2) to recognize or interpret human emotional movement.

Evidence from independent studies in neuroscience, social psychology, and behavioral studies strongly suggest that our ability to perceive the actions of others results from the massive experience we have accumulated in planning and executing self-produced actions. Sensory representations used during action perception and motor representations for action planning are coded in the same brain areas. Different connectivity between these areas and additional signals determine whether the movement will be perceived as own or observed. This common coding principle (as known in psychological studies), or mirror neuron principle (as known in neuroscience) is a basis for designing the social interaction systems [14, 15].

A ‘social perception’ system that is directly connected to, and partially overlapping with the mirror neuron system, processes emotionally-rich facial expressions, bodily posture and actions of others, and triggers an appropriate emotional responses. It detects other persons’ assumed social motor intentions, triggers automatic social/emotional response plans, and so facilitates automatic intentional and emotional responses.

If these ideas are used in an interactive robot the same modules will be used for recording and interpreting emotional movements. The common coding principle makes it possible to create an interchangeable interaction through whole body movement between two robots, as shown in [14]. We want to extend this motor interaction to emotional body language that is expressed by movement. Moreover, we would like to make it possible to have mutual interaction and understanding between a robot and a human. This implies that a high level representation of movement primitives is characterized by certain emotional patterns. Such a high level representation which characterizes the human movement in terms of expressed and perceived intentions and emotions is searched in the framework of Laban movement analysis.

In order to identify the contribution of body movements to the recognition of emotion, it is important to have a clear and suitable description of these movements. The Laban Movement Analysis [28] is a well-established, effective method for observing, describing, annotating, and interpreting human movement, however it does not provide a straightforward way to assign quantitative measures to the movement qualities. It provides descriptors for the content of human body movements in terms of the following categories: Body, Space, Effort and Relationship. The Effort category relates to the dynamic and expressive characteristics of the movement. It is comprised of four movement qualities: weight, space (not to be confused with the Space category), time, and flow. Each quality represents a continuum between opposite polarities: weight varies between strength and lightness, space, between direct and indirect, time between sudden and sustained and flow between bound and free. Effort qualities usually appear in combinations called “states” or “drives”. When two motion qualities are combined, it is called inner attitudes or incomplete effort. When three motion qualities are combined, they form externalized drives. Of special interest is the Passion drive, which combines time, weight, and flow. Using these motion determinants that body takes in space gives a way to noticeably differentiate expressive and emotional actions.

For instance, the difference between punching someone in anger and reaching for a glass is slight in terms of body organization—both rely on extension of the arm. However, the attention to the strength of the movement, the control of the movement and the timing of the movement are very different. This example shows how the three qualities, namely weight, flow, and time, respectively, help characterize the emotional load of movements.

According to [36], the Laban Effort parameters can be translated into low-level movement parameters such as curvature, velocity, and acceleration.

After consulting a certified Laban movement analyst, and reviewed the related literature [36–39] we concluded that this relation is bidirectional, i.e. based on the physical characteristics of the movement we can determine the Laban characteristics of the movement, and vice versa. If the factors that are the components of the Passion drive (i.e. time, weight, and flow) are combined, a judgment about the emotional load of the movement is possible to be made. For this purpose, based on the expert’s opinion, our knowledge of modeling movement, and making a number of simple experiments we concluded that the acceleration, the velocity, and the profile (curvature, amplitude, and variability) of the movement can be used in the following way:

  • Time can be characterized by looking at the movement’s acceleration.

  • Weight is characterized by a combination of acceleration and velocity.

  • Flow needs the values of all three components in order to be assessed.

This relation is visualized in the Figs. 1, 2 and 3:

Fig. 1
figure 1

The relation between the dynamic characteristics of the movement and the Laban movement qualities. Needles to say that the transformation from the upper layer also can be used as an input and the lower layers as an output if we could get measurable weight, time and flow factors

Full size image
Fig. 2
figure 2

Transformation of effort factors into 4 basic emotions. A neural classifier is a good choice to realize this transformation

Full size image
Fig. 3
figure 3

NAO robot

Full size image

Connecting the Laban factors to emotional states is a two stage process. First, the right correspondence between the Laban factors and emotions has to be established. Camurri et al. [37] and Fagerberg et al. [38], have independently used LMA to classify dance gestures in terms of basic emotions; anger, fear, grief and joy. Their analysis overlaps for 3 of these emotions, namely anger, fear, and joy (happiness). We use the coding that corresponds to the relation between Laban movement qualities and emotions as introduced by Camurri et al.

Second, since we work with partially subjective parameters a classification method that can deal with sufficient degree of uncertainty has to be deployed. For this purpose we used variations of neural classifiers for both analysis and synthesis of emotional movement.

In summary, the framework for analyzing and simulating emotional movements takes the following steps.

  1. 1.

    Description of movement that allows for an easy bidirectional transformation between the dynamic characteristics of a signal (velocity, acceleration and variability) to Laban Effort factors.

  2. 2.

    Learning the dependence between Laban Effort factors and the basic emotions.

  3. 3.

    Prediction of movement as an element of the interaction.

4 Examples: using the framework for expressing and interpreting emotional movements by robots

The framework was tested in several prototypes of behaving objects for social play that correspond to one of the levels of awareness of the robots as listed in Sect. 2. Once more we would like to state that our definition of a robot is a device or everyday object with sensory, motor, and processing capabilities. Movement is the most recognizable expression of behavior, but other controlled behavioral expression, such as speech, light changes are also possible and considered as valid motor expression.

4.1 Simple behavioral response based on interpreting the emotional state of human players

Analyzing the emotional state of the human player is a very complex task. The complexity also depends on the sensors, which can provide simple data in the range of binary values (i.e. on and off) to camera images that require complex signal processing. We restrict the complexity of the analysis by assuming that either the robot or the player will be static at every moment, so when the robot observes moving body parts it does not need to recalculate them with respect to its own movement.

In order to abstract information about the movement qualities by a static object with sensory-motor features, we constructed a behaving walk-in closet equipped with 3 types of sensors and a learning algorithm that decides the closet behavior. We placed sensors in the surrounding floor area, in the shelves and in the vertical space in the walk-in closet. Human players were asked to act out several scenarios. From the analysis of the sensor data it became apparent that the walking movement and the hand movement are most informative about the emotional state of the human subjects. These two movements relate to two kind of movement analysis: micro analysis of the movements in the kinesphere of the user (that relates to a body part in our case arms) and macro analysis to describe the larger (whole body) movements in the space. To capture macro movement we placed 12 sensor mats (595 mm × 170 mm) in the floor which are able to capture the walking movement in one dimension. In the shelves we placed 8 passive infrared sensors to measure micro movement activity. The sensors were connected through three Arduino microcontrollers to a PC. A Neural gas learning algorithm was implemented on Max/MSP and connected to the actuators that controlled the intensity, the speed, and the order of lighting of the lamps.

The analysis of the data showed that the movement qualities space and time were sufficient for the learning algorithm embedded in the closet to distinguish 4 different states of the players. For every combination one scenario was created which would be tested in an experiment. All the scenarios had as main context a selection of clothes for different occasions. The goal of the scenario was to evoke a certain state on the user, such as the feeling of hurry, relaxation, or enjoyment.

The closet is currently not prepared to distinguish between the different players but its behavior can be changed by increasing the activity in the object handling or in whole body movements of the game participants. So far the collective play behavior is not tested systematically. Our focus has been on the mood analysis and on the simulation of submissive and dominant behaviors of the closet.

4.2 Social mediation of play based on recognition and expression of emotions by movement

For the purpose of mediation of play between children, a robot that can engage in reciprocal social interaction through movement was used. In particular, we design behaviors to support collective games for the humanoid robot NAO [40]. The underlying idea is that the robot can recognize and imitate human movements and give feedback on the behavior of the human. The game scenarios that can be constructed on the basis of this interactive behaviors aim to help children with autism learn to recognize emotional movements from a robot partner and to produce similar movements.

For this type of interaction behavior a more precise method for analyzing the human movement is needed. The NAO robot has 25 degrees of freedom, 5 in each leg and arm, and 1 in each hand. Further it has 2 degrees of freedom in its head and one in the pelvis. The platform contains 2 color cameras with a maximum resolution of 640 × 480 pixels at a speed of 30 frames per second. The platform contains an embedded AMD Geode 500 MHz processor and is shipped with an embedded Linux distribution. A software library called NaoQi is used to control the robot. This API provides an easy to use C++ interface to the robot’s sensors and actuators. Due to this library it is relatively easy to control the robots actuators and make use of advanced routines that let the robot move and talk using text to speech conversion.

Challenging, however, remains the recognition of the emotional behaviors, which we resolved within the framework outlined in Sect. 3. A performer and certified Laban movement analyst was asked to enact waving behaviors with different emotional coloring, namely angry, happy, sad and polite. The waving was chosen for several reasons. First it is a behavior that is used exclusively in a social context. Second, it can relatively easily be tracked by a single camera, and third, several emotional states can naturally be expressed through waving. In fact our goal is not to confine to the type of the movement, but the extraction of a dynamic primitives that are typical for a certain emotion.

Figure 5 depicts pairs of 4 emotional waving patterns recorded in an experimental scenario as shown in Fig. 4, using color images of 640 × 480 pixels at a speed of 29 frames per second. The image processing consists of a combination of skin color and motion detection with the aim of tracking a single body part per person that can be associated with the emotional waving movement. The black rectangular regions capture the center and the boundaries of the skin color areas, green areas capture moving objects and the blue areas give a combination of a moving skin colored area. Due to parallel processing on GPU’s, multiple moving hands can be followed, which makes this method suitable for collective games.

Fig. 4
figure 4

Marked regions of interest. Black and white rectangular regions denote skin and motion area, respectively. A black rectangular region in the right snapshot denotes the average position of the fastest moving skin color objects. Tracking of multiple objects is possible because of the parallel processing framework

Full size image

Fifteen recordings of 20 seconds have been made with waving patterns that enact happiness, anger, sadness, or politeness (Fig. 5). At each row two examples of the same enacted emotion are plotted. Note that the trajectories of the movements have been different by enacting the same emotional state. For instance for angry emotional state the performer acted movement resembling pushing away a fly and a movement resembling an angry mother calling for her child to come. The first movement starts close to the body and moves laterally away. The second starts frontally, away from the body and finishes frontally, close to the body. The plots, however, show the acceleration profile of the movement which is similar for each emotion, independently of the trajectory of the enacted movement. From Fig. 6 the following can be observed:

Fig. 5
figure 5

Waving patterns. First to fourth row show acceleration profiles for happiness, anger, sadness, and politeness for two differently enacted emotional waving patterns

Full size image
Fig. 6
figure 6

Distinct emotion profiles are revealed by average frequency and acceleration

Full size image
  1. 1.

    Happy waving provides a regular waving pattern with a relatively high frequency.

  2. 2.

    Angry waving demonstrates bursts with tremendous acceleration

  3. 3.

    Sad waving demonstrates a profile of low acceleration; its frequency is relatively low and appears to have a lower frequency compared to the other three emotions.

  4. 4.

    Polite waving is a regular pattern with a relatively high frequency that is obtained by using minimal energy.

To analyze whether the emotional patterns can be properly classified by a robot, an average acceleration-frequency matrix was plotted. Figure 6 shows that four distinctive clusters are formed.

4.3 Expressing emotional behaviors in social game with robots

The Laban movement guidelines as suggested by Camurri et al. [37] were used to design emotional behaviors on a mobile robot. Note that for this experiment we did not use directly the human movement primitives extracted from the signals that were analyzed in Sect. 4.2. This choice was made since the recordings that are illustrated in Sect. 4.2 are not made with a representative group of participants. To test solely the perceived emotion from the movement, the experiments with the e-puck [41] robot that does neither have anthropomorphic nor zoomorphic shape were performed. A control user group of 42 typically developing children were tested to observe the emotional behaviors enacted by the robots. The outcome of the tests showed a good recognition of several basic emotions (Table 1).

Table 1 Recognition rate of basic emotions from the designed movement behaviors
Full size table

The chart shows the recognition rate in percents of the designed emotional behaviors for each emotion. It is important to mention that the children were not provided with a list of possible emotions. They were asked how they think that the robot is feeling at the moment.

The low recognition rate on anger and happiness emotions was the reason to construct the confusion matrix out from the user test outcomes. From this matrix become apparent that the two emotional movement behaviors, anger and happiness were mistaken for each other. These two emotions are characterized with high intensity but have different valence (i.e. happiness is a strong positive emotion, while anger is a strong negative emotion). Moreover, the Laban descriptions of these emotions, as suggested by [37] have many common elements, as evident from Table 2.

Table 2 The descriptions of emotional movements for Anger and Joy (Happiness) as concluded by Camurri et al. [37]
Full size table

In contrast, the acceleration plots as made in our experiments with robot vision and described in Sect. 4.2 show very different acceleration profiles. The recorded waving pattern that represents Happiness provides a regular waving pattern with a relatively high frequency (Fig. 5a, b), while the Anger waving demonstrates bursts with tremendous acceleration, and pauses in between (Fig. 5c, d).

To analyze the robot behaviors we attached the Wiimote to the robot when it performed the emotional movements. The plots did not show the typical acceleration profile as by the emotional movements enacted by a human demonstrator. The recording of the acceleration profiles from human motion patterns are a substantial step for redesigning the robot emotional behavior.

The so designed emotional behaviors were included in a game for promoting associative play [1] by children with autism. To design the game the following shortcomings of the children were targeted: inability to share and socially interact, inability to understand expression of emotions and link them to context, preference to learn by examples and logic rather than by trial and error.

To account for these problems, a combined approach of a game that will require negotiations and working towards a common goal, together with recognition of emotional states was made. The game uses a storyline that describes various situations involving different emotions. When the children recognize the emotion described in the story they have to command a robot to either perform or contradict this emotion. The robot is commanded by the collective physical behavior of the children. At least two children have to step on one site of a large disk to make the disk tilt (Fig. 7). The disk can be tilted in several directions denoted with colored LED-lighted arrows. Every tilt direction will provoke movement behavior of the robot that expresses particular emotion. The tilted disk will trigger a movement of a robot that corresponds to the corresponding emotion. The LED arrows have the color that resembles the emotion in a similar way as the traffic light metaphor that was used in schools for autistic children to illustrate children’s emotions. The bottom line of the game is that the children have to identify the correct emotion based on the story and movement. The teacher can use different storylines referring to an emotion, that will consequently be acted by the movement of the robot. With this multivalent approach we aimed at an integrated understanding of the acted emotion.

Fig. 7
figure 7

Game situations and flowchart of the game platform. The robot in the game situations is drawn as a small object near the platform

Full size image

For the purpose of the game a huge round disc that could accommodate several children was made (Fig. 7). The disc can control the movement of a robot by being tilted in a certain direction. The robot in the game situations is drawn as a small object near the platform. It resembles the shape and the size of the e-puck robot [41] that was used for this experiment. The e-puck is a two-wheel mobile robot that was originally developed at Swiss Federal Institute of Technology (EPFL). The robot is equipped with a dsPIC processor and can be controlled by blue tooth through the computer, or a very simple program can be uploaded on the dsPIC processor. We used computer-mediated control.

The children, standing on top of the platform, had to negotiate their positions since a single child could not tilt the disc. Once multiple children moved to the right position and the disc tilted at that direction, the corresponding behavior of the robot was triggered. To change the robot emotional behavior, the children had to agree on their next position and move together. When conflicting views occur, it was an opportunity for the children to learn to negotiate and get aware that they need others’ help.

5 Discussion

We proposed a framework for expressing and interpreting emotional movements that is based on actual recording, analyzing and emulating of emotional movements on robots. It uses the same way for representing the emotional behavior in the agent which expresses and the agent which observes (or interacts). In this way the proposed framework enables with the same modeling, sensing, and analysis tools to design games and scenarios with robots and other intelligent objects that have different degrees of awareness. By using this framework, different levels of skills of the robots, namely their abilities to analyze the emotional state of the human player, to express own emotional social behavior, or take part in reciprocal social interaction can be achieved with no need for re-design.

Laban movement analysis was used to provide the possibility for a unified way of expressing and interpreting emotional movements. A distinctive feature of scenarios that use Laban movement analysis is that they allow for realistic interpretation of the human (emotional) state. Differently from the existing studies that employ the Laban movement analysis we choose for analysis of the qualities of the actual movements with elaborate image processing techniques. The analysis of the human movements with a camera, showed different acceleration profiles of the movement as the one that were suggested by other studies on emotional movements that use the Laban framework, for instance by [37]. This fact suggests that using signal analysis can improve the designed emotional and social behaviors by capturing features of the movement that are omitted by observations. Indeed, when the robot behaviors for the described game were designed purely on the recommendations of [37], two emotions were largely confused by the participants in the user test. After the acceleration patterns of the so created emotional movements were recorded and compared with the emotional movement by the human observer, there was an obvious difference in the acceleration profiles of the misinterpreted signals. Moreover, our approach is straightforward for the design of interactive behaviors, since it gives a clear description of the behavior.

Last, but not least, the parallel processing framework that we previously suggested in [43] and used in the experiments shown in Sect. 4.2, allows for parallel tracking and interacting with multiple human or robot agents, which makes our framework especially suitable for games with multiple human participants, and therefore for design for social and physical play. This is a subject for our future research.