Sensing spatial and temporal coordination in teams using the smartphone

Team coordination in safety-critical environments has been assessed using different methodologies. A common approach includes behavioral observation. By observing recorded videos of the team interaction and encoding predefined behaviors, researchers can investigate temporal aspects of team processes such as patterns of interaction and changes over time [9]-[11]. Behavioral observation, however, is very resource-intensive and impractical for applied settings.

Another approach to team processes focuses on the structural characteristics of teamwork. Crawford et al. recently introduced a theoretical framework that considers structure [12]. By drawing on social network analysis (SNA), their theory proposes different types of networks to provide a more comprehensive explanation of the relationship between team processes and performance. SNA expresses the social environment as patterns or regularities in relationships among interaction units [13]. These relationships (i.e., ties) can be of different types. For example, a communication network could capture which members of a department communicate with each other on a regular basis. Such a network can reveal those members that are central to the dissemination of information within this department. The relationship data that makes up a social network can be represented and analyzed in different ways [14]. For a quantitative analysis, different relationship metrics can be derived from the data. These metrics can describe properties of individual members or of the whole team. The most common team-level metrics include density and centralization [15]. Network density is defined as the ratio between the actual number and the total number of ties in a network and is often used as an indicator of cohesion [13]. Centralization refers to the variance in ties per team member; low values indicate a structure in which each member has the same numbers of ties. Centralization reflects aspects of work organization and hierarchy.

Researchers have applied SNA to teams in organizations. For example, it has been suggested that centralization has a negative impact on team performance in complex tasks [16]. Zohar et al. showed that the density of a military teams communication network mediated the effects of transformational leadership on climate strength [17]. Likewise, a series of case studies with police and firefighting teams suggests that both teams have different network architectures (distributed vs. split) [18].

Despite the potential of SNA to uncover the underlying structure of team processes, the number of studies using SNA in team research is small [12],[15]. We believe that part of this problem lies in the method itself as SNA, like behavioral observation, is very resource-intensive and often impractical for applied settings. One way to address this issue includes taking advantage of new developments from the field of mobile behavior sensing.

Mobile Behavior Sensing aims at measuring and analysing human behavior from sensor data recorded with mobile devices [8],[19],[20]. Research in wearable and ubiquitous computing has shown how user context and behavior can be inferred from the smartphone’s sensor data using signal processing and machine learning techniques. The integrated sensors capture device interaction, body movement, location and speech of the user as well as characteristics of the user’s environment such as ambient light and sound. Characteristics features are then extracted from the sensor signals to make inferences about the context, state and behavior of a user.

Farrahi et al. used coarse location information from cell towers and clustered individual location traces to discover daily routines such as “going to work at 10am” or “leaving work at night” [21].

In order to give semantic meaning to recorded location information, features from ambient sound and video were fused to categorize location into place categories such as “college/education”, “food/restaurant” or “home” [22],[23].

The audio modality has been analyzed in the mobile setting to detect conversations, recognize speakers and estimate speaking duration [24]-[26], perceived basic emotions [27], perceived stress during a street promotion tasks [28] and to quantify sociability as one aspect of well-being [29].

On a macro level, Eagle et al. have first shown how mobile phones can be used to infer proximity networks of communities [20]. Relying on repeated Bluetooth scans, mobile phones were used to detect other nearby devices to estimate proximity between individuals. On the same dataset, topic models were later used to discover human interactions from the proximity data [30].

Before the smartphone was available, Choudhury et al. introduced the sociometer, a wearable device, to automatically sense body motion, communication and proximity networks [24]. Extending this line of research, Olguin et al. used a new version of the sociometer to collect behavioral data of nurses in a hospital. The results showed a positive relationship of group motion energy and speaking time with group productivity [31].

In previous work, we adopted the idea to use motion and speech activity to monitor teams. Our feasibility study showed that speech and motion activity are promising performance indicators in firefighting teams [32] which motivated us to design, build and distribute our mobile sensing app CoenoFire in a real fire brigade [33]. In this paper, we build on our approach to sense team proximity dynamics with the smartphone [34].

Our approach is schematically presented in Figure 1 and consists of the following steps:

In the example presented in Figure 1, person A is standing still while being in proximity with the running persons B and C. Person D on the other hand is walking behind a wall and is therefore not in any sub-group with another person. This leads to the presented sub-group network. As person A is in-sight with persons B and C, the sub-group graph shows them to be in one group, whereas person D is indicated to be alone. The movement alignment network shows person B and C to be best aligned because they are both running, whereas person A is worst aligned as she is the only person not moving. From the sub-group and movement alignment networks SNA metrics are derived to characterize overall network structures in order to capture team coordination indicators.

Our data collection framework called CoenoFire is illustrated in Figure 2 and consists of two parts: the smartphone data recording app as the sensing front-end and a database and visualization server at the back-end. CoenoFire was developed to monitor firefighters during real-world incidents. Details on the system architecture were previously presented in [33].

For data collection, we used the Sony Xperia Active smartphone which is dust and water-resistant, has a 3-inch capacitive touchscreen and a built-in ANT-radio (http://​www.​thisisant.​com). ANT is a low power wireless protocol that was developed to connect fitness devices such as heart-rate-belts and pedometers with sport watches; however, in this work we use it for proximity estimation. We developed an Android app to continuously record data of the phone’s built-in sensors. Therefore, we extended the funf-open-sensing-framework [35] to also detect nearby devices by transceiving ANT-radio messages and to save the raw sensor data locally to the memory card.

Data was recorded from the following built-in sensors: acceleration and orientation sensors were used to measure body movement, the barometer measured atmospheric pressure and was used to infer whether individuals were on the same floor level and ANT-radio messages were sent and received to find out which team member was in proximity to another one.

As our goal was to sense team behaviors, all devices carried by the team members needed to be synchronized to allow comparison of the sensor signals across team members. Therefore, we measured the offset between system time and a common reference time each 5 min using the network time protocol. With this approach, we were able to achieve a time synchronisation across devices with a maximum time difference of 500 ms. To enable remote monitoring, we configured the framework to upload every five minutes a subset of calculated features, such as the battery level to a central server. Because we used the smartphone as a sensing platform, we installed our app as the default homescreen and blocked all soft buttons. In this way, our app was always visible and the use of the smartphone was restricted to our data collection.

At the back-end, we ran one web server to receive and store the data from the smartphones in a central database. A second web server provided a web-based user interface that offered real-time monitoring of the system. A screen shot of the web interface showing the battery status of the devices is presented in the right of Figure 2. The interface also allows visualization of real time data of the firefighters’ movement and speech activity.

In order to validate our approach, we tested it in a sample of firefighting teams completing a training scenario. The study was conducted in cooperation with the Zurich fire department and approved by the Ethics committee of the University of Zurich. Written consent from all participants was obtained prior to data collection.

In our previous work [34], we have shown how moving sub-groups within teams can be detected from radio-based proximity data obtained with smartphones. The detected sub-groups can be visualized in the form of narrative charts to display which team members were in sub-groups at each point in time and show how sub-groups merge and split over time. The narrative chart presented in Figure 4a illustrates the moving sub-groups of firefighters during the described training scenario. The chart allows for example to identify the points in time when the first (1) and second troop (3) reached the top of the turntable ladder and when the first troop entered the building (2). As can be seen in the narrative chart the lines representing the members of the first troop (T1a, T1b, T1c) split from the other lines (other team members) at time point (1) when the first troop forms a sub-group and uses the turntable ladder to reach the roof window. At time point (2) two members (T1a, T1b) of the first troop enter the building and team member T1c remains outside on the top of the ladder. In the narrative chart this is shown by the splitting of the yellow line from the orange and purple lines. At time point (3) members of the second troop (T2a, T2b) climb the turntable ladder and join team member T1c which is shown in the chart by the merging of the red and brown lines with the yellow line.

Having detected moving sub-groups, we are able to calculate a sub-group network that summarizes which team member was for how long in a sub-group with another team member. Thus, the sub-group network captures the overall spatial structure of the team during a mission. In Figure 4a the corresponding sub-group network is presented on the right of the narrative chart. The network graphs highlights three sub-groups that belong to the first and second troop that enter the building via the turntable ladder as well as the ground support team that includes the incident commander, turntable ladder operator and the engineer. In the graph darker links between nodes correspond to team members that were longer than 60% of the mission together in a sub-group.

In the following we briefly describe our method to detect moving sub-groups using radio-based proximity data. Please refer to [34] for more details. We follow a two stage approach to detect moving sub-groups: We first calculate the proximity matrix D ^t for consecutive time intervals t of length L=5 s. Each binary element D ij t indicates whether device i received any ANT message of device j during time interval t. Considering proximity to be undirected, we further symmetrize the proximity matrix to obtain D sym t.

In the second stage, moving sub-groups are clustered from the proximity data. Clusters are first identified independently from the symmetrized proximity matrices of each time interval and secondly, the clustering output is smoothed by applying a temporal filter, so that clusters last for at least 10 s. We cluster each symmetrized proximity matrix D sym t using the single-link criterion. As a result, if group member A is connected with B and B with C, but not with A, all three devices are still clustered into one group.

Using only radio based proximity information might lead to individuals on different height levels to be clustered into one group. To address this problem, we added height information derived from the atmospheric pressure sensor. If the absolute atmospheric pressure difference between two devices is greater than a predefined threshold, the two devices are considered to be on different height levels and are thus not clustered to the same sub-group. To obtain the sub-group network, we average the clustering results over all time intervals.

As the ANT-radio protocol operates in the 2.4 GHz band, radio signals are particularly influenced by the surrounding environment. In our experiments, we observed that depending on the relative orientation and environment of the individuals, the maximal transmit distance varied in the range of 1 m to 20 m. In [34] we evaluated our algorithm to detect moving sub-groups of firefighters during the described training scenario by comparing the results to a manually annotated ground truth. On average, team members were assigned to the correct sub-group with 95% accuracy.

In order to capture the temporal aspect of coordination in teams, we measure and compare activity levels of individual team members. Thus, we assume that well coordinated team members change their activity level at similar points in time.

We define the activity level to be the fraction of time that an individual is active within a moving window of length L. The activity level increases when individuals become active and decreases as soon as team members stop moving. The window length L determines the slope of the activity level and the minimum time that an individual needs to be active to reach the maximum activity level. The value of L also affects the temporal resolution, a small value requires individuals to change their activity closer in time, whereas a larger value allows for a delay between activity changes, as the activity level is calculated over a longer period.

Figure 5 illustrates the calculation of the motion activity level. In a first step, we detect when a team member is active, by thresholding the moving standard deviation of the linear acceleration magnitude. When a predefined threshold is exceeded, motion activity is detected (top in Figure 5). In a second step, the motion activity level is calculated as the percentage of time that motion activity was detected within a hopping window of length L and step size S (bottom in Figure 5). For further processing, the continuous activity level is linearly quantized into 10 discrete activity levels {0..9}.

In order to compare two motion activity level signals X,Y∈{0..9} of two team members, we use the mutual information as similarity measure. In general, mutual information measures the dependency between two random variables, that is how much information two variables share and is defined as:

The dependency between X and Y is expressed by the joint distribution p(x,y) and compared to the joint distribution when independence is assumed, in which case p(x,y)=p(x)p(y). Thus, I(X,Y) is zero if and only if X and Y are independent.

Two examples of activity level alignment that occurred during the firefighting training scenario are presented in Figure 4b. The two activity levels presented in the top graph belong two team members from the first troop (T1a, T1b), whereas the activity levels shown in the bottom graph belong to team member T1b and the incident commander. While the activity levels of the troop members change often together in time and are well aligned, the activity levels of the troop member T1b and the incident commander are not well aligned in time. Consequently, the observed mutual information is higher between the activity levels of the troop members as opposed to those of troop member T1b and the incident commander.

In order to summarize the temporal alignment for the whole team, the mutual information between all pairs of activity levels are calculated. This results in the activity alignment network. An example is presented on the right side of Figure 4b. As can be seen, troop member T1b had highest activity alignment with troop member T1a and lowest with the incident commander.

On each of the extracted team networks (sub-group network and activity alignment network), we calculate network density and degree centralization in order to characterize the global network structure. In summary, we extract the following team coordination indicators from the team networks:

In the following, we give the definition of network density and centralization. Degree centrality of a node in the network captures how well each node (team member) in the network is connected to other nodes (other team members). Degree centrality of node i is defined by

with a _ij ∈[0..1] being an element of the adjacency matrix defining the network and i,j∈[1..N], with N being the number of nodes in the network. Network density D is the average degree centrality and thus captures how well nodes in the network are connected with each other. Network density is given by

Degree centralization DC measures how central its most central node (highest degree) is in relation to how central all the other nodes are. Thus, it is a measure of degree variation and is zero in a homogeneously connected network where each node has the same degree and one in a star network. Degree centralization is defined by

with d^* being the maximum degree observed in the network.

In order to evaluate our approach, we correlated the proposed coordination indicators derived from the sensor data with three validation criteria. We used perceived implicit and explicit coordination to validate the indicators. Explicit coordination includes those actions intentionally used for team coordination and is achieved by means of verbal communication. By contrast, implicit coordination refers to the anticipations of others members’ actions and the dynamic adjustment of one’s own actions accordingly, without the need for overt communication [37].

We additionally used time to complete the training mission as objective validation criteria. As time is critical in firefighting our reasoning was that well coordinated teams would be faster.

Our findings reveal significant relationships between both subjective and objective validation criteria as can be seen from Table 1. Figure 6 shows the relationships graphically. In summary, we find:

The finding that faster teams had a sub-group network with higher degree of centralization makes sense, because the chosen scenario demanded teams to split into at least three sub-groups of different size: the troop that went inside the building, the firefighter on top of the ladder that helped with the fire hose and the remaining team members on the ground outside the building. Thus, faster teams organized their spatial structure more efficiently than slower teams.

In terms of activity alignment, the results showed that faster teams exhibited higher temporal movement coordination. This finding seems reasonable as it indicates that firefighters in faster teams worked well together and aligned their movements accordingly. Thus, faster teams moved on average more synchronously than slower teams.

The correlation analysis did not indicate any significant relationships between the proposed team coordination indicators and perceived explicit coordination. This is likely due to the fact that the proposed coordination indicators measure spatial and temporal aspects and did not include direct verbal communication which is essential for explicit coordination.

To further analyse the correlation results, we visually inspect the team networks. In Figure 7 the sub-group networks of the teams are presented. As can be seen, faster teams tended to have one firefighter on top of the ladder (red circle) and two well connected sub-groups, the troop and the remaining firefighters on ground. The observed high degree centralization stems from the fact that faster teams split quickly into these three groups so that the firefighter on top of the ladder was relatively long alone, which is the reason for the low degree (small circle). As the other firefighters were part of a sub-group, their respective degrees were higher (larger circle). Due to this difference in the individual degrees the overall degree centralization of the network is high. While not being statistically significant, it can also be observed from the networks, that slower teams showed higher density, meaning that their team members stayed longer in well connected sub-groups, which indicates that they needed more time to split into the required sub-groups.

The movement alignment networks of all groups are presented in Figure 8. As can be seen, faster teams (top row) have better connected networks and thus higher network density. Further, the networks show that the highest activity coordination occurred between members that worked close together such as the troop inside the building. Troop members of faster teams had higher alignment between their activity levels (represented by bigger circles) than those of slower teams (represented by smaller circles).

We see four main implications of our method:

Our proposed team sensing approach also has potential implications for practitioners. Instructors of first responder teams can use the smartphone during training to collect objective data on team processes. These data could then serve as additional input for debriefing and thus enables data supported training feedback. Even more so, as the data can be illustrated using different graphs. For example, the narrative chart (compare Figure 4) can be used to get a quick overview of when sub-groups formed and disbanded over the course of a mission.

As the smartphone allows for continuous, real-time assessment of teams, it could also be used during actual missions to monitor performance and coordination. This has a large potential for error prevention. For example, using radio-based proximity data, mission commanders can detect when a team member moves out of sight of the rest of the team without relying on GPS or any installed infrastructure. Being alone may pose a threat to this person because it will be more difficult for his teammates to recognize potential dangers and to provide timely backup. As smartphones are widely used, it would not be difficult to implement such a monitoring system.

In order to detect moving sub-groups over time, we make use of radio based proximity estimation. The accuracy of nearby device detection dependents on architectural constraints and consequently varies across locations. Experiments showed that the maximum detection distance varies between 1 m to 20 m. This accuracy proved to be good enough for the detection of moving sub-groups in the firefighting scenario. We therefore believe our method to be also applicable to other first responder teams. Having room-level accuracy, the approach is also useful to track white-collar workers in office buildings in order to capture their co-location networks, which could be used to identify important persons in a social network. However, in a social event in which individuals stand close together the spatial resolution is likely not to be sufficient to reliably detect social interaction. In such scenarios, the used technique can only give rough proximity cues.

Further, we measured temporal coordination as simultaneous change in activity level. As the activity level captures the amount of body movement, it is only a rough estimate of action coordination. For firefighting teams and most likely also for other first responder teams, the simultaneous change in body movement is clearly related to team coordination because in such teams it is important to move together to solve the task. For white-color workers in an office building this simultaneous change in movement however has no clear meaning. In typical office work, body movement itself is not a driving process behind team performance.