Who talks to whom: an evaluation of a call log visualization

The visualization of dynamic networks has become an active research discipline with increasing relevance for various communities. Different concepts of layout, visualization and interaction techniques can be found in the literature as analyzed by von Landesberger et al. (2011) and surveyed by Shaobo and Lingda (2017). Several ways to map the time dimension to the visualization of dynamic graphs have been explored. The two most common are to either map time to time, which results in an animation or to map time to space which creates a timeline within the visualization (Beck et al. 2017). Most commonly, a series of diagrams gets animated. To follow and understand changes over time, mental representations need to be preserved, which can become critical in animation as limitations of memory and cognitive overload can become a problem  (Archambault et al. 2014; Beck et al. 2014). The cognitive load theory describes that humans can improve thinking and reasoning by using external structures to reduce the need to keep information in memory (Sweller et al. 2011). In that sense, showing diagrams next to each other as small multiples is the better alternative as each slice in time can be observed at once.

Evaluations of graph representations shed light on the suitability for specific tasks and scalability for appropriate dataset sizes. The evaluation of Ghoniem et al. (2004) compares a node-link diagram to a matrix-based representation, which results in both being able to show advantages depending on graph size and density. Simple time to space mappings, such as time to color, saturation, glyphs, etc., are rarely applied independently and rather combined with other techniques (Beck et al. 2014). A challenge in representing dynamic graphs is to provide overview and detail. Burch et al. (2015) present edge-stacked timelines using color to show individual weighted connections of a graph. They use two linked representations side by side, one showing the graphs of each point in time in a minimized form on a narrow timeline. Each graph can then be selected to be shown in more detail for further investigation on the timeline in the second representation. Only through interaction one can follow edge changes on the overview timeline and obtain details about which nodes are connected in the second visualization.

To avoid timeslicing on continuous datasets, Simonetto et al. (2018) developed a dynamic graph drawing algorithm to be able to show dynamic graphs in a space-time cube. They argue that despite being intuitive it is often unclear how many timeslices to select to get a reliable picture of the graph structure while not slowing down the visualization by too many timeslices.

A considerable amount of evaluation studies can be found in the field of information visualization. Beck et al. (2017) categorized visualization papers on the basis of the evaluation approach that was used. They describe seven types of evaluations: case study, user study, algorithmic, expert, survey, theoretical and none (for papers not mentioning an evaluation). They further state that deciding whether a visualization technique is helpful or not in the end always requires to involve users. User studies, for example, can be used to compare visualization approaches or test visualization parameters under controlled conditions. Often, specific visualizations have been compared in that way to demonstrate the superiority of one visualization over the other. The user studies of Farrugia and Quigley (2011), e.g., report response times with static representations being significantly faster for most tasks compared to the animation of two dynamic graph series.

Response times alone can not provide a conclusive picture of the qualities of a visualization. Qualitative methods can elicit user preferences and thoughts and, thus, provide insights about how humans make sense of visualized data. Lam et al. (2012) introduce seven evaluation scenarios divided into process and visualization categories. The first category allows analysts to understand the process and the roles played by visualizations, while the second category focuses on analyzing the underlying visualization itself, e.g., by testing its usability and making design decisions.

The role of aesthetic criteria in perceived usability has been the subject of several evaluation studies. Beck et al. (2009) provide a framework for a standardized qualitative assessment to identify strengths and weaknesses of visualizations. They evaluated three dynamic graph visualizations in this way, while pointing out that this kind of assessment is subjective to some extent. Burch (2017) sets aesthetic criteria as targets for the design of the Dynamic Graph Wall, a dynamic graph visualization using multiple visual metaphors. The author considers node-link diagrams, adjacency matrices and adjacency lists to automatically render the appropriate use case based visualization in a separate view as graph properties change over time.

Research shows that the interaction design is a critical factor for the success of scalable graph visualizations (Pienta et al. 2015). One approach to analyze the dynamics in a graph is to show differences in specific points in time. NetEvViz enables to select two time points on a timeline to show the differences in a node-link representation (Khurana e al. 2011). Another approach is to use multiple parallel links between nodes (Beck et al. 2014). Node-link diagrams with multivariate edges were presented as multiple threads by Ko et al. (2014), who used colored parallel lines. However, colors can be difficult to distinguish and the approach does not scale well.

NetVisia, an approach to show evaluation in social networks (Gove et al. 2011), combines heat maps and matrices to support the exploration of temporal changes in networks. It was evaluated with regards to potential usability problems as well as new data insights. Think-aloud protocols of the users were analyzed to see which kind of insights could be gained through the visualization. Results show that novice users could gain insights within a few minutes despite observed usability problems.

Dynamic networks are relevant for various fields. Social network is only one specific application example. Other applications for dynamic network visualization can be found in the domain of security data (Zhao et al. 2014) or ranking data (Lei et al. 2016). Stoiber et al. (2019) present netflower, a visual analytics tool for the open exploration for data journalists. Netflower uses a Sankey diagram, a flow diagram in which the width of the arrows is proportional to the flow rate as the main visualization, to represent network change over time. The tool extends the main visualization by bar chart sparklines on both sides to provide temporal overviews of data attributes.

Visual analytics tools can also provide advantages in crime investigations where multiple heterogeneous data sources need to be combined as well. Seidler et al. (2016a) introduced a combined node-link and matrix visualization system to visualize dynamics of criminal behavior. They evaluated the technique with domain experts using think-aloud protocols with observation. They showed that using both, a node-link and a matrix visualization, strengthens the identification of harmful developments. Moreover, Gove et al. (2011) designed a novel social network system which includes heat maps and matrices to support users exploring temporal changes in networks. Hlawatsch et al. (2014) introduced a visualization for dynamic, weighted graphs based on adjacency lists which is timeline based and juxtaposed. They represent nodes and corresponding links separately on two horizontal axes and use colors to further distinguish the nodes. A quantitative user study confirms the suitability for dynamic weighted graphs, especially for the analysis of link distribution in graphs with asymmetric characteristics. Based on their results, aesthetic criteria, such as compactness and visual clutter are further discussed.

To communicate results and insights gained from data exploration another aspect of visualization comes into play: being able to tell a story. Benjamin et al. (2016) showed that graph comics are a good storytelling medium as complex temporal changes in networks can be conveyed with minimal textual annotations and no training of the audience. The structure of a network can provide the analyst with information of different roles of the actors. As Liu et al. (2017) analyze connected nodes, they distinguish the nodes of an intermediary to the ones of a professor. An intermediary may know many people who are not interconnected as they often do not know each other. Whereas professors know many students that may have many links that tie them to each other (as they probably visit the same classes and are in the same social circles). As soon as these roles or professions change, the social network changes as well which might lead to new trends and patterns.

There are two approaches that were a significant inspiration for the design of the proposed visualization of this work. In the approach of Burch et al. (2015), stacked-edges show individual connections of a graph. The main difference to our design is that we do not use color to show weighted graphs. Another inspiring approach was developed by Kriglstein et al. (2016) who examined visualizations for group meetings. In their representations, letters denote locations, numbers denote hourly intervals and colors denote different individuals. Especially, their augmented matrix was an inspiration for this work. The idea to connect persons in a matrix structure via links inspired the approach of this work in that the person’s row can be scanned to see who the person connects with. In their empirical evaluation, Kriglstein et al. could show that matrix representations were superior to other forms of visualizations.

An aesthetically pleasing design can significantly influence the usability, readability and ultimately the usage and success of a visualization or of any product for that matter. The aesthetic criteria discussed in the literature often also include usability considerations. Edge crossing is one of the most often mentioned of these criteria. Avoidance of edge crossing does not only result in a more pleasing appearance, but also in an increased readability. Beck et al. (2009) addressed aesthetic dimensions of dynamic graph visualizations in an effort to help designers come up with applicable new designs in practice as well as to compare and evaluate them. They point out that modified design criteria are necessary for dynamic graphs compared to static graphs. In their work, they attempted to translate vague aesthetic aspects into specific criteria that are directly applicable to arbitrary dynamic graph visualizations. They consider a graph readable or aesthetic when the user is both able to access detail information and to uncover general regularities and anomalies of the graph structure. Detail information may include the point in time and weights of edges, the specific nodes that are connected by an edge and paths. General regularities and anomalies could be clusters of vertices, outliers, trends, symmetries or patterns (Beck et al. 2009). They organized these aesthetic criteria into the categories “general criteria,” “dynamic criteria” and “scalability criteria.” As these criteria (Beck et al. 2009) were specifically tailored to dynamic graph visualizations and could therefore directly be applied, we chose to consider these for designing and evaluating the visualization.

For creating the visualization, we used an artificial social network with 98 phone calls between 26 individuals during a period of 6 consecutive days. The dataset contains private calls only, i.e., undirected connections between two persons. Each call lasts for a period of time, which is mapped as the intensity per connection, i.e., the call duration. The network connections are undirected, i.e., the dataset does not indicate who starts the conversation, but rather focuses on the duration of a call.

The main purpose of visualizing social networks is to be able to make sense of the social network that is represented within the data. For this reason, the dataset was created with special attention to its content in order to support storytelling, i.e., the data suggest the interpretation of circles of friends or families so that stories can unfold during the analysis.

The names of the 26 individuals each start with a letter from the alphabet to avoid confusing names and simplify reading the visualization. It is a common practice in social network analysis to prepare the dataset in a first step to assign distinct actor names, for example, aliases in crime analysis or a composition of surname and institution for scientific co-authorship network analysis. The dataset structure containing two actors per call, a numeric value representing different days and an intensity value is shown in Table 1. Each column represents a phone call and has an intensity assigned to reflect the length of the call in minutes.

In the user study, a different dataset was used. The underlying structure is the same, but the original dataset used for creating the visualization was limited to 20 actors and 56 phone calls within 7 days. The number of persons and, thus, the number of calls was reduced in order to keep the testing sessions within an appropriate time frame. The network size was still complex enough for realistic tasks to meet the goals of the study. The number of calls per day is evenly distributed showing 7–9 calls per day. The average duration of the phone calls is 17,13 minutes, with a minimum of 1 minute to a maximum of 44 min. Both datasets are provided in the "Appendix".

It is usually necessary to use an artificial dataset to cover all possible tasks for a user study. Real datasets are often messy and do not allow systematic testing.

The newly developed visualization for dynamic social networks uses a timeline-based approach, featuring characteristics of node-link diagrams and matrix-based representations in an attempt to leverage their respective advantages, i.e., showing individual links between actors and enabling an analysis in a structured way. The visual representation of the phone calls network used in the user study depicts each conversation on a daily basis, i.e., it shows precisely who talked to whom on which day and for how long, compare Fig. 1.

In this representation, the different points in time (in our case: days) are shown next to each other, i.e., they are juxtaposed. The phone calls are shown on the x-axis and consecutively numbered for identification (ID 1-98, see Fig. 2). The x-axis is labeled on the top with this ID as well as the day of the call, and on the bottom with intensities representing the duration of the call. Calls are assigned intensities, which are given below each link. The calls are sorted by intensity on a daily basis, i.e., the most intense (longest) calls are shown first in the segment for one specific day.

The y-axis lists all actor names in an alphabetical order. Vertical, blue lines symbolize a connection between two persons which are indicated by the connected dots along the horizontal line to their name on the y-axis. Additionally, the names of the pair are shown along the vertical connection line to facilitate identifying the connected individuals. The length of the blue lines has no meaning, it only connects two persons. To get an overview of the connections per person, the horizontal auxiliary lines which are kept in a light gray tone can be followed. When a user wants to know how many phone calls, e.g., Emily had on day 1, she will follow the horizontal gray line and notice that there are three blue dots that represent three calls (to Anna, Olivia and Daniel). The visualization enables users, for example, to see whether a person has frequent phone calls with another (see, e.g., the phone calls between Emily and Anna) and whether the intensity (duration) of these phone calls is similar or not. Users can refer to a specific connection via the presented IDs, which can be helpful when several analysts talk about their findings. The colors are taken from the default color palette of the visualization software described in the following section.

The visualization is in its early stages and interactivity is limited. Users can interact via zooming, scrolling and displaying additional information when hovering on connections. Hovering over a link shows the label of the actors, i.e., the names of the participating parties. The analysis might be more effective through the use of more advanced interaction opportunities and features such as filtering. To be able to show communications per person might be, for example, of interest. At the current stage, two views of this filter were explored and designed but not fully implemented. So far, the visualization also does not use color coding, or the thickness of lines or position encoding. It is, of course, possible to use such features to show additional variables. We kept the visualization simple for testing purposes to be able to validate the core functionality. In future work, we intend to develop a more complex visualization using all or most of the features discussed above.

The interactive visualization was created with Tableau, using an artificially constructed phone call dataset consisting of 26 individuals and six points in time, see Sect. 3.2. Figure 2 shows the stimuli used in the user study. Tableau includes a graphical system that allows users to explore and analyze their own datasets in a simple, quick and uncomplicated way, as described by Wesley et al. (2011). It is based on Polaris (Stolte et al. 2002) and has become a powerful addition to the technology stack of many organizations. Tableau enables users to create interactive visualizations using data from either an online data source or an offline copy of the data. The visualizations can either be saved locally offline or can be collaboratively shared on their server (Wesley et al. 2011).

We evaluated the proposed visualizations from two different points of view. On the one hand, we conducted a heuristic evaluation that tries to identify advantages and disadvantages through comparison with a theoretical model. In the context of the heuristic evaluation, we were also able to compare the proposed visualization to alternative solutions. We also conducted a user study to assess the usability and utility of the system. The specific strength of user studies is its ability to identify usability problems that never came up in a heuristic evaluation. Usability experts who conduct such evaluations are sometimes unable to anticipates problems encountered by inexperienced users. This makes usability studies especially valuable. On the other hand, it is difficult to compare different visualizations in the context of usability studies because tasks will always favor one or the other visualization. Therefore, heuristic evaluation and user study will complement each other because they have different strengths and weaknesses. We found that it is more useful to compare different visualizations using an heuristic evaluation and analyze the concrete interactions of users with the visualization in user studies.

Beck et al. (2009) influenced the design of the visualization proposed in this paper, but we also used their criteria for evaluation purposes. We evaluated whether the aesthetic criteria by Beck et al. (2009) were truly fulfilled in an heuristic evaluation of the visualization we developed. We further wanted to clarify whether individual criteria might contradict each other. Beck et al. (2009) used their criteria to compare three different visualizations qualitatively and to assess their respective advantages and disadvantages. We compared our visualization to two generic solutions for presenting social networks: node-link diagrams and matrices, see Figs. 3 and 4 respectively.

The research question we wanted to investigate in the heuristic evaluation was:

The heuristic evaluation was conducted by two usability experts who were also knowledgeable in the area of visualizations. One of the experts had 25 years of expertise, while the other expert had only a few years of expertise. They used the criteria from Beck et al. (2009) to evaluate the visualizations systematically. Both the evaluations were collated to provide a comprehensive overview of their analyses. In this process, it was possible to integrate the different points of view of the two experts. On the whole, the views did not differ considerably. The analysis distinguishes general from dynamic aesthetic criteria, as well as aesthetic scalability criteria. The heuristic evaluation allowed to compare the proposed visualization with other alternative kinds of visualization.

We conducted a small user study with a qualitative methods approach to test the usability of the visualization with regards to social network tasks and receive first feedback for improvements. The work of Lam et al. (2012) was taken as a basis for forming our research questions as well as for designing the user study. These allow the analysts to focus on analyzing the visualization itself, for example, testing its usability and making design decisions as well as the process and role of the visualization. As the primary goal of the study is to see how the visualization is accepted by users, the tasks were based on the two scenarios for evaluating user performance and user experience by Lam et al. (2012). Our research questions were as follows:

The visualization appeared to be less suitable for path-related tasks. Therefore, an additional visual representation, for example, a node-link representation which allows to easily find paths, could support the current visualization.

The day separator could be designed to be more prominent such as with a darker color or a thicker line as it was often overlooked.

The current version of the visualization uses three colors to differentiate the components. Adding more colors could provide a better differentiation between the different elements.

The vertically oriented text was hard to read for the participants and resulted in a poor reading performance. In order to avoid this issue, the text should be oriented horizontally which however would necessitate more spacing between the connections.