Ranking and grouping social media requests for emergency services using serviceability model

“Big Crisis Data” from social media has such a high volume, variety, and velocity, that it can overwhelm response services (Castillo 2016). Crisis informatics (Palen and Anderson 2016) has investigated the use of social media for emergency services. Quantitative approaches have focused on studying public behavior in specific emergencies while addressing problems of data collection and filtering, classification, summarization as well as visualization (Imran et al. 2015). Prior research has identified information overload as a key challenge and a barrier for the efficient use of social media communication by emergency services (Hiltz et al. 2014; Castillo 2016). Information overload originates from a variety of factors including the large scale, unstructured, and noisy nature of social media content. Furthermore, the characteristics of relevant social media requests that must be prioritized are not well understood.

In the emergency management domain, Public Information Officers (PIOs) play the role of serving information to the public or sourcing relevant information from public sources for the response agencies or an EOC, by leveraging various information communication technologies including social media (Hughes and Palen 2012). PIOs are provided guidelines for communication with the public (FEMA 2017), and have the responsibility to communicate critical information and respond promptly to requests. Over the last few years, PIOs have increasingly used social media to communicate effectively with the public. Reports and surveys of emergency services (US Homeland Security 2014, 2016; Reuter and Spielhofer 2017) recognize social media as a valuable information channel for improving their operational response coordination; however, they also recognize the necessity to effectively filter, prioritize, and organize information from this channel.

The literature provides some guidance on modeling requesting behavior or information seeking intent across different domains (Mai 2016), including Q&A forum (Vasilescu et al. 2014), email communication (Yang et al. 2017), and social media platforms (Ferrario et al. 2012; Sachdeva and Kumaraguru 2017; Purohit et al. 2013). In online fora, researchers have found request behavior in varied contexts such as urgency, informational intent, and social support. However, prior research on information seeking behavior is generic for all types of users and often not targeted toward seeking answers from a specific agency, organization, or group of organizations, as we focus in this study. In email communications (Yang et al. 2017), researchers explored the characteristics of ranking messages for replying and created predictive models for prioritization. However, the length of emails provides a greater context to express the request behavior, which does not apply to typically shorter social media messages.

The most relevant line of work for our analysis is request behavior on social media, which has been defined by researchers as ranging from explicit requests for organizational users (Sachdeva and Kumaraguru 2017) to implicit requests for seeking donations and resources (Purohit et al. 2013; Varga et al. 2013) and other actions (Zade et al. 2018; Ranganath et al. 2017) during disasters. In particular for explicit requests to agencies, Sachdeva et al. (Sachdeva and Kumaraguru 2017) defined requests to which police agencies should respond, evaluate, or take action as serviceable requests, by analyzing the messages on a Facebook page of a police department. Likewise, Ferrario et al. (Ferrario et al. 2012) analyzed the #bbcqt hashtag used for BBC Question Time (a current affairs discussion program broadcast on BBC One in the UK) to find actionable tweets. References (Purohit et al. 2013; Nazer et al. 2016; He et al. 2017) proposed methods to identify implicit request messages for seeking or offering help during disaster relief, however, not specifically targeted to emergency services. Ranganath et al. (Ranganath et al. 2017) created a method to identify users who can provide timely and relevant responses to actionable questions posted on social media, but not specific factors for organizational agency users. Recently Zade et al. (Zade et al. 2018) presented a survey-based study with practitioners on defining actionability of social media messages during disaster events. Although relevant to our concept of serviceability, the actionability of content is considered generically and thus, it is not specific to requesting help and seeking response by the emergency services. To complement the prior research on social media for request behavior, we focus on creating a generalizable model for serviceability characteristics of requests targeted to organizational emergency services.

Prior research in Information Retrieval has explored various ways to organize search ranking results and enable users to better interact with information and to reduce the cognitive effort of scanning through a long list (Wang and Zhai 2007; Osiński and Weiss 2005). \(Carrot^2\)¹ is an example of a retrieval system for web search that groups web results under different topical categories (Osiński and Weiss 2005). Recent works (Wasilewski and Hurley 2016; Cobos et al. 2014) have continued exploring semantic grouping of ranked items for the different ways of incorporating diversity in ranking. However, to the best of our knowledge previous work has not investigated the grouping and ranking of short, informal language text of social media messages, especially in a time-sensitive scenario like disaster events.

We consider a general class of emergency service requests, following official guidelines from the US FEMA (Federal Emergency Management Agency).

 FEMA (2017), which include intended actions such as a request for resources (e.g., emergency medical assistance for an injured person) as well as information (e.g., a request for a phone number to get information on missing people). The key characteristic of a serviceable request message is that it requests a resource that can be provided, or asks a question that can be answered. For instance, we do not consider messages whose sole purpose is to congratulate/praise or complaint as serviceable; in our framework, a serviceable message must contain an explicit request for resources or a concrete question.

The serviceability of a social media message is also determined by whether it is correctly addressed to an organization that can provide the resource or information. Most social media platforms include features for sending publicly or privately a message addressed to a specific user. Thus, a citizen seeking an action or answer from an organization can address the request to that organization’s account.

We note that each organization or agency usually has its own protocols to determine if and how a request should be answered. However, the knowledge of such protocols is acquired by the service personnel through training guidelines, and may remain in the form of tacit knowledge instead of structured knowledge that could be used to automate responses.

Finally, serviceability not only refers to the topicality of the request and to the fact that it must be addressed correctly, but also to whether it contains required details, such as time, place, or context. In summary, we propose the following definition of a request on social media with the serviceability characteristics.

Our definition 1 describes an ideal serviceable message, but serviceability is a matter of degree. To quantify this, we associate a score to each of the three types of serviceability characteristics for a given message m, for instance by using a 5-points Likert Scale (Likert 1932):

Our quantitative serviceability model defines a message m as a function of these characteristics f(E(m), A(m), C(m), D(m)). In this study, we automatically learn this scoring function instead of manually providing it, by estimating a relevance function learned via a Learning-to-Rank (Liu 2009) algorithm. We describe its implementation to rank messages next.

We first collected data from Twitter for six disaster events from the last 6 years, using the keyword-based crawling approach. We collected tweets during hurricane Harvey in 2017 and Louisiana floods in 2016 using the CitizenHelper system (Karuna et al. 2017) and for prior events, we re-used datasets available from previous works (Sheth et al. 2014; Imran et al. 2016). Following the recommendation of collecting “contextual streams” (Palen 2014), we further extended each event collection with messages that belonged to conversation chain (a Reply message thread on Twitter), where a conversation chain contained at least one message from an event dataset. To collect such conversation chain messages, we “scrapped” web pages of conversations using tweet id in each of our event datasets (this allows to recover more public messages than using Twitter’s API, which does not provide conversation chains). Specifically, the conversation chain for tweet with id TWEETID authored by a user with handle USER is available at http://​twitter.​com/​USER/​status/​TWEETID. Table 2 shows a summary of the characteristics of our dataset.

We asked crowdsourcing annotators for rating the individual serviceability characteristics of a message, as we describe in this section. We also requested domain practitioners for the ground truth annotation of the overall serviceability of a message (described in the next section.)

For rating individual serviceability characteristics, we provided instructions and examples to the crowdsourcing annotators (specifically, university student volunteers) based on the model described in Sect. 3. Given a message, three annotators associated a numerical rating between 1 and 5 to each serviceability characteristic. We also solicited the rating on an additional attribute of the message to indicate non-serviceable aspects such as complaints, gratitude, congratulations, and advertisements because these are not a priority for operational response. For example, through this message “@account1 wow that photo made me tear up! Just amazing what u do! #yycflood”, a user is only trying to praise and express gratitude toward the officials. It is not asking for any operational resource or requesting any information. We provided the following annotation task description:

For the annotation task on the conversational dataset of an event, we selected a biased set of messages using the following two equal-sized samples, in order to increase the recall of potential serviceable requests. The first sample selects all the messages that were directly addressed or targeted to official accounts (i.e., that start with ‘@account’) and that were posted in a conversation chain before an official reply was posted. The second sample randomly selects messages from the remaining event dataset after excluding the first sample and the messages authored by official accounts. We collected the set of official accounts of relevant response organizations for an event through the official reports, including those from FEMA and news sources. For example, @account in the examples of Table 1 is the Twitter user account of the emergency management unit of a county government responding to hurricane Harvey. (The target official accounts of each event are provided in our data release.)

After execution of the event-specific annotation task by three annotators per message, we computed the average ratings of characteristics per message. Table 3 shows examples of messages with average ratings.

To validate our model for serviceability, we required a gold standard set of serviceable requests. The set was designed with the help of domain experts by labeling the annotated message set from the previous section. For this task, we asked domain experts to label if a request message would qualify as serviceable or not serviceable, according to their experience. We also provided them an optional field for entering comments on their choice of label. Our domain experts are three active professionals in the emergency management domain located in the USA, Canada, and Nepal, who have had roles in the public communications in a response agency. Specifically, the US-based expert labeled messages from Harvey, Louisiana, Oklahoma, and Sandy events, the expert based in Canada labeled the Alberta event, and the expert based in Nepal labeled the Nepal earthquake event.

Table 2 summarizes the resultant label distribution from the three domain experts. The comments they provided during their label annotation process show some insights.

First, experts in general considered that serviceable messages were a subclass of the messages they would respond to during a disaster; in other words, that they would not answer only to requests for actions or information, but sometimes, also to other classes of messages, for example M8 in Table 3.

Second, a disagreement between the experts was observed in relation to messages that only express gratitude (such as M3 in Table 1). One of the three experts considered such messages as serviceable, with the rationale that replying to gratitude messages could help strengthen trust within a community. In contrast, the other two experts considered gratitude messages as not serviceable, as they did not consider them a priority for operational response like the other actionable messages. We sided with the majority opinion and resolved to keep the gratitude messages under the not serviceable category.

Third, experts identified that in some cases a message should be answered to provide reassurance or restate facts. This was considered a good strategy to counter rumors, in particular, highly alarming or easily falsifiable ones.

Overall, we found a correlation between the gold standard annotations of serviceability by experts and the average ratings by non-expert crowdsourcing workers for the proposed serviceability characteristics, as shown in Fig. 2.

For filtering and prioritizing serviceable requests, we propose a supervised learning method for automatic classification and ranking. In automatic classification, the objective is to classify a message as either serviceable or not serviceable. In automatic ranking, the objective is to order a list of messages according to how serviceable they are. The Learning-to-Rank methodology (Liu 2009) is suitable for jointly meeting these objectives. This method could use any kind of relevance levels of serviceability for training purpose. In our experiments, we have considered only binary levels, but we remark the method is general. Figure 1 depicts how learning-to-rank fits within this process, as we explain next.

Formally, we consider that each event \(i = 1, 2, \ldots , m\) determines a query \(q^i\), and \(D^i = \{ d^i_{1}, d^i_{2}, \dots , d^i_{n\_i} \}\) are all the messages relevant to that event, which need to be ranked. Each message is associated to a label in \(Y = \{ y_1, y_2, \ldots , y_\ell \}\) representing its level of serviceability. (A total order between the graded levels exist.) The training set corresponds to tuples \(\left\langle q^i, D^i, Y^i \right\rangle \) containing queries, documents for each query, and sets of labels for each document, with \((Y^i)_{j}\) indicating the label for document \(d^i_{j}\).

In this context, the goal of a learning-to-rank method is to learn a ranking model that associates to each query \(q^i\) a permutation of the documents in \(D^i\) that matches as much as possible the training labels, in the sense that higher graded labels in Y are associated to documents appearing near the top of the ranking (being more serviceable). In particular, we consider learning-to-rank models characterized by functions \(f_i: D^i \times D^i \rightarrow \{ -1, +1 \}\) that associate to each pair of documents \((u,v) \in D^i\) a score \(-1\) if u should be ranked above v for \(q^i\) and \(+1\) otherwise. Specifically:To solve this problem, we use the SVM-Rank algorithm (Joachims 2006).

Feature extraction Query-document pairs \((q^i, d^i_{j})\) are represented by feature vectors, which include the following:

We create two types of clusters of the individual ranked requests:

The fine-granularity clustering method represents each request message using the bag-of-words model after preprocessing text of requests. We use TF-IDF weighing scheme for a word feature and compute cosine distance between two word vectors of request messages for the clustering process (Baeza-Yates et al. 2011). Our preprocessing replaces URLs, numbers, and user mentions by generic tokens, and removes stopwords using a dictionary plus the top 3% frequent words in the vocabulary. We used hierarchical clustering with no predefined criterion on the desired number of clusters.

We perform the coarse-granularity clustering using a distributed semantic representation of words as vectors (a word embedding), specifically, the pre-trained embeddings of Wikipedia concepts and entities (e.g., ConVec Sherkat and Milios 2017.) We compute the cosine distance-based similarity to match the embedding representations of a request message (averaging over message tokens’ embeddings) and the concept of the relevant top-level category from the DBpedia ontology as shown in Table 4. Thus, this approach can facilitate a top-down faceted browsing of requests under various ontological sub-categories. We determine a request item’s membership to a concept-category-based cluster by the highest matched concept category. Relevant concept categories such as Medicine or MeanOfTransportation are determined based on the framework of well-known Incident Command System (ICS) (FEMA 2017) used by emergency management practitioners. We consider the set of requests under the same concept category as a cluster, which can have sub-clusters.

The advantage of the resulting clustered messages using such hierarchical approach is that we can further navigate the top-level concept cluster for sub-categories in the hierarchy of the predefined domain ontology, or the dendogram of the fine-granularity clusters. The flexibility for faceted navigation is useful for emergency responders with different roles for response (e.g., officers responding to transportation queries and within that, road vs. rail transport).

For each of the clustering approaches, we retrieve clusters that have different sizes and that involve messages with different relevance scores from the individual ranking method described in the previous section. Our aim is to rank the clusters, where the cluster importance is measured using the relevance rank of individual request messages contained in the cluster.

The importance function denoted by \(score\_rank(c_k)\) for a cluster \(c_k\) is computed by following Borda Count (Yang and Guo 2016), which is a preference aggregation method where scores are given to each candidate cluster in reverse proportion to their ranking. Thus, the higher-ranked candidates receive more points and we sort candidate clusters by \(score\_rank(c_k)\) values, defined as:where \(c_k\) refers to kth cluster with individual request members \(d^i_{j}\),

\(MAX\_RANK\) denotes the maximum rank a cluster could achieve (i.e., the total number of clusters), and \(rank(c_k)\) function provides the representative position of the cluster \(c_k\) among the candidate clusters. This is estimated by ordering the best relevance scored member (representative) of each of the candidate clusters (i.e., the member \(d^i_{j}\) with the highest relevance score in the individual serviceability ranking among the request members of \(c_k\)).

Table 5 compares the performance of different ranking model schemes described in Sect. 6, in terms of nDCG values of the first 5 positions \((nDCG\hbox {@5})\) and 10 positions \((nDCG\hbox {@10})\) of ranking. For a qualitative analysis of the resulting ranking systems, we also present top-2 and bottom-2 ranked items from T+I+S method in Table 6. We observe the following from these tables:

We first study the distribution of resulting clusters from our two methods of fine- and coarse-granularity in Figs. 3 and 4, followed by the analysis of cluster-wise group ranking performance in Fig. 5 and Table 7. We note the following observations:

While our presented method can prioritize and group serviceable messages, it has some limitations.

First, the analyzed data contain only English language messages, which is the dominant language in the data collections we use. However, we anticipate the core information characteristics of serviceability for the messages to be the same, or similar, across messages written in other languages. Similarly, we note that all our dataset come from the same platform, i.e., Twitter, and we would need to evaluate messages in other platforms where populations and norms may be different and hence, serviceability might differ.

Third, we focused on the semantic grouping of requests based on textual content, while there can be also grouping by spatial and temporal semantics. We plan to explore the ensemble of diverse semantic grouping approaches in a future study.

Fourth, there is a possibility of temporal significance of relevancy for the posting time when a request was posted, such as when the hurricane landfall occurs. However, in this research, the data collection was focused on the period of active response to the emergency. This translates to the observation from our temporal analysis that the percentage of relevant messages per day never fell below a certain threshold in an event. Thus, a system based on this method would require some pre-filtering of relevant messages, especially if applied outside the active response period. A future work could explore this direction further for temporal significance of requests to develop a generic social media filtering and ranking system for all phases of the disaster management.

Fifth, there is no availability of real-time data for the actual social network structure, which could be leveraged for generating more features based on social network characteristics to benefit the real-time serviceability ranking applications.

Sixth, there is a likelihood of a user sending emotional messages for requesting help, or using certain patterns of vocabulary that is used by people under extreme stress and can be a part of conversational context. Therefore, features capturing such subjectivity could be explored for ranking serviceable requests in the future work.

Anonymized messages, a list of official accounts, the crowdsourced and expert labels as well as codes are available at: http://​shorturl.​at/​hnqCH.