Word statistics in Blogs and RSS feeds: Towards empirical universal evidence

Abstract

We focus on the statistics of word occurrences and of the waiting times between such occurrences in Blogs. Due to the heterogeneity of words’ frequencies, the empirical analysis is performed by studying classes of “frequently-equivalent” words, i.e. by grouping words depending on their frequencies. Two limiting cases are considered: the dilute limit, i.e. for those words that are used less than once a day, and the dense limit for frequent words. In both cases, extreme events occur more frequently than expected from the Poisson hypothesis. These deviations from Poisson statistics reveal non-trivial time correlations between events that are associated with bursts of activities. The distribution of waiting times is shown to behave like a stretched exponential and to have the same shape for different sets of words sharing a common frequency, thereby revealing universal features.

Introduction

Web logs, also known as Blogs, have become an influential medium Hammersley, 2005, Glance et al., 2004, Thelwall et al., 2006, that encompasses a broad variety of subjects, e.g. politics and science, and are participative by nature. They involve a huge number of interacting users that belong to several layers of the population, from topic specialists to average people. This variety suggests that Blogs could be an efficient information source for identifying, tracking and modeling the spread of ideas and opinion formation, for example in public debates over political questions. Indeed, the democratic nature of Blogs allows us to examine how trends develop from the interactions of decentralized bloggers and to follow dynamic opinion changes over a wide and diverse sample of the population. This is in contrast with the main media where relatively few journalists are involved. Precise knowledge of word statistics in Blogs is consequently of interest in order to make coherent statistical tests for automatically detecting critical events, e.g. trends or media shocks Kleinberg, 2002, Kleinberg, 2008.

The most basic time statistics ignoring correlations between events can be modeled by Poisson distributions. This distribution concerns independent events: the number n of events arriving during some time interval $\Delta$ occurs with a probability

$$P(n|a)=\frac{a^n}{n!}{\text{e}}^{-a}\text{,}$$

where a is the arithmetic average number of events during this time interval. Moreover, the distribution of waiting times between two successive Poisson events is the negative exponential:

$$f(\tau )={\tau }_{\text{c}}^{-1}\exp \left(-\frac{\tau }{{\tau }_{\text{c}}}\right)\text{,}$$

where ${\tau }_{\text{c}}=\Delta /a$ is the average characteristic waiting time between events. This distribution is well-known to apply to nuclear disintegration but it has also been used for describing the time gaps between shoppers entering a store (Kan & Fu, 1997), the number of failure of products (Gregory, 2005), the number of terrorist acts (Telesca & Lovallo, 2006) as well as the number of airplane accidents as a function of time (Ausloos & Lambiotte, 2006a). An increasing amount of empirical evidence indicates, though, that human activity patterns do not fit this model. It has been shown by many other authors that human processes are rather heterogeneously distributed in time, with short periods of high activity Kleinberg, 2002, Kleinberg, 2008, Willinger and Paxson, 1998, or bursts, separated by long periods of inactivity Barabási, 2005, Vázquez et al., 2006, Dewes et al., 2003, Paxson and Floyd, 1995, Dezsö et al., 2006, Vázquez, 2005, Ebeling and Neiman, 1995, Gopikrishnan et al., 2001, Sabatelli et al., 2002. This heterogeneity is characterized by a distribution of waiting times which deviates from the exponential (2) and which, usually, presents a so-called heavy tail.

In this paper, we focus on the statistics of such waiting times between word occurrences in Blogs (and other similar periodically updated web sources) and also on the statistics of the number of word occurrences per day. To do so, we focus on texts published in 68022 RSS feeds during a period of 214 days and analyze two limiting cases. On one hand, we focus on very rare “events”, namely words that occur on average less than once per day. It is shown that the frequency of words is very heterogeneous, so that the time statistics have to be measured in classes of “frequently-equivalent” words, i.e. words are discriminated through their total number of occurrences during the whole time period. This discrimination allows us to show that the distribution of waiting times deviates from the exponential (2), i.e. it is fitted by a stretched exponential and therefore presents an overpopulated tail. The deviation from the pure exponential is evaluated with the quantity $\zeta$ that measures the importance of the second moment of the time statistics. Interestingly, it is found that the shape of the distribution as well as the value of $\zeta$ do not depend on the class of words in which they are measured. On the other hand, we focus on events that occur many times per day on average. In that case, scaling laws are applied in order to smoothen the empirical results. Deviations from the Poisson statistics (1) are also found. Consequently, our results not only confirm that the dynamics of topics in Blogs present bursts of activity Kleinberg, 2002, Kleinberg, 2008, Willinger and Paxson, 1998 but they also provide tools in order to measure the importance of such bursts by comparing the empirical word statistics to a Poisson uncorrelated process.