Diachronic Variation of Temporal Expressions in Scientific Writing Through the Lens of Relative Entropy

Key characteristics. Temporal expressions have three important key characteristics (cf. Alonso et al. (2011); Strötgen and Gertz (2016)). First, they can be normalized, i.e., expressions referring to the same semantics can be normalized to the same value. For example, March 11, 2017 and the 2nd Saturday in March of this year point to the same point in time, even though both expressions are realized in different ways. Second, temporal expressions are well-defined, i.e., given two points in time X and Y, the relationship between these two points can always be determined, e.g., as X is before Y (cf. Allen (1983)). Third, they can be organized hierarchically on a granularity scale (from coarser to finer granularities and vice versa such as day, month or year). Relevant in our analysis are normalization and granularity. Normalized values are used to compare temporal expressions across time periods instead of considering only the single lexical realizations. In terms of granularity, we consider granularity scales to determine diachronic changes of temporal expressions.

Types. According to the temporal markup language TimeML (cf. Pustejovsky (2005)), there are four types of temporal expressions (cf. also Strötgen and Gertz (2016)):

In the analysis, we consider all these four types showing how their use has changed diachronically in scientific writing.

For temporal tagging we use HeidelTime (Strötgen and Gertz 2010), a domain-sensitive temporal tagger. HeidelTime supports normalization strategies for four domains: news, narrative, colloquial, and autonomous. Although HeidelTime has been applied to process scientific documents using the autonomous domain, these scientific documents have been very specific, relatively short (biomedical abstracts) with many so-called autonomous expressions (i.e., expressions not referring to real points in time, but to references in a local time frame).

In contrast, our corpus is quite heterogeneous, containing letters and reports in the earlier time periods and full articles in the later time periods. Thus, we expect that most of the documents are written in such a way that the correct normalization of relative temporal expressions can be reached by using the document creation time as reference time. This makes the documents similar to news-style documents according to HeidelTime’s domain definitions. Thus, we apply HeidelTime with its news domain setting. Note, however, that in our analysis we use only normalized values of Duration and Set expressions, which are normalized to the length and granularity of an expression but not to an exact point in time. Thus, our findings still hold if some of the occurring temporal expressions are not normalized correctly to a point in time.

HeidelTime uses TIMEX3 tags, which are based on TimeML (Pustejovsky et al. 2010), the most widely used annotation standard for temporal expressions. In the following, we briefly explain the value attribute of TIMEX3 annotations of Duration and Set expressions, as we do consider their normalized values for a deeper analysis of the occurring temporal expressions. The value attribute of Duration and Set expressions contains information about the length of the duration that is mentioned, starting with P (or PT in case of time level durations) followed by a number and an abbreviation of the granularity (e.g., years: Y, month: M, week: W, days: D; hours: H, minutes: M). In addition, fuzzy expressions are referred to by X instead of precise numbers, e.g., several weeks is normalized to PXW, monthly is normalized to XXXX-XX and annually to XXXX.

For meaningful analysis and substantiated conclusions of temporal expressions in our diachronic corpus, the extraction (and normalization) quality of the temporal tagger should be reliable. Although HeidelTime has been extensively evaluated before on a variety of corpora², our corpus is quite different from standard temporal tagging corpora as it contains scientific documents from multiple scientific fields published across several centuries. Creation of a proper gold standard with manual annotations covering all scientific fields across all time periods would not be feasible in an appropriate time frame. Instead, for a valuable statement of temporal tagging quality on our corpus, determining the correctness of expressions tagged by the temporal tagger would be meaningful.

For this, we use precision, i.e., we randomly sample 250 instances for each time period, and manually validate whether the automatically annotated temporal expressions are correctly extracted.³ Here, we consider correctly extracted instances (right) and wrongly extracted instances (wrong). The latter are either cases of ambiguity (e.g., spring as ‘season’ or ‘water spring’ or current meaning ‘now’ or ‘electric current’) or wrongly assigned temporal expressions to numbers occurring in the text. On top we differentiate correctly assigned but not relevant instances (other) due to noise in the data itself. These are, e.g., temporal expressions assigned to reference sections (especially in the 1950–2000 periods) or used within tables (mostly in the earlier time periods).

Table 2 presents precision information and the number of instances per assigned category of right, other, and wrong. We consider the other instances to be correct in terms of precision of extraction. Across periods, precision achieves 0.89 to 0.96.

Comparing temporal types across fifty years time periods in terms of frequency (see Fig. 1 showing log of frequency per million (pM)), Date is the most frequent type, followed by Duration. Set and Time expressions are less frequent. In addition, while Date remains relatively stable over time, expressions of Duration, Set and Time drop quite a bit from 1850 onwards, getting relatively rare.

Inspecting diachronic change through the lens of relative entropy (as described in Sect. 5) allows us to consider temporal expressions typical of one time period when compared to the other time periods. We study each type of temporal expression and carefully select the base of comparison.

Date Considering Date expressions, instead of comparing single dates (which mostly occur only once in the corpus, such as June, 3, 1769), we take a level of abstraction and consider part-of-speech (POS) sequences of annotated Date expressions to better inspect the types of changes that might have occurred over time. For each Date expression, we extract POS sequences and use relative entropy to detect typical POS sequences of temporal expressions for each time period.

Table 3 shows POS sequences typical of one time period vs. 6-4 other time periods (see column comp.)⁴. For example, for 1650 the POS sequences Determiner-Noun (DT-NN) and Proper Noun (NP) are quite typical, which are all temporal expression referring to seasons in terms of lexical realizations (see Example 1). Both POS sequences are typical of 1650 vs. 1750 to 2000 (i.e., 5 comparisons). If we consider the POS sequences that are typical across time periods and their lexical realizations, there seems to be a development in terms of specificity and interval (see Fig. 2).

To capture the notion of specificity, we consider how many pieces of temporal information are given by a POS sequence to make a temporal expression most specific, with a scale from 1 to 4, where 1 is least specific (e.g., NP denoting seasons such as Winter as we do not know of which year etc.) and 4 is most specific (e.g., NP CD, CD such as June 3, 1769 which gives us an exact date)⁵. For comparison across time periods, Fig. 2 shows the average of the specificity count over all typical POS sequences of a time period (black line). For the interval of typical Date expressions, the amount of days⁶ the expressions refer to is used (shown in log in Fig. 2, gray line). The more specific an expression becomes, the smaller the interval it refers to and vice versa.

Figure 2 also shows how temporal expressions move from relatively unspecific (e.g., in the Spring in 1650) to very specific (June 18, 1784 in 1800) and back to unspecific expressions (e.g., the last decades in 2000). The interval moves instead from a wider to a smaller span and back to a wider span in 2000. Investigating the contexts, in which these expressions arise, gives further insights. While in the early time periods, season mentioning is typical, from 1800 to 1850, temporal expressions are typical with exact date, year or month expressions. These expressions are used to present exact dates of observations made by a researcher at several points in time, especially in the field of astronomy (see Example 1). From the 1950 onwards, typical Date expressions become less explicit, relating to broader (e.g., the 1970s in Example 3) and less specific (e.g., current literature in Example 3) temporal reference. These expressions are used, e.g., in the context of previous work descriptions in introduction sections of research papers.

Time To investigate typical Time expressions, similarly to Date expressions, we consider their POS sequences (see Table 4).

It can be seen that only the intermediate time periods show typical expressions (1750 to 1850). In terms of granularity, in the period of 1750, expressions are less granular pointing to broader sections of a day (e.g., morning, evening) mostly used to describe observations made (see Example 4). In the 1850 period, expressions point to specific hours of a day (e.g., 9 A.M.) mostly in descriptions of experiments.

Duration For Duration we consider their TIMEX3 value, as it directly encodes normalized information on the duration length and granularity of temporal expressions. Figure 3 shows typical TIMEX3 values (e.g., P1D for expressions such as one day) of specific time periods⁷. The y-axis shows the duration length in seconds on a log scale. In general, duration length gets lower from 1750 to 1850 (with expressions of seconds and hours, which are more granular) and higher in 1950 and 2000 (with expressions of decades, which are less granular).

We then again consider the contextual environments of these typical expressions. In the earlier time periods (1650 and 1700), day and year expressions are typical, mostly relating to observations or experiment descriptions (see Example 5).

From the period of 1750 to 1950, duration length is relatively low with expressions of seconds, minutes and hours being typical of these time periods. These expressions are mainly related to observations in the 1750 period and experiment descriptions from 1800 to 1950 (see Example 6).

In the 1950, besides weeks and minutes, related to experiment descriptions (see Example 7), expressions of decades are typical. The latter is also true for the 2000. In both periods, expressions relating to decades refer to previous work (see Example 7).

Set For Set expressions again their TIMEX3 value is considered. Figure 4 shows typical expressions with the times per year of a Set expression on the y-axis in log⁸, mirroring also less granular (annually) and more granular (every day) expressions. As we have seen from Fig. 1, Set expressions are relatively rare in scientific writing and strongly decrease over time (see Sect. 6.1). This is also reflected in the few temporal expressions typical of each time period in Fig. 4. In terms of granularity, there is a shift from day to month expressions (see Example 8). Interestingly, for the latter, there has been a move from a noun phrase expression (every/each month) to an adverb expression (monthly). While, in the intermediate periods (1800 and 1850) both expressions are typical, in 1950 only monthly is typical. In 1750 to 1850, every/each month expressions relate to observations done on a monthly basis of which the mean or average is drawn and the same applies for monthly used with mean as a term (see Example 8). In 1950, instead, monthly is solely used as an adverb. Thus, there is a replacement of longer noun phrase expressions (every/each month) by the shorter adverb expression monthly.