1 Introduction
mod
attribute that allows the modification of expressions, but only in a very constrained way (12 preset non-disjoint modifiers). In order to overcome this lack, existing approaches [20, 32, 38, 39] use fuzzy sets to represent individual timexes and relations. However, they describe specific historical events or generic periods of time (e.g. holidays), relying on external sources of data, such as the result of Internet search queries or image timestamps collected from social media, and they do not provide a generic or reusable methodology for the normalisation of imprecise timexes. In these situations, the normalisation is done based on the extracted time spans, which are often focused on one kind of expression and with restricted interpretation of the timexes, being difficult to be applied to broader domains.2 Background and related work
2.1 Temporal information extraction
TIMEX3
tags [7, 18]. In Llorens et al. [30], authors use the argument that temporal expression normalisation can only be effectively performed with a large knowledge base and set of rules.2.2 Normalisation of temporal expressions
Category | Examples |
---|---|
Temporal units | Day, month, year |
Temporal modifiers | Last, previous, next |
Temporal quantifiers | Several, few |
Temporal directions | Ago, further, later |
Temporal approximators | Almost, about |
Day names | Monday, Tuesday |
Month names | January, February |
Cardinal numbers | One, 1, two, 2 |
Ordinal numbers | First, 1st, second, 2nd |
Coreference timex | Period, time |
Fixed timex | Today, yesterday, now |
2.3 Imprecise temporal representation
TIMEX3 value=“P1M” mod=“LESS_THAN”
>. As a consequence, when interpreting this expression and its annotated features, it is not clear whether we should consider each possible number of days between 0 and 30 as equally likely, or whether for example, 20–25 days ago is more likely than 5–10 days ago or even “yesterday”.3 Imprecise temporal data in text
3.1 Quantifying imprecise timexes
Corpus | Lang | Docs | Description |
---|---|---|---|
AQUAINT | En | 73 | News reports, also referred to as the Opinion Corpus, annotated with time expressions [36] |
TE3 Platinum | En | 20 | The corpus used to rank participant systems in the TempEval-3 evaluation exercise, consisting of newswire documents and blog posts annotated for events, time expressions, and relations [46] |
TE3 Silver | En | 2452 | Documents automatically annotated as a silver standard in TempEval-3 [46] |
TimeBank | En | 183 | News articles annotated with temporal information, events, times, and temporal links between events and times [35] |
WikiWars | En | 22 | Documents sourced from Wikipedia, within the domain of military conflicts, containing timex annotated with TIMEX2 [31] |
CSTNews4 | Pt | 50 | A discourse-annotated corpus for fostering research on single- and multi-document summarisation from news texts [13] |
THYME\(^\mathrm{a}\) | En | 248 | Clinical narratives data sets used in SemEval-2015 Clinical TempEval Task [6] |
SLAM\(^\mathrm{a}\) | En | 1000 | Medical records without any pre-annotated timexes provided by the Biomedical Research Centre and Dementia Biomedical Research Unit at South London and Maudsley NHS Foundation Trust and King’s College London [40] |
InfoSaude\(^\mathrm{a}\) | Pt | 3360 | Medical records without any pre-annotated timex extracted from the InfoSaude system, Public Health Department in Brazil [11] |
Total number of timexes | Imprecise timexes | Imprecise (%) | |
---|---|---|---|
(a) Non-clinical corpora | |||
AQUAINT | 463 | 35 | 7.6 |
TE3 Platinum | 158 | 20 | 12.7 |
TE3 Silver | 15,191 | 863 | 5.7 |
TimeBank | 478 | 60 | 12.6 |
WikiWars | 862 | 112 | 13.0 |
CSTNews4 | 444 | 32 | 7.2 |
Total (micro) | 17,596 | 1122 | 6.4 |
Total (macro) | 9.8 | ||
(b) Clinical corpora | |||
Thyme | 3358 | 659 | 19.6 |
SLAM | 35,120 | 12,226 | 34.8 |
InfoSaude | 503,005 | 53,830 | 10.7 |
General | 134,388 | 13,785 | 10.3 |
Gynaecology | 66,021 | 5452 | 8.3 |
Nutrition | 64,282 | 6286 | 9.8 |
Psychiatry | 238,314 | 28,307 | 11.9 |
Total (micro) | 541,483 | 66,715 | 12.3 |
Total (macro) | 21.7 |
Temporal granularity | Non-clinical corpora (%) | Clinical corpora (%) |
---|---|---|
Year | 28.5 | 21.1 |
Month | 20.1 | 21.2 |
Week | 7.7 | 6.8 |
Day | 10.7 | 17.6 |
Time (hour, minute and second) | 4.9 | 2.8 |
Undefined | 23.8 | 15.2 |
Others\(^\mathrm{a}\) | 4.3 | 15.3 |
Corpus | DATE | TIME | ||||
---|---|---|---|---|---|---|
Tot | Imp | % | Tot | Imp | % | |
(a) Classes DATE and TIME | ||||||
THYME | 2588 | 460 | 17.8 | 118 | 13 | 11.0 |
SLAM | 22,678 | 9296 | 41.0 | 919 | 27 | 2.9 |
SMS | 210,596 | 19,082 | 9.1 | 63,468 | 71 | 0.1 |
General | 59,835 | 4838 | 8.1 | 15,530 | 11 | 0.1 |
Gynaecology | 33,965 | 1642 | 4.8 | 3996 | 4 | 0.1 |
Nutrition | 23,324 | 1969 | 8.4 | 8444 | 15 | 0.2 |
Psychiatry | 93,472 | 10,633 | 11.4 | 35,498 | 41 | 0.1 |
Avg (micro) | 235,862 | 28,838 | 12.2 | 64,505 | 111 | 0.2 |
Avg (macro) | 22.6 | 4.7 |
Corpus | DURATION | SET | ||||
---|---|---|---|---|---|---|
Tot | Imp | % | Tot | Imp | % | |
(b) Classes DURATION and SET | ||||||
THYME | 434 | 150 | 34.6 | 218 | 36 | 16.5 |
SLAM | 8001 | 2801 | 35.0 | 1558 | 102 | 6.5 |
SMS | 190,411 | 34,524 | 18.1 | 38,530 | 153 | 0.4 |
General | 49,829 | 8900 | 17.9 | 9194 | 36 | 0.4 |
Gynaecology | 24,088 | 3783 | 15.7 | 3972 | 23 | 0.6 |
Nutrition | 26,933 | 4285 | 15.9 | 5581 | 17 | 0.3 |
Psychiatry | 89,561 | 17,556 | 19.6 | 19,783 | 77 | 0.4 |
Avg (micro) | 198,846 | 37,475 | 18.8 | 40,306 | 291 | 0.7 |
Avg (macro) | 29.2 | 7.8 |
3.2 Classification of imprecise timexes
Imprecise type | Clinical corpora | ||
---|---|---|---|
THYME | SLAM | InfoSaude | |
PR | 55.7% | 58.0% | 30.2% |
MV | 15.5% | 6.6% | 27.0% |
IV | 11.9% | 14.4% | 24.9% |
RV | 10.2% | 4.0% | 13.6% |
PP | 6.2% | 3.2% | 4.3% |
GE | 0.5% | 13.8% | 0.0% |
Total | 659 | 12,229 | 53,830 |
4 Normalisation of imprecise timexes
4.1 Specification of the input data
Imprecise type | Question type | #Questions | #Answers (avg) | Fleiss’ kappa agreement | |||
---|---|---|---|---|---|---|---|
Port | Eng | Port | Eng | Port | Eng | ||
MV | Approximately | 30 | 38 | 70.4 | 88.7 | 0.329 | 0.322 |
Less than | 18 | 26 | 71.3 | 88.8 | 0.285 | 0.324 | |
More than | 24 | 26 | 70.5 | 89.5 | 0.248 | 0.347 | |
IV | Imprecise value | 41 | 48 | 70.2 | 89.4 | 0.198 | 0.201 |
PR | Present reference | 12 | 12 | 69.4 | 91.2 | 0.321 | 0.427 |
Total | 125 | 150 | 70.3 | 89.3 | 0.268 | 0.297 |
4.2 Membership functions
4.3 Normalisation models
Parameter | Description (value) |
---|---|
(a) Features | |
Granularity | Four input values to set the temporal granularity —“Val” variation |
Reference value | Number extracted for MV expressions —“Val” variation |
Reference days | Number of days extracted for MV expressions —“Day” variation |
Temporal context | Number of days that represents the temporal context – IV expressions |
Imprecise value | Five input values to set the imprecise value—IV expressions |
(b) Training parameters | |
maxIteration | Maximum number of training iterations to be performed (5000) |
minIteration | Minimal number of iterations to be performed before stopping (1000) |
maxNoBetter | Training stops after 200 iterations with no improvement (200) |
K
| Number of folds in K-fold cross-validation (4) |
learningRate | Learning rate used by the backpropagation algorithm (0.05) |
(c) MLP layers | |
hiddenLayer | Number of neurons in the hidden layer(\((inputLayerSize-1)*(outputLayerSize-1)\) ) |
outputLayer | Number of neurons in the hidden layer to produce trapezoidal MSFs (4) or trapezoidal MSFs (6) |
MANY
, for example, the range value [6, 8] is equivalent to a MSF(x, [5, 6, 8, 9]).5 Evaluation
5.1 Modified value (MV) expressions
Method | Var | Portuguese | English | ||||
---|---|---|---|---|---|---|---|
\(M_4\)
|
\(M_6\)
| Avg |
\(M_4\)
|
\(M_6\)
| Avg | ||
Baseline | 0.673 | 0.646 | 0.660 | 0.741 | 0.731 | 0.736 | |
Regression | Lin(A) | 0.635 | 0.558 | 0.597 | 0.615 | 0.613 | 0.614 |
Regression | Lin(0) | 0.762 | 0.740 | 0.751 | 0.797 | 0.794 | 0.796 |
Regression | Log(A) | 0.772 | 0.746 |
0.759
| 0.814 | 0.806 |
0.810
|
Regression | Log(0) | 0.669 | 0.661 | 0.665 | 0.678 | 0.693 | 0.686 |
MLP | Day/Lin | 0.321 | 0.584 | 0.452 | 0.340 | 0.514 | 0.427 |
MLP | Day/Log | 0.729 | 0.755 | 0.742 | 0.679 | 0.786 | 0.733 |
MLP | Val/Lin | 0.785 | 0.742 |
0.763
| 0.738 | 0.787 | 0.763 |
MLP | Val/Log | 0.757 | 0.738 | 0.747 | 0.760 | 0.774 |
0.767
|
p value=0.000481
when comparing the results between regression-Log(A) and MLP-Val/Log approaches; (b) in Portuguese, p value=0.003243
when comparing the results between regression-Log(A) and MLP-Val/Lin approaches. In addition, we also compared the results between regression-Lin(0) and regression-Log(A), from which we found no significant differences for both languages (p value=0.183702
for English; p value=0.314776
for Portuguese. The Lin(0) variation does not rely on logarithmic transformations, and this model can be directly calculated by applying simple linear transformations on the input imprecise expression.
Modifier | Pt | En |
---|---|---|
\([B_p,B_r,B_s,B_v]\)
|
\([B_p,B_r,B_s,B_v]\)
| |
Approx(\(N_{tgran}\)) | [0.7185, 0.9375, 0.9964, 1.2335] | [0.7101, 0.9325, 1.0602, 1.2965] |
LessThan(\(N_{tgran}\)) | [0.6921, 0.8290, 0.8554, 0.9888] | [0.3693, 0.7964, 0.9371, 1.0803] |
MoreThan(\(N_{tgran}\)) | [0.9705, 1.2111, 1.2605, 1.4995] | [0.8799, 1.0704, 1.2036, 1.7093] |
5.2 Imprecise value (IV) expressions
Method | Var | Portuguese | English | ||||
---|---|---|---|---|---|---|---|
\(M_4\)
|
\(M_6\)
| Avg |
\(M_4\)
|
\(M_6\)
| Avg | ||
Baseline | 0.325 | 0.311 | 0.318 | 0.318 | 0.298 | 0.308 | |
Mean | Day/Lin | 0.661 | 0.847 | 0.754 | 0.866 | 0.847 |
0.857
|
Day/Log | 0.657 | 0.848 | 0.753 | 0.867 | 0.846 | 0.856 | |
Val/Lin | 0.669 | 0.850 |
0.760
| 0.859 | 0.845 | 0.852 | |
Val/Log | 0.656 | 0.847 | 0.751 | 0.844 | 0.831 | 0.837 | |
Regression | Day/Lin | 0.660 | 0.850 | 0.755 | 0.892 | 0.871 | 0.881 |
Day/Log | 0.660 | 0.846 | 0.753 | 0.884 | 0.868 | 0.876 | |
Val/Lin | 0.673 | 0.858 |
0.765
| 0.889 | 0.877 |
0.883
| |
Val/Log | 0.668 | 0.841 | 0.755 | 0.847 | 0.848 | 0.848 | |
MLP (granularity) | Day/Lin | 0.610 | 0.779 | 0.695 | 0.792 | 0.827 | 0.809 |
Day/Log | 0.694 | 0.728 | 0.711 | 0.849 | 0.814 | 0.831 | |
Val/Lin | 0.820 | 0.767 |
0.793
| 0.848 | 0.832 | 0.840 | |
Val/Log | 0.751 | 0.726 | 0.738 | 0.848 | 0.819 |
0.834
| |
MLP (imprecise value) | Day/Lin | 0.626 | 0.582 | 0.604 | 0.760 | 0.757 | 0.759 |
Day/Log | 0.712 | 0.551 | 0.632 | 0.862 | 0.843 |
0.853
| |
Val/Lin | 0.784 | 0.738 | 0.761 | 0.821 | 0.811 | 0.816 | |
Val/Log | 0.762 | 0.766 |
0.764
| 0.841 | 0.762 | 0.802 |
p value=0.075434
when comparing the mean Day/Lin and the regression (Val/Lin) approaches; (b) in Portuguese, p value=0.199188
when comparing the mean Val/Lin and the MLP-Val/Lin (granularity) approaches. The main advantage of the mean approach refers to the fact it can be applied independently of an input value or temporal context.
5.3 Present reference (PR) expressions
Modifier | Granularity | Val/Lin[p, r, s, v] | Day/Lin [p, r, s, v] |
---|---|---|---|
(a) Portuguese | |||
<none> | Days | [1, 2, 3, 10] | |
Weeks | [1, 3, 3, 7] | [9, 18, 22, 50] | |
Months | [1, 2, 4, 11] | [39, 65, 110, 312] | |
Years | [1, 3, 10, 14] | [485, 1259, 3577, 4802] | |
Few | Days | [1, 2, 4, 14] | |
Weeks | [1, 2, 3, 7] | [8, 12, 23, 50] | |
Months | [1, 2, 3, 7] | [43, 58, 96, 198] | |
Years | [1, 2, 3, 6] | [183, 588, 1164, 2274] | |
Some | Days | [1, 2, 5, 14] | |
Weeks | [1, 3, 3, 6] | [6, 18, 23, 44] | |
Months | [1, 2, 4, 9] | [38, 65, 129, 250] | |
Years | [1, 2, 5, 8] | [345, 671, 1681, 2789] | |
Many | Days | [3, 5, 11, 30] | |
Weeks | [1, 3, 3, 9] | [9, 19, 23, 64] | |
Months | [3, 6, 8, 21] | [76, 195, 254, 630] | |
Years | [1, 10, 13, 16] | [356, 3784, 4528, 5764] | |
Several | Days | [2, 8, 12, 28] | |
Weeks | [1, 3, 3, 9] | [9, 19, 23, 64] | |
Months | [1, 3, 5, 22] | [57, 93, 128, 663] | |
Years | [1, 3, 10, 16] | [596, 1029, 3428, 5849] | |
(b) English | |||
<none> | Days | [1, 3, 5, 14] | |
Weeks | [1, 2, 4, 11] | [7, 18, 30, 73] | |
Months | [1, 3, 5, 10] | [26, 92, 134, 296] | |
Years | [1, 3, 4, 16] | [239, 1125, 1498, 5550] | |
Few | Days | [1, 2, 4, 8] | |
Weeks | [1, 3, 3, 9] | [9, 20, 25, 59] | |
Months | [1, 3, 4, 7] | [25, 80, 107, 205] | |
Years | [1, 3, 4, 8] | [315, 993, 1304, 2806] | |
Some | Days | [1, 3, 6, 29] | |
Weeks | [1, 2, 3, 6] | [6, 18, 22, 45] | |
Months | [1, 2, 4, 11] | [27, 72, 120, 310] | |
Years | [1, 3, 5, 13] | [235, 1134, 1776, 4675] | |
Many | Days | [2, 5, 13, 37] | |
Weeks | [1, 4, 5, 15] | [6, 31, 36, 102] | |
Months | [2, 6, 8, 17] | [59, 191, 233, 504] | |
Years | [2, 4, 7, 12] | [709, 1737, 2573, 4150] | |
Several | Days | [1, 4, 5, 10] | |
Weeks | [1, 3, 5, 11] | [8, 24, 34, 76] | |
Months | [1, 3, 5, 14] | [52, 81, 145, 401] | |
Years | [1, 3, 5, 14] | [261, 1280, 1583, 5081] |
Lang | IV approach | PR expression |
\(IV_{days}\)
|
\(IV_{weeks}\)
|
\(IV_{months}\)
|
\(IV_{years}\)
|
---|---|---|---|---|---|---|
Pt | Mean | Now | 0.729 | 0.182 | 0.074 | 0.015 |
Recently | 0.313 | 0.462 | 0.168 | 0.056 | ||
Currently | 0.379 | 0.308 | 0.231 | 0.081 | ||
Regression | Now | 0.784 | 0.130 | 0.075 | 0.011 | |
Recently | 0.325 | 0.439 | 0.157 | 0.079 | ||
Currently | 0.512 | 0.379 | 0.095 | 0.013 | ||
En | Mean | Now | 0.385 | 0.175 | 0.338 | 0.102 |
Recently | 0.180 | 0.528 | 0.281 | 0.010 | ||
Currently | 0.343 | 0.390 | 0.208 | 0.059 | ||
Regression | Now | 0.557 | 0.161 | 0.339 | \(-\) 0.056 | |
Recently | 0.239 | 0.574 | 0.189 | \(-\) 0.003 | ||
Currently | 0.437 | 0.474 | 0.078 | 0.012 |
5.4 Comparing languages
Imprecise type | F1 |
\(\hbox {\textit{F}1}_{3\mathrm{D}}\)
|
---|---|---|
MV | 0.731 | 0.692 |
IV | 0.767 | 0.719 |
PR | 0.391 | 0.304 |