2009 | OriginalPaper | Buchkapitel
Correlation of Term Count and Document Frequency for Google N-Grams
verfasst von : Martin Klein, Michael L. Nelson
Erschienen in: Advances in Information Retrieval
Verlag: Springer Berlin Heidelberg
Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by
For bounded datasets such as the TREC Web Track (WT10g) the computation of term frequency (TF) and inverse document frequency (IDF) is not difficult. However, when the corpus is the entire web, direct IDF calculation is impossible and values must instead be estimated. Most available datasets provide values for
term count (TC)
meaning the number of times a certain term occurs in the entire corpus. Intuitively this value is different from
document frequency (DF)
, the number of documents (e.g., web pages) a certain term occurs in. We investigate the relationship between
TC
and
DF
values of terms occurring in the Web as Corpus (WaC) and also the similarity between
TC
values obtained from the WaC and the Google N-gram dataset. A strong correlation between the two would gives us confidence in using the Google N-grams to estimate accurate IDF values which for example is the foundation to generate well performing lexical signatures based on the TF-IDF scheme. Our results show a very strong correlation between
TC
and
DF
within the WaC with Spearman’s
ρ
≥ 0.8 (
p
≤ 2.2×10
− 16
) and a high similarity between
TC
values from the WaC and the Google N-grams.