A Scalable Model for Frequency Distribution of Low Occurrence Multi-words Towards Handling Very Large Spectrum of Text Corpora Sizes

This chapter delves into the development of a scalable model for predicting the frequency distribution of low occurrence multi-words in large text corpora. The model focuses on n-grams, sequences of n consecutive words, and their statistical distributions, which are crucial for applications such as indexing, term extraction, compression, cache design, and translation. The chapter highlights the importance of understanding n-gram distributions as a function of corpus size for guiding tokenization strategies and corpus analysis for pre-training language models. It addresses the limitations of traditional models that consider only moderate-sized corpora and single words, emphasizing the relevance of multi-word n-grams in capturing semantic specificity and language structure. The proposed model predicts the cumulative number of distinct n-grams and their sizes, achieving very low and stable average relative errors across a wide range of corpus sizes, from hundreds of millions to hundreds of billions of words. The chapter also discusses the methodology for estimating model parameters, including the use of cross-validation and spline-based regression, and presents experimental results that demonstrate the model's accuracy and stability. The findings suggest that the proposed approach is promising for addressing the challenges posed by very large-scale corpora sizes and opens possibilities for handling relevant low occurrence multi-words in emerging applications based on large language models.