Comparison of Perplexity Scores of Language Models for Telugu Data Corpus in the Agricultural Domain

The agricultural domain has a lack of readily available resources, especially in regional languages. For a country like India, which majorly relies on agriculture as its main source of GDP and employment, it becomes vital to develop a corpus that spans across multiple topics of this domain. Post-collection of data, there must be a language model (LM) that can be implemented to assess the use of this data collected. Perplexity is a measure of how well a probability distribution model can predict a sample. Based on the lowest perplexity score of all models, we determine which LM performs the best. This paper compares three different LMs—n-gram, LSTM and Transformers. The perplexity of LSTM and Transformers was found to be 23.127 and 12.3 respectively, on the Telugu language dataset that was built by collecting data via web-scraping of links on the internet. The alignment of the theoretical knowledge and observed results of perplexity scores validates that the prepared Telugu agricultural dataset can be used for further NLP applications.