A Lexicon LDA Model Based Solution to Theme Extraction of Chinese Short Text on the Internet

Chinese short text has become the main content of the Internet. Accurately extracting thematic terms is the basis of content analysis, query suggestions, document classification, and text clustering and other tasks for Chinese short text on the Internet. Since Chinese short text is short on the Internet, unbalanced and less of context information, the traditional text clustering model is not immediately appropriate. This paper presents a simple and generic theme model named Lexicon LDA for Chinese short text on the Internet, by using the sentence structure within the document, to enrich the context of the common Chinese word semantics. Words of each sentence which is divided by punctuation marks compose a word set. Unlike the previous method, the model distributes the theme for each word set, rather than for each document. When the data set presents a strong theme distribution, it can significantly improve the effect of the theme model through experiments. The conclusion is that extracting thematic terms of Chinese short text on the Internet is related both to the word itself and to the sentence where the word is located.