DEC-transformer: deep embedded clustering with transformer on Chinese long text

Long text clustering is of great significance and practical value in data mining, such as information retrieval, text integration, and data compression. Compared with short text clustering, long text clustering involves more semantic information representation and processing, making it a challenging problem. Most recent techniques merely rely on dynamic word embeddings from pre-training as a transfer learning or only based on a simple combination of transformers and traditional clustering methods, which still need to be expanded to short text due to the quadratic computational complexity. In this paper, a novel model combining transfer learning and dynamic feedback called deep embedded clustering with transformer(DEC-transformer) is proposed. To better capture the semantic relationships between sentences in documents, a novel transfer learning technology is firstly applied to long text clustering tasks for pre-training. Unlike previous papers, a two-stage training task is designed by treating semantic representation and text clustering as a united process, and the parameter is dynamically optimized by adaptive feedback to further improve efficiency. Experimental results on the test set show that the proposed model has made great progress in accuracy compared with several benchmarks. Furthermore, it also has good robustness and can get good performance on noisy datasets.