Chinese Medical Text Classification with RoBERTa

Many existed Chinese text classification solutions are successful, but the gap is that they are also limited by the models applied by themselves, so it’s available to consider a solution for advancing the Chinese text classification performance, especially in TCM (Traditional Chinese Medicine) text classification task. Assembled by Encoder element and Decoder element, Transformer and others X-former models have shown an outstanding performance in different NLP tasks, and among them BERT has succeeded in text representation and text classification tasks, but it has the possibility to be improved. Here we show our solution and experiment. In many NLP tasks, RoBERTa, which is based on BERT, has s a state-of-the-art performance than BERT. The classified sample data is selected from TCM workbench and tokenized by the Tokenizer we build based on pretrained RoBERTa, which was processed by RoBERTa_TCM, the RoBERTa model fine-tuned with our own data. In order to evaluate the vectorization performance and text classification performance of our Tokenizer-RoBERTa_TCM solution, we select some wild-range-applied language model: Word2Vec, LSTM, Bi-LSTM, contributing 4 baselines: Word2Vec-LSTM, Word2Vec-BiLSTM, Tokenizer-LSTM, Tokenizer-BiLSTM. We find out that the Tokenizer-RoBERTa_TCM model has shown a state-of-the-art classification ability with 90.88% average precision, 91.05% average recall and 90.72% average F1. All of them were the highest results among the baselines. It means that compared to regular text classification models (LSTM, Bi-LSTM, etc.), our RoBERTa_TCM model has an obvious improvement. This solution has the potential application research value in the text classification of TCM text.