Intent Identification Using Few-Shot and Active Learning with User Feedback

Collaboration tools contain many intents in workplace conversation, and identifying these intents is important to increase workplace productivity. However, labelling these intents for a large collection of conversations is expensive and data availability for new intents is quite limited. The pre-trained models show outstanding results in text classification tasks and large language models have recently produced accurate models with few samples. We explored few-shot learning methods and active learning strategies for this problem. Our proposed method, “SetFit with AL” is a combination of Sentence Transformer Fine-tuning (SetFit) and active learning for intent identification. This method fine-tunes a sentence-transformer model to develop accurate models. The intent classification evaluation dataset was used to evaluate this method. The results show that our proposed method outperforms state-of-the-art large language model GPT-3.5 and is comparable to GPT-4. This method also can utilize user feedback to adapt to new data and develop personalized models. Thus, the contribution of this paper is that fine-grained intents are identified using minimal data and the model is adaptable based on user feedback.