BERT-based semi-supervised domain adaptation for disastrous classification

Currently, with the rapid development of social media platforms represented by twitter, multimodal content on social media provides critical information during major disastrous events. Different from traditional text news, major disastrous events posted on Twitter often attach videos or images. Although many studies have shown that both image and text contents are useful for humanitarian aid and disaster response, this research focuses on analyzing multimodal information in social media to effectively analyze sudden disastrous events. We propose an end-to-end model, the BERT-based semi-supervised domain adaptation for multimodal disastrous events classification (BSSDA). BSSDA is composed of two main modules: a classifier with minimax entropy domain adaptation and a multimodal feature extractor. First, we perform multimodal feature extraction on our three modalities (image, text and image description). Here, we generate image description with the NeuralTalk and extract both image description features and text features with BERT, and extract image features with pretrained VGG. Then, the multimodal features are concatenated and fed to the classifier with minimax entropy domain adaptation module. The purpose of adding the domain adaptation module is to map the multimodal features of different disastrous events to the same feature space by using a good amount of unlabeled data of the unforeseen real-time disastrous event. Our experimental results of three different types of disastrous events show that our BSSDA model has significant improvements for disastrous events classification.