Multi-level and Multi-granularity Fusion Network for Multimodal Named Entity Recognition

The vast flow of information on social media drives the spread of users’ opinions, and posts combining text and images contain rich information. In this context, the multimodal named entity recognition (MNER) task aims to identify entities and categorize them in social media posts using images. However, the textual expressions in those posts are often informal and lack contextual representation, making it difficult to determine entity types. Besides, accompanying images may either contain no relevant objects or have a significant amount of distracting information. To this end, we propose a Multi-level and Multi-granularity Fusion network (MMF) for the MNER task. First, multi-granularity multimodal features are considered to enhance the input representation. Then, we build a Transformer-based fusion network to achieve multi-layer semantic interaction between text and visual features at different granularities. Finally, the text features are connected with the obtained visual guidance, and conditional random fields (CRF) are used for entity label classification. Extensive experiments on the Twitter2015 and Twitter2017 datasets demonstrate the effectiveness and superiority of our proposed method.