CLIP-AMR-GPT: Enhancing Image Captioning via Cross-Modal Semantics Fusion and GPT-Based Re-ranking

Current image captioning models still face several critical challenges: insufficient exploitation of semantic knowledge, the absence of effective mechanisms to integrate heterogeneous feature sources, and limitations in generating natural and coherent output captions. To address these issues, this paper proposes a novel image captioning model, CLIP-AMR-GPT, built upon an encoder–decoder architecture that integrates multi-source knowledge. Specifically, the encoder combines vision–language features extracted from the CLIP model, relational graph embeddings representing semantic relationships among objects, and Abstract Meaning Representation (AMR) embeddings derived from ground-truth captions. The decoder employs an adaptive attention mechanism to dynamically regulate the influence of AMR-like graph embeddings at each word generation step, thereby enabling flexible exploitation of semantic structural information. Furthermore, a GPT-2-based re-ranking module is incorporated to evaluate and select captions with the highest linguistic likelihood, enhancing fluency and coherence. Experimental evaluations on the MS COCO benchmark dataset demonstrate that the proposed model outperforms state-of-the-art methods, validating the effectiveness of integrating visual, semantic, and linguistic knowledge into image captioning models.