Image Caption with Prior Knowledge Graph and Heterogeneous Attention

Currently, most image description models are limited in their ability to generate descriptions that reflect personal experiences and subjective perspectives. This makes it difficult to produce relevant and engaging descriptions that truly capture the essence of the image. To address this issue, we propose a novel approach called Subject-awareness-driven Heterogeneous Attention (SCHA). SCHA leverages users’ knowledge and expertise to generate content-adaptive image descriptions that are more human-like and reflective of personal experiences. Our approach involves a carefully designed heterogeneous cascade annotation model that captures scene information from multiple perspectives. We also incorporate a prior knowledge graph with textual information to enhance the richness and relevance of the generated descriptions. Our method has great potential for industrial production detection and can open up new possibilities for increasing the flexibility and variety of detection steps. When compared to the results of MSCOCO and Visual Genome datasets, our approach produces richer and more adaptive descriptions than widely used baseline models.