Conditional Embedding Pre-Training Language Model for Image Captioning

The pre-trained language model can not only learn language representations with different granularity from a large number of corpus, but also provide a good initialization for downstream tasks. Aggregation or alignment of text features and visual features as input of pre-training language model is the mainstream approach to deal with visual-language tasks. People can accurately describe an image by constantly referring to the visual information and key text information of the image. Inspired by this idea, we no longer follow mainstream approach, and propose to adjust the pre-training language model processing by using high-low visual features as conditional inputs. Specifically, conditional embedding layer normalization (CELN) we proposed is an effective mechanism for embedding visual features into pre-training language models for feature selection. We apply CELN to transformers in the unified pre-training language model (UNILM). This model parameter adjustment mechanism is an innovative attempt in pre-training language model. Extensive experiments on two challenging benchmarks (MSCOCO and Visual Genome datasets)demonstrate that this seminal work is effective. Code and models are publicly available at: https://​github.​com/​lpfworld/​CE-UNILM.