Video captioning based on vision transformer and reinforcement learning

Model structure

As shown in Fig. 1, the model includes three modules: feature extraction, video caption generation and reinforcement learning feedback mechanism. The feature extraction module extracts the features of the segmented video frame through a convolution neural network. The video caption generation module adopts an encoding-decoding framework to encode the features before decoding to generate the caption of the video content. The reinforcement learning feedback mechanism module takes the model as the agent, the video data and the real caption as the environment, and optimizes the caption of the video content based on the CIDEr index.

Feature extraction

Video data is composed of objects, scenes, people and other static elements in the spatial domain, and its structure is composed of multiple continuous video frames. There are changes in motion trajectories between frames, which contain rich temporal motion information (Wang, 2020). Hence, both frame level static features and temporal motion features of videos need to be extracted.

In deep learning methods, we usually extract richer features by continuously stacking the number of network layers. However, the accuracy of the model gradually begins to saturate and rapidly decline as neural networks deepen. It will lead to the disappearance of the gradient and the degradation of accuracy. In response to this problem, He Kaiming et al. designed ResNet network and introduced deep residual learning module (Ioffe & Szegedy, 2015). It fits the residual mapping by stacking layers, so that the accuracy of the model increases as the number of network layers increases. In addition, the deeper network layers will increase the computational complexity of the model. However, ResNeXt network can effectively solve such defects. To sum up, ResNet and ResNeXt (Xie et al., 2017) networks are selected to extract static features and temporal motion features in the video. To be clear, we used the ResNeXt pretraining model trained on Dataset Kinetics (Carreira & Zisserman, 2017).

Before extracting the video features, we first segment the video data into $224\times 224$ video frames using FFMPEG tool. We do not limit the number of video frames, but uniformly process 50 frames of equal length before inputting the reference model. Then, all video frames are fed into the feature extraction network to obtain the complete features of the video. Suppose that the video is segmented into N video frames, and the frame sequence is $\mathrm{V}={\left\{X_i,y\right\}}_{i=1}^N$. We extract 2,048-dimensional static feature r_i and dynamic feature e_i for each frame respectively. The sum result x_i of the two features is used as the overall feature of the video. The sequence of visual features is $x_v=\left\{x_1,x_2,x_3,\cdots ,x_N\right\}$, which is calculated as Eqs. (1)–(3).

(1) $$r_i=f_r(X_i)$$

(2) $$e_i=f_e(X_i)$$

(3) $$x_i=r_i+e_i$$where r_i represents the result of static feature extraction, e_i is the result of temporal motion feature extraction, f_r and f_e represent static and dynamic feature extraction functions respectively, $X_i\in R^{C\times H\times W}$ represents the i-th video frame, $x_i\in R^{d_{visiual}}$, d_visiual is the dimension of the video feature. We set the size of d_visiual to 4,096 dimensions. C, H and W are the number of channels, height, and width of the video frame. Their values are 3, 224, and 224.

Video captioning generation

The video caption generation phase consists of the encoding and decoding of the video features, as shown in Fig. 2. In the encoding stage, the embedded feature vectors are fed into the encoder to encode the video features globally. In the decoding stage, the encoding result is taken as the input to the decoder, and its output is the video captioning statement.

Encoder

The vision transformer (ViT) model proposed in 2021 can encode image features with the global field of view, and tackle the problem that convolution networks are highly sensitive to the high-frequency information in the image. Inspired by literature (Mubashira & James, 2020), we use the transformer encoder of ViT as the model encoder, and effectively use the intermediate state of the encoder to implement the global encoding of video features.

The encoder of the benchmark model is made up of a stack of 12 single Vision Transformer encoding blocks. Each block consists of Multi-Head Attention (MHA) and MultiLayer Perceptron (MLP) Block, as shown in Fig. 3. To ensure the stability of the distribution of data features, the data is normalized by Layer Norm (LN) before each block is executed.

In the x-th time step shown in Fig. 3, it is assumed that the video feature extracted by the convolution network is C. First, we use MHA function to calculate the normalized characteristic C of the previous time step. Then, apply the MLP function to calculate the output of the coding block. Finally, the N features are normalized. The result is expressed as the final feature of the encoder, and its output size is 1,024 dimensions, which is calculated as Eqs. (4)–(6).

(4) $$\begin{array}{c}{z}_l^{\mathrm{\prime }}=MSA(LN(z_{l-1}))+z_{l-1}l=1\dots N\end{array}$$

(5) $$\begin{array}{cc}{z}_l=M\mathrm{L}\mathrm{P}(LN(z_l^{\mathrm{\prime }}))+z_l^{\mathrm{\prime }}& l=1\dots N\end{array}$$

(6) $$\begin{array}{cc}{x}_l=LN(z_l)& l=1\dots N\end{array}$$where $z_l^{\mathrm{\prime }}$ represents the output of the multi-head attention mechanism, z_l is the output of the multilayer perceptron, x_l is the output of the encoder at time l, which is the result of the global feature encoding. N is the total time step length.

Decoder

Taking into account the timing relationship between video frames, the benchmark model in this paper uses a multi-layer Long Short-Term Memory neural network (LSTM) to construct the decoder. The LSTM consists of input gate, forgetting gate, output gate and memory unit. The network structure of the LSTM unit is shown in Fig. 4. The LSTM network transmits cell state as well as hidden state in forward propagation. This effectively solves the problem that the parameters of other recurrent neural networks cannot be continuously optimized due to the disappearance of gradient in the process of back propagation (Yang et al., 2018). Therefore, we construct an LSTM decoder to remember the context timing relationship while retaining the video content information, and generate a more logical caption.

As shown in Fig. 4, let the coding result of the t time model be 1,024-dimensional vector x_t, the hidden layer feature corresponding to the input feature be h_t−1, and the cell memory unit of the LSTM network be c_t. Then the activation function $\sigma$ is used to obtain the input feature vector i_t of the LSTM unit. Similarly, the forgetting feature f_t and the output feature o_t can be obtained as Eqs. (7)–(12).

(7) $$i_t=\sigma (W_{xi}{x}_t+W_{\mathrm{h}i}{h}_{t-1}+b_i)$$

(8) $$f_t=\sigma (W_{xf}{x}_t+W_{\mathrm{h}f}{h}_{t-1}+b_f)$$

(9) $$o_t=\sigma (W_{xo}{x}_t+W_{\mathrm{h}\mathrm{o}}{h}_{t-1}+b_o)$$

(10) $$g_t=\varphi (W_{xg}{x}_t+W_{\mathrm{h}\mathrm{g}}{h}_{t-1}+b_g)$$

(11) $$c_t=f_t\odot c_{t-1}+i_t\odot g_t$$

(12) $${\mathrm{h}}_t=o_t\odot \varphi (c_t)$$where $\odot$ is the Hadamard product operation, i_t represents the inputs, f_t is the forgetting feature, o_t is the output gate, g_t is the input modulation gate, W and b are the parameters to be optimized. We use the sigmoid activation function and the tanh activation shown in Eqs. (13) and (14). Adding forget gates and memory gates in the decoding process will enable the video captioning model to memorize the video content in the time domain, and generate a more logical caption.

(13) $$\sigma (x)={\displaystyle \frac{1}{1+e^{-x}}}$$

(14) $$\varphi (x)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}$$

The detailed process of the video feature decoding is shown in Table 1.

Table 1:

The detailed process of the video feature decoding.

Algorithm 1: Video feature decoding
Inputs	Initialize weights $W_{hi}$, $W_{xi}$ and encoded feature $x_t$
1	Calculate the forgetting gate eigenvector $f_t$ using Eq. (8);
2	Use Eq. (7) to calculate the input gate feature vector $i_t$;
3	Calculate the input modulation gate feature vector gt using Eqs. (10) and (11), and update ct−1 to ct
4	Calculate the output gate eigenvector Ot using Eq. (9);
5	Repeat steps (1) to (4) until all features are decoded, and the input of the last decoding unit is the output result of the decoder
Output	Decoded feature vector

DOI: 10.7717/peerj-cs.916/table-1

Furthermore, in order to enable mapping between the decoded result and the text, we preprocess the captioning text tags corresponding to the video. First, we use the word embedding method to encode each word in the caption into a 512-dimensional vector $y_t\in Y(y_1,y_2,y_3,\dots ,y_{512})$. We use a dataset in which the maximum length of all captions is 20 words. Therefore, we set the length of the video captioning sentence to 20. When embedded, captions of less than 20 words are represented by the number 0. Assuming the caption generated by the model be expressed as $y_t^{\mathrm{\prime }}\in Y^{\mathrm{\prime }}(y_1^{\mathrm{\prime }},y_2^{\mathrm{\prime }},y_3^{\mathrm{\prime }},\dots ,y_{\mathrm{m}}^{\mathrm{\prime }})$, then the conditional probability representation of $Y^{\mathrm{\prime }}$ with respect to X is shown in Eq. (15).

(15) $$P(Y^{\mathrm{\prime }}|X,Y)=P(y_1^{\mathrm{\prime }},\cdots ,y_m^{\mathrm{\prime }}|x_1,\cdots ,x_n;y_1,\cdots ,y_n)=\prod _{t=1}^{m}P(y_t^{\mathrm{\prime }}|h_{n+t-1},y_{t-1}^{\mathrm{\prime }})$$where the conditional probability $P(y_t^{\mathrm{\prime }}|h_{n+t})$ represents the probability value of all words in the corpus corresponding to the softmax layer. We represent the string [END] as the end of the sentence. The end marker not only prompts the model to switch coding and decoding in the training stage, but also can be used as a marker to describe the completion of generation in the test stage.

In summary, at time step t, the model first encodes the video features through the transformer encoder block, and then sends it to the multi-layer LSTM network to decode and generate the caption $y_t^{\mathrm{\prime }}$. Each time a caption is generated during the training process, the model calculates the cross-entropy loss based on the generated sentence and the real captioning, and continuously updates and optimizes the model parameters. The calculation is shown in Eqs. (7)–(12).

Reinforcement learning optimization

In order to improve the accuracy of video captioning model, a reinforcement learning method is introduced to learn the strategy gradient ${\pi }_{\theta }$, where $\theta$ represents the model parameters. Specially, benchmark models were used as Agent, video and captioning as Environment in reinforcement learning. In each time interval of the model, Agent generates a word accordingly. When the generated word is the end-of-sequence token [END], Environment calculates the reward value R(t) nd feeds it back to the Agent. The model optimization process is shown in Fig. 5.

When optimizing the model with reinforcement learning, let the sequence of words, status values and rewards generated by a video be $\tau =\left\{s_1,a_1,r_1,s_2,a_2,r_2,\cdots ,s_t,a_t,r_t\right\}$, where s_t represents the status of the Environment at time t, at is the word generated at time t. Finally, the model calculates the loss function gradient and optimizes the model parameters according to the loss value and the reward value as follows:

(16) $$L(\theta )=-{\displaystyle \frac{1}{N}\sum _{\tau }R(\tau )\log {\pi }_{\theta }(\tau )=-E_{\tau \sim {\pi }_{\theta }}[R(\tau )]}$$

(17) $${\mathrm{\nabla }}_{\theta }L(\theta )=-E_{\tau \sim {\pi }_{\theta }}[R(\tau )\cdot \mathrm{\nabla }\log {\pi }_{\theta }(\tau )]$$where ${\pi }_{\theta }(\tau )$ is the probability that the model is generated and described as $\tau$, and N is the number of samples. In order to improve the readability and fluency of the generated captions, we refer to the literature (Pasunuru & Bansal, 2017b) using the method of mixed loss in reinforcement learning to enhance learning. The proportion of cross entropy loss L_ce and reinforcement learning loss L_rl is adjusted by super parameter $\gamma$. Reinforcement learning is expressed as shown in Eq. (18).

(18) $$L_{mix}=(1-\gamma )L_{ce}+\gamma L_{rl}$$

In video captioning, traditional reward methods include CIDEr, BLUE and METEROR. Among them, CIDEr is a weighted evaluation index, which pays more attention to whether the generated captions contain the focus of the image content. The evaluation index is more consistent with the human evaluation method. Consequently, that we used cider as the reward index of reinforcement learning.

Datasets and evaluation indicators

We chose the MSR-VTT dataset commonly used in the field of video captioning, which contains 10,000 videos. Each video in the dataset contains 20 manually annotated reference captions, which are partitioned by Xu et al. (2016) before training. Specifically, it is divided into 6,513 as training data and 497 as verification data, and the rest as test data. In addition, we extract video features through the concept-v4 network proposed by Szegedy et al. (2017). The English annotation sentences in the above dataset were selected for model training.

Four common evaluation indicators of ROUGE-L, METEOR, BLEU-4 and CIDEr-D were used when evaluating the model (Denkowski & Lavie, 2014; Lin, 2004; Papineni et al., 2002; Vedantam, Lawrence Zitnick & Parikh, 2015). The ROUGE-L index considers the order of words in sentences and evaluates the meaning of sentences. The METEOR indicator is based on the single-precision weighted harmonic average and the single word recall rate. The evaluation results of this indicator are more relevant to the results of manual evaluation. Bleu-4 index measures the semantic similarity between the generated result and the target sentence by defining the number of 4-gram. The CIDEr index is often set as an evaluation method in the field of image or video captioning. The method represents a caption generated by the model and a real caption as a word frequency vector and inverse word frequency vector, and uses cosine similarity to measure the captioning performance. The evaluation index has higher reference value in the field of video and image content captioning (He et al., 2020). The higher the percentage score of these four standard evaluation indicators, the closer the generated captioning semantics to the real captioning.

Parameter setting

During feature extraction, videos will be randomly segmented into $224\times 224$ frames. A feature vector with 4,096 dimensions corresponding to the number of frames will be obtained. Then process all the feature data into the same dimension $50\times 4,096$. The LSTM decoder of the baseline model has a hidden layer size of 1,024. Before inputting the video features into the model, the 4,096-dimensional feature is mapped to 1,024-dimensional, and the word embedding is expressed as a 512-dimensional vector. The experiment uses the Adam optimizer to train the network, and the initial value of the learning rate is set to 0.0001. It can be reduced with the iteration of training. To prevent overfitting, we introduce the vertical connection dropout method proposed by Zaremba, Sutskever & Vinyals (2014), which can achieve the regularization effect. The initial value of all weights that need gradient update is set to a uniform distribution on the interval [−0.08, 0.08]. The width size of Beam Search is 5 in the testing phase.

Result Analysis

In order to verify the efficiency of the video content description model in this paper, the current mainstream video description models are constructed under the same dataset and evaluation index system. Among them, the POS-CG uses both a POS sequence generator and a description generator. It uses reinforcement learning to optimize the model end-to-end (Wang et al., 2019). SAAT model strengthens the focus on predicates and actions in sentences and enhances the recognition of action words. It gives more consideration to the interaction between objects within the video (Zheng, Wang & Tao, 2020). Therefore, improve the logic and readability of the description. The Cident-RL model uses a mixture of cross entropy loss and reinforcement learning loss. It adds Entailment score into the reward mechanism to improve the readability of generated description (Pasunuru & Bansal, 2017b). The SGN model mines the semantic information in consecutive video frames and divides the video segments into units of different information according to the semantics (Ryu et al., 2021). The experimental results are shown in Table 2.

It can be seen from Table 2 that the method in this paper obtains BLEU-4 of 42.0, METEOR of 28.8, ROUGE-L of 62.0 and CIDEr-D of 54.2, respectively. Compared with the best SGN model among all comparison models, the BLEU-4 indicator is improved by 2.9%; Compared to the top-scoring model CIDEnt-RL on metrics such as METEOR, ROUGE-L and CIDEr-D, our method scores 1.4%, 0.9% and 4.8% higher, respectively. The reason is that the model in this paper inputs the extracted features into the encoder composed of Vision Transformer after extracting features from the video using CNN network. It can pay attention to local information and consider global features. In addition, the introduction of LSTM networks preserves contextual timing information, which makes the generated description more logical.

Video content captioning aims to generate more consistent captioning of video content. Figure 6 shows some examples of our model in the MSR-VTT dataset. Each video data in the figure lists three manually annotated captions and one model-generated captioning. The model in this paper has the ability to generate more accurate and more readable captions compared the captions marked manually. The reason is that the model pays more attention to the global feature of the video, so that it can consider the overall structure of the video. In addition, manual annotation captions are often limited to personal knowledge domains, interests, and language skills. As a result, the model can generate better captioning, which also verifies the effectiveness of the proposed method.

Ablation experiment

In order to verify the advantages of the encoder module and reinforcement learning in the video content captioning model, we completed an ablation experiment on the MSR-VTT dataset. In detail, the LSTM-LSTM model generates video content captioning by using the LSTM network as encoder and decoder. LSTM-LSTM-RL integrates reinforcement learning on this basis, so that the weight parameters of the model are further optimized. Compared with the original model LSTM-LSTM, the model has increased by 0.5%, 0.5%, 1.4% and 6.4% respectively under the four evaluation indexes. The experimental results are shown in Table 3. It proves the effectiveness of reinforcement learning to optimize video content captioning model. However, both of them have the disadvantage of directly taking the final hidden layer state of the encoder as the input of the decoder. The models lose the content of the middle-hidden layer of the encoder, resulting in low model scores. Transformer LSTM model solves the problem, which replaces the encoder with vision transformer coding block. The model can globally encode video features and make full use of coding results in the decoding stage. Compared with the LSTM network as the encoder, the model with the Vision Transformer coding block as the encoder has achieved significant better results. The experimental results correspond to the Transformer-LSTM-RL are shown in Table 3. In summary, updating the encoder and introducing a reinforcement learning method improves the accuracy of the video content captioning task.