Discriminative and efficient non-local attention network for league of legends highlight detection

Research on video highlight detection is mainly performed along two directions: (a) keyframe-based methods and (b) structure-driven methods. Keyframe-based methods use a subset of representative keyframes from the original video to generate highlights. Most early video highlight detection methods focus on extracting keyframes independently and using them as a classification task. Borth et al. proposed a keyframe extraction approach in which the video is first segmented into shots using shot boundary detection, and then obtained keyframes using the k-means algorithm [15]. Lin et al. used a context-specific highlight support vector machines (SVM) model to summarize video sequences without watching the entire video by predicting the contextual information of each video segment [16]. Even though these methods achieved remarkable performance, they only extract low-level features and ignore the temporal dependency, which describes the relationship between highlight and non-highlight frames.

Unlike keyframe methods, structure-driven methods exploit a well-defined data structure in the detected video, such as audience cheering and chatting, score changes, or other special events. Therefore, structure-based methods are suitable for sports video highlight detection, and they have attracted the attention of many researchers [6, 17, 17]. Zhao et al. proposed a highlight detection model based on audio energy and motion activity. Hsieh et al. proposed a more flexible solution for finding important and meaningful events in sports games by analyzing the messages shared between users on microblog services [18]. While sports video highlight detection has improved, these methods rely on audio, textual, and psychological data, which are not always easy to obtain.

With the recent rapid development of eSports, video highlight detection has attracted the attention of both industry and academia. Fu et al. proposed a CNN-LSTM model for LoL that combine visual features and real-world audience discourse. Song et al. proposed a cascaded prediction approach for learning convolution filters of visual effects for detecting video highlights in Heroes of the Storm, LoL, and Dota2 [19]. Wang et al. [20] proposed a multi-stream framework to fuse spatial, temporal information, and audio features extracted from Honor of King videos.

Recent eSports video highlight detection attempted to address these problems from a cross-modal perspective. However, for real-world applications, cross-modal information, such as audience chat or audio signal, is either difficult to obtain or requires additional computation cost. Therefore, we attempt to consider eSports video highlight detection using only visual features.

The objective of attention-based video highlight detection is to find what the user is most paying attention to, which is highly correlated with highlights. Ma et al. presented a generic framework in which computational attention models based on the modeling of viewer’s attention were used [21]. Ma et al. presented a generic framework for a user attention model, which estimates the how much attention viewers may pay to video content. Ejaz et al. proposed an efficient visual attention model based on the key frame extraction method and reduced the computational cost using the temporal gradient based on dynamic visual saliency detection [22]. Additionally, some studies integrated spatial–temporal clues into attention-based video highlight detection, with the goal of determining when and where users are most interested. A novel 3D dimensional attention model was proposed, which can automatically localize the key elements in a video without any extra supervised annotations [23].

A self-attention mechanism was proposed to model the long-range dependency in machine translation [24]. Inspired by self-attention mechanism [24] and the non-local means algorithm [25], Wang et al. proposed the non-local attention [13], which computes the response at one position as a weighted sum of the features at all positions, capturing long-range spatial–temporal dependency for video representation. Our work is similar to [23], but [23] computes coarse-grained spatial temporal attention in which the attention matrix only models the relationship between different channels and shares along spatial and temporal dimensions. Conversely, DENAN can capture a fine-grained relationship at the pixel level, capturing the relationship at all positions along spatial and temporal dimensions.

Figure 2 shows an overview of our proposed network, which consists of three parts: (1) a LoL video encoder, which is used to convert LoL video sequences into deep feature representations. ShuffleNetV2 [14] is used as the encoder, (2) non-local attention module, which is used to capture both spatial–temporal and long-range dependencies in LoL video sequences, and (3) discriminative loss function for training. Triplet loss (TL) [26] and cross entropy (CE) loss are used to optimize the entire framework.

We adopt an efficient video encoder with low computation costs to meet the requirement for a real-time video highlight detection system. Inspired by the recent studies [14, 27], which designed an efficient and light-weight 2D universal CNN, we adopt ShuffleNetV2 [14] (Fig. 3) as a frame-level video encoder for LoL.

The number of feature channels in light-weight network is limited by the computing resources available. Compared with ShuffleNetV1 [27], the main improvement of ShuffleNetV2 [14] is channel split operation. Particularly, the total number of channels of the input feature map is divided into one branch with \({C}^{^{\prime}}\) channels and the other with \(\left(C-{C}^{^{\prime}}\right)\) channels (\(C\) is the total number of channels). After a three-layer convolution operation for each branch, the features of the two branches are combined. Then, a channel shuffle operation is applied to enable two branches of information to interact.

Given a set of video sequences \({\{{X}_{i}\}}_{i=1}^{N}\), where \({X}_{i}\) denotes one LoL video sequence and \(N\) represents the total number of video sequences. Each sequence contains \(T\) frames, where \(T\) is the video sequence length. Frame-by-frame video images for one sequence are input to the video encoder to obtain a set of frame-level feature sequences \({\chi }_{i}={\{{x}_{i}^{t}\}}_{t=1}^{T}\), where \({x}_{i}^{t}\) represent the feature map for \({t}\)th image in \({i}\)th sequence. Then, the output from the video encoder is fed into the non-local attention module to further capture the spatial–temporal features of LoL video.

The input of non-local attention module are the frame-level feature sequences \({\chi }_{i}={\{{x}_{i}^{t}\}}_{t=1}^{T}\). Following the non-local form proposed in [13, 25], the non-local operation could be defined as follows:where \({\chi }_{i}\) represent the extracted feature sequences, \(j\) represent the computed position index of the response, \(k\) represent the possible positions of all input feature sequences in both spatial and temporal dimensions, \(s\left({\chi }_{i,j},{\chi }_{i,k}\right)\) denotes the relationship between position \(j\) and position \(k\) of the input feature sequences, \(g({\chi }_{i,k})\) computes the feature representation of input \({\chi }_{i}\) at position \(k\), and \(N({\chi }_{i})\) is the normalization factor.

We design a non-local attention module based on the non-local operation [13] to capture spatial and temporal pixel dependencies in LoL video (Fig. 4). The definition of non-local operation is similar to Eq. (1), which captures the spatial and temporal long-range pixel relationship. Then, the definition of non-local attention is defined as follows:where \({\stackrel{\sim }{\mathcal{F}}}_{i,j}\) represent a specific position element of \({\stackrel{\sim }{\mathcal{F}}}_{i}\) on position \(j\), \(k\) are all possible positions of \({\chi }_{i}\) to be computed. Here, we project \(\chi_{i,j} \; {\text{and}}\; \chi_{i,k}\) to embedding space using a linear transformation. Therefore, \(\theta \left({\chi }_{i,j}\right)={W}_{\theta }{\chi }_{i,j}\), \(\phi \left({\chi }_{i,k}\right)={W}_{\phi }{\chi }_{i,k}\), and \(g\left({\chi }_{i,j}\right)={W}_{g}{\chi }_{i,j}\), where \(W_{\theta } ,W_{\phi } ,\;{\text{and}}\; W_{g}\) are weights to be learned. Equation (2) is similar to the self-attention mechanism proposed in [24].

To project \({\stackrel{\sim }{\mathcal{F}}}_{i}\) into the original space, the non-local operation is wrapped into non-local attention module, as shown in Fig. 4, which is defined as follows:where \({W}_{z}\) represent a linear projection matrix, which is implemented using a \(1\times 1\times 1\) convolution, and \({Z}_{i}\) represent the final output of non-local attention module. A video highlight is a duration of continuous frames that are not time-independent. However, determining whether a particular frame is a highlight must consider the long-range pixel dependency in the spatial dimension. Therefore, we should consider the relationship between LoL videos from both spatial and temporal dimensions.

To further refine the spatial–temporal information of the learned features, we use global average pooling (GAP) to aggregate information in spatial and temporal dimensions as follows:where \(\stackrel{\sim }{{z}_{i}}=[\stackrel{\sim }{{z}_{i}^{1}},\dots ,\stackrel{\sim }{{z}_{i}^{C}}]\) is the feature representation vector of \({\chi }_{i}\), \(c\) denotes the position index of \(\stackrel{\sim }{{z}_{i}^{c}}\) in \(\stackrel{\sim }{{z}_{i}}\), and \(c=1024\) in our proposed DENAN.

To extract discriminative and robust features from DENAN, we combined CE loss with TL [26] to train our framework for LoL video highlight detection. The CE loss is defined as follows:where \(f\left(\cdot \right)\) denotes softmax, \({W}_{y}\) represent the weight, the output \(f\left(\stackrel{\sim }{{z}_{i}}\right)\) denotes the probability of whether the input frame is a highlight, \(N\) is the number of LoL video sequences in a mini-batch, and \(j\) are all classes, including highlight and non-highlight.

Meanwhile, the discriminativeness of the features extracted from DENAN was improved using TL [26]. The batch-hard TL is defined as follows:where \({\tilde{z }}_{a},{\tilde{z }}_{p},{\tilde{z }}_{n}\) are features extracted from the anchor, positive and negative samples, respectively, and \(\mathrm{\alpha }\) is the margin hyper-parameter to control the differences between intra- and inter-distance. Here, positive and negative samples refer to the LoL video sequence with the same or different class from the anchor.

In summary, the loss function for DENAN training is a combination of TL and CE losses, which is defined as follows:where \(\lambda \) controls the balance of TL and CE loss, \(\lambda \) is 1 in this work.

We trained and evaluated our proposed DENAN on two datasets: NALCS [28] and LMS [28]. NALCS contained 218 videos of LoL in the 2017 Spring, with 128 videos divided into a training set, 40 videos divided into a validation set, and 50 videos divided into a test set. The average length of each video is between 30 and 50 min, with both highlight and non-highlight frames. The data labeling process has been described in detail [28].

The LMS dataset contained 103 videos of LoL, including 57 training videos, 18 validation videos, and 28 testing videos. In our experiment, training and validation sets of these two datasets were used for training, while test sets were used for testing.

Based on the commonly used metrics in video summarization tasks [6, 8, 29], we use precision (\(\mathrm{P}\)), recall (\(\mathrm{R}\)), and Fl-score (\(\mathrm{F}1\)) as evaluation metrics in our experiments to evaluate the performance of DENAN. Let \(\mathrm{TP}\) denotes positive frames that are correctly predicted, \(\mathrm{FP}\) denotes positive frames with an incorrect prediction, \(\mathrm{FN}\) denotes negative frames with an incorrect prediction. Then, the \(\mathrm{P}\), \(\mathrm{R}\), and \(\mathrm{F}1\) are calculated as follows:

To evaluate the computation complexity of DENAN, we also introduce floating point operations (FLOPs) followed by number of parameters (Num Params) [14, 30]. In particular, FLOPs represents the number of floating point operations the model performs when processing a sequence of data, while Num Params represents the number of DENAN parameters.

In LoL, the attractive action, such as escape and kill, occurs in the later part of the highlight fragment. Because no one can predict what will happen until the last moment, the later part of the highlight fragment is highly related to the final result. To generate a proper data format, we sampled 5000 positive frames from the last labeled positive frames as the real positive for training on NALCS and LMS datasets. Additionally, another 5000 negative frames were sampled over all negative frames. These sampled frames were considered the first frame for each video sequence, and the remaining frames were sampled every ten frames for each video sequence in a 720P 30 FPS video. During testing, the frames were evaluated every 30 frames in a 720P 30 FPS video.

Our proposed DENAN uses ImageNet-pretrained ShuffleNetV2 [14] as a backbone. The last two layers of ShuffleNetV2 (GAP and FC layers) are removed. All the frames in LoL videos are resized to \(224\times 224\). There are 32 LoL video sequences in each mini-batch, each of which contains 16 frames, resulting in 512 frames. During the training stage, the initial learning rate is set to 0.01 and is decreased to 0.001 in the 20th epoch. We set the maximum epoch of iterations to 60, which is sufficient to reach convergence. The SGD algorithm is used for optimizing the parameters. The momentum and weight decay for SGD are set to 0.9 and 10^–4, respectively.

In this section, we conduct a series of experiments on NALCS and LMS datasets to demonstrate the validity of all components in our proposed DENAN. Additionally, we investigated the effect of the TL margin and the length of the video sequence on model performance.

Table 1 shows the results of an ablation study conducted for each component in DENAN. The ShuffleNetV2 network was trained with CE loss on the NALCS and LMS datasets as the baseline. NAN stands for non-local attention network and TL denotes triplet loss. Compared with the baseline, baseline + NAN improves the P, R, and Fl by 0.04, − 0.02, and 0.01, and 0.07, − 0.04, and 0.01 on NALCS and LMS datasets, respectively. Baseline + TL means that TL is combined with model training over baseline, which improves P, R, Fl by 0.01, 0.01, and 0.01, and 0.02, 0.01, and 0.01 on NALCS and LMS datasets, respectively. The objective of TL is to learn discriminative and robust feature representations. The objective of non-local attention is to capture the long-range dependencies in spatial and temporal dimensions. When TL and non-local attention are combined, the discriminativeness of feature representations learned by non-local attention network is improved. The P, R, and Fl are improved by − 0.03, 0.05, and 0.01 on the NALCS dataset and − 0.03, 0.04, and 0.01 on the LMS dataset over baseline + NAN, respectively. These experimental results show the effectiveness of non-local attention in capturing spatial and temporal long-range dependencies.

In Fig. 5, we conduct experiments on both NALCS and LMS datasets with baseline + NAN + TL to evaluate the performance in terms of the TL margin, which controls the minimum distance between the hardest positive and hardest negative. In these experiments, our framework achieves the best result on NALCS and LMS datasets when \(\alpha =0.1\). Smaller values for the TL margin yield promising results because TL focuses directly on the feature representation and can enhance the robustness of an only CE loss-trained backbone. When the TL margin is set to a larger value, DENAN overfitting occurs.

Table 2 shows the experimental results on NALCS and LMS datasets for an ablation study based on the length of input LoL video sequences. For a fair comparison, we use the model trained with a video sequence length of 16 and evaluate the performance of different lengths (4, 8, and 16). Compared with \(T=4\), when \(T=16\), P, R, and F1 increased by 0.11, 0.04, and 0.08, respectively, on the NALCS dataset, whereas on the LMS dataset, P, R, and F1 increased by 0.08, 0.08, and 0.08, respectively. This is because our proposed DENAN captures the spatial–temporal dependency between different frames. In theory, as the length of the video sequence grows, the performance of the model should increase. However, the length is limited by the GPU memory, and we can only set a maximum length of 16.

As shown in Table 3, our approach significantly outperforms the state-of-the-art methods based on the evaluation metrics with P, R, Fl, FLOPs, and Num Params on NALCS and LMS datasets.

In Table 3. P, R, and Fl of the proposed DENAN are 0.77, 0.74, and 0.76, respectively, on the NALCS dataset and 0.73, 0.78, and 0.76, respectively, on the LMS dataset. In particular, the Num Params and FLOPs of the proposed DENAN are 3.35 and 246.77 M, respectively. DR-DSN [8] develops video summarization as a sequential decision-making process and a deep summarization network to indicate the likelihood of a frame being selected to summarize the video. According to the probability distributions, selected frames are used as video highlights. Our method is 0.06 higher for Fl on the NALCS dataset, and 0.08 higher on the LMS dataset. In terms of Num Params and FLOPs, DENAN outperformed DR-DSN. However, DR-DSN first extracts frame-level features using GoogLeNet [31], and the extracted features are always local. Therefore, our model still has a computational complexity advantage. Iv-LSTM [28] is proposed for video highlight prediction in LoL based on joint visual features and textual analysis of the audience commentary. It achieves 0.79, 0.7, and 0.75, and 0.72, 0.68, and 0.7 on the NALCS and LMS datasets, respectively. Our proposed method outperforms Iv-LSTM by − 0.02, 0.04, and 0.01, and 0.01, 0.1, and 0.06 on the NALCS and LMS datasets, respectively. DENAN outperforms Iv-LSTM with only visual features, which demonstrates the effectiveness of DENAN for LoL video highlight detection.

We compare the results of LoL video highlight made by Tencent and our proposed DENAN. Figure 6a, c show the LoL video highlight replay made by Tencent. Figure 6b, d show the LoL video highlight detection made by our proposed DENAN. Highlight frames and non-highlight frames are marked with green and red blocks, respectively. Most highlight frames are detected correctly, and our proposed DENAN outperforms state-of-the-art methods, including Tencent’s current methods. Our proposed DENAN not only achieves accurate video highlight detection (higher P, R, and Fl), but it also has lower FLOPs and Num Params, enabling automatic and real-time video highlight detection.

Figure 7 shows the visualization of our proposed method on LoL videos from the NALCS and LMS datasets. We denote video highlight and non-highlight frames as 1 and − 1. The upper part of Fig. 7a, b shows a comparison of ground truth (label) and prediction (predict), where the turquoise line represents the ground truth of each frame in the video, and the light-pink line represents the prediction of our proposed DENAN. The lower part shows the overlap between ground truth and prediction, with green indicating correct prediction and red indicating the opposite. As shown in Fig. 7, most positive labels are predicted correctly, which demonstrates the effectiveness of our method.