AI and data-driven media analysis of TV content for optimised digital content marketing

Content Wizard is an extension of a professional-grade tool for social media marketing, integrating the new components for trans-vector publishing developed in the ReTV project. The currently manual, labour-intensive task of selecting and adapting video content for publication to a growing set of digital channels (vectors) can be supported and, in many cases, automated by the newly developed online components which communicate with each other via secure Web services to cover all aspects needed:The aforementioned set of components forms a distributed platform following a service-oriented architecture, which we have called the Trans-Vector Platform (TVP).¹¹ The Content Wizard is the user facing application on top of this architecture, re-using some of its components as illustrated in Fig 2.

The Prediction Service component uses collected metrics about past topics of online discussion to predict the next trending topics (cf. Sect.  3.2). These predictions guide both the recommendation of which content to publish and the scheduling of when it should be published; for this, the Recommendation and Scheduling component makes use of both the Binary Repository containing the video assets and a Video Feature Storage where extracted metadata for each video asset is stored (containing structural metadata like scenes, shots and sub-shots as well as descriptive metadata like concept and object detection). Topics of audience interest are matched to topics in video assets using a text-to-video matching module (cf. Sect.  3.3). The Video Adaptation and Re-purposing component takes a video asset and produces a modified copy according to the topic(s) to focus on and the target publication channel while aiming to retain as much as possible the original content of the video asset (cf. Sect.  3.4). The remaining components ensure that data visualisations are available to the Content Wizard user to guide their decision making in the workflow (Visualisation Engine) and that the final (digital video) posting can be published at the optimal time (cf. Sect.  3.5).

Trending topic detection is a part of predictive analytics, i.e. the use of algorithms, increasingly from the machine learning domain, to predict the future value of a chosen variable based on its past values and optionally additional features which exhibit a correlation to its values. To be able to offer a prediction, appropriate data has to be available. The online discourse on both Web and social media vectors has been collected by the webLyzard platform for several years for the news domain, while in the ReTV project we began both crawling TV/radio-specific Websites and social media monitoring for TV/radio related terms.¹² Our pipeline performs keyword and entity extraction from the text of the collected documents; we then calculate metrics related to communication success for each of those keywords or entities such as frequency (of mentions) and sentiment (based on the surrounding text). Since data collection takes place continually, these metrics can be expressed as time series data at daily or hourly granularity. We consider sets of keywords and entities as “topics” (from specific topics such as “Eurovision” which can include alternative labels - “Eurovision Song Contest” or “Grand Prix Eurovision” - as well as common hashtags or abbreviations, e.g. “ESC”, to broad topics such as “cultural events” which might encompass matches with museum exhibitions, art gallery vernissage, concerts etc.). For any topic we can extract the time series data for its frequency of mentions for at least the past two and a half years.

Our heuristic is that the frequency of mentions of a topic on a channel positively correlates with the reach and engagement for content about that topic (posted on the same channel at the same time), just as tweets are considered more likely to get more reach and engagement when associated with a trending topic. We consider topics—sets of keywords—instead of single keywords in the prediction as we recognise that there are constantly new keywords emerging in the online discourse for which broader topics can still be effective (e.g. “non-fungible token” (NFT) may not be present in past online documents but “blockchain” would be). We use the frequency metric as the target variable for prediction of topic presence (and, by corollary, popularity) by channel and time, so that we could suggest the optimal time for publication of content about that topic on that channel. By comparing a group of topics on the same channel for a certain time, we can also suggest the optimal topic for which to choose or create content for publishing at that time.

Following linear regression and ARIMA based approaches reported in previous work [39, 40], we chose to look at LSTMs (Long Short Term Memories) which have been reported as being particularly effective for the forecasting task [41, 42]. We wanted to perform multi-step forecasting directly from the prediction model so we looked at seq2seq models which were also being reported in the literature as the best performing LSTMs for multi-step forecasting [43]. Considering current research, we tested with the addition of attention mechanisms (Luong attention [44]), which further improved the accuracy. Finally, we also experimented with different activation functions and found a function proposed by Google engineers called Swish [45] to be the superior approach for our prediction task. Our final LSTM model, based on comparison with different configurations of accuracy with different evaluation datasets (based on keyword frequencies extracted from the webLyzard platform), is an Encoder-Decoder model with input–output sequences of length (200, 30), using Luong attention and Swish activation function.

Prediction accuracy for this LSTM model is illustrated below in Table 1 and Fig. 3 with one evaluation dataset for the topic “cycling” (frequency metric from global news media, 995 data points each where 1 data point represents 1 day). We provide MAE and RMSE metrics on the ‘val’(idation) data which is the next 30 days of actual values (not in the input dataset for training/testing), where a comparison of predicted values is made to the 7-day moving average of the actual values (as explained in,¹³ this better captures audience attention, as a recently present topic is not immediately forgotten). MAE and RMSE metrics are reported as the average from 5 runs. Different models are compared: E-D refers to Encoder-Decoder, AR to autoregression for single-step forecasts and TS is use of a TimeSeriesGenerator (from the Keras library for deep learning). The input–output sequences used are (7,1) and (200,30). Our SARIMAX model is also given as a comparative benchmark (baseline). The chart shows the similarity between the best performing model and the actual 7-day moving average. While this is just one evaluation, we tested across multiple topics. The main requirement for a good result was that the topic has not emerged recently and exhibits seasonality, i.e. past trends are indicative of future ones. Sports, for example, works well as the peaks and troughs of sports coverage—tied to events—tend to be similar each year (even when the average values have been lower for 2020 with the cancellations of sports events, there was still online discussion about the events around the time when they would have taken place).

Given that time series forecasting requires sufficient, relevant past data, we had to generate and pre-process almost three years of past data to predict the next 30 days for the LSTM evaluations (making it impossible to apply this approach for predicting e.g. the topic coronavirus which would encounter almost zero data for the period before January 2020). Here, shorter time periods could be used for training with single-step auto-regressive models when topics are ‘recently emerged’. However, this would still not be able to anticipate future appearances of the topic unrelated to past trends, e.g. due to a new event already known to be taking place in the future. We have, therefore, also processed all past documents to detect textual references to future dates, meaning we can also calculate a metric for the number of documents that associate a given topic with a given date. This metric, known as “temporal reference detection” is used in the prediction model as an additional feature to anticipate future spikes in mentions of a topic based not on past trends (which may not be present) but on a future event identified in the document corpus. The topic “elections” may be explored as an example of a future “out-of-trend” event. Consider a prediction task at the end of October 2020 for the next 30 days (being November 1–30, 2020) based on training with past data from 13 months (October 1, 2019 – October 31, 2020). Since an election is typically every four years, for example, the training data will not contain any frequency peaks for the topic ”elections” (no election took place), and if there is an election in the prediction period, it can not be anticipated from this past data—as would be the case with the 2020 US election on November 3, 2020.

To evaluate, we use the frequency metrics from global English language news media for training, testing, and validation with the prediction time period of November 1–30, 2020. The training/testing dataset (past 13 months) has 397 data points. The validation dataset has 30 data points and uses the 7-day moving average of the frequency metric. As seen in Fig. 4, the predicted values from our best performing model (orange line), which are based on the “usual” news cycle without major election events, are much lower than the actual as the forecasting is unable to take into account the US election on 3 November. Looking at our “temporal reference detection” metric for the topic in the month of November, there is a significant peak on 3 November (as we would expect, with many documents referencing “election” and the 3rd November together) and generally a much larger association of the topic with dates in the first half of the month. The blue line is the real (7-day moving average) values for the topic “elections’ in November. If we consider the addition of the prediction values with the temporal reference detection metrics (grey line), it can be seen that the predictions come closer to the actual values and interestingly the predicted value for 3 November is almost the same as the actual. The subsequent predicted values do drop off heavily from the actual values but it would probably have been impossible to anticipate the longer period of focus on the election several days after election day itself due to ongoing counts and disputes.

We conclude that the temporal reference detection metric can be valuable in combination with the LSTM-based forecasting model to incorporate out-of-trend peaks in topic popularity that would otherwise not be part of our predictions. We can then use the prediction model to identify which topic(s) can be expected to attract more attention from an online audience on some future date (and thus recommend a relevant video for publication). In an on-the-fly calculation situation, we can provide predictions almost immediately using only the temporal reference detection metric. Forecasting using LSTMs is more resource intensive and it is therefore not feasible to expect the same on-the-fly responsiveness yet such predictive capabilities can be provided using pre-calculations when the user knows the specific topics for which they want to forecast popularity over a future date range. The resulting prediction model enables organisations to anticipate future moments of heightened interest in any preselected topic, with the intention to publish relevant content at that time such that it has optimal chances to achieve maximal reach and engagement metrics.

Cross-modal retrieval is used for performing a retrieval task across different modalities such as image-text, video-text, and audio-text. In Content Wizard, we employ such techniques to address the text-video cross-modal retrieval task. Given a set of unlabeled video shots and an unseen textual query, this task aims to retrieve video shots from a media collection ranked from the most relevant to the least relevant shot for an input textual query.

Inspired by the dual encoding network presented in [46], the ATT-ATV network that encodes video-caption pairs into a common feature subspace is created [35]. This network utilizes attention mechanisms for more efficient textual and visual representation and exploits the benefits of richer textual and visual embeddings. Let \(\textbf{V}\) be a media item (e.g., an entire video or a video shot) and \(\textbf{S}\) the corresponding caption of \(\textbf{V}\). Both \(\textbf{V}\) and \(\textbf{S}\) are translated into a new common feature space \(\Phi (\cdot )\), resulting in two new representations \(\Phi (\textbf{V})\) and \(\Phi (\textbf{S})\) that are directly comparable. For this, two similar modules, consisting of multiple levels of encoding, are utilised for the visual and textual content, respectively. As discussed in detail in [35], the levels of encoding that each module consists of are: (i) mean-pooling, (ii) attention-based bi-GRU sequential model, and (iii) CNN layers; and, the improved marginal ranking loss [47] is used to train the entire network. The produced representations are highly effective in generating a ranked list of relevant video shots, given a free-text query of what the user is looking for. The ATT-ATV network is trained using a combination of four large-scale video captioning datasets: MSR-VTTT [48], TGIF [49], ActivityNet [50] and Vatex [51]. In [35], for calculating the visual embeddings that are used as input to the visual module, a ResNext-101 network (trained on the ImageNet-13k dataset) and a ResNet-152 network (trained on the ImageNet-11k dataset) were used. Also, two word embeddings were used as input to the textual module: (i) a Word2Vec model trained on the English tags of 30K Flickr images, and (ii) a pre-trained language representation BERT, trained on Wikipedia content. Here, to improve the performance, we re-train the ATT-ATV architecture using additional visual and textual features, coming from the trained CLIP model (ViT-B/32) [52]; both visual and textual embeddings are calculated, by utilising the corresponding CLIP image or textual encoder. Table 2 presents the results of the utilised (re-trained) ATT-ATV network and comparisons with state-of-the-art literature methods, including the initial ATT-ATV network of [35], on three Trecvid AVS benchmark datasets (AVS16, AVS17, and AVS18) [53]. The presented results are those reported in the original paper of each method. We use the mean extended inferred average precision (MxinfAP) as an evaluation measure, as proposed in [54] since it is the typical evaluation measure when working with these datasets. Our method outperforms all competitors on every dataset.

In the context of video content discovery in the Content Wizard, we focused on making it easier for editors to find videos in their collections that match the topics predicted to be more popular at the planned publication time. Specifically, given a text as search input, we want to return a set of relevant videos from the user’s media collection (see Fig. 5). We pass the text (e.g. the set of labels associated with the topic chosen for the content publication) into the newly developed text-to-video search module. The text of the search query is mapped into a common embedding space with the video shots from the media collection, both represented via 2048-dimensional vectors. We compare the embedding vector of the text to all of the embedding vectors of the video shots that have been computed for each video at the time that it is ingested in the archive and return a ranked list of the closest matches.

We use Elasticsearch to store and query these embedding vectors, using the special, optimised field type DenseVector. Elasticsearch also efficiently supports the cosine similarity metric between the vectors to find the shots that are closest to our text embedding, where we only need to look for the n closest shots so the search does not have to be exhaustive. As we have an embedding vector for each shot of a video, we get a matching score for each shot yet some shots may come from the same video so we have to aggregate the results. We highlight how the scores of different shots are aggregated can change the search results. Let us assume we have two videos, each containing ten shots. The first video contains a scene that discusses a dinosaur movie and nine other scenes with no relation to the cinema or dinosaurs. The second video is a sequence of 10 shots of a dinosaur museum. For a search on dinosaurs and film, one could argue that the scene in the first video is the most relevant, but the second video would usually be ranked as more relevant on average due to many more shots of the video being close to our search. Therefore, we decided to aggregate the distance metrics for each video shot in such a way as to rank the first video higher based on our observation that social media managers prefer short content that is highly relevant to longer content with lower relevance. Our cross-modal video retrieval uniquely prefers the video with the single highly relevant shot whereas traditional systems would rank the other video with more relevant shots, due to our use case being the selection of suitable short form content for social media publication.

Video summarisation aims to generate a concise synopsis of a full-length video that retains the most significant parts. Based on this, viewers can have a quick overview of the whole content without having to watch the entire video. In the context of Content Wizard, a new video summarisation method is utilised to create a media asset out of a longer form video into a topic-focused summary in the optimised short form for publication on social media.

In the core of our newly developed summarisation pipeline is an AC-SUM-GAN, an unsupervised deep-learning-based method that embeds an Actor-Critic model into a Generative Adversarial Network. The Actor and the Critic take part in a game that incrementally leads to the selection of the video key-fragments. Their choices at each step of the game result in a set of rewards from the Discriminator, according to the proximity of video-level representations of the original and the summary-based reconstructed version of the video in a learned latent space. The applied training workflow allows the Actor and Critic to discover a space of actions and states, and automatically learn a value function (Critic) and a policy for selecting the most important parts of the video (Actor). The pre-trained model of AC-SUM-GAN that is integrated into the Content Wizard, gets as input deep representations of the video frames that are obtained using the output of the pool5 layer of GoogleNet [57]. It is composed of the parts of the network architecture that are used at the inference stage, namely: (a) a linear layer for dimensionality reduction (from 1024 to 512), (b) the State Generator (formed using a 2-layer bi-directional LSTM with 512 hidden units) that formulates the state for the Actor’s choices, and (c) the Actor (formed using four fully-connected layers that are followed by a softmax layer) that progressively picks \(15\%\) of the video fragments, based on the generated state and a categorical distribution of probabilities. Further details about the network architecture and the processing pipeline at the training and inference stages can be found in [16]. According to a recent survey on deep learning methods for video summarisation [58], AC-SUM-GAN is ranked first (on average) among the top-5 unsupervised video summarisation approaches of the literature. As shown in Table 3, AC-SUM-GAN produces state-of-the-art results on two benchmarking datasets (SumMe and TVSum), reaching an F score that exceeds 50% on SumMe and 60% on the TVSum dataset (following the relevant established evaluation protocols).

To further adapt the results of this automatic summary generation to the video content retrieved by our previously described component, the analysis outcomes of AC-SUM-GAN are additionally combined with a set of rules and requirements about the content and characteristics of an optimal summary for social media, devised by considering requirements identified in the use cases of the ReTV project (e.g. “avoid talking-head shots”) as well as the feedback provided by test users from ReTV partners in response to indicative summarisation results that were shown to them. As a result of this, the final video summarisation pipeline implements the following rules:To incorporate these additional rules into the video summarisation process, we proceeded as follows. First, the video is analysed by the initial video summarisation method which produces a set of frame-level scores indicating the importance of the visual content of each frame. Then, taking into account the extracted information about the shot segments of the video, we calculate shot-level importance scores by averaging the scores of the frames within each shot. Following this, we rank the shots based on the computed shot-level scores, thus producing a first ranking of the shots. Subsequently, we compute another ranking after filtering out less appropriate shots based on the set of predefined rules, making use of various hand-crafted features and concept detection results. The hand-crafted features include the Edge Change Ratio (ECR) measure and the variance of the Laplacian of each frame—which is used as a measure of the blurriness of an image—to avoid the selection of shots with extreme camera movement. Regarding concept detection, we employed publicly available pre-trained models for the ImageNet [62] and Places365 [63] datasets, and trained a new model of the EfficientNet B3 architecture [64] on the TRECVID SIN dataset [65], thus generating detection scores for more than 1.5K concept labels. We then selected and utilised specific concept labels to detect and exclude shots that include anchorpersons or related concepts (e.g. TV studio, TV talk show, speaking to camera, etc). Finally, the computed rankings are combined by averaging the rank of each video shot to form the final ranking.

To produce the summary, the shot with the highest rank (i.e. the most appropriate for being included in the video summary) is selected as a candidate and is removed from the ranked list. We then check if the next candidate shot is visually similar to the set of the already selected shots for inclusion in the summary, skipping it if it is and selecting it if not. To assess the visual similarity of shots, we compare their distance (computed based on their representations in intermediate layers of a CNN we use for processing the video for visual concept detection) against a pre-defined min_visual_distance threshold. Following this, two different pre-defined thresholds min_shot_duration and max_shot_duration are utilised to filter out very short and very long shots, in such a way to control the pace of change between video segments. The values of min_shot_duration and max_shot_duration were set as 1 and 5 seconds respectively. If the duration of a candidate shot is greater than min_shot_duration and lower than max_shot_duration, then the whole shot is included in the summary. However, if this duration is greater than max_shot_duration, then we select a sub-part of the shot that lasts max_shot_duration seconds, by maximising its importance based on the computed frame-level importance scores by the video summarisation method [16]. We continue until the desired duration of the video summary is reached.

The aforementioned shot selection process is different from the one implemented in [16], where the summary is constructed by solving the 0/1 Knapsack problem. The new shot selection process enables the use of a user-adjustable rhythm parameter to modify min_shot_duration and max_shot_duration thresholds. Specifically, setting a \(\textit{rhythm} < 1\), since we multiply max_shot_duration to rhythm, the value of max_shot_duration results being below 5 seconds, consequently leading to the selection of smaller segments of the original video for the summary. Thus a fast-paced summary is constructed, i.e. with quickly alternating shots. Respectively, setting \(\textit{rhythm} > 1\), results in a summary with a slower rhythm of alternating shots. Additionally, plainly solving the knapsack problem would often select very small segments to fulfil the time budget requirements, as observed during preliminary experiments. This is avoided by employing the shot selection process discussed above.

An indicative example that demonstrates the impact of the additional rules developed in the ReTV project for video summarisation (suitable for social media publication) is presented in Fig. 6. The original video, which contains the start of a BBC news broadcast (the original video can be viewed at ¹⁴), is 77 s long, for which we aim to generate a 20 s summary (e.g. for TikTok or an Instagram Reel). Constructing a summary, with the selection of segments being based solely on the AC-SUM-GAN method’s frame importance scores, will select shots #4, #8, #9, #15 and #17, considering that a) the shots are the top-5 scoring shots (notice the height of the respective grey boxes at the middle of Fig. 6), and b) these shots’ total duration suffices to construct a 20-second summary. However, employing the editor-based rules introduced in ReTV, shots #4 and #17 are excluded since they contain frames of an anchorperson, while shot #15 is excluded due to the included synthetic graphics (see the key-frames of the respective shots at the top of Fig. 6). Therefore, our algorithm selected segments from shots #1, #5, #6, #8, #9 to generate the summary (see the key-frames of the selected shots at the bottom of Fig. 6).

Content Wizard is implemented as a Web-based tool built on top of the social media management tool Levuro Engage.¹⁵ Levuro Engage is a modern web application using React¹⁶ for the front end. The backend is distributed among multiple servers. The processing of uploaded videos is done using AWS encoding servers and the processed videos are stored in S3 buckets. This distributed approach scales well with an increasing number of videos. The social media and Web monitoring backend deployed by webLyzard¹⁷ which provides the documents used in the topic prediction uses Elastic indexes to scale up to support hundreds of millions of documents. The ReTV project did not only develop the innovative backend components for transvector publishing presented in the previous sub-sections which are called via REST interfaces by the Content Wizard but also Web-based user interfaces to interact with their functionality within the application.

The Content Wizard works with the video collections of the user. Since the process that determines the predicted popularity of topics is not instantaneous, several topics of interest may be predefined by the user and their predictions pre-computed for immediate access, with the user being able to request a recalculation using the latest Web and social media documents. The extraction of the text-to-video embeddings is also time-consuming, so we also pre-compute embeddings for the video collection and make them available in a database that allows for fast querying (also the user can request recalculations when new videos are added to the collection). To begin the transvector publication workflow, the user can look for future dates when a topic of interest to their audience is predicted to peak in interest, or simply see which of their topics are predicted to be rising in popularity on a specific future date. This view is available in the “trending stories” interface of the Tools tab of the Content Wizard (Fig. 7 shows the predicted popularity of the topic “James Bond” calculated on 25 March 2021 for the next four weeks. A peak in interest is predicted on 2 April 2021 - which was the originally planned release date of the next James Bond film “No Time To Die”).

Since the process of identifying relevant topics and matching them to appropriate videos in the users media collection is often laborious, a user can select a topic in the “trending stories” interface and choose directly the presented option “Video Search”. We immediately show the most related videos from their collection based on the keywords used by the topic as the textual input into the text-to-video search. The text-to-video search turns the list of keywords into an embedding vector and compares this embedding vector to all of the embedding vectors of video shots that are available in the collection and returns the best matches. For example, the videos matched to the topic “James Bond” all relate to film or cinema despite the search input not mentioning either term (see Fig. 8), an association which has been learnt by the embeddings layer. The user of the Content Wizard can then select the video, summarise it if desired using the ReTV video summarisation component, and post it on social media with an accompanying message.

When the user selects a video for editing in the Content Wizard, they are given the option to have it automatically summarised. The summarisation is then requested from the newly developed Video Adaptation and Re-purposing component. The video analysis process necessary for the video summarisation has been run in advance for all videos in the collection and the extracted features are stored in the Video Feature Storage, therefore the main summarisation process is almost instant (again, when new videos are introduced, the user can request their analysis so that their features are also available to the summarisation step). Figure 9 illustrates how the video editor cuts the video into the segments of the original video that the Video Adaptation and Re-purposing component proposed. The user can still make manual adjustments if so desired. Through this process, the video content is optimised for a particular publication vector.

Finally, the user can publish to multiple social networks connected to the Levuro publication module, which currently supports Twitter, Instagram, YouTube, Facebook, and LinkedIn. Accompanying the video, a text can be drafted. When publishing to a social media channel, the Content Wizard allows the user to schedule the post (see Fig. 10). The Content Wizard now calls the Recommendation and Scheduling component to suggest the ideal date of publication. Specifically, the prediction model is run against the text of the posting (using keyword extraction to generate a list of keywords from the text, which are treated as a “topic” for publication) and the future date is selected where the predicted frequency for mentions of that topic is highest (within the given publication date range, e.g. from 1 to 14 days in the future).

This combination of steps aided by the user-friendly interface is intended to ensure that users of the Content Wizard can still retain control over the digital video content marketing decisions while supported by the automated functionality to find relevant topics, select and re-purpose appropriate video content in their collections, and publish it on digital channels at the optimal moment, enabled by the AI and data-driven media analysis implemented in the ReTV project.