Video deepfake detection using Particle Swarm Optimization improved deep neural networks

One of the first end-to-end trainable architectures for video classification using CNN and RNN was proposed in 2015 by Liang and Hu [7]. Their work exploited recurrent CNN (RCNN) for undertaking object recognition. In 2016, Donahue et al. [8] studied Long-Term Recurrent Convolutional Network (LRCN), where a CNN processed raw visual input and fed it to a stack of recurrent sequential models for spatial–temporal feature extraction. Specifically, the LRCN model adopted CNNs to learn visual features from video frames and passed a sequence of image embeddings through Long Short-Term Memory (LSTM) networks for video classification.

In 2018, after the first deepfakes appeared, the idea to use the hybrid architecture for deepfake detection was first researched and published combining the advantageous characteristics of the RNN to enhance the performance of the CNN. According to Sabir et al. [2], the body of literature that has been most explored to gain insight about video classification for deep fake detection was video action recognition [9‐12] because of the extensive development in the field in recent years and similar spatial–temporal processing nature to that of deepfake detection. One of the main methods of human action recognition is a “two-stream" network methodology, which processes video frames and optical flow in two different branches before fusing them for video classification. [13] presented a deepfake detection with this two-stream technique. In addition, an RCNN model was employed by [2] for deepfake detection. It processed each frame using a CNN, and the extracted spatial features were further processed using an RNN for video forgery identification. Other strategies utilized biological signals and attention layers [14, 15] to enhance the efficiency for manipulated video detection. In particular, these techniques paid special attention to lip movements and eye gaze to check for inconsistencies.

Moreover, Sabir et al. [2] focused on using a CNN followed by a recurrent model with the input as the query video frame sequences. Their model exploited frame-to-frame temporal differences. Their work claimed that since image manipulations were conducted frame-by-frame and temporal discrepancies were expected, low-level face manipulation techniques should show temporal artefacts with inconsistent features across frames. Their work thus aimed to identify such temporal inconsistencies. Specifically, they used a DenseNet to extract features like discontinuous jawlines and blurred eyes and then retrieved the RNN’s final output rather than averaging recurrent features across all time steps as in traditional video classification pipelines.

Hyperparameter configurations of deep neural networks have significant effects in reducing or preventing oscillations in gradient descents as well as correcting gradient directions for weight adjustment towards global optima. The local optima, plateau and saddle points in the loss space are the major challenges that deep networks encounter. If hyperparameters of deep neural networks are not appropriately optimized, the networks’ performance will be affected significantly by the above factors. Although methods, such as grid and random search, work well for hyperparameter search with discrete values in a small search space, there are other more effective methods like employing swarm-based metaheuristic methods to determine optimal hyperparameter configurations specially in a continuous large search space like loss spaces in deep networks. We explore such an option through an evolutionary algorithm called PSO, which is simple to implement and has been proven by the literature to produce great robustness for learning configuration selection in neural networks.

The evolutionary algorithms are optimization methods that take inspiration from biological processes. The PSO algorithm was proposed by [16] in 1995, which takes inspiration from fish or bird swarm movement.

Because PSO does not rely on gradient descent, one can use an objective function to optimize deep network parameters without relying on its derivatives [9]. In this work, hyperparameters of deep learning models such as the learning rate, dropout rate, image input size and number of frames extracted from videos will be optimized. The objective function will be associated with loss function of the deep learning model to advise search of optimal learning settings.

The way PSO works is by initializing a group of particles in the search space of the function randomly, and at each iteration it checks which particle achieves the lowest value (i.e. the most optimal loss) on the objective function. At each following step, each particle uses the information of the best solutions found by the swarm and itself, along with random exploration factors, to guide the particle’s movement [16].

Taking k as the iteration number, the velocity of a given particle i is given by:whereThe position of the particle i at iteration \(k+1\) will be updated with the velocity as follows:The inertia weight controls how much of the particle’s previous velocity is kept in the update. A greater w setting indicates that the particle’s previous velocity has strong effects to the new velocity generation. The cognitive and social weights define how much the particle is impacted by its own best past experiences (pbest) and the best experiences of the other particles in the swarm (gbest), respectively [17]. In general, the values of \(c_1\) and \(c_2\) should be greater than 0, and less than or equivalent to 2.5. Setting these values too low may lead the particles to fail to sufficiently explore the search space, whereas setting them too high may cause the particles to become extremely sensitive to changes in the swarm and exhibit suboptimal behaviour.

In short, PSO is a powerful optimization technique that has been used to solve a wide range of optimization problems. The velocity update formula is critical in establishing how the swarm particles move and update their positions in search of a satisfactory optimal solution. Variant methods have also been proposed to tackle local optima traps of the original PSO algorithm, which were widely adopted in hyperparameter and architecture search in deep networks [18‐20].

There are inspiring related studies for hyperparameter and architecture search using multi-task learning. For example, automatic generation of multi-task learning models was conducted by Zhang et al. [21] for solving a variety of semantic segmentation problems. The automation process utilized a randomly assigned backbone network in conjunction with a set of tasks as inputs with the attempt to generate a multi-task model with a reasonable trade-off between performance and cost. A gradient-based search method was used for architecture search. The optimization process determined the assignment of different network nodes for each task and how these selected nodes were shared with other tasks. A unique characteristic of their work was the adoption of parameter sharing at the operator (neuron) level via a joint optimization of shared policies and network weights. Their yielded multi-task model showed great capabilities in tackling diverse multi-class semantic segmentation problems. In addition, automated production of search parameter and search mechanisms for metaheuristic algorithms was exploited by Stützle and López-Ibáñez [22]. Such techniques were capable of developing optimizers with effective new search strategies. They also showed great efficiency in enhancing existing search methods’ performance via optimal parameter selection. Furthermore, an adaptive hybridized multi-task learning framework was developed by Lialestani et al. [23] pertaining to temperature prediction at different depth levels. Their work performed architecture generation of a multi-task multilayer perceptron neural network using a FA variant developed by Shahri et al. [24]. The FA variant conducted multi-task network architecture generation, where the absorption and randomization parameters were fine-tuned by the population brightness variance.

Cheng et al. [25] developed a multi-task learning model with a hybrid CNN-transformer encoder for simultaneous image segmentation and classification using multimodal MRI image inputs. A U-Net-like encoder-decoder architecture was proposed with an additional transformer unit embedded in the bottom of the CNN-based encoder. The hybrid CNN-transformer encoder fused high-level spatial and global features extracted by a CNN-stream and a transformer-based operation, respectively. The joint learning of both segmentation and classification tasks was conducted via a compound loss function integrating segmentation and classification losses with uncertain weights. To tackle data sparsity and unlabelled data, a semi-supervised joint learning mechanism was deployed to enhance classification performance by integrating with uncertainty-based label selection.

Besides PSO, in recent years, a number of new state-of-the-art swarm intelligence algorithms have been proposed including Spotted Hyena Optimizer, Symbiotic Organisms Search, Tree Seed Algorithm, Sparrow Search Algorithm and Tunicate Swarm Algorithm, for tackling engineering, mathematics and image processing optimization problems. Proposed by Cheng and Prayogo [26], Symbiotic Organisms Search employs mutualism, commensalism and parasitism processes to simulate mutual interaction of two organisms to lead the search of global optimality. Firstly, during mutualism, the mean position vector of the current search agent and another randomly selected organism is calculated, which is used in conjunction with the global best solution to update the positions of both the current and randomly selected individuals. Their offspring solutions are used to replace them if the new solutions are fitter. Secondly, for the commensalism stage, the difference between the global best solution and a randomly selected individual is used to guide the movement of the current search agent. Subsequently, the parasitism operation randomly mutates the dimensions of the current search agent, which is used to substitute a randomly selected organism if this mutated solution is fitter. The effectiveness of Symbiotic Organisms Search was evidenced by its capabilities in handling diverse engineering and benchmark optimization problems, as indicated in a related study [27]. Modified Symbiotic Organisms Search algorithms and its hybridation with other search methods were also extensively studied in [27] to guide future development. Motivated by the cluster hunting behaviours of the spotted hyenas, the search process of Spotted Hyena Optimizer [28] comprises encircling/hunting and attack mechanisms. The encircling/hunting operation performs local exploitation and intensifies the search around the global best solution. Specifically, the leader spotted hyena with the best fitness score is used to re-allocate the remaining spotted hyenas to its optimal neighbouring regions. Those spotted hyenas with high correlations to the leader form a cluster where their mean position is used to generate a new swarm leader. Adaptive search coefficients are exploited to balance local and global search operations. Diverse Spotted Hyena Optimizer variant methods including the integration with other swarm intelligence algorithms such as PSO and Simulated Annealing (SA) as well as other local/global search strategies were extensively studied in Ghafori and Gharehchopogh [29]. Their flexibilities were further demonstrated in solving a variety of complex single and multi-objective optimization problems [29].

A Sparrow Search Algorithm was developed by Xue and Shen [30] where the population was composed of top ranking producer and lower ranking scrounger subswarms. In addition, 10%-20% of the sparrows are capable of perceiving danger. The top ranking producers perform global exploration when a randomly generated alarm coefficient is lower than the pre-defined safety threshold, otherwise the producer subswarm conducts local exploitation using Gaussian distribution. The lower ranking scrounger subswarm is guided by the producers to exploit optimal local regions of the respective producers. The sparrows with the capabilities of sensing danger follow the swarm leader while staying away from the global worst solution. The algorithm outperformed other classical search methods for solving a number of numerical optimization problems. To overcome slow convergence of the model, a number of variant methods of the Sparrow Search Algorithm were studied by Gharehchopogh et al. [31]. These include the incorporation of PSO, Firefly Algorithm (FA) [32], Differential Evolution (DE) and Since Cosine Algorithm (SCA) [33] with Sparrow Search Algorithm, respectively. Other enhancement mechanisms such as random walk based on Levy flights and chaotic map-based swarm initialization are also exploited to increase search robustness. Related studies of neural architecture and hyperparameter search using the Sparrow Search Algorithm were also investigated in [31]. A Tree Seed Algorithm was exploited by Kiran [34]. A swarm of tree solutions is randomly initialized. For each tree, a number of seed solutions are generated. Specifically, each new seed solution is generated using two sub-dimension-based search operations. One is guided by the best tree solution and a randomly selected tree position while the other is led by the current tree position and a randomly selected tree location. For the generation of a specific dimension of a seed solution, the selection of these two search strategies is controlled by a randomly generated threshold parameter. The number of the new seed solutions that can be produced for each tree is dynamic between 10% and 25% of the population size, in order to increase search exploitation. If the best offspring seed solution is fitter than the tree solution, it is used to substitute the tree solution. The algorithm obtained competitive performance in comparison with other search methods such as PSO and FA for solving 24 numerical test functions. A comprehensive survey of the Tree Seed Algorithm was conducted by [35] where a variety of variants of Tree Seed Algorithm were analysed. The variant methods included the combination of Tree Seed Algorithm with other swarm intelligence algorithms such as Artificial Bee Colony (ABC) [36] and SCA. Improvement strategies such as Levy and Gaussian distributions were also utilized to enhance flexibility of Tree Seed Algorithm. Moreover, the effectiveness of Tree Seed Algorithm was also ascertained by handling a variety of real-world optimization problems such as feature selection and image compression. A variant method of Tunicate Swarm Algorithm was studied by Gharehchopogh [37], which included Quantum Rotation Gate (QRG) and mutation operators based on Cauchy, Gaussian and Levy distributions to increase search robustness of the original method. In particular, besides using QRG, their work explored the effectiveness of the combinations of any two out of the three mutation operators as well as the integration of all three random walk strategies. The superiority of the full model integrating all mutation operators along with QRG was ascertained by solving a set of 52 unimodal, multimodal, composition and hybrid test functions, as well as several other engineering optimization problems.

A new FA variant was developed by Shahri et al. [24] by incorporating a brightness expectation value and a generalized weighted average of a random brightness. It exploited an adaptive absorption coefficient and an adaptive randomization search step to better balance the search between intensification and diversification. The population fitness variance was used to adjust these adaptive search parameters after a number of iterations, which was calculated using the difference between the fitness of each firefly and the mean fitness of the overall swarm, divided by a dynamic normalization factor. Owing to the adaptive adjustment of the search parameters based on the fitness variations during the search process, their method showed better capabilities in overcoming local optima traps in comparison with FA for solving several benchmark functions as well as multi-objective blasting engineering problems.

Motivated by foraging behaviours of social spiders, Social Spider Optimizer (SSO) [38] first generates a vibration intensity of each spider whereby a better fitness score aligns with a stronger vibration. The strongest vibration intensity generated by other spiders and sensed by the current spider is extracted. A randomly generated binary mask is used to select either this new best vibration intensity or another vibration intensity generated by a random individual in each dimension for the construction of new personal leader signal. This new elite leader signal is used to guide a random walk operation for position updating. Boundary checkings are also performed after position updates. SSO shows competitive performance as compared with a number of state-of-the-art search algorithms for tackling diverse numerical optimization problems. Besides the above, there are also other swarm intelligence algorithms developed for handling feature selection, hyperparameter search, deep neural architecture generation with respect to image segmentation/classification [19, 39‐41], human action recognition [42] and environmental sound classification [18], as well as solving other engineering and mathematical optimization problems [43‐49].

The initial stage of the training pipeline involves extracting and pre-processing the first 150 frames of each video. The Python OpenCV library was used to extract the image frames, and then, the faces on each frame were processed through the Multi-task cascaded CNN (MTCNN) face detector [50] for cropping. After that, the face crops were organized into folders and saved as image files within the file system. In particular, the cropped facial regions from the real videos are augmented during training by flipping them horizontally to increase real sample sizes. Figure 1 shows the detailed preprocessing pipeline for face cropping.

Proposed by Zhang et al. [50], the MTCNN model is used for face detection. Specifically, the model is able to perform face classification, facial region bounding box generation and facial alignment. MTCNN firstly deploys a proposal CNN to perform binary (face and non-face) classification and generate a number of candidate bounding box regression vectors. The nonmaximum suppression (NMS) method is used to merge highly overlapped bounding boxes. A second CNN model is subsequently utilized to further refine the bounding box regression results by rejecting remaining candidate false positives. A third comparatively deeper CNN is used in this stage to determine the final bounding box output as well as generate a set of facial landmarks indicating positions of both eye centres, left and right mouth corners and the nose tip. The MTCNN model outperformed other face detection benchmarks while maintaining efficient computational cost.

In this research, we employ MTCNN to perform real-time facial bounding box regression for all sampled frames extracted from the video, without using associated facial landmark outputs. The detected facial regions determined by the bounding box regression vectors are cropped out for subsequent classification. Owing to the fact that a region of interest containing the entire face is tracked through the overall video using bounding boxes, the false positives of face classification and localization are greatly reduced. This in turn significantly improves real and manipulated video classification performance. Figure 2 shows the results for face detection and cropping for a sample video.

We use two well-known video deepfake datasets, i.e. Celeb-DFv2 and DFDC, for model evaluation. Precisely, the Celeb-DFv2 dataset [51] consists of 590 genuine and 5,639 fake videos. The official split of the dataset shows 5,711 and 518 videos for training/validation and test, respectively. We adopt this official split in our experiment.

The DFDC dataset [52] has a total of 23,654 real and 104,500 synthetic videos. We extract a subset of 1,016 original and 8,425 tampered videos in our experiments. A test set of 206 real and 1,636 fake videos is used for testing with the remaining videos for training and validation in our experiment. We further split the training and validation sets using a ratio of 80-20.

Besides the evaluation of each of the above datasets, we also generate a customized dataset by combining the above two datasets with a video face recognition database, i.e. YouTube Faces Database [53], for model evaluation. The YouTube Faces Database is designed for video face recognition and consists of 3,425 videos from 1,595 subjects, with an average of 181.3 frames per video. It is used to increase the genuine video sample sizes to balance the large numbers of fake instances provided by Celeb-DFv2 and DFDC. To be specific, at the training stage, all real videos from the official training set in Celeb-DFv2, a comparatively larger number of real videos from DFDC, as well as 1,618 videos from the Youtube Faces Database with more than 50 frames, were combined together to construct the customized genuine training video set. In addition, a balanced number of fake videos are drawn from the official training set of Celeb-DFv2 and our DFDC subset in order to obtain a ratio of approximately 50%-50% between fake and real videos, in the constructed training set. We further split the combined training set by a ratio of 80-20 for training and validation, respectively. The real and fake samples from the official test set of Celeb-DFv2 and a comparatively larger DFDC unseen test set are used for model evaluation in this experiment.

Table 1 shows the detailed training/validation and test sample sizes for each dataset.

The final step was to send the data to the Pytorch dataloader so that the models could be trained and validated for each experimental setting. During the training process, the transformations serve to supplement the data by changing each frame from real videos at each epoch with a random component. This is accomplished through the use of augmentation. For the training dataset, we initially used random rotation with 20 degrees and gaussian blur. The images were initially resized and then normalized before being used in either the training/validation sets or the test set. For the oversampling of frames from real videos, the RandomHorizontalFlip() function is utilized.

We firstly employ transfer learning using a CNN model with the EfficientNet architecture for deepfake detection. Figure 3 shows the overall dataflow using transfer learning for synthetic video classification.

EfficientNet was designed with the goal of scaling CNNs more efficiently than other deep networks proposed previously [54]. Since its inception, this CNN architecture has shown to be among those that achieve the highest performance when tested against various image classification benchmarks.

EfficientNet makes use of a compound scaling strategy, which involves scaling the network’s width, resolution and depth uniformly to whatever degree is required to make optimal use of the computational resources available. A grid search is usually used to find the scaling constants [54].

The network design of EfficientNet makes use of mobile inverted bottleneck convolution (MBConv), which is analogous to MobileNetV2 convolutional block but slightly larger. In order to maximize precision and FLOPS, a neural architecture search was employed in the construction of the baseline model. After that, a family of EfficientNet models was obtained by scaling it up using such a strategy. Within the context of this research, the version known as EfficientNet-B0 was employed [54]. The overall architecture specifically used in this study including the fully connected layers is shown in Table 2 below. The pure EfficientNet was also used by the winning solution of the Deepfake Detection Tournament hosted by the DFDC dataset authors in 2019 [52].

The MBConv blocks consist of residual blocks like ResNet that connect the beginning of the block with the end using a skip connection. The difference from the original block from ResNet is that, regarding the number of channels, it follows a narrow-wide-narrow approach instead of the traditional wide-narrow-wide strategy [54].

The EfficientNet-B0 model was initially trained using ImageNet. We further fine-tune the model using the training/validation sets of the frames of each deepfake dataset in our experiments. The fine-tuned model is used for the identification of fake/real videos. In addition, a new PSO variant is used to fine-tune network hyperparameter, i.e. learning rate, dropout rate, image size and number of frames, with the attempt to further enhance performance.

Specifically, a random swarm is firstly initialized in a search space of [0, 1]. Each particle has four dimensions to represent the four optimized hyperparameters. The proposed PSO search operations are used to guide the particle movement in the search space for hyperparameter search. We evaluate each particle’s fitness by converting its position into valid network learning configurations, which are used to set up transfer learning process. The network performance on the validation set is used as the fitness measure of each particle. The most optimal solution identified by the proposed optimizer is used as the recommended best learning configurations of EfficientNet-B0. The optimized EfficientNet-B0 model is then trained using the combined training set with larger numbers of epochs and tested with the respective test sets for deepfake detection.

Besides using transfer learning for video forgery detection, motivated by [2, 55‐57], a hybrid CNN-RNN architecture is proposed in this research for distinguishing fake from real videos. Specifically, this hybrid model uses EfficientNet serialized with a GRU layer for spatial–temporal feature extraction to inform video classification. The proposed PSO algorithm is used for identifying optimal hyperparameters. Figure 4 shows the system dataflow. The detailed network architecture is shown in Table 3.

As shown in Table 3, a latent representation of 1,280 dimension output extracted from the last convolutional layer of EfficientNet is taken as input from each frame and these features of each frame are concatenated to be passed on to the GRU layer. The GRU layer is used to take advantage of the spatial temporal features from the sequence of frames verifying if it is either a deepfake or a real video. The process is different from the pure CNN (i.e. the aforementioned EfficientNet) that takes the average of all frames for deepfake detection as in the transfer learning process. The GRU component has thus 1,280 latent dimensions and 1,280 hidden layers as the layer configurations in this study.

The optimizer used is ADAM that combines features of optimization algorithms such as RMSProp and ADAGrad. It is used to adjust the learning rate for each weight on the fly using exponential weighted moving average to get the first and second moments of the gradient estimates [56]. The loss function opted is cross-entropy loss. It gives two likelihoods for real and fake labels using a softmax function [56]. In addition, to improve discriminative feature learning, we fine-tune the weights of ImageNet pre-trained EfficientNet-B0 embedded in the proposed EfficientNet-GRU model using the combined training set with a small number of epochs (i.e. 5 epochs), before passing on features to the GRU layer. Moreover, the proposed PSO model is used to fine-tune hyperparameters of this hybrid network during the training stage, similar to the process discussed earlier for parameter search using transfer learning. Specifically, we optimize the learning rate, dropout rate, image size and number of video frames, owing to their significance to network performance.

A new PSO variant is proposed for hyperparameter search for both EfficientNet-GRU and EfficientNet-B0 in this research. In order to tackle limitations of the original PSO algorithm, it incorporates nonlinear functions for composite leader generation and a reinforcement learning strategy for dynamically adjusting the search process. As such, different search actions led by different hybrid leaders and the global best solution are dynamically dispatched based on the reward schemes of the reinforcement learning algorithm. Figure 5 shows the overall proposed algorithm. The detailed search strategies are presented below.

In the initial experiments, the models were trained with hyperparameter searched manually. The process entails individually experimenting with a range of hyperparameters selected. The following hyperparameters are optimized:The training and validation sets of the combined dataset are used for hyperparameter search. For each of the aforementioned hyperparameters, a set of three values was chosen in order to manually fine-tune the model and identify the configuration with the lowest loss. The process was repeated for each one of the four hyperparameters. The values that composed the set are illustrated in Table 4.

A total of 5 epochs were run to obtain the loss value for each hyperparameter set, as this was the number of epochs that demonstrated satisfactory stability in the preliminary results. To choose the optimal ones in each of them, the algorithm was run through their possible values, while the other hyperparameters remained constant. The parameter setting with the smallest loss error was then selected. These manually selected optimized settings are used to set up each network, which is further tested with test sets of Celeb-DFv2, DFDC and the combined datasets, respectively.

Besides manual hyperparameter selection, automatic hyperparameter search is also performed. We employ the proposed PSO model, as well as 8 classical search methods and 4 PSO variant algorithms, for hyperparameter search, including PSO, ABC [36], Salp Swarm Algorithm (SSA) [44], SSO [38], Bare-bones PSO (BBPSO) [63], Flower Pollination Algorithm (FPA) [64], FA [32], Dragonfly Algorithm (DA) [65], Genetic PSO (GPSO) [40], PSO with sine coefficients (SPSO) [20], a BBPSO variant with attractiveness and evade actions (BBPSOV) [49] and PSO with adaptive sine, circle and spiral coefficients (ACPSO) [66]. We adopt variable settings of the above algorithms from their original studies in our experiments.

The following experimental settings are adopted. For each search method, a total of 10 search agents are created and the maximum number of 20 generations is used for hyperparameter search. All the search methods perform the same number (i.e. 200) of function evaluations. A set of 5 runs was conducted for each search method. For both EfficientNet-B0 and EfficientNet-GRU, Table 9 shows the search ranges of different hyperparameters. These search ranges are obtained via trial-and-error as discussed in Section 4.1. A set of five evaluation metrics, i.e. the mean precision, recall, accuracy, AUC scores and the Wilcoxon rank sum (RS) test, is used for performance comparison. The details of the experimental studies are presented as follows.

Again the training and validation sets of the combined dataset are used for the proposed PSO-based hyperparameter search, owing to their representative capabilities. Because of the large sample sizes of this combined video deepfake dataset, in order to reduce the high computational cost of hyperparameter search, the training and validation sets of the combined dataset were sampled for 5% and 25%, respectively. The validation cross-entropy loss function was used as the objective function for evaluating each particle. The experiments ranged from 10 to 20 hours for each of the two models using the sampled training and validation sets for each hyperparameter search.

We conduct performance comparison between our optimized EfficientNet-B0 and EfficientNet-GRU models and other networks including ResNet50-GRU, ResNet101-GRU, GoogLeNet-GRU, as well as 3D CNNs such as Inflated-3D (I3D), a Mixed Convolution Network (MC3), 3D ResNeXt101 and 3D ResNeXt50, using the Celeb-DFv2, DFDC and combined datasets, respectively. These baseline state-of-the-art networks are selected because of their significant discriminative capabilities in video classification [9, 42, 60, 67]. Similar to our work, for the hybrid networks, i.e. ResNet50-GRU, ResNet101-GRU and GoogLeNet-GRU, the respective CNN (ResNet50, ResNet101 and GoogLeNet) models are pretrained using ImageNet and their successive GRU models are trained from scratch using our combined deepfake training set.

In addition, for 3D CNNs, 3D convolutions instead of 2D convolutions are used for feature learning, except for MC3 where mixed convolutions are used. Precisely, MC3 employs 3D convolutions in first two groups and 2D convolutions from group 3 onwards [68]. 3D ResNeXt101 and 3D ResNeXt50 are variants of 3D ResNet, which introduce a cardinality parameter to control the number of parallel paths within each residual block [69]. All the above 3D CNNs are pre-trained using a large human action dataset, i.e. Kinetics, consisting of 306,245 videos from 400 classes, then fine-tuned using the combined deepfake training set for real/manipulated video classification.

All the selected hybrid and 3D CNN baseline networks are equipped with the following learning settings, i.e. learning rate = 0.0001, dropout rate = 0.3, image size = 112 and number of frames = 40. Tables 18, 19 and 20 show the detailed evaluation results, while Figs. 16, 17 and 18 illustrate respective ROC curves, for the Celeb-DFv2, DFDC and combined test sets, respectively.

As indicated in Tables 18, 19 and 20 and Figs. 16, 17 and 18, the proposed PSO-optimized EfficientNet-B0 and EfficientNet-GRU models show competitive performances as compared with those of the aforementioned three hybrid and four 3D CNN models, across datasets. The proposed optimizer employs multiple composite elite signals and Q-learning-based search operation allocation with bespoke search behaviours of each particle, to boost model capabilities in tackling stagnation. Our optimized networks are thus equipped with better learning settings and show enhanced capabilities in spatial–temporal feature learning for video forgery classification. In addition, ResNet101-GRU shows better results than those of ResNet50-GRU and GoogLeNet-GRU, because of its more effective feature learning capabilities using deeper residual blocks. Among the 3D CNNs, I3D illustrates more discriminative capabilities by inflating all the filters and pooling kernels in a 2D architecture through the insertion of an additional temporal dimension and thus achieves the most reliable performance. It outperforms MC3, 3D ResNeXt101 and 3D ResNeXt50 for most test scenarios.

The confusion matrices of the proposed PSO-optimized EfficientNet-B0 and EfficientNet-GRU models with respect to the Celeb-DFv2, DFDC and combined test sets are provided in Figs. 19 and 20, respectively. The built-in scikit-learn packages in the Python library are used to generate these confusion matrix results, along with those for other evaluation metrics (i.e. accuracy, precision, recall and AUC scores) in this research.

Since all the search methods employ the same number of function evaluations for hyperparameter search, with deep learning-based fitness function evaluation as the most costly process, these methods have a similar overall cost for optimal parameter selection at the training stage. We provide the average cost of each algorithm with one function evaluation over 5 runs for computational efficiency comparison. Such a mean cost for each trial is calculated by averaging the cost for hyperparameter search by the number of function evaluations performed. This includes the cost of dedicated search operations embedded in each algorithm along with one fitness evaluation of the recommended hyperparameters using either EfficientNet-B0 or EfficientNet-GRU. The cost variations are mainly caused by different search principles operated in the search methods. Table 21 depicts the detailed cost comparison.

We conduct the computational cost comparison using a NVIDIA RTX 3090 consumer GPU. As indicated in Table 21, the proposed model shows moderate mean computational costs per function evaluation over 5 runs for both networks. ABC, SSA, FA, FPA and BBPSO have lower mean computational costs owing to their comparatively simpler search strategies by following randomly selected (ABC), neighbouring (SSA and FA) and global best (BBPSO and FPA) solutions, respectively. DA employs a search action combining separation, alignment, cohesion, attraction and evading mechanisms, also showing relatively light computational costs. SSO performs vibration intensity-based actions with dynamic leader generation, resulting in slightly higher but reasonable costs. In contrast, BBPSOV and ACPSO have the highest computational costs due to diverse embedded search actions, e.g. evading/attraction-inspired mechanisms in BBPSOV, and three subswarms with adaptive sine, circle and spiral search coefficients in ACPSO. The proposed model, GPSO and SPSO show moderate costs because of the deployment of reinforcement learning-based action selection and hybrid leader generation in our model, crossover and mutation operations in GPSO, and adaptive sine-based search coefficients in SPSO, respectively. The optimal configurations identified by each method in the training process are used to construct the final network at the test stage for performance comparison.

We also compare our optimized networks with other existing studies for Celeb-DFv2 and DFDC datasets. The selected existing studies were evaluated using the official Celeb-DFv2 test set and the DFDC subset, respectively, as performed in our experiments. But since these related studies were trained using a variety of deepfake training databases for evaluating both datasets, they are used for loose performance comparison. Tables 22 and 23 illustrate the performance comparison with state-of-the-art existing studies using the Celeb-DFv2 and DFDC datasets, respectively.

We used the official split to evaluate the Celeb-DFv2 dataset. Specifically, for the official Celeb-DFv2 test set, the selected existing studies shown in Table 22 obtained accuracy rates ranging from 0.8074 to 0.8989 and AUC scores ranging from 0.696 to 0.9003. Our transfer learning using EfficientNet-B0 with the proposed PSO-based hyperparameter fine-tuning obtained a competitive benchmark with an accuracy rate of 0.9247 and an AUC score of 0.8985, while EfficientNet-GRU with the proposed PSO-based hyperparameter selection achieved a better accuracy rate of 0.9382 and a better AUC result of 0.9141. Both of our models outperform most of these related studies by a sufficient margin.

Table 23 shows the comparison between our optimized networks and existing studies for the DFDC test set. Again our optimized EfficientNet-GRU and EfficientNet-B0 models with the proposed PSO-based hyperparameter fine-tuning obtain more reliable performance in comparison with those of existing studies. Specifically, the EfficientNet-B0 model with the proposed PSO-based hyperparameter selection achieves a competitive mean accuracy rate of 0.9414 and AUC score of 0.9161, and our optimized EfficientNet-GRU obtains a better mean accuracy rate of 0.9517 and a better AUC score of 0.9346.

In short, owing to the efficient search capabilities of the proposed PSO model guided by composite leaders and optimized search strategies governed by the fitness evaluations, our optimized networks outperform existing studies for both Celeb-DFv2 and DFDC datasets and show great efficiency in tackling challenging manipulated video classification tasks. They can be deployed as effective substitute methods for deepfake content identification. In addition, our work also highlights the importance of hyperparameter selection in deep learning networks and the potential use of evolutionary algorithms in such tasks for video deepfake classification.