Realistic Speech-Driven Facial Animation with GANs

The relationship between speech and facial motion has been exploited by some CG methods, which assume a direct correspondence between basic speech and video units. Cao et al. (2005) build a graph of visual representations called animes which correspond to audio features. The graph is searched in order to find a sequence that best represents a given utterance under certain co-articulation and smoothness constraints. Additionally, this system learns to detect the emotion of the speech and adjust the animes accordingly to produce movements on the entire face. The final result is obtained by time-warping the anime sequence to match the timing of the spoken utterance and blending for smoothness. Such methods use a small set of visual features and interpolate between key frames to achieve smooth movement. This simplification of the facial dynamics usually results in unnatural lip movements, which is why methods that attempt to model the facial dynamics are preferred over these approaches.

Some of the earliest methods for facial animation relied on Hidden Markov Models (HMMs) to capture the dynamics of the video and speech sequences. Simons and Cox (1990) used vector quantization to achieve a compact representation of video and audio features, which were used as the states for their fully connected Markov model. The Viterbi algorithm was used to recover the most likely sequence of mouth shapes for a speech signal. A similar approach is used in Yamamoto et al. (1998) to estimate the sequence of lip parameters. Finally, the Video Rewrite method (Bregler et al. 1997) relies on the same principles to obtain a sequence of triphones, which are used to look up mouth images from a database. The final result is obtained by time-aligning the images to the speech and then spatially aligning and stitching the jaw sections to the background face.

Since phonemes and visemes do not have a one-to-one correspondence some HMM-based approaches replace the single Markov chain approach with a multi-stream approach. Xie and Liu (2007) propose a coupled HMM to model the audio-visual dependencies and compare the performance of this model to other single and multi-stream HMM architectures.

Although HMMs were initially preferred to neural networks due to their explicit breakdown of speech into intuitive states, recent advances in deep learning have resulted in neural networks being used in most modern approaches. Like past attempts, most of these methods aim at performing a feature-to-feature translation. A typical example of this, proposed in Taylor et al. (2017), uses a deep neural network (DNN) to transform a phoneme sequence into a sequence of shapes for the lower half of the face. Using phonemes instead of raw audio ensures that the method is subject independent.

Most deep learning approaches use convolutional neural networks (CNN) due to their ability to efficiently capture useful features in images. Karras et al. (2017) use CNNs to transform audio features to 3D meshes of a specific person. This system is conceptually broken into sub-networks responsible for capturing articulation dynamics and estimating the 3D points of the mesh.

Analogous approaches,which are capable of generating facial descriptors from speech using recurrent neural networks (RNNs) have been proposed in Fan et al. (2015), Suwajanakorn et al. (2017), Pham et al. (2017). In particular, the system proposed in Suwajanakorn et al. (2017) uses Long Short Term Memory (LSTM) cells to produce mouth shapes from Mel-Frequency Cepstral Coefficients (MFCCs). For each generated mouth shape a set of best matching frames is found from a database and used to produce mouth images. These mouth shapes are blended with the frames of a real target video to produce very realistic results.

Although visual features such as mouth shapes and 3D meshes are very useful for producing high quality videos they are speaker specific. Therefore, methods that rely on them are subject dependent and require additional retraining or re-targeting steps to adapt to new faces. For this reason methods like the one proposed in Zhou et al. (2018) use speaker independent features such as visemes and Jaw and Lip (JALI) parameters.

Finally, Chung et al. (2017) proposed a CNN applied on MFCCs that generates subject independent videos from an audio clip and a still frame. The method uses an \(L_1\) loss at the pixel level resulting in blurry frames, which is why a deblurring step is also required. Secondly, this loss at the pixel level penalizes any deviation from the target video during training, providing no incentive for the model to produce spontaneous expressions and resulting in faces that are mostly static except for the mouth.

The recent introduction of GANs in Goodfellow et al. (2014) has shifted the focus of the machine learning community to generative modelling. GANs consist of two competing networks: a generative network and a discriminative network. The generator’s goal is to produce realistic samples and the discriminator’s goal is to distinguish between the real and generated samples. This competition eventually drives the generator to produce highly realistic samples. GANs are typically associated with image generation since the adversarial loss produces sharper, more detailed images compared to \(L_1\) and \(L_2\) losses. However, GANs are not limited to these applications and can be extended to handle videos (Mathieu et al. 2015; Li et al. 2017; Vondrick et al. 2016; Tulyakov et al. 2018).

Straight-forward adaptations of GANs for videos are proposed in Vondrick et al. (2016) and Saito et al. (2017), replacing the 2D convolutional layers with 3D convolutional layers. Using 3D convolutions in the generator and discriminator networks is able to capture temporal dependencies but requires fixed length videos. This limitation was overcome in Saito et al. (2017) but constraints need to be imposed in the latent space to generate consistent videos. CNN based GAN approaches have been used for speech to video approaches such as the one proposed in Zhou et al. (2019).

The MoCoGAN system proposed in Tulyakov et al. (2018) uses an RNN-based generator, with separate latent spaces for motion and content. This relies on the empirical evidence shown in Radford et al. (2015) that GANs perform better when the latent space is disentangled. MoCoGAN uses a 2D and 3D CNN discriminator to judge frames and sequences respectively. A sliding window approach is used so that the 3D CNN discriminator can handle variable length sequences. Furthermore, the GAN-based system proposed in Pham et al. (2018) uses Action Unit (AU) coefficients to animate a head. A similar approach is used in the GANimation model proposed in Pumarola et al. (2018). These approaches can be combined with speech-driven animation methods (Pham et al. 2017) that produce AU coefficients which drive facial expressions from speech.

GANs have also been used in a variety of cross-modal applications, including text-to-video and audio-to-video. The text-to-video model proposed in Li et al. (2017) uses a combination of variational auto encoders (VAE) and GANs in its generating network and a 3D CNN as a sequence discriminator. Finally, Chen et al. (2017) propose a GAN-based encoder-decoder architecture that uses CNNs in order to convert audio spectrograms to frames and vice versa. This work is extended in Chen et al. (2019), using an attention mechanism which helps the network focus on frame regions that correlate highly with the audio. However as a result this method neglects other areas such as the brow and eyes.

The generator accepts as input a single image and an audio signal, which is divided into overlapping frames corresponding to 0.2 s. Each audio frame must be centered around a video frame. In order to achieve this one-to-one correspondence we zero pad the audio signal on both sides and use the following formula for the stride:The generator network has an encoder-decoder structure and can be conceptually divided into sub-networks as shown in Fig. 3. We assume a latent representation that is made up of 3 components which account for the speaker identity, audio content and spontaneous facial expressions. These components are generated by different modules and combined to form an embedding which can be transformed into a frame by the decoding network.

Our system uses multiple discriminators in order to capture different aspects of natural videos. The Frame Discriminator achieves a high-quality reconstruction of the speakers’ face throughout the video. The Sequence Discriminator ensures that the frames form a cohesive video which exhibits natural movements. Finally, the Synchronization Discriminator reinforces the requirement for audio-visual synchronization.

The Frame discriminator (\(D_{img}\)) is trained on frames that are sampled uniformly from a video x using a sampling function S(x). Using the process shown in Fig. 6 we obtain in and out of sync pairs \(p_{in}\), \(p_{out}\) from the real video x and audio a and a fake pair \(p_{f}\). We use these pairs as training data for the Synchronization discriminator (\(D_{sync}\)). Finally the Sequence Discriminator (\(D_{seq}\)), classifies based on the entire sequence x. The total adversarial loss \(\mathcal {L}_{adv}\) is made up of the adversarial losses associated with the Frame (\(\mathcal {L}^{img}_{adv}\)), Synchronization (\(\mathcal {L}^{sync}_{adv}\)) and Sequence (\(\mathcal {L}^{seq}_{adv}\)) discriminators. These losses are described by Eqs. 2–4. The total adversarial loss is an aggregate of the losses associated with each discriminator as shown in Eq. 5, where each loss is assigned a corresponding weight (\(\lambda _{img}\), \(\lambda _{sync}\), \(\lambda _{seq}\)).

An \(L_1\) reconstruction loss is also used to help capture the correct mouth movements. However we only apply the reconstruction loss to the lower half of the image since it discourages the generation of facial expressions. For a ground truth frame F and a generated frame G with dimensions \(W \times H\) the reconstruction loss at the pixel level is Eq. 6.The loss of our model, shown in Eq. 7, is made up of the adversarial loss and the reconstruction loss. The \(\lambda _{rec}\) hyperparameter controls the contribution of of the reconstruction loss compared to the adversarial loss and is chosen so that, after weighting, this loss is roughly triple the adversarial loss. Through fine tuning on the validation set we find that the optimal values of the loss weights are \(\lambda _{rec}= 600\), \(\lambda _{img}= 1\), \(\lambda _{sync}= 0.8\) and \(\lambda _{seq}= 0.2\). The model is trained until no improvement is observed in terms of the audio-visual synchronization on the validation set for 5 epochs. We use pre-trained lipreading models where available or other audio-visual synchronization models to evaluate the audio-visual synchrony of a video.We used Adam (Kingma and Ba 2014) for all the networks with a learning rate of 0.0001 for the Generator and Frame Discriminator. The Sequence Discriminator and Synchronization Discriminator use a smaller learning rate of \(10^{-5}\). Smaller learning rates for the sequence and synchronization discriminators are required in order to avoid over-training the discriminators, which can lead to instability (Arjovsky and Bottou 2017). The learning rate of the generator and discriminator decays with rates of \(2\%\) and \(10\%\), respectively, every 10 epochs.

In order to quantify the effect of each component of our system we perform an ablation study on the GRID dataset (see Table 5). We use the metrics from Sect. 5 and a pre-trained LipNet model which achieves a WER of \(21.76 \%\) on the ground truth videos. The average value of the ACD for ground truth videos of the same person is \(0.98 \cdot 10^{-4}\) whereas for different speakers it is \(1.4 \cdot 10^{-3}\).

The model that uses only an \(L_1\) loss achieves better PSNR and SSIM results, which is expected as it does not generate spontaneous expressions, which are penalized by these metrics unless they happen to coincide with those in ground truth videos. We also notice that it results in the most blurry images. The blurriness is minimized when using the frame adversarial loss as indicated by the higher CPBD scores. This is also evident when comparing video frames generated with and without adversarial training as shown in Fig. 10.

The Average Content Distance is close to that of the real videos, showing that our model captures and maintains the subject identity throughout the video. Based on the results of the ablation study this is in large part due to the Frame Discriminator. Furthermore, this indicates that the identity encoder has managed to capture the speaker identity. Indeed, when plotting the identity encoding (Fig. 11) of 1250 random images taken from the GRID test set using the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm Van Der Maaten and Hinton (2008) we notice that images of the same subject have neighbouring encodings. Additionally, we notice that the data points can be separated according to gender.

The Sequence Discriminator is responsible for the generation of natural expressions. To quantify its effect we compare the distribution of blinks for videos generated by the full model to those generated without the Sequence Discriminator. This is shown in Fig. 12, where it is evident that removing the sequence discriminator drastically reduces blink generation. Furthermore, we note the similarity of the generated and real distribution of blinks and blink duration. The average blink rate in videos generated by our model is 0.4 blinks/s with the median blink lasting 9 frames (0.36s). Both the average blink rate and median duration are very close to those found in the ground truth videos in Table 4.

We also notice that the removal of the sequence discriminator coincides with a an increase in PSNR and SSIM, which is likely due to the generation of blinks and head movements. We test this hypothesis by calculating the PSNR only on the lower half of the image and find that gap between the non-adversarial model and our proposed model reduces by 0.3 dB.

The effect of the synchronization discriminator is reflected in the low WER and high AV confidence values. Our ablation study shows that the temporal discriminators have a positive contribution to both the audio-visual synchronization and the WER.

Our method is capable of producing realistic videos of previously unseen faces and audio clips taken from the test set. The same audio used on different identities is shown in Fig. 13. From visual inspection it is evident that the lips are consistently moving similarly to the ground truth video.

Our method not only produces accurate lip movements but also natural videos that display characteristic human expressions such as frowns, blinks and angry expressions, examples of which are shown in Fig. 14. In these examples we highlight the regions of the frames that exhibit the most movement using motion maps. These maps are obtained by calculating the optical flow between consecutive frames, reflecting the angle of movement in the hue and assigning the magnitude of the motion to the value component in the Hue Saturation Value (HSV) color-space.

The amount and variety of expressions generated is dependent on the amount of expressions present in the dataset used for training and hence faces generated by models trained on expressive datasets such as CREMA-D will exhibit a wider range of expressions. This is illustrated in Fig. 15, where the facial expressions reflect the emotion of the speaker.

The works that are closest to ours are those proposed in Suwajanakorn et al. (2017) and Chung et al. (2017). The former method is subject dependent and requires a large amount of data for a specific person to generate videos. There is no publicly available implementation for the Speech2Vid method proposed in Chung et al. (2017) but a pre-trained model is provided, which we can use for comparison. For completeness we also compare against a GAN-based method that uses a combination of an \(L_1\) loss and an adversarial loss on individual frames. We consider this approach as the baseline GAN-based approach. Finally, we also compare with the ATVGNet model proposed in Chen et al. (2019), which is pretrained on the LRW dataset (Fig. 16).

When silent audio is provided as input to our model the lips do not form words. However, in the case where the initial frame exhibits a facial expression (i.e. smile) it is suppressed gradually over a sequence of frames. We verify this by using silent audio with a small additive pink noise to drive the generation process. The results in Fig. 17 show how smiles naturally transform to more neutral expressions. If the initial expression is neutral it is unaltered. It is important to note that videos will continue to exhibit spontaneous facial movements such as blinks even when the audio is completely silent.

Since the baseline and the Speech2Vid model are static methods they produce less coherent sequences, characterized by jitter, which becomes worse in cases where the audio is silent (e.g. pauses between words). This is likely due to the fact that there are multiple mouth shapes that correspond to silence and since the model has no knowledge of its past state it generates them at random. Figure 18 highlights such failures of static models and compares it to our method.

The Speech2Vid model only uses an \(L_1\) reconstruction loss during training. This loss penalizes spontaneous expressions which mostly occur on the upper part of the face and is therefore likely to discourage their generation. In order to examine the movement we use optical flow and create a heatmap for the average magnitude of movement over a set of 20 videos of the same subject from the LRW test set. The heatmaps shown in Fig. 19 reveal the areas of the face that are most often animated. Videos generated using our approach have heatmaps that more closely resemble those of real videos. The static baseline is characterized by considerably more motion on the face which likely corresponds to jitter. The Speech2Vid and ATVGNet models do not animate the upper part of the face. This means that that these methods do not capture speaker’s tone and cannot therefore generate matching facial expressions. An example of this shortcoming is shown in Fig. 20 where we compare a video generated from the CREMA-D dataset using the Speech2Vid model and our proposed method.

We measure the performance of our model on the GRID, TCD TIMIT, CREMA-D and LRW datasets using the metrics proposed in Sect. 5 and compare it to the baseline and the Speech2Vid model. For the LRW dataset we also compare with the ATVGNet GAN-based method proposed in Chen et al. (2019), for which we use the provided pretrained model. The preprocessing procedure for ATVGNet is only provided for the LRW dataset hence we do not compare with this model on other datasets.

The results in Table 6 show that our method outperforms other approaches in both frame quality and content accuracy. For the LRW dataset our model is better not only from the static approaches but also from ATVGNet. Our model performs similarly or better than static methods when in terms of frame-based measures (PSNR, SSIM, CBPD, ACD). However, the difference is substantial in terms of metrics that measure content such as lipreading WER. Also our method achieves a higher AV confidence, although it must be noted that based on the offset estimated using the SyncNet model our videos generated for the CREMA-D dataset exhibit a slight lag of 1 frame compared to the Speech2Vid method. Finally, we emphasize that our model is capable of generating natural expressions, which is reflected in the amount and duration of blinks (Table 6), closely matching those of the real videos, shown in Table 4.

We note that the Speech2Vid and ATVGNet methods are not capable of generating any blinks. For the Speech2Vid model this due to using only an \(L_1\) loss and for the ATVGNet this is likely due to the attention mechanism which focuses only on the mouth since it is the region that correlates with speech. The static baseline is capable of producing frames with closed eyes but these exhibit no continuity and are characterised by very short duration as shown in Table 6.

We further note the differences in the performance of our method for different datasets. In particular we note that the reconstruction metrics are better for the GRID dataset. In this dataset subjects are recorded under controlled conditions and faces are not characterised by much movement. Synthesized faces will mimic the motion that is present in the training videos, generating emotions and head movements. However since these movements cause deviation from the ground truth videos and therefore will be penalized by reference metrics such as PSNR and SSIM. Performance based on reconstuction metrics becomes worse as datasets become less controlled and exhibit more expressions. Another noteworthy phenomenon is the drop in audio-visual correlation, indicated by the lower AV confidence for the TCD TIMIT and CREMA-D datasets compared to GRID and LRW. We attribute to this drop in performance to the fact that the TCD TIMIT and CREMA-D are smaller datasets. It is therefore likely that the datasets do not have the sufficient data for the models to capture the articulation as well as for larger datasets.

Human perception of synthesized videos is hard to quantify using objective measures (Figs. 17, 18, 19, 20, 21). Therefore, we further evaluate the realism of generated videos through an online Turing test.² In this test users are shown 24 videos (12 real–12 synthesized), which were chosen at random from the GRID, TIMIT and CREMA datasets. We have not included videos from the LRW since uploading them publicly is not permitted. Users are asked to label each video as real or fake. Responses from 750 users were collected with the average user labeling correctly \(52\%\) of the videos. The distribution of user scores is shown in Fig. 21.