A coupled HMM approach to video-realistic speech animation

Abstract

We propose a coupled hidden Markov model (CHMM) approach to video-realistic speech animation, which realizes realistic facial animations driven by speaker independent continuous speech. Different from hidden Markov model (HMM)-based animation approaches that use a single-state chain, we use CHMMs to explicitly model the subtle characteristics of audio–visual speech, e.g., the asynchrony, temporal dependency (synchrony), and different speech classes between the two modalities. We derive an expectation maximization (EM)-based A/V conversion algorithm for the CHMMs, which converts acoustic speech into decent facial animation parameters. We also present a video-realistic speech animation system. The system transforms the facial animation parameters to a mouth animation sequence, refines the animation with a performance refinement process, and finally stitches the animated mouth with a background facial sequence seamlessly. We have compared the animation performance of the CHMM with the HMMs, the multi-stream HMMs and the factorial HMMs both objectively and subjectively. Results show that the CHMMs achieve superior animation performance. The ph-vi-CHMM system, which adopts different state variables (phoneme states and viseme states) in the audio and visual modalities, performs the best. The proposed approach indicates that explicitly modelling audio–visual speech is promising for speech animation.

Introduction

Speech-driven talking faces are playing an increasingly indispensable role in multimedia applications such as computer games, online virtual characters, video telephony, and other interactive human–machine interfaces. For example, talking faces can provide visual speech perceptual information for hearing-impaired people to better communicate with machines through lipreading [1]. Recent studies have shown that the trust and attention of humans towards machines can be significantly increased by 30% if humans are interacting with a human face instead of text only [2]. Current Internet videophones can transmit videos, but due to bandwidth limitations and network congestion, facial motion accompanied with audio often appears jerky since many frames are lost during transmissions. Therefore, a video-realistic speech-driven talking face may provide a good alteration.

The essential problem of speech-driven talking faces is speech animation—synthesizing speech-related facial animation from audio. Despite of decades of extensive research, realistic speech animation still remains to be one of the challenging talks due to the variabilities of human speech, mostly commonly the coarticulation phenomenon [3]. Various approaches have been proposed during the last decade, which significantly improve the animation performance. Some approaches use a 3D mesh to define the head shape and map a face texture to the mesh [4], [5], [6]. Others realize photo- or video-realistic animation from recorded image sequences of face and render facial movements directly at the image level [6], [7], [8], [9], [10].

Despite of different head models, according to the audio/visual conversion method, speech animation can be categorized into speech classes from audio and animation parameters from audio [6]. In the former approaches, audio is first segmented to a string of speech classes (e.g., phonemes) manually or automatically by a speech recognizer. Subsequently these units are mapped simply to lip poses, ignoring dynamic factors such as speech rate and prosody. The latter approaches derive animation parameters directly from speech acoustics, where speech dynamics are preserved. During the last two decades, machine learning methods, such as neural networks [11], Gaussian mixture models (GMMs) and hidden Markov models (HMMs) have been extensively used in the audio/visual conversion.

In GMM-based approaches [12], [13], Gaussian mixtures are used to model the probability distribution of audio–visual data. After the GMM is learned using the expectation maximization (EM) algorithm and the audio–visual training data, the visual parameters are mapped analytically from the audio. The universal mapping of the GMM ignores the context cues that are inherent in speech. To utilize the context cues, a mapping can be tailored to a specific linguistic unit, e.g., a word. Hence, HMM-based approaches have been recently explored.

To the best of our knowledge, Yamamoto et al. [14] were the first to introduce HMMs into speech animation, which has led the way to some new developments [13], [15], [16], [17], [18], [19], [20]. Their approach, namely Viterbi single-state approach, trained HMMs from audio data, and aligned the corresponding animation parameters to the HMM states. During the synthesis stage, an optimal HMM state sequence is selected for a novel audio using the Viterbi alignment algorithm [21]; and the visual parameters associated with each state is retrieved. Such approaches produce jerky animations since the predicted visual parameter set for each frame was an average of the Gaussian mixture components associated with the current single state, and it was indirectly related to the current audio input. In some other techniques, e.g., the mixture-based HMM [13] and the remapping HMM in Voice Puppetry [15], the visual output was made dependent not only on the current state, but also the audio input, resulting in improved performance. The mixture-based HMM technique [13] trained joint audio–visual HMMs which encapsulate the synchronization between the two modalities of speech. Recently, Aleksic et al. [20] proposed a correlation-HMM system using MPEG-4 visual features, which integrated independently trained acoustic HMM and visual HMM, allowing for increased flexibility in model topologies.

However, all the above methods heavily rely on the Viterbi algorithm that lacks robustness to noise [17]. If speech is contaminated by ambient noise, the animation quality will suffer greatly [15]. Moreover, the Viterbi sequence is deficient for speech animation in that it represents only a small fraction of the total probability mass, and many other slightly different state sequences potentially have nearly equal likelihoods [22].

Moon et al. [23] proposed a hidden Markov model inversion (HMMI) method for robust speech recognition. Choi et al. [16], [17] extended this method to audio–visual domain for speech animation, in which audio and video were jointly modelled by phoneme HMMs. They were able to generate animation parameters directly from the audio input by a conversion algorithm considered as an inversion of EM-based parameter training. In this way, they managed to avoid using the Viterbi algorithm, and made use of all possible state sequences to represent a quite large fraction of the total probability mass. More recently, Fu et al. [22] have demonstrated that the HMMI method outperforms the remapping HMMs [15] and the mixture-based HMMs [13] on a common test bed.

The conventional one-chain HMMs do have limitations in describing audio–visual speech: (1) we know that due to the difference in discrimination abilities, audio speech and visual speech can be categorized to different speech classes—phonemes and visemes [24]. Therefore, the bimodal speech is better modelled by different atoms explicitly. (2) speech production and perception are inherently coupled processes with both synchrony and asynchrony between the audio and visual modalities [25]. In previous HMM-based speech animations, the bimodal speech is modelled by a single Markov chain, which cannot reflect the above important facts.

The above two facts are important in audio–visual speech recognition (AVSR) or automatic lipreading [26]. These facts also affect the performance of the inverse problem (i.e., speech animation) since human perception system is sensitive to artifacts induced by loss of synchrony or asynchrony. Therefore, in order to make animation look more natural, it is necessary to take these facts into consideration. In this paper, we propose a coupled HMM (CHMM) approach to video-realistic speech animation, in which we use CHMMs to model the above characteristics of audio–visual speech.

In the following section, we describe the diagram of our speech animation system. Section 3 presents the AV-CHMMs used in our speech animation system, including our motivations, the model structures and the model training procedure. Section 4 derives the EM-based A/V conversion algorithm for the AV-CHMMs. Section 5 describes the audio-visual front-end. Our facial animation unit is presented in Section 6. Section 7 gives the comparative evaluations both objectively and subjectively. Finally, conclusions and future work are given in Section 8.