Sparse coding over redundant dictionaries for fast adaptation of speech recognition system

Highlights

• A novel use of sparse coding is done for the on-line adaptation of HMM based ASR systems.

• The target is first sparse coded using OMP over exemplar/learned speaker dictionaries.

• Adapted model is obtained by the maximum likelihood scaling of the sparse coded target.

• Performance same as the existing techniques is obtained but with much lower complexity.

Abstract

This work presents a novel use of the sparse coding over redundant dictionary for fast adaptation of the acoustic models in the hidden Markov model-based automatic speech recognition (ASR) systems. The presented work is an extension of the existing acoustic model-interpolation-based fast adaptation approaches. In these methods, the basis (model) weights are estimated using an iterative procedure employing the maximum-likelihood (ML) criterion. For effective adaptation, typically a number of bases are selected and as a result of that the latency of the iterative weight estimation process becomes high for those ASR tasks that involve human-machine interactions. To address this issue, we propose the use of sparse coding of the target mean supervector over a speaker-specific (exemplar) redundant dictionary. In this approach, the employed greedy sparse coding not only selects the desired bases but also compresses them into a single supervector, which is then ML scaled to yield the adapted mean parameters. Thus reducing the latency in the basis weight estimation in comparison to the existing fast adaptation techniques. Further, to address the loss in information due to reduced degrees of freedom, we have also extended the proposed approach using separate sparse codings over multiple (exemplar and learned) redundant dictionaries. In adapting an ASR task involving human-computer interactions, the proposed approach is found to be as effective as the existing techniques but with a substantial reduction in the computational cost.

Introduction

The automatic speech recognition (ASR) systems are traditionally developed using the Gaussian mixture model (GMM) based context-dependent hidden Markov model (CD-HMM). Recently, the deep neural network (DNN) (Hinton et al., 2012) has been proposed to overcome the inefficiency of the GMM in the modeling of the events that lie on or near a nonlinear manifold in the data. With the advent of the DNN for generating the observation probabilities, the ASR systems based on DNN–HMM are fast becoming popular¹. In general, the ASR systems are trained on speech data from a large number of speakers (male and female). This pooled training is primarily intended towards making the statistical models speaker independent (SI). Unlike an speaker dependent (SD) system, the SI systems have to deal with both the intra-speaker and the inter-speaker variability. As a result of that, an SI system is reported to be 2 to 3times inferior in comparison to an SD system when both the systems are trained with an equal amount of data (Woodland, 2001). The SD systems, though quite effective, are infeasible to be built for each of the speakers in the user population on account of requiring a large amount of speech data per speaker. Consequently, the speaker adaptation techniques have been developed which intend to modify the parameters of the SI system to better suit a particular speaker given the limited amount of data from that speaker.

Some of the applications of ASR are information retrieval, language learning tools, voice-based search and entertainment (Eskenazi, 2009, Gray, Willett, Pinto, Lu, Maergner, Bodenstab, 2014, Schalkwyk, Beeferman, Beaufays, Byrne, Chelba, Cohen, Kamvar, Strope, 2010, Wang, Yu, Ju, Acero, 2008). In the case of ASR tasks involving human-machine interactions, the system is required to recognize speech from the adult as well as the child (male/female) speakers. Further in such ASR tasks, the adaptation data is made available to the ASR system in an incremental manner. Moreover, the duration of the data is generally very small. Consequently, the conventional adaptation techniques like the maximum a-posteriori (MAP) (Gauvain and Lee, 1994) and the maximum likelihood linear regression (MLLR) (Digalakis, Rtischev, Neumeyer, 1995, Leggetter, Woodland, 1995) are found to be unsuitable. This is so because the available adaptation data is insufficient to estimate a large number of parameters. On the other hand, the fast adaptation techniques try to modify the model parameters even with a small amount of adaptation data (Gales, 1999, Hazen, Glass, 1997, Kenny, Boulianne, Dumouchel, 2005, Kuhn, Junqua, Nguyen, Niedzielski, 2000). In these techniques the adapted model parameters are derived by a linear interpolation of a set of bases (predefined acoustic models) spanning a low (K) dimensional subspace. Consequently, only a few interpolation weights (or the direction coordinates) need to be estimated. Such a reduction in complexity makes these approaches amenable for aforementioned adaptation tasks. The fast adaptation approaches are found to be effective for the adaptation of the Gaussian means as well as the mixture-weights (Duchateau, Leroy, Demuynck, Van hamme, 2008, Hahm, Ohkawa, Ito, Suzuki, Ito, Makino, 2010) of the acoustic model.

In the initial works on fast adaptation (Gales, 1999, Hazen, Glass, 1997, Kenny, Boulianne, Dumouchel, 2005, Kuhn, Junqua, Nguyen, Niedzielski, 2000), the K bases to be interpolated were kept fixed. The recent works (Mak, Lai, Hsiao, 2006, Teng, Gravier, Bimbot, Soufflet, 2009) have shown that an improved recognition performance can be achieved for the given test data by selecting K bases from a predefined set of acoustic models. In those approaches, speaker adapted (SA) models corresponding to each of the N speakers in the training set are derived first. This is usually done by adapting the mean parameters of the SI model with the speaker-specific data while keeping the covariance matrices and the mixture-weights unchanged. Given the test data, a set of K models (bases) are then selected from the N acoustic models and are linearly combined. The K interpolation weights required for linear combination are estimated using an iterative maximum-likelihood (ML) approach. It is worth mentioning here that there do exist some fast adaptation approaches in which the bases are derived using the given adaptation data rather than using predefined ones (Gales, 1999, Kenny, Boulianne, Dumouchel, 2005).

The aforementioned fast adaptation techniques differ mainly in the way the K bases are created/selected. The work presented in Mak et al. (2006) employs a Viterbi-alignment-based ML search for the basis selection. The ML search becomes very cumbersome when N is large and hence is not suitable for adapting interactive ASR tasks. The approach reported in Teng et al. (2009) requires the estimation of the interpolation weights for all the N models using a single iteration of ML estimation. The K acoustic models having the largest magnitude for the interpolation weights are selected for deriving the adapted model. The interpolation weights for those K models are then re-estimated iteratively. The complexity introduced in estimating weights for all the N models limits the feasibility of this approach in the case of ASR tasks involving human-machine interactions.

The work presented in this paper is inspired by the recent works employing a dynamic selection of acoustically close bases. In our earlier work (Shahnawazuddin and Sinha, 2014a), techniques employing the sparse representation (SR) (Elad, 2010) over a redundant dictionary were explored for the dynamic selection of the bases. In that work, the sparse coding was used only for the basis selection while the basis coefficients in the sparse coding were discarded. Like the other model-interpolation-based techniques, the interpolation weights were estimated iteratively in the ML sense. The latency in the basis selection process was reduced due to the greedy SR-based approaches but the computational cost of the weight estimation remained the same. In contrast to our earlier work, in this paper we propose a novel use of the sparse coding to derive the adapted model mean parameters.² The main contributions of the work are as follows:

• Exploration of the sparse-coding-based basis selection approach to reduce the computational cost in learning of the interpolation weights for the selected bases.

• Enhancement of the proposed approach with sparse coding over multiple redundant dictionaries which relaxes the degrees of freedom in the weight estimation without much increase in the computational cost.

• Exploration of the proposed approach in the context of highly mismatched acoustic conditions as well as in combination with other existing fast adaptation approaches.

The remaining of this paper is organized as follows: In Section 2, a brief review of the SR-based basis selection technique is given. In Section 3, we describe the proposed scheme that employs sparse coding to derive the Gaussian mean parameters. The multiple dictionary-based approach to overcome the loss in information is presented in Section 4. The experimental setup and evaluation of the proposed schemes are discussed in Section 5. Finally the paper is concluded in Section 6.