Developing sequentially trained robust Punjabi speech recognition system under matched and mismatched conditions

In real-life applications, a speech signal heard in one’s ear is a continuous mixture of different kind of signals which are originated from diverse environments and recording conditions. These adverse conditions greatly impact the performance of state-of-the-art recognition systems due to presence of unwanted information in an input speech signal [1]. Despite this, humans are well able to distinguish different sounds from multiple sources. In recent years, communicating with speech-oriented technological devices has become a part of daily usage for billions of people around the globe in form of voice assistant applications: Amazon Alexa, Apple Siri, Google Assistance, Microsoft Cortana [2, 3], etc. Moreover, human generally feel more comfortable by communicating in their native language and thus making use of their devices in real-life applications: military operations, education and medical research. These social interactions help parents in teaching their children in own native language which eventually preserve two objectives: (i) keeping first language alive, (ii) preserving literature and the pride of cultural roots. Nowadays, languages are depleting at an alarming rate, in fact, it has been reported that within next century, almost 50% of existing languages enter into an endangered stage [4]. Although, during spread of Covid-19 pandemic, a demand for deployment of children-based speech technologies have become an important as well as challenging task for various researchers. Many research works have been presented on adult, children, or mismatched training and testing conditions with an objective of removing the constraints of validating speech styles, classes, vocabularies as well as distorted channels [5‐8]. These variations in children’s speech arises an urge for developing a robust ASR system which will help in reflecting the future of human–computer interactions in form of technological education. The other major challenge is building of children ASR system in their native language. To fulfill this requirement, one must have effective resource (in form of application or device interface) as well as adequate quantity of training data.

Hence, many approaches for artificial augmentation-based generation of synthetic data under real-life adverse conditions have been worked upon for fulfilling objective of large training data requirement. Likewise, some researchers have investigated automatic recognition of children’s speech under mismatched conditions, i.e., training adaptation to adult speech corpus but it has been a well-known challenging problem due to acoustic differences in speech of adult and child speakers [9‐11]. Apart, feature extraction exhibits a compact representation of an input speech signal which is a foremost step in development of an efficient ASR system. Since it is not feasible to recognize speech signals from digitized waveforms due to large-scale variations, thereby, better aspect of the application of noise-robust feature extraction techniques is required to be considered such that the variability among matched or mismatched systems is removed. The extracted feature vectors are though well efficient in capturing relevant information while discarding the redundancies originated due to presence of noise in an input speech signal. Therefore, various feature extraction techniques: RASTA-PLP [12], MFCC [13], GFCC [14] and PNCC [15] have been investigated by various researchers with an effort of deploying an effective noise-robust ASR system. For the past many years, HMM has been a widely adapted modeling technique for efficient learning of parameters corresponding to an acoustic model [16]. With progressive developments in ASR, the flexibility and prediction power of deep learning algorithms have enabled researchers to generate observational probabilities for different HMM states. However, most of the hybrid DNN–HMM architectures being trained for development of speech recognition have been based upon their individual classification of frames or on the basis of cross-entropy. It helped in reduction of frame error rate. Apart, the generation of speech which further processed using method of ASR is considered a sequence classification problem. Therefore, to better match the decision rule in case of matched and mismatched system, various forms of sequence-discriminative training: MMI, MPE, bMMI and sMBR training criteria by employing lattices are being evaluated by earlier researchers [17‐20]. The resultant techniques utilized GMM–HMM- and DNN–HMM-based architecture which has resulted into the continuous gains [21]. Subsequently, researchers have expressed some of the disagreements pertaining to comparative analysis of resultant techniques where MMI has been performed better than that of MPE [22] and in [23, 24] sMBR which resulted into effective gains in terms of accuracy. Later, these front- and back-end approaches capabilities are employed in Indian Punjabi language.

In this paper, a robust Speaker-Independent ASR framework has been presented on four front-end approaches: MFCC, RASTA-PLP, GFCC and PNCC to explore effectiveness of Punjabi speech recognition system on various matched (adult speech in train and test) or mismatched (per-mutational mix of adult and child speech in train and test sets) systems. While in Punjabi language, the work on children speech is almost zero and in adult at infant stage because of the non-availability of child and very less adult speech and text-labeled dataset. The implementation has been performed on large adult data and very less children speech on original and synthetic noise injected at lower SNRs. The impact of various types of noises—Volvo, babble, pink, white and factory has been analyzed alone or through pooling of all noise dataset. Likewise, the problem of data scarcity has also been overcome through creation of synthetic noise dataset by pooling it with original corpora using out-data augmentation [25] strategy. Accordingly, in this research, the training child data have been augmented with enough available or self-created adult data with an effort of handling the problem of data scarcity and tried to boost the performance of system in mismatched conditions. Moreover, the geometry of vocal organs of child and adult differs considerably (smaller in children), which arises an urge for scaling of fundamental frequency or pitch. Thereby, an effort for removal of inter-speaker variations and mismatch conditions among test and train datasets has been investigated by utilization of the methodology of VTLN [26]. In addition, the acoustic optimization methodologies based upon the procedure of inter-frame-based discriminative training have been further employed and the performance is monitored for each framework on hybrid front-end techniques.

The remainder part of the paper is organized as follows: the next section describes the related work, and in the third section, the technologies study employed are discussed. The fourth section presents corpus use, and in the fifth section, proposed approaches on heterogeneous robust ASR framework using sequencing discriminative training are processed in matched and mismatched systems using original and synthetic dataset are outlined. The sixth and the last sections present experimental study along with conclusion and future work, respectively.

Earlier adaptations for development of an ASR system were based upon the interpretation of phonemes for effective creation as well as recognition of vowel sounds. However, development of noise-robust ASR systems has been greatly affected by acoustic environments in the presence of background noise, reverberation and other distortions caused due to interfering signals [27]. The primary requirement of increasing efficiency of an ASR system in native language is important and, basically, dependent upon the representation of compact information by utilizing various technique of filtering noise and undesired information present in an input speech signal. Gong et al. [28] analyzed the impact of noisy conditions on building a robust ASR system by portraying a survey of 250 publications related to the techniques while discarding undesired information present in form of noise from an input speech signal. The researcher highlighted the importance of categorization based on measurement and analysis of noise-resistant features. The techniques for speech enhancement and hidden Markov model adaptation for the compensation of unwanted noise are also presented in it. Likewise, Diethorn et al. [29] highlighted the use of noise-reducing processors in modern daily-life communication systems. It consists of telephone handsets, mobile phones, teleconferences and in-home-based telephonic appliances and speaker phones. The researcher focused on the methods of extracting useful information from a single-channel noisy system by utilizing the techniques of short-time spectral modification. Further, Farahani et al. [30] based upon the higher value of SNR highlighted the minor adaptations in a signal with performance degradation due to mismatch conditions between train and test set. The researcher highlighted the replacement of speech signal features with features generated on the basis of autocorrelation sequence. Ma et al. [31] experimented on novel noise reduction algorithm by utilization of Wiener gain function by exploring it on bias and variance properties of the multi-taper spectrum. MFCC has been a broadly used method for feature extraction; however, degradation of performance of most of the system is seen under noisy conditions. Kadyan et al. [32] presented heterogeneous feature vectors using MFCC and PLP fusion along with RASTA which further utilized GA + HMM- and DE + HMM-based hybrid classifiers under both clean and real conditions. The researchers concluded with an overall improvement using hybrid classifiers by ~ 13% with MFCC and DE + HMM when compared with RASTA-PLP. On the other hand, many advanced noise-robust features: GFCC, PNCC and their comparative analysis with different SNRs have been presented by Zhang et al. [33]. The outcome revealed noise robustness and effectiveness of PNCC feature extraction methodology under lower SNRs as compared to that of traditional modifications of feature vectors using MFCC and GFCC approaches. Moreover, the concatenated features GF-MFCC for performance improvement in both clean and noisy environments were investigated by Dua et al. [34]. Since, the speech recognition was named as a sequence classification problem such that there is an effective need for consideration of inter-frame constraints that helped in optimization of HMM parameters alongside the phonetic word references and powerful language model. On the other hand, the estimation of HMM model parameters was made by maximizing the likelihood when the states of model were paired in a supervised manner [35]. However, Nádas et al. [36] concluded that non-consideration of other possible word strings during MLE training frequently leads to an increase in likelihood of word corresponding to its transcribed utterances. Later Povey et al. [37] experimented the comparison of the use of another discriminative training MMIE and generally utilized MLE. The outcomes showcased a significant increase in the performance of the system using MMIE as compared to that of MLE on very large data sets. Finally, Veselý et al. [22] represented the comparison of different sequence-discriminative training criteria: MMIE, MPE, sMBR and bMMI. The outcomes of the comparison have demonstrated an average relative improvement of 8–9% by utilizing the cross-entropy-based DNN model. Finally, in this paper, an effort has been made to analyze the characteristics of different front and back-end approaches on less resource language like Punjabi. Consequently, processing of inter-speaker variability in test set along with efficient modeling of model parameters with various discriminative approaches on train set.

The processing of clean speech always generates better output in any ASR system but it becomes challenging with real environment or synthetic noisy dataset. The real-life speech when tested on a clean train system degrades the performance of the ASR system. Therefore, an effort has been made to evaluate the characteristics of different front-end approaches to find an optimal approach that can yield better output for both types of system (generally the system with environment differences between train and test set) using Kaldi toolkit [41]. Initially, the original clean signal is injected with different types of artificial noise using Algorithm 1. It is possible through augmentation strategy which tries to fulfill the requirement of data scarcity problem of training dataset. Later pooling of such dataset has been performed such that Fig. 1 demonstrates the method of noise-based data augmentation through injection of background noise at varying SNRs into different combinations of clean datasets as detailed in Table 1. Although the perception of an individual identifying with respect to frequency context present in corresponding signal is elucidated to be non-linear. The case of machine processing a real-world input speech signal is always challenging due to various inbuilt parameters like environment, speaker and other acoustic features. To tackle such issue at training and test end, initially four front-end feature extraction approaches: MFCC, RASTA-PLP, GFCC, and PNCC are being investigated with a target of extracting robust feature vectors that helped in extraction of relevant information in spite of the presence of noisy background. The main focus of the feature extraction process is the improvement of cepstral representation by extraction of information which is nearly close to the human perception. First, conventional MFCC is a widely used feature extraction approach which is based upon the typical 40-channel Mel-Filterbank with a frame size of 25 ms and frame shift of 10 ms. The perceptual sensitivity on the magnitude axis is taken into account by expressing magnitude upon log-spectrum motivated by the use of mel-scale. However, MFCCs are not robust to noise such that the performance is degraded in the presence of an additive noisy environment. Second, RASTA-PLP is more robust to steady-state spectral features. In this technique, the temporal derivatives of critical log-spectrum are estimated using a regression line based on first-order IIR filtering. Here, while performing the process of integration, the pole of the system (\(z=0.98\)) through Eq. (2) is initialized. Therefore, the separate channel estimation phase in the process of RASTA-PLP helped in reduction of convolution noise which is quite different from that of processes being involved in techniques with the change in transfer function. However, the accuracy evaluated on the frequency scale is quantized based upon the selection of different criteria of information extraction process. In this way on third, the equivalent of MFCCs, which is GFCC is computed based upon its 64-channel Gammatone filter-bank using a frame size of 25 ms and a frame shift of 10 ms. Later, PNCC is employed which makes the use of typical frame sizes of 25 ms and 10 ms just like MFCC, RASTA-PLP and GFCC approaches. In this process, every frame in a particular audio signal is processed using Povey window [41] and furthermore FFT is being evaluated on 256-bit resolution. The initial processing stages for evaluation of PNCC are quite similar to that of the stages of MFCC and PLP. However, the difference lies in the process of the analysis of frequency performed which are utilized using gamma-tone filters. Further, the long duration temporal analysis accomplished using noise reduction is evaluated by a series of non-linear time which lies on the varied operations being performed.

Further, the final refined feature vectors are classified by computing the cepstral mean and variance normalization (CMVN) which are being evaluated using the following equation for each process. It helps in fixing the data samples such that they remain in an appropriate format as required for the process of acoustic modeling.

These features are further processed to remove inter-speaker variability factors. In the first phase of training procedure, mono-phone (mono) models are generated for very small quantities of data. Further triphone models are trained which includes the process of computation of the delta features (tri1) and delta–delta features (tri2). However, the process of splicing helped in extraction of 13-dimensional features across ± 4 frames. It resulted in generation of 117 dimensional vectors. Thus, it is difficult to evaluate upon a large number of vectors so the procedure of LDA + MLLT (tri3) estimation is applied with an objective to reduce the dimensions from 117 to 30. Finally, a global fMMLR is applied to align the reduced dimensions to normalize the inter-speaker variations. Finally, the different systems are trained on hybrid DNN–HMM acoustic models as shown in Fig. 4.

Figure 5 demonstrates the proposed system utilized the noise-robust PNCC features being experimented on a noise-augmented pooled dataset for adults and the combination of adult and child dataset. These extracted features are further normalized using CMVN and further trained on mono-phone (mono) and tri-phones (tri1, tri2, tri3) models as in the baseline system. The inter-speaker variations among children and adults are key parameters which try to enhance the performance of the system. Therefore, the intuitive method of VTLN has been implemented by warping the spectrum in a frequency axis particularly on the test dataset. This type of normalization helps in the reduction of inter-speaker variability by relatively placing the format positions in its normalized spectrum. However, the current ASR systems are mostly trained with MLE and further methods of sequence-discriminative training have been experimented. Moreover, the process of generation of lattices is also employed at the modeling phase which serves as an important aspect. It acts as an intermediate format between interoperation format and the corresponding recognition passes. Lattices related to the certain utterances are created through utilization of the arrangement of back pointers for which a solitary Viterbi back pointer is being stored at the word level. The visualization of the lattice for a Punjabi word sequences is shown in Fig. 6. It is converted into non-compact structural form such that the comparing arcs are being removed alongside the addition of acoustic and language model costs.

While implementing MMI as an objective function for parameter optimization, maximization of the numerator (reference labels) and minimization of the denominator (chance of others) is performed. The generated lattices are expanded to HMM such that different pronunciations for the words are accounted for by just considering only certain word sequences available in the transcript. Finally, the state occupancy probability (γ) for both the numerator and the denominator lattice occurrence is separately computed through the following equations:

Here, the number of iterations (i = 1 to 8) for model-space training in MMIE is experimented which helped in the reduction of the likelihood of word sequences apart from the reference utterances. MMI usually works by considering large segments of multiple patterns corresponding to the utterance whereas MPE is focused on the optimization at the sub-`string pattern level. Therefore, the major impact is of phones being implied such that the different language models substituting the value (n = 1, 2, 3, 4) in Eq. (18) are being experimented. Moreover, the parameters for boosting the likelihood of the word sequences (boost factor) which range from 0.05 to 0.25 is considered for the process of bMMI. It is well known that the MBR family use for optimization was designed with an objective of minimizing the error rate in reference to the different granularity of labels. The average accuracy of the given states referring to every path of lattice corresponding to reference is evaluated. This helps in the calculation of MBR posterior which is computed over the denominator lattices for utterance through Algorithm 2.

In view of a certain state-etiket, the undermined Markov speaker model requires some very simple expressions: it is simply the product of the probabilities of sound characteristics for every frame which fits into this label (i.e., the acoustic classifier results, known as the acoustic score) and the probability of each frame being multiform. In addition, there may be a much greater state than the label, for example the number of steps taken inside that phone, the mode of the phrase and the preceding words. The significance of using word sequence-based decoding process is being utilized in sequences of separate decision problems involving tiny sets of confusing words for segment lattices being created for developing a general-purpose automated speech recognition (ASR) system. Likewise, in successive rescoring SMBR passes, acoustic models which distinguish between competing words in such classes are subsequently employed. Hence, the refinement of the search area which permits specialized models of discrimination is proven to be an advantage over rescoring with classically trained models of discrimination. Finally, these specialized models of discriminative training involving adult dataset and adult–child mix training are trained on hybrid DNN–HMM models considering both clean and noisy environments as a test set. The key impetus, however, is to train the model, which is effective in capturing long-term dependence between the missing acoustic characteristics. Thus, the capture of these long-term maladjusted relationships became efficient later on with another modified neural network design of TDNN. Thus, both DNN and TDNN architecture are ultimately trained for speed disturbed data via sequence-based training optimization of acoustic modeling. The corresponding performance is represented in the form of Word Error Rate (WER) and RI using the following equations, respectively:

In this study, heterogeneous front-end: MFCC-, GFCC-, RASTA-PLP-, and PNCC-based robust ASR framework has been systematically presented that provides better accuracy using various parameter optimized sequence-discriminative training approaches on acoustic modeling phase. These approaches have been implemented on large adult speech and very low child speech on true matched and mismatched systems. Further, the issue of data scarcity caused due to small original train speech is resolved using out-domain augmentation strategy. These results in large training complexity because of the multi-style data augmentation strategy employed through pooling of original speech and noise injected at different SNR level synthetic speech. It resulted into over fitting and confusion of acoustic model information, so it is additionally processed using parameter optimization of feature vectors by MMI, MPE, bMMI, and sMBR approaches which are processed on the basis of lattice generation, and adjustments of learning rates. It tried to be demonstrated for developing an effective training system. Moreover, the adequate gender-based selection concerning adult data has solved for the problem of the data shortage as well as reduced differences of acoustic mismatched parameters including frequency and vocal tract length has led to substantial improved performance of the system. Further, this paper also included additional inter-speaker variability reduction methods between adult and child speech using the VTLN approach in the test set only. It is found to be efficient in normalization of training and testing dataset differences caused due to varying vocal length through optimal selection of warp factor. The experiment results showed that ASR frameworks investigated on PNCC + VTLN approaches are found to be effective with only test normalized systems on TDNN-sMBR-optimized acoustic models. However, the results show a relative improvement of 47.51% on mismatched, 40.18% on matched systems and 49.87% on adequate gender-selected systems than other ASR frameworks, respectively. Further work can be extended by speech rate rhythmically parameter-based classification approach for normalization of individual adult and child speech trained systems on true matched and semi- or mismatched conditions on the basis of test speech. Further to that, a robust switch to process separate clean and noisy environment dataset is also required to implement an efficient front-end approach that wishes to address the drawbacks of the proposed approach.