A DCRNN-based ensemble classifier for speech emotion recognition in Odia language

Classifier plays an essential role for the SER system. Researchers have proposed various deep learning algorithms to represent an efficient classifier to distinguish emotional classes. Some popular emotion classifiers are the K-Nearest Neighbor (KNN) algorithm [20], Hidden Markov Models (HMMs) [21], Gaussian Mixture Models (GMMs) [22], SVM) [23], and Softmax function [24]. In the above classifiers, if the training data is much greater than the number of features (p >  > q), KNN is better, but for lesser training data, SVM outperforms KNN and GMM. Currently, most researchers use Softmax and SVM classifiers rather than KNN and GMM classifiers, which makes them more popular and reliable. Softmax and SVM are the most useable classifiers in speech-related tasks and the performance difference is usually very small. But sometimes, selected features are not robust enough to design a classifier for speech emotion recognition. So classifiers are trained and tested on the same data. To integrate the merits of classifiers, we ensemble the SVM and Softmax classifiers and evaluate the performances in terms of accuracy for speech emotion recognition.

The feature extraction process is a primary task for building an SER system. This process can reduce noise from raw audio data and generate highly effective features in learning the emotions from the SER model. Various features were used for SER systems, such as acoustic features, context information, hybrid features, and linguistic features. Among these, acoustic features are mainly used in the emotion recognition domain, containing local and global features [16]. Acoustic features are separated into four groups: spectral features, prosodic features, voice quality features, and other features. Above all, prosodic and spectral features are used more commonly in SER systems. [31] utilized 79 emotional-related prosodic features. Kachele et al. [25] integrated prosodic features and speech quality information to identify more robust speech emotion systems than prosodic features. To obtain a better result, many traditional linear spectral correlation features have been investigated, such as Zero Crossing Rate (ZCR), Chromagram, Linear Prediction Coefficients (LPC), Root-Means-Square (RMS), Mel-Frequency Cepstrum Coefficients (MFCC), Log-Frequency Power Coefficient (LFPC), Log Mel-spectrogram, and Linear Prediction Cepstrum Coefficients (LPCC) [14, 26], and energy-related features. Prosody features such as Pitch, formant frequency, duration, and loudness were also commonly used [27]. Some voice quality features such as shimmer (amplitude irregularity), jitter (pitch irregularity), fundamental frequency, duration, harmonics-to-noise ratio (HNR), and power are also used [11]. In addition, a combination of prosodic features and voice quality information gives better information about the identification of emotion in comparison with only prosodic features. Some context information has also been studied [28] for emotion recognition. In [28], the authors present an SER system based on cultural information. The authors proved that cross-cultural-based SER performs better than multi-cultural and intra-cultural paradigms. Since these features mentioned above are low-level, they may not contain enough emotional information to identify the subjective emotional state. It may be possible to employ deep learning strategies to learn high-level features that are automatically effective for speech emotion recognition to address this problem.

The spectral features can identify emotional details better in time–frequency correlation and extract high-level information from the spectra using 2-D images. Therefore the researcher gives more attention on spectral features for speech emotion recognition [17, 29]. Generally, durations of speech signals are different, but most of the deep learning models require a fixed size of input. Representations of incomplete feature maps may not detect the correct emotional state of an utterance. To overcome this drawback, we extract 3-D log Mel-spectrograms (Static, Delta, Delta-delta) from the 1-D raw speech signal as inputs to our proposed DCRNN model, to minimize the loss of emotional information.

The process of creation of three-channel utterance level log Mel-spectrograms are as follows. (1) We use Librosa Audio Library [30] to originate the log Mel-spectrograms from the speech signals under 16 kHz sample rate. (2) Then, we apply first and second-order derivative on the 2-D static log Mel-spectrogram along with the time axis to find the delta and delta-delta log Mel-spectrograms. So it creates itself a 3-D log Mel-spectrogram. The 3-D log Mel-spectrogram of each emotion on the Odia database is shown in Fig. 1. Finally, we resize the log Mel-spectrograms to 277 × 227 × 3 because pre-trained (AlexNet) DCNN requires an input size of 277 × 227 × 3.

After generating log Mel-spectrograms, we performed pre-trained AlexNet [19] DCNN for feature extraction. The AlexNet is a powerful model capable of achieving competitive performance on challenging small datasets. On the other hand pre-trained networks like VGG, ResNet, and efficient networks are much deeper and has more parameters, which require a very large number of inputs to achieve high performance [31]. We were inspired by the uses [8, 16] of pre-trained AlexNet. In the AlexNet network, the original parameters remain the same, and the layers are used to generate features. The AlexNet deep neural network contains several convolutional layers (CL), dropout, max-pooling layers, fully-connected layer, and the ‘Relu’ (rectified linear unit) activation function. The detailed description of DCNNs layers as follows.

The AlexNet model accepts with a fixed input size of 277 × 227 × 3. Each convolutional layer consists of several filters, kernel size, non-linear activation function, and padding. The convolutional layer is used to extract local patterns of the input and generates the feature maps. The AlexNet model has five convolutional layers (CL1, CL2, CL3, CL4, and CL5). The CL1, CL2, and CL5 are followed by the max-pooling layer, as shown in Fig. 2. The CL1 layer has a 96 kernel filter and a kernel size of 11 × 11 with a stride number of 4. The size of the CL2 layer is 5 × 5 with 256 kernels and a stride of 1. The CL3 layer has a size of 3 × 3 with 384 kernels connected to the outputs of the CL2 layer, and the CL4 layer has a size 3 × 3 with 384 kernels. The Relu activation function is used in each convolutional layer which increases the training process.

The max-pooling layer reduces the feature maps by utilizing maximum filter activation to get more high-level features. The output from the last max-pooling layers is fed to the fully connected layer. In the AlexNet) [19] model, the fully connected layers are FCL6, FCL7, and FCL8. FCL6 and FCL7 build a 4096-dimensional feature vector, whereas the FCL8 layer contains a 1000-dimensional feature vector because of 1000 types of categories on the Image Net data. We have not used FCL8; the FCL7 produces a 4096-D feature vector connected to the Bi-GRU layer.

RNN [39] is proposed to solve the problem with time-series data. Most of the DNN, namely, convolutional neural network and multi-layer perceptron neural network are built with the help of weight connection of every layer, and the nodes between every layer are separated. So, nodes are independent of each other. However, in real life, the utterance is a time series with a variable length [32, 33]. Therefore, the previous utterance of the speaker is strongly related to the present utterance, which requires a model that can review the past information and process information of the different length of time-series data. To solve these issues, Hochreiter and Schmidhuber implemented Long Short Term Memory (LSTM) [34]. But the LSTMs take longer time, and require more memory to train than GRUs. And GRUs perform better than LSTMs in small amounts of training data.

Transformer architecture is often another common choice. However, this architecture does not capture the input order information. In [35], the authors point out that the Transformer only starts to outperform CNNs when data is more to train for the classification tasks. However, audio datasets typically do not have large amounts of data, which motivates us towards the use of Bi-GRU. The GRU cell is a particular type of RNN and the modified version of LSTM, as shown in Fig. 3. In the GRU cell, the cell state and the hidden state merge, and the input gate and the forgotten gate are combined and built as an update gate. Output of the last fully connected layer (4096-D) of DCNN network is connected to the Bi-GRU network with a sequences input of \(\left\{{x}_{1},{x}_{2,}{x}_{3}\dots \dots {x}_{t}\right\}\) and we get output sequentially as \(\left\{{y}_{1},{y}_{2,}{y}_{3}\dots \dots {y}_{t}\right\}\) by calculating each of input using activations (‘Relu’) functions in the network on the basic formulations from time \(t=1\) to \(t=T\)

From the Fig. 3. We get,

Here, \({z}_{t}\) defines update gate, \({\tilde{h }}_{t}\) represent the current value of the present hidden state, \({\tilde{h }}_{t}\) denotes the activation value of the current hidden state, and \({h}_{t-1}\) is the activation value of previous hidden state.where, \({\upsigma }\) defines the sigmoid function, \({x}_{t}\) is the input of the GRU cell, \(\odot \) defines the element-wise multiplication operation,\(\varnothing \) is the tanh function, and \(W\) and \(U\) are the weight matrix, used during the training operation. The main contribution of \(r\) is to control how much information passes through \({x}_{t}\) and how much previous information will affected by\({\tilde{h }}_{t}\). In essence, the GRU unit can overcome the problem of long-term distance information learning and at the same times also overcomes the problem of gradient dissent.

In this study, we use a speech spectrogram to detect emotion categories using a sequence labelling task. The unidirectional GRU cell cannot handle well a large number of speech samples of different durations. Therefore, Bi-GRU (Bi-direction Gated Recurrent Unit) [39] is utilized to learn present information and the past information of a variable-length of sound samples.

Select an efficient classifier is vital for final classification of emotion. Most of the deep learning model uses Softmax [23] classifier. For example, \(n\) possible classes has \(n\) nodes in the Softmax layer denoted by \({c}_{j}\).

Where \({c}_{j}\) is defined as the discrete probability distribution.

Output of the Softmax functions formula is as follows: where, \({d}_{j}\) is sum of the input into a Softmax layer which can be define as:

Here, \({d}_{j}\) is the activation in the second last layer and \({W}_{ij}\) define the weight associate between the second last layer and Softmax layer. Thus the final prediction class \(\widehat{j}\) would be:

The easiest way to extend SVMs for multiclass classification problems is using the one-vs-all method [23, 36]. For example, in class classification problems, \(n\) number of linear SVMs will be independently used, where the data from the other classes form the negative cases. The output representation of the \({n}^{th}\) SVM is:

And, the final predicted category is calculated using Eq. (10):

The SVM classifier’s prediction is the same as the Softmax classifier demonstrated in Eq. (8). The only difference between multiclass SVM and Softmax is in their parameters-weight matrices \(W\). Softmax classifier layer minimizes cross-entropy loss, while SVM classifier tries to find the maximum boundary between the data points concerning the classes.

Ensembles classifiers are generally proposed by a combination of two or more classifications. To improve over the best performing classifier, ensemble classifiers must comprise accurate base classifiers [37]. This paper has used the prediction probabilities of Softmax (marked as \({P}^{\mathrm{Softmax}}\)) and SVM (indicated as \({P}^{\mathrm{SVM}}\)) individually. Then we combine Softmax and SVM classifier to ensemble their probability as \({P}^{\mathrm{Ensemble}}\) to predict the final class. We use their maximum probabilities from individual classifiers, as mention in Eq. (11).

This strategy has improved our accuracy significantly. Comparing the results, our method with two different classifiers (SVM and Softmax) shows varied effectiveness. The experimental results show that the DRCNN with ensemble classifier is extremely accurate in identifying emotion.

To illustrate the performances, we tested our model on the Odia dataset and one popular public dataset (RAVDESS). Here, we use RAVDESS [23] dataset to validate our Odia dataset, which is widely used in speech emotion recognition.

The Odia dataset consists of 60 different utterances with seven different emotions: anger, surprise, fear, sadness, happiness, neutral, and disgust [38]. In our previous work, we have used six discrete emotions and a total of 3240 utterances. The dataset is collected from three different Odia dialects (Sambalpuri, Cuttacki, Berhampuri). Each dialect is recorded by six different speakers (three male and three female) whose ages gap between 19 to 40 years. We use ten different Odia emotion sentences, and every sentence was repeated three times in all three dialects. So, in total, 18 (six speakers and three dialects) × 10 (number of sentences) × 7 (number of emotion) × 3 (number of repetition) = 3780 utterances are collected. All the utterances are recorded at a sampling frequency of 8.1 kHz with 16-bit quantization.

The RAVDESS speech emotional corpus [23] was recorded in the English language. The whole speech corpus consists of 1440 emotional audio (.wav) files with eight discrete emotional states: fear, calm, neutral, angry, disgust, sadness, boredom, and happiness. The dataset is completed by twenty-four (twelve male and twelve female) North American professional actors and the average time duration of each audio file is 3 s. The complete recording process was done at a sampling rate of 48 kHz have 16-bit quantization. The details of each emotional state of the RAVDESS and Odia datasets are shown in Table 1.

It is essential to find out the suitable number of Bi-GRU layers and how many neurons are needed per layer to achieve the best-optimized model. To further investigate, we have studied the effect of different classifiers with and without pre-trained models. So, we first optimize the Bi-GRU layer, which is employed with the output of deep CNNs. The hidden layer and neurons are heavily dependent on the performance of the deep learning model. We train and validate our model with different numbers of Bi-GRU (with 1, 2, 3) layers with varying number of neurons (with 128, 256, 512) on the Odia and RAVDESS datasets using Softmax and SVM classifier. After conducting experiments on various layers, we concluded that BiGRU²₁₂₈ (two Bi-GRU layers with 128 neurons) on the RAVDESS dataset and Bi-GRU²₂₅₆ (two Bi-GRU layers with 256 neurons) on the Odia dataset performs better on different classifiers.

We also trained and tested our method without pre-trained DCNN instead of pre-trained DCNN (AlexNet) on the same model of our proposed method. Each parameter of the convolutional layers was randomly initialized with a standard normal distribution. The overall accuracy found without pre-training architecture was 81.24% of Odia dataset and 74.32% of RAVDESS with ensemble classifier. Then, we use the pre-trained ImageNet (AlexNet) model. The pre-trained ImageNet (AlexNet) model showed improved performance by 4.07% and 3.22%, respectively, compared to without the pre-trained model with ensemble classifier as shown in Table 2. The result demonstrates that using a pre-trained DCNN model improves the recognition accuracy and convergence rate.

Here, we represent the confusion matrix for investigation of the performances of the DCRNN model using Softmax classifier, SVM classifier, and the ensemble classifiers on Odia dataset and RAVDESS dataset shown in Figs. 4 and 5. Figure 4 represents the confusion matrix of the DCRNN model using Softmax classifier on Odia dataset; ‘neutral’ and ‘surprise’ achieves the highest recognition rate of 91.18% and 92%, respectively. In comparison ‘disgust’ is the lowest accuracy rate of 56.76%, and the other four emotions are obtained with accuracies below 90% with an overall accuracy of 81.53%, as illustrated in Table 2.

Figure 4b shows the confusion matrix of the DCRNN model using SVM classifier; ‘surprise’, ‘sadness’, and ‘disgust’ emotions show increased recognition rates of 8% (from 92 to 100%), 13.28% (71.43–85.71%), and 10.81% (56.76–67.57%), while ‘angry’ and ‘neutral’ show decreased rates of 3.18% (78.12–75%) and 5.89% (91.18–85.29%). Recognition rates of the two other emotions remain relatively same. The overall recognition rate increased from 81.53 to 83.62%, which shows that the SVM classifier performs slightly better than the Softmax classifier.

And Fig. 4c displays the confusion matrix of the proposed DCRNN model with an ensemble classifier. The confusion matrix illustrates that the recognition rate of ‘angry’, ‘fear’, ‘happiness’, and ‘sadness’ are 81.25%, 85.29%, 97.14%, and 88.57%, which indicates that 3.13%, 5.88%, 11.43%, and 17.14% as compared to Softmax classifier and an increase of 6.25%, 5.88%, 11.43%, and 2.86% when compared with the SVM classifier. The overall accuracy rate is found 85.31%, which is 3.78% and 1.71% better than the Softmax and SVM classifier.

On the other hand, Fig. 5 shows the performance on the RAVDESS dataset with Softmax, SVM, and an ensemble of Softmax and SVM classifiers with eight emotions. Figure 5a states the performance of the DCRNN model using the Softmax classifier on the RAVDESS dataset. From Fig. 5a, we observed that ‘calm’, ‘happiness’, and ‘surprise’ are recognized well with an accuracy rate of 91.30%, 87.88%, and 86.84%, whereas ‘neutral’ classified relatively well with a recognition rate of 77.42%. The other four emotions classified less than 70%. The overall accuracy observed is 73.52%, as shown in Table 2. Figure 5b shows the confusion matrix of the DCRNN model using SVM classifier; ‘neutral’, ‘calm’, ‘happiness’, and ‘surprise’ can be recognized with a recognition rate of 80.65%, 80.43%, 87.88%, and 84.21% respectively. At the same time the other four emotions are indicate a recognition rate below 75%. The overall recognition rate is 74.91%, which shows that our SVM classifier performs better than the Softmax classifier.

Finally, we demonstrate that the recognition rate of ensemble classifier of ‘sadness’, ‘angry’, ‘fear’, and ‘disgust’ are classified 20.93%, 7.14%, 15%, and 8.57% greater accuracy, than Softmax classifier. The accuracy figures of ‘sadness’, ‘fear’, ‘angry’, and ‘disgust’ are 9.30%, 10%, 7.14%, 2.86% and ‘calm’ as 2.18% better performs than SVM classifier as shown in Fig. 5c. On the other side, the accuracy rate of ‘happiness’, ‘calm’, and ‘surprise’ decreases 8.69%, 6.06%, and 2.63%, respectively, compared to the results of the Softmax classifier; ‘neutral’ and ‘happiness’ indicate, 3.23% and 6.06% decrease in the recognition rate from the SVM classifier.

The overall average recognition accuracy rate of the ensemble classifier of 77.54%, reveals that it outperforms the Softmax and SVM classifiers by 4.02% and 2.63%, respectively as shown in Table. 2.

In addition, we also report the value of the F1-score for each emotional state to calculate the statistical importance of our experimental results; the F1 represents the harmonic mean of precision and recall. Tables 3 and 4 represent the statistical performance on the Odia and RAVDESS databases.

From the results of the F1-score, we demonstrate that each dataset illustrates different issues in recognizing a particular emotional state. On the Odia dataset, ‘disgust’ is recognized slightly lower than all the other emotions by all the classifiers, whereas ‘angry’, ‘disgust’, and ‘sadness’ perform relatively below all three classifiers on the RAVDESS.

Finally, for further analysis of the effectiveness of the proposed model, the training losses curve of two datasets (Odia and RAVDESS) are represented in Figs. 6 and 7. It can be noticed that training through 150 epochs of two datasets has slight fluctuation in convergence, maybe for the duration of emotion samples were not equal. The average duration of Odia is 4.5 s, and the RAVDESS is 3 s.

Furthermore, we compare the result of our proposed method with several recent studies as well. Table 5 demonstrates a comparison between the results of our proposed method and previously public results with the model and features on the two datasets. Distinctly, on the RAVDESS dataset, our proposed method clearly expresses better with the performance level of 17.44%, 11.57%, 8.14%, 4.04%, and 5.93% collated with [3, 7, 17, 40, 41].

Our proposed method on the Odia dataset achieves 18.61% and 10.72% better accuracy compared to [38]. They used only prosodic features such as pitch, energy, format, etc., associated with SVM and GMM classifiers.