When Old Meets New: Emotion Recognition from Speech Signals

Sentiment analysis and affective computing have been receiving a surge of interest from both the academic and business communities in recent years due to the proliferation of opinion-rich social media data and their increasing applications in different use cases. The research in this field has been traditionally focused on analyzing textual data. However, the limitations of interpreting emotions from text features alone have led to studies devoted to predicting emotional states from other forms of modalities, such as speech [1]. Speech signals are the most natural, intuitive and fastest means of interaction for humans and are known to carry much more information than text or spoken words [2].

Emotion recognition from speech signals has been studied for decades. However, finding effective features and training machine learning models that can generalize well to real-world applications are still challenging tasks. This undertaking, along with the emergence of speech-based virtual assistants that provide readily available platforms for voice-based emotion recognition systems [3], drives a growing interest in the speech emotion recognition (SER) body of research [2].

With advancements in technology and the heightened optimism in deep learning, SER applications are expanding rapidly in different domains. In the healthcare sector, for instance, an SER system can be used as a medical diagnostic tool in which the acoustic properties of speech are analyzed as indicators of depression and suicide risk [4]. In [5], the authors developed a real-time SER for online learning. Additionally, the automotive industry can benefit from SER for the purpose of automatically recognizing the emotional state of the driver and responding intelligently in order to improve the driving experience and encourage better driving [6].

A variety of approaches have been proposed in terms of preprocessing, feature extraction and classification algorithms using different SER databases. Traditional SER systems involve the calculation of statistical functions for low-level descriptors such as pitch, zero-crossing rate (ZCR), energy and mel-filter bank features, which are fed into traditional supervised learning algorithms such as support vector machines (SVMs) [7]. Recently, the growing interest in deep learning has led to more advanced approaches that use, for instance, deep neural networks (DNNs) to automatically learn speech features [8]. Various DNN architectures for classifying emotions have also been presented and have exhibited the viable results [9].

A major challenge in fully exploiting the potential of DNNs for the task of emotion recognition is the lack of a large number of emotion-labeled speech datasets, which inhibits the training of a deep network from scratch. For this reason, traditional speech features such as mel-frequency cepstral coefficients (MFCCs) are still widely used due to their remarkable performance in SER experiments [3]. In this paper, we leverage the strengths of both traditional and deep learning approaches by combining the classical MFCC features with image spectrogram features extracted by means of a pretrained DNN.

In terms of classification algorithms, SVM is considered one of the most popular options in SER [7] and is among the best known classical methods for image classification [10]. A long short-term memory (LSTM) network, on the other hand, is one of the most powerful algorithms for modeling sequential data, which are inherent in speech features [8]. Hence, we carried out various experiments on the abovementioned features using the two classification algorithms SVM and LSTM. In particular, the main contributions of this paper are as follows:

The classification results of the competing techniques are illustrated in Table 1, which contains the F1-score on the test set for each emotional state as well as two global measures represented by the overall F1-score and the overall accuracy. Here, the term overall refers to the average of the outcomes across the different prediction tasks. In particular, the global measures were computed as the mean over the six F1-score and accuracy values weighted by the number of samples belonging to the different classes. The use of the F1-score and the weighted accuracy as performance metrics ensures that the class imbalance in the dataset is accounted for. To refine the performance evaluation, we also combined the individual outcomes in Table 1 across the emotional states. Specifically, for each method, we computed a score given by the ratio between its error rate, defined as 1 minus the F1-score, and the lowest error rate on a given emotion. For a model, the sum of the scores on all emotions can be seen as a measure of its ability to generate predictions that are close, if not equal, to the best ones in terms of accuracy. The results of this analysis are depicted in Fig. 3.

From the outcomes of Table 1 and Fig. 3, some empirical conclusions can be drawn. By focusing on the use of SVM as the classifier, we observe that the proposed hybrid input feature vector outperformed MFCCs and spectrogram-based features taken individually. In particular, compared to MFCC-SVM and SpecsResNet-SVM, Hybrid-SVM led to an improvement in the overall F1-score and accuracy of at least 2% and 2.3%, respectively. Note that the results of Hybrid-SVM are fairly superior to the accuracy achieved on the same RAVDESS dataset in prior studies that use SVM-based methods for classification. For example, Zhang et al. [48] investigated the usefulness of alternative groups of acoustic features represented by different types of low-level descriptors. Among these, in the emotion recognition task, MFCCs derived by all (spoken and sung) utterances and combined with SVM achieved an accuracy of 0.538, which dropped to 0.487 when only spoken utterances were used. Our computational tests, therefore, support the effectiveness of a hybrid representation relying on MFCCs and spectrogram-based features, where the latter descriptors enhance the predictive ability of the former when they are used in conjunction with classical learning methods such as SVM. In particular, our findings shed light on the effectiveness of a pretrained CNN model, namely ResNet50, used as a feature extractor from speech spectrogram images. For the task of emotion prediction, indeed, this approach also showed the interesting results on certain emotional states (i.e., surprise) when it was exploited to generate a unique input feature vector to feed an SVM classifier (SpecsResNet-SVM model). Note that this method eliminates the lengthy and resource-intensive process of network training from scratch. Therefore, it deserves attention as a promising novel approach for emotion classification from speech data when this task is formulated as an image classification problem, as suggested in [53]. Finally, our experiments also confirmed the suitability of SVM when hybrid representations are involved as input for classification. Indeed, our results agree with prior studies that demonstrated the effectiveness of SVM for emotion prediction when the learning is based on multimodal or fused data sources [54].

When deep learning algorithms (i.e., LSTM) were exploited for classification, we observed the superiority of MFCCs, which, taken as unique input, were able to outperform to a large extent the spectograms and hybrid features combined with LSTM and to achieve a slightly higher accuracy compared to that of Hybrid-SVM. This noteworthy behavior is confirmed by the score analysis. Indeed, MFCC-LSTM reached the highest total score, being associated with ratios that were more densely distributed around 1 compared to those of the other techniques (Fig. 3). The success of MFCC-LSTM shows that, despite being traditional speech descriptors, MFCCs can still dominate more sophisticated deep-learning features. We observe that a known benchmark on the RAVDESS dataset was achieved in [49], where the authors reached an accuracy of 0.787 by using MFCCs and variants of artificial neural networks. Their result, however, is not directly comparable with our outcome for certain reasons. In particular, they cast the problem as a 5-class prediction task by including the unified anxiety/stress emotion in place of fear and surprise, where the former is the most difficult emotion to detect, as reported by the last row in Table 1, which indicates, for each emotion, the average F1-score over all methods. Moreover, they built the model on a larger training set by selecting 25% of the examples for testing. Nevertheless, the result achieved in our more difficult 6-class prediction problem can be judged to be at least on par with the benchmark obtained for the 5-class learning task. Furthermore, it is worth noting that the superiority of MFCC-LSTM over Hybrid-SVM is evaluated based on the global F1-score and the global accuracy, which both account for the class imbalance. Consequently, the difference in performance between MFCC-LSTM and Hybrid-SVM, where the latter prevailed over the former for disgust and surprise and was equally accurate on anger, can be significantly reduced by including more samples for disgust and surprise, which are precisely the underrepresented classes in the dataset.

By looking closely at the outcomes for the six emotion types, we observed that fear and happiness emerged as the most difficult to predict, while anger was consistently well detected by all models (Fig. 3). These findings are aligned with previous studies that showed that happiness is difficult to recognize due to its specific characteristics (i.e., it relies on distinct contextual information more than other emotion types) [9], whereas anger is often associated with the highest prediction accuracy and is therefore clearly the easiest emotion to predict from speech features [2, 55].

The direct comparison between the two top-ranked models, MFCC-LSTM and Hybrid-SVM, also revealed that the performance gap largely depends on happiness, which is more heavily misclassified by the latter. To investigate this discrepancy we analyzed the confusion matrices (Fig. 4), which showed that, for models exploiting spectrogram-based features, happiness is more often confused with sadness, fear and anger. This behavior suggested that a possible cause could be sought in the spectrograms of the aforementioned emotions, which make their mutual distinction more difficult. For this reason, we further analyzed the spectrograms by evaluating their similarity, based on the idea that features extracted from highly similar spectrograms may hinder the classification of the related emotions in the subsequent prediction task. To this end, we computed two indices between the spectrograms for each pair of emotions, represented by the cosine similarity and the structural similarity (SSIM) index [56], which accounts for image luminance l, contrast c and structural similarity s, as follows:where x and y are two image signals and \(\alpha\), \(\beta\) and \(\gamma\) are nonnegative parameters used to regulate the importance of the three components, which are all set to 1 in our tests.

The results of this analysis are depicted in Fig. 5, which contains the average pairwise cosine similarity of the six emotions (part (a)) and a graph-based representation of the SSIM index (part (b)). These results support our conjecture regarding the misclassification of happiness by spectrogram-based models, which can be misleading due to the high degree of spectrogram similarity between happiness and sadness, fear and anger, as emerges clearly from Fig. 5. The same analysis highlights also a substantial similarity between fear and sadness. This may explain the performance on these two emotions of the spectrogram-based methods, which labeled examples as sad and fearful in at least 15.9% of cases. Hence, as a general remark, spectrogram-related features can be useful for the prediction of some emotions (i.e., sadness, anger, disgust, and surprise), but not all. This paves the way for further investigation of the effectiveness of such descriptors by devising novel features to be extracted that are able to better grasp and reflect the inherent diversity among the visual representations of the spectrum of frequencies of emotions’ signals.

By adopting the best classification model based on our experiments (i.e., MFCC-LSTM), we built a real-time SER tool that takes speech signals as input and makes instantaneous emotion predictions. In particular, to increase the generalizability of the application, we retrained the final model on a larger sample obtained by combining different publicly available datasets: the Surrey Audio-Visual Expressed Emotion [57] dataset, Crowd-Sourced Emotional Multimodal Actors Dataset [58], Interactive Emotional Dyadic Motion Capture [59] dataset, Toronto Emotional Speech Set [60] and Berlin Database of Emotional Speech [61]. The collection of these data provided a total of 14,309 instances of 2–5 seconds of speech.

Figure 6 illustrates the graphical user interface of the SER application, which displays the instantaneous probabilities for each of the six emotions as well as the overall emotion probabilities of the speech stream over time. These probabilities are derived from the softmax layer of the LSTM network. In particular, the overall predicted emotion \(\hat{e}\) was computed over the entire length of the input speech signals by means of a hard majority voting scheme. Thus, it was defined as the most frequent emotion label assigned to the entire speech stream from time \(t_1\) to time \(t_n\), according to the following expression:where \(e_{t_i}\) is the predicted emotion at time \(t_i, i = 1,...,n\).

The performance of real-time SER systems is subject to the length of time necessary to process the speech features. The use of our MFCC-LSTM model enables the real-time feasibility of our tool primarily because the extraction and preprocessing of the MFCC features require minimal computational effort. As shown in Table 2, the process of feature extraction and preprocessing is on the order of milliseconds, allowing for real-time prediction of emotional states. Note that this analysis was performed on a machine with 8 GB RAM and a 4-core CPU.

The proposed SER tool, through the related application programming interface (API), can be effectively framed within intelligent systems in order to automatically recognize emotions from speech, contribute to a better interpretation of verbal communication and enhance their capacity to interact with humans in a natural way. This plays a fundamental role currently, when it is believed that artificial systems need to become more sensitive toward human emotions in order to be more user friendly, trustworthy and effective. The developed SER application is available upon request.

In this paper, we investigated the fusion of speech MFCCs and image features extracted from signal spectrograms for the task of emotion recognition. In particular, we evaluated the effectiveness of different combinations of supervised learning algorithms, namely SVM and LSTM, and feature sets, represented by MFCCs, spectrogram images and their hybrid fusions, on the benchmark RAVDESS dataset. By using the best-trained model, we built a real-time SER tool that takes in input speech signals and predicts instantaneous and overall emotions over time.

Our results demonstrated that with a suitable classifier, in the hybrid fusion approach, MFCCs and spectrogram images do complement each other, resulting in an improvement in the prediction accuracy. Concerning the classification algorithms, our experiments revealed that classical SVM is recommended for handling such hybrid features, whereas LSTM provides the most favorable results when the more conventional MFCC features are taken as unique input. In regard to the six basic emotions, happiness emerged as one of the most difficult to predict when using speech features. Indeed, as noted previously in the literature, it relies more on context information than the other emotion types [9]. To build a real-time SER tool, we used the MFCC-LSTM model, which was shown to be the best in our experiments. It is worth noting that the extraction and preprocessing of the MFCC features in real time require less computational effort than the hybrid approach. This enhances the feasibility of the real-time system. By means of the related API, our SER application can be integrated into any intelligent system and can help practitioners who are working on the development of real-time emotion recognition tools. The developed SER software can be useful in several areas. For example, it could effectively support activities in a call center, where emotions stemming from phone conversations could be monitored in real time. This would enable agents to better handle phone calls and ultimately increase customer satisfaction [62, 63]. Team collaboration measurement tools, such as the Happimeter [64], are other promising use cases for our work.

Despite the novel contributions of this study, we are aware of certain challenges and limitations. First, the limited number of emotion-labeled speech datasets prevented us from building and evaluating our models on larger and more diverse datasets. The generalizability of our prediction models could be further enhanced by using a larger speech training dataset collected from diverse populations in terms of language or culture, when such a dataset becomes available. Second, we found it challenging to predict happiness from speech features. Future works can therefore focus on finding novel descriptors that best represent this emotional state. For example, happiness could be better detected by means of other forms of modalities such as text, where the context can be better captured, or facial expressions and physiological characteristics such as changes in heart rate, which have been studied in multimodal emotion recognition research [42, 54]. Finally, our work evaluated two well-known algorithms for classification. Our findings, however, encourage future research directions devoted to exploring more sophisticated ensemble learning algorithms as well as alternative fusion methods.