Personality trait estimation in group discussions using multimodal analysis and speaker embedding

The aspects of nonverbal communication have become important focuses in human–computer interaction (HCI) studies. This is because nonverbal aspects are naturally delivered in human-to-human communication. When we interact with other people, we consider not only what they are saying but also how they are speaking. If nonverbal aspects were not considered, communication would become very unnatural or robot-like.

The study of of the nonverbal aspects, e.g., personality, has attracted much attention in HCI. Personality extensively influences human life, in areas such as decision making, preferences, and reactions. It comprises the patterns of the habitual behaviors, emotions, and cognition of a person [34]. We could achieve a better understanding of ourselves and other people around us by understanding personality.

The integration between personality science and HCI studies has been emerging since the mid-2000s, and thus, the term personality computing (PC) was established as a research field [33, 53]. Vinciarelli and Mohammadi [53] argued that three phenomena fuel PC from a technological perspective: (1) the availability of personal information in social networks, (2) the possibility of data collection via mobile technology on a daily basis, (3) the consideration of social and affective intelligence in the computing and machinery research. Subsequently, three major problems are addressed in PC, i.e., automatic personality recognition, perception, and synthesis tasks [53]. The features for these tasks are extracted from personality-expressive signals, such as behavioral modalities from various data sources [33].

The most popular and influential personality taxonomy is the Big Five personality trait system [31, 34]. Since it is relatively stable in time as well as applicable across various cultures and trait measures, the Big Five personality trait system is accepted in a wide range of areas, including in PC [33, 34, 53]. This measurement classification system comprises five traits: In an earlier study, manual assessment was conducted by using a standardized factor analysis of personality description questionnaires to determine one’s Big Five personality traits. However, this type of manual assessment is very costly and thus not applicable for HCI interfaces. Accordingly, automatic personality trait assessment studies have attracted great attention in recent years.

Several techniques have been proposed from the perspectives of various modalities for automatic personality trait estimation. For instance, personality detection studies based on facial expression analysis were reviewed in [17] using image processing techniques. Concurrently, studies on speech personality trait recognition have also progressed in the speech research community, especially since the Interspeech 2012 Speaker Trait Challenge was released [42]. Other approaches using language models have also been widely employed to estimate personality traits, such as those derived from conversations through social media [10, 56]. Instead of focusing on one modality, several studies have also used multimodal analysis to infer personality traits [9, 22, 25, 29].

Despite the growth in the number of automatic personality detection studies, the reliability of detection performance is still far from ideal. Most of the existing studies focused on inferring individually perceived personality traits in self-presentation scenarios [5, 42], which is not ideal for representing personality. McCrae and Costa (1996) reported that personality shows the basic tendencies of a person, particularly in dealing with social interactions [31]. Manifesting personality traits in interactions is more meaningful than self-presentation.

In recent years, several studies have considered the automatic inference of personality traits from interaction processes, such as small group interactions [19, 22, 25, 29]. Okada et al. [29] proposed a personality trait estimation method based on a co-occurrent multimodal event discovery approach using the audio-visual (AV) subset of a group meeting from the Emergent Leadership (ELEA) corpus (ELEA-AV). Subsequently, the study of Kindiroglu et al. [19] demonstrated a multidomain and multitask approach for predicting the extraversion and leadership traits in the ELEA corpus. Additionally, prior work in [25] focused on personality trait estimation by using multimodal features and communication skills indices for datasets with multiple discussion types.

Many transformer-based methods and various types of multimodal fusion techniques have been proposed for solving various computing tasks [21]. Most of the methods required a large-scale dataset which is difficult to fulfill for analyzing a social interaction, such as the main task addressed in this study. With a relatively smaller size of data, we focus on how to handle individuality features and how to mitigate the issue of individual differences in more diverse group discussion corpora (different language and environment settings).

This study aims to address two novel points. First, we investigate the relationships between the state-of-the-art speaker individuality feature extracted from speech, namely, the identity vector (i-vector), and the Big Five personality traits. Our hypothesis is that speaker individuality is interrelated with personality. Second, we investigate the effectiveness of multimodal features regardless of the selected language. In this study, we consider two group discussion datasets, including the Multimodal Task-Oriented Group Discussion (MATRICS) corpus (in Japanese) and the ELEA-AV corpus (in European languages), to infer the Big Five personality traits. By the end of this paper, we will discuss the following key questions: The rest of this paper is organized as follows. Section 2 describes works that are closely related to this study. In Sect.3, we introduce the utilized multimodal corpora. Subsequently, we describe the employed feature representation approach in Sect. 4. The experimental settings and results are summarized in Sect. 5. In Sect. 6, we discuss the results and answer the key questions in this study. Finally, this paper is concluded in Sect. 7.

Automatic personality computing is useful for many HCI applications because it can model the relationships between stimuli and the outcomes of social perception processes. In other words, an automatic personality computing method models or estimates how our responses or impressions towards others are based on every observable action performed by the subject. Efforts on personality trait analysis with the consideration of multimodality are relatively extensive. For instance, a study on personality trait recognition in social interactions using audio and visual features was conducted by Pianesi et al. [35]. Their study aimed to automatically predict personality traits obtained from self-reported questionnaires. In [1], Aran et al. investigated video blogs in a small group meeting to predict personality traits, especially extraversion traits. Another work from Jayagopi and Gatica-Perez [18] attempted to propose a solution for predicting group performance and personality traits by mining typical behavioral patterns. Subsequently, a mining approach for extracting co-occurrent events from multimodal time-series data for personality estimation was also proposed by Okada et al. [29]. Batrinca et al. [6] conducted a comparative analysis to observe the difference between the personality trait recognition accuracies obtained for a human-machine interaction (HMI) scenario and a human-human interaction (HHI) scenario.

In addition to the studies mentioned above, several studies specifically focused on improving Big Five personality trait prediction. For instance, Fang et al. [16] used three nonverbal features, including intrapersonal features, dyadic features, and one-vs.-all features, to predict the Big Five model. Lin et al. [22] developed a Big Five predictor based on the use of an interaction mechanism in bidirectional long short-term memory (BLSTM) to model the vocal behaviors of participants. In the prior study [25], communication skills and task types were considered for estimating the Big Five personality traits.

Our current work differs significantly from the existing studies in terms of the utilized features and dataset dependency. In most studies, low-level features were extracted for the estimation process. We consider the transfer learning technique by extracting higher-level features using state-of-the-art pretrained speaker embedding models (i-vector and x-vector extractors [13, 47]). To ensure the effectiveness of our proposed system regardless of the selected language, we use two different language corpora, i.e., a European language corpus and a Japanese corpus (Table 1).

In this study, we utilized two multimodal data corpora, i.e., the MATRICS corpus and ELEA-AV corpus. Figure 1 presents an overview of these corpora. The MATRICS corpus was used as the main dataset for analyzing the effectiveness of each modality. In addition, the ELEA-AV corpus was used to analyze the speaker individuality features as audio-related features despite the different nature of this dataset.

In this study, we extracted three modality groups (i.e., audio, language, and motion & visual groups) and communication skills indices as the inputs for Big Five estimation. Table 2 shows a summary of the multimodal features.

In our preliminary study [25], one of the objectives was to clarify the effectiveness of verbal and nonverbal features and CS indices for estimating the Big Five personality traits. We extracted audio-related features in the same manner as the baseline system in the Interspeech 2012 Speaker Trait Challenge [42] designed for estimating perceived speaker traits from single speaker utterances. In contrast, we aimed to thoroughly study which audio-related features are more suitable for estimating the self-assessed speaker traits of each participant in a group discussion, as provided in the MATRICS corpus. Self-assessed speaker traits are more robust than perceived speaker traits, regardless of the speech content and environment. Since the sizes of the group discussion corpora are relatively limited, we also considered performing transfer learning by using the state-of-the-art speaker individuality features for estimating the Big Five personality traits. Figure 7 shows the main ideas of our experimental process.

The experiment in the current study aimed to investigate the effectiveness of (1) speaker individuality features (i-vector and x-vector) (Sect. 5.2.1); (2) nonverbal behaviors, e.g., face gestures (Sect. 5.2.2); and (3) a combination of modality groups (Sect. 5.2.3) for Big Five personality trait estimation in both the MATRICS and ELEA-AV corpora. Accordingly, we conducted unimodal analysis followed by multimodal analysis by considering each modality group. An ablation test was also conducted to study the importance of each modality group. In this study, the experiment was conducted as a binary classification task (similar to [2, 29]). The input was the combination of features explained in Sect. 4, and the targets were the Big Five personality trait scores, i.e., neuroticism (N), extraversion (E), openness (O), agreeableness (Ag), and conscientiousness (C). As mentioned above, the Big Five scores were obtained from a self-assessed questionnaire, which is usually more accurate but more difficult to predict than the perceived Big Five scores used in prior studies [22, 29].

In this section, we discuss the key information obtained in this study. We also discuss the prospective multimodal interfaces that utilize the results of our findings. Finally, the limitations and the future direction to address the remaining issues in this study will be discussed.

From the experimental results, as shown in Sect. 5.2, we can discuss two main points that answer the following key questions. Most well-known studies for personality trait estimation focused on self presentation scenarios. For instance, the Speaker Trait Challenge 2012 [42] and the ChaLearn Looking at People 2016 [36]. However, the findings from these studies might be limited because psychological science suggested that situations and social interactions are highly associated with personality states [31]. Only a few studies worked on predicting personality traits in social interactions, including [19, 22, 25, 29]. This study specifically addressed the personality traits estimation using the speaker individuality and multimodal cues in multiple languages group discussion corpora (i.e., MATRICS and ELEA-AV).

As one of the primary key findings, the speaker individuality feature is considered beneficial for estimating neuroticism and extraversion traits in the European or Japanese language group discussion corpus. The neuroticism and extraversion traits are statistically significant in stimulating peoples’ attitudes when receiving or making a call at public places [24]. Hence, the estimated personality can be utilized in a virtual call center for giving customer-centric responses. Besides, we can also build an interface based on speaker embedding to detect the user’s attitude. Similarly, the multimodal analysis results of this study could also be used for developing a virtual agent for group interactions that can respond appropriately to each participant based on the estimated personality traits. An appropriate response could lead to a smooth conversation.

While this study provides several key findings, the corpora used in this study might be considerably small-size, limited to the group discussion settings, and consist of only European and Japanese languages. The investigation of more diverse corpora will be considered as a future direction. In this study, we did not focus on analyzing the recent advanced machine-learning algorithms. Instead, we focus on mitigating individual differences from the relatively smaller size but more diverse group discussion corpora, which could be analyzed using classical machine learning algorithms. In future work, we will thoroughly consider how to model personality traits, and other internal properties based on the recent trends in multimodal machine learning [21].

This paper analyzed the effectiveness of the state-of-the-art speaker individuality feature, namely, the i-vector, to predict the Big Five personality traits in two different group discussion datasets. Our experimental results showed that this feature could effectively estimate the Big Five personality traits in both datasets, i.e., MATRICS and ELEA. A significant improvement was obtained when predicting the neuroticism and extraversion traits. Subsequently, a multimodal analysis was also carried out to compare the effectiveness of each modality and psychological feature. The psychological features included CS and Ld indices. The results showed that the audio-related features contributed most significantly to this task. An improvement could be achieved by using motion-related features, especially for predicting the conscientiousness trait. Furthermore, the i-vector speaker embedding system could improve the estimation results of personality traits, even when only using one modality (audio-related).

In our future work, we will develop a multimodal interface based on speaker embedding for automatic personality trait estimation using multimodal features. For instance, an interface can give adequate personalized feedback to the user based on the estimated traits. Additionally, recent multimodal machine learning approaches, the relationship between personality traits and other internal properties, and the explainability of the multimodal cues will be thoroughly investigated.