Emotions and Personality in Personalized Services

Personalized systems traditionally used the traces of user interactions to learn the user model, which was used by sophisticated algorithms to choose the appropriate content for the user and the situation. Recently, new types of user models started to emerge, which take into account more user-centric information, such as emotions and personality. Initially, these models were conceptually interesting but of little practical value as emotions and personality were difficult to acquire. However, with the recent advancement in unobtrusive technologies for the detection of emotions and personality these models are becoming interesting both for researchers and practitioners in the domain of personalized systems. This chapter introduces the book, which aims at covering the whole spectrum of knowledge needed to research and develop emotion- and personality-aware systems. The chapters cover (i) psychological theories, (ii) computational methods for the unobtrusive acquisition of emotions and personality, (iii) applications of personalized systems in recommender systems, conversational systems, music information retrieval, and e-learning, (iv) evaluation methods, and (v) privacy issues.

The work overviews the theoretical models of emotions mainly used by computer scientists in the area of user modeling and sentiment analysis. Central in this regard are the dimensional models in which the body side is crucial, and the cognitive ones in which the evaluation processes give rise to emotions. Special attention is devoted to a socio-cognitive model of emotions in terms of goals and beliefs, focusing on social emotions, both related to image (admiration, bitterness, enthusiasm) and to self-image (pride, shame). Nature, function, and typical body signals of these emotions are overviewed.

In this chapter, we introduce and discuss some of the most important and widely used models of personality. Focusing on trait theories, we first give a brief overview of the history of personality research and assessment. We then move on to discuss some of the most prominent trait models of the nineteenth century—including Allport’s trait theory, Cattell’s 16 Factor Model, Eysenck’s Giant Three, and the Myers–Briggs Type Indicator (MBTI)—before focusing on the Big Five Model (Five Factor Model), which is the most widely accepted trait model of our time. Next, we introduce alternatives to the Big Five that appear to be useful in the context of personalized services (the HEXACO and RIASEC models), and subsequently outline the relationships between all the models discussed in the chapter. Finally, we provide an outlook on innovative methods of predicting personality with the help of digital footprints.

This chapter gives a brief overview on the state of the art in emotion and affect acquisition for various modalities and representation forms—mainly discrete and continuous. Its main content covers a survey of existing computational models and tools (i.e. off-the-shelf solutions), before looking at future efforts needed covering the current trends and the open issues.

This chapter provides an overview of the methods that can be used to automatically acquire information about an individual’s personality. In particular, we focus our attention on the sources of data (e.g. text, audio, video, mobile phones, wearable sensors, etc.) and the features used to automatically infer personality. For each data source, we discuss the methods of extracting the cues used to detect personality, as well as the major findings. Lastly, we refer some limitations of the current research which is relevant for the advancement of the state of the art.

Social signal processing is the domain aimed at modelling, analysis and synthesis of nonverbal communication in human–human and human–machine interactions. The core idea of the field is that common nonverbal behavioural cues—facial expressions, vocalizations, gestures, postures, etc—are the physical, machine-detectable evidence of social phenomena such as empathy, conflict, interest, attitudes, dominance, etc. Therefore, machines that can automatically detect, interpret and synthesize social signals will be capable to make sense of the social landscape they are part of while, possibly, participating in it as full social actors.

In this chapter, we review and discuss the state of the art on sentiment analysis in social streams—such as web forums, microblogging systems, and social networks, aiming to clarify how user opinions, affective states, and intended emotional effects are extracted from user generated content, how they are modeled, and how they could be finally exploited. We explain why sentiment analysis tasks are more difficult for social streams than for other textual sources, and entail going beyond classic text-based opinion mining techniques. We show, for example, that social streams may use vocabularies and expressions that exist outside the mainstream of standard, formal languages, and may reflect complex dynamics in the opinions and sentiments expressed by individuals and communities.

The Experience Sampling Method (ESM) introduces in-situ sampling of human behaviour, and provides researchers and behavioural therapists with ecologically valid and timely assessments of a person’s psychological state. This, in turn, opens up new opportunities for understanding behaviour at a scale and granularity that was not possible just a few years ago. The practical applications are many, such as the delivery of personalised and agile behaviour interventions. Mobile computing devices represent a revolutionary platform for improving ESM. They are an inseparable part of our daily lives, context-aware, and can interact with people at suitable moments. Furthermore, these devices are equipped with sensors, and can thus take part of the reporting burden off the participant, and collect data automatically. The goal of this survey is to discuss recent advancements in using mobile technologies for ESM (mESM), and present our vision of the future of mobile experience sampling.

In this chapter we describe publicly available datasets with personality and affective parameters relevant to the research questions covered by this book. We briefly describe the available data, acquisition procedure, and other relevant details of these datasets. There are three datasets acquired through the users’ natural interaction with different services: LDOS CoMoDa, LJ2M and myPersonality. Two datasets were acquired in controlled, laboratory settings: LDOS PerAff-1 and DEAP. Finally, we also mention four stimuli datasets from the Media Core project: ANET, IADS, ANEW, IAPS, as well as the 1000 songs dataset. We summarise this information for a quick reference to researchers interested in using these datasets or preparing the acquisition procedure of their own.

With the growing number of conversational systems that find their way in our daily life, new questions and challenges arise. Even though natural conversation with agent-based systems has been improved in the recent years, e.g., by better speech recognition algorithms, they still lack the ability to understand nonverbal behavior and conversation dynamics—a key part of human natural interaction. To make a step towards intuitive and natural interaction with virtual agents, social robots, and other conversational systems, this chapter proposes a probabilistic framework that models the dynamics of interpersonal cues reflecting the user’s social attitude within the context they occur.

Diversity is increasingly recognized as an important metric for evaluating the effectiveness of online recommendations. However, few studies have fully explored the possibility of realizing personalized diversity in recommender systems by taking into account the individual user’s spontaneous needs. In this chapter, we emphasize the effect of users’ personality on their needs for recommendation diversity. We start with a review of the two branches of research in this area, diversity-oriented recommender systems (RS) and personality -based RS. We then report the results from a user survey that we conducted with the aim of identifying the relationship between personality and users’ preferences for recommendation diversity. For instance, the personality trait of conscientiousness can affect users’ preferences not only for diversity in respect of a particular attribute (such as movie genre, country, or release time), but also their preference for overall diversity when all attributes are considered. Motivated by the survey findings, we propose a personality -based diversity-adjusting strategy for recommender systems, and demonstrate its significant merit in improving users’ subjective perceptions of the system’s recommendation accuracy. Finally, we consider implications and suggestions for future research directions.

This chapter reports how affective computing (in terms of detection methods and intervention approaches) is considered in adaptive e-learning systems. The goal behind is to enrich the personalized support provided in online educational settings by taking into account the influence that emotions and personality have in the learning process. The main contents of the chapter consist in the review of 26 works that present current research trends regarding the detection of the learners’ affective states and the delivery of the appropriate affective support in diverse educational settings. In addition, the chapter discusses open issues regarding affective computing in the educational domain.

Music and places can both trigger emotional responses in people. This chapter presents a technical approach that exploits the congruence of emotions raised by music and places to identify music tracks that match a place of interest (POI). Such technique can be used in location-aware music recommendation services. For instance, a mobile city guide may play music related to the place visited by a tourist, or an in-car navigation system may adapt music to places the car is passing by. We address the problem of matching music to places by employing a controlled vocabulary of emotion labels. We hypothesize that the commonality of these emotions could provide, among other approaches, the base for establishing a degree of match between a place and a music track, i.e., finding music that “feels right” for the place. Through a series of user studies we show the correctness of our hypothesis. We compare the proposed emotion-based matching approach with a personalized approach where the music track is matched to the music preferences of the user, and to a knowledge-based approach which matches music to places based on metadata (e.g., matching music that was composed during the same period that the place of interest was built in). We show that when evaluating the goodness of fit between places and music, personalization is not sufficient and that the users perceive the emotion-based music suggestions as better fitting the places. The results also suggest that emotion-based and knowledge-based techniques can be combined to complement each other.

Recommender systems are decision aids that offer users personalized suggestions for products and other items. Context-aware recommender systems are an important subclass of recommender systems that take into account the context in which an item will be consumed or experienced. In context-aware recommendation research, a number of contextual features have been identified as important in different recommendation applications: such as companion in the movie domain, time and mood in the music domain, and weather or season in the travel domain. Emotions have also been demonstrated to be significant contextual factors in a variety of recommendation scenarios. In this chapter, we describe the role of emotions in context-aware recommendation, including defining and acquiring emotional features for recommendation purposes, incorporating such features into recommendation algorithms. We conclude with a sample evaluation, showing the utility of emotion in recommendation generation.

This chapter presents our findings on emotional and color perception of music. It emphasizes the importance of user-aware music information retrieval (MIR) and the advantages that research on emotional processing and interaction between multiple modalities brings to the understanding of music and its users. Analyses of results show that correlations between emotions, colors and music are largely determined by context. There are differences between emotion-color associations and valence-arousal ratings in non-music and music contexts, with the effects of genre preferences evident for the latter. Participants were able to differentiate between perceived and induced musical emotions. Results also show how associations between individual musical emotions affect their valence-arousal ratings. We believe these findings contribute to the development of user-aware MIR systems and open further possibilities for innovative applications in MIR and affective computing in general.

Recommender systems analyze a user’s past behavior, build a user profile that stores information about her interests, maybe find others who have a similar profile, and use that information to find potentially interesting items. The main limitation of this approach is that provided recommendations are accurate, because they match the user profile, but not useful as they fall within the existing range of user interests. This drawback is known as overspecialization. New methods are being developed to compute serendipitous recommendations, i.e. unexpected suggestions that stimulate the user curiosity toward potentially interesting items she might not have otherwise discovered. The evaluation of those methods is not simple: there is a level of emotional response associated with serendipitous recommendations that is difficult to measure. In this chapter, we discuss the role of emotions in recommender systems research, with focus on their exploitation as implicit feedback on suggested items. Furthermore, we describe a user study which assesses both the acceptance and the perception of serendipitous recommendations, through the administration of questionnaires and the analysis of users’ emotions. Facial expressions of users receiving recommendations are analyzed to evaluate whether they convey a mixture of emotions that helps to measure the perception of serendipity of recommendations. The results showed that positive emotions such as happiness and surprise are associated with serendipitous suggestions.

The rising interest in socially assistive robotics is, at least in part, stemmed by the aging population around the world. A lot of research and interest has gone into insuring the safety of these robots. However, little has been done to consider the necessary role of emotion in these robots and the potential ethical implications of having affect-aware socially assistive robots. In this chapter we address some of the considerations that need to be taken into account in the research and development of robots assisting a vulnerable population. We use two fictional scenarios involving a robot assisting a person with Parkinson’s disease to discuss five ethical issues relevant to affect-aware socially assistive robots.