Skip to main content
Top
Published in: Minds and Machines 1/2024

Open Access 01-03-2024

Philosophical Lessons for Emotion Recognition Technology

Author: Rosalie Waelen

Published in: Minds and Machines | Issue 1/2024

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Emotion recognition technology uses artificial intelligence to make inferences about a person’s emotions, on the basis of their facial expressions, body language, tone of voice, or other types of input. Underlying such technology are a variety of assumptions about the manifestation, nature, and value of emotions. To assure the quality and desirability of emotion recognition technology, it is important to critically assess the assumptions embedded in the technology. Within philosophy, there is a long tradition of epistemological, ontological, phenomenological, and ethical reflection on the manifestation, nature, and value of emotions. This article draws from this tradition of philosophy of emotions, in order to challenge the assumptions underlying current emotion recognition technology and to promote a more critical engagement with the concept of emotions in the tech-industry.
Notes
Rosalie Waelen is the sole author of this paper.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

1 Introduction

Emotion recognition technology (or just ‘emotion recognition’) is one of many possible applications of artificial intelligence (AI). The technology is sometimes also referred to as ‘affect recognition’ or ‘sentiment analysis’. Similarly, throughout the history of philosophy of emotions we find references to ‘affects’, ‘sentiments’, or ‘passions’. AI-enabled emotion recognition can take several forms, which has to do with the goal of a specific application as well as the type of data that is used. The most popular input for emotion recognition are facial expressions – meaning images or videos of people’s faces. Other potential data sources for emotion recognition are gestures and body language (visual data), spoken or written words (textual data), the tone of one’s voice (audio), or body temperatures (physiological data).
The main techniques used now to analyze these types of data are computer vision and natural language processing, both of which build predominantly on neural networks. Developments in deep learning in the last decade therefore also boosted research and development of emotion recognition. Work in “computer-assisted quantification of facial expressions” started already in the 1990’s though (Abhang et al., 2016, p. 8). In the same decade, Rosalind Picard introduced the term ‘affective computing’ and started researching other ways of automating emotion recognition (Picard, 1997). In 2009, Picard co-founded the company Affectiva, which uses ‘emotion AI’ to help businesses to better understand their customers, to help the automotive industry to improve driver safety, and to support behavioral research.1 Other potential uses of emotion recognition are: assessing job applicants, analyzing audiences’ responses to content, predictive policing, helping people with autism in social interactions, and improving human-computer interaction (Whittaker et al., 2018; Hippe, 2016; Picard & Klein, 2002).
Like with other emerging technologies, philosophers and ethicists can support emotion recognition research by reflecting on the potential ethical and societal implications of the technology. However, there is another role to play for philosophers when it comes to emotion recognition. Building on a long tradition of philosophy of emotions, philosophers can critically assess the assumptions that underly emotion recognition applications. Fundamental philosophical questions about the manifestation, nature, and value of emotions appear to be neglected within emotion recognition research. This article looks into the epistemological, ontological, phenomenological, and moral debates about emotions and asks what lessons can be drawn from the philosophical tradition that is the philosophy of emotions in the context of emotion recognition technology.
The core of the article is divided into three parts: a section on the manifestation of emotions, discussing whether and how emotions can be observed empirically (Sect. 2); a section on the nature of emotions, dealing with ontological and phenomenological questions about emotions (Sect. 3); and a section on the evaluation of emotions, explaining different normative views on emotions (Sect. 4). The conclusion (Sect. 5) summarizes which lessons the philosophy of emotions can teach those who develop, sell, or use emotion recognition tools.

2 The Manifestation of Emotions

The history of the philosophy of emotions goes back to the very beginning of (western) philosophy. The pre-Socratics already discussed emotions, as did Socrates, Plato, and Aristotle (Solomon, 2008). For the longest time, questions about the nature and manifestation of emotions were addressed by philosophers. Nowadays, science and philosophy have become more divided, and different scientific disciplines are concerned with different aspects of emotions. Questions about the manifestation of emotions are therefore not merely philosophical questions, but also of relevance to behavioral sciences.
The question ‘How can we recognize or observe an emotion?’ is an empirical question. Observable aspects of an emotion are either physiological, expressive, or behavioral (Scarantino & de Sousa, 2021). Physiological signs are for example heart rate and body temperature. Bodily expressions of emotions can be facial movements or body language. Other ways of expressive emotions are for example through the tone of one’s voice or the words one chooses. Behavioral signs of emotions are behavioral patterns that commonly follow a given emotion. Physiology, bodily expressions, and behavior are not the only aspects of emotions though. Other aspects of emotions are mental states, experiences, and evaluations. Note that, although these aspects are not observable in the same way as physiological changes, bodily expressions, and behavioral patterns, that does not mean that they are not measurable. Mental states, experiences, and evaluations are discussed in more detail in Sects. 3 and 4.
Determining if and how an emotion can be observed is of course crucial to the success of emotion recognition technology. Because physiology, expressions, and behavior are the (most) observable aspects of emotions, these aspects are usually also the basis for emotion recognition research and development. Emotion recognition can be based on one of these aspects or on a combination of the three. The remainder of this section discusses each of these three observable aspects in relation to emotion recognition. The section concludes with three questions about the manifestation of emotions, that are important for emotion recognition research and development.

2.1 Expressions

Expressions of emotions can take many forms. We express our emotions through our body language and facial movements, but also with our voice and through our language. Body language and facial movements can be recognized, i.e. categorized, by means of computer vision models. The recognition of expressions in written and spoken language can be automized using natural language processing. Although both areas are currently heavily researched, facial expressions are probably the most popular basis for emotion recognition. The analysis of facial expressions by means of computer vision is for example the basis for the aforementioned uses of emotion recognition to analyze how audiences respond to certain content or to analyze the emotional state of drivers in a car.
Emotion recognition by facial analysis is mostly based on the research of psychologist Paul Ekman, who argued that humans everywhere, irrespective of their cultural background, express a set of six basic emotions with the same facial expressions (Ekman, 1973). The six emotions that Ekman distinguished are happiness, anger, sadness, fear, disgust, and surprise. This set of basic emotions is often used in emotion recognition research and development, even for emotion recognition applications that are not based on facial expressions. However, Ekman’s work on basic emotions and facial expressions does not provide sufficient scientific basis for emotion recognition, according to Whittaker and colleagues (2018). The latter point out that “the basic foundations of these theories remain controversial among psychologists” and that Ekman’s six categories are too “rigid” and his understanding of the physiological causes of emotions too “simplistic” (2018, p. 14). There is no consensus among psychologists that emotions are natural kinds in the first place, which means that it cannot be considered a scientific fact that there is a set of emotions that is experienced and expressed in the same way by all humans throughout history and across cultures (Barrett, 2006). Barrett et al. (2019) argue that more research is needed about the ability to read emotional states from facial movements or expressions and that it is premature for tech-companies to try to develop emotion recognition technology based on facial analysis.
So, despite its popularity, emotion recognition on the basis of facial analysis has received a lot of criticism as well. Among other things, emotion recognition on the basis of facial analysis is accused of being unscientific. Members of the AI Now Institute write that emotion recognition on the basis of facial analysis is “not backed by robust scientific evidence” and used in ways that “recall the pseudosciences of phrenology and physiognomy” (Whittaker et al., 2018, p. 4). Furthermore, given that Ekman’s categories of basic emotions were meant to categorize facial expressions, research and development in emotion recognition should not use Ekman’s categories in other contexts (e.g. emotion analysis of language), unless there is sufficient scientific evidence that the basic emotion categories apply in other contexts.

2.2 Physiology

Physiological signs are another popular basis for emotion recognition. Much of emotion recognition research and development is dedicated to the possibility of recognizing emotions on the basis of body temperatures, in particular through heat maps of the face (e.g. Cruz-Albarran et al., 2017; Ordun et al., 2020). These face-heatmaps usually make use of Ekman’s six categories of basic emotions as well. The underlying idea of this approach to emotion recognition is that different emotions correspond to different temperatures in certain areas of the face or other parts of the body.
The common view within psychology and the philosophy of emotions is that experiencing or feeling an emotion is followed by corresponding physiological responses. When I feel shame, for example, that feeling is followed by my blushing. When I feel fear, that feeling is followed by an increased heart rate. In contrast to this common view, however, James (1884) argued that emotions emerge from their manifestation. According to James, emotions are the result of bodily changes, not the cause of those changes. Hence, for James emotions are an experience or awareness of something happening in the body. Following this view, we are sad because we cry, not the other way around. Whether James’ divergent view is correct or not, it teaches emotion recognition developers that they need to be careful when making assumptions about the relation between different aspects of emotions – such as a physiological state and an experience or cognitive state.

2.3 Behavior

A third manifestation of emotions is behavior. Emotions are usually thought of as motivating certain behavioral patterns. This means that certain behaviors can be traced back to a mental state, the feeling of an emotion, and linked to corresponding expressions or physiological signs. For example, anger can be recognized on the basis of a frowning face and increased heartbeat, but also through aggressive forms of behavior. Emotion recognition technology based on behavior technology can be used for educational purposes or support predictive policing. Behavior analysis technology can also improve behavioral science, especially when combined with facial analysis, speech analysis, or physiological analysis.
For behavioralists, behavioral patterns are central to their understanding of emotions. Well-known for defending this view is Ryle (1951). According to Ryle (1951), emotions are mere dispositions to behave in a certain way (although Ryle did admit that these dispositions are accompanied by certain feelings or agitations). The behavioralist view again shows that the ways in which emotions may manifest themselves and the meanings of these manifestations are not a given, but a topic of scientific debate.
That we can observe facial expressions, physiological changes, and behavioral patterns is obvious. Also clear is that AI can be a useful tool in identifying or tracking our expressions, physiology, and behavior. Less of a given is what conclusions we can draw about a person’s emotions on the basis of observations about their expressions, physiology, and behavior.
A first critical question to guide emotion recognition research and development is: Are the aspects to be observed necessary or sufficient to identify an emotion? If, for instance, an emotion recognition system is programmed in such a way that it is assumed that a smiling face is a necessary condition for feeling joy, then those who do not smile or have an unusual smile will never get recognition for their joy. If, on the other hand, a smiling face is considered a sufficient condition for labeling someone as joyous, people can fake a smile and pass the joy-test. Hence, uncritical assumptions about the necessity or sufficiency of certain observable aspects of emotions can lead to false and potentially problematic conclusions.
A second question to ask has to do with the relation between different aspects of emotions: Which conclusions can we draw about a person’s mental state or future behavior, on the basis of an observation of their body temperature? Understanding the importance of certain aspects of emotions and the relationship between different aspects of emotions is crucial to emotion recognition technology, because this understanding determines what information can be inferred upon recognizing an emotion. Without a clear understanding of the relation between different aspects of emotions, the possible uses of emotion recognition technology remain limited. Take the example of predictive policing. A person’s movements and body language may be recognized as angry by an emotion recognition system. As a result, the person is preemptively stopped by a police agent or security guard, in order to prevent any aggressive behavior. However, for such predictive policing to be ethically acceptable, there would have to be scientific evidence about the relation between expressions and behavior, that is, the likelihood that somebody who expresses anger will also act violently.
A third and final takeaway is that emotions may not manifest themselves in a universal manner. The critique raised against the use of Ekman’s theory about basic emotions and their facial expressions needs to be taken seriously (Barrett, 2006; Whittaker et al., 2018). Factors such as culture, age, gender, and more, may affect the ways in which people express their emotions or behave in response to an emotion. If an emotion recognition system is based on the way emotions are expressed within a certain culture, and emotional expressions are not universal, that technology would be useless in a different cultural context.

3 The Nature of Emotions

Questions about the manifestation of emotions are asked by psychologists, neuroscientists, anthropologists, and other scientists interested in the phenomenon of emotions. Questions about the nature and value of emotions are more exclusive to philosophy. Some fundamental questions about the nature of emotions are: What are emotions? What are necessary and sufficient aspects of emotions? How do emotions differ from one another? And how do emotions relate to reason? The previous section discussed three observable aspects of emotions: expressions, physiological changes, and behavioral patterns. This section looks at the phenomenological, cognitive, and evaluative aspects of emotions.

3.1 Experience

Emotions can be understood not just in terms of physiological changes, behavioral patterns, or brain activity, but also as a subjective experience. Phenomenological questions about emotions have to do with what it means to experience or feel an emotion. A first important question in this context is whether it is necessary that people feel or experience an emotion in order to say that they have an emotion. Could it be possible to say that someone is ashamed of something (for example because they blush) even if that person does not report feeling shame? Another important question in the context of experiencing emotions has to do with the differences between people. In the previous section, it was already mentioned that emotional expressions may vary due to factors such as culture, age, or gender. With regards to experience, the question arises whether people experience or feel emotions in the same way – and whether different experiences can still be categorized under the same label. Factors that may affect the experience of emotions are, for instance, neurodiversity or being under influence of alcohol or drugs. These questions about differences in experiences are also important in the context of emotion recognition technology. If there are indeed differences in experience, even when observable aspects of an emotion are the same, that means that emotion recognition technology based on the categorization of observable aspects, tells us little about a person’s subjective feeling or experience.

3.2 Cognition

The relation between cognition and emotion is another major topic in the philosophy of emotions and in research on emotions in general. If we agree with Hegel, we can interpret the history of philosophy is a history of the development or pursuit of reason. Throughout the history of philosophy, emotions have mainly been seen as the opposite of reason or a threat to reason. Emotions were taken to be something subconscious, nothing more than a subjective feeling, while rationality was associated with objectivity. Much of the engagement with the topic of emotion has therefore been about controlling emotions (Solomon, 2008). In contrast to the view that emotions are opposite to reason and rationality, some philosophers and scientists argue for a cognitive theory of emotions, suggesting that emotions are cognitive states. Nussbaum (2001), for instance, argues that emotions are (evaluative) judgements. According to this view, emotions are not opposed to rationality, but simply a different type of rationality and intelligence.
The relation between emotion and cognition is empirically investigated through fMRI scans of activity in the brain and its connection to people’s subjective reports of their emotional states. To that extent, the cognitive aspect of emotions is observable and measurable. Up for debate, however, is whether activity in the brain is a response to a feeling or is the feeling. AI could, of course, play a role in analyzing brain scans by learning which patterns of brain activity correspond to which (subjectively reported) emotions. Note, however, that such use of AI is not what is usually meant when we talk about AI-based emotion recognition.

3.3 Evaluation

It is widely accepted that emotions have intentionality (Solomon, 2008). That emotions have intentionality means that emotions are always about something; they are always directed at an object. To fully understand an emotion, then, one would also want to know the object of the emotion. Those who support a cognitive view of emotions, like Nussbaum (2001), defend that emotions have an evaluative component. The evaluative component of emotions entails that an emotion represents a normative judgement about the object at which the emotion is directed. For instance: if I am angry with my neighbor, my neighbor is the object of my anger, and my anger expresses the judgement that my neighbor has wronged me.
Given the intentional and evaluative component of emotions, one can again conclude that the information revealed by emotion recognition systems based on observable components like physiology or expressions, may be limited. This limitation is not necessarily ethically or scientifically problematic, but reduces the usefulness of such emotion recognition technology in practice.
Emotion recognition research and development should not focus solely on the manifestation of emotions – how to recognize emotions – but also consider questions about the nature of emotions. A tool that supposedly recognizes emotions automatically makes assumptions about what emotions are. In light of the ideal of transparency in AI, it is important to make those assumptions explicit. Such transparency can be realized, for instance, by providing a working definition of emotions that gives users and other parties an insight into the ways in which a specific model or a company approaches the complex topic of emotions. Making assumptions about the nature of emotions explicit is also valuable because it forces developers to think about philosophical questions like: What does it mean to experience an emotion? How do different aspects of emotions relate to one another? Do people experience or show their emotions differently?

4 The Evaluation of Emotions

In addition to the manifestation and nature of emotions, the philosophical debate about the normative evaluation of emotions is also important for research and development on AI-based emotion recognition. In the previous section, it was mentioned that emotions themselves can be understood as evaluations of their objects. Emotions are always about something. I am sad, for example, because I lost something or someone I cared about. The topic of this section is not about the evaluative aspect of emotions, but about the ways in which emotions are evaluated. Other people, cultural norms, and societal structures, evaluate or judge our emotions as being good or bad, weak or strong, right or wrong. Emotions also play a role in different theories of virtue and value (for more on this, see Tappolet (2023). Being aware of the fact that emotions can be evaluated in different ways is important in the context of emotion recognition, because it is unavoidable that these AI systems make assumptions about the value of individual emotions. To offer an example of a philosophical debate about the evaluation of emotions, this section discusses the Stoic view on emotions (based on Frede, 2001; Graver, 2017; Sellars, 2009; Vogt, 2017) and Nussbaum’s neo-stoic philosophy of emotions (as developed in Nussbaum, 2001).
As already mentioned, the relation between emotions and reason has been a major topic of debate throughout the history of philosophy. Until today, there is no consensus on the topic. For a long time, philosophers mostly portrayed emotions as opposed to reason and as a threat to reason. The Stoics are perhaps most famous for this view – although their stance on emotions’ relation to reason really was more nuanced than they often get credit for. Stoicism was an important school of thought in ancient Athens and Rome. Stoicism is first and foremost a philosophy about the good life, as goes for other ancient philosophical schools. The Stoic motto is ‘living in accordance with nature’, understanding nature as driven by an immanent divine form of reason (logos). Because the good life, nature, and reason were intimately connected for the Stoics, physics and epistemology are deeply intertwined with Stoic ethics. Stoicism is still known today for being anti-emotions, but the details of the Stoic philosophy of emotions are less known. Unlike Platonists and Aristotelians, Stoics actually believed that emotions were a part of reason. The Stoics developed and defended a cognitive-evaluative theory of emotions, meaning that they understood emotions as judgements. Most emotions are irrational or unintelligent, according to Stoics, not because they oppose a person’s own reason and rationality, but because they oppose the divine reason (logos) that is immanent in nature. Living in accordance with nature – i.e., living the good life – entails having correct judgements that align with the divine logos.
A misconception about the Stoic view on emotions is that Stoics condemned all emotions. This is not entirely true. The “passions” (pathē) that Stoics specifically condemned were pleasure (hēdonē), desire (epithumia), fear (phobos), and distress (lupē). According to Stoicism, pleasure, desire, fear, and distress are rational in nature (namely a product of human reason), but irrational or unintelligent with regards to their content (namely in contrast with divine reason). Opposed to these irrational emotions are a set of correct affective judgements: joy (chara), hope (boulēsis), and caution (eulabeia) (Graver, 2017).2 Hence, even for Stoics there is some room for emotion in the good life. Stoics argued that “bad” emotions (i.e., ones that do not align with divine logos) can be replaced by “good” ones. When one learns to follow the divine reason that is immanent to nature, one will only have appropriate, intelligent emotions.
The idea that emotions are cognitive and that one can rationally control their emotions, rather than being controlled by emotions, is what sets the Stoic philosophy of emotions apart from its contemporaries.
Nussbaum (2001) argues for a neo-stoic philosophy of emotions. Her philosophy of emotions is Stoic, because she adopts the Stoic cognitive-evaluative conception of emotions (as mentioned also in the previous section). To support the argument that emotions are cognitive, Nussbaum points out that emotions always have an object. The directedness of an emotion stems from a certain view and interpretation of the world, which is the product of a person’s beliefs, values, convictions, and so on. Hence, an emotion is not only a judgement about an object, but an evaluative and normative judgement. Nussbaum explains that emotions, despite being rational judgements, can nevertheless overcome and overwhelm us, because they are always a reaction to circumstances, and we cannot control or predict those circumstances. The stronger our emotional response, the stronger the values or beliefs that underly our judgement.
Because Nussbaum does not adopt the entire Stoic worldview, she does not have to accept the Stoic proposition that pleasure, desire, fear, and distress are necessarily incorrect judgements. Nussbaum argues for a different evaluation of emotions. Unlike the Stoics, Nussbaum does not perceive the fact that emotions make one vulnerable to and dependent on external circumstances as a threat to happiness. Instead, she argues that emotions improve one’s life by giving shape, with both highs and lows, to an otherwise flat landscape (hence the title of her 2001 book: Upheavals of thought). Because emotions give shape and meaning to our lives, Nussbaum concludes that not only the cognitive nature of emotions, but also the content of emotions makes them rational or ‘intelligent’ (hence the subtitle of her 2001 book: The intelligence of emotions). However, she also acknowledges that emotions can sometimes be wrong judgements. Emotions that resemble wrong judgements might not threaten a happy life, as the Stoics believed, but, Nussbaum argues, they are a poor guide for moral action. Given the relation of emotions to both happiness and moral action, Nussbaum concludes that emotions should be a part of ethical theory and ethical deliberation processes in practice.
This little digression into the Stoic and Nussbaum’s neo-stoic ethics of emotions goes to show that emotions can be normatively evaluated in different ways. In other words, there is no uniform ethics of emotions. Different philosophers have attached different values to emotions (Tappolet, 2023). According to the ancient Stoics, there is only room for a select set of emotions in the good life – all other emotions need to be controlled. According to Nussbaum, on the other hand, emotions make life worth living. So, to Nussbaum, all emotions are valuable with respect to happiness. However, she also argues that some emotions are objectionable to the extent that they motivate morally problematic behavior. Nussbaum, in line with other contemporary philosophers (e.g. Roeser, 2011), argues that emotions play an important part in moral action. In her work on emotions, Nussbaum promotes emotions like compassion and forgiveness, while condemning emotions like anger and disgust (e.g. Nussbaum, 2016). So, albeit for different reasons, Nussbaum’s ethics of emotions shares with Stoic ethics the idea that certain emotions should be avoided.
The lesson here for emotion recognition, is that there is no universally agreed upon ethics of emotions that an emotion recognition system can or should adopt. Different individuals, cultures, political groups, etc., will disagree on which emotions are morally good and bad, and which contribute to a happy life. However, emotion recognition technology, whether developers realize or not, will necessarily represent a particular normative view on emotions. Many philosophers of technology have argued that technology is not neutral (e.g. Winner, 1980). Like any technology, emotion recognition systems too contain implicit normative assumptions, namely assumptions about the value of particular emotions and emotions in general. Imagine the following scenario: Smart cameras in a public space (like an airport or a football stadium) are used to improve security by identifying people, tracking them, and analyzing their emotions through their body language and facial expressions. Whenever the system identifies an angry person, they send a signal to the security guards on site. This emotion recognition system is based on and reproduces the normative position that anger is an undesirable or dangerous emotion. Another example: From the tone of one’s voice, an AI assistant (like Siri or Alexa) recognizes that a user is sad. In response to this observation, the assistant recommends an up-beat playlist to cheer the user up. In such a case, the recommendation system interprets sadness as an undesirable state, which should be avoided and replaced by joy. These examples are not necessarily problematic uses of emotion recognition, but they are meant to show that the ways in which an emotion recognition system interprets and responds to emotions are inherently normative.3
Emotion recognition research and development should critically reflect on the values that an emotion recognition system should endorse and also consider whether these values align or clash with the values of those who will be analyzed by the technology. Another issue that should be considered is the possibility that a system ends up disciplining people into showing certain emotions and suppressing others. In doing so, emotion recognition systems would not only embody norms about good and bad conduct, but also endorse a vision of the good life.

5 Conclusion

The philosophy of emotions refers to a long history of epistemological, ontological, phenomenological, and normative debates about emotions. The aim of this article was to show, through a brief overview of different discussions within the philosophy of emotions, that it is important for research and development on the topic of AI-based emotion recognition to consider the different types of philosophical questions about emotions. This fifth and final section sums up three main lessons learned from the overview of debates in the philosophy of emotions.
A first lesson is that the type of data that an emotion recognition system is based on reflects assumptions about the manifestation of emotions and the relationship between different aspects of emotions. An assumption about the manifestation of emotions is for example that emotions are always observable through one’s facial expressions. Another epistemological assumption in this context is that people always express certain types of emotions with the same types of facial expressions. With respect to the relationship between different aspects of emotions, one needs to be careful when making inferences about other aspects of emotions upon observing one aspect. For instance, upon observing certain physiological changes, one needs to ask critically what this aspect of an emotion can really reveal about other aspects of emotions, such as behavioral patterns or subjective experiences.
A second lesson is that differences between people and differences between individual emotions need to be considered. When exporting a product that works in one place to another country or context, it should first be determined whether the aspect of emotions that the to-be-exported emotion recognition system is based on, applies in a different cultural context. Certain facial expressions, gestures, behaviors, or even uses of one’s voice, may be culturally dependent. Without clear evidence that an emotion recognition system is universally applicable, the product is not only unscientific but also useless. Furthermore, different types of emotions may relate differently to the various aspects of emotions. A system might work great for anger and happiness, but is not as useful when it comes to learning about sadness or empathy.
The third lesson is that the unavoidable fact that an emotion recognition system ascribes certain values to an emotion needs to be handled with care. The conclusions that a system is programmed to draw upon recognizing a specific emotion, and the actions it takes or prescribes in response to that emotion, are inherently normative. Emotion recognition systems have the power to prescribe and promote certain emotions and their corresponding behavioral patterns, while condemning others. By doing so, these systems express a specific view of moral conduct as well as a conception of the relation of emotions to the good life. However, the philosophy of emotions teaches us that there is no universal ethics of emotions. The normative views that are imbedded in an emotion recognition system can clash with the views of individual users or the society in which the system is used. Because of these value conflicts and the potential disciplinary power of emotion recognition technology, it is important that emotion recognition developers are aware of the normative assumptions that underly their technology.
These three lessons are meant to function as a guide for AI research and development, to help to identify and assess the philosophical and scientific assumptions that underly emotion recognition technology. Take the example of improving driver safety by analyzing drivers’ facial expressions (mentioned in the introduction). To successfully implement emotion recognition in vehicles, with improved driver safety as a result, it is important to research the relation between facial expressions and driving behavior, consider whether the technology relevant for all members of society and across cultures, and reflect on which values a system should attach to which emotions and, consequently, endorse. Future research could look into the importance and applicability of these lessons in the context of human-computer interaction (HCI), where emotions also play a central role.

Declarations

Competing interests

The author states that there are no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://​creativecommons.​org/​licenses/​by/​4.​0/​.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Footnotes
1
See https://​www.​affectiva.​com/​. Accessed July 3rd, 2023.
 
2
According to the Stoics, joy is the correct counterpart of pleasure, hope the correct version of desire, and caution the replacement for fear. One could argue, in the light of Seneca’s De Clementia, that mercy forms the correct counterpart of distress (Seneca, 2009).
 
3
These two examples are fictive, although technologically possible.
 
Literature
go back to reference Cruz-Albarran, I. A., et al. (2017). Human emotions detection based on a smart-thermal system of thermographic images. Infrared Physics & Technology, 81, 250–261.CrossRef Cruz-Albarran, I. A., et al. (2017). Human emotions detection based on a smart-thermal system of thermographic images. Infrared Physics & Technology, 81, 250–261.CrossRef
go back to reference Ekman, P. (1973). Darwin and facial expression. Academic. Ekman, P. (1973). Darwin and facial expression. Academic.
go back to reference Frede, M. (2001). On the Stoic Conception of the Good. Topics in Stoic Philosophy. Edited by Ierodiakonou. Clarendon. 71–94. Frede, M. (2001). On the Stoic Conception of the Good. Topics in Stoic Philosophy. Edited by Ierodiakonou. Clarendon. 71–94.
go back to reference Graver, M. (2017). The Stoics’ Ethical Psychology. The Cambridge Companion to Ancient Ethics. Edited by C. Bobonich. Cambridge University Press. 200–217. Graver, M. (2017). The Stoics’ Ethical Psychology. The Cambridge Companion to Ancient Ethics. Edited by C. Bobonich. Cambridge University Press. 200–217.
go back to reference James, W. (1884). What is an emotion? Mind, 9. 188– 205. James, W. (1884). What is an emotion? Mind, 9. 188– 205.
go back to reference Nussbaum, M. (2001). Upheavals of thought: The intelligence of emotions. Cambridge University Press. Nussbaum, M. (2001). Upheavals of thought: The intelligence of emotions. Cambridge University Press.
go back to reference Nussbaum, M. (2016). Anger and forgiveness: Resentment, generosity, Justice. Oxford University Press. Nussbaum, M. (2016). Anger and forgiveness: Resentment, generosity, Justice. Oxford University Press.
go back to reference Ordun, C., Raff, E., & Purushotham, S. (2020). The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data. Presented at AAAI FSS-20: Artificial Intelligence in Government and Public Sector, Washington, DC, United States. Ordun, C., Raff, E., & Purushotham, S. (2020). The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data. Presented at AAAI FSS-20: Artificial Intelligence in Government and Public Sector, Washington, DC, United States.
go back to reference Picard, R. W. (1997). Affective Computing. MIT Press. Picard, R. W. (1997). Affective Computing. MIT Press.
go back to reference Roeser, S. (2011). Moral emotions and intuitions. Palgrave Macmillan. Roeser, S. (2011). Moral emotions and intuitions. Palgrave Macmillan.
go back to reference Ryle, G. (1951). The concept of mind. Barnes & Noble. Ryle, G. (1951). The concept of mind. Barnes & Noble.
go back to reference Sellars, J. (2009). The art of living: The stoics on the nature and function of philosophy. Bristol Classical. Sellars, J. (2009). The art of living: The stoics on the nature and function of philosophy. Bristol Classical.
go back to reference Seneca, L. A. (2009). De Clementia. Oxford University Press. Seneca, L. A. (2009). De Clementia. Oxford University Press.
go back to reference Solomon, R. C. (2008). The Philosophy of Emotions. Handbook of Emotions, third edition. Edited by Lewis, Haviland-Jones, and Barrett. Guilford Press. 3–16. Solomon, R. C. (2008). The Philosophy of Emotions. Handbook of Emotions, third edition. Edited by Lewis, Haviland-Jones, and Barrett. Guilford Press. 3–16.
go back to reference Tappolet, C. (2023). Philosophy of emotion. A contemporary introduction. Routledge. Tappolet, C. (2023). Philosophy of emotion. A contemporary introduction. Routledge.
go back to reference Vogt, K. M. (2017). The Stoics on Virtue and Happiness. The Cambridge Companion to Ancient Ethics. Edited by C. Bobonich. Cambridge University Press. 183–199. Vogt, K. M. (2017). The Stoics on Virtue and Happiness. The Cambridge Companion to Ancient Ethics. Edited by C. Bobonich. Cambridge University Press. 183–199.
go back to reference Whittaker, M., et al. (2018). AI now Report 2018. AI Now Institute. Whittaker, M., et al. (2018). AI now Report 2018. AI Now Institute.
go back to reference Winner, L. (1980). Do Artefact Have Politics? Daedalus, 109(1). 121–136. Winner, L. (1980). Do Artefact Have Politics? Daedalus, 109(1). 121–136.
Metadata
Title
Philosophical Lessons for Emotion Recognition Technology
Author
Rosalie Waelen
Publication date
01-03-2024
Publisher
Springer Netherlands
Published in
Minds and Machines / Issue 1/2024
Print ISSN: 0924-6495
Electronic ISSN: 1572-8641
DOI
https://doi.org/10.1007/s11023-024-09671-3

Other articles of this Issue 1/2024

Minds and Machines 1/2024 Go to the issue

Premium Partner