Skip to main content
SpringerLink
Log in

Modeling Multimodal Expression of User’s Affective Subjective Experience

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

With the growing importance of information technology in our everyday life, new types of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major role not only in an individual’s cognitive processes but also in an individual’s interaction with others. We identify three key points to be considered when developing systems that capture affective information: embodiment (experiencing physical reality), dynamics (mapping experience and emotional state with its label) and adaptive interaction (conveying emotive response, responding to a recognized emotional state). We present two computational systems that implement those principles: MOUE (Model Of User Emotions) is an emotion recognition system that recognizes the user’s emotion from his/her facial expressions, and from it, adaptively builds semantic definitions of emotion concepts using the user’s feedback; MIKE (Multimedia Interactive Environment for Kansei communication) is an interactive adaptive system that, along with the user, co-evolves a language for communicating over subjective impressions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Averill, J.: 1994, I Feel, Therefore I Am - I Think. In: P. Ekman and R. Davidson (eds.): The Nature of Emotion: Fundamental Questions. Oxford University Press.

  • Bartlett, M., Hager, J., Ekman, P. and Sejnowski, T.: 1999, Measuring facial expressions by computer. Psychophysiology 36(2), 253–263.

    Article  PubMed  Google Scholar 

  • Bianchi, N., Mussio, P., Padula, M. and Rinaldi, G. R.: 1996, Multimedia Document Management: An anthropocentric Approach. Information Processing and Management 32(3), 287–304.

    Article  Google Scholar 

  • Bianchi-Berthouze, N., Berthouze, L. and Kato, T.: 1999, Understanding Subjectivity: An Interactivist View. In: J. Kay (eds.): Proceedings of the International Conference on User Modeling, Banff, Canada. CISMcourse and lectures No. 407. Wien, New York: Springer, 3–12.

    Google Scholar 

  • Bianchi-Berthouze, N. and Kato, T.: 2000, Querying and Personalizing the Web: A Multimedia Personal Assistant. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Nashville, TE.

  • Bianchi-Berthouze, N.: 2001, Kansei-Mining: Identifying visual impression as patterns in images. To appear in: Proceedings of the International Conference IFSA/NAFIPS, Vancouver (CA).

  • Bianchi-Berthouze, N. and Berthouze, L.: 2001, Exploring Kansei in multimedia information.To appear in: International Journal on Kansei Engineering.

  • Biocca, F.: 1997, The Cyborg’s Dilemma: Embodiment in Virtual Environments. In: J. Marsh, C. Nehaniv, and B. Gorayska (eds.): Cognitive Technology. 12–26.

  • Black, M. and Yacoob, Y.: 1997, Recognizing Facial Expressions in Image Sequences using Local Parametric Models of Image Motion. International Journal of Computer Vision 25(1), 23–48.

    Article  Google Scholar 

  • Bradshaw, J.: 1997, An Introduction to Software Agents. In: J. Bradshaw (ed.): Software Agents. MIT Press, 3–46.

  • Cahn, J.: 1990, The generation of affect in synthesized speech. Journal of the American Voice I/O Society 8, 1–19.

    Google Scholar 

  • Chauvin, Y. and Rumelhart, D.: 1995, Backpropagation: Theory, Architectures, and Applications. Lawrence Erlbaum Associates.

  • Cipolla, R. and Pentland, A. E.: 1998, Computer Vision for Human-Machine Interaction. Cambridge, MA: Cambridge University Press.

    Google Scholar 

  • Damasio, A.: 1994, Descartes’ Error. Avon Books.

  • Davidson, R.: 1994, Honoring biology in the study of affective style. In: P. Ekman and R. Davidson(eds.): The Nature of Emotion: Fundamental Questions. Oxford University Press, 321–328.

  • Ekman, P. and Rosenberg, E.: 1997, What the Face Reveals: Basic and Applied Studies of Spontaneous ExpressionUsing the Facial Action Coding System. Oxford University Press.

  • Ekman, P., Levenson, R. W. and Friesen, W.: 1983, Autonomic System Activity Distinguishes among Emotions. Science 221, 1208–1210.

    PubMed  Google Scholar 

  • Essa, I.: 1999, Computers Seeing People. AI Magazine 20(2), 69–82.

    Google Scholar 

  • Essa, I. and Pentland, A: 1997, Coding, analysis, interpretation, and recognition of facial expressions. IEEE transactions on pattern analysis and machine intelligence 19(7).

  • Fleming, M.: 1997, Neural Network Model of Micro and Macroprosody. Ph.D. Thesis Psychology Department Stanford University.

  • Frijda, N. H.: 1986, The Emotions. NewYork: Cambridge University Press.

    Google Scholar 

  • Gross, J.: 1996, Antecedent and Response-Focused Emotion Regulation: Divergent Consequences for Experience. Journal of Personality and Social Psychology 73, 435–448.

    Google Scholar 

  • Haralick, R. M., Shanmugan, K. and Dinstein, I.: 1973, Texture Features for Image Classification. IEEE Transactions Systems, Man and Cybernetics 3, 610–621.

    Google Scholar 

  • Hattori, R., Fujiyoshi, F. and Iida, M.: 1999, An Education System on WWW for Study Color Impression of Art Paintings Applied NetCatalog. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 218-223.

    Google Scholar 

  • Hayes-Roth, B., Ball, G., Lisetti, C. and Picard, A. and Stern, R.: 1998, Panel on Affect and Emotion in the User Interface. In: Proceedings of the 1998 International Conference on Intelligent User Interfaces (IUI’98), 91–94.

  • Imai, T., Yamauchi, K. and Ishi, N.: 1999, Color Coordination System on Case Based Reasoning System using Neural Networks. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 218–229.

    Google Scholar 

  • Inder, R., Bianchi-Berthouze, N. and Kato, T.: 1999, K-DIME: A Software Framework for Kansei Filtering of Internet Material. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 358–363.

    Google Scholar 

  • Isomoto, Y., Yoshine, K. and Yamasahi Ishi, N. H.: 1999, Color, Shape and Impression keywords as Attributes of Painting for Information Retrieval. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 257–262.

    Google Scholar 

  • Izard, C.: 1990, Facial Expressions and the Regulation of Emotions. Journal of Personality and Social Psychology 58(3), 487 498.

    Article  PubMed  Google Scholar 

  • Izard, C.: 1993, Four Systems for Emotion Activation: Cognitive and Noncognitive Processes. Psychological Review 100(1), 68–90.

    Article  PubMed  Google Scholar 

  • Izard, C. E.: 1977, Human Emotions. New York: Plenum Press.

    Google Scholar 

  • Jameson, A., Paris, C. and Tasso, C. (eds.): 1997, User Modeling: Proceedings of the sixth International Conference (UM’97). New York: Springer.

    Google Scholar 

  • Kato, T.: 1996, Cognitive User Interface to Cyber Space Database: - Human Media Technology for Global Information Infrastructure -. In: Proceedings of the International Symposium on Cooperative Database Systems for Advanced Applications, Kyoto, Japan, 184–190.

  • Kirsh, R.: 1971, Computer Determination of the Constituent Structure of Biomedical Images. Computers and Biomedical Research 4(3), 315–328.

    Article  PubMed  Google Scholar 

  • Kobayashi, S.: 1998, Colorist: a practical handbook for personal and professional use. Tokyo: Kodansha International.

    Google Scholar 

  • Kobsa, A. and Wahlster, W.: 1989, User Models in Dialog Systems. Berlin: Springer-Verlag.

    Google Scholar 

  • Lee, S. and Harada, A.: 1998, A Design Approach by Objective and Subjective Evaluation of KANSEI Information. In: Proceedings of the International Workshop on Robot and Human Communication, RoMan’98, Hakamatsu, Japan, 327–332.

  • Levenson, R.: 1992, Autonomic Nervous System Differences Among Emotions. Psychological Science 3(1), 23–27.

    Google Scholar 

  • Leventhal, H. and Scherer, K.: 1987, The Relationship of Emotion to Cognition: A Functional Approach to a Semantic Controversy. Cognition and Emotion 1(1), 3–28.

    Google Scholar 

  • Lisetti, C.: 1997, Motives for intelligent agents: Computational scripts for emotion concepts. In: G. Grahne (ed.), Proceedings of the Sixth Scandinavian Conference on Artificial Intelligence (SCAI’97), 59–70.

  • Lisetti, C.: 1998, Invited Book Review: Affective Computing (MIT Press). Pattern Analysis and Applications 1, 71–73.

    Article  Google Scholar 

  • Lisetti, C.: 1999, A User-Model of Cognition-Emotion. In: Proceedings of the UserModelling (UM’99) First Workshop on Attitude, Personality, and Emotions in User-Adapted Interaction, 25–35.

  • Lisetti, C., Douglas, M. and LeRouge, C.: 2001, Intelligent Affective Interfaces: A User-Modeling Approach for Telemedicine. In: Proceedings of the First International Conference on Universal Access in Human-Computer Interaction (UAHCI’01).

  • Lisetti, C. and Rumelhart, D.: 1998, Facial expression recognition using a neural network. In: D. Cook (ed.), Proceedings of the 1998 International Florida Artificial Intelligence Research Symposium Conference (FLAIRS’98), 328–332.

  • Lisetti, C. and Schiano, D.: 2000, Automatic Facial Expression Interpretation: Where Human Computer Interaction, Artificial Intelligence, and Cognitive Science Intersect. Pragmatics and Cognition (Special Issue on Facial Information Processing: A Multidisciplinary Perspective) 8(1), 185–235.

    Google Scholar 

  • Lisetti, C. L., Rumelhart, D. E. and Holler, M.: 1998, An Environment to Acknowledge the Interface between Affect and Cognition. In: Proceedings of the AAAI Spring Symposium Series on Intelligent Environments. (also AAAI Technical Report SS-98-02).

  • Lutz, C. and White, G.: 1986, The anthropology of emotions’. Annual Review of Anthropology 15, 405–436.

    Article  Google Scholar 

  • Maes, P.: 1990, Designing Autonomous Agents. Cambridge, MA: MIT Press.

    Google Scholar 

  • Maetama, S., Yuta, S. and Harada, A.: 1998, Mobile Robot in the Remote Museum for Modeling the Evaluation Structure of KANSEI. In: Proceedings of the International Workshop on Robot and Human Communication, RoMan’98, Hakamatsu, Japan, 315–320.

  • Mandler, G.: 1975, Mind and Emotion. New York: Wiley.

    Google Scholar 

  • Merelli, M., Mussio, P. Padula M.: 1985, An approach to the Definition, Description and Extraction of structures in Binary Digital Images. Computer Vision Graphics and Image Processing 31, 19–49.

    Google Scholar 

  • Miyake, N.: 1997, Making Internal Process External for Constructive Collaboration. In: J. Marsh, C. Nehaniv, and B. Gorayska (eds.), Cognitive Technology. IEEE Computer Society, 119–123.

  • Mojsilovic, Z., Kovacevic, J., Hu, J., Safranek, R. J. and Ganathy, S. K.: 2000, Matching and Retrieval Based on Vocabulary and Grammar of Color Patterns. IEEE Transaction on Image Processing 9(1), 38 55.

    Article  Google Scholar 

  • Nagao, K. and Takeuchi, A.: 1994, Speech Dialogue with Facial Displays: Multimodal Human-Computer Conversation. In: 32nd Annual Meeting of the Association of Computational Liguistics (ACL-94).

  • Nass, C., Kim, E. and Lee, E.: 1998, When my face is the interface: Experimental comparison of interacting with one’s own face or someone else’s face. In: Proceedings of the International Conference on Human Factors in Computing Systems (CHI’98), 148–154.

  • Norman, D.: 1998, The Invisible Computer:Why Good Products Fail, Why the Personal Computer Is So Complex, and How To Do It Right. Cambridge, MA: MIT Press.

    Google Scholar 

  • Ortony, A., Clore, G. and Collins, A.: 1988, The Cognitive Structure of Emotions. Cambridge University Press.

  • Padgett, C. and Cottrell, G.: 1998, A simple neural network models categorical perception of facial expressions. In: Proceedings of the Twenty First Annual Cognitive Conference of the Cognitive Science Society, 806–811.

  • Picard, R. W.: 1997, Affective Computing. Cambridge, MA: M.I.T. Press.

    Google Scholar 

  • Rumelhart, D. E., Hinton, G. E. and Williams, R.: 1986, Learning Internal Representations by Error Propagation. In: D. Rumelhart and J. McClelland (eds.): Parallel Distributed Processing: Explorations in the Microstructures of Cognition Volume 1: Foundations.

  • Rybak, I. A., Gusakova, V. I., Golovan, A. V., Podladchikova, L. N. and Shevtsova, N. A.: 1998, A Model of Attention-Guided Visual Perception and Recognition. Vision Research 38, 2387–2400.

    Article  PubMed  Google Scholar 

  • Saracevic, T., Spink, A. and Wu, M.: 1997, Users and Intermediaries in Information Retrieval: What Are They Talking About. In: A. Jameson, C. Paris, and C. Tasso (eds.): User Modeling: Proceedings of the Sixth International Conference, UM’97, Sardinia, 43–54.

  • Sato, T.: 1996, Expressive Robot with Touching Behavior. Private communication.

  • Schachter, S.: 1964, The Interaction of Cognitive and Physiological Determinants of Emotional States. In: L. Berkowitz (ed.): Advances in Experimental Social Psychology Vol. 1, 529–564.

  • Selker, T.: 1994, COACH: A teaching agent that learns. Communications of the ACM 37(7), 92–99.

    Article  Google Scholar 

  • Shibata, T. and Kato, T.: 1999, Kansei Image Retrieval Systems for Street Landscape. Discrimination and Graphical Parameters based on correlation of Two Images. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 247–242.

    Google Scholar 

  • Sloman, A.: 1990, Motives, Mechanisms, and Emotions. In: M. Boden (ed.), The Philosophy of Artificial Intelligence.

  • Sproull, L., Subramani, M., Kiesler, S., Walker, J. and Waters, K.: 1996, When the Interface Is a Face. Human-Computer Interaction 11, 97–124.

    Article  Google Scholar 

  • Tanguy, L.: 1997, Computer-Aided Language Processing. In: J. Marsh, C. Nehaniv, and B. Gorayska (eds.), Cognitive Technology, 136–145.

  • Weizenbaum, J.: 1966, ELIZA - A Computer Program for the Study of Natural Language Communication between Man and Machine. Communications of the Association for Computing Machinery 9, 36–45.

    Google Scholar 

  • Wierzbicka, A.: 1992a, Defining Emotion Concepts. Cognitive Scence 16, 539–581.

    Article  Google Scholar 

  • Wierzbicka, A.: 1992b, Talking about Emotions: Semantics, Culture, and Cognition. Cognition and Emotion 6, 285–319.

    Google Scholar 

  • Winograd, T.: 1998, A human-centered interaction architecture. In: Working Paper of The People, Computer, and Design Project.

  • Yoshida, K., Kato, T. and Yanoru, T.: 1999, A Study of Database Systems with Kansei Information. In: Proceedings of the IEEE International Conference on Systems, Men and Cybernetics, Tokyo, Japan, Vol. 6, 253–256.

    Google Scholar 

  • Zajonc, B. and Markus, H.: 1984, Affect and Cognition: The Hard Interface. In: J. Izard, C. Kagan and R. Zajonc (eds.), Emotion, Cognition and Behavior, 73–102.

  • Zajonc, R.: 1980, Feeling and Thinking: Preferences Need No Inferences. American Psychologist 35, 151–175.

    Article  Google Scholar 

  • Zakia, R.: 1997, Perception and Imaging. Focal Press.

Download references

Author information

Authors and Affiliations

  1. Computer Software Department, University of Aizu, Aizu Wakamatsu, Fukushima, Japan

    Nadia Bianchi-Berthouze

  2. School of Computer Science, University of Central Florida, Orlando, FL, USA

    Christine L. Lisetti

Authors
  1. Nadia Bianchi-Berthouze
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christine L. Lisetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bianchi-Berthouze, N., Lisetti, C.L. Modeling Multimodal Expression of User’s Affective Subjective Experience. User Model User-Adap Inter 12, 49–84 (2002). https://doi.org/10.1023/A:1013365332180

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1013365332180

Search

Navigation