Abstract
Knowledge about the users emotional state is important to achieve human like, natural Human Computer Interaction (HCI) in modern technical systems. Humans rely on implicit signals like body gestures and posture, vocal changes (e.g. pitch) and mimic expressions when communicating. We investigate the relation between them and human emotion, specifically when completing easy or difficult tasks. Additionally we include physiological data which also differ in changes of cognitive load. We focus on discriminating between mental overload and mental underload, which can e.g. be useful in an e-tutorial system. Mental underload is a new term used to describe the state a person is in when completing a dull or boring task. It will be shown how to select suited features, build uni modal classifiers which then are combined to a multimodal mental load estimation by the use of Markov Fusion Networks (MFN) and Kalman Filter Fusion (KFF).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)
Glodek, M., Reuter, S., Schels, M., Dietmayer, K., Schwenker, F.: Kalman filter based classifier fusion for affective state recognition. In: Zhou, Z.H., Roli, F., Kittler, J. (eds.) MCS 2013. LNCS, vol. 7872, pp. 85–94. Springer, Heidelberg (2013). doi:10.1007/978-3-642-38067-9_8
Glodek, M., Schels, M., Schwenker, F., Palm, G.: Combination of sequential class distributions from multiple channels using Markov fusion networks. J. Multimodal User Interfaces 8(3), 257–272 (2014)
Held, D.: Bimodale Erkennung affektiver Zustnde durch Ensemble Methoden anhand von Audio- und Biosignalen. Master’s thesis, Ulm University (2016)
Hihn, H., Meudt, S., Schwenker, F.: Inferring mental overload based on postural behavior and gestures. In: Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems. ACM (2016)
Kalman, R.E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)
Kipp, M., Martin, J.-C.: Gesture and emotion: can basic gestural form features discriminate emotions? In: 3rd International Conference on Affective Computing and Intelligent Interaction Workshops, pp. 1–8. IEEE (2009)
Meudt, S., Zharkov, D., Kächele, M., Schwenker, F.: Multi classifier systems and forward backward feature selection algorithms to classify emotional coloured speech. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 551–556. ACM (2013)
Picard, R.W.: Affective Computing, vol. 252. MIT Press, Cambridge (1997)
Plesa-Skwerer, D., Faja, S., Schofield, C., Verbalis, A., Tager-Flusberg, H., Dykens, E.M.: Perceiving facial and vocal expressions of emotion in individuals with Williams syndrome. Am. J. Ment. Retard. 111(1), 15–26 (2006)
Russell, J.A., Bachorowski, J.-A., Fernández-Dols, J.-M.: Facial and vocal expressions of emotion. Annu. Rev. Psychol. 54(1), 329–349 (2003)
Schels, M., Glodek, M., Meudt, S., Scherer, S., Schmidt, M., Layher, G., Tschechne, S., Brosch, T., Hrabal, D., Walter, S., et al. Multi-modal classifier-fusion for the recognition of emotions. In: Coverbal Synchrony in Human-Machine Interaction (2013)
Schüssel, F., Honold, F., Bubalo, N., Huckauf, A., Traue, H., Hazer-Rau, D.: In-depth analysis of multimodal interaction: an explorative paradigm. In: Kurosu, M. (ed.) HCI 2016. LNCS, vol. 9732, pp. 233–240. Springer, Cham (2016). doi:10.1007/978-3-319-39516-6_22
Shan, C., Gong, S., McOwan, P.W.: Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing, 2005. ICIP 2005, vol. 2, p. II-370. IEEE (2005)
Vogt, T., André, E., Bee, N.: EmoVoice — a framework for online recognition of emotions from voice. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS, vol. 5078, pp. 188–199. Springer, Heidelberg (2008). doi:10.1007/978-3-540-69369-7_21
Acknowledgments
The authors of this paper are partially funded by the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Kindsvater, D., Meudt, S., Schwenker, F. (2017). Fusion Architectures for Multimodal Cognitive Load Recognition. In: Schwenker, F., Scherer, S. (eds) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. MPRSS 2016. Lecture Notes in Computer Science(), vol 10183. Springer, Cham. https://doi.org/10.1007/978-3-319-59259-6_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-59259-6_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59258-9
Online ISBN: 978-3-319-59259-6
eBook Packages: Computer ScienceComputer Science (R0)