Skip to main content
Erschienen in: Neural Computing and Applications 23/2020

22.07.2020 | S.I. : Emerging applications of Deep Learning and Spiking ANN

Affective analysis of patients in homecare video-assisted telemedicine using computational intelligence

verfasst von: A. Kallipolitis, M. Galliakis, A. Menychtas, I. Maglogiannis

Erschienen in: Neural Computing and Applications | Ausgabe 23/2020

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The affective/emotional status of patients is strongly connected to the healing process and their health. Therefore, being aware of the psychological peaks and troughs of a patient provides the advantage of timely intervention by specialists or closely related kinsfolk. In this context, this paper presents the design and implementation of an emotion analysis module integrated in an existing telemedicine platform. Two different methodologies are utilized and discussed. The first scheme exploits the fast and consistent properties of the speeded-up robust features algorithm in order to identify the existence of seven different sentiments in human faces. The second is based on convolutional neural networks. The whole functionality is provided as a Web service for the healthcare platform during regular video teleconference sessions between authorized medical personnel and patients. The paper discusses the technical details of the implementation and the incorporation of the proposed scheme and provides the initial results of its accuracy and operation in practice.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Ayadi MM, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit 44:572–587CrossRef Ayadi MM, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit 44:572–587CrossRef
2.
Zurück zum Zitat Bay H, Tuytelaars T, Gool VG (2008) Speeded up robust features. Comput Vis Image Underst 110(3):346–359CrossRef Bay H, Tuytelaars T, Gool VG (2008) Speeded up robust features. Comput Vis Image Underst 110(3):346–359CrossRef
3.
Zurück zum Zitat Bouchiha R, Besbes K (2013) Automatic remote-sensing image registration using SURF. Int J Comput Theory Eng 5(1):88–92CrossRef Bouchiha R, Besbes K (2013) Automatic remote-sensing image registration using SURF. Int J Comput Theory Eng 5(1):88–92CrossRef
4.
Zurück zum Zitat Buades A, Coll B, Morel JM (2005) A non-local algorithm for image denoising. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), vol 2, pp 60–65 Buades A, Coll B, Morel JM (2005) A non-local algorithm for image denoising. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), vol 2, pp 60–65
5.
Zurück zum Zitat Chen M, Zhang Y, Li Y, Hassan MM, Alamri A (2015) AIWAC: affective interaction through wearable computing and cloud technology. IEEE Wirel Commun 22(1):20–27CrossRef Chen M, Zhang Y, Li Y, Hassan MM, Alamri A (2015) AIWAC: affective interaction through wearable computing and cloud technology. IEEE Wirel Commun 22(1):20–27CrossRef
6.
Zurück zum Zitat Ciresan DC, Meier U, Masci J, Gambardella LM, Schmidhuber J (2011) Flexible, high performance convolutional neural networks for image classification. In Twenty-second international joint conference on artificial intelligence Ciresan DC, Meier U, Masci J, Gambardella LM, Schmidhuber J (2011) Flexible, high performance convolutional neural networks for image classification. In Twenty-second international joint conference on artificial intelligence
7.
Zurück zum Zitat Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a matlab-like environment for machine learning. In: NIPS 2011 Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a matlab-like environment for machine learning. In: NIPS 2011
8.
Zurück zum Zitat Ekman P, Davidson RJ (eds) (1994) Series in affective science. The nature of emotion: fundamental questions. Oxford University Press, Oxford Ekman P, Davidson RJ (eds) (1994) Series in affective science. The nature of emotion: fundamental questions. Oxford University Press, Oxford
9.
Zurück zum Zitat Filntisis PP, Efthymiou N, Koutras P, Potamianos G, Maragos P (2019) Fusing body posture with facial expressions for joint recognition of affect in child–robot interaction. IEEE Robot Autom Lett 4:4011–4018CrossRef Filntisis PP, Efthymiou N, Koutras P, Potamianos G, Maragos P (2019) Fusing body posture with facial expressions for joint recognition of affect in child–robot interaction. IEEE Robot Autom Lett 4:4011–4018CrossRef
10.
Zurück zum Zitat Forstall S, Chaudhri I, Chaudhri IA (2006) Webview applications. U.S. Patent Application 11/145,560 Forstall S, Chaudhri I, Chaudhri IA (2006) Webview applications. U.S. Patent Application 11/145,560
11.
Zurück zum Zitat Fridenson-Hayo S, Berggren S, Lassalle A et al (2017) 'Emotiplay': a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. Eur Child Adolesc Psychiatry 26:979–992CrossRef Fridenson-Hayo S, Berggren S, Lassalle A et al (2017) 'Emotiplay': a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. Eur Child Adolesc Psychiatry 26:979–992CrossRef
12.
Zurück zum Zitat Gadaf R, Besar B (2017) The effects of emotional intelligence on employees performance. Int J Bus Glob 18(4):467–479CrossRef Gadaf R, Besar B (2017) The effects of emotional intelligence on employees performance. Int J Bus Glob 18(4):467–479CrossRef
14.
Zurück zum Zitat Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hammer B, Zhou Y (2013) Challenges in representation learning: a report on three machine learning contents. In: International conference on neural information processing, pp 117–124 Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hammer B, Zhou Y (2013) Challenges in representation learning: a report on three machine learning contents. In: International conference on neural information processing, pp 117–124
15.
Zurück zum Zitat Gudi A, Tasli HE, Uyl TM, Maroulis A (2015) Deep learning based FACS Action Unit occurrence and intensity estimation. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 06, pp 1–5 Gudi A, Tasli HE, Uyl TM, Maroulis A (2015) Deep learning based FACS Action Unit occurrence and intensity estimation. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 06, pp 1–5
16.
Zurück zum Zitat Gulli A, Sujit P (2017) Deep learning with Keras. Packt Publishing Ltd, Birmingham Gulli A, Sujit P (2017) Deep learning with Keras. Packt Publishing Ltd, Birmingham
17.
Zurück zum Zitat Holmgård C, Yannakakis G, Karstoft KI, Andersen H (2013) Stress detection for PTSD via the StartleMart Game. In: Proceedings—2013 humane association conference on affective computing and intelligent interaction, ACII 2013, pp 523–528 Holmgård C, Yannakakis G, Karstoft KI, Andersen H (2013) Stress detection for PTSD via the StartleMart Game. In: Proceedings—2013 humane association conference on affective computing and intelligent interaction, ACII 2013, pp 523–528
18.
Zurück zum Zitat Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:​1704.​04861
19.
Zurück zum Zitat Izquierdo-Reyes J, Ramirez-Mendoza RA, Bustamante-Bello MR, Navarro-Tuch S, Avila-Vázquez R (2018) Advanced driver monitoring for assistance system (ADMAS). Int J Interact Des Manuf 12:187–197CrossRef Izquierdo-Reyes J, Ramirez-Mendoza RA, Bustamante-Bello MR, Navarro-Tuch S, Avila-Vázquez R (2018) Advanced driver monitoring for assistance system (ADMAS). Int J Interact Des Manuf 12:187–197CrossRef
20.
Zurück zum Zitat Juan L, Gwun O (2010) SURF applied in panorama image stitching. In: 2010 2nd international conference on image processing theory, tools and applications, pp 495–499 Juan L, Gwun O (2010) SURF applied in panorama image stitching. In: 2010 2nd international conference on image processing theory, tools and applications, pp 495–499
21.
Zurück zum Zitat Kallipolitis A, Galliakis M, Menychtas A, Maglogiannis I (2019) Emotion analysis in hospital bedside infotainment platforms using speeded up robust features. In: 15th IFIP international conference on artificial intelligence applications and innovations (AIAI), pp 127–138 Kallipolitis A, Galliakis M, Menychtas A, Maglogiannis I (2019) Emotion analysis in hospital bedside infotainment platforms using speeded up robust features. In: 15th IFIP international conference on artificial intelligence applications and innovations (AIAI), pp 127–138
22.
Zurück zum Zitat Katarya R, Verma O (2016) Recent developments in affective recommender systems. Phys A Stat Mech Appl 461:182–190CrossRef Katarya R, Verma O (2016) Recent developments in affective recommender systems. Phys A Stat Mech Appl 461:182–190CrossRef
24.
Zurück zum Zitat Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187MathSciNetCrossRef Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187MathSciNetCrossRef
25.
Zurück zum Zitat Krakovsky M (2018) Artificial (emotional) intelligence. Commun ACM 61:18–19CrossRef Krakovsky M (2018) Artificial (emotional) intelligence. Commun ACM 61:18–19CrossRef
26.
Zurück zum Zitat Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: NIPS Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: NIPS
29.
Zurück zum Zitat Lee CM, Yildirim S, Bulut M, Kazemzadeh A, Busso C, Deng Z, Lee S, Narayanan SS (2004) Emotion recognition based on phoneme classes. To appear in Proc. ICSLP’04 Lee CM, Yildirim S, Bulut M, Kazemzadeh A, Busso C, Deng Z, Lee S, Narayanan SS (2004) Emotion recognition based on phoneme classes. To appear in Proc. ICSLP’04
30.
Zurück zum Zitat Liu Z, Wu M, Cao W, Mao J, Xu J, Tan G (2018) Speech emotion recognition based on feature selection and extreme learning machine decision tree. Neurocomputing 273:271–280CrossRef Liu Z, Wu M, Cao W, Mao J, Xu J, Tan G (2018) Speech emotion recognition based on feature selection and extreme learning machine decision tree. Neurocomputing 273:271–280CrossRef
31.
Zurück zum Zitat Lopez-de-la-Calleja M, Nagai T, Attamimi M, Nakano-Miyatake M, Perez-Meana H (2013) Object detection using SURF and superpixels. J Softw Eng Appl 06:511–518CrossRef Lopez-de-la-Calleja M, Nagai T, Attamimi M, Nakano-Miyatake M, Perez-Meana H (2013) Object detection using SURF and superpixels. J Softw Eng Appl 06:511–518CrossRef
32.
Zurück zum Zitat Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRef Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRef
33.
Zurück zum Zitat Lucey P, Cohn JF, Kanade T, Saragih JM, Ambadar Z, Matthews IA (2010) The Extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition—workshops, pp 94-101 Lucey P, Cohn JF, Kanade T, Saragih JM, Ambadar Z, Matthews IA (2010) The Extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition—workshops, pp 94-101
34.
Zurück zum Zitat Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings third IEEE international conference on automatic face and gesture recognition, pp 200–205 Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings third IEEE international conference on automatic face and gesture recognition, pp 200–205
35.
Zurück zum Zitat Martín A, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th symposium on operating systems design and implementation, vol 16, pp 265–283 Martín A, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th symposium on operating systems design and implementation, vol 16, pp 265–283
36.
Zurück zum Zitat Mavridou I, McGhee J, Hamedi M, Fatoorechi M, Cleal A, Balaguer-Ballester E, Seiss E, Cox G, Nduka C (2017) FACETEQ: a novel platform for measuring emotion in VR. In: 2017 IEEE virtual reality (VR), pp 441–442 Mavridou I, McGhee J, Hamedi M, Fatoorechi M, Cleal A, Balaguer-Ballester E, Seiss E, Cox G, Nduka C (2017) FACETEQ: a novel platform for measuring emotion in VR. In: 2017 IEEE virtual reality (VR), pp 441–442
38.
Zurück zum Zitat Noroozi F, Corneanu C A, Kaminska D, Sapinski T, Escalera S, Anbarjafari G (2018) Survey on emotional body gesture recognition. arXiv:1801.07481 Noroozi F, Corneanu C A, Kaminska D, Sapinski T, Escalera S, Anbarjafari G (2018) Survey on emotional body gesture recognition. arXiv:​1801.​07481
39.
Zurück zum Zitat Panagopoulos C, Menychtas A, Fouskas G, Plagianakos V, Maglogiannis I, Delimpasis K, Galliakis M, Petropoulos D, Gkartzios C, Koumpoulis C (2019) A smart infotainment system equipped with emotional intelligence. Stud Health Technol inform 262:214–217 Panagopoulos C, Menychtas A, Fouskas G, Plagianakos V, Maglogiannis I, Delimpasis K, Galliakis M, Petropoulos D, Gkartzios C, Koumpoulis C (2019) A smart infotainment system equipped with emotional intelligence. Stud Health Technol inform 262:214–217
41.
Zurück zum Zitat Pizer SM, Amburn EP, Austin JD et al (1987) Adaptive histogram equalization and its variations. Comput Vis Graph Image Process 39:355–368CrossRef Pizer SM, Amburn EP, Austin JD et al (1987) Adaptive histogram equalization and its variations. Comput Vis Graph Image Process 39:355–368CrossRef
42.
Zurück zum Zitat Rami AR, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F et al (2016) Theano: a python framework for fast computation of mathematical expressions. arXiv:1605.02688 Rami AR, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bastien F et al (2016) Theano: a python framework for fast computation of mathematical expressions. arXiv:​1605.​02688
43.
Zurück zum Zitat Rao Q, Qu X, Mao Q, Zhan Y (2015) Multi-pose facial expression recognition based on SURF boosting. In: 2015 international conference on affective computing and intelligent interaction (ACII), pp 630–635 Rao Q, Qu X, Mao Q, Zhan Y (2015) Multi-pose facial expression recognition based on SURF boosting. In: 2015 international conference on affective computing and intelligent interaction (ACII), pp 630–635
44.
Zurück zum Zitat Rousseeuw P (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65CrossRef Rousseeuw P (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65CrossRef
48.
Zurück zum Zitat Tivatansakul S, Ohkura M, Puangpontip S, Achalakul T (2014) Emotional healthcare system: emotion detection by facial expressions using Japanese database. In: 2014 6th computer science and electronic engineering conference (CEEC), pp 41–46 Tivatansakul S, Ohkura M, Puangpontip S, Achalakul T (2014) Emotional healthcare system: emotion detection by facial expressions using Japanese database. In: 2014 6th computer science and electronic engineering conference (CEEC), pp 41–46
49.
Zurück zum Zitat Wang S, Phillips P, Dong Z, Zhang Y (2018) Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm. Neurocomputing 272:668–676CrossRef Wang S, Phillips P, Dong Z, Zhang Y (2018) Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm. Neurocomputing 272:668–676CrossRef
52.
Zurück zum Zitat Yamauchi T, Xiao K (2018) Reading emotion from mouse cursor motions: affective computing approach. Cogn Sci 42:771–819CrossRef Yamauchi T, Xiao K (2018) Reading emotion from mouse cursor motions: affective computing approach. Cogn Sci 42:771–819CrossRef
53.
Zurück zum Zitat Yannakakis GN (2018) Enhancing health care via affective computing. Malta J Health Sci 5:38 Yannakakis GN (2018) Enhancing health care via affective computing. Malta J Health Sci 5:38
Metadaten
Titel
Affective analysis of patients in homecare video-assisted telemedicine using computational intelligence
verfasst von
A. Kallipolitis
M. Galliakis
A. Menychtas
I. Maglogiannis
Publikationsdatum
22.07.2020
Verlag
Springer London
Erschienen in
Neural Computing and Applications / Ausgabe 23/2020
Print ISSN: 0941-0643
Elektronische ISSN: 1433-3058
DOI
https://doi.org/10.1007/s00521-020-05203-z

Weitere Artikel der Ausgabe 23/2020

Neural Computing and Applications 23/2020 Zur Ausgabe

S.I. : Emerging applications of Deep Learning and Spiking ANN

A CNN–LSTM model for gold price time-series forecasting

S.I. : Emerging applications of Deep Learning and Spiking ANN

Anomaly detection via blockchained deep learning smart contracts in industry 4.0

S.I. : Emerging applications of Deep Learning and Spiking ANN

A semi-self-taught network intrusion detection system