Mi Li
Skip Abstract Section
Abstract
At present, there have been many studies on the methods of using the deep learning regression model to assess depression level based on behavioral signals (facial expression, speech, and language); however, the research on the assessment method of anxiety level using deep learning is absent. In this article, pupil-wave, a physiological signal collected by Human Computer Interaction (HCI) that can directly represent the emotional state, is developed to assess the level of depression and anxiety for the first time. In order to distinguish between different depression and anxiety levels, we use the HCI method to induce the participants’ emotional experience through three virtual reality (VR) emotional scenes of joyful, sad, and calm, and construct two differential pupil-waves of joyful and sad with the calm pupil-wave as the baseline. Correspondingly, a dual-channel fusion depression and anxiety level assessment model is constructed using the improved multi-scale convolution module and our proposed width-channel attention module for one-dimensional signal processing. The test results show that the MAE/RMSE of the depression and anxiety level assessment method proposed in this article is 3.05/4.11 and 2.49/1.85, respectively, which has better assessment performance than other related research methods. This study provides an automatic assessment technique based on human computer interaction and virtual reality for mental health physical examination.
- [1] . 2015. The economic burden of adults with major depressive disorder in the United States (2005 and 2010). Journal of Clinical Psychiatry 76, 2 (2015), 155–162.Google Scholar
Cross Ref
- [2] 2015. Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: A systematic analysis for the global burden of disease study 2013. The Lancet 386, 9995 (2015), 743–800.Google ScholarCross Ref
- [3] . 2008. The Global Burden of Disease: 2004 Update. Geneva, Switzerland: World Health Organization.Google Scholar
- [4] . 2017. Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiology and Psychiatric Sciences 26, 1 (2017), 22–36.Google ScholarCross Ref
- [5] . 2015. Generalized anxiety disorder. In Encyclopedia of Psychopharmacology. Berlin, Germany, Springer, 699–702.Google Scholar
- [6] . 1996. Comparison of beck depression inventories-I and-Ⅱ in psychiatric outpatients. Journal of Personality Assessment 67, 3 (1996), 588–597.Google ScholarCross Ref
- [7] . 2009. The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders 114, 1–3 (2009), 163–173.Google ScholarCross Ref
- [8] . 2002. The PHQ-9: A new depression diagnostic and severity measure. Psychiatric Annals 32, 9 (2002), 509–521.Google ScholarCross Ref
- [9] . 2006. A brief measure for assessing generalized anxiety disorder. Archives of Internal Medicine 166, 10 (2006), 1092–1097.Google ScholarCross Ref
- [10] . 2008. The Psychology of Human Computer Interaction. CRC Press.Google Scholar
- [11] . 2010. Defining virtual reality: Dimensions determining telepresence. Journal of Communication 42, 4 (2010).Google Scholar
- [12] . 2004. Fear conditioning in virtual reality contexts: A new tool for the study of anxiety. Biological Psychiatry 55, 11 (2004), 1056–1060.Google ScholarCross Ref
- [13] . 2016. The differential effects of agency on fear induction using a horror-themed video game. Computers in Human Behavior 56, (2016), 142–146.Google ScholarDigital Library
- [14] . 2016. Decision tree based depression classification from audio video and language information. In Proceedings of the ACM Multimedia Conference Amsterdam the Netherlands. 89–96.Google Scholar
- [15] . 2018. Automated depression diagnosis based on deep networks to encode facial appearance and dynamics. IEEE Transactions on Affective Computing 9, 4 (2018), 578–584.Google ScholarDigital Library
- [16] . 2018. Automated depression analysis using convolutional neural networks from speech. Journal of Biomedical Informatics 83, (2018), 103–111.Google ScholarCross Ref
- [17] . 2018. Measuring depression symptom severity from spoken language and 3D facial expressions. arXiv preprint arXiv:1811.08592 (2018).Google Scholar
- [18] . 2018. Detecting depression with audio/text sequence modeling of interviews. In Proceedings of the Interspeech 2018. 1716–1720.Google Scholar
- [19] . 2016. Detecting depression using vocal, facial and semantic communication cues. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. 11–18.Google Scholar
- [20] . 2018. Integrating deep and shallow models for multi-modal depression analysis-hybrid architectures. IEEE Transactions on Affective Computing 12, 1 (2018), 239–253.Google ScholarCross Ref
- [21] . 2020. Automatic assessment of depression from speech via a hierarchical attention transfer network and attention autoencoders. IEEE Journal of Selected Topics in Signal Processing 14, 2 (2020), 423–434.Google ScholarCross Ref
- [22] . 2019. Combining global and local convolutional 3D networks for detecting depression from facial expressions. In Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition. 1–8.Google Scholar
- [23] . Combining Global and Local Convolutional 3D Networks for Detecting Depression from Facial Expressions.Google Scholar
- [24] . 2018. Measuring emotion: Development and application of an instrument to measure emotional responses to products. In Funology. Berlin, Germany, Springer, 111–123.Google Scholar
- [25] . 2018. The influence of emotional stimuli on the oculomotor system: A review of the literature. Cognitive Affective & Behavioral Neuroscience 18, 2 (2018), 411–415.Google ScholarCross Ref
- [26] . 2008. The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45, 4 (2008), 602–607.Google ScholarCross Ref
- [27] . 2008. Presence and emotions in virtual environments: The influence of stereoscopy. Cyberpsychology & Behavior 11, 1 (2008), 1–8.Google ScholarCross Ref
- [28] . 2006. Changing induced moods via virtual reality. In Proceedings of the 1st International Conference on Persuasive Technology for Human Well-being. 7–15.Google Scholar
- [29] . 2011. An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders. Biomedical Signal Processing and Control 6, 3 (2011), 261–268.Google ScholarCross Ref
- [30] . 2014. Context-based signal descriptors of heart-rate variability for anxiety assessment. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’14). 3631–3635.Google Scholar
- [31] . 2016. Predicting generalized anxiety disorder among women using random forest approach. In Proceedings of the 2016 3rd International Conference on Computer and Information Sciences (ICCOINS’16). 37–42.Google Scholar
- [32] . 2017. Separating generalized anxiety disorder from major depression using clinical. Hormonal, and structural MRI data: A multimodal machine learning study. Brain and Behavior 12, 3 (2017), e00633.Google Scholar
- [33] . 2000. The role of rumination in depressive disorders and mixed anxiety/depressive symptoms. Journal of Abnormal Psychology 109, 3 (2000), 504–511.Google ScholarCross Ref
- [34] . 2007. Rumination and attention in major depression. Behaviour Research and Therapy 45, 11 (2007), 2664–2678.Google ScholarCross Ref
- [35] . 2016. Distributed and parallel time series feature extraction for industrial big data applications. arXiv preprint arXiv:1610.07717 (2016).Google Scholar
- [36] . 2019. Automatic assessment of depression based on visual cues: A systematic review. IEEE Transactions on Affective Computing 10, 4 (2019), 445–470.Google ScholarCross Ref
- [37] . 2013. AVEC 2013: The continuous audio/visual emotion and depression recognition challenge. In Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge (AVEC'13). 3–10.Google Scholar
- [38] . 2014. AVEC 2014: 3D dimensional affect and depression recognition challenge. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (AVEC'14). 3–10.Google Scholar
- [39] . 2016. AVEC 2016: Depression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge (AVEC'16). 3–10.Google Scholar
- [40] . 2017. AVEC 2017: Real-life depression, and affect recognition workshop and challenge. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge (AVEC'17). 3–9.Google Scholar
- [41] , Yona Falinie Binti A. Gaus, and . 2018. Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Transactions on Cognitive and Developmental Systems 10, 3 (2018), 668–680.Google ScholarCross Ref
- [42] . 2021. Video-based depression level analysis by encoding deep spatiotemporal features. IEEE Transactions on Affective Computing 12, 1 (2018), 262–268.Google ScholarCross Ref
- [43] . 2020. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing 11, 3 (2020), 542–552.Google ScholarCross Ref
- [44] . 2020. AudVowelConsNet: A phoneme-level based deep CNN architecture for clinical depression diagnosis. Machine Learning with Applications 2, (2020), 100005.Google ScholarCross Ref
- [45] . 2018. Automated depression analysis using convolutional neural networks from speech. Journal of Biomedical Informatics 83, (2018), 103–111.Google ScholarCross Ref
- [46] . 2016. Emotion, working memory, and cognitive control in patients with first-onset and previously untreated minor depressive disorders. Journal of International Medical Research 44, 3 (2016), 529–541.Google ScholarCross Ref
- [47] . 2016. Emotional experience and the mood-congruent working memory effect in first-onset and untreated depressive disorder patients. Journal of Psychiatry 19, 4 (2016), 1000379.Google Scholar
- [48] . 2015. A review of depression and suicide risk assessment using speech analysis. Speech Communication 71, (2015), 10–49.Google ScholarDigital Library
- [49] . 2015. Automated audiovisual depression analysis. Current Opinion in Psychology 4, (2015), 75–79.Google ScholarCross Ref
- [50] . 2019. Automatic assessment of depression based on visual cues: A systematic review. IEEE Transactions on Affective Computing 10, 4 (2019), 445–470.Google ScholarCross Ref
- [51] . 2018. Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE Journal of Biomedical and Health Informatics 22, 2 (2018), 525–536.Google ScholarCross Ref
- [52] . 2013. Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging. JAMA Psychiatry 70, 1 (2013), 87–97.Google ScholarCross Ref
- [53] . 2021. 1D convolutional neural networks and applications: A survey. Mechanical Systems and Signal Processing 151 (2021), 107398.Google ScholarCross Ref
- [54] . 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems-Volume 1 (NIPS'12). 1097–1105.Google Scholar
- [55] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- [56] . 2015. Going deeper with convolutions. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 1–9.Google Scholar
- [57] . 2021. Res2Net: A new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 2 (2021), 652–662.Google ScholarDigital Library
- [58] . 2020. HS-ResNet: Hierarchical-split block on convolutional neural network. arXiv preprint arXiv:2010.07621 (2020).Google Scholar
- [59] . 2016. End-to-end attention-based large vocabulary speech recognition. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’16). 4945–4949.Google Scholar
- [60] . 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). 6000–6010.Google Scholar
- [61] . 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2020), 2011–2023.Google ScholarDigital Library
- [62] . 2019. ECA-Net: Efficient channel attention for deep convolutional neural networks. arXiv preprint arXiv:1910.03151 (2019).Google Scholar
- [63] . 2018. CBAM: Convolutional block attention module. In Proceedings of the 2018 European Conference on Computer Vision. 3–19.Google Scholar
- [64] . 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7794–7803.Google Scholar
- [65] . 2019. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3146–3154.Google Scholar
- [66] . 2020. Pyramidal convolution: Rethinking convolutional neural networks for visual recognition. arXiv preprint arXiv:2006.11538 (2020).Google Scholar
- [67] . 2018. Pupil localization in image data acquired with near-infrared or visible wavelength illumination. Multimedia Tools and Applications 77, 1 (2018), 1041–1067.Google ScholarDigital Library
- [68] . 2017. Smoothening of iris images and pupil segmentation using fractional derivative and wavelet transform. In Proceedings of the 2017 4th International Conference on Signal Processing, Communication and Networking (ICSCN’17). 1–6.Google Scholar
- [69] . 2018. PuRe: Robust pupil detection for real-time pervasive eye tracking. Computer Vision and Image Understanding 170, 4 (2018), 40–50.Google ScholarDigital Library
- [70] . 1999. Robust pupil center detection using a curvature algorithm. Computer Methods and Programs in Biomedicine 59, 3 (1999), 145–157.Google ScholarCross Ref
- [71] . 2007. Eye tracking methodology. Theory and Practice 328, 614 (2007), 2–3.Google Scholar
- [72] . 2018. Pupil detection based on color difference and circular though transform. International Journal of Electrical and Computer Engineering 8, 5 (2018), 3278–3284.Google Scholar
- [73] . 2016. An approach of noisy color iris segmentation based on hybrid image processing techniques. In Proceedings of the 2016 International Conference on Cyberworlds (CW’16). 183–188.Google Scholar
- [74] . 2015. Refining Width Parameter of Gaussian Kernel Function for Tuning Coherent Point Drift (CPD) Registration. (2015).Google Scholar
- [75] . 2015. Normalization: A preprocessing stage. International Advanced Research Journal in Science, Engineering and Technology 2, 3 (2015), 20–22.Google ScholarCross Ref
- [76] . 2021. Coordinate attention for efficient mobile network design. arXiv preprint arXiv:2103.02907 (2021).Google Scholar
- [77] . 2019. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 510–519.Google Scholar
- [78] . 2017. Multimodal measurement of depression using deep learning models. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge.Google Scholar
Index Terms
- Automatic Assessment of Depression and Anxiety through Encoding Pupil-wave from HCI in VR Scenes
Recommendations
Automatic detection and classification of emotional states in virtual reality and standard environments (LCD): comparing valence and arousal of induced emotions
AbstractThe following case study was carried out on a sample of one experimental and one control group. The participants of the experimental group watched the movie section from the standardized LATEMO-E database via virtual reality (VR) on Oculus Rift S ...
Reflecting on Emotions within VR Mood Worlds
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable ComputersVirtual reality applications can be an effective tool for emotion regulation and therefore support emotional well-being and mental health. However, many VR applications focus on inducing (positive) emotions and there is little research on how users can ...
A widget framework for augmented interaction in SCAPE
UIST '03: Proceedings of the 16th annual ACM symposium on User interface software and technologyWe have previously developed a collaborative infrastructure called SCAPE - an acronym for Stereoscopic Collaboration in Augmented and Projective Environments - that integrates the traditionally separate paradigms of virtual and augmented reality. In ...
Comments