1 Introduction
Connected to the rising prevalence of mental health problems, there is an increasing gap between the demand for mental health care and the resources of health services [7]. As patients are subject to substantial delays when waiting for therapy [7, 9] new ways of combating this global shortage of mental health services have to be found. One possible solution lies in the usage of conversational agents (CAs). CAs can be used to overcome certain barriers, such as long waiting lists and geographical problems that prohibit the attendance of face-to-face counseling [7]. Additionally, CAs offer unique benefits to mental health, such as being continuously available, responding to context of both the user and the user’s language, the capacity to give responses based on clinically relevant mental health theories and the avoidance of stigmatization of affected individuals [23]. However, thus far CAs are not deployed widely and research to assess the potential impact of this type of assistive technology in mental health is still in its early stages [23]. For young adults, the lack of professional support services poses a major problem since precisely the age group between 17 and 25 visits health care professionals particularly rarely [15]. Hence, low-threshold services are needed that reach those affected in an early stage of mental health problems, such as depression [15]. In Germany, every sixth student is affected by mental health problems [15] and the prevalence of depressive disorders is especially high in adolescents and people in their 20s in comparison to other age groups [19]. Most students are at this critical age. In addition to the general consequences which depression may cause, students might face further challenges in the academic context. Deroma et al. [10] revealed that depressive symptoms in students are associated with lower academic performance. Hunt el al. [17] found a connection between depression and higher drop-out rates. The psychological strain on students further worsened through the COVID-19 pandemic due to the lack of social contacts [1].
Empathy-driven chatbots (CBs) using methods based in cognitive behavioral therapy (CBT) for mental health counseling have already been proposed and first studies have shown the efficacy in treating symptoms of depression [7, 9, 13, 18]. The central goal of CBT is to identify the patient’s problematic patterns in cognition and behavior in order to modify those through cognitive reconstruction and behavior changes [3, 12]. While it can be applied to many psychiatric conditions, from its beginnings CBT was created with a focus on depressive disorders [34]. However, so far there is no research on how empathy-driven dialogues affect the usability and acceptance of voice assistant (VA) based CBT. Therefore, we address this gap by proposing the concept for an empathy-driven VA-based CBT system for the mental support of students with depression. The usability and acceptance of the concept is evaluated through a user study with the target group based on a developed prototype. Specifically, the effects of emotion sensitivity on usability and acceptance are investigated through a single blind randomized controlled A/B testing experiment.
Anzeige
In Sect. 2 the fundamentals of emotion sensitive CA-based CBT will be laid out, while incorporating the current state of research. Subsequently, Sect. 3 will introduce the concept and prototype of an empathy-driven VA capable of delivering CBT-related psycho-education to students. Section 4 will describe the methodology, execution and results of a conducted study to investigate the effects of sentiment recognition and empathy-driven dialog management on the acceptance and usability of the created system. Consequently, implications and shortcomings will be discussed. Section 5 will complete the paper through a concise conclusion and an indication of future work.
2 State of the Art: Empathy-Driven CA-Based CBT
Norcross et al. [33] demonstrated, that regardless of the type of psychotherapy chosen for treatment, the therapeutic relationship plays a crucial role in therapy outcome. Among others, empathy has been identified as an effective element of the therapeutic relationship [11]. Hence, there has been ongoing research with empathy-driven CBs and Embodied Conversational Agents (ECAs) delivering chosen methods based in CBT.
Wysa is an empathy-driven CA that aims to increase mental resilience and uses sentiment recognition and dialog adaptation in text-based conversations1. In 2017 a study was conducted by Becky et al. [18] to investigate the efficacy of the CA in mediating techniques based in positive psychology and CBT in order to increase the mental well being of users. Participants that used the app in between measurements of their mental state had a significant improvement in depressive symptoms in comparison to control.
Fitzpatrick et al. [13] designed and implemented a CA called Woebot as a CB application intended for daily usage. The therapeutic approach of the application consists of psycho-educational content oriented toward self-help for CBT and therapeutic process-oriented features. These include, among others, empathetic responses, meaning that Woebot adapts its responses to the user’s mood, and tailoring, meaning that the content provided to the user is also adapted to their mood. The conversational style is based on both the dynamics of social interactions and clinical decision making. Fitzpatrick et al. evaluated the feasibility, acceptability and preliminary efficacy of Woebot on a sample of college students with symptoms of anxiety and depression. The results indicated that Woebot could be a suitable option of CBT delivery.
Anzeige
Miranda et al. [22] developed a mobile-based ECA for detecting and preventing suicidal behavior called HelPath. Users can make inputs only through predefined interaction options with the GUI. Each interaction is organized as a short therapy session, starting with a query of the user’s current mood. The subsequent procedure of therapy sessions may vary depending on two factors. Firstly, the user is able to decide, which kind of activity they want to do. Secondly, the application is designed to respond empathetically in order to increase the success of the therapeutic interventions. One of the strategies for generating empathetic reactions includes explaining and suggesting suitable CBT activities based on the user’s emotional state. This empathetic strategy also contributes to the variability of therapy sessions. Another empathy-driven strategy implemented in HelPath is the generation of emotional feedback to the user’s utterances. To identify the user’s emotional state from their utterances, the cognitive appraisal theory of emotions [28] is used. Usability, adherence and acceptability of HelPath was evaluated in a study with individuals affected by suicidal thoughts [22]. The results indicated, that HelPath is perceived as emotionally competent and participants evaluated its level of adherence as positive [22].
Ring et al. [25] proposed an ECA that communicates CBT principles verbally and non-verbally to users with the goal of recreating face-to-face conversations. The system’s interface allows speech input as well as input via touch screen. However, user input is constrained to pre-formulated answers that can be chosen. Two sub-systems are used for speech recognition and to detect affect from user input. The basic building block of the affect detection system consists of a classifier which assigns the user’s utterance to one of three valence categories (happy, neutral and sad). In addition, the user’s facial expression is used for affect detection. The generation of empathetic responses to negative affect relies on Russell’s circumplex model of emotion [27]. For empathetic responses, suitable items are picked from a hand-created list of responses. The feasibility and acceptance of the counseling system was evaluated in a pilot study on young individuals aged 18 to 28, who had symptoms of depression [25]. The study showed that the system is capable of evoking and responding to the affective state of users in real time and that the users felt understood by the agent [25].
In summary, while there has been promising research in the field of empathy-driven CB-based and ECA-based therapy approaches, research on the effects of empathy-driven dialogues in VA-based systems for CBT on usability and acceptance is still lacking.
3 Design and Prototype of an Empathy-Driven VA for CBT Delivery
Fig. 1.
Procedure of initial therapy session
×
![]()
We designed an empathy-driven VA capable of delivering psycho educational content as one selected method from CBT to students in a human centered design approach, based on a thorough analysis of related work and of the context of use. The interactions with the VA are designed as short therapy sessions. In order to track the course of depressive symptoms of users, they are asked to evaluate the severity of their symptoms before the actual therapy session starts. If the user’s scores indicate that they suffer from severe depression, they are encouraged to seek professional psychological help and the session terminates. Afterwards, the user is asked about their current emotional state. Following the concept of Ghandehrioun et al. [14], the user’s utterances are classified into an extended version of Russell’s circumplex model [27] to identify the user’s emotional state. The extension of the circumplex model includes the addition of a neutral category to the four quadrants and the inclusion of more mood-describing words to make the model more precise. The system adapts its response accordingly to the identified emotional category. Therefore, different empathetic techniques are used i.e. mirroring, empathetic listening, cheering up or calming [30]. As a sentiment recognition strategy a rule-based approach is used. Hence, keywords describing the user’s mood are looked up in lexical resources and matched to emotions. Subsequently, therapeutic content is delivered in a quiz format, containing content regarding depression and CBT, to support long-term retention of information as shown by Roediger and Butler [26]. During the quiz, the user is given statements about depression, psychotherapy and psychological treatment possibilities and has to decide, if those statements are true or false. Afterwards, the user is given immediate feedback on the correctness of their answer through empathetic responses - as those have been shown to be important for the therapeutic relationship [11, 21] as well as for the interaction with CAs in general [8]. Hence, feedback is adapted depending on the correctness of the user’s answer and detailed information on the respective CBT-related topic is given. To ensure controllability and customizability as important interaction design principles [29], the user is able to chose a quiz on a specific CBT-based topic during the course of the interaction with the system. The procedure of the VA-based therapy session can be seen in Fig. 1.
4 Evaluation of Acceptance and Usability
To assess the usability and acceptance (cf. ISO 9241-11 [31]) of the developed system we conducted a single blind randomized controlled A/B testing experiment. Specifically, we wanted to investigate, how empathy-driven dialog management affects usability and acceptance of the system in students. Participants were recruited using social media. All participants had to sign a privacy policy and consent form in order to comply with data protection provisions.
4.1 Participants
To be eligible to participate in the study, subjects had to be a current or recent student and be able to speak German fluently (as system’s utterances were in German). While symptoms of depression were measured during testing using the Patient Health Questionnaire (PHQ9) [20], none of the participants were diagnosed with a current depressive episode before taking part in the study in order to comply with ethical concerns. The implications of this predicament will be discussed later. Although, 16 participants took part in the study, 6 participants had to be excluded due to incomplete data provision. Overall, 10 subjects completed the survey and testing (5 females and 5 males). The average age of participants was 25,6 (ranging from 21 to 30).
4.2 Methodology and Material
Subjects were randomly assigned to test either an empathy-driven VA delivering psycho-educational content (as described in Sect. 3) in group A or to a VA without empathetic dialog management, delivering the same psycho-educational content, in group B. The existence of different experimental groups was not disclosed to participants. To measure differences in usability, the System Usability Scale (SUS) [2] was used. Consumer satisfaction and acceptance in the context of digital mental health interventions was assessed using the adapted Client Satisfaction Questionnaire for internet-based interventions (CSQ-I) [6]. The Net Promoter Score (NPS) [16] was applied as an additional indication of the perceived usability and acceptance. As subjects’ technology commitment might effect their assessment of acceptance and usability of the proposed system, it was measured using the subscales technology acceptance and technology competence belief of a questionnaire for measuring technology commitment (TB) [24]. Participants first had to answer questionnaires regarding demographic information, experience with depression and psychotherapy, the PHQ9, the subscales of the TB, and experience with VAs. Afterwards, participants tested the VA-based system (provided with either the empathetic or the non-empathetic version). Finally, participants had to fill out the SUS, CSQ-I, items included as a manipulation check and the NPS. Furthermore, they were given the opportunity to provide additional feedback.
4.3 Results
PHQ9 scores ranged from 1 to 16, whereby scores ranged from 2 to 7 in the experimental group and from 1 to 16 in the control group. Referring to Kroenke et al. [20], these scores indicate that all subjects in the experimental group showed none or mild depressive symptoms while in the control group, one subject showed moderate depressive symptoms and one even showed moderately severe symptoms . However, differences between groups in depressive symptoms were not significant (M\(_{exp}\) = 4.0, SD\(_{exp}\) = 1.9; M\(_{contr}\) = 8.0, SD\(_{contr}\) = 5.02, p = .17).
Regarding the technology commitment, no significant differences could be found between groups either, neither in the subscale technology acceptance (M\(_{exp}\) = 12.6, SD\(_{exp}\) = 3.93; M\(_{contr}\) = 15.6, SD\(_{contr}\) = 3.26, p = .27) nor in the subscale technology competence beliefs (M\(_{exp}\) = 17.4, SD\(_{exp}\) = 2.65; M\(_{contr}\) = 18.2, SD\(_{contr}\) = 3.12, p = .71), although the control group showed slightly higher values in both subscales.
The second part of the survey contained the SUS, the CSQ-I, items for manipulation check and the NPS. Overall, the mean SUS score was 79.25 (SD\(_{overall}\) = 10.67), indicating a usability above average2. While no significant differences were found regarding SUS scores between groups (p = .95), the control group rated the usability slightly higher than the experimental group (M\(_{exp}\) = 79.0, SD\(_{exp}\) = 9.82; M\(_{contr}\) = 79.5, SD\(_{contr}\) = 11.45).
Similar results were found regarding the CSQ-I. The mean CSQ-I score over all subjects was 23.7 (SD\(_{overall}\) = 4.86) and while no significant differences were found between groups (p = .13), the control group assessed the application slightly higher than the experimental group (M\(_{experimental}\) = 21.2, SD\(_{exp}\) = 4.58; M\(_{contr}\) = 26.2, SD\(_{contr}\) = 3.71). Regarding the items used for manipulation check, significant differences occurred only for the item asking how well subjects felt understood by the respective VA (item Comprehension Skills), whereby the comprehension skills of the neutrally acting VA were rated higher than those of the emapthetically acting one (M\(_{exp}\) = 2.6, SD\(_{exp}\) = 0.49; M\(_{contr}\) = 4.0, SD\(_{contr}\) = 0.89, p = .025). When asked for rating the appropriateness of the VA’s answers, the assessments in both groups were exactly the same (M\(_{exp}\) = 4.2, SD\(_{exp}\) = 0.4; M\(_{contr}\) = 4.2, SD\(_{contr}\) = 0.4). For the assessment of perceived aloofness, no significant differences between groups could be found either (p = 0.76) although participants of the experimental group gave slightly smaller ratings (M\(_{exp}\) = 2.2, SD\(_{exp}\) = 1.17; M\(_{contr}\) = 2.4, SD\(_{contr}\) = 0.49). The overall NPS was −10. When considering both groups separately, the NPS was higher in the control group (NPS\(_{contr}\) = 20) than in the experimental group (NPS\(_{exp} = -40\)).
4.4 Discussion
On sample level, a SUS score above average and a CSQ-I score above mean were obtained, indicating a high usability and acceptance of the proposed application. Those results support other research indicating that VA technology seems suitable for delivering psychotherapy digitally [4, 32]. Nevertheless, here presented application differs from those proposed within other studies. In contrast to a VA-based application supporting young adults receiving behavioral activation therapy for depression treatment in doing their therapeutic homework, here proposed application does not require users to already receive therapy [4]. Thereby, it addresses the problem of insufficient treatment options and provides more versatile possibilities of use.
Striegl et al. (2022) [32] also proposed a VA-based application which does not require users to receive therapy as a preliminary for its use. However, this application was intended to be used by elderly individuals suffering from depression whereas the application proposed here is intended to be used by students. Furthermore, as Striegl et al. (2022) [32] did not investigate the influence of empathetic DM on users’ usability ratings, they only equipped their VA with rudimentary empathetic capabilities.
Although SUS scores and CSQ-I scores were high, the overall NPS score was -10 indicating that subjects would rather not recommend the proposed application. Furthermore, no significant differences in SUS scores and CSQ-I scores could be found between groups meaning that the implementation of empathetic DM did not have effects on subjects’ usability and acceptance ratings. However, among manipulation check items, significant differences were found in the item Comprehension Skills, whereby control group ratings were higher than those in the experimental group.
Several implications could be derived from the obtained results. First, within this study, empathetic DM might have not influenced subjects’ usability and acceptance assessment. However, the strategies chosen for equipping the proposed application with empathetic abilities might have not been sufficient, as the neutrally acting prototype was perceived as being better in understanding subjects than the empathetically acting prototype. The other manipulation check results - which indicate that the empathetically acting prototype was not perceived as less aloof and its reactions were not assessed as more appropriate - further support this hypothesis. Additionally, mixed findings exist in this area of research in general. Ghandeharioun et al. (2019) [14] also used a control group design evaluating an empathetically responding and a neutrally responding mental health application and could not find any significant differences between groups. However, instead of using questionnaires requiring subjective evaluations, they used behavioral data as indicators for group differences. Bickmore et al. (2005) [5], on the other hand, found beneficial effects of the implementation of relational strategies such as empathy on working alliance and the desire to keep working with their proposed application continuously. However, this study is only partly comparable as it investigated a computer agent supporting users in changing automated health behavior, whereas the here presented study focused on a VA-based application delivering CBT.
Overall, it is difficult to compare the here presented results to those of other studies in the field of empathetic conversational agents, as most other research relies on different implementation and/or assessment methods. Therefore, further research is needed in this area.
A major limitation concerning the evaluation procedure and the obtained results of this study lies in the occurrence of several technical problems. Those problems negatively affected the evaluation in different ways. First, technical problems lead to the exclusion of participants resulting in a final sample size of only 10 subjects (5 in each group). A further consequence was that several testings had to be accompanied by the facilitator, which might have led to social desirability biases. Additionally, some participants could not interact with the VA properly which might have influenced their perception of the VA negatively. As those problems did not occur equally distributed over groups, they potentially influenced the evaluation systematically and thereby negatively influenced the internal validity of this study. Furthermore, they might have contributed to higher SUS, CSQ-I and NPS scores in the control group. Additionally, as stated before, while several of the participants showed singes of mild to in one case even moderately severe symptoms of depression, none of the test subjects were diagnosed with depression before taking part in the study. A followup study should therefore be carried out with students with diagnosed depression and with an improved prototypical system.
5 Conclusion
Within this study, no significant effects could be found between a group testing an empathetically acting VA and a group testing a neutrally acting VA delivering chosen CBT-based methods to students with depression. This could indicate that the implementation of empathetic DM does not influence the usability and acceptance of VA-based CBT delivery. With reference to the manipulation check results, another explanation might be that the selected strategies to implement empathetic behavior were not sufficient. Nevertheless, definite conclusions cannot be drawn as several technical problems - resulting in a small sample size and a potential systematic bias - impaired the evaluation. Therefore, a follow-up study with a larger sample size should investigate whether the proposed strategies are suitable to implement empathetic behavior in VAs and whether empathetic DM influences the usability and acceptance of VA-based CBT delivery.
Despite those problems, the obtained results on sample level are highly promising regarding the suitability of VA-based CBT delivery to students with depression. Additionally to high usability and acceptance ratings, qualitative feedback was very encouraging and emphasized the importance of research in the area of digital solutions for students with depression.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.