Designing Human-centered AI for Mental Health: Developing Clinically Relevant Applications for Online CBT Treatment

3.2.1 Study Theme “Understanding Patient Mental Health: Risks, Progress, and Barriers”.

A key focus in the review practices of PWPs is to gain a sufficient understanding of their patients’ mental health. In the first instance, this includes (i) gaining an understanding of the person's main mental health problem to be able to identify the right treatment, and effectively focus or adapt the treatment to the person. To this end, supporters collate patient information from initial screening and treatment calls that are recorded in a patient boarding card within the care providers EHR system together with other relevant health information.

During regular patient reviews, supporters then assess: (ii) whether there are any indicators of risk that have to be responded to immediately, and (iii) how the patient is progressing in their mental health throughout treatment, as one of the first and most important steps in their review. Predominantly, this is done by examining changes in the patients’ mental health symptoms as reported via standardized clinical questionnaires of PHQ-9 [60] (a 9-item measure of depression symptoms) and GAD-7 [64] (a 7-item measure of anxiety symptoms).

A “mental health risk” is typically indicated through a “risk tab” within the supporter interface that flags- when a patient scores higher on question nine of the PHQ-9, which asks about the person's thoughts on self-harm and intent to end their life. In some cases, mental health risk indicators are also picked up in patient's messages, or through changes in treatment engagement (i.e., absence of communication with the supporter, reduced program use). In cases of an identified risk, the supporter would often take action by calling the patient to assess what is going on for them and to clarify if their current scores actually reflect how they are feeling.

For assessing the “mental health progress” of a patient, PWPs review trends in patients’ clinical scores. If those scores show a gradual improvement, little time is spent reviewing these further. However, if the scores remain unchanged; change suddenly or significantly in either direction; or indicate a decline in mental health, then the supporter would take further action. For example, if the patient scores are not improving or are not at the level that the supporter expected them to be, they will look for potentially “barriers”. This can include strategies such as: (i) asking the patient in an online message or call how they are feeling to better understand their circumstances; or (ii) taking a closer look at key aspects of clinical questionnaire items on which the patient may score highly that could be indicative for example of “persistent sleep difficulties” or “high levels of anxiety”.

In cases where a patient does not improve after several weeks, they are brought to case management supervision (CMS) to re-assess the suitability of the current treatment approach for them. In general, all patients are regularly reviewed in CMS, typically: when they are new to the service; four weeks into an intervention; had no changes in their scores; or showed risks. In CMS, supporters discuss the patients’ main mental health problem and current treatment plan and consider their level of program engagement and clinical scores to identify risks of drop-out and strategies to address any treatment struggles (i.e., by changing the intervention, or stepping the person up to a more intensive care approach).

It is important to note that some supporters also described the need for caution in interpretations of the clinical scores, as they may not exactly reflect what is going on for the person. While frequent clinical assessments can provide a numerical indicator and trend of that person's mental health progression, these should not be given too much weight or consideration in isolation. Supporters, therefore, described efforts to assess clinical scores in the context of other information provided in patient messages and conversations. For example, additional information can help mental health assessment by providing contextual insights into why someone might be struggling with stress and worries (e.g., exam times for students), and can clarify if someone has easy access to the means to commit suicide (e.g., an isolated farmer) that require action.

3.2.2 Summary and Opportunities for AI.

In summary, supporters often gain an important understanding of their patients’ main mental health problem and life circumstances through an initial assessment call. Combined with program engagement reviews, these serve as contextual resources to evaluate patients’ mental health progression and guide treatment. A key focus in assessing patient mental health progress involves the regular review of clinical scores and their changes over time to detect any mental health risks, potential barriers to improvement, and requirements to re-evaluate the patients’ suitability for a particular treatment and adapt their care. Responding to these tasks, we identified the following (not fully exhaustive) list of questions and opportunities for AI that could be investigated further:

5.1.1 Adapting Patient Treatment for Improved Outcomes.

In keeping with the clinical literature on FIT, having access to the prediction was perceived as a useful indicator for assessing patient mental health progress by providing an additional perspective—based on data. In particular, supporters regarded a positive prediction (RI) to serve as “confirmation” that a patient is “on track” and to add “confidence” that they can continue with their chosen treatment approach. In these instances, the data was considered to “back-up what clinical scores may already suggest”, providing reassurance to PWPs that the treatment is working. P13 for example explains:

(…) I think it would be a really good boost if you found out that, actually, this client should be on track and that they should be likely to reliably improve. I think that would be just quite a nice boost that, ‘Oh, good. This is really going to be something that helps them’.

Contrarily, supporters assessed a negative prediction (No RI) as a prompt to explore why the treatment may not be working for a patient, and how it could be adjusted to better meet their needs. To this end, PWPs described taking a number of different actions. They would engage in more conversation and “funneling” with the patient to better understand why they might not be on the right path, what they find (un)helpful and discuss any changes that might need to happen. Related to this, supporters would work harder to identify any treatment, engagement, or mental health barriers that the patient might encounter through a closer review of available data; as well as increase their level of support for patients who “need a bit more attention in the review” through activities such as: switching the patient from online to telephone reviews; breaking down therapy contents more; providing access to additional resources; increasing the relevance of treatment contents (via tailoring); adding more treatment sessions or increasing the frequency of review sessions. P18 explicates:

I think with an online review client in particular if I was able to tell, say if we had this data really early on and we could see that it wasn't going to be effective, then what that will allow is actually I could call that client, e-mail them, have that more of a verbal discussion to see what the issues are, how can we support them in a better way. Whereas previously I might have relied on the information that they're leaving me to highlight if it was or wasn't working (…).

Furthermore, a negative prediction could also lead supporters to re-assess the fit of the chosen SilverCloud program for the patient or whether the therapy itself needed to be changed (i.e., to a face-to-face format) such that the patient would get the right kind of help sooner. For this, supporters described the benefits of using the prediction as data evidence for guiding decisions within CMS:

Firstly, a negative RI prediction for a patient can support decisions if a patient needs to be stepped-up to different care “sooner”. Regarding the prediction as additional, more “solid, objective data evidence”, supporters assessed it as helpful for building the rationale and backing-up their clinical judgment to change treatment; especially if the prediction aligns with their intuition that the patient might not be benefitting from current therapy. This provides opportunities for improved care planning through an earlier re-distribution of care resources and reduced delays in having a patient attend for too long the wrong care pathway. P6 states:

Just hopefully the chance to change it. If you think that person's not going to get into like reliable change there, and identifying the correct treatment options, you know, the earlier on that you do that, the more likely treatment is to work, so if it was flagging that actually, yeah, when you've discussed it with your supervisor this maybe isn't the right option, it gets them into the right one quicker. Yeah, it just makes it a bit more personalized I think, as well.

Secondly, the prediction can also assist in decisions about how many (more) treatment sessions supporters should administer to a patient. A positive RI prediction can confirm that the usual 4–6 review sessions might suffice, whereas a negative RI prediction as early as review session 3, would enable an earlier change in intervention; as well as indicate that patients may need a longer treatment period if the change is unlikely to be expected within that time; offering useful insights to resource management. P15 expands:

I think at Step 2 ⁵ with PWPs the most useful thing would be to not overuse sessions. So if you could predict early on that a patient at session three isn't going to experience RI it could change the intervention more quickly. So at the moment you might have a PWP who's working with a patient and they might do six sessions before they then move them to the next intervention, but if you can tell that somebody is not going to benefit or get to recovery from that intervention you might want to think about changing it sooner. How I see it would be useful would be in the step-up or changing intervention process.

Despite multiple proposals for how especially a negative RI prediction can provide a useful indicator that a patient is not “on track”, it does not provide specific, diagnostically-useful insights that could assist supporters’ understanding of potential treatment problems, nor does the AI provide any therapy specific recommendations (i.e., for content tailoring). P1 describes this limitation and a desire for potentially more “actionable” data insights:

“I suppose the difficulty with a prediction is it is ultimately a figure that we're going to be receiving. And that doesn't actually give me detailed information about my practice. That's where I get a bit, because it is just a number, so I might get a number saying the prediction is, there's a 63% chance this person will improve and recover, great. But that doesn't tell me what I need to do. (…).”

In summary, supporters described multiple ways in which they envisioned the proposed outcome prediction feedback may serve as a useful “flag” for reviewing their practices that would allow them to adapt treatment choices. This approach to personalization contrasts with other possible tactics whereby the AI could, i.e., help identify specific problems that a patient may encounter and provide to the PWPs specific suggestions for action. While potentially more insightful, we will discuss later our choice to pursue a simpler data insight in lieu of more complex inferences and conclusions that could be drawn by the AI to reduce risks that may otherwise arise in cases where the AI prediction is false; as well as to not interfere with supporters “sense of agency” in making their own professional assessments nor reduce opportunities for upskilling novices.

5.1.2 Early Indicator in Performance Monitoring.

While supporters saw the most potential in the use of outcome feedback for treatment adaptation, some envisioned its use in performance monitoring, by providing insights into (i) the effectiveness of individual PWPs; (ii) how well the health service is achieving its treatment targets or required resourcing; and (iii) anticipated success rates of a digital approach to help the uptake of online patients by supporters new to services like SilverCould.

PWPs in team lead and supervisory roles described how the predictions could provide “really specific data” on how well their supporters are doing, potentially allowing them to better monitor and assist staff performance. In this instance, negative RI predictions are regarded as a “training tool” that provides supervisors with an opportunity to work more closely with a PWP to help move forward their thinking and decision-making in how best to support their patients; and to do so in a very sensitive and carefully scaffolded manner to ensure they empower (rather than criticize) the PWP. P6 explains the benefits of identifying guidance or training needs early:

Yeah I think it would actually end up having quite a big impact in that you'd just be making sure that your PWPs were doing the right stuff and had enough support, because I know that, for example, at the moment the only thing we can really go by is recovery rates but that's often at the end, or just waiting to see if people complain about your PWPs, which isn't nice! Whereas I think if you could see on SilverCloud that actually there were quite a few people who maybe weren't quite getting there it's just it would encourage me to then have that gentle conversation about ‘do they feel like they know enough about SilverCloud’, ‘are they supported enough’, and then could in theory make more clients get into recovery (…). This would give me like really specific data which would be really helpful.

Furthermore, a few supporters considered the value of the predictions not just on an individual supporter basis but for providing a useful “helicopter perspective” to senior management and coordinators on how well the specific health service (i.e., NHS trust) is performing. Interpreting the prediction as a “forecast” on key performance indicators, they described its potential in clarifying if a service is due to hit its targets; or whether additional staff may be needed due to an identification of, for example, large numbers of complex patient cases.

Similarly, as a measure of service performance, the prediction is regarded as providing supporters, who are new to SilverCloud, with important insights into its effectiveness as a treatment. Especially, where predictions are predominately positive, PWPs described the prospect of it offering encouragement to clinicians who may be hesitant to get started with a digital treatment approach. P18 explains:

But I think this will be really valuable for people who, well clinicians who are new to the service and new to using this treatment because actually that will help build their belief I think quite early on that it does work, whereas I think I have that experience to know that it works and it works with a lot of people so I've got a lot of practice evidence that it works, whereas some people who've not worked in the service or not worked with SilverCloud may be a bit more hesitant of the digital work.

To conclude, supporters described the potential benefits of the prediction as an early indicator in performance monitoring to enable more timely adaptations to their practices; better resource planning at a service level; and for communicating the prospective effectiveness of SilverCloud to supporters new to the iCBT service.

5.2.1 Negative RI Predictions: Demoralizing Supporters to Take Action and Increasing Performance Pressures.

While a positive RI prediction can boost supporter confidence in their treatment choices; there were mixed results in supporters’ responses to negative RI predictions.

On the one hand, they describe negative RI predictions to serve as an important indicator for treatment adaptation and as a positive “challenge for them to improve the patient's situation”, providing them with a “push” to explore more how to improve the patient's mental health. With this mindset, they ascribed negative RI predictions not necessarily to their own work practices but attributed it to limitations of the “online” medium; other “situational patient factors”; and possibilities of the “data not being 100% accurate”. P4 explains:

(…) it's data. It's not going to be 100% accurate, because it's going to miss out little things. I wouldn't let it dishearten me, because I ultimately have that contact with the client and get to hear their feedback themselves that there may be small things that are stopping them from getting to reliably improve, like situational things. I guess I just wouldn't let it impact on me.

On the other hand, the majority of supporters also raised concerns how access to such data insight might “negatively impact on assessments of their performance and competence as a therapist”. Here, predictions that their patients are not going to improve as desired can contribute to feelings that they may not provide “the right level of care”; or are “not doing a good job”. P9 states:

(…) you might take it as a personal, not attack, but a personal question mark over your competencies or why they're not going to improve (…)

In instances, where negative RI predictions sustain for individual patients, supporters described how this could have a “demoralizing” effect and might lead them to give up. Expressing a reduced sense of agency and feelings that there is nothing they can do to change such outcome, P1:

(…) you've worked with them for three sessions and they're unlikely to make a RI, you just think, ‘What's the point? Let's just wait another session and then I'll take it to CMS and deal with it’, rather than actually being like, ‘Oh, what's going on?’.

Whether PWPs are more likely to perceive negative RI predictions as a “challenge to achieve improvement” or “thread to assessments of their professional competence” was described to depend on: (i) their personality or general mindset; (ii) their workday as determined by their caseload, associated time pressures and stress levels (i.e., high stress levels can increase sensitivity to take outcomes personally rather than to explore external factors); and (iii) the frequency of exposure to, and distribution of, negative predictions across their patient cohort. For instance, P1 explains how frequent exposure to negative predictions, especially in situations where they are very invested in patients, can impact their motivation and self-confidence:

I think that's where I feel this is tricky. This is where I feel a bit conflicted about how often we would see the stat, because I don't think it would… I don't think necessarily it would impact my level of how motivated I would be to support the patient, but more I would find it demoralizing. If I felt I was working really hard with that patient and I've reflected on it and I'm trying to do this, that and the other, and it keeps going back, I'm like, ‘Is this me?’ So, I guess the danger is you start kicking yourself constantly about it, after you've done all the exploration. So, I guess you try and pull out all the stops, yeah. But then there are kinds of gets that line where you're like, ‘Okay, how constructive is this?’ it's finding that balance, I think.

This suggests the need for careful design choices as to where, when, and how often supporters should interface with the prediction to reduce risks to supporter agency, confidence, and motivation. This may include decisions to only show the prediction to PWPs at specific (clinically relevant) times in the treatment journey to aid perceptions of it serving as a “flag” for reflection on patient outcomes.

Furthermore, how the prediction is positioned and contextualized within the supporter interface can invite patient outcome comparisons and additionally contribute to performance pressures. For example, the Client Dashboard (Table 1, concept 5) shows the prediction outcome for the entire patient cohort of a supporter. While this can help identify those at greatest need for more support, it depends on the distribution of negative RI outcomes whether a supporter feels motivated or demoralized to affect positive change. P18 explains how a predominance of no-RI predictions may feel threatening to self-perceptions of their professional capabilities:

(…) [If] I was to see that actually there was quite a low likelihood of all those people in the top 10 that I've got listed as soon as I logged onto SilverCloud every day, I think it would demotivate me but also, I think if I knew that the algorithm suggested it wasn't going to work it might affect the way that I approach that client if I could see that all my clients weren't going to work. I think it would definitely filter into some negative thoughts about my work practice abilities!

Similarly, supporters raised concerns about the Population Comparison (Table 1, concept 4), which was seen as another way to potentially monitor and compare supporter performance. For example, in instances where a patient is predicted to not improve and is less likely to improve than the general population statistics suggest, this could be seen as an indicator of the supporter under-performing. This can add to performance pressures within a work context that already “strongly emphasizes statistics and numbers to justify pay’ as well as invite ‘unhealthy competitive thought” whereby supporters might start to compare themselves to one another, which could detract from the important focus on the patient. P20 describes:

I think with the population comparison it comes back to that kind of concern around that bit of competitiveness (…) you might start thinking, ‘Well, I'm not doing as well as this person’, and, ‘They've got scores of lower than majority’ and that just might have negative implications for me as a practitioner because I'm thinking, ‘Oh, I'm not doing that well’. I don't want to feel like I'm doing a crappy job. (…)

All this suggests for design to focus supporter attention on an individual patient's care for improving that person's mental health outcomes, rather than to encourage cross-patient or cross-supporter comparisons.

5.2.2 Uncritical Treatment/Over-reliance on Prediction Outcome.

High caseloads, fast-paced work, and emotional stresses can mean supporters are not able to pause and reflect about a prediction in a “balanced manner” and could become overly reliant on it. This could manifest in PWPs uncritically accepting and over-trusting the data, which some may regard as a useful “shortcut” to making “quicker” decisions; potentially disregarding their own reflections, and what they know about the patient. P3 expands:

Well, our job is really fast paced. Most of the time it is hard to allocate a time to actually reflect of where this client is now, where we're heading, what do I do with it? And I think the downside of this is that therapists will disregard that reflection and just rely on the machine. ‘I'm going to just click on this and see what the machine tells me’, and, like I said, just disregard that side of reflection whether is this right based on what I know so far myself and having that relationship with the client? Do I feel that this fits in the situation? I guess that will depend from therapist to therapist, but then, like I said, our culture being fast paced, we can easily be drawn into shortcuts, give me the answer.

This tendency for over-reliance is moderated by assumptions that high confidence predictions provide “good data” that is “based on tens of thousands of previous cases”, which many PWPs described as potentially a “more reliable”, “more objective, evidence-based” indicator than their own (human) judgment. P16 explains:

(…) I know my intuition is not perfect and I know the data is not perfect, so it just opens up those conversations, doesn't it, which it needs to, and I'd be inclined to go with the data most of the time anyway, unless I've got a good reason not to. I'd probably trust the prediction accuracy as long as it was good predictability, I'd trust that more than my own intuition unless I had a really good reason not to.

Individual assessments of patient outcomes can also be more difficult to develop: (i) early on within treatment; (ii) for more complex cases; and (iii) where PWPs are less experienced in their work (i.e., trainees, novice PWPs, and other non-trained practitioners using SilverCloud), which increases chances that PWPs uncritically accept a prediction outcome “as is”. Interlinked with a potential over-reliance on the predictions are concerns about reduced support and possible neglect of the more individualized side of care, which we will expand on next.

In a few cases, supporters remarked how seeing a positive trend may suggest that their help could be less intense and take more the shape of “check-ins” with the person; encouraging them to continue with what they are doing; and shifting the focus to the online program as the main source of support as opposed to the PWP. In other words, some PWPs may—consciously or unconsciously—reduce their level of support for patients predicted to improve, assuming their recovery and use of SilverCloud in a self-sufficient way. While some supporters describe the benefits of being able to expand their “scope to help those patients who are unlikely to get to RI”, others expressed worries that over-trust in the prediction could mean some PWPs make rushed decisions to change or not change treatments without considering the individual case as much. P9 expresses:

I'm just scared that there might be negligence in the sense that therapists may become too reliant on it, that they may … ‘Oh well, they're going to recover in four weeks’ time anyway, so let's just quickly push them through the treatment’ and not really look at longevity and making sure that there is going to be … (…) I don't need to do this bit or that bit or we can skip this corner here or if they need it, it's always going to be there available to them in SilverCloud.

In such instances, the prediction is seen to potentially “disrupt” the usual efforts of PWPs to engage with patients in dialogue to identify how best to support them, or to gather additional information about the patient that could enable a more balanced assessment. P3 states:

This is what I'm saying as a disadvantage. You then lose the interpersonal side of things of have I checked if this person can actually then do it on their own without offering two more sessions? Because it can be just reliant on let me check in with the machines and see what he says about this.

All this emphasizes the need to ensure—through appropriate PWP training, supervision, and continued patient monitoring—that interpretations of the prediction should not mean a reduction in care that could risk having adverse effects on patient recovery, nor to forget about the all-important individualized side of care.

5.2.3 Need for Balanced Data Assessments: Imperfect Prediction Outputs are Only “One piece of Data”.

In keeping with the above, the majority of the supporters themselves were very mindful of and expressed the need for, the prediction to not take away from the “human-centred care” that is tailored to the individual. Cautioning that patient scores alone—which the predictions are based on—may not accurately depict a person's mental health, many supporters reiterated the importance to not treat the “patient as a number” and put clinical decision-making as “black and white”; advocating instead for supporters to use their professional skillsets and talk to the person to understand their struggles.

Furthermore, the vast majority of PWPs generally acknowledged that they would not fully rely on, or 100% trust the prediction at all times; describing their intent to treat the prediction as “one bit of information” that can serve as a “helpful indicator or guide”, but should not be treated in isolation. Instead, PWPs stated that they would evaluate the prediction in the context of other (i.e., situational) factors that influence, change, or otherwise explain patient outcomes. Considering the often unique circumstances of each patient, they explained how the main rationale for their any decision-making should still be based on what they are picking up from their patient, and their own clinical judgment. P6 describes:

(…) So I think it gives a bit of backup to our rationale I think as long as we're using it to back it up and not just as its own thing, then I think that would be helpful. (…) especially as we can never 100% predict what someone's going to do.

Their feedback suggests that incorporations of the prediction within their work practices should nurture a more symbiotic relationship between the data and clinician assessments. To supporters, the prediction presents “a suggestion based on a pattern”; and they regard the recognition of that pattern in itself to have value as it enables them to then pick up and pinpoint extra aspects in their care with the patient that the machine may have missed in its data. P4 articulates this as follows:

(…) it might be missing certain things. It's not going to pick up everything, and we know it's not going to be 100% accurate. It's going to be as accurate as it can be. But I guess it's just useful, isn't it? Because we as therapists are able to point out those things that are maybe not in the patterns, or pick up on those things that are not in the patterns as we talk to our clients. That would be my only concern is maybe not picking up everything and including everything in that pattern, so I probably wouldn't rely on it 100%.

To achieve a more balanced, symbiotic relationship in the way that supporters should come to work with the predictions to mitigate identified risks, requires careful PWP training, explanation, and interface design to effectively communicate what may constitute an “appropriate” data use: what is the prediction for; what are its limitations; and how can it be (un)helpful to supporter practices?

5.2.4 Implications of “False” Predictions.

While PWPs generally assessed 87% as a good prediction accuracy, they were mindful that the ‘machine could be false’, cautioning them to not treat the prediction “as a fact”. When asked during the interview how supporters would feel if they discovered predictions were false, they described this as ‘irritating’ and to cause them to become less trusting and more dismissive of the data, especially if it was false often. Discussing possible implications for patients, they considered false predictions however to only have a “small impact”. They ascribe this low impact, firstly, to the use of the prediction tool in conjunction with other available patient information (as detailed above). Secondly, PWPs highlighted the importance of continued monitoring of treatment outcomes following any adaptations made. Reflecting about the consequences of a false prediction, P13:

(…) We're still going to look into ways to still support that client, even if we don't think that it's going to be of any benefit. We're still going to look into barriers and how to still make the best of it and get the most from that programme, even if it's not going to be the sole thing that [this] person needs. So, I think if it's incorrect, I don't suppose it would make too much of a difference, because like we said, we're monitoring them anyway. So even if it says, ‘Yes, they're going to be fine’. Like that one says, ‘98%, it's going to improve, you've got nothing to worry about’, we would still be monitoring that anyway. So I don't think it needs to be 100%.

However, false predictions were also described to be disruptive to a patient's treatment journey if they caused unnecessary or unhelpful adaptations that could lead to poorer patient experiences or meant that the patient misses out on support needed or be delayed in their recovery. About the importance of correct predictions, P20:

I think it would have to be very important because we're kind of getting to the point where we're drawing a line between helping somebody for what they need versus just going back and forth between different things because we think that's okay and that's the right thing to do. Which I'm happy to do if they're going to benefit from it, but if that information wasn't that accurate I'd feel really bad for the patient because I'm just jimmy jamming back and forth for no reason really, or not a good enough reason.

Interlinked are concerns how prediction errors may falsely represent the performance and anticipated recovery rates of the health service, which can impact future funding. Prediction errors may also lead supporters to administer more treatment sessions than needed, impacting their caseload and care capacity as well as potentially adding to supporter fatigue and burnout.

Despite those concerns, supporters generally assessed the benefits of predominantly correct predictions to potentially outweigh risks of false predictions due to the prospect of overall improved decision-making and achieving better patient recovery rates than what they would achieve through current practices. P16 explains:

Well it can only be correct X% of the time, right? So it don't bother me if it gets it wrong, it's telling me it's going to get it wrong. It's just the best that it can do based on loads of data, which I don't have otherwise. (…) machines make false predictions, it's just saying nine in ten people, so one in ten times, it's going to give you a wrong prediction. (…) for that individual client, it's a pain but that's IAPT, you have to make these decisions anyway. At least you're getting it right 90% of the time now instead of, you know, 60%. (…) it's about improving my decision making, isn't it? Can this improve my decision making by 10%? If it gives me a false prediction, but overall, I know, my recovery rate increases by 10% and clients are getting better, then I'm happy because patients are always not going to get well.

Thus, while overall risks to patients and their care might be low, supporters cautioned how high rates of false information through prediction errors could negatively impact patient experiences; health service outcomes; and supporter care capacity. Simultaneously, we find that their feedback reflects a very balanced assessment of the risks overall through expressions of awareness that AI predictions will not always be correct, and suggestions that—on a large-scale service delivery level—the combined benefits of sufficiently accurate predictions are likely to outweigh the risks. Remaining mindful that limitations introduced by prediction errors and model uncertainty are difficult to overcome fully, it is nonetheless important: (i) to carefully explain to, and remind supporters—both in PWP training and within the user interface—that prediction outputs are probabilistic in nature, and thus, can be fallible; and (ii) to encourage them to contextualize and balance the prediction outputs with their own professional assessment as well as other information about their patient.

5.3.1 Identified Requirements and Sensitivities for Design.

In reviewing and comparing the feasibility of the design concepts, we identified four key sensitivities for design:

Create a context-appropriate visual that is easy-to-interpret at a glance: Due to busy schedules, the supporters unambiguously expressed the need for the prediction representation to be very easy-to-comprehend and quick-to-interpret. Here, many valued the symbolism of the Wheel Indicator whose graded chart with its green and red areas enables a “near instant understanding of a lot of information”. Yet, one supporter perceived the “red NO” label as a too “stark, attacking” visual, describing it alike a warning signal in cars that “screams failure” or “danger”. There was also potential for confusion if the Wheel was read as showing the prospective spectrum from deterioration (far left) to improvement (far right) rather than as a binary (yes/no) indicator for RI. As a more “neutral”, non-threatening, and simple design, the Visual Cue + Text (concept 3) felt more appropriate and was received best. It was assessed as a “clear, succinct and easy-to-access” presentation that solely focused on the RI prediction metric. The other concepts of Multiple Outcomes Table, Population Comparison, and Client Dashboard, all entailed more information and complexity that, whilst generally providing additional insights, were regarded as “too overwhelming” and to take up “too much time to read” and unpack. Given the constraints on supporter time, all this suggests the need for a fairly neutral, simple design with low information complexity. This necessitates a difficult tradeoff between providing as little information as possible whilst ensuring that the AI prediction becomes correctly understood for what it is, and appropriately interpreted for informing treatment. To manage information complexity, PWPs also recommended to communicate only “one concept” upfront, with the option to include other outcome definitions or additional context-information “on demand”.

Cater for diverse information representation preferences to help map prediction (category) to action: Overall, preferences for representation modalities were mixed and varied considerably across the supporters. In keeping with the above-described importance of time efficiency, most PWPs preferred designs with simple, clear visuals (concepts 1 + 3) that enable them to “easily see rather than having to digest the information of numbers” or text. Least liked were communications of the prediction output through text alone, as in the Multiple Outcomes Table. The text labels were assessed as “too vague”; making it difficult to understand the nuances or categories that distinguish, i.e., “very likely” from “likely” labels. This often led to expressions of a clear preference for, and proposals to add, numerical values. Especially for the Wheel indicator, supporters explicitly expressed their liking of the confidence percentages above text labels and simpler yes-no indications. Some, however, also raised concerns about how these numbers are to be interpreted, describing the potential to get hung up on the detail of these numbers, and to not have time to reflect about them under time pressure. P9 explains:

(…) I feel like I'd get too hung up on, say for example, if it's 95 versus 98, I'm like well what's that 3%? Whereas if you have a word, it's objective. If you can see where I'm coming from. It's less subjective, like well okay, this one's only 58, that's fine but it could be 60 but it … so you … it's hard …(…) if you've got a word that's objective, you interpret it, you understand it. It's a universal indicator, clearly. I appreciate numbers are universal as well, but unless we've got a clear scale, you don't really know what you're working with.

Whether supporters found text, visual or numerical information easier to process depending on their individual preferences and ability to map the prediction result to a “category” or “clear scale” that allows them to make sense of the data in their mind such that they can link it with subsequent action. While for some, text labels can feel less ambiguous than numbers, it is reverse for others; suggesting a hybrid design that includes multiple formats to account for diversity in preferences and aid meaningful matches of prediction (category) to action.

Facilitate PWP work practices in providing patient-focused, individualized care (not work pressures): Preferences for the various design concepts were, of course, also moderated by the anticipated use scenario and how the prediction data could potentially be actioned. Most values were ascribed to the use of RI prediction in assisting the individualized care practices of PWPs by enabling targeted adaptations of the patient's treatment for improved outcomes (Section 5.1.1). To this end, design concepts that showed the prediction in the context of an individual patient's profile were preferred over, i.e., the Client Dashboard (concept 5). While the dashboard enables the spotting of trends in supporter performance across patients, which some supervisors regarded as useful insight; others raised concerns about it potentially reflecting badly on their work practices, especially if many patients are predicted as unlikely to improve. What might complicate these reviews is that the predictions shown in the table reflect the outcomes of patients at different stages in their treatment journey, which might be less appropriate to compare across. Similarly, for assessing an individual patient, the Population Comparison (concept 4) was evaluated as “least helpful”. While some supporters saw the benefits of being able to give oneself a “pat on the back” if the visual showed for their patient a higher chance of RI than for “similar patients”; this was a concern for others, who saw additional statistics that showed comparisons as adding to supporter stress and competition, and who questioned the relevance of such comparison for aiding individualized, patient-centered care or discussions of an individual patients’ circumstances during CMS meetings. P1 states:

(…) And as a practitioner, personally, I don't think that's helpful for how I would treat my patients because it's hard enough personalizing treatment already, let alone being like, ‘Oh, well, compared to others who are in the same situation, they should be...’ And people shouldn't be fitting in with everybody else. That's the whole point of mental health. It's not about should, it's about ‘where are you’. So, I wouldn't want to see that comparison.

All this re-emphasizes the importance of supporting “individualized care” practices and avoiding threads to supporter motivation or adding to performance pressures through patient comparisons.

Help ground prediction outcome in real-world data: Despite the above critique of the Population Comparison concept, it is noteworthy that some supporters assessed the additional information it provides to offer a useful context for interpretations of the prediction and treatment choices. For example, a less than average chance of RI may influence how closely a supporter monitors or supports a patient. Here, some of the PWPs suggested that the comparison information “gives more rationale to why the machine has decided about the chances of RI” and “more explanation why the data should be considered”. In other words, some of the supporters described how having this additional context information can help ground the prediction within real-world data, and how this can help their understanding of, and trust in, the AI-generated data. P6 explains:

I think it's just a bit more interesting than just having a prediction, I think that's really good but because it's drawing on real life examples, it's taking you back to people who have actually used the program rather than it just being a number. I think it makes it a bit more solid.

Thus, while patient/ group comparisons may be less appropriate for this specific mental health context, and more complex information likely ignored due to work stresses, providing nonetheless access to information (on demand) that details the source data may aid to the credibility and potential acceptance of the prediction within clinical care. For this, it is paramount that the design aim remains to encourage understanding of the prediction rather than to persuade supporters to uncritically accept the data output.

5.3.2 Design Rationale: Choices Made in a First UI Integration for RI Outcome Prediction.

Responding to these findings, the UX team at SilverCloud designed how the RI improvement prediction would interface within the Supporter Intervention Management site, as is shown in Figure 4.

Fig. 4.

View Figure

Patient-focused Indicator: Aiming to serve as a useful indicator to inform and potentially adapt the treatment of an individual patient, the prediction is placed prominently within the “patient review status” section at the top of the patient profile page. This enables supporters to quickly assess how actively engaged the patient is with the iCBT program; whether they are due a review; and what their prognosis is for achieving RI. This placement allows for the prediction to be seen and assessed in the context of all other patient information, including the trajectories of their depression and anxiety scores over time, which are the inputs to that prediction.

Succinct, Clear and Diverse: A key driver in the design is ensuring that the RI prediction can be understood quickly, at a glance, by a non-technical audience with varying levels of expertise as PWPs and varying experience in using SilverCloud. Building on the Visual Cue + Text, the design charts five boxes which map to a five-point Likert scale that is articulated through short text labels that categorize the prediction as a range from “very unlikely” to “very likely” for a patient to achieve RI. Color gradients of grey-to-green help create a neutral, non-threatening visual. To not overcrowd the design and unnecessarily draw too much attention to the detail of numbers, percentages are included in a drop-down menu to account for diverse information review preferences.

Text Clarity and Explaining Non-Predictions: To ensure that the words chosen for labels and contextual explanations are precise, help an accurate understanding of the prediction, and reduce risks of the information being misinterpreted, requires multiple iterations and considerations, including whether to provide explanations, i.e., for “empty state” conditions for which we discussed that it would be better to not show a prediction. This pertains to (i) patients, who have not yet completed three clinical questionnaires required to make an accurate enough prediction; (ii) those whose starting PHQ or GAD are below the thresholds for RI (PHQ-9 score <=6, GAD-7 score <=4) and thus, achieving RI is numerically impossible; and (iii) those whose score(s) are below caseness.⁶ For patients below caseness, it is numerically very difficult (albeit possible) to achieve RI, which makes negative RI predictions highly probable. Yet, those predictions are less likely to indicate poor patient progress or need for more support, as these individuals are already in the desired score bounds of “recovery”.

Explanation vs. Openness for Interpretation and Appropriation: We also discussed including additional explanations for patients in the minimal-to-mild score ranges to explain that these “typically won't show significant improvement” and to suggest that “the best outcome for these users is the maintenance of low symptoms”. While there can be the desire to provide extra information for cases that can be more difficult to interpret, we decided to include those only within a general but not instance-specific explanation to avoid over-interpreting individual cases for the supporters. These are deliberate considerations in order to: (i) avoid for PWPs to ignore the prediction unless it was for more serious cases; (ii) nor to restrict how PWP may make use of the prediction within their clinical practice. Partly, our research seeks to better understand exactly their clinical decision processes as well as any unanticipated use scenarios for the prediction, especially for unusual patient cases. Given the often unique, specific circumstances of each mental health patient, to generalize or narrow interpretations of the prediction, or to suggest what (concrete) actions to take in response, may risk reducing the applicability and potential usefulness of the data insight for a wider variety of patients. It could also feel disempowering to supporters, and impact their sense of agency, if they felt the prediction (and proposed actions) would try to replace their professional assessments, or otherwise limit their choices.

6.1.1 Positioning AI-derived Data Insights as Inputs to Human Sense- and Decision-making Processes.

Our user research identified two main ways in which predictions of RI outcomes could assist the work practices of iCBT supporters. They could serve as: (i) a “validator” to help confirm supporter decisions in cases where positive predictions align with their own clinical assessments, potentially boosting supporter confidence; and as (ii) a “flag” for negative prediction cases or where predictions were incongruent with supporter assessments, inviting pause to reflect and re-evaluate the patients’ current situation that can prompt for adaptations to existing practices. As such, our AI output does not provide any more specific (i.e., diagnostic) information that could assist supporters understanding, e.g., of the patients’ mental health state or potential treatment blockers, nor does it provide any concrete recommendations for what actions to take or propose to a particular patient. While more advanced AI applications are technically possible and could offer valuable additional insight, there are a number of reasons why we pursued a more general, less directive approach for generating AI insights:

Tradingoff the Risks and Benefits of Designing Complex System Inferences vs. Simpler Data Insights: Firstly, whilst the delivery of more complex data insights is an exciting prospect, it can be more challenging to achieve sufficiently robust and reliable data models. This is particularly pronounced in mental health due to general difficulties to establish what would constitute an optimal (aka ground truth) approach to treatment for a specific patient, even amongst health professionals [33]. In other words, while more ambitious algorithmic modeling efforts may propose greater gains, these can also come at an increased risk for cases where patients are falsely predicted for [101]. We believe that this presents a key challenge, especially for the design of personalized interventions that seek to increase the relevance and outcome of treatment for a specific individual. Yet, in cases where more specific, tailored recommendations may fail to deliver on their promise and mismatch the needs of patients or care providers, this can have opposite effects on patient engagement and health, and diminish AI utility and trust. Being mindful that AI systems are rarely, if ever, 100% accurate, we were very deliberate in our choices to explicitly position AI-outputs as part of human (expert) assessment and decision-making processes as a mechanism for managing those risks. In doing so, this leaves the human, rather than the machine, accountable for interpreting each patient's unique circumstances and, in response, determining appropriate actions forward. It also broadens the scope for other, potentially unanticipated use scenarios of RI prediction, and ensures its application to a wide range of patients. Thus, tradingoff risks and benefits, we consider this “AI informed” approach to human-supported care delivery [70] as a more ethical and responsible path towards early introductions of AI insights into mental healthcare contexts.

Designing for Human Expertise and Agency in AI-Informed Work Practices: Secondly, our research investigates how we can empower clinical supporters with AI. Thus, our aim is not to reduce the need for supporter input and analytical effort (in favor of the technology), but to explore how AI-insights could help maximize the impact of their “human” involvement in patient reviews. For this it is paramount that the supporters do not perceive the provision of AI-insights as competing with, or replacing them in, their professional expertise as this could unnecessarily undermine them in their role; as well as reduce their willingness to support the development and adoption of AI approaches in their work [100]. Thus, by creating AI outputs that serve merely as a useful “flag” to inform clinical care, supporters remain “in charge” of examining more closely the circumstances and potential reasons for a particular prediction outcome and determining directions forward. The hope is that this can help preserve a sense of agency and purpose in their role, which is important for supporter motivation and job satisfaction. Other research exploring decision-support [52] goes one step further and argues for AI systems to explicitly suggest appropriate next steps within the technology design to help clinicians make the connection between AI output and their healthcare practices. Either way, for HCI research this suggests opportunities for interface design to—implicitly or explicitly—aid supporters in the identification of the right subsequent actions, which in our case may involve explicit design decisions to assist supporters in their search for explanatory information (i.e., by encouraging them to look for certain mental health blockers).

Understanding the Impact of Design Choices on Work Practices and Workflow Integration Challenges: Our study findings also identified how specific design choices such as the frequency of (especially negative) prediction outcomes or their positioning and contrasting with other information (especially comparisons across patients) could add to work pressures, cause demotivation, and a reduced sense of agency in supporters that their actions can indeed affect positive change. All this warrants careful considerations in future design and research to study the actual impact of achieved AI predictions: (i) how AI applications can help care providers make more-informed, confident treatment choices for improved patient outcomes; as well as (ii) how integrations of AI outputs within everyday healthcare come to shape clinicians understanding of their own role; and how their design can help to minimize disruptions to their clinical expertise and work culture (cf. [95]). All this can help advance learnings how AI technology may best assist health care providers in their practices.

6.1.2 Protecting the “Human-ness” in Human-supported, Digital Healthcare Delivery.

While, as described above, there can be many different visions for how AI technology could come to transform (mental) healthcare, we have chosen to focus our efforts on identifying strategies forward for empowering (rather than replacing) clinicians with AI. Especially in the context of psychotherapy, we further acknowledge the importance for technology to not unnecessarily disrupt the interpersonal relationship between patient and care providers; seeking to protect the all-important “human touch”, “genuine sense of care” and “empathic understanding” that often characterizes these relations, and are crucial for treatment success [92, 115]. However, as indicated in our initial user study (see [99] and Appendix A for main findings), trying to foster a connection between supporters and patients within a remote, self-administered therapy format that involves the asynchronous sending of online messages can already put into question the authenticity of supporters’ identity as “real” humans. To counteract this, our findings describe supporters active work in carefully crafting their feedback messages to patients such that they convey a “sense of care” by including personable expressions; person-specific guidance; communicating that they heard the person's concerns; and ensuring that they respond to these concerns in an “empathic way” by building on their own life and professional experiences.

Integrating AI Insights Sensibly within Interpersonal Dynamics and for Supporting Human Relations: Given the importance of developing a genuine bond between supporter and patient within a computer-mediated setting, we were therefore deliberate in our choices for the AI and possible optimizations to supporter work to not go down routes towards standardizing or otherwise automating existing processes. Aiming to protect the “human-ness” of supporter communications we would favor, for example, the personal look and handcrafted feel of their personalized messages that bring forward individual communication styles, over more templated, machine-led communication approaches that may increase efficiency in message production, but at the costs of inviting perceptions of a “robotic, auto-responder system”. We believe that if we move beyond common development goals of “improving productivity” and considered more closely what may constitute a desirable use and integration of AI insight within healthcare from a patient and care provider point-of-view, this can open-up many important additional routes for AI application. Using goals of “protecting or nurturing supporter-patient relationships” as an example, future work may explore uses of AI to: (i) help increase patient awareness of the supporter's role and investment in their therapeutic success to foster their bond and associated benefits; and (ii) assist supporters to more closely connect with their patients. In this regard, our feedback-informed approach (RI prediction) in itself is sought to give supporters an additional view-point on their patients to enable them to be more responsive to those most in need of additional care, which can aid their therapeutic relationship. Future work may also focus more explicitly on the relational needs of the supporters by generating, i.e., data insights that foreground: how their actions came to matter to patients (i.e., highlighting support successes, or what types of actions are most helpful to their patients); what communication styles their patients may respond to best (see work by [24] as an example); or otherwise expand ways in which supporters specific skills and expertise can become leveraged more. Such efforts can aid a feeling of “congruence” on the part of the supporter for investing in the patient's treatment success, keeping them engaged and motivated in the process, which is often rooted in an underlying desire to “be helping others”.

AI Acceptance vs. Perceptions of AI Dehumanizating Healthcare: Such considerations of where AI technology might come into interpersonal dynamics and caring relationships with a view to both sustain or extend human relations and avoid undermining health professionals in their roles and expertise may further play a key role in improving acceptance of AI applications within such care contexts. Especially in healthcare, there are increasing concerns about the role that AI might play in “dehumanizing medicine” [95]. Above and beyond already existing trends within health services to “continuously monitor” outcomes and focus on success “metrics”, there are tendencies within AI work to treat individuals as “data points” in algorithmic modeling [20] by transforming a person's individual (mental) health experience into compressed mathematical representations that allow for the identification of large-scale patterns [100]. This is a tension that we also saw in our user research findings that highlighted concerns about the introduction of a binary prediction to lead to simplified interpretations whereby patients become treated as a “number” and prediction outcomes simply read as “black and white”. Thus, in dealing with imperfect AI technology, as we will discuss next, it is paramount that we ensure in design and training that healthcare providers can maintain a more holistic view of their patients and focus on individualized care.