Learning machines in Internet-delivered psychological treatment

A learning machine can be defined as an autonomous self-regulating open reasoning system that actively learns in a decentralized manner, over multiple domains [6], or from one abstract model of its world to another. The basic definition of machine learning is as follows. “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E” [19, 2]. A learning machine, by contrast, not only learns, but also meta-learns, since it has to adjust its behavior according to the individual machine learning modules that it continually receives and assesses output from. This leads to: A computer program learns to learn if its performance of each task in T improves with experience E and also with the number of tasks (cf. [29]). Feedback allows it to self-correct its models and its processes of learning. Autonomy yields the possibility of sustained autonomous learning [14]. Openness indicates that new machine learning modules could be added at any time to the learning machine. That its learning is active means that it can pursue learning goals without explicitly being told to do so, guided by self-testing in accordance with the principles of operant conditioning [27].

Important aspects of learning machine “education” were mapped out by Turing [31]. The machine trains and dynamically adjusts its perception, after which it can move into reasoning, which requires further education, and then to interaction. The machine can then interact with humans, e.g., by presenting a prediction of an outcome, or by explaining its reasoning steps. The behavior of a learning machine is ideally interpretable to authorized observers, i.e., they should be able to follow its reasoning and not accept it as merely an educated black box. In some cases, less than optimal performance (cf. [21]) could be accepted to provide interpretability, e.g., when explicating a model by reference to its education from training data [17].

The learning machine has internal states, and rules for switching between them, like an ordinary Turing machine. Intuitively, each patient in treatment will pass through different states of a much simpler representation, in the form of finite automata: one automaton per patient. An end state can correspond to labels indicating, e.g., level of engagement with the treatment program, or successful response to it. To learn under which circumstances state-switching in the learning machine controlling this array of automata occurs is an important component of meta-learning. For a self-supervised Turing machine to be able to study its own tape operations and state changes, self-observation must be possible. This separates the learning machine from the much simpler machine learning modules that it incorporates, as finite automata are too simple models for managing introspection. The learning machine will not monotonically improve its performance over time and over tasks, but will have test modes in which experimental reasoning and inference will be self-evaluated and possibly lead to revisions of state-switching rules, e.g., probability or utility values.

To generalize from one domain to another, as in, e.g., switching between groups of patients having been referred to different Internet treatment programs, learning machines must adopt their learning to the settings in which they are situated. Such multitask learning is realized by means of inductive bias created from training inputs of related tasks [8]. This allows for perception, reasoning, and interaction about things pertaining to more than one domain [13]. In return, the reasoning may be directed by meta-rules that help steer cross-domain inference. Such steering involves ethics, norms, rules, laws, and all other forms of constraints. The interplay between perception, reasoning, and interaction makes reductionist approaches unsuitable for learning machines. Instead, a systemic methodology in which the machine can learn more than the sum of its learning from each task performed, should be adopted for its design. Given the generality of a learning machine, very few currently exist, and for very specialized tasks, or subject to meta-level constraints on data or its distributions [21]. Our aim is therefore to provide a constructive example of what a learning machine can be and how it can be applied to a case of practical significance. The hypothesis is that a learning machine can provide robust decision support to clinicians. This hypothesis will be evaluated in 2020 in a large and pre-planned randomized controlled trial. The confirmatory role of that evaluation will determine the importance of the contribution from the clinical side, thereby assessing the value of the exploratory search for regularities in the data.

The data analyzed by the learning machine in the here presented case study are unique in that it represents virtually all of the information passed between patients and therapists in a particular mental health care setting. Because care is Internet based, there are few subtle pieces of information missed out on. Given such a special data set, why is it reasonable to think that learning machines can contribute to the task of drawing important conclusions about patients? Deductive inference from data can be valuable for clinical decision support (in spite of the fact that deductive methods only make explicit what is already there) because patient data are multimodal. Methods for multimodal fusion let different modalities be represented in one database (or knowledge base, or expert system component, or some other external explicit representation), allowing for new propositions to be deduced that could not be deduced from either of its constituents. Inferring unknowns from knowns, where the latter come in the form of data, is the general challenge for drawing useful inductive conclusions. To generalize from what has been observed in order to predict future data with a minimum of error is done through classification or regression. Precisely which results pattern matching methods can achieve has been mapped out in statistical learning theory [32].

Data are not the sole resource for machine learning, as in most clinical applications there is expert knowledge available on, e.g., diagnoses, treatments, substances, and prevention. In the ideal situation, all such knowledge is assembled into a knowledge graph, indicating how pieces of knowledge are interrelated. Even if knowledge elicitation can be costly in that clinical staff is a scarce resource, it is generally worth the effort, as the knowledge graph is so useful. Knowledge fusion therefore sometimes involves knowledge from experts being integrated into learning machines, e.g., for bootstrapping training by finding reasonable initial weights instead of just starting from random values.

A unique opportunity for testing learning machines recently arose at the Internet Psychiatry Clinic at Psykiatri Sydväst in Stockholm. Since 2008, the clinic has offered Internet-based Cognitive Behavioral Therapy (ICBT) for depression, social anxiety, and panic disorder with documented significant treatment effects [30]. The clinic would in the future like to predict, as early as possible, if the current treatment will have a positive effect on the primary problems the treatment aims to reduce, e.g., on symptoms of social anxiety. Over time, the goal of introducing a learning machine for ICBT decision support will become an important part of an adaptive treatment strategy that has already been tested for one of the treatments [11], but could be further improved.

At the clinic, 10-week ICBT interventions with diagnose-specific content based on established CBT techniques are divided into step-wise modules, with therapist support chiefly via asynchronous messages on a secure platform. The therapists normally see their patients face to face neither during treatment nor during intake and follow-up. The ICBT platform collects large amounts of mostly self-reported data for each patient before, during, and after treatment. Both established self-reporting questionnaires and more anamnestic and open-ended questions are used. Patients are largely self-referred and apply for ICBT via a public eHealth platform. A screening is then completed, including standard symptom scales for depression, social anxiety, and panic disorder, which also constitutes the primary outcome for each respective diagnose-specific ICBT program. Also included are scales for insomnia, general quality of life and functioning, use of health care, work ability, sick leave, and domestic functioning, problematic use of alcohol and drugs, and ADHD symptoms. Many of these measurements are then repeated at treatment start as well as post-treatment. Structured data from the psychiatric assessment are summarized, registered, and quantified in the platform together with some administrative data. During treatment, weekly questionnaires of primary symptoms, depression, and suicidal ideation are filled out, and data from interactive worksheets, patients’ reports on their use of therapeutic techniques, and messages between therapist and patient are collected. Moreover, data on the patients’ behavior within the platform, for example number of logins and number of clicks, and accessed treatment modules, are also recorded for possible future use.

Covering relevant parts of the clinic, a conceptual model in the form of an entity–relationship (ER) diagram [10] was built to represent the ICBT information flow (Fig. 1). This abstract model describes all events (e.g., a self-evaluation questionnaire being submitted) that result in bursts of information being added, and it thus constitutes a dynamic model useful to suggesting types of data susceptible to analytics [5]. The model has been consolidated with ICBT experts and serves as an ontological frame for discussing data and processes in a structured and unambiguous way. It constitutes an important step toward a finished knowledge graph for the clinic.

Machine learning has only recently been introduced in psychiatric research, for example in suicidality prediction, using fMRI to distinguishing CBT responders from non-responders, and measuring brain resting-state network connectivity to predict responders in electroconvulsive therapy [23]. Automated analyses of free speech have been able to predict later development of psychosis [2] and to estimate long-term severity of depression [16]. Another study used mostly patient self-reports to identify those likely to respond to a specific anti-depressant and validated the results in an external sample [9]. Machine learning methods can thus successfully learn to predict prospective events that are clinically relevant, and this can enable the identification of patients who are likely to respond to a particularly well-standardized intervention. The predictive power depends on the number of available predictors and their level of independence, data quality, sophistication of the learning algorithms, and how well they match the nature of data. In the design of a learning machine for ICBT, each classifier and each component of predictive analytics that is deemed relevant to test by the domain experts is a potentially valuable input signal.

The learning parameter space and the combinatorial space of processing multiple input signals in order to produce a maximally useful output signal are both hard to characterize mathematically, making empirical exploratory work important as a complement to formal arguments. Grounded in a clinical application, such progress can be slow and costly, as improving clinical outcome and counting costs produce uncertain results [7]. Learning machines that perform well on a task do so in part because the domain-specific bias obtained from performing earlier tasks has been coded efficiently. In the case analyzed here, the relatively large amount of patients will allow a learning machine to (with some still unknown rate of success) solve the meta-level learning problem [24]. The learning machine learns features with the help of expert bias (cf. [1]), and the general idea used to fuse and unify knowledge here was first proposed by Jacobs and Hinton in 1988 to overcome problems with interference between subtasks when using backpropagation [15, 79]:

If one knows in advance that a set of training cases may be naturally divided into subsets that correspond to distinct subtasks, interference can be reduced by using a system composed of several different “expert” networks plus a gating network that decides which of the experts should be used for each training case. ...The idea behind such a system is that the gating network allocates a new case to one or a few experts, and, if the output is incorrect, the weight changes are localized to these experts (and the gating network).

In the perception–reasoning–interaction loop, the machine will store perceived input signals and recall them as part of its reasoning. The directly visible actions of the machine are restricted to its interaction; answering a specified set of questions, such as Will this patient receive enough symptom reduction?, sending out private messages to the therapist, and similar. In the future, an example of a question-driven activity (pertaining to a future in which the adaptive behavior platform is in place in full) would be a therapist asking the machine the equivalent of the human to human question Will the patient 2535 complete the planned exposure? An example of autonomous reasoning leading to a message being sent from the learning machine to a therapist would be Please check status of Patient 2535 with respect to possible dropout.

Previous research done at the Internet Psychiatry Clinic showed that simple regression using change scores on patients’ primary symptom measure 4 weeks into treatment can explain between 34 and 43% of the variance in outcome at the end of treatment [26]. This indicates that moving away from baseline only predictions to instead monitor progress and predict outcomes after treatment has started can yield large information gains. Inspired by those findings, and by previous research on continuous monitoring of psychological treatment [18], the concept of an adaptive treatment strategy has been tested in a randomized controlled trial at the clinic [11]. In this trial, patients undergoing ICBT for insomnia were classified as at risk of failure (i.e., not benefiting from treatment) or not 4 weeks into treatment. The classifier was a multi-step algorithm using patient rating as well as clinician ratings in a simple spreadsheet that made calculations based on coefficients from previous predictor studies and rules based on clinical experience and hunches put in by the researchers (i.e., a rather rudimentary procedure from a statistical and computational point of view). Those classified as at risk were then randomized either to continue treatment as normal or to get extra attention and support from their therapist. Out of those not classified as at risk, 23% ended treatment with a poor outcome, whereas 64% of those classified as at risk who did not get extra help ended treatment with a poor outcome, indicating a clinically meaningful accuracy of the classifier. Those classified as at risk who did get extra help only had poor outcomes in 37% of cases, indicating that the predicted failures can be avoided in many cases. The trial by Forsell et al. [11] shows that waiting a few weeks into treatment and then making even a relatively simple prediction can have a large clinical impact if that information is clearly stated and then acted upon by the therapist. These studies demonstrate the potential of predictions using data from both baseline and the early weeks in treatment. However, apart from less than perfect predictions, Schibbye et al. [26] leave a lot of available information unused, and Forsell et al. use an algorithm that is partially manual and often requires input and effort from the therapist. Therefore, there is still room for improvement that could potentially be partially covered with a learning machine.

A finite number of different machine learning methods are employed for basic tasks of classification and prediction, and the learning machine is thus using ensemble learning [22] for efficient fusion and unification of the individual methods. To classify patients into those that responded well to the program and those that did not, for instance, a score function from the data into a set of expert classifications and assessments is defined. The function is typically parameterized and linear, and its value range is sometimes referred to as the ground truth since it is against these labels any machine learning method will be validated initially. A loss function that measures the discrepancy between this ground truth and the predictions given the trained parameter set is defined next. Because the loss function is defined over a high-dimensional space, even a linear classifier weight matrix will have tens of thousands of elements. To explore this feature space, it is possible to fix points (e.g., via a random weight assignment) and then travel along lines (dim 1) or planes (dim 2), computing loss along the way. The final step is to optimize by minimizing the loss. The overarching representation in the here presented model will be a distributed probabilistic continuous neural network with iterative parameter tuning [3]. The methods for basic tasks are thus not one time only, but classification and prediction are done repeatedly over time, as new data are made available to the learning machine. As its score function is not linear, a model that can handle nonlinearity must be chosen, e.g., a recurrent convolutional neural network.

Since most of the data can be represented as time series, a finite mixture model [20] is also employed to detect various kinds of latent behavior change in the patients over the course of the ICBT program. In historical data, a hidden patient behavior pattern is assumed to exist. A finite automaton describes the different health or intervention states a patient may be in, observed as well as latent states, and the states can be identified with the help of machine learning methods (e.g., an HMM, see [25]). The transition probabilities between states are in the model computed for each patient. This individualized approach still allows for reasoning about classes of patients, formed, e.g., from mapping behaviors onto class membership. An example class label is Patients in the depression program under 30 years of age that only reply to therapists messages late at night. Another example is comorbidity: Patients in the depression program that suffer from irritable bowel syndrome.

The model is naïve in the sense that it is defined to capture all relevant properties of the data. One way of explaining why it needs to be systemic in order to learn how to learn is to say that it codes for a family of models that all have to be able to grow in complexity and predictive power as the data grow. Therefore, a prior distribution must be developed that can encompass this family, producing a nonparametric model [12]. This Bayesian approach to modeling will employ distributed representations, just like in the networks employed in the model, since these are suitable for overlapping population clusters like the patient classes just mentioned. The clusters themselves are inductively inferred from the mixture model representation, and will form a hierarchy (cf., e.g., [28]). What becomes known about other patients in the same classes can then benefit an individual patient. The model is generative in the sense that each set of individual patient data is a random mixture over latent patient class descriptions, similar to parameterized topic models based on latent Dirichlet allocation [4].