Long-Term Effects of a Social Intervention for Vulnerable Youth: A Register-Based Study

The number of children with conduct disorder problems is a growing concern in many countries. Early life problems can settle into permanent life problems with huge individual and societal costs. Young people with conduct disorder problems are likely to perform poorly in school and often face difficulties pursuing further education or employment. They often face a risk of entering a life path involving criminal activity, drug abuse, and mental health problems. The evidence on effective interventions for disadvantaged adolescents is limited, as stressed by, for example, Nobel Laureate James Heckman and co-authors (Heckman et al. 2014). Some of these young people are placed outside their homes, in foster care, although little is known about the effectiveness of foster care (e.g., Doyle 2007; Frederiksen 2012).

Multisystemic therapy (MST) is an intervention developed in the United States (US) in the 1990s by psychologist Scott Henggeler and colleagues (e.g., Henggeler et al. 1992), with a focus on 12- to 17-year-old violent juvenile offenders. It is an intensive family-based intervention based on a systemic approach: the therapist meets the child in the family home or in other familiar surroundings and is available on call 24 h a day, 7 days a week. MST has been evaluated in 55 published studies based on 25 randomized controlled trials and has been adopted in more than 34 US states and 15 countries worldwide (MST Services 2017). While this is a strong evidence base, meta-analysis shows great effect heterogeneity, especially across countries. MST has been implemented in Denmark since 2003, but to date there have only been two evaluations of the Danish program. The two evaluations applied a before–after design (Greve 2006; Greve and Thastum 2008), but no effect measurement was conducted.

Because the Danish implementers of the MST intervention have gathered social security numbers for all the participants, information from various public administrative registers can be linked by Statistics Denmark.

This chapter shows how the administrative register data can be used to construct effect measurements of MST in the Danish setting. It is based on a quasi-experimental effect design that utilizes access to data for the entire Danish youth population and their parents, from the time of the children’s birth. By using such register data, some of the pitfalls in the existing studies can be avoided: most are based on small samples and have self-reported outcomes, measured over short time horizons. This creates a risk of self-evaluation bias and attrition bias, which can be avoided in an analysis based on administrative register data.

The chapter is organized as follows. In the next section, previous findings on the effect of MST are summarized. This is followed in Sect. 3 by a description of the data, and a description of the constructed control group is provided in Sect. 4. Section 5 presents the effect estimates. In Sect. 6 the findings are summarized, and the section concludes with a further discussion of the benefits of using register data for impact measurements.

MST was developed in the late 1980s and 1990s by psychologist Scott Henggeler and colleagues at the Family Services Research Centre at University of South Carolina. More than 55 studies have been published on the effect and implementation of MST for different subpopulations of youth.

Many of these studies have been evaluated using a randomized controlled trial. Most of these have had a limited number of participants, from 16 (Borduin et al. 1990) to 176 (Borduin et al. 1995), and with Glisson et al. (2010) being an exception, with 615 observations.

Many of the studies have shown promising results on young people’s behavior, family relations, and, for example, relapse into crime for juvenile offenders (see, e.g., Curtis et al. 2004). However, a Cochrane review from 2005 concluded that the evidence was not strong enough to make final recommendations for the use of MST in preference to alternative interventions (Littell et al. 2005). However, this review was based on only eight randomized controlled trials, and it should be stressed that no harmful effects of MST compared with alternative interventions were found and that the overall effect was positive, but not significantly different from zero. Three possible reasons for the difference in results when compared with previous reviews (e.g., Littell 2005) are highlighted in the Cochrane review: (1) previous reviews were narrative rather than systematic; (2) the Cochrane review excluded unpublished studies; and (3) the Cochrane review did not include studies not using the intention-to-treat principle. Two Dutch studies using administrative data point toward the aforementioned self-reporting bias as another potential explanation, finding positive effects for parent- and youth-reported problem behavior but not for criminal convictions (Asscher et al. 2013, 2014). Finally, there is a large amount of effect heterogeneity across countries and across different comparison groups. In some studies, MST is compared against another specific intervention, while in others it is compared against treatment as usual. This could in itself explain the divergent results.

The most recent review of the effects of MST includes 22 studies involving 4066 young people. This study finds that, after correcting for publication bias, there is no effect of MST on specific delinquency but a small, significant, and positive effect on general delinquency, psychopathology (e.g., externalizing and internalizing behavior), and family factors (e.g., self-reported family functioning, parenting skills, or parental health). The study confirms the finding of great effect heterogeneity, e.g., showing that the effect on delinquency is larger if MST is compared with a single alternative intervention (as opposed to multiple interventions) and if it is conducted in the United States. It also shows that, although study quality matters, the use of randomized versus quasi-experimental designs did not affect the results. The fact that the results of good quasi-experimental designs do not differ from those of randomized controlled trials has been reported elsewhere (e.g., Card et al. (2015) in the case of active labor market interventions), but of course this may differ from case to case (e.g., a famous study from economics by Lalonde (1986) and elaborations on his findings, e.g., by Dehejia and Wahba (1999)).

It is worth mentioning specifically the results from Norway and Sweden because the implementation setting in these countries is likely to most closely resemble the Danish implementation setting. Both were evaluated using a randomized controlled trial. While the Norwegian studies found that MST improves participants’ internalizing symptoms and decreases out-of-home placements (Ogden and Halliday-Boykins 2004), the Swedish experience found no effect of MST when compared with standard alternative social interventions (Sundell et al. 2008).

The results from the reviews—reinforced by the divergent Scandinavian findings—show the need for case-by-case evaluation, which may well be conducted using quasi-experimental designs, at the very least as a cheap first-hand examination.

The present analysis is based on information from Danish administrative register data. The relevant group of children were all observed from birth, and, furthermore, one can link information on parents to information on their children. The data come from various registers on education, health, crime, drug abuse treatment, labor market attachment, and the use of social measures. All the data were collected for administrative purposes, e.g., tax information in employment and health-care use, or for purposes of tax-financed reimbursement of health-care providers. The data are therefore considered to be highly reliable. As all Danish citizens have a unique identifier (CPR number), information from various public registers can be linked by Statistics Denmark.

The effect analysis is based on data on 436 participants who received MST at some point during the years 2007–2011. The participants were located in three municipalities (Herning, Aalborg, and Aarhus). At the time of entering MST treatment, the individuals were aged 12–17 years, and, since data are available up until 2013, the early participants can be followed up in the register data to the age of 22.

Table 1 shows the distribution of MST participation by age and the year of entering MST treatment. The majority of the participants entered the program around the age of 14–16 years, and there is a decline in the use of MST over the period.

Table 2 presents means of selected characteristics for MST participants and all other vulnerable young people aged 12–17 years in 2007–2011. “Other vulnerable young people” are defined as young people aged 12–17 receiving alternative social measures. The social measures under consideration are out-of-home placement (institution care or foster care) and preventive measures, such as a steady contact person. These measures are all offered with reference to the Danish Act on Social Services. The act implies both that vulnerable young people have a right to receive such measures and that local municipalities must pay for them.

The group means in Table 2 are given prior to any correction for background characteristics. The table shows that the two groups (MST participants and other vulnerable young people) are quite similar in terms of, for example, family background, parental characteristics, and health treatment. For instance, 13% of the MST participants have fathers with a criminal conviction. This is true for 12% of other vulnerable young people. It is also seen that a very large share of young people in both groups have parents who were in psychiatric treatment or hospitalized in the year before they received MST or an alternative social treatment. In contrast to these results, Table 2 also shows that the share of children and young people that were placed out of the home in the years before the comparison year was substantially higher for the group of other vulnerable young people than for the MST participants. These patterns of similarity between the groups with respect to health-care usage and parental characteristics but divergence with respect to previous social interventions—are also found when looking further back in time during the children’s lives. Figure 1 shows, as an example, the similarity in father’s outcome for MST participants and other vulnerable young people, averaged over the years when the child was aged 0–5 years and when the child was aged 6–11 years. In contrast to this result, there is a marked difference between MST participants and other vulnerable young people with respect to the share who have previously received social treatment. This is shown in Fig. 2.

In a similar manner, the paths for some of the outcome variables of the analysis are compared in Fig. 3. Figure 3a–c shows the share of young people that was convicted in a given year. The conviction rates are shown both before and after the year of treatment. In the pretreatment period, the shares were close to zero for both groups, which is explained by the fact that the age of criminal responsibility in Denmark in the period of observation was 15 years. For all types of convictions, the rates increased after the treatment year, particularly for the MST participants. The gaps between MST participants and other vulnerable young people are as high as 10–15 percentage points. Hence, these descriptive measures do not indicate that MST has a crime prevention impact. It is, however, stressed that these numbers have not yet been corrected for differences in background characteristics between MST participants and other vulnerable young people and hence should not be interpreted causally. The corrected estimates are shown in the next section.

Figure 3d presents the share of young people sentenced to imprisonment. The difference between the two groups is lower and shifts over time. As imprisonment is a sentence reserved for more severe crimes, this indicates that the higher rates of convictions for MST participants can primarily be attributed to less severe types of crime.

Evaluating an intervention such as MST is a challenge when the participants are not randomly selected from among the group of vulnerable young people. To handle the nonrandom selection, propensity score matching is applied to evaluate the effects of MST. This procedure matches each MST participant to a number of other vulnerable young people with a propensity to be treated with MST similar to that of the actual participants.

The MST participants were located in three municipalities (Herning, Aalborg, and Aarhus), and in this study, these participants are matched with similar young people in the same age group, who were living in other municipalities and receiving social measures. The matching is performed using the rich Danish register information on their childhood, health, family situation, and parents, observed every year from birth until the age at MST intervention.

As seen in the preceding section, the MST participants and other vulnerable young people are very similar, even without matching. What seems to be a major difference between the two groups is the extent to which they received social measures prior to the age at which they are compared. Prior social care is thus a key observable characteristic for which MST participants need to be matched with other vulnerable young people. Prior social care is also likely to be related to prior occurrence of conduct disorder and family problems. The design is illustrated in Fig. 4, highlighting that both the MST participants and their comparison groups are identified as vulnerable young people and therefore to a large extent similar at the outset. This study measures outcomes in the years after treatment and is able to follow MST participants and their comparison group for up to 5 years. Therefore, those who are MST participants at the age of 12 are tracked until the age of 17, and likewise those who are MST participants at the age of 17 are tracked until the age of 22. The follow-up period also depends upon the nature of the outcome, because education, crime, and employment outcomes are relevant at different ages.

The matching procedure includes register information on a very long list of variables including age, sex, family type, health-care history measures for children and their parents, criminal records, psychiatric treatment, drug abuse treatment, education of children’s parents, and previous use of social measures.

Matching on this very rich set of characteristics of the child’s past—including detailed annual lagged information on the social intervention history—enables a reliable comparison group to be identified for the effect evaluation (Lechner and Wunsch 2013).¹

This study estimates effects of MST on outcomes measured from both short- and long-term perspectives. The outcomes include secondary education, youth education, employment, welfare dependency, criminal convictions, and use of social measures after MST treatment. All estimates are obtained by the use of matched controlled groups, as described in the previous section.

This chapter has presented an example of how to use administrative register data to obtain new knowledge about the effectiveness of social interventions. It has examined the case of a small-scale social intervention—multisystemic treatment (MST)—which is targeted toward vulnerable young people and which has been applied to young people with conduct disorder problems and juvenile offenders. The intervention has been evaluated numerous times in various countries using randomized controlled trials but often with highly divergent results across target groups and across countries.

MST has been used in Denmark for more than 13 years, but its effect there has never been evaluated. The gold standard for effect evaluation is a randomized controlled trial. However, randomized controlled trials are expensive and are often difficult to conduct because of resistance from social workers and decision-makers when it comes to social interventions. But randomized controlled trials are not without other limitations, both with respect to potential contamination of control and treatment groups and because they are often based on small-scale interventions that are sometimes difficult to extrapolate to real-world settings. They are often also dependent upon survey measurements of outcomes, with problems of attrition over time and measurement error or self-reporting biases.

This is not to say that randomized controlled trials are not worth pursuing, but alternatives might be worth pursuing in difficult cases. Administrative register data are a hugely valuable source of information which, applied in the right way, can provide new insights at relatively low costs. This chapter has shown how to obtain effect measures, 13 years after the first use of MST in Denmark. The approach benefited from access to life-cycle data on children and their parents and the opportunity to track children and young people over time with very limited attrition and measurement error. The possibility that the estimates may have biases cannot be excluded, but it seems very unlikely that the biases are large enough to remove the negative effects documented.