Identifying the determinants of lapse rates in life insurance: an automated Lasso approach

The data set comprises a set of insurance contracts written over a period of roughly 12 years which are observed for almost 10 additional run-off years (with no new contracts written after the first 12 years). Note that the proposed methodology can also be applied to situations where the portfolio is not in run-off. The data set is well structured and without missing values. There are \(n = 501,251\) observations (104,555 unique contracts) with \(J = 13\) covariates: contract duration (number of years between inception and observation time), insurance type (traditional or unit-linked), country (four European countries), gender, payment frequency (e.g. monthly or yearly), payment method (e.g. debit advice or depositor), nationality (whether or not the country, in which the insurance was sold equals the nationality of the policyholder), dynamic premium increase percentage,¹ entry age, original term of the contract, premium payment duration, sum insured and yearly premium. Note that the original data set contains even more covariates, but many of them are omitted because they do not contain predictive information (e.g. name of the insured person).

In order to automatically model structures and complex trends with the Lasso, one pre-processing step appears useful. In this step, we transform the continuous covariates into discrete covariates by partitioning them into a finite set of classes (bins). We may choose the bins based on expert knowledge, in order to get reasonable intervals, or based on quantiles. It turns out that a fully data driven model performs best, where the bins are based on a univariate decision tree. Each decision tree is grown with the restriction that the number of observations in a terminal leaf should be at least 5% of the overall data set. Hence we implicitly set the maximal number of bins for each covariate to 20. Obviously, deeper trees increase the number of category levels rapidly and thus are prone to overfitting. Hence, this seems to be a reasonable and robust restriction. With this transformation of the continuous covariates, we can model more complex trends, rather than modelling just the overall trend by estimating a single parameter for those numeric covariates. This pre-processing step implies that the subsequent model calibration takes less than 10 minutes on a standard computer.

Note that the data set (and other typical data sets in the context of lapse analysis) contains information over several observation years. In each observation year, every contract may lapse. For modelling purposes, a separate row is created for each in force contract in each observation year. As a consequence, one single contract may occur in several rows in the data set—once for each observation year where the contract is still in force. Note that by this modification, the rows of the data set are no longer fully independent. It also causes a selection bias, since contracts who lapse early on end up having less observations in the data set. Given the size of the data set, this seems acceptable. A thorough analysis of the impact of this assumption seems advisable.

Unlike other papers, e.g. Eling and Kiesenbauer [10], we do not consider the calendar year as a covariate. Even though it may have an impact on lapse rates in specific situations (e.g. higher lapse rates during a financial crisis), it cannot be used as a covariate for future lapse rates. Therefore, for risk management, we explicitly exclude the calendar year from the model and focus on the contract duration to capture the development over time. However, to quantify the calendar year effect, we include the calendar year covariate and other macroeconomic covariates ((AMECO swap rate [3], ECB inflation rate [8] and ECB Eurostoxx performance [7]) in a sensitivity analysis in Sect. 4.

Overall, this analysis requires only limited pre-processing of the data set, since the Lasso itself performs the main variable selection.

In practical business applications, it appears to be common practice to apply univariate smoothing algorithm for modelling biometric assumptions, e.g. the so-called Whittaker–Henderson approach. It was derived around 100 years ago, as described in Joseph [16]. Applications of this smoothing technique are manifold. For example, the German actuarial society (DAV) uses Whittaker–Henderson to smooth the crude death rates to derive the mortality rates (DAV2008T, without security loadings) used in most German life insurance companies. Another straight-forward and commonly used application of Whittaker–Henderson is the modelling and smoothing of lapse rates over time, see for example SAV [28]. Here, the raw lapse rates for the different values of the covariate contract duration are used to fit a smooth curve, resulting in estimates for the lapse rate depending on the contract duration. The upper panel of Fig. 1 shows the smoothed lapse rates for the analysed insurance portfolio. The bars show the exposure (right y-axis), and the points (connected with the dashed line) show the observed lapse rates for each contract duration (left y-axis). There is a decreasing trend in lapse rates, meaning that more contracts are terminated at the beginning of the contract, which appears intuitive. The blue line is the result of the Whittaker–Henderson smoothing approach. At first glance, the approach works quite well, since it captures the main trend, does not seem to overfit, and is very simple.

However, the Whittaker–Henderson approach is a univariate approach and therefore can only capture the effect of one covariate on lapse rates. Hence, it does not predict lapse rates on an individual policy level, as simply all insured with identical contract duration get equal predictions. To illustrate this, we add the country as second covariate. The lower panel of Fig. 1 shows the result of smoothing the covariate contract duration with the Whittaker–Henderson approach (blue line) visualised for two covariates (contract duration and country). The points (connected with a dashed line) show the observed lapse rates for each contract duration for the different countries and the blue line—which has not changed—shows the result of the Whittaker–Henderson approach. Clearly, the countries show significantly different lapse rates—in both magnitude and shape. In order to model structures with several covariates, perhaps even on an individual policy level, sub-groups have to be analysed independently. In this example of adding the covariate country, we would have to build four individual Whittaker–Henderson curves for each of the sub-groups (countries).

First of all, these sub-portfolios are subjective and require expert judgement. Secondly, the approach still ignores all other covariates. The problem is not only ignoring some information, but also the risk of misinterpreting the information. One might see and interpret trends in the univariate perspective that actually come from other covariates which are not included. For instance, it is possible that the contracts in country four show higher lapse rates because they belong to a specific entry age group or insurance type that has a higher lapse risk. Finally, another problem in building sub-groups is that the segments may become very small and hence the number of observations falling in a specific segment is not sufficient to derive a reliable estimate.

The GLM models the expected value of a target variable Y, a random outcome according to a distribution of the exponential family, as a linear combination of the explanatory covariates \(X_i\):The link function g connects the linear combination of the explanatory covariates to the target variable—hereafter lapse. The choice of the link function must reflect possible entry values of the target variable, in this case lapse \((Y=1)\) or no lapse \((Y=0)\). This can be achieved with the logit link, \(g(x) = \log (x/(1-x))\), which is also the natural link function implied by the exponential family. Other link functions are possible, e.g. a probit link or a cloglog link function. However, the natural link function has some advantages, e.g. unbiased estimates, see Wuthrich [34]. The parameters of the model (\(\beta\)) can be estimated efficiently by optimising the likelihood, assuming a Binomial distribution for the target variable lapse. Maximising the likelihood is equivalent to minimising the negative of the log-likelihood:Based on this likelihood function, we can use the residual deviance as a performance measure. It is a generalisation of the mean squared error (MSE) for normally distributed data to other distributions. The residual deviance is two times the difference of the log likelihood of a fully saturated model that fits the data perfectly and the model under consideration, i.e. a measure for residuals for arbitrary distributions. Starting from a model that only contains an intercept (so-called intercept only model with the null deviance), one can analyse how much better a model is compared to other models and also compared to the intercept only model.

The calibration of a competitive GLM with best possible residual deviance requires further modifications of the covariates.² For instance, a GLM that models the covariate entry age with only one coefficient typically shows a poor performance (underfitting). The model can then only detect one general trend for that covariate but cannot incorporate the different effects on lapse for different entry age groups (marginals), say if the lapse rate is increasing up to a certain entry age and decreasing afterwards. Conversely, a GLM that determines an estimate for each entry age individually will presumably overfit, i.e. the performance on new test data will be considerably worse than on training data. Therefore, a covariate like entry age is typically divided into bins, in order to achieve a more accurate model using pre-specified entry age groups: \(\beta _{entry\,age} \cdot X_{entry\,age} = \beta ^1_{entry\,age} \cdot X^{1}_{entry\,age} +\cdots + \beta ^{p_{entry\,age}}_{entry\,age} \cdot X^{p_{entry\,age}}_{entry\,age}\), where \(p_j\) is the number of categories for covariate j, e.g. the number of entry age intervals for the covariate entry age. Here, \(X_{entry\,age}^j = 1\) if the entry age of an observation lies within the j-th entry age interval and zero otherwise. Alternatively, Generalised Additive Models (GAM) can be used to incorporate functional transformations using splines or polynomials of the covariates.

However, in both approaches the selection of the number of entry age groups, or the number of splines/polynomials is crucial and needs to be specified priorly. Without deeper knowledge of the structures of the marginals, it is hard to identify the ‘correct’ number and comes along with a considerable effort and model risk. In a multivariate model, the ‘correct’ number of groups also depends on the other covariates in the model. Therefore, the process of identifying a good and robust multivariate model is even more complex and only possible with a high effort by iteratively refitting and readjusting the GLM. Hereafter, we show how an extension of the GLM can solve this problem. We will see that the predictions of our lapse model remain fully explainable, and the coefficients can be used to interpret the lapse rate on an individual contract level, as in a standard GLM. Note that most other machine learning techniques lack one or even more of these properties. Often, higher accuracy in forecasts comes along with a loss of interpretability or explainability. For many applications in the insurance sector, however, the ability to explain and interpret the models outcome is of decisive importance, e.g. in risk modelling or pricing.

One of the major challenges in training a competitive model is to avoid overfitting. Most popular in statistical learning is the use of cross-validation, where several models are built and compared with respect to their accuracy in order to choose the best model among them. The k-fold cross-validation (see James et al. [15]) splits the data set randomly in k distinct groups, where \(k=5\) or \(k=10\) are typical choices. Afterwards, each model is built on \(k-1\) data sets (training data) and validated on the remaining data set (validation data), by using the deviance of the model on the validation data set. This step can be repeated k times, and each time, a different group is treated as the validation data set. For each model, the k-fold cross-validation gives k estimates of a performance measure on a changing validation data set. Then one can decide which configuration of the model performs best on average.

In addition to the optimum, the k-fold cross-validation gives an estimate for the variation of the performance measure. Similar to the optimum, this can be estimated directly from the k-fold repetitions. When choosing a model, the optimum is clearly a plausible choice. But sometimes it can also be advantageous to choose a model that is less complex, e.g. a GLM with fewer coefficients. As long as this model is still within the expected variation of the performance measure, this is also statistically justified. A frequent choice here is the one-standard-error (1-SE) rule, so choosing the least complex model which is still within one standard error from the optimum. A model with fewer coefficients and a similar performance may be preferable, since the model is likely to be more robust and to recognise the patterns in the data well without overfitting. The selection of a robust model also implies that a similar performance can be expected on new data.

In contrast to the beginnings of statistical learning applications, nowadays there is usually a lot of data available. As a result, it can often happen that, through the application of cross-validation alone, models are selected that still have a large number of parameters, e.g. coefficients in the GLM. In addition to a good cross-validation performance it is therefore often important to select a parsimonious model with fewer parameters.

The Lasso is an extension of the GLM using regularisation and is able to handle overfitting and generating a parsimonious model simultaneously. Here, the number of parameters is reduced by a penalty term and a more robust and stable model is developed through the parameter estimation itself.

Introduced by Tibshirani [29], the least absolute shrinkage and selection operator (Lasso) is a GLM with regularisation. Lasso adjusts the maximum likelihood estimation of the GLM by adding a penalty term. The optimisation function extends to:We use the index \(L_j\) to specify different version of the Lasso for each covariate. R stands for regular Lasso where the penalty term is defined as:Note that this penalty term equals the \(L_1\) norm of the coefficient vector \(\beta _j\). Other norms are also possible, e.g. the \(L_2\) norm, which corresponds to the Ridge Regression, see Hoerl et al. [14]. Optimising Eq. (2) will result in a similar GLM as in Eq. (1) but with different values for \(\beta\). The Lasso has advantageous properties: For some values of \(\lambda >0\), selected coefficients will be set to zero, and thus they will be effectively excluded from the model. Therefore, Lasso reduces the complexity of a model systematically. Here, the regularisation shrinks the \(\beta _j\) of different covariates simultaneously towards zero, as the left panel of Fig. 2 shows for two covariates with parameters \(\beta _1\) and \(\beta _2\). The \(L_1\) norm as a penalty term implies that the optimum for different values of \(\lambda\) (visualised by the ellipses) intersects the penalty term (diamond) where some of the coefficients are zero, here \(\beta _1\). The amount of shrinkage is controlled by the tuning parameter \(\lambda\). Set to zero, the resulting GLMs of Eqs. (1) and (2) coincide. Set very high, the resulting GLM of Eq. (2) will be an intercept only model, see right panel of Fig. 2. The specification of a parameter \(\lambda\) that produces a robust GLM in-between those extremes can be done with cross-validation. Note that compared to cross-validating several different GLMs with various parameters, the Lasso requires a cross-validation for only one tuning parameter. Cross-validation can show which values of the tuning parameter \(\lambda\) perform well, measured by the residual deviance of the model. The 1-SE rule gives the range of \(\lambda\) where the performance is within one standard error of the cross-validated optimum. Within this range, higher values of \(\lambda\) shrink even more coefficients to zero and produce easier and probably less complex models without a significant loss of performance. In what follows, we will use a 5-fold cross-validation with this 1-SE rule to determine the tuning parameter \(\lambda\).

The coefficients in the so-called fused Lasso, see Devriendt et al. [6] and Tibshirani et al. [33], do not reflect the difference of a category level to the intercept (upper panel of Fig. 3) but to the adjacent category level, cf. middle panel of Fig. 3. Without regularisation, this formulation results in identical predictions. After regularisation, however, some of the \(\beta _j\) will again be zero, thus adjacent category levels, e.g. entry age groups are fused together forming larger entry age groups. This means that the originally very time-consuming task of identifying the ‘correct’ number of entry age groups can be replaced by an automated and fully data-driven process.

The number of observations in the category levels has an important impact on the estimation. Since category levels with many observations have more weight in the (penalised) maximum likelihood estimator, the model will particularly focus on these areas. Adjacent category levels with fewer observations are combined more frequently by the fused Lasso and receive a common estimator. Hence, the fused Lasso is also well suited for smoothing the oscillating edges of covariates with fewer observations which is often a challenge in practical applications.

The fused Lasso \((L_{j}= F)\) applies the \(L_1\)-penalty to the differences of subsequent coefficients:Similar to the notation in Tibshirani [31], the \(p_j \times p_j\) dimensional transformation matrix \(D^{(1)}\) can be expressed as³:Note that the coefficient \(\beta _{j,1}\) (corresponds to the first row of the matrix) is typically not penalised, since it is part of the intercept. For the fused Lasso, the choice of the category level which serves as the intercept does not change the model results.

An equivalent formulation uses a contrast matrix, which is defined as the inverse of the transformation matrix:Using the fused Lasso penalty for covariate j then leads to the transformation of the design matrix \(X_j^{new} = X_j C^{(1)}\).

In practical applications, there are often covariates that show a monotonically increasing or decreasing behaviour with respect to the target variable. Fusing adjacent category levels together can be insufficient to model this monotone behaviour precisely. In this case, common actuarial practice is to use GAMs, i.e. to model these effects with splines or polynomials. As mentioned above, this comes along—again—with the crucial and complex choice of the ‘correct’ degree of polynomials or the knots of splines. The trend filtering, see Kim et al. [19], was developed for these applications and is another extension of the Lasso. It works similar as the fused Lasso, but also allows for non-zero slopes. So rather than penalising the change between adjacent levels, this approach penalises the change of the slope between category levels, cf. lower panel of Fig. 3. Hereby, changing linear trends and monotone structures within a covariate can be identified efficiently. The trend penalty can be expressed as:In the formula, \(|\beta _{j,1}|\) is treated as the intercept (which is not penalised), while \(|\beta _{j,2}-2\beta _{j,1}|\) penalises the change in trend compared to the initial trend (with slope 0)—again see lower panel of Fig. 3. The intercept is rather arbitrary and could also be set to any other category level of the covariate, e.g. the middle category level or the last category level. Since the term \(|\beta _{j,2}-2\beta _{j,1}|\) is penalised, the choice of the initial trend does affect the result of the model.

Here, the \(p_j \times p_j\) dimensional transformation matrix \(D^{(2)}\) can be expressed as:In terms of the contrast matrix, we can write:Using the trend filtering penalty for covariate j then leads to the transformation of the design matrix \(X_j^{new} = X_j C^{(2)}\).

We could also consider other functions in a similar setting. Note that the transformation matrix for higher orders can be defined recursively, see Tibshirani [31], i.e. \(D^{(k)} = D^{(k-1)} D^{(1)}\). Thus, also higher orders can be included straightforwardly, e.g. with cubic structures. Again, the time-consuming task of identifying the ‘correct’ degree or knots can be replaced adequately by an automated and fully data-driven process.

Multivariate models like a GLM can jointly assess the impact of covariates on the target variable. This allows to identify the factors triggering a high or low estimate of the target variable. Sometimes, the influence of a covariate on the target variable also depends on the value of another covariate. In that case, common actuarial practice is to include an interaction term in the model. Including an interaction between covariate \(X_i\) and \(X_j\), the GLM formula extends to:By this definition, the model gets \(p_i \cdot p_j\) additional coefficients. Through the interaction of \(X_i\) with \(X_j\), further structures can now be included in the model. For example, if adjacent category levels of \(X_i\) show differences depending on their value of \(X_j\). E.g. for certain values of \(X_j\) the observed lapse rates may be increasing in \(X_i\), whereas for other values of \(X_j\) the observed lapse rates may be decreasing in \(X_i\). This can be reflected in the model with an interaction term.

Again, the choice of which interaction to include in a model is crucial and comes along with a considerable effort. Here, especially overfitting is a potential risk, as interactions incorporate a quadratic number of additional coefficients in the model. The Lasso proves to be a helpful tool for including only the most important coefficients (here interactions) in the model. For example, we may apply the regular Lasso to these coefficients, i.e.where \(p_i\) (\(p_j\)) is the number of different category levels of covariate \(X_i\) (\(X_j\)). After regularisation, only the most important coefficients remain in the model. This can significantly reduce the number of coefficients (here interactions) in the model. Yuan and Lin [37] apply a grouped Lasso, where all or no coefficient of a covariate remain in the model. For our data set, however, the grouped Lasso leads to overfitting as too many parameters remain in the model.

Another example of penalising the interaction term is in case of possible levels or trends in \(X_i\) or \(X_j\). We can then apply a fused Lasso or a trend filtering approach for one direction (or both). This is illustrated in Fig. 4 for an interaction of \(X_i\) and \(X_j\). Say that in the data, \(X_j\) is not ordinal scaled and hence the regular Lasso is used for that direction (y-direction). Hence, we compare each coefficient with the intercept—visualised by the green arrows. \(X_i\) however is ordinal scaled and for the interaction term, a fused Lasso penalty is chosen for that direction (x-direction). Hence, we compare each coefficient with its subsequent coefficient—visualised by the red arrows.

Similar to Eq. (4), we can formalise the penalty term for this example:Note that for this example, we assumed that \(\beta _{i,j,1,1}\) is included in the intercept and is therefore not penalised.

Fusing in both directions \(X_i\) and \(X_j\) leads to the 2d-fused Lasso, see again Devriendt et al. [6] and Tibshirani et al. [33]. This approach can be generalised using an arbitrary graph indicating which two points in the plane should be considered for the fusing. One example is given in Tibshirani et al. [33], where they use a graph with the US (mainland) states as nodes and edges whenever two states share a border. One difficulty, however, is that in general, the number of edges is greater than the number of nodes. Consequently, we need additional second-order conditions to obtain a unique optimum. In order to avoid this case and the more complex optimisation associated with it, we have chosen a regular Lasso for one dimension in the numerical example in the next section.

The implementation uses the R [26] interface for h2o, see LeDell et al. [20]. Note that the R package smurf, see Reynkens et al. [27], which is based on the analysis of Devriendt et al. [6] already introduces different regularisation types within a Lasso setting (SMURF algorithm). However, it does not contain the trend filtering, so that we ended up developing a new implementation within the h2o framework.

For covariates with fused Lasso or trend filtering penalty, we use the contrast matrices \(C^{(1)}\) (fused Lasso) and \(C^{(2)}\) (trend filtering) to adjust the structure in the design matrix, as described in Sect. 3. To be more precise: We apply the contrast matrices to the original design matrix X using the contrast argument in the R built-in model.matrix function. By that, the corresponding columns for each covariate are changed accordingly. This is equivalent to:where \(C_i\) is the \(p_i \times p_i\) contrast matrix for the i-th covariate and 0 is a matrix of zeros of corresponding size.

The h2o command h2o.glm for building the actual model is then straightforward. The necessary arguments are the response (lapse), the design matrix (\(X^{new}\)), the family (binomial), the penalty type (Lasso), the number for the cross-validation (=5) and the lambda search (true), which automatically screens a range of possible values for \(\lambda\). By default, the command standardises the data set. Note that the h2o.glm command itself uses the regular Lasso and does not account for different penalty types. This has already been done by adjusting the design matrix accordingly. There are also other implementations for Lasso in R. We chose h2o, since it outperforms the smurf package with respect to computing time. Additionally, it also satisfies the consistency condition, that the fit for \(\lambda = 0\) coincides with a GLM fit. Other packages like glmnet, see Friedman et al. [12] do not satisfy this condition.

We apply the different versions of the Lasso to model the lapse rates of a European insurance portfolio, see Sect. 2. For each covariate, one of the regularisation types regular Lasso, fused Lasso or trend filtering needs to be specified. There are some recommendations for the choice of the regularisation type but no strict rules. For example, binary or unordered nominal covariates are typically modelled with a regular Lasso. Otherwise, the arbitrary selection of the order of the category levels would impact the predictions. Ordinal covariates with many category levels often perform best with a fused Lasso that forms groups with joint coefficients. Note that a certain degree of expert judgement is required for ordinal covariates, especially when there are only a few category levels. It is difficult to specify whether or not an effect of a covariate follows monotone trends with slopes not equal to zero before analysing the data and building a model. The decision between fused Lasso and trend filtering is essentially a choice whether adjacent categories should be fused together to one level or one common trend. Hence, the model specification should always be checked carefully for effects in a multivariate setting. An example for this analysis is the covariate entry age, which appears to exhibit an increasing trend at the beginning and a negative slope for the other entry age groups in a univariate view, as shown in Fig. 6. However, we decided to fuse different entry age groups together since this appeared more adequate after accounting for the impact of other covariates (light blue line).⁴

We can summarise the individual penalty types for each covariate in the model formula:

Here, the letters T, F and R denote trend filtering, fused Lasso and regular Lasso, respectively. For the trend filtering, we start with the category level in the middle. This decision can be justified by the exposure (starting in an area with higher exposure) or by the expected trend from a univariate analysis of the covariate. A sensitivity analysis shows that this decision has very little impact to our data set.

Based on Zou [38], Devriendt et al. [6] propose different penalty weights for the different regularisation types, since standard Lasso can lead to inconsistent variable selection. However, these weights have little influence for this data set and we chose to not use any penalty weights in this analysis.

As explained in Sect. 3.2, the shrinkage factor \(\lambda\) significantly changes the result of the model, where an optimal value can be estimated by cross-validation. We want to visualise this effect with Fig. 5, where results of four different values of \(\lambda\) (very high—high—optimal—low) are shown for the covariate contract duration. The panels have the same structure as in Fig. 1, so the bars show the exposure and the points show the observed lapse rates. However, this time the results of the model are twofold, as the GLM is a multivariate model. The dark blue line shows the average prediction for each contract duration and the light blue line shows the marginal effect of the covariate, i.e. the \(\beta\) for all relevant contract durations (all other \(\beta\)’s are manually set to zero for this visualisation). In a good model, the dark blue line should reflect the points reasonably well. From the shape of the light blue line, it is possible to explain and interpret the impact of the covariate, here contract duration, on lapse rates.

The top left panel shows the results of the model for a very high value of \(\lambda\), i.e. a large penalisation of coefficients. Since the penalty is high, we get a very simple model (low variance). However, we are not able to detect the main trends of the covariate (high bias) and presumably end up underfitting. The fact that the dark blue line and the light blue line are not identical shows that there are other covariates with coefficients not equal to zero. Decreasing \(\lambda\) will reduce the penalisation, and hence more and more coefficients remain in the model. This can be seen in the panel on the top right. Here, the effect of the trend filtering can be seen, since certain areas of the light blue line follow a trend and in between are several trend changes, for example in contract duration eleven. Remember, that all these trends and trend changes are identified purely data driven and are not based on any manual adjustments. The bottom left panel shows the result for \(\lambda\) based on the 1-SE rule, which we consider the optimal \(\lambda\) for this illustration. Based on visual inspection, the model seems to be able to find the main trends without overfitting the data. From the shape of the marginal effect, we can learn that higher contract durations imply a lower lapse rate. However, the degree of reduction is not constant but is tremendously reduced after three years. If we further decrease lambda and thus further reduce penalisation, the model appears to overfit the data set (as shown in the bottom right panel). The overall prediction is good (low bias), but the model is probably no longer robust (high variance). Note that for the case \(\lambda = 0\), Eq. (2) reduces to Eq. (1), and we have a logistic regression model without any penalisation and presumably no coefficient would be set to zero.

Note that the shrinkage parameter \(\lambda\) is fitted using cross-validation in one step for all covariates jointly. This is one of the main advantages of the Lasso specification in this paper, since it replaces the iterative process of fitting a GLM by a one-step automated process.

The covariate entry age is modelled with a fused Lasso in order to find entry ages that can be grouped together beyond the univariate pre-processing in Sect. 2. Figure 6 shows the result for the optimal value of \(\lambda\). Here, the shape of the light blue line and the dark blue line differ. This illustrates that a univariate approach can be misleading for such a covariate as the slopes of the two lines have different signs in some areas. Table 1 shows the coefficients for entry age. To be more specific: the first row (coefficient) corresponds to the actual coefficients from the model. The second row (coefficient cumulated) translates those coefficients according to the contrast matrix we used for that covariate—here fused Lasso, see Eq. (3). The third row (marginal lapse rate) then applies the inverse of the logit function and therefore gives the marginal lapse rate, which corresponds to the light blue line from Fig. 6. This highlights the advantage of our approach since each coefficient can be explained and interpreted.

The fused Lasso identifies many coefficients to be 0 and therefore effectively removes them from the model. The corresponding category levels are fused together, and the light blue line is constant. However, a closer inspection of the coefficients shows that some of the grouped coefficients are very close to zero, yet not equal to zero, see for example the coefficient for the level [37, 41). This phenomenon of the Lasso is already known from other applications, see Zou [38]. The reason for this is that the \(L_1\) norm is the best approximation of a sparse solution with a convex optimisation, where the actual sparse solution would be with respect to the \(L_0\) pseudo-norm. In a comment on the presentation of Tibshirani [30], Peter Bühlmann, therefore, suggested to “interpret the second ‘s’ in Lasso as ‘screening’ rather than ‘selection’”. For the modelling of lapse rates, this phenomenon may not per se be problematic. If a parsimonious model is required or desirable, the screened parameter can be adjusted manually, as we will illustrate in Sect. 4.5.

So far we have not included any interaction terms, leading to a model with one estimated coefficient for each category level of each covariate. But what if the lapse rate for one category level depends on the category level of another covariate? This appears to be the case for the covariates country and contract duration. The top panel of Fig. 7 shows the observed lapse rates for different values of contract duration on the x-axis and for the four countries on the y-axis. We observe a decreasing trend for all countries with increasing contract duration. However, as in Fig. 1, both level and slopes differ by country.

For example, the observed lapse rate in country four is higher for all values of contract duration. A different level could be modelled without interactions, as the coefficient for country four would be higher in the multivariate model. However, structurally different slopes can only be modelled with interactions, as illustrated in the second and third panel of Fig. 7. The second panel shows the overall prediction of the model without interactions which corresponds to the dark blue line in Fig. 5. The third graph shows the marginal effect of the covariates contract duration and country for the model without interactions, which corresponds to the light blue line in Fig. 5. The model without interactions captures the overall decreasing trend and the different level of the lapse rates per country, e.g. country four has a higher coefficient. But the different slopes for increasing contract durations for different countries cannot be detected by this model. For example, the estimates for country three appear too high for contract durations three to ten.

To capture these differences, it is necessary to include the interaction term for contract duration \(\times\) country in the model. In our framework, we can include interactions straightforwardly and just need to assign a sub-penalty for this new interaction, see Sect. 3.4. We choose a penalty such that the different countries are penalised with the regular Lasso, since there is no ordinal scale for country, and the different values for contract duration are penalised with the fused Lasso for the interaction term. The model formula can be adjusted accordingly:

The resulting model with the interaction term contract duration \(\times\) country is then able to identify the overall main trend, the different overall level of lapse rates per country and the different slopes for the different countries. This can be seen in the fourth and fifth panel of Fig. 7: Now the different lapse rates in country three are captured.

If necessary, further interactions can be added to the model formula in a similar fashion. However, interactions always come with a significant number of additional parameters. In our example, the interaction contract duration \(\times\) country added \(4 \times 20 = 80\) potential additional parameters to the model.⁵ Among them, the Lasso can again identify the most important parameters, such that only 35 parameters are effectively added. All parameters (marginal effects and interactions) are again estimated in one step using cross-validation. Sometimes the addition of an interaction parameter may make a marginal parameter obsolete. This may also be justified from a performance point of view as it is controlled by the cross-validation. However, for practical applications it may be desirable that most of the effects are explained by the univariate effects and only structures going beyond that are explained by interaction terms. To achieve this, it is possible to include interactions in two steps. In a first step, a model without interactions is built. This model produces forecasts for each line item. In a second step, interactions can be analysed with a similar Lasso as in step one, but now without the marginal effects (and therefore only the additional interaction terms) and the forecast of the first step as an offset, i.e. a bias for each line item. By such a two-step approach, the model is able to capture the marginal effects within the offset and can then add relevant interaction terms if needed.

To illustrate this, we start with the Lasso model without interactions and add the interaction term contract duration \(\times\) country. Again, \(\lambda\) is chosen based on a 5-fold cross-validation using the 1-SE rule. In this setup, only 20 parameters are effectively added.

In order to quantitatively assess and compare the different models, we first implement an intercept only model as a baseline. As the name suggests, the intercept only model consists of the intercept but no other covariates. Essentially, the predicted lapse rate for each observation is just the average lapse rate of the portfolio. Several modelling approaches have been discussed so far: First of all the Whittaker–Henderson approach. We implement one model using only the contract duration, resulting in one curve—the blue line from Fig. 1 and one model using contract duration and country, which would lead to four curves. Secondly, we implement a regular GLM without interactions. Thirdly, we implement the proposed Lasso without interaction terms with \(\lambda\) based on the 1-SE rule. This model serves as the base model afterwards.

We consider several sensitivity analyses:For the specification of an optimal model with cross-validation, we already used the deviance. Therefore, a measure based on the deviance allows for a consistent comparison of the models. In addition to the deviance, we also consider the quantity \(1 - \frac{D_M}{D_0}\) which is similar to the \(R^2\) measure for normal distributed data. Here, \(D_M\) is the deviance of the model and \(D_0\) is the null deviance, i.e. the deviance of the intercept only model. This quantity can be interpreted as the improvement over the reference model. We also include the area under the curve (AUC) as another well-known measure. Table 2 summarises the results for the full data set.

Among the analysed models, the Whittaker–Henderson approach has by far the worst performance. This is mainly due to the fact that the approach is univariate. Including another dimension, e.g. country, would slightly improve the results, but also the number of estimated parameters is multiplied by the number of covariate levels (four in the case of the country). Note that some combinations do not contain any observations, see Fig. 7, leading to 69 parameters. This already exceeds the number of parameters of the multivariate Lasso base model with 44 parameters. For the GLM, the number of estimated parameters is quite high (77). Note that we have used the same specification as for the Lasso which implies that for each category level of each covariate a separate parameter is estimated. In order to get a robust GLM, the manual and iterative process of grouping levels would start here.

The alternative proposed in this paper is to use the Lasso to identify these structures automatically. The Lasso model shows a similar good fit (12.14% vs 12.36% improvement over the intercept only model) as the (presumably overfitted) GLM but only has 44 parameters. Using the optimal \(\lambda\) (without the 1-SE rule), the improvement over the intercept only model increases slightly (12.23% vs. 12.14%). However, the number of parameters increases accordingly (54 vs. 44). The models using only one penalty type (all regular and all fused, respectively) show a similar performance with respect to deviance and AUC. But the number of parameters is significantly higher, especially for the all regular Lasso. This is mainly due to the fact that adjacent category levels are not fused together. The sensitivity of the different regularisation types shows a similar effect as the sensitivity of the penalty type: Again, the performance remains very similar and the number of parameters increases, especially for the elastic net with \(\alpha = 0.0\). Just like the GLM (with no regularisation), this model leads to 77 coefficients, since Ridge does not perform variable selection. Increasing \(\alpha\) to 0.5 reveals the selecting property of the Lasso, leading to 55 parameters.

Adding additional information to the model will almost certainly improve the performance. We analyse the effect of both new covariates and an interaction term of existing covariates: Firstly, the additional macroeconomic covariates improve the model performance (13.32% vs. 12.14%), but also increase the complexity (72 vs. 44 parameters). Interestingly, the model with the additional calendar year covariate performs equally well to the model with additional macroeconomic covariates. It seems like the additional information can mostly be described through the calendar year effect and not through the structure of the macroeconomic covariates. Secondly, the interaction term increases the predictive power (12.86% vs. 12.14%). Again, this additional covariate increases the number of parameters (79 vs. 44). The offset model shows a similar effect (12.72% vs. 12.14% and 64 vs. 44 parameters).

The reduced design matrices used in the manually selected models lead to very parsimonious models with 30, 20 and 10 parameters, respectively and decreasing performances. However, the manual model with 30 parameters shows that the number of parameters can be reduced significantly (30 vs. 44) without loosing too much predictive power (12.11% vs. 12.14%). The manual Lasso with only 10 parameters is still better than the Whittaker–Henderson models—increased performance and less complexity. Lastly, the binning effect is analysed with the Lasso with expert judgement binning. We can see that the choice of the bins in the pre-processing has an impact on the results.