A hybrid landmark Aalen-Johansen estimator for transition probabilities in partially non-Markov multi-state models

When the multi-state process is Markov, a consistent estimator of these transition probabilities is provided by the Aalen-Johansen estimator (see Aalen and Johansen 1978). For this, consider n i.i.d. realisations \(X_i(t)\) of X(t), where for subject i we define the at risk process for state j as \(Y_j^{(i)}(t) = 1\{ X_i(t-) = j\}\) and the transition counting process for transition \(j \rightarrow k \in E\) as \(N_{jk}^{(i)}(s,t) = \sum _{u \in (s,t]} 1\{X_i(u-) = j, X_i(u) = k\}\). We assume that all transition counting processes are locally finite i.e. at most a finite number of transitions happens on bounded intervals, hence justifying the summation index for the counting process just defined. Then, the aggregated at risk processcorresponds to the number of individuals in state j just before time t, and the transition counting processescorresponds to the number of transitions directly from state j to state k in the interval (s, t]. Let \(\overline{\mathbf {Y}}(t) := (\overline{Y}_{1}(t), \ldots , \overline{Y}_{K}(t))\) and \(\overline{Y}_{\bullet }(t) = \sum _{j=1}^K \overline{Y}_j(t)\) be the total number of subjects at risk at time t. For \(J_{j}(t) := 1\{\overline{Y}_j(t) > 0\}\) letbe the Nelson-Aalen estimator of the transition rates and \(\widehat{{\varvec{{\Lambda }}}}(t)\) a \(K \times K\) matrix with (j, k)th element (\(j \ne k\)) equal to \(\widehat{{\varLambda }}_{jk}(t)\) and diagonal elements \(\widehat{{\varLambda }}_{jj}(t) = - \sum _{k \not = j} \widehat{{\varLambda }}_{jk}(t)\). The Aalen-Johansen (AJ) estimator of the transition probability matrix \(\mathbf {P}(s, t)\), with elements \(P_{jk}(s,t)\), is then given bywhere the product is ordered according to the time ordering and consist of a finite number of terms due to the locally finite behaviour of the counting processes. Estimated state occupation probabilities at time t may be obtained bywhere \(\widehat{{\pi }}(0)\) is the row vector of empirical state occupation probabilities at \(t=0\), given by \(\widehat{\pi }_j(0) := \overline{Y}_j(0+) / \overline{Y}_{\bullet }(0+)\). Here, the \(+\) in \((t+)\) means that the numbers observed to be in state j at time t, rather than just before time t are to be taken. Datta and Satten (2001) argued that the estimated state occupation probabilities in (3) are consistent, even if the multi-state model is non-Markov. The Aalen-Johansen (AJ) estimator of the transition probabilities \(\mathbf {P}_l(s, t)\) from (1) is then given bywhere \(e_{l}\) is a vector with the lth element equal to 1, and all other elements 0.

Building on the results of Datta and Satten (2001), Putter and Spitoni (2018) defined the landmark Aalen-Johansen (LMAJ) estimator of transition probabilities. This estimator uses the landmark population \(\{i: X_i(s) = l\}\) for estimation of the transition intensities. In what follows we consider the landmark time s and landmark state l, on which we condition, as fixed. Suppressing the dependence on s and l in the notation, we defining the landmark counting process and at risk process as \(Y_j^{(i, \text {LM})}(t) := 1\{X_i(s) = l\}Y_j^{(i)}(t)\) and \(N_{jk}^{(i, \text {LM})}(t) := 1\{X_i(s) = l\}N_{jk}^{(i)}(s,t)\). We define the aggregated at risk and counting processes based on the landmark population asLet \(\overline{\mathbf {Y}}^{(\text {LM})}(t) := (\overline{Y}_{1}^{(\text {LM})}(t), \ldots , \overline{Y}_{K}^{(\text {LM})}(t))\) and \(\overline{Y}_{\bullet }^{(\text {LM})}(t) := \sum _{j = 1}^{K} \overline{Y}_{j}^{(\text {LM})}(t)\).

For \(J_{j}^{(\text {LM})}(t) := 1\{\overline{Y}_{j}^{(\text {LM})}(t) > 0\}\) define the landmark Nelson-Aalen estimator of the transition rates asand let \(\widehat{{\varvec{{\Lambda }}}}^{(\text {LM})}(t)\) be the matrix with (j, k)th element \(\widehat{{\varLambda }}_{jk}^{(\text {LM})}(t)\) and diagonal element \(\widehat{{\varLambda }}_{jj}^{(\text {LM})}(t) = - \sum _{k \not = j} \widehat{{\varLambda }}_{jk}^{(\text {LM})}(t)\). Then the landmark Aalen-Johansen (LMAJ) estimator of (1) presented by Putter and Spitoni (2018) is given by

An undesirable feature of landmark subsampling is a reduction of the number of individuals at risk used for estimation. As defined, the subsample reduce the at risk set for all transitions, including Markov transitions if such exist. An easy way of improving the estimation is by plugging in Nelson-Aalen estimates based on the landmark sample in the Aalen-Johansen (AJ) estimator only for transitions where the difference between the landmark Nelson-Aalen estimate and the full sample Nelson-Aalen estimates differ substantially. This is the main idea behind what we will refer to as the hybrid Aalen-Johansen (HAJ) estimator. We are therefore in particular interested in methods for detecting and deciding when differences in the hazard estimates are large enough to prefer the landmark estimate. In Section 3.2 we describe how a simple two-sample testing procedure can be used to identify the set of such non-Markov transitions. If this is only a subset of the full set of transitions E, we refer to the multi-state processes as partially non-Markov and denote the set of non-Markov transitions as \(A \subset E\). More formally partially non-Markov means that a subset of transitions satisfies \(E[\mathrm {d}N_{jk} \vert \mathcal {F}_{t-}, Y_{j}(t) = 1] = E[\mathrm {d}N_{jk} \vert Y_{j}(t) = 1]\), where \(\mathcal {F}\) is the natural filtration of X. The HAJ estimator is a plug-in estimator of a transition rate estimator where rates are estimated as:Define \(\widehat{{\varvec{{\Lambda }}}}^{(\text {H})}(t)\) to be the matrix with (j, k)th element \(\widehat{{\varLambda }}_{jk}^{(\text {H})}(t)\) and diagonal element \(\widehat{{\varLambda }}_{jj}^{(\text {H})}(t) = - \sum _{k \not = j} \widehat{{\varLambda }}_{jk}^{(\text {H})}(t)\). Now, the HAJ estimator of (1) isObserve that for \(A = \emptyset \) we get the classical AJ estimator, while for \(A = E\) we get the LMAJ estimator. Suppose that only a subset of transitions satisfy the Markov property above, and we wish to form the HAJ estimator. This can be done by applying the standard AJ estimator to a particular subset. More specifically such a subset is obtained by removing individuals that are not in the landmark state at the landmark time point from the specific risk sets used for estimating intensities for the non-Markov transitions. Applying the AJ estimator to such a reduced dataset from the landmark time point and onward will produce the HAJ estimate. As already mentioned, the LMAJ estimator is expensive; meaning that data reduction reduces precision (increases variance). The HAJ estimator will guarantee equal or better precision (relative to the LMAJ estimator), at the possible expense of introducing bias. In Sect. 4 we will study how these opposing effects balance out.

Estimation of transition probabilities in multi-state models relies on a special relation between conditional probabilities and cumulative hazard rate functions. The relation is the multi-state version of the argument for the Kaplan-Meier estimator in classical time-to-event models. For Markov multi-state models the result is due to Gill and Johansen (1990) and says that if \(\mathbf {P}(s, t)\) is the transition probability matrix of a Markov multi-state model and \({\varvec{\Lambda }}\) the cumulative hazard rate matrix, then we havewhere the limits are taken over refinements of (s, t], and \(s =t_0< t_1< \cdots <t_M=t\). As a consequence, the product integral, when considered as a functionalis a convenient construct for producing plug-in estimators of probabilities in multi-state models based on estimators of the cumulative hazard rate matrix. In the Markov case Duhamel’s equation (see e.g. Andersen et al. (1993, Chapter 2)) and smoothness properties of (7) can be used to establish consistency and large sample properties either through Martingale methods (see e.g. Andersen et al. (1993, p. 320)) or through the continuous mapping theorem and the functional delta method. These results rely almost entirely on the properties of (7). Hence if, in the non-Markov case, one can establish (6), or rather a suitably modified version of (6), then there is good reason to believe that consistency and large sample behavior follow as well. We shall first argue heuristically that such a suitable modification of (6) exists in the non-Markov case and as in the Markov case there are two ways of deriving this result. In the Markov case, Gill and Johansen (1990) derived (6) directly by considering limit behaviour of the two products considered in (6). An alternative proof of (6) in the Markov case is due to Aalen et al. (2001), and alternatively by Andersen et al. (1993, p. 296), using what we shall refer to as a book keeping argument.

We shall first consider the direct approach and here we will rely entirely on the results of Overgaard (2019) (who derives the results for state occupation probabilities) and simply claim that the same arguments carry over. The one line argument is that “transition probabilities are occupation probabilities when we stratify to the conditioning event” (i.e. for the landmark data). Consider a fixed landmark time point s and landmark state l and define \(\mathbf {P} ^{\text {LM}}(t, u) = \Bigl ( P_{jk}^{\text {LM}}(t, u) \Bigr )\) to be the matrix of transition probabilities in the landmark population, withBy construction, and regardless of the Markov assumption, we have for \(s \le t\)for \(s = t_{0}< t_{1}< \cdots < t_{M} = t\). For a sufficiently fine partition of the interval (s, t] consider the approximation \(\mathbf {P}^{\text {LM}}(t_{l-1}, t_{l}) \approx I + {\varvec{\Lambda }}^{(\text {LM})}(t_{l}) - {\varvec{\Lambda }}^{(\text {LM})}(t_{l-1})\) where \({\varLambda }_{jk}^{(\text {LM})}(\mathrm {d}t)\) is the transition rate of the landmark population, i.e.The desired result is then to achieve the equalitywhere the limits are taken over refinements of (s, t]. A detailed argument for this will take us too far astray and we will instead refer to Overgaard (2019) Theorem 2 and Theorem 5. Now we turn to the book keeping argument.

Assuming no censoring one can, as pointed out by Aalen et al. (2001) and alternatively by Andersen et al. (1993, p. 296), derive the landmark estimator of \(\mathbf {P}_{l}(s,t)\) pointwise for (s,t) from a bookkeeping argument, which establishes an empirical version of (8). In order to do so let us briefly defineA natural way of estimating the transition probabilities \(\mathbf {P}_{l}(s,t)\) for specific time points s and t is to consider the fraction of empirical means \(\mathbf {\overline{Y}}^{(\text {LM})}(t) / \overline{Y}_{\bullet }^{(\text {LM})}(s)\). The number of individuals from the landmark sample in state j just after time s, \(\overline{Y}_{j}^{(\text {LM})}(s + \varDelta t)\), may be expressed as those who were in state j at time s plus the net arrivals in the small time frame \(\varDelta t\). That isfor \(\varDelta t\) \(\rightarrow 0\), where \([\mathbf {I} + \varDelta \widehat{{\varvec{{\Lambda }}}}^{(\text {LM})}(s +)]_{j}\) denotes the jth column of \(I + \varDelta \widehat{{\varvec{{\Lambda }}}}^{(\text {LM})}(s +)\). From this observation one obtains the following algebraic relationWe may recognize the right hand side as a continuous functional of the Nelson-Aalen transition rate matrix based on the landmark data. Hence consistency will follow from convergence of the left hand side to the desired transition probability, consistency of the rate matrix estimator and the continuous mapping theorem. The only difficulty here is the consistency of the rate matrix. For that we refer to appendix A or alternatively Nießl et al. (2020).

If \(X_{1}, X_{2}, \ldots \) are partially non-Markov, then there is potential gain in terms of power and variance when using the HAJ estimator. Note thatwhich for Markov transitions \((jk) \in A\) implies that \(\mathrm {d}{\varLambda }_{jk}^{(\text {LM})}(t) = \mathrm {d}{\varLambda }_{jk}(t)\). Likewise if we let \(u \rightarrow {\varvec{\Lambda }}^{(\text {H})}(u)\) be the hybrid cumulative hazard rate matrix based on \({\varLambda }_{jk}\) for \((jk) \in A\) and \({\varLambda }_{jk}^{(\text {LM})}\), for \((jk) \in A^{C}\), we have \({\varvec{\Lambda }}^{(\text {LM})} = {\varvec{\Lambda }}^{(\text {H})}\). Thus, when (9) holds, the same holds in the partially non-Markov setting using \({\varvec{\Lambda }}^{(\text {H})}\) and consistency of the HAJ estimator follows by continuity of (7).

As pointed out by Titman and Putter (2020), from (11) it is clear that a test of the Markov assumption for a specific jump transition process from state j to state k can be obtained by comparing intensities based on disjoint landmark states. One can use a two-sample test of the hypothesisNote that the two disjoint landmark states ensure independence between samples in the estimation procedure. Typically, \(l_1\) would be the landmark state of main interest and \(l_2\) the set of all other remaining possible states. In the application and the simulation study we use two different tests as selection criteria for determining Markov and non-Markov behaviour for specific transitions. Denote the log-rank test statistic for transition \(j \rightarrow k\) from landmark time point s by \(\mathfrak {X}_{s}\). This test statistic is the basis for a test referred to as the point test. Since the above test statistic is dependent on s, Titman and Putter (2020) suggest a more global test of the Markov assumption for transition \(j \rightarrow k\) based on \(\mathfrak {X} := \max _{i} \mathfrak {X}_{s_{i}}\), over a suitable grid \(s_{1}, \ldots , s_{k}\). We refer to this as the grid test. For further discussion of the use of two sample tests to identify non-Markov transition see Online Resource C. We also stress that these tests are considered primarily as diagnostic tools and we do not address issues related to multiple testing.

For simplicity, we have so far not considered censoring. Censoring is not used in the simulation experiment and is not a major issue in the practical application. In time-to-event analysis censoring is generally the rule rather than the exception and a comment on the matter is appropriate. First of all, in the setting of multi-state models, stronger censoring assumptions are typically needed, compared to the regular survival setting (see e.g. Aalen et al. (2008, p. 123)). Overgaard (2019) derives the result (6), which would extend to (9), based on a form of independent censoring coined “the status independent observation assumption” and Glidden (2002) considers right censoring with strong independence assumptions requiring censoring to be independent of states. A similar independent censoring assumption is needed for the LMAJ and HAJ estimators, as discussed by Putter and Spitoni (2018). For dependent censoring, Datta and Satten (2002) consider an IPCW version of the AJ estimator of state occupation probabilities. This weighted estimator was empirically investigated by Gunnes et al. (2007) and showed reasonable results for non-Markov behaviour induced by a joint frailty at baseline and various dependent censoring regimes. If present, dependent censoring can affect selection into the landmark sample, and a similar IPCW version of LMAJ or HAJ should be considered.

We have not considered covariate based hazard rate models in our formal treatment of the HAJ estimator but the results are expected to generalize to classical covariate models e.g. Cox or additive hazard regression due to the smoothness property of (7). See e.g. Hoff et al. (2019) for a discussion of covariate based models for the transition rates in relation to the LMAJ estimator.

Regarding variance estimates for the LMAJ and HAJ estimator we recommend bootstrapping. The LMAJ and HAJ estimator are constructed by applying the AJ estimator to different subsets of the entire data set. Non-parametric bootstrap estimates of the variance and point wise 95 percentile intervals can be obtained in the following way: First perform random re-sampling of subjects with replacements, then obtain estimates of transition probabilities (AJ, LMAJ and HAJ) by applying the AJ estimator to the different subsets of each of the the bootstrap samples. Variance and percentile estimates can then be obtained by pointwise sample variance estimates and pointwise sample percentile estimates. Since we use a test statistic to create the Hybrid estimator a natural question is whether to base the test on the original sample or to apply it to each bootstrap sample. In the practical application the tests are based on the original sample. Many of the classical variance estimators for the AJ estimator rely on the transition rates not depending on the history of the multi-state process. This is true under the Markov assumption but if we relax the Markov assumption additional variation is introduced for the Nelson-Aalen estimator (See appendix A for a formal discussion) and in turn for the plug-in estimator of transition probabilities. In the first simulation experiment we study the empirical coverage of the Greenwood type estimator of confidence intervals and in the practical application we compare such confidence intervals to bootstrapped confidence intervals. The results are included in Online Resource A.2 and B.2. The results suggest only minor deviations of the Greenwood type estimator, but it is unclear whether this holds in general.

From Fig. 2 we see that, for non-zero frailty variance, the HAJ estimator performs at least as good as or better than the AJ estimator and better than the LMAJ estimator. In other words, the interpretation of the HAJ estimator as an intermediary between the AJ estimator and the LMAJ estimator seems to hold. In the Markov case, i.e. \(\sigma ^{2} = 0\), we see almost no performance difference between the HAJ and the AJ estimators. Due to the in-built error from testing the Markov assumption we generally expect better performance of the AJ estimator over the HAJ estimator for Markov models, in particular for models with many transitions. For \(\sigma ^{2} = 0.4\), the LMAJ estimator performs worse than the AJ estimator. The HAJ estimator is comparable to AJ for \(\sigma ^2 = 0.4\) but starts to outperform AJ for larger values of \(\sigma ^2\). In this experiment, HAJ always performed better than LMAJ. In other words, Fig. 2 suggests that if a perturbation of the transition intensity is sufficiently large, so as to induce significant non-Markov behaviour for the transition probabilities, then the HAJ estimator outperforms the AJ and the LMAJ estimator. However, as we shall see in the next experiment, this conclusion has its limitations. In the experiment all transitions are tested and results from the point tests and grid tests are included in Online Resource A.1.1.

Regarding the bias-variance trade-off, we see from Fig. 3 that the AJ estimator overestimates the transition probability \(P_{21}(s, t)\), whereas the LMAJ and the HAJ estimator are close to the true transition probability. We also see that the HAJ estimator has smaller variance than the LMAJ estimator. This is exactly what we would expect from the HAJ estimator. Through the selection method (the grid test applied to all testable transitions) it accounts for the partly non-Markov and partly Markov behaviour, resulting in smaller bias than the AJ estimator and less variance than the LMAJ estimator.

From Fig. 4 we see that for the estimated transition probability from state 2 to 1 and state 2 to 3 the HAJ estimator performs slightly worse than the LMAJ estimator. This is expected when the non-Markov behaviour is strong enough, simply because the selection procedure for the HAJ estimator will make the wrong choice in some percentage of cases based on the significance level. The point at which the HAJ estimator is favourable to either the LMAJ or the AJ estimator depends partly on sample size and partly on how attenuated the non-Markov behaviour is. A good selection mechanism should take both elements into account. If we use a test as selection mechanism one could regard the significance level as a parameter deciding what “too much attenuation” should mean. From this perspective a standard 5 percent level need not be the optimal choice. In the experiment all transitions are tested and results from the point tests and grid tests are included in Online Resource A.1.2.

In our first example, we calculate transition probabilities from being on sick leave at day 100. The main motivation is to compare the AJ estimator with the LMAJ and HAJ estimator of transition probabilities for starting points after the original time of inclusion in the study (as this is where the AJ estimator may be biased due to violations of the Markov assumption). Note however, that, as in this example where time is measured as within year calendar time, the scientific question at t= 100 and t = 0 could indeed be slightly different. For example, using this time scale allows for investigating seasonal effects on the probability of returning to work from sick leave. In this particular analysis, the full dataset is reduced from 184 951 individuals to 23 288 by fixing a number of covariates. Specifically, we consider high school completers attending general education (non-vocational) that scored between 7 and 9 (i.e. high scores) on the cognitive test during military conscript examinations (scores range between 1-9 where 9 is considered the best). The landmark subset then consists of only 72 individuals. Due to the heavy restriction on covariates, we expect the sample population to now be fairly homogeneous compared to the full cohort. If the limited sample size makes the LMAJ low-powered for certain transitions, we would expect to see differences in the the LMAJ and HAJ estimates.

We start by looking at cumulative transition intensities in the landmark sample and compare them with cumulative transition intensities calculated from the full dataset. The various intensities are shown in Fig. 6. Differences between the curves from the two data samples indicate a violation of the Markov assumption. Inspection of Fig. 6 suggests that transitions exhibiting similar intensities in the landmark sample and the full sample are \(1 \rightarrow 2\), \(4 \rightarrow 1\) and \(4 \rightarrow 2\). In addition to visual inspection, we can test for non-Markov behaviour using the point test described in Sect. 3.2. Results from this test, found in Online Resource Table C.1, imply that transitions \(1 \rightarrow 2\), \(2 \rightarrow 5\), \(4 \rightarrow 1\) and \(4 \rightarrow 2\) are not significantly non-Markov. Based on the above results, the HAJ estimator will here utilize all available data for transitions (\(1 \rightarrow 2\), \(2 \rightarrow 5\), \(4 \rightarrow 1\) and \(4 \rightarrow 2\)) and only the landmark data for the other transitions. We illustrate the resulting transition probabilities from the landmark state (sick leave) into work and into education in Fig. 7. Estimated transition probabilities to all states based on the HAJ and LMAJ estimators are found in Online Resource B.2.1.

In this example, LMAJ and HAJ estimates are in close agreement and differ substantially from the AJ estimates. The AJ transition probability estimates are far greater than estimates produced by the LMAJ and HAJ estimators, indicating that the AJ estimates are biased. Figure 7 show that for transition 3 \(\rightarrow \) 1, the HAJ estimates appears slightly smoothed compared to LMAJ estimates, and for transition 3 \(\rightarrow \) 4, the pointwise confidence intervals for the HAJ estimates are narrower at late times than the confidence intervals for the LMAJ estimates. The AJ estimates have more narrow confidence intervals due to the much larger sample size. Confidence intervals for the two landmark estimators are based on bootstrapping, while for AJ estimator, Greenwood plug-in estimates for standard errors were used. In Online Resource B.2.1, we also compare plug-in estimates of standard errors with bootstrap estimates for the HAJ estimator and find that they are very similar, with the bootstrap standard errors being slightly larger.

In our second example we still use data on high school completers of general education, but now consider individuals with cognitive scores between 4 and 6 (medium scores) and only individuals with parents who completed high school as their highest formal education. This amounts to a total of 10 451 individuals. Day 3000 is chosen as the landmark time point, and the landmark state is now unemployment. The landmark subset consists of 463 individuals. Results of log-rank tests for identifying Markov and non-Markov transitions can be found in Online Resource Table C.2. The results indicate that transitions \(2 \rightarrow 5\), \(3 \rightarrow 2\), \(3 \rightarrow 4\), \(3 \rightarrow 5\), \(4 \rightarrow 1\) and \(4 \rightarrow 3\) are Markov. Thus, all available data are used for these transitions when constructing the hybrid estimator.

Estimates of transition probabilities from unemployment to education and disability are presented in Fig. 8. Compared to our previous example, AJ estimates seem less biased when compared to the estimates from the two landmark methods, but do fail to capture the development in the first one third of the time period. In terms of precision, the HAJ estimator seems to give higher precision than the LMAJ estimator for both the showcased transitions.

Estimated transition probabilities to all states based on the HAJ and LMAJ estimators and the corresponding comparison of bootstrap and Greenwood type estimates of standard errors for the HAJ estimator in this example can be found in Online Resource B.2.2.