Variable Selection for Hidden Markov Models with Continuous Variables and Missing Data

Assuming independence between units, the log-likelihood referred to the observed data can be written asIn the above expression, \(\varvec{\theta }\) is the vector of all model parameters and \(f(\varvec{y}_{i}^o)\) is the manifest distribution of the observed response variables defined in Eq. 2. In order to estimate these parameters, we maximize \(\ell (\varvec{\theta })\) by an EM algorithm (Baum et al., 1970; Dempster et al., 1977), which is based on the complete-data log-likelihood expressed asIn the previous formulation, \(z_{itu}=I(u_{it}=u)\) is an indicator variable equal to 1 if individual i is in latent state u at time t and \(z_{it\bar{u}u}=z_{i,t-1,\bar{u}}\;z_{itu}\) is the indicator variable for the transition from state \(\bar{u}\) to state u of individual i at time occasion t. Also, note that \(\ell ^*(\varvec{\theta })\) is the sum of three components that may be maximized separately. Regarding the first component, we have to consider that the model assumptions imply thatand, therefore, this component simplifies asIn practice, the E-step of the EM algorithm computes the posterior expected value of the dummy variables involved in \(\ell ^*(\varvec{\theta })\) given the observed data and the current value of the parameters. In particular, these expected values correspond to the following quantities:They are computed by means of forward-backward recursions of Baum et al. (1970) and Welch (2003); for an illustration see Bartolucci et al. (2013, Chapter 3). With missing observations and under the MAR assumption, the E-step also includes the computation of the following expected values whereandAt the M-step of the EM algorithm, we update the model parameters by considering the following closed form for the means:moreover, \(\varvec{\Sigma }\) is updated as follows:Finally, the initial and transition probabilities may be obtained on the basis of the usual formulation, that is,The E- and M-step are repeated until convergence, which is checked on the basis of the relative log-likelihood difference, that iswhere \(\varvec{\theta }^{(s)}\) is the vector of parameter estimates obtained at the end of the s-th iteration of the M-step and \(\epsilon \) is a suitable tolerance level (e.g., \(10^{-8}\)).

A crucial aspect of the estimation process is the initialization of the EM algorithm as the model log-likelihood is typically multimodal. In such a situation, a multi-start strategy, based both on a deterministic and a random starting rule, is required. Overall, for a given k, the inference is based on the solution corresponding to the largest value of the log-likelihood at convergence, which typically corresponds to the global maximum. The parameter estimates obtained in this way are collected in the vector \(\hat{\varvec{\theta }}\). To obtain the corresponding standard errors, we can rely on a non-parametric bootstrap procedure (Davison and Hinkley, 1997), which is performed by drawing, with replacement, a number of bootstrap samples from the original one, and estimating the proposed HM model with the selected number of states on these samples. This procedure is easily implemented and it has the advantage of the robustness of the results.

With reference to a given number of states, k, the dynamic assignment of units to the latent states is of particular interest. As usual, the estimation algorithm directly provides the estimated posterior probabilities of \(U_{it}\), as defined in Eqs. 3 and 4. These probabilities can be directly used to perform local decoding so as to obtain a prediction of the latent states of every unit i at each time occasion t. In particular, the predicted latent state at occasion t for a sample unit i, denoted by \(\hat{u}_{it}\), is obtained as the value of u corresponding to the maximum of \(\hat{z}_{itu}\). The entire sequence of predicted latent states resulted by the local decoding is denoted as \(\hat{u}_{i1},\ldots , \hat{u}_{iT}\). The so-called global decoding, which is based on an adaptation of the Viterbi algorithm (Viterbi, 1967), may also be adopted to obtain the prediction of the latent trajectories of a unit across time, that is, the a posteriori most likely sequence of latent states.

Finally, note that it is also possible to perform multiple imputations of the missing responses conditionally or unconditionally to the predicted latent state. In more detail, in the conditional case the predicted value is simply \(\hat{\varvec{y}}_{it}=\textrm{E}(\varvec{Y}_{it} | \varvec{y}_{it}^o, \hat{u}_{it})\), where \(\hat{u}_{it}\) is the predicted latent state and the expected value is defined in Eq. 5. The unconditional prediction of the missing responses is instead obtained as

Starting from an initial set of variables, the proposed method is aimed at finding the set of items that provides the best value of the BIC index (Schwarz, 1978), which may be seen as an approximation of the Bayes factor (Kass and Raftery, 1995). In particular, for a given model, the index may be defined aswhere \(\hat{\ell } \) denotes the maximum log-likelihood of the model of interest and \(\#\text {par}\) denotes the corresponding number of free parameters. The same criterion is used to perform the model selection step as illustrated in the following. Note that, with respect to other implementations of the BIC available in the literature about HM models (for a review see Bacci et al., 2014), we rely on a stronger penalization based on the overall number of observations equal to nT rather than on n only. We prefer this choice that leads to a more parsimonious selection of the relevant variables, providing also better results within the simulation study and easiness of comparisons with alternative procedures.

Let \(\mathcal {Y}\) be the set of clustering variables and \(\overline{\mathcal {Y}}\) the complement of \(\mathcal {Y}\) with respect to the full set, that is, the set of remaining variables. The variable selection procedure is based on finding the set of relevant variables, that is, the set of the most useful variables for clustering, which minimizes the following indexIn the previous expression, \(\text {BIC}_k(\cdot )\) is the BIC index computed as in Eq. 6 under the proposed HM model with k latent states, fitted on the set of clustering variables indicated in parentheses, and \(\text {BIC}_{\text {reg}}(\cdot )\) is the BIC index referred to the multivariate linear regression for the irrelevant or noise variables, \(\overline{\mathcal {Y}}\), on the set \(\mathcal {Y}\), which are assumed to be independent of the cluster memberships and thus not providing additional information about clustering.

In applying this procedure, attention must be paid to the estimation of the above linear regression models when missing data are present in the set of the independent variables. In this case, imputation is necessary and, as discussed in Sect. 3.1, in particular we adopt a sort of multiple imputation based on the posterior expected values in Eq. 5 obtained at convergence of the EM algorithm used to estimate the proposed HM model on the full set of variables. In this preliminary stage, a model selection procedure is also performed, aimed at selecting the proper number of states based on the \(\text {BIC}_k\) index, for increasing values of k. When missing data are present in the subset of variables considered as dependent variables in the linear regression model, maximum likelihood estimation of model parameters is performed under the MAR assumption, and the corresponding \(\text {BIC}_{\text {reg}}\) is computed.

We propose a greedy inclusion-exclusion procedure that starts with an initial set of clustering variables, denoted by \(\mathcal {Y}^{(0)}\), an initial number of latent states, denoted by \(k^{(0)}\), and the corresponding \(\text {BIC}_{\text {tot}}(\mathcal {Y}^{(0)},k^{(0)})\) index. At the h-th iteration, the algorithm performs the following two steps:The algorithm ends when no variable is added to or is removed from \(\mathcal {Y}^{(h)}\). It is worth mentioning that the proposed approach may be influenced by the choice of the initial set of responses, \(\mathcal {Y}^{(0)}\); therefore, some preliminary or sensitivity analyses at this aim are required. In general, starting with the complete set of variables is also possible, so relying on a backward scheme. Otherwise, it is possible to start with an empty set and follow a forward procedure. The latter should be preferred, especially when there are many response variables. In the simulation study presented in the next section and in the applicative example illustrated in Sect. 6, we opt for a forward procedure.

Taking inspiration from Maugis et al. (2009b), we also propose an improved version of the above algorithm in which the variables not included in the HM model are regressed to a subset of those included in this model, and this subset is selected by a suitable forward procedure. Obviously, the resulting algorithm is slower than the initial version but, as we experimented, it leads to a more reasonable choice of the relevant variables. The results of the simulation study and the application reported in the next sections are obtained with this improved version, while those obtained with the initial version are available upon request to the authors.

Within the simulation study, we randomly drew \(B = 100\) samples of size \(n =250, 500, 1,\!000\) from an HM model of the type formulated in Sect. 2, with a number of hidden states equal to \(k = 2, 3\). We assumed \(r=2,4\) continuous outcomes and \(T=5,10\) time occasions. Regarding the measurement model, the following values for the conditional means are considered: \(\varvec{\mu }_1 = (0,0)^\prime \) and \(\varvec{\mu }_2 = (4,0)^\prime \), with \(k = 2\) states and \(r = 2\) response variables, whereas with \(r = 4\) variables we set \(\varvec{\mu }_1 = (0,0,0,0)^\prime \) and \(\varvec{\mu }_2 = (4,0,4,0)^\prime \). Moreover, with \(k = 3\) latent states and \(r = 2\) outcomes we set \(\varvec{\mu }_1 = (0,0)^\prime \), \(\varvec{\mu }_2 = (4,0)^\prime \), and \(\varvec{\mu }_3=(4,2)^\prime \), by duplicating the means of the variables when \(r=4\) as before. When \(k=2\) and \(r=2\), an additional scenario characterized by less separated states is also considered, by letting \(\varvec{\mu }_1 = (0,0)^\prime \) and \(\varvec{\mu }_2 = (2,0)^\prime \). We also assumed a variance-covariance matrix constant across states, with all variances equal to 1 and covariances equal to 0.5. Equally likely states are considered at the first time occasion, that is, \(\pi _u = 1/k\), \(u=1,\ldots ,k\), whereas time-homogeneous transition probabilities, with persistence probabilities equal to \(\pi _{u|\bar{u}} = 0.8\), when \(u=\bar{u}\), and off-diagonal probabilities equal to \(\pi _{u|\bar{u}}=0.2/(k-1)\), when \(u\ne \bar{u}\), \(u=1,\ldots ,k\), are assumed.

In order to evaluate the performance of the variable selection procedure, a total of \(J=15\) variables are included, with the r clustering variables generated according to the HM model defined above. The \(J-r\) irrelevant variables are instead simulated according to a multiple regression model depending on the relevant clustering variables, with the intercept varying on an equally spaced grid of values from −2 to 2, and the regression coefficients fixed to (−1, 1) for all \(J-r\) variables when \(r=2\). An alternative setting is also considered, by simulating the \(J-r\) noise variables from a standard multivariate Gaussian distribution, so that they are assumed to be independent of the relevant clustering variables.

A varying proportion of intermittent missing responses, \(p_{miss} = 0, \ 0.05, \ 0.1, \ 0.25\), is also assumed for the full set of variables. Finally, in order to evaluate the robustness of the method in the presence of mild skewness in the data, an alternative scenario in which the relevant variables are simulated according to a Chi-squared distribution with 10 degrees of freedom is also implemented.

Overall, a total of 194 different scenarios are considered. In the following, we report the results obtained under the most informative scenarios in order to assess the effect of the different design factors on the selection of the proper set of clustering variables and of a valid number of latent states.

The results reported in the following are based on a series of different scenarios:In Table 1, for each scenario described above, we report the frequency of samples in which the correct variable partition is chosen together with the frequency of samples in which the true number of states is selected. We also report the average computing time (in seconds) required by the greedy algorithm to reach the convergence and the average adjusted Rand index (ARI, Hubert and Arabie, 1985), which is aimed at evaluating the agreement between the estimated and the true latent structure. In particular, this index attains its upper bound equal to 1 when there is a perfect agreement between the true and estimated clustering structure and to 0 otherwise.

From the results, we observe that the true clustering variables are always correctly selected under all scenarios, except for scenario 8, that is, when the proportion of missing responses increases. In such a situation, the relevant variables are selected in just over half of the samples. However, we notice that in all samples the true clustering variables are always selected, in addition to some irrelevant variables. This worsening in the results is likely due to the imputed explanatory variables in the regression model estimated on the noise variables.

Referring to the performance of the simultaneous model selection procedure, we observe that the number of latent states is correctly chosen in all samples, regardless of the simulated scenario, apart from scenario 11, where the clustering variables are generated from a Chi-squared distribution. Here, the true number of states is always overestimated, due to the presence of skewness in the data, leading to selecting \(k=3\) in 3 samples out of 100, \(k=4\) in 73 samples out of 100, and \(k=5\) in 24 samples.

Looking at the results in terms of clustering performance, measured by the average ARI over the simulated samples, it is evident that the proposed approach performs well under all scenarios. In particular, the clustering quality worsens as the number of states increases and with lower separation of the states due to the increasing complexity of the model associated with a higher uncertainty on the latent structure. Moreover, as expected, the average ARI across simulations reduces when there is an increasing proportion of missing responses or when the simulated data deviate from a Gaussian distribution. On the other hand, increasing the number of relevant variables and the number of time occasions allows us to recover in a better manner the latent structure of the model, thanks to a higher amount of available information.

In evaluating the performance of the proposed approach, it is also relevant to take into account the computational cost of the corresponding greedy search algorithm. All scenarios are run on an Apple M1 Pro with 16 Gb for a fair comparison. As mentioned, when considering the improved version of the algorithm that includes an additional forward step, the computing time necessary to reach convergence increases. In particular, it varies across simulated scenarios, on average, from a minimum of about 245 s to a maximum of about 622 s under the most complex scenario characterized by a high proportion of missing variables.

It is also interesting to compare our proposal with a simplified version of the model, which does not take into account the longitudinal structure of the data and avoids missing values. In particular, we first imputed the missing responses before starting the variable selection algorithm by using function imp.mix of the R package mix (Schafer, 2022). Then, we run the clustvarsel algorithm that performs variable selection for Gaussian model-based clustering according to similar greedy search algorithm (Scrucca and Raftery, 2018), following the classical approach of Raftery and Dean (2006). In our longitudinal context, the responses of the same unit to the different time occasions are considered independent, and a finite mixture model of Gaussian distribution with a variance-covariance matrix common to all components is estimated. We observe that, under the first six scenarios reported above, the simplified procedure performs well in selecting the clustering variables and the true number of states but it leads to a reduction of the average ARI. Obviously, the computational time is reduced with respect to our proposal due to the simplified version of the model and the variable selection algorithm. On the other hand, in more complex scenarios, characterized by a large proportion of missing values, less separation of the hidden states, and in the presence of mild skewness in the data, the performance of the clustvarsel algorithm gets worse, especially concerning the correct estimation of the number of clusters. Consequently, this approach also attains poor results when considering the clustering quality, substantially reducing the average ARI. Results are available upon request by the authors.

In the analysis, using the approach proposed in the paper we selected the best subset of indicators able to highlight the patterns of disparities among countries and to provide a picture of how countries are performing over time. Thus we shed light on the way to classify or cluster countries based on their level of socioeconomic development. As illustrated at the end of Sect. 4.2, the greedy search algorithm is implemented by performing a preliminary inclusion step, in which each variable is singly proposed in turn for inclusion in the initial set. At the same time, for each candidate variable, a model selection step is performed to choose the optimal number of states.

The procedure led us to select a model with \(k = 6\) latent states including \(r= 12\) indicators, showing a log-likelihood value at convergence equal to -20,998.52 with 665 parameters and a BIC value of 47,496.78. Table 3 reports the estimated cluster conditional means \(\varvec{\mu }_u\), \(u=1,\ldots ,6\), referring to the selected set of indicators.

For a more straightforward interpretation, the latent states are increasingly ordered according to the values of the estimated means of the variables Lit and Ele, and decreasingly ordered according to values of Fert. For a visual comprehension of the different features among selected clusters, Fig. 1 shows the resulting heatmap. We note that:Table 4 shows the estimated covariances (lower part), variances (diagonal), and the partial correlations (upper part) among the selected variables. Looking at the partial correlations we observe that Rese and Sch2 have a quite high correlation, whereas Saf has a negative correlation with Rese, and a positive correlation with Gsav; Lit is negatively associated with Fert given all the remaining indicators.

The estimated parameters of the latent model referred to the initial probabilities in 2000 and transition probabilities from 2000 to 2001 are reported in Table 5. The statistical significance of the coefficients is established according to the estimated standard errors obtained by the non-parametric bootstrap procedure described in Sect. 3.1, based on 300 samples. Tables with the estimated standard errors are reported in the SI (see Tables 11 to 14).

At the beginning of the study, some countries belong to the 6th group (about 28%), while around 18% of countries belong to the 1st cluster. From 2000 to 2001, we mainly observed mobility of the countries of the 3rd cluster to the 4th (4.5%) and the 6th (3.2%) and from the 5th to 3rd (2.9%) and 4th (3.1%). Countries in the 1st group have a probability of around 0.01 of moving to the 2nd group. We note a very high persistence probability for the 4th and the 6th groups.

The estimated transition probabilities referred to the period just before the global financial crisis (2005 to 2006) are reported in Table 6. Two upward transitions are visible for countries in clusters 3 and 4: (i) a probability of 0.07 of moving from the 3rd to the 4th state and of 0.05 to the 5th; (ii) a probability of 0.06 of moving from the 4th to the 5th cluster.

Probabilities referred to the transitions in years 2010 to 2011 are reported in Table 7. Three years after the global economic downturn, we notice a probability of: (i) 0.1 of moving from the 2nd to the 3rd cluster; (ii) 0.04 from the 3rd to the 6th cluster; (iii) 0.10 from the 4th to the 6th. Transitions estimated for the last period of observation from 2016 to 2017 are reported in Table 8; 30% of countries in the 4th cluster are moving to the 6th cluster, thus showing a general growth for emerging economies.

Using global decoding, we can inspect the most dynamic countries in terms of changing clusters over time. Figure 2 shows the proportion of countries predicted in each cluster at every time occasion. It results that the frequencies of countries predicted in the 1st, 3rd, 4th, and 5th clusters are lower at the end of the period, whereas those of the 6th and 2nd clusters are higher. A better understanding of the predicted dynamics can be gained by the estimated relative frequencies of each cluster reported in Table 9 for years 2000, 2006, 2011, and 2017.

Looking at the predicted sequence of latent states, we also observe that twenty countries illustrated in Fig. 3 are predicted to have three transitions across states during the whole period except Bhutan (Thunder Dragon Kingdom), which results in the unique country to show four transitions. In particular, we notice that Afghanistan moves from the 1st up to the 4th cluster. El Salvador and Dominican Republic transit from the 4th to the 6th during the last period. Also, Ghana and Timor-Leste move from the 1st up to the 3rd cluster, during 2007–2008 and 2009–2010, respectively. These results are in line with the fact that, for example, Ghana is often considered one of the fast-developing countries in Africa due to a government that promoted a stable political environment and effective management of the significant natural resources present in the country. Timor-Leste is recognized as a fast-developing country in Southeast Asia due to natural resource revenue and poverty reduction after its independence in 2002; see the reports of The World Bank Group (2015, 2022), and Zallé (2019), for further details.

Figure 4 shows the maps of the countries according to the predicted states both in 2000 and 2017. At the beginning of the period, thirty-nine countries are allocated in the 1st cluster, and only twenty of them remained in the same cluster at the end of the period,⁴ thus confirming as the worst countries in terms of socioeconomic development. From the maps, we notice that many Central and Latin American countries made significant progress as well as India. In particular, the Republic of Ecuador has made progress in social and human development and Vietnam was the fastest growing economy in Asia.⁵ On the other hand, Libya changed from the 5th to the 3rd cluster probably due to the civil war that started in 2014.⁶

We assessed the clustering quality by the Q statistic proposed in Hennig and Coretto (2022). It measures non-parametrically how close the within-cluster distributions are to the elliptical unimodal distributions with the only mode on the mean. The realized values of the statistics are near zero (best value) for all the variables and clusters at every time occasion. The highest values of around 0.21 are observed only in some clusters and time occasions for the variable Ele.

The classification uncertainty is measured by the entropy calculated according to the posterior probabilities, which is 158.38 and, compared with its maximum 6,998.61, indicates that the model provides a suitable clustering structure while accounting for parsimony.