Some dimension reduction strategies for the analysis of survey data

The idea in principal component analysis (PCA) is to transform the predictor variables into linearly independent variables—or principal components—such that the first principal component has the largest variance, the second principal component has the second largest variance and is orthogonal to the first principal component, and so on. More formally, let \(\Sigma \) be the covariance matrix of \(\mathbf {X}.\) We want to find p linear combinations of \(\mathbf {X}\) such that they are uncorrelated with each other. We construct these components such that the first component’s variance is the maximum among all the linear combinations, the second component’s variance is the second largest and uncorrelated to the first component, the third component’s variance is the third largest and uncorrelated to both the first and the second component, etc. In other words, we wish to find the appropriate vectors \(\mathbf a _1,\mathbf a _2,\ldots ,\mathbf a _p\) such that the following holds:The construction of the principal components are guaranteed by the following proposition, which is a condensed version of Result 8.1 in Johnson and Wichern [16]:

We can use, for example, the singular value decomposition of the covariance matrix \(\Sigma \) to find the eigenvalues and eigenvectors. \(\Sigma \) is also referred to as the kernel matrix. For consistency with notation used later, we denote \(M_{PC} = \Sigma \) as the kernel matrix for PCA. This is estimated by \(\hat{M}_{PC} = \hat{\Sigma },\) which is based on the sample data.

Principal component analysis is, perhaps, the oldest dimension reduction technique that is still widely used today [17, 18]. Consequently, PCA has been applied to numerous important data problems spanning a wide array of scientific fields. For example, PCA has been used for facial image recognition in image analysis [19], for the analysis of hormone profiles to assess the productivity of plants [20], and as part of a robust decision support tool for facilitating industrial production scheduling [21]. PCA has been applied for various survey data analyses, but due to the sometimes large number of binary or categorical variables in such data, it does not always provide reliable results [22].

Formally, a dimension reduction is a function \(\mathcal {R}(\mathbf X )\) that maps \(\mathbf X \) to a k-dimensional subset of the reals such that \(k < p.\) Specifically, we let \(\mathcal {R}(\mathbf X )=\mathbf B ^{\text {T}}{} \mathbf X, \) where \(\mathbf B \) is a \(p\times k\) matrix. We say that a dimension reduction is sufficient if the distribution of \(Y|\mathcal {R}(\mathbf X )\) is the same as that for \(Y|\mathbf X, \) which is the original conditional distribution of interest in regression models. Combining the notions of dimension reduction and sufficiency, sufficient dimension reduction [11, 15, 23] is used to detect a lower dimension subspace of the predictor space, such that the response variable is independent with the predictor vectors providing all the information of this subspace.

Without loss of information, \(\mathbf {X}\) can be replaced by \(\mathbf {\eta }^\text {T}\mathbf {X},\) where \(\mathbf {\eta }\in \mathbb {R}^{p\times d},\) \(d < p.\) The subspace spanned by the columns of \(\varvec{\eta }\) is called a dimension reduction subspace for the regression of Y on \(\mathbf {X}.\) The intersection of all dimension reduction subspaces is called the central subspace (CS), which we denote by \(\mathcal {S}_{Y|\mathbf {X}}\) with dimension \(d = \text {dim}(\mathcal {S}_{Y|\mathbf {X}}).\) The basis \(\varvec{\beta }\in \mathbb {R}^{p\times d},\) \(d<p\) of the CS has the property that \(Y \perp \mathbf {X} |\mathbf {\beta }^{T}\mathbf {X},\) which is to say that the conditional distribution of \(Y|\mathbf {X}\) is the same as the conditional distribution of \(Y|\varvec{\beta }^{T}\mathbf {X}.\) Under mild conditions [23], the CS exists and is unique.

Sometimes, the mean function \(\text {E}(Y|\mathbf {X})\) may be of primary interest instead of the conditional distribution \(Y|\mathbf {X}.\) For such settings, Cook and Li [24] introduced the following. Let \(\varvec{\beta }\in \mathbb {R}^{p\times d},\) \(d<p\) now be the basis for the subspace for \(Y \perp \text {E}(Y|\mathbf X ) |\mathbf {\beta }^{T}\mathbf {X}.\) This subspace is then called the mean dimension reduction subspace. The intersection of all mean dimension reduction subspaces is called the central mean subspace (CMS), which we denote by \(\mathcal {S}_{\text {E}(Y|\mathbf {X})}.\)

For estimating the CS or the CMS, the following two conditions are assumed for many dimension reduction methods:In the above, \(\mathbf P _{\mathcal {S}}\) is a projection matrix onto the subspace \(\mathcal {S},\) which is either \(\mathcal {S}_{Y|\mathbf X }\) or \(\mathcal {S}_{\text {E}(Y|\mathbf X )}\) for estimating the CS or CMS, respectively.

It is important to emphasize the fundamental difference between PCA and sufficient dimension reduction. PCA reduces the number of predictors without considering the response variables, and choosing the number of principal components is not done through any formal inference paradigm. However, the idea of sufficient dimension reduction is to attain a sufficient subspace, which includes all of the information we need. There are asymptotic results for determining the number of dimensions in sufficient dimension reduction. These asymptotic results are derived and/or discussed in the references we cite for the dimension reduction techniques that we discuss below. Thus, using PCA is somewhat limited because it does not consider the response variable(s), nor does it have a formal inference mechanism for choosing the “best” number of principal components.

\(\text{ E }(Y|\mathbf X )\) is a p-dimensional surface where, for now, we assume that all of the variables represented by the columns of \(\mathbf X \) are continuous. The notion of inverse regression works with the curve computed by \(\text{ E }(\mathbf X |Y),\) which consists of p one-dimensional regressions. Li [11] introduced sliced inverse regression (SIR), which involves dividing the range of the response Y into H non-overlapping intervals called slices. Letting \(\Sigma = \text {Var}(\mathbf {X})\) and \(\mathbf {Z} = \Sigma ^{-1/2} (\mathbf {X}- \text {E}(\mathbf {X})),\) we then see that \(\mathcal {S}_{Y|\mathbf {X}} = \Sigma ^{-1/2}\mathcal {S}_{Y|\mathbf {Z}}\) [25]. Hence, we can work on the scale of \(\mathbf {Z}.\) Moreover, under the linearity condition, \(\mathcal {S}_{\text {E}(\mathbf {Z}|Y)} \subset \mathcal {S}_{Y|\mathbf {Z}}\) and \(\mathcal {S}\{\text {Var}[\text {E}(\mathbf {Z}|Y)]\} = \mathcal {S}_{\text {E}(\mathbf {Z}|Y)}\) [25]. Thus, we can form the kernel matrix \(M_{SIR} = \text {Var}[\text {E}(\mathbf {Z}|Y)].\)

Let \(\mathbf x _i,\) \(\mathbf z _i,\) and \(y_i\) be the sample versions of their respective unobserved quantities. The algorithm for SIR [11] is as follows:Li [11] contrasted SIR with PCA by noting that the sampling properties of SIR are easy to understand and, thus, make subsequent inference using SIR fairly straightforward. SIR also is developed naturally in the regression setting, whereas PCA has to be applied to the multivariate data consisting of only the predictors and then the response variables are regressed against the transformed predictors; i.e., this approach is called principal components regression. SIR has provided critical insight into various applications, such as to understand the electrochemical process of aluminum smelter plants [26] and for the purpose of direct marketing and new product design for managers of data-rich marketing environments [27].

We next consider the case where \(\mathbf X \) can consist of both continuous and categorical predictor variables. In order to accommodate this in a setup similar to SIR, we need to use the notion of partial dimension reduction as introduced in Chiaramonte et al. [12]. Let W be a categorical variable with K levels and define the partial central subspace relative to \(\mathbf {X}\) as the intersection of all subspaces spanned by \(\varvec{\eta }\in \mathbb {R}^{p\times d}\) such that \(Y\perp \mathbf {X} | (\varvec{\eta }^T \mathbf {X}, W).\) Denote the partial central subspace as \(\mathcal {S}^{W}_{Y|\mathbf {X}}.\) The relationship between partial and conditional dimension reduction is \( \mathcal {S}^W_{Y|\mathbf {X}} = \bigoplus ^K_{k=1} \mathcal {S}_{Y_k|\mathbf {X}_k}, \) where \(\mathcal {S}_{Y_k|\mathbf {X}_k} \) is the CS conditioned on level k and \(\bigoplus \) is the direct sum.

For each level, the mean and covariance matrix of \(\mathbf {X}_k\) are \(\varvec{\mu }_k\) and \(\Sigma _k,\) respectively. We further assume that the covariance structures are the same across the levels; i.e., \( \Sigma _k = \Sigma _{pool},\) \(k= 1,\ldots , K.\) Now, letting \(\mathbf {Z}_k = \Sigma ^{-1/2}_{pool}(\mathbf {X}_k - \varvec{\mu }_k)\) results in \( \mathcal {S}^W_{Y|\mathbf {X}} = \Sigma ^{-1/2}_{pool}\bigoplus ^K_{k=1} \mathcal {S}_{Y_k|\mathbf {Z}_k}. \) Then, we can use SIR for each level to find the kernel matrix \(M_k.\) After averaging these kernel matrices over different levels, we get \(M^W = \sum ^{K}_{k=1} \text {Pr}(W=k) M_k.\)

We now present the algorithm for calculating the sample version of \(M^W\) and finding the estimated directions for \(\mathcal {S}^W_{Y|\mathbf {X}}\):The literature on partial SIR has mostly focused on theoretical developments of the approach [12, 28, 29]. However, the efficacy of partial SIR was demonstrated in an analysis involving genomic data and clinically-relevant information in predicting survival of diffuse large-B-cell lymphoma [30]. The utility of partial SIR was also briefly highlighted in production and efficiency analyses of Norwegian electricity distribution networks [31].

One disadvantage of SIR is that it cannot detect symmetric structure of predictors; however, the sliced average variance estimate (SAVE) method [13] can find the directions, even in the presence of symmetric structures. Under the linearity and constant variance condition, \(\mathbf I _p - \text {Var}(\mathbf {Z}|Y) \in \mathcal {S}_{Y|\mathbf {Z}},\) where \(\mathbf I _p\) is the \(p\times p\) identity matrix. Hence, \(M_{SAVE} = \text {E}[\mathbf I _p - \text {Var}(\mathbf {Z}|Y)]^2.\)

The algorithm for SAVE [13] is as follows:SAVE has also been used for various applications and different data structures. For example, Bura and Pfeiffer [32] successfully used SAVE for class prediction of DNA microarray data. They also provided a discussion of some visualizations that can be used in such an analysis. SAVE has also been used as an effective tool in image analysis to distinguish between different facial expressions on the same person’s face [33].

SIR and SAVE are both inverse regression approaches; i.e., we treat Y as if it were the independent variable and \(\mathbf X \) as if it were the dependent variable. Another type of dimension reduction is the principal Hessian direction (PHD) method [14, 15], which is a correlation (or joint) approach. Two common types of PHDs—of which there are others—are calculated as follows:The algorithm for PHD is identical to that used for SIR and SAVE, except we use the sample version of the kernel matrix corresponding to whichever of the PHDs above is of interest.

PHD has also been used for other complex data problems. For example, Cheng and Li [34] demonstrated the efficacy of using PHD in designed experiments having a large number of factors, with particular attention given to factorial designs and rotatable response surface designs. Lue [35] used PHD in the context of a regression analysis when the predictors are known to have measurement error. Lue et al. [36] showed how an imputed-spline modification to PHD yields an effective framework for conducting dimension reduction in survival regressions with censored data.

For our analysis, we use the 2015 Planning Database (PDB) [48], a publicly available Census Bureau dataset that contains housing, demographic, socioeconomic, and Census operational data. The variables and counts in the PDB are from the 2010 Census and select 5-year estimates from the 2009–2013 ACS. The data are aggregated at the block-group level. A census block is the smallest geographic unit used by the Census Bureau, and a block group comprises multiple blocks, usually containing between 600 and 3000 people. The PDB comprises approximately 220,000 block groups.

Three separate response variables are investigated for our analysis: the number of people with no health insurance coverage \((Y_1),\) the number of people with one type of health insurance coverage \((Y_2),\) and the number of people with two or more types of health insurance coverage \((Y_3)\). While these could be treated as a multivariate response, we will analyze three separate models to be consistent with the dimension reduction procedures in “Dimension reduction techniques” section, which were developed assuming a univariate response. A total of 15 variables were identified as relevant candidate predictor variables. The descriptions from the PDB documentation for these variables are given in the Additional files 1, 2.

There are a total of 220,354 records in the 2015 PDB for potential analysis. We first excluded observations from the Commonwealth of Puerto Rico, which is often done due to different laws and demographic considerations involving the Commonwealth; see “Dimension reduction techniques” section of Young et al. [9] for an example of excluding Puerto Rico. The number of Puerto Rico records is 2594, which is about 1.18% of the total number 2015 PDB records. We then omitted records that had missing values for any of the variables under consideration. There are 8754 such records, which is about 3.98% of the total number of 2015 PDB records. This left us with 209,006 records for our analysis. We then transformed the predictors using the maximum likelihood approach of Box and Cox [49] in order to ensure that the linearity condition for dimension reduction is satisfied.

The first part of our analysis focuses on reducing the dimension of our data. Each of the dimension reduction techniques discussed in “Dimension reduction techniques” section are able to be performed using functions available for the R programming language [50].

We first assess the presence of multicollinearity. Table 1 provides the variance inflation factors (VIFs)—a measure of the severity of multicollinearity in an ordinary least squares setting—for the 15 predictor variables. While we are not simply fitting linear models for our analysis, the use of VIFs in this context still provides a reasonable assessment of multicollinearity. Typically, VIF values greater than 10 indicate possible influence on the least squares estimates [51, Chapter 9]. In Table 1, five variables exceed this threshold. Two of these variables—\(X_{4}\) and \(X_{5}\)—would be expected to yield high VIFs. They are both measures of the number of ACS households with individuals who live “alone,” but the former is in the context of those who live with non-relatives. While these could be essentially measuring the same effect, we will retain both of these variables for the purpose of demonstrating the efficacy of the different dimension reduction methods.

We use PCA to characterize those principal components explaining the most variation among the dataset. While Johnson and Wichern [16] state that there is “no definitive answer” to determine “how many components to retain,” we proceed to use a scree plot. The scree plot consists of the principal components ordered according to their amount of variability explained on the x-axis and the cumulative proportion of the variability explained on the y-axis. The scree plot for the health insurance data is given in Fig. 1. We use 0.90 as the threshold to determine the number of principal components to select. Using this criterion, we select six principal components, which will be used for comparison with the subsequent analysis. These cumulative probabilities are also reported in Table 2.

Principal component analysis does not depend on the response variable, so the same six principal components would then be used as the predictors in the principal components regression for each of the three responses. The sufficient dimension reduction methods do, however, take into consideration the value of the independent variable. Thus, we could potentially find a different dimension for each of the three response variables.

For each of the sufficient dimension reduction procedures, testing is done to determine the dimensions. These marginal tests, based on the work in Cook [52] and Shao et al. [53], are also available in R. The tests are done sequentially, where we first test 0 dimensions versus 1 dimension, 1 dimension versus 2 dimensions, etc. Based on these tests, the dimensions selected for each of the sufficient dimension reduction procedures are summarized in Table 3. The full test results are given in the Additional files 1, 2.

SAVE, yPHD, and rPHD did not reduce the number of dimensions much or at all. Recall that partial SIR can be used when including categorical variables. A categorical variable was constructed where we partitioned the 50 states and the District of Colombia using the nine Census-designated geographical divisions [54]. The inclusion of this categorical predictor only yielded a moderate reduction according to partial SIR. The only sufficient dimension reduction procedure that noticeably reduces the dimension for each of the three responses is SIR. Therefore, the remainder of our analysis will focus on the results from PCA and SIR.

In order to use our results from PCA and SIR, we first transform the original predictor variables using the computed principal components and directional vectors, respectively. For each of the three responses, the coefficients for both methods are given in the Additional files 1, 2. We then fit an additive model [55] to each response; i.e., we fit the modelsfor \(k=1,2,3,\) where the \(f_j\) are unknown smooth functions of the transformed data, \(d_k\) is the dimension for the PCA or SIR results, and \(\gamma _0\) is an intercept term. Thus, a total of six additive models are estimated.

For each estimated additive model, approximate t-tests can be constructed to determine the significance of each smooth term. For each of the three additive models constructed using the PCA results, all six terms are highly significant at the \(\alpha =0.05\) level. For the SIR results we found the following:Thus, we drop the fourth and sixth terms from our models with \(Y_1\) and \(Y_2\) as the response, respectively, and refit the additive models. The approximate tests for all of the remaining terms yield significant results.

We also assessed the partial residual plots for each of the six fits. In the context of our additive models, the partial residual plots help us assess the relationship between the response variable and each smooth term, given that the other smooth terms are in the model. Figures 2 and 3 are the partial residual plots for the additive models based on the PCA predictors and SIR predictors, respectively, with \(Y_1\) the response. In Fig. 2, the additive model captures some curvature for the effects due to the first four principal components (labeled as “dimensions”) in this fit. In Fig. 3, the additive model captures some curvature for the effect due to the first dimension in this fit. Similar assessments can be made for the remaining four additive models. The corresponding partial residual plots are included in the Additional files 1, 2.

We next calculated the Bayesian information criterion (BIC) and adjusted \(R^2\) values to compare the estimated additive models for each response. These results are given in Table 4. For each of the three responses, SIR yields the better BIC and adjusted \(R^2\) values. While these measures do not provide direct comparisons between the models based on the different responses, it is worth noting that the adjusted \(R^2\) values for the models with one insurance as a response (\(Y_2\)) are quite high relative to the other models. This indicates that there is little improvement that could be made to those estimated models by adding another set of PCA-transformed or SIR-transformed predictors.

Finally, we also assess the residuals from the additive models at the state level. Figure 4 provides maps of the United States, where the states have been shaded according to the mean of the residuals from the respective additive model built using the PCA-transformed predictors (maps in the left column) and the SIR-transformed predictors (maps in the right column). The three rows of maps correspond to those models for individuals with no insurance (\(Y_1\)), with only one insurance (\(Y_2\)), and with more than one insurance (\(Y_3\)). Notice that each pair of maps for a given response (i.e., the maps within each row) show similar distributions of the mean residuals at the state level. In particular, the maps corresponding to the additive models for \(Y_1\) (Fig. 4a, b both show the same states with larger positive residuals, which have darker shading. These states include Nevada, Texas, Florida, and Alaska. The maps corresponding to the additive models for \(Y_2\) (Fig. 4c, d both show that regions with larger negative residuals (lighter shading) appear mostly in the Western states while regions with larger positive residuals (darker shading) appear mostly in the Midwest. Finally, the maps corresponding to the additive models for \(Y_3\) (Fig. 4e, f both have shading indicating residuals with overwhelmingly small magnitude. However, the one state indicated with a larger positive residual on both maps is Hawaii. Overall, these maps indicate that both dimension reduction strategies yield similar results for the models built for each of the three responses. Further improvements could be explored using models that, for example, include a spatial component.