The Multiple Facets of Partial Least Squares and Related Methods

Large-scale data, where the sample size and the dimension are high, often exhibits heterogeneity. This can arise for example in the form of unknown subgroups or clusters, batch effects or contaminated samples. Ignoring these issues would often lead to poor prediction and estimation. We advocate the maximin effects framework (Meinshausen and Bühlmann, Maximin effects in inhomogeneous large-scale data. Preprint arXiv:1406.0596, 2014) to address the problem of heterogeneous data. In combination with partial least squares (PLS) regression, we obtain a new PLS procedure which is robust and tailored for large-scale heterogeneous data. A small empirical study complements our exposition of new PLS methodology.

Partial least squares (PLS) was first introduced by Wold in the mid 1960s as a heuristic algorithm to solve linear least squares (LS) problems. No optimality property of the algorithm was known then. Since then, however, a number of interesting properties have been established about the PLS algorithm for regression analysis (called PLS1). This paper shows that the PLS estimator for a specific dimensionality S is a kind of constrained LS estimator confined to a Krylov subspace of dimensionality S. Links to the Lanczos bidiagonalization and conjugate gradient methods are also discussed from a somewhat different perspective from previous authors.

Given q + p variables (q endogenous variables and p exogenous variables) and the covariance matrix among exogenous variables, how to compute the covariance matrix implied by a given recursive path model connecting these q + p variables? The finite iterative method (FIM) was recently introduced by El Hadri and Hanafi (Electron J Appl Stat Anal 8:84–99, 2015) to perform this task but only when all the variables are standardized (and so the covariance matrix is actually a correlation matrix). In this paper, the extension of FIM to the general case of a covariance matrix case is introduced. Moreover, the computational efficiency of FIM and the well-known Jöreskog’s method is discussed and illustrated.

PLS-PM presents some inconsistencies in terms of coherence with the direction of the relationships specified in the path diagram (i.e., the path directions). The PLS-PM iterative algorithm analyzes interdependence among blocks and misses to distinguish explicitly between dependent and explanatory blocks in the structural model. This inconsistency of PLS-PM is illustrated using the simple two-blocks model. For the case of more than two blocks of variables, it is necessary to have a close look at the different criteria optimized by PLS-PM to show this issue. In general, the role of latent variables in the structural model depends on the way the outer weights are calculated. A recently proposed method, called Non-Symmetrical Component-based Path Modeling, which is based on the optimization of a redundancy-related criterion in a multi-block framework, respects the direction of the relationships specified in the structural model. In order to assess the quality of the model, we provide a new goodness-of-fit index based on redundancy criterion and prediction capability. Furthermore, we provide a procedure to address the problem of multicollinearity within blocks of variables.

“Imaging genetics” studies the genetic contributions to brain structure and function by finding correspondence between genetic data—such as single nucleotide polymorphisms (SNPs)—and neuroimaging data—such as diffusion tensor imaging (DTI). However, genetic and neuroimaging data are heterogenous data types, where neuroimaging data are quantitative and genetic data are (usually) categorical. So far, methods used in imaging genetics treat all data as quantitative, and this sometimes requires unrealistic assumptions about the nature of genetic data. In this article we present a new formulation of Partial Least Squares Correlation (PLSC)—called Mixed-modality Partial Least Squares (MiMoPLS)—specifically tailored for heterogeneous (mixed-) data types. MiMoPLS integrates features of PLSC and Correspondence Analysis (CA) by using special properties of quantitative data and Multiple Correspondence Analysis (MCA). We illustrate MiMoPLS with an example data set from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) with DTI and SNPs.

Correlation Partial-Least Squares (PLSC) provides a robust model for analyzing functional neuroimaging data, which is used to identify functional brain networks that show the largest covariance with task stimuli. However, neuroimaging data tend to be high-dimensional (i.e., there are far more variables P than samples N), with significant noise confounds and variability in brain response. It is therefore challenging to identify the significant, stable components of PLSC analysis. Empirical significance estimators are widely used, as they make minimal assumptions about data structure. The most common estimator in neuroimaging PLS is Bootstrapped Variance (BV), which tests whether bootstrap-stabilized mean component eigenvalues (i.e., covariance) are significantly different from a permuted null distribution; however, recent studies have highlighted issues with this model. Two alternatives were proposed that instead focus on reliability of the PLSC saliences (i.e., singular vectors): a Split-half Stability (SS) model that measures the consistency of reconstructed components for split-half data, and Split-half Reproducibility (SR) which measures the reliability across independent split-half analyses. We compare BV, SS, and SR estimators on functional Magnetic Resonance Imaging (f MRI) data, for both simulated and experimental datasets. The SS and SR methods have comparable sensitivity in detecting “brain” components for most simulated and experimental conditions. However, SR shows consistently greater sensitivity for “task” components. We demonstrate that this is due to relative bias in the SS model: both “brain” and “task” components have biased null distributions, but for the low-dimensional “task” vectors, this bias becomes sufficiently high that it is often impossible to distinguish a significant effect from the null distribution.

In studies involving genetic data, the correlations between X and Y scores obtained from PLS regression models can be used as measures of association between genome-level measurements, X, and phenotype-level measurements, Y. These correlations may be overestimated due to potential overfitting (i.e., they may be vulnerable to optimism bias). We evaluate the optimism bias through simulations and examine the effect of increasing sample size and strength of correlation. We assess the effectiveness of bootstrap-based and permutation-based bias correction methods. We also investigate the selection of the appropriate number of components for PLS regression. We include an analysis of genetic data consisting of genotypes and phenotypes related to Attention Deficit Hyperactivity Disorder (ADHD).

A multiway Fisher Discriminant Analysis (MFDA) formulation is presented in this paper. The core of MFDA relies on the structural constraint imposed to the discriminant vectors in order to account for the multiway structure of the data. This results in a more parsimonious model than that of Fisher Discriminant Analysis (FDA) performed on the unfolded data table. Moreover, computational and overfitting issues that occur with high dimensional data are better controlled. MFDA is applied to predict the long term recovery of patients after traumatic brain injury from multi-modal brain Magnetic Resonance Imaging. As compared to FDA, MFDA clearly tracks down the discrimination areas within the white matter region of the brain and provides a ranking of the contribution of the neuroimaging modalities. Based on cross validation, the accuracy of MFDA is equal to 77 % against 75 % for FDA.

Regularized Generalized Canonical Correlation Analysis (RGCCA) extends regularized canonical correlation analysis to more than two sets of variables. Sparse GCCA (SGCCA) was recently proposed to address the issue of variable selection. However, the variable selection scheme offered by SGCCA is limited to the covariance (τ = 1) link between blocks. In this paper we go beyond the covariance link by proposing an extension of SGCCA for the full RGCCA model (τ ∈ [0, 1]). In addition, we also propose an extension of SGCCA that exploits pre-given structural relationships between variables within blocks. Specifically, we propose an algorithm that allows structured and sparsity-inducing penalties to be included in the RGCCA optimization problem.

We address component-based regularization of a multivariate Generalized Linear Model (GLM). A set of random responses Y is assumed to depend, through a GLM, on a set X of explanatory variables, as well as on a set T of additional covariates. X is partitioned into R conceptually homogeneous blocks X ₁, …, X _R , viewed as explanatory themes. Variables in each X _r are assumed many and redundant. Thus, generalized linear regression demands regularization with respect to each X _r . By contrast, variables in T are assumed selected so as to demand no regularization. Regularization is performed searching each X _r for an appropriate number of orthogonal components that both contribute to model Y and capture relevant structural information in X _r . We propose a very general criterion to measure structural relevance (SR) of a component in a block, and show how to take SR into account within a Fisher-scoring-type algorithm in order to estimate the model. We show how to deal with mixed-type explanatory variables. The method, named THEME-SCGLR, is tested on simulated data, and then applied to rainforest data in order to model the abundance of tree-species.

This paper introduces structural equation modeling for imprecise data, which enables evaluations with different types of uncertainty. Coming under the framework of variance-based analysis, the proposed method called Partial Possibilistic Regression Path Modeling (PPRPM) combines the principles of PLS path modeling to model the network of relations among the latent concepts, and the principles of possibilistic regression to model the vagueness of the human perception. Possibilistic regression defines the relation between variables through possibilistic linear functions and considers the error due to the vagueness of human perception as reflected in the model via interval-valued parameters. PPRPM transforms the modeling process into minimizing components of uncertainty, namely randomness and vagueness. A case study on the motivational and emotional aspects of teaching is used to illustrate the method.

The paper aims to introduce assessment and validation measures in Quantile Composite-based Path modeling. A quantile approach in the Partial Least Squares path modeling framework overcomes the classical exploration of average effects and highlights how and if the relationships among observed and unobserved variables change according to the explored quantile of interest. A final evaluation of the quality of the obtained results both from a descriptive (assessment) and inferential (validation) point of view is needed. The functioning of the proposed method is shown through a real data application in the area of the American Customer Satisfaction Index.

Functional linear model with functional predictors and scalar response is a simple and popular model in the field of functional data analysis. The slope function is usually expanded on some basis functions, such as spline and functional principal component (FPC) basis, and then the model can be converted into a multivariate linear model. The FPC basis can keep most variance information of the functional data, but the correlation with response is not considered. Motivated by this, we use partial least square basis to expand the slope function. Meanwhile, considering the functional predictors are not all significant and variable selection procedure is implemented. In this process, group variable selection is introduced to identify the significant predictors. Then the proposed method is used to analyse the relationship between number of monthly emergency patients and some environmental factors in functional form, and some meaningful results are obtained.

We address the problem of investigating the relationships between (K + 1) blocks of variables (i.e., K blocks of independent variables and one block of dependent variables), where the observations are a priori divided into several known groups. We propose a simple procedure called multiblock and multigroup PLS regression—which is a straightforward extension of multiblock PLS regression—that takes into account the group structure of the observations. This method of analysis is illustrated with a large, questionnaire based, survey exploring, in 2011, the cannabis consumption of teenagers of thirteen European countries (the European School Survey Project on Alcohol and other Drugs).

In this paper we propose a new approach to study the properties of the Partial Least Squares (PLS) vector of regression coefficients. This approach relies on the link between PLS and discrete orthogonal polynomials. In fact many important PLS objects can be expressed in terms of some specific discrete orthogonal polynomials, called the residual polynomials. Based on the explicit analytical expression we have stated for these polynomials in terms of signal and noise, we provide a new framework for the study of PLS. We show that this approach allows to simplify and retrieve independent proofs of many classical results (proved earlier by different authors using various approaches and tools). This general and unifying approach also sheds light on PLS and helps to gain insight on its properties.

We develop a new universal stopping criterion in components construction, in the sense that it is suitable both for Partial Least Squares Regressions (PLSR) and its extension to Generalized Linear Regressions (PLSGLR). This criterion is based on a bootstrap method and has to be computed algorithmically. It allows to test each successive components on a significant level α. In order to assess its performances and robustness with respect to different noise levels, we perform intensive datasets simulations, with a preset and known number of components to extract, both in the case N > P (N being the number of subjects and P the number of original predictors), and for datasets with N < P. We then use t-tests to compare the predictive performance of our approach to some others classical criteria. Our conclusion is that our criterion presents better performances, both in PLSR and PLS-Logistic Regressions (PLS-LR) frameworks.

In this paper we propose an extension to the PATHMOX segmentation algorithm to detect which endogenous latent variables and predictors are responsible for heterogeneity. We also address the problem of factor invariance in the terminal nodes of PATHMOX. We demonstrate the utility of such methodology on real mental health data by investigating the relationship between dementia, depression and delirium.

The partial least squares (PLS) is a popular path modeling technique commonly used in social sciences. The traditional PLS algorithm deals with variables measured on interval scales while data are often collected on ordinal scales. A reformulation of the algorithm, named Ordinal PLS (OrdPLS), is introduced, which properly deals with ordinal variables. Some simulation results show that the proposed technique seems to perform better than the traditional PLS algorithm applied to ordinal data as they were metric, in particular when the number of categories of the items in the questionnaire is small (4 or 5) which is typical in the most common practical situations.