Skip to main content
SpringerLink
Log in

From simple structure to sparse components: a review

  • Original Paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

The article begins with a review of the main approaches for interpretation the results from principal component analysis (PCA) during the last 50–60 years. The simple structure approach is compared to the modern approach of sparse PCA where interpretable solutions are directly obtained. It is shown that their goals are identical but they differ by the way they are realized. Next, the most popular and influential methods for sparse PCA are briefly reviewed. In the remaining part of the paper, a new approach to define sparse PCA is introduced. Several alternative definitions are considered and illustrated on a well-known data set. Finally, it is demonstrated, how one of these possible versions of sparse PCA can be used as a sparse alternative to the classical rotation methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bach F (2008) BOLASSO: model consistent LASSO estimation through the bootstrap. In: ICML ’08 proceedings of the 25th international conference on machine learning. ACM Press, New York, pp 33–40

  • Browne MW (2001) An overview of analytic rotation in exploratory factor analysis. Multivar Behav Res 36:111–150

    Article  Google Scholar 

  • Cadima J, Jolliffe IT (1995) Loadings and correlations in the interpretations of principal components. J Appl Stat 22:203–214

    Article  MathSciNet  Google Scholar 

  • Cadima J, Jolliffe IT (2001) Variable selection and the interpretation of principal subspaces. J Agric Biol Environ Stat 6:62–79

    Article  MathSciNet  Google Scholar 

  • Cai T, Ma Z, Wu Y (2012) Sparse PCA: optimal rates and adaptive estimation. http://arxiv.org/abs/1211.1309

  • Candès EJ, Tao T (2007) The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). Ann Stat 35:2313–2351

    Article  MATH  Google Scholar 

  • Candès EJ, Wakin M, Boyd SP (2008) Enhancing sparsity by reweighted \(\ell _1\) minimization. J Fourier Anal Appl 14:877–905

    Article  MATH  MathSciNet  Google Scholar 

  • Chipman HA, Gu H (2005) Interpretable dimension reduction. J Appl Stat 32:969–987

    Article  MATH  MathSciNet  Google Scholar 

  • Chu MT, Trendafilov NT (1998) ORTHOMAX rotation problem. A differential equation approach. Behaviormetrika 25:13–23

    Article  Google Scholar 

  • d’Aspremont A, Ghaoui L, Jordan M, Lanckriet G (2007) A direct formulation for sparse PCA using semidefinite programming. SIAM Rev 49:434–448

    Article  MATH  MathSciNet  Google Scholar 

  • d’Aspremont A, Bach F, Ghaoui L (2008) Optimal solutions for sparse principal component analysis. J Mach Learn Res 9:1269–1294

    MATH  MathSciNet  Google Scholar 

  • Diele F, Lopez L, Peluso R (1998) The Cayley transform in the numerical solution of unitary differential systems. Adv Comput Math 8:317–334

    Article  MATH  MathSciNet  Google Scholar 

  • Ding X, He L, Carin L (2011) Bayesian robust principal component analysis. IEEE Trans Image Process 20:3419–3430

    Article  MathSciNet  Google Scholar 

  • Donoho DL, Johnstone IM (1994) Ideal spatial adaptation via wavelet shrinkage. Biometrika 81:425–455

    Article  MATH  MathSciNet  Google Scholar 

  • Eckart C, Young G (1936) The approximation of one matrix by another of lower rank. Psychometrika 1:211–218

    Article  MATH  Google Scholar 

  • Edelman A, Arias TA, Smith ST (1998) The geometry of algorithms with orthogonality constraints. SIAM J Matrix Anal Appl 20:303–353

    Article  MATH  MathSciNet  Google Scholar 

  • Enki D, Trendafilov NT (2012) Sparse principal components by semi-partition clustering. Comput Stat 4:605–626

    Article  MathSciNet  Google Scholar 

  • Enki D, Trendafilov NT, Jolliffe IT (2013) A clustering approach to interpretable principal components. J Appl Stat 3:583–599

    Article  MathSciNet  Google Scholar 

  • Friedlander M, Tseng P (2007) Exact regularization of convex programs. SIAM J Optim 4:1326–1350

    MathSciNet  Google Scholar 

  • Guan Y, Dy J (2009) Sparse probabilistic principal component analysis. Proc Twelfth Int Conf Artif Intell Stat 5:185–192

    Google Scholar 

  • Guo F, Gareth J, Levina E, Michailidis G, Zhu J (2010) Principal component analysis with sparse fused loadings. J Comput Graph Stat 19:947–962

    Article  Google Scholar 

  • Hannachi A, Jolliffe IT, Stephenson DB, Trendafilov NT (2006) In search of simple structures in climate: simplifying EOFs. Int J Climatol 26:7–28

    Article  Google Scholar 

  • Harman HH (1976) Modern factor analysis, 3rd edn. University of Chicago Press, Chicago

    Google Scholar 

  • Hausman RE (1982) Constrained multivariate analysis. In: Zanakis SH, Rustagi JS (eds) Optimization in statistics. North-Holland, Amsterdam, pp 137–151

  • Hotelling H (1933) Analysis of a complex of statistical variables into principal components. J Educ Psychol 24(417–441):498–520

    Article  Google Scholar 

  • Jeffers JNR (1967) Two case studies in the application of principal component analysis. Appl Stat 16:225–236

    Article  Google Scholar 

  • Jennrich RI (2007) Rotation methods, algorithms, and standard errors. In: Cudeck R, MacCallum RC (eds) Factor analysis at 100. Lawrens Erlbaum Associates, Mahwah, NJ, pp 315–335

  • Johnstone IM, Lu AY (2009) On consistency and sparsity for principal components analysis in high dimensions. J Am Stat Assoc 104:682–693

    Article  MathSciNet  Google Scholar 

  • Jolliffe IT (2002) Principal component analysis, 2nd edn. Springer, New York

    MATH  Google Scholar 

  • Jolliffe IT, Uddin M (2000) The simplified component technique: An alternative to rotated principal components. J Comput Graph Stat 9:689–710

    MathSciNet  Google Scholar 

  • Jolliffe IT, Trendafilov NT, Uddin M (2003) A modified principal component technique based on the LASSO. J Comput Graph Stat 12:531–547

    Article  MathSciNet  Google Scholar 

  • Journée M, Nesterov Y, Richtárik P, Sepulchre R (2010) Generalized power method for sparse principal component analysis. J Mach Learn Res 11:517–553

    MATH  MathSciNet  Google Scholar 

  • Lu Z, Zhang Y (2012) An augmented Lagrangian approach for sparse principal component analysis. Math Program Ser A 135:149–193

    Article  MATH  Google Scholar 

  • Marshall A, Olkin I (1979) Inequalities: theory of majorization and its applications. Academic Press, London

    MATH  Google Scholar 

  • MATLAB (2011) MATLAB R2011a. The MathWorks, Inc., New York

  • Moghaddam B, Weiss Y, Avidan S (2006) Spectral bounds for sparse PCA: exact and greedy algorithms. Adv Neural Inf Process Syst 18:915–922

    Google Scholar 

  • Mulaik SA (2010) The foundations of factor analysis, 2nd edn. Chapman and Hall/CRC, Boca Raton, FL

    Google Scholar 

  • Paul D, Johnstone IM (2007) Augmented sparse principal component analysis for high dimensional data. http://arxiv.org/abs/1202.1242

  • Pearson K (1901) On lines and planes of closest fit to systems of points in space. Philos Mag 2:559–572

    Article  Google Scholar 

  • Qi X, Luo R, Zhao H (2013) Sparse principal component analysis by choice of norm. J Multivar Anal 114:127–160

    Article  MATH  MathSciNet  Google Scholar 

  • Richtárik P, Takáč M, Ahipaşaoğlu SD (2012) Alternating maximization: unifying framework for 8 sparse PCA formulations and efficient parallel codes. http://www.maths.ed.ac.uk/~richtarik/24AM.pdf

  • Rousson V, Gasser T (2004) Simple component analysis. Appl Stat 53:539–555

    MATH  MathSciNet  Google Scholar 

  • Shen H, Huang JZ (2008) Sparse principal component analysis via regularized low-rank matrix approximation. J Multivar Anal 99:1015–1034

    Article  MATH  MathSciNet  Google Scholar 

  • Sriperumbudur BK, Torres DA, Lanckriet GRG (2011) A majorization-minimization approach to the sparse generalized eigenvalue problem. Mach Learn 85:3–39

    Article  MATH  MathSciNet  Google Scholar 

  • Thurstone LL (1935) The vectors of mind. University of Chicago Press, Chicago, IL

    MATH  Google Scholar 

  • Thurstone LL (1947) Multiple factor analysis. University of Chicago Press, Chicago, IL

    MATH  Google Scholar 

  • Tibshirani R (1996) Regression shrinkage and selection via the LASSO. J R Stat Soc 58:267–288

    MATH  MathSciNet  Google Scholar 

  • Trendafilov NT (1999) A continuous-time approach to the oblique Procrustes problem. Behaviormetrika 26:167–181

    Article  Google Scholar 

  • Trendafilov NT, Jolliffe IT (2006) Projected gradient approach to the numerical solution of the SCoTLASS. Comput Stat Data Anal 50:242–253

    Article  MATH  MathSciNet  Google Scholar 

  • Trendafilov NT, Lippert RA (2002) The multimode Procrustes problem. Linear Algebra Appl 349(1–3):245–264

    Article  MATH  MathSciNet  Google Scholar 

  • Vichi M, Saporta G (2009) Clustering and disjoint principal component analysis. Comput Stat Data Anal 53:3194–3208

    Article  MATH  MathSciNet  Google Scholar 

  • Vines SK (2000) Simple principal components. Appl Stat 49:441–451

    MATH  MathSciNet  Google Scholar 

  • Witten DM, Tibshirani R, Hastie T (2009) A penalized matrix decomposition, with applications to sparse principal components and canonical correlation. Biostatistics 10:515–534

    Article  Google Scholar 

  • Wright S (2011) Gradient algorithms for regularized optimization. SPARS11, Edinburgh, Scotland. http://pages.cs.wisc.edu/~swright

  • Zou H, Hastie T, Tibshirani R (2006) Sparse principal component analysis. J Comput Graph Stat 15:265–286

    Article  MathSciNet  Google Scholar 

  • Zou H, Hastie T, Tibshirani R (2007) On the ”degrees of freedom” of the LASSO. Ann Stat 35:2173–2192

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

I thank the Editor, the Associate Editor, and the anonymous reviewers for their careful work and for the many helpful comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Mathematics and Statistics, The Open University, Walton Hall, Milton Keynes, MK7 6AA, UK

    Nickolay T. Trendafilov

Authors
  1. Nickolay T. Trendafilov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nickolay T. Trendafilov.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Trendafilov, N.T. From simple structure to sparse components: a review. Comput Stat 29, 431–454 (2014). https://doi.org/10.1007/s00180-013-0434-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00180-013-0434-5

Keywords

Search

Navigation