Skip to main content
Erschienen in: International Journal of Machine Learning and Cybernetics 6/2014

01.12.2014 | Original Article

A comparison of ℓ1-regularizion, PCA, KPCA and ICA for dimensionality reduction in logistic regression

verfasst von: Abdallah Bashir Musa

Erschienen in: International Journal of Machine Learning and Cybernetics | Ausgabe 6/2014

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Relevant information extraction and dimensionality reduction of the original input features is an interesting research area in machine learning and data analysis. Logistic regression (LR) is a well-known classification method that has been used widely in many applications of data mining, machine learning, and bioinformatics. However, its performance is affected by the multi-co-linearity among its predictors, and the features’ redundancy. ℓ1-regularizion and features extraction methods are commonly used to enhance the performance of logistic regression under multi-co-linearity and ovefitting problems, and to reduce computational complexity by discarding less relevant or redundant features. These methods include principal component analysis, kernel principal component analysis and independent component analysis. Recently, ℓ1-regularized logistic regression has received much attention as a promising method for features selection in classification tasks. So there is a great need to be compared with these existing methods. In this paper, we assess the performance of the aforementioned feature selection methods on LR and ℓ1-regularized logistic regression using different statistical measures. A variety of performance metrics has been utilized: accuracy, sensitivity, specificity, precision, the area under receiver operating characteristic curve and the receiver operating characteristic analysis. This study is distinct by its inclusion of a comprehensive statistical analysis.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Weitere Produktempfehlungen anzeigen
Literatur
1.
Zurück zum Zitat Hosmer DW, Lemeshow S (2000) Applied logistic regression. Wiley series in probability and statistics, 2nd edn. Wiley, New York Hosmer DW, Lemeshow S (2000) Applied logistic regression. Wiley series in probability and statistics, 2nd edn. Wiley, New York
2.
Zurück zum Zitat Menard S (2002) Applied logistic regression analysis, 2nd edn. Sage publications Inc, UK Menard S (2002) Applied logistic regression analysis, 2nd edn. Sage publications Inc, UK
3.
Zurück zum Zitat Neter J, Kutner MH, Nachtsheim CJ, Wasserman W (1996) Applied linear statistical models, 4th edn. Irwin, Chicago Neter J, Kutner MH, Nachtsheim CJ, Wasserman W (1996) Applied linear statistical models, 4th edn. Irwin, Chicago
4.
Zurück zum Zitat Ryan TP (2008) Modern regression methods, 2nd edn. Wiley, New York Ryan TP (2008) Modern regression methods, 2nd edn. Wiley, New York
5.
Zurück zum Zitat Brzezinski JR Knafl GJ (1999) Logistic regression modeling for context-based classification. In: Proceedings tenth international workshop on database and expert systems applications 1999, pp 755–759. doi:10.1109/DEXA.1999.795279 Brzezinski JR Knafl GJ (1999) Logistic regression modeling for context-based classification. In: Proceedings tenth international workshop on database and expert systems applications 1999, pp 755–759. doi:10.​1109/​DEXA.​1999.​795279
6.
Zurück zum Zitat Liao JG, Chin K-V (2007) Logistic regression for disease classification using microarray data: model selection in a large p and small n case. Bioinformatics 23(15):1945–1951 Liao JG, Chin K-V (2007) Logistic regression for disease classification using microarray data: model selection in a large p and small n case. Bioinformatics 23(15):1945–1951
7.
Zurück zum Zitat Sartor MA, Leikauf GD, Medvedovic Lrpath M (2008) A logistic regression approach for identifying enriched biological groups in gene expression data. Bioinformatics 25(2):211–217CrossRef Sartor MA, Leikauf GD, Medvedovic Lrpath M (2008) A logistic regression approach for identifying enriched biological groups in gene expression data. Bioinformatics 25(2):211–217CrossRef
8.
Zurück zum Zitat Asgary MP, Jahandideh S, Abdolmaleki P, Kazemnejad A (2007) Analysis and identification of β-turn types using multinomial logistic regression and artificial neural network. Bioinformatics 23(23):3125–3130 Asgary MP, Jahandideh S, Abdolmaleki P, Kazemnejad A (2007) Analysis and identification of β-turn types using multinomial logistic regression and artificial neural network. Bioinformatics 23(23):3125–3130
10.
Zurück zum Zitat Kwak N, Kim C, Kim H (2008) Dimensionality reduction based on ICA for regression problems. Neurocomputing 71(13–15):2596–2603 Kwak N, Kim C, Kim H (2008) Dimensionality reduction based on ICA for regression problems. Neurocomputing 71(13–15):2596–2603
11.
12.
Zurück zum Zitat Wainwright M, Ravikumar P, Lafferty J (2007) High-dimensional graphical model selection using ℓ1-regularized logistic regression. To appear in advances in neural information processing systems (NIPS) 19 Wainwright M, Ravikumar P, Lafferty J (2007) High-dimensional graphical model selection using ℓ1-regularized logistic regression. To appear in advances in neural information processing systems (NIPS) 19
14.
Zurück zum Zitat Genkin A, Lewis DD, Madigan D (2007) Large-scale Bayesian logistic regression for text categorization. Technometrics 49(3):291–304CrossRefMathSciNet Genkin A, Lewis DD, Madigan D (2007) Large-scale Bayesian logistic regression for text categorization. Technometrics 49(3):291–304CrossRefMathSciNet
15.
Zurück zum Zitat Wang X, Dong L, Yan J (2012) Maximum ambiguity based sample selection in fuzzy decision tree induction. IEEE Trans Knowl Data Eng 24(8):1491–1505CrossRef Wang X, Dong L, Yan J (2012) Maximum ambiguity based sample selection in fuzzy decision tree induction. IEEE Trans Knowl Data Eng 24(8):1491–1505CrossRef
16.
Zurück zum Zitat Cao LJ, Chua KS, Chong WK, Lee HP, Gu QM (2003) A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine, Neurocomputing 55(1–2):321–336 Cao LJ, Chua KS, Chong WK, Lee HP, Gu QM (2003) A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine, Neurocomputing 55(1–2):321–336
17.
Zurück zum Zitat Cai LJ, Zhang JQ, Zongwu CAI, Kian Guan LIM (2006) An empirical study of dimensionality reduction in support vector machine. Neural Network World, pp 177–192 Cai LJ, Zhang JQ, Zongwu CAI, Kian Guan LIM (2006) An empirical study of dimensionality reduction in support vector machine. Neural Network World, pp 177–192
18.
Zurück zum Zitat Cao LJ, Chong WK (2002) Feature extraction in support vector machine: a comparison of PCA, XPCA and ICA. In: Proceedings of the 9th international conference on neural information processing 2002, ICONIP ‘02, vol 2, pp 1001–1005. doi:10.1109/ICONIP.2002.1198211 Cao LJ, Chong WK (2002) Feature extraction in support vector machine: a comparison of PCA, XPCA and ICA. In: Proceedings of the 9th international conference on neural information processing 2002, ICONIP ‘02, vol 2, pp 1001–1005. doi:10.​1109/​ICONIP.​2002.​1198211
19.
Zurück zum Zitat Lerner B, Guterman H, Aladjem M, Dinstein I (1999) A comparative study of neural network based feature extraction paradigms. Pattern Recogn Lett 20(1):7–14 Lerner B, Guterman H, Aladjem M, Dinstein I (1999) A comparative study of neural network based feature extraction paradigms. Pattern Recogn Lett 20(1):7–14
20.
Zurück zum Zitat Ekenel HK, Sankur B (2004) Feature selection in the independent component subspace for face recognition. Pattern Recogn Lett 25:1377–1388CrossRef Ekenel HK, Sankur B (2004) Feature selection in the independent component subspace for face recognition. Pattern Recogn Lett 25:1377–1388CrossRef
21.
Zurück zum Zitat Aguilera AM, Escabias M, Valderrama MJ (2006) Using principal components for estimating logistic regression with high-dimensional multi collinear data. Comput Stat Data Anal 50(8):1905–1924CrossRefMATHMathSciNet Aguilera AM, Escabias M, Valderrama MJ (2006) Using principal components for estimating logistic regression with high-dimensional multi collinear data. Comput Stat Data Anal 50(8):1905–1924CrossRefMATHMathSciNet
22.
Zurück zum Zitat Xiang D (2010) The listed company’s financial evaluation based on PCA-logistic regression model. In: Second international conference on multimedia and information technology (MMIT) 2010, vol 2, pp 168–171, 24–25. doi:10.1109/MMIT.2010.148 Xiang D (2010) The listed company’s financial evaluation based on PCA-logistic regression model. In: Second international conference on multimedia and information technology (MMIT) 2010, vol 2, pp 168–171, 24–25. doi:10.​1109/​MMIT.​2010.​148
23.
Zurück zum Zitat Liu Z, Chen D, Bensmail H (2005) Gene expression data classification with kernel principal component analysis. J. Biomed Biotechnol l2:155–169 Liu Z, Chen D, Bensmail H (2005) Gene expression data classification with kernel principal component analysis. J. Biomed Biotechnol l2:155–169
24.
Zurück zum Zitat Gao Q-S, Xue F-Z (2011) Applications of the kernel principal component analysis-based logistic regression model on nonlinear association study. J Shandong Univ (health sciences) doi:10.1186/1471-2156-12-75 Gao Q-S, Xue F-Z (2011) Applications of the kernel principal component analysis-based logistic regression model on nonlinear association study. J Shandong Univ (health sciences) doi:10.​1186/​1471-2156-12-75
25.
Zurück zum Zitat Villa A, Chanussot J, Jutten C, Benediktsson JA, Moussaoui S (2009) On the use of ICA for hyperspectral image analysis. In: IEEE international, IGARSS 4:IV-97-IV-100 geoscience and remote sensing symposium. doi:10.1109/IGARSS.2009.5417363 Villa A, Chanussot J, Jutten C, Benediktsson JA, Moussaoui S (2009) On the use of ICA for hyperspectral image analysis. In: IEEE international, IGARSS 4:IV-97-IV-100 geoscience and remote sensing symposium. doi:10.​1109/​IGARSS.​2009.​5417363
26.
Zurück zum Zitat Widodo A, Yang B-S (2007) Application of nonlinear feature extraction and support vector machines for fault diagnosis of induction motors. Expert Syst Appl 33(1):241–250CrossRef Widodo A, Yang B-S (2007) Application of nonlinear feature extraction and support vector machines for fault diagnosis of induction motors. Expert Syst Appl 33(1):241–250CrossRef
27.
Zurück zum Zitat Yu S-N, Chou K-T (2008) Integration of independent component analysis and neural networks for ECG beat classification. Expert Syst Appl 34(4):2841–2846 Yu S-N, Chou K-T (2008) Integration of independent component analysis and neural networks for ECG beat classification. Expert Syst Appl 34(4):2841–2846
28.
Zurück zum Zitat Liwei F (2010) Independent component analysis for naive classification, PhD thesis, National University of Singapore, Singapore Liwei F (2010) Independent component analysis for naive classification, PhD thesis, National University of Singapore, Singapore
29.
Zurück zum Zitat Deniz O, Castrillon M, Hernandez M (2003) Face recognition using independent component analysis and support vector machines. Pattern Recogn Lett 24:2153–2157CrossRef Deniz O, Castrillon M, Hernandez M (2003) Face recognition using independent component analysis and support vector machines. Pattern Recogn Lett 24:2153–2157CrossRef
30.
Zurück zum Zitat Xiang S, Nie F, Meng G, Pan C, Zhang C (2012) Discriminative least squares regression for multiclass classification and feature selection. IEEE Trans Neural Netw Learn Syst 23(11):1738–1754CrossRef Xiang S, Nie F, Meng G, Pan C, Zhang C (2012) Discriminative least squares regression for multiclass classification and feature selection. IEEE Trans Neural Netw Learn Syst 23(11):1738–1754CrossRef
31.
Zurück zum Zitat Kim S-J, Koh K, Lustig M, Boyd S, Gorinevsky D (2007) An interior-point method for largescale l1-regularized least squares. IEEE J Sel Top Sign Process 1(4):606–617CrossRef Kim S-J, Koh K, Lustig M, Boyd S, Gorinevsky D (2007) An interior-point method for largescale l1-regularized least squares. IEEE J Sel Top Sign Process 1(4):606–617CrossRef
32.
Zurück zum Zitat Jolliffe IT (2002) Principle components analysis, 2nd edn. Springer, Verlag Jolliffe IT (2002) Principle components analysis, 2nd edn. Springer, Verlag
33.
Zurück zum Zitat Escabias M, Aguilera AM, Valderrama MJ (2004) principal component estimation of functional logistic regression: discussion of two different approaches. J Nonparametric Stat 16(3–4):365–384 Escabias M, Aguilera AM, Valderrama MJ (2004) principal component estimation of functional logistic regression: discussion of two different approaches. J Nonparametric Stat 16(3–4):365–384
34.
Zurück zum Zitat van der Maaten LJP, Postma EO, van den Herik HJ (2008) Dimensionality reduction: a comparative review. Neurocomputing van der Maaten LJP, Postma EO, van den Herik HJ (2008) Dimensionality reduction: a comparative review. Neurocomputing
35.
Zurück zum Zitat Scholkopf B, Burges C, Smola A (eds) (1999) Advances in kernel methods—support vector learning, MIT Press, Cambridge, pp 327–352 Scholkopf B, Burges C, Smola A (eds) (1999) Advances in kernel methods—support vector learning, MIT Press, Cambridge, pp 327–352
37.
38.
Zurück zum Zitat Tipping ME (2001) Sparse kernel principal component analysis. In: Advances in neural information processing systems 13:633–639 Tipping ME (2001) Sparse kernel principal component analysis. In: Advances in neural information processing systems 13:633–639
39.
40.
Zurück zum Zitat Hyvärinen A, Karhunen J, Oja E (2001) Independent component analysis. Wiley, New YorkCrossRef Hyvärinen A, Karhunen J, Oja E (2001) Independent component analysis. Wiley, New YorkCrossRef
41.
Zurück zum Zitat Hyvarinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw 13:411–430CrossRef Hyvarinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw 13:411–430CrossRef
42.
Zurück zum Zitat Hyvarinen A, Oja E (1997) A fast fixed-point algorithm for independent component analysis. Neural Comput 9(7):483–1492CrossRef Hyvarinen A, Oja E (1997) A fast fixed-point algorithm for independent component analysis. Neural Comput 9(7):483–1492CrossRef
43.
Zurück zum Zitat Musa AB (2012) Comparative study on classification performance between support vector machine and logistic regression. Int J Mach Learn Cybern. doi: 10.1007/s13042-012-0068-x Musa AB (2012) Comparative study on classification performance between support vector machine and logistic regression. Int J Mach Learn Cybern. doi: 10.​1007/​s13042-012-0068-x
44.
Zurück zum Zitat Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30 Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30
45.
Zurück zum Zitat van der Maaten L (2010) Statistical pattern recognition toolbox for Matlab (stprtool) version 2.11, version 0.7.2b van der Maaten L (2010) Statistical pattern recognition toolbox for Matlab (stprtool) version 2.11, version 0.7.2b
46.
Zurück zum Zitat Gavert H, Hurri J, Sarela J, Hyvarinen A (2005) Fast ICA for Matlab 7.x and 6.x, Version 2.5 Gavert H, Hurri J, Sarela J, Hyvarinen A (2005) Fast ICA for Matlab 7.x and 6.x, Version 2.5
48.
Zurück zum Zitat Maloof M (2002) On machine learning, ROC analysis, and statistical tests of significance. In: Proceedings of the sixteenth international conference on pattern recognition, pp 204–207 Maloof M (2002) On machine learning, ROC analysis, and statistical tests of significance. In: Proceedings of the sixteenth international conference on pattern recognition, pp 204–207
49.
Zurück zum Zitat Liang G, Zhu X, Zhang C (2012) The effect of varying levels of class distribution on bagging for different algorithms: an empirical study. Int J Mach Learn Cybern (IJMLC). doi:10.1007/s13042-012-0125-5 Liang G, Zhu X, Zhang C (2012) The effect of varying levels of class distribution on bagging for different algorithms: an empirical study. Int J Mach Learn Cybern (IJMLC). doi:10.​1007/​s13042-012-0125-5
50.
Zurück zum Zitat Xiang S, Nie F, Zhang C (2008) Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recog (PR) 41(12):3600–3612 Xiang S, Nie F, Zhang C (2008) Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recog (PR) 41(12):3600–3612
Metadaten
Titel
A comparison of ℓ1-regularizion, PCA, KPCA and ICA for dimensionality reduction in logistic regression
verfasst von
Abdallah Bashir Musa
Publikationsdatum
01.12.2014
Verlag
Springer Berlin Heidelberg
Erschienen in
International Journal of Machine Learning and Cybernetics / Ausgabe 6/2014
Print ISSN: 1868-8071
Elektronische ISSN: 1868-808X
DOI
https://doi.org/10.1007/s13042-013-0171-7

Weitere Artikel der Ausgabe 6/2014

International Journal of Machine Learning and Cybernetics 6/2014 Zur Ausgabe