Skip to main content
Erschienen in: International Journal of Machine Learning and Cybernetics 7/2020

03.01.2020 | Original Article

Unsupervised feature learning with sparse Bayesian auto-encoding based extreme learning machine

verfasst von: Guanghao Zhang, Dongshun Cui, Shangbo Mao, Guang-Bin Huang

Erschienen in: International Journal of Machine Learning and Cybernetics | Ausgabe 7/2020

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Extreme learning machine (ELM) is a popular method in machine learning with extremely few parameters, fast learning speed and model efficiency. Unsupervised feature learning based ELM receives rising research focus. Recently the ELM auto-encoder (ELM-AE) was proposed for this task, which develops the ELM based compact feature learning without sacrificing elegant solution. Compared with ELM-AE and following \(\ell _1\)-regularized ELM-AE, we introduce a sparse Bayesian learning scheme into ELM-AE for better generalization capability. A parallel training strategy is also integrated to improve time-efficiency of multi-output sparse Bayesian learning. Furthermore, pruning hidden nodes for better performance and efficiency according to estimated variances of prior distribution of output weights is achieved. Experiments on several datasets verify the effectiveness and efficiency of our proposed ELM-AE for unsupervised feature learning, compared with PCA, NMF, ELM-AE and \(\ell _1\)-regularized ELM-AE.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Weitere Produktempfehlungen anzeigen
Literatur
1.
Zurück zum Zitat Rumelhart DE, Hinton GE, Williams RJ et al (1988) Learning representations by back-propagating errors. Cognit Model 5(3):1MATH Rumelhart DE, Hinton GE, Williams RJ et al (1988) Learning representations by back-propagating errors. Cognit Model 5(3):1MATH
2.
Zurück zum Zitat Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892 Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892
3.
Zurück zum Zitat Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(2):513–529 Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B (Cybernetics) 42(2):513–529
4.
Zurück zum Zitat Huang G-B, Zhu Q-Y, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501 Huang G-B, Zhu Q-Y, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501
5.
Zurück zum Zitat Nizar AH, Dong ZY, Wang Y (2008) Power utility nontechnical loss analysis with extreme learning machine method. IEEE Trans Power Syst 23(3):946–955 Nizar AH, Dong ZY, Wang Y (2008) Power utility nontechnical loss analysis with extreme learning machine method. IEEE Trans Power Syst 23(3):946–955
6.
Zurück zum Zitat You ZH, Lei YK, Zhu L, Xia J, Wang Bing (2013) Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis. Bmc Bioinform 14(S8):S10 You ZH, Lei YK, Zhu L, Xia J, Wang Bing (2013) Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis. Bmc Bioinform 14(S8):S10
7.
Zurück zum Zitat Zhige Xie X, Kai SW, Liu L, Xiong Y, Hui Huang (2015) Projective feature learning for 3D shapes with multi-view depth images. Comput Graphics Forum 34(7):1–11 Zhige Xie X, Kai SW, Liu L, Xiong Y, Hui Huang (2015) Projective feature learning for 3D shapes with multi-view depth images. Comput Graphics Forum 34(7):1–11
8.
Zurück zum Zitat Huang G, Song S, Gupta JND, Cheng W (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern 44(12):2405–2417 Huang G, Song S, Gupta JND, Cheng W (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern 44(12):2405–2417
9.
Zurück zum Zitat Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with extreme learning machine for big data. IEEE Intell Syst 28(6):31–34 Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with extreme learning machine for big data. IEEE Intell Syst 28(6):31–34
10.
Zurück zum Zitat Kasun LLC, Yang Y, Huang G-B, Zhang Z (2016) Dimension reduction with extreme learning machine. IEEE Trans Image Process 25(8):3906–3918MathSciNetMATH Kasun LLC, Yang Y, Huang G-B, Zhang Z (2016) Dimension reduction with extreme learning machine. IEEE Trans Image Process 25(8):3906–3918MathSciNetMATH
11.
Zurück zum Zitat Liyanaarachchi LCC, Tianchi L, Yan Y, Zhiping L, Guang-Bin H (2015) Extreme learning machine for clustering. In: Proceedings of ELM-2014, volume 1. Springer, pp 435–444 Liyanaarachchi LCC, Tianchi L, Yan Y, Zhiping L, Guang-Bin H (2015) Extreme learning machine for clustering. In: Proceedings of ELM-2014, volume 1. Springer, pp 435–444
12.
Zurück zum Zitat Tang J, Deng C, Huang G-B (2015) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27(4):809–821MathSciNet Tang J, Deng C, Huang G-B (2015) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27(4):809–821MathSciNet
13.
Zurück zum Zitat Luo J, Vong C-M, Wong P-K (2013) Sparse bayesian extreme learning machine for multi-classification. IEEE Trans Neural Netw Learn Syst 25(4):836–843 Luo J, Vong C-M, Wong P-K (2013) Sparse bayesian extreme learning machine for multi-classification. IEEE Trans Neural Netw Learn Syst 25(4):836–843
14.
Zurück zum Zitat Hoerl AE, Kennard W (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67MATH Hoerl AE, Kennard W (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67MATH
15.
Zurück zum Zitat Huang G-B (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cognit Comput 6(3):376–390MathSciNet Huang G-B (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cognit Comput 6(3):376–390MathSciNet
16.
Zurück zum Zitat Ding C, He X (2004) K-means clustering via principal component analysis. In: International Conference on machine learning Ding C, He X (2004) K-means clustering via principal component analysis. In: International Conference on machine learning
17.
Zurück zum Zitat Pearson K (1901) Liii. on lines and planes of closest fit to systems of points in space. Lond Edinb Dublin Philos Mag J Sci 2(11):559–572MATH Pearson K (1901) Liii. on lines and planes of closest fit to systems of points in space. Lond Edinb Dublin Philos Mag J Sci 2(11):559–572MATH
18.
Zurück zum Zitat Hotelling H (1933) Analysis of a complex of statistical variables into principal components. J Educ Psychol 24(6):417MATH Hotelling H (1933) Analysis of a complex of statistical variables into principal components. J Educ Psychol 24(6):417MATH
19.
Zurück zum Zitat Yoo Y, Oh S-Y (2016) Fast training of convolutional neural network classifiers through extreme learning machines. In: 2016 International Joint Conference on neural networks (IJCNN). IEEE, pp 1702–1708 Yoo Y, Oh S-Y (2016) Fast training of convolutional neural network classifiers through extreme learning machines. In: 2016 International Joint Conference on neural networks (IJCNN). IEEE, pp 1702–1708
20.
Zurück zum Zitat Wang Y, Xie Z, Kai X, Dou Y, Lei Yuanwu (2016) An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning. Neurocomputing 174:988–998 Wang Y, Xie Z, Kai X, Dou Y, Lei Yuanwu (2016) An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning. Neurocomputing 174:988–998
21.
Zurück zum Zitat Low C-Y, Teoh AB-J (2017) Stacking-based deep neural network: Deep analytic network on convolutional spectral histogram features. In: 2017 IEEE International Conference on Image Processing (ICIP). IEEE, pp 1592–1596 Low C-Y, Teoh AB-J (2017) Stacking-based deep neural network: Deep analytic network on convolutional spectral histogram features. In: 2017 IEEE International Conference on Image Processing (ICIP). IEEE, pp 1592–1596
22.
Zurück zum Zitat Congdon P (2007) Bayesian statistical modelling, vol 704. Wiley, HobokenMATH Congdon P (2007) Bayesian statistical modelling, vol 704. Wiley, HobokenMATH
23.
Zurück zum Zitat Soria-Olivas E, Gomez-Sanchis J, Martin JD, Vila-Frances J, Martinez M, Magdalena JR, Serrano AJ (2011) Belm: Bayesian extreme learning machine. IEEE Trans Neural Netw 22(3):505–509 Soria-Olivas E, Gomez-Sanchis J, Martin JD, Vila-Frances J, Martinez M, Magdalena JR, Serrano AJ (2011) Belm: Bayesian extreme learning machine. IEEE Trans Neural Netw 22(3):505–509
24.
Zurück zum Zitat Bishop CM (2006) Pattern recognition and machine learning. Springer, BerlinMATH Bishop CM (2006) Pattern recognition and machine learning. Springer, BerlinMATH
25.
Zurück zum Zitat MacKay DJC (1992) The evidence framework applied to classification networks. Neural Comput 4(5):720–736 MacKay DJC (1992) The evidence framework applied to classification networks. Neural Comput 4(5):720–736
26.
Zurück zum Zitat Nabney IT (2004) Efficient training of RBF networks for classification. Int J Neural Syst 14(3):1–8 Nabney IT (2004) Efficient training of RBF networks for classification. Int J Neural Syst 14(3):1–8
27.
Zurück zum Zitat Tipping ME (2001) Sparse Bayesian learning and the relevance vector machine. J Mach Learn Res 1(Jun):211–244MathSciNetMATH Tipping ME (2001) Sparse Bayesian learning and the relevance vector machine. J Mach Learn Res 1(Jun):211–244MathSciNetMATH
28.
Zurück zum Zitat MacKay DJC (1992) Bayesian interpolation. Neural Comput 4(3):415–447MATH MacKay DJC (1992) Bayesian interpolation. Neural Comput 4(3):415–447MATH
29.
Zurück zum Zitat Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M et al (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M et al (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:​1603.​04467
30.
Zurück zum Zitat Lee DD, Sebastian Seung H (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788MATH Lee DD, Sebastian Seung H (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788MATH
31.
Zurück zum Zitat Michael L, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition, pp 200–205. IEEE Michael L, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition, pp 200–205. IEEE
32.
Zurück zum Zitat Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Proceedings of 1994 IEEE Workshop on Applications of Computer Vision. IEEE, pp 138–142 Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Proceedings of 1994 IEEE Workshop on Applications of Computer Vision. IEEE, pp 138–142
33.
Zurück zum Zitat Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on computer vision and pattern recognition Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on computer vision and pattern recognition
Metadaten
Titel
Unsupervised feature learning with sparse Bayesian auto-encoding based extreme learning machine
verfasst von
Guanghao Zhang
Dongshun Cui
Shangbo Mao
Guang-Bin Huang
Publikationsdatum
03.01.2020
Verlag
Springer Berlin Heidelberg
Erschienen in
International Journal of Machine Learning and Cybernetics / Ausgabe 7/2020
Print ISSN: 1868-8071
Elektronische ISSN: 1868-808X
DOI
https://doi.org/10.1007/s13042-019-01057-7

Weitere Artikel der Ausgabe 7/2020

International Journal of Machine Learning and Cybernetics 7/2020 Zur Ausgabe

Neuer Inhalt