Abstract
Some novel strategies have recently been proposed for single hidden layer neural network training that set randomly the weights from input to hidden layer, while weights from hidden to output layer are analytically determined by pseudoinversion. These techniques are gaining popularity in spite of their known numerical issues when singular and/or almost singular matrices are involved. In this paper, we discuss a critical use of Singular Value Analysis for identification of these drawbacks and we propose an original use of regularisation to determine the output weights, based on the concept of critical hidden layer size. This approach also allows to limit the training computational effort. Besides, we introduce a novel technique which relies an effective determination of input weights to the hidden layer dimension. This approach is tested for both regression and classification tasks, resulting in a significant performance improvement with respect to alternative methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ajorloo H, Manzuri-Shalmani MT, Lakdashti A (2007) Restoration of damaged slices in images using matrix pseudo inversion. In: Proceedings of the 22nd international symposium on computer and information sciences
Bache K, Lichman M (2013) UCI machine learning repository, University of California, School of Information and Computer Sciences, Irvine. http://archive.ics.uci.edu/ml
Badeva V, Morozov V (1991) Problèmes incorrectement posés: théorie et applications en identification, filtrage optimal, contrôle optimal, analyse et synthèse de systèmes, reconnaissance d’images. Série Automatique, Masson
Bengio Y, Glorot X (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of AISTATS 2010, vol 9, pp 249–256
Bishop CM (2006) Pattern recognition and machine learning (information science and statistics). Springer, Secaucus
Cancelliere R (2001) A high parallel procedure to initialize the output weights of a radial basis function or bp neural network. In: Proceedings of the 5th international workshop on applied parallel computing, New Paradigms for HPC in industry and academia, PARA ’00. Springer, London, pp 384–390
Cancelliere R, Gai M, Artières T, Gallinari P (2012) Matrix pseudoinversion for image neural processing. In: Huang T, Zeng Z, Li C, Leung C (eds) Neural information processing, Lecture notes in computer science, vol 7667. Springer, Berlin, pp 116–125
Fuhry M, Reichel L (2012) A new tikhonov regularization method. Numer Algorithms 59(3):433–445
Gallinari P, Cibas T (1999) Practical complexity control in multilayer perceptrons. Sig Process 74:29–46
Golub GH, Van Loan CF (1996) Matrix computations, 3rd edn. Johns Hopkins University Press, Baltimore
Helmy T, Rasheed Z (2009) Multi-category bioinformatics dataset classification using extreme learning machine. In: Proceedings of the eleventh conference on congress on evolutionary computation, CEC’09, Piscataway, NJ, USA, IEEE Press, pp 3234–3240
Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501
Kohno K, Kawamoto M, Inouye Y (2010) A matrix pseudoinversion lemma and its application to block-based adaptive blind deconvolution for mimo systems. Trans Cir Sys Part I 57(7):1449–1462
LeCun Y, Bottou L, Orr G, Müller K-R (1998) Efficient backprop. Springer, Berlin, pp 9–50
Malathi V, Marimuthu N, Baskar S (2010) Intelligent approaches using support vector machine and extreme learning machine for transmission line protection. Neurocomputing 73(1012):2160–2167 Subspace Learning/Selected papers from the European Symposium on Time Series Prediction
Miche Y, van Heeswijk M, Bas P, Simula O, Lendasse A (2011) Trop-elm: a double-regularized elm using lars and tikhonov regularization. Neurocomputing 74(16):2413–2421
Minhas R, Baradarani A, Seifzadeh S, Wu QJ (2010) Human action recognition using extreme learning machine based on visual vocabularies. Neurocomputing 73(1012):1906–1917 Subspace Learning/Selected papers from the European Symposium on Time Series Prediction
Nguyen TD, Pham HTB, Dang VH (2010) An efficient pseudo inverse matrix-based solution for secure auditing. In: Proceedings of the IEEE international conference on computing and communication technologies, research, innovation, and vision for the future, IEEE International Conference
Penrose R, Todd JA (1956) On best approximate solutions of linear matrix equations. In: Mathematical Proceedings of the Cambridge Philosophical Society, null, pp 17–19
Poggio T, Girosi F (1990) Networks for approximation and learning. Proc IEEE 78(9):1481–1497
Rao C, Mitra S (1971) Generalized inverse of matrices and its applications. In: Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. Wiley, New York
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1. MIT Press, Cambridge, pp 318–362
Sun Z-L, Choi T-M, Au K-F, Yu Y (2008) Sales forecasting using extreme learning machine with applications in fashion retailing. Dec Support Syst 46(1):411–419
Tikhonov AN (1963) Solution of incorrectly formulated problems and the regularization method. Sov Math Dokl 4:1035–1038
Tikhonov AN, Arsenin VY (1977) Solution of Ill-posed Problems. In: Scripta series in mathematics. Winston & Sons, Washington
Wang G, Zhao Y, Wang D (2008) A protein secondary structure prediction framework based on the extreme learning machine. Neurocomputing 72(13):262–268 Machine Learning for Signal Processing (MLSP 2006)/Life System Modelling, Simulation, and Bio-inspired Computing (LSMS 2007)
Yu D, Deng L (2012) Efficient and effective algorithms for training single-hidden-layer neural networks. Pattern Recogn Lett 33(5):554–558
Acknowledgments
The activity has been partially carried on in the context of the Visiting Professor Program of the Gruppo Nazionale per il Calcolo Scientifico (GNCS) of the Italian Istituto Nazionale di Alta Matematica (INdAM).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by José Mario Martínez.
Rights and permissions
About this article
Cite this article
Cancelliere, R., Deluca, R., Gai, M. et al. An analysis of numerical issues in neural training by pseudoinversion. Comp. Appl. Math. 36, 599–609 (2017). https://doi.org/10.1007/s40314-015-0246-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40314-015-0246-z