An improved evolutionary extreme learning machine based on particle swarm optimization
Introduction
Single-hidden-layer feedforward neural networks (SLFN) have been proven to be universal approximator and are widely used for regression and classification problem [1]. Gradient-based learning algorithms such as backpropagation (BP) and Levenberg–Marquardt (LM), have been extensively used in the training of SLFN due to their reasonable performance [2], [3], [4]. However, these gradient-based algorithms are apt to be trapped in local minima and very time-consuming due to improper learning steps in many applications [5], [6], [7], [8], [9]. Moreover, they only consider the desired input/output information without the structure properties of network, thus their generalization performance are limited [10], [11], [12].
In order to overcome the drawbacks of gradient-based methods, extreme learning machine (ELM) for SLFN was proposed in 2004 [13]. ELM randomly chooses the input weights and hidden biases and analytically determines the output weights of SLFN through simple generalized inverse operation of the hidden layer output matrix. ELM not only learns much faster with better generalization performance than traditional gradient-based learning algorithms but also avoids many difficulties faced by gradient-based learning methods such as stopping criteria, learning rate, learning epochs, and local minima [14]. However, it is also found that ELM tends to require more hidden neurons than traditional gradient-based learning algorithms as well as result in ill-condition problem due to randomly selecting input weights and hidden biases [14], [15].
In the literature [14], a evolutionary ELM (E-ELM) was proposed which used the differential evolutionary algorithm to select the input weights and Moore–Penrose (MP) generalized inverse to analytically determine the output weights. The evolutionary ELM was able to achieve good generalization performance with much more compact networks. In the literature [15], in order to improve the conditioning of traditional ELM, an improved ELM was proposed by selecting input weights for an ELM with linear hidden neurons. This approach maintains testing accuracy with stable condition, but it was only limited to ELM with linear hidden neurons.
In recent years, particle swarm optimization (PSO) has been used increasingly as an effective technique for search global minima [16]. PSO has no complicated evolutionary operators and less parameter need to adjust [17]. Therefore, the hybrids of PSO and ELM should be promising for training feedforward neural networks. In the literature [18], particle swarm optimization (PSO) [19], [20] was used to optimize the input weights and hidden biases of the SLFN to solve some prediction problems, which mainly encoded the boundary conditions into PSO to improve the performance of ELM.
In this paper, a new method combining ELM with an improved PSO called as IPSO-ELM is proposed. In the proposed algorithm, the improved PSO is used to optimize the input weights and hidden biases, and the MP generalized inverse is used to analytically calculate the output weights. The improved PSO mainly focuses on decreasing the norm of the output weights of the SLFN and constraining the input weights and hidden biases within a reasonable range to improve the convergence performance of ELM.
The rest of paper is organized as follows. Section 2 introduces some preliminaries of ELM and PSO. The proposed method is proposed in Section 3. In Section 4, experiment results and discussion for regression and classification problems are given to demonstrate the effectiveness of the proposed algorithm. Finally, the concluding remarks are offered in Section 5.
Section snippets
Extreme learning machine
For N arbitrary distinct samples (xi,ti), where xi=[xi1,xi2,…xin]T∈Rn, . A SLFN with H hidden neurons and activation function g(·) can approximate these N samples with zero error. This means thatwherewhere is the weight vector connecting the i-th hidden neuron and the input neurons, is the weight vector
The improved extreme learning machine (IPSO-ELM)
ELM need not spend much time to tune the input weights and hidden biases of the SLFN by randomly choosing these parameters. Since the output weights are computed based on the input weights and hidden biases, there inevitably exists a set of nonoptimal or unnecessary input weights and hidden biases. Thus, two problems may be resulted from randomly choosing parameters in ELM. One is that ELM may require more hidden neurons than conventional gradient-based learning algorithms in some applications,
Experiment results and discussion
In this section, the IPSO-ELM are compared with PSO-ELM [18], E-ELM [14], LM (one of the fastest implementation of BP algorithms and is provided in the neural networks tools box of MATLAB.) and ELM. The parameters in all algorithms in all experiments are determined by trial and error. For IPSO-ELM, PSO-ELM and E-ELM, the maximum optimization epochs are 20, and the population size is 200. For IPSO-ELM and PSO-ELM, the initial inertial weight, wmax, and the final inertial weight, wmin, are
Conclusions
In this paper, an improved evolutionary extreme learning machine based on particle swarm optimization (IPSO-ELM) was proposed. In the new algorithm, an improved PSO was used to optimize the input weights and hidden biases, and minimum norm least-square scheme to analytically determine the output weights. In the process of selecting the input weights and hidden biases, the improved PSO considered not only the RMSE on validation set but also the norm of the output weights as well as constrained
Acknowledgements
This work was supported by the National Natural Science Foundation of China (nos. 61271385, 60702056), Natural Science Foundation of Jiangsu Province (nos. BK2009197, BK2009199) and the Initial Foundation of Science Research of Jiangsu University (no. 07JDG033).
Fei Han received the M.A. degree from Hefei University of Technology in 2003 and the Ph.D. degree from University of Science and Technology of China in 2006. He is currently an Associate Professor of computer science at Jiangsu University. His principal research interests are intelligent computing and intelligent information processing, including neural networks, particle swarm optimization, and bioinformatics.
References (24)
- et al.
Extreme Learning Machine: theory and applications
Neurocomputing
(2006) - et al.
Improved extreme learning machine for function approximation by encoding a priori information
Neurocomputing
(2006) - et al.
Dilation method for finding close roots of polynomials based on constrained learning neural networks
Phys. Lett. A
(2003) - et al.
Evolutionary extreme learning machine
Pattern Recognition
(2005) - et al.
OP-ELM: Optimally Pruned Extreme Learning Machine
IEEE Trans. Neural Networks
(2010) Radial basis probabilistic neural networks: model and application
Int. J. Pattern Recognition Artif. Intell.
(1999)Neural Networks: A Comprehensive Foundation
(1999)- et al.
A constructive hybrid structure optimization methodology for radial basis probabilistic neural networks
IEEE Trans. Neural Networks
(2008) A constructive approach for finding arbitrary roots of polynomials by neural networks
IEEE Trans. Neural Networks
(2004)- et al.
Finding roots of arbitrary high order polynomials based on neural network recursive partitioning method
Sci. China Ser. F. Inf. Sci.
(2004)
A neural root finder of polynomials based on root moments
Neural Comput.
The local minima free condition of feedforward neural networks for outer-supervised learning
IEEE Trans on Syst. Man Cybern.
Cited by (217)
Automobile insurance fraud detection using data mining: A systematic literature review
2024, Intelligent Systems with ApplicationsCement-based grouting material development and prediction of material properties using PSO-RBF machine learning
2024, Construction and Building MaterialsGeometric constructive network with block increments for lightweight data-driven industrial process modeling
2024, Journal of Process ControlAn evolutionary supply chain management service model based on deep learning features for automated glaucoma detection using fundus images
2024, Engineering Applications of Artificial IntelligenceA novel method for determining the particle breakage contribution of high-speed railway graded aggregate and its application in vibratory compaction
2023, Case Studies in Construction Materials
Fei Han received the M.A. degree from Hefei University of Technology in 2003 and the Ph.D. degree from University of Science and Technology of China in 2006. He is currently an Associate Professor of computer science at Jiangsu University. His principal research interests are intelligent computing and intelligent information processing, including neural networks, particle swarm optimization, and bioinformatics.
Hai-Fen Yao is a graduate student in the School of Computer Science and Telecommunication Engineering at Jiangsu University. She received the B.S degree from Suzhou University in 2000. Her research interests include neural networks, particle swarm optimization and intelligent computing.
Qing-Hua Ling received the B.S. degree from Nanjing Normal University in 2002 and the M.A. degree from Hefei Institute of Intelligent Machines, Chinese Academy of Sciences in 2005. She is currently a lecturer of computer science at Jiangsu University of Science and Technology. Her research interests include neural networks, particle swarm optimization and bioinformatics.