Elsevier

Neurocomputing

Volume 116, 20 September 2013, Pages 87-93
Neurocomputing

An improved evolutionary extreme learning machine based on particle swarm optimization

https://doi.org/10.1016/j.neucom.2011.12.062Get rights and content

Abstract

Recently Extreme Learning Machine (ELM) for single-hidden-layer feedforward neural networks (SLFN) has been attracting attentions for its faster learning speed and better generalization performance than those of traditional gradient-based learning algorithms. However, ELM may need high number of hidden neurons and lead to ill-condition problem due to the random determination of the input weights and hidden biases. In this paper, a hybrid learning algorithm is proposed to overcome the drawbacks of ELM, which uses an improved particle swarm optimization (PSO) algorithm to select the input weights and hidden biases and Moore–Penrose (MP) generalized inverse to analytically determine the output weights. In order to obtain optimal SLFN, the improved PSO optimizes the input weights and hidden biases according to not only the root mean squared error (RMSE) on validation set but also the norm of the output weights. The proposed algorithm has better generalization performance than traditional ELM and other evolutionary ELMs, and the conditioning of the SLFN trained by the proposed algorithm is also improved. Experiment results have verified the efficiency and effectiveness of the proposed method.

Introduction

Single-hidden-layer feedforward neural networks (SLFN) have been proven to be universal approximator and are widely used for regression and classification problem [1]. Gradient-based learning algorithms such as backpropagation (BP) and Levenberg–Marquardt (LM), have been extensively used in the training of SLFN due to their reasonable performance [2], [3], [4]. However, these gradient-based algorithms are apt to be trapped in local minima and very time-consuming due to improper learning steps in many applications [5], [6], [7], [8], [9]. Moreover, they only consider the desired input/output information without the structure properties of network, thus their generalization performance are limited [10], [11], [12].

In order to overcome the drawbacks of gradient-based methods, extreme learning machine (ELM) for SLFN was proposed in 2004 [13]. ELM randomly chooses the input weights and hidden biases and analytically determines the output weights of SLFN through simple generalized inverse operation of the hidden layer output matrix. ELM not only learns much faster with better generalization performance than traditional gradient-based learning algorithms but also avoids many difficulties faced by gradient-based learning methods such as stopping criteria, learning rate, learning epochs, and local minima [14]. However, it is also found that ELM tends to require more hidden neurons than traditional gradient-based learning algorithms as well as result in ill-condition problem due to randomly selecting input weights and hidden biases [14], [15].

In the literature [14], a evolutionary ELM (E-ELM) was proposed which used the differential evolutionary algorithm to select the input weights and Moore–Penrose (MP) generalized inverse to analytically determine the output weights. The evolutionary ELM was able to achieve good generalization performance with much more compact networks. In the literature [15], in order to improve the conditioning of traditional ELM, an improved ELM was proposed by selecting input weights for an ELM with linear hidden neurons. This approach maintains testing accuracy with stable condition, but it was only limited to ELM with linear hidden neurons.

In recent years, particle swarm optimization (PSO) has been used increasingly as an effective technique for search global minima [16]. PSO has no complicated evolutionary operators and less parameter need to adjust [17]. Therefore, the hybrids of PSO and ELM should be promising for training feedforward neural networks. In the literature [18], particle swarm optimization (PSO) [19], [20] was used to optimize the input weights and hidden biases of the SLFN to solve some prediction problems, which mainly encoded the boundary conditions into PSO to improve the performance of ELM.

In this paper, a new method combining ELM with an improved PSO called as IPSO-ELM is proposed. In the proposed algorithm, the improved PSO is used to optimize the input weights and hidden biases, and the MP generalized inverse is used to analytically calculate the output weights. The improved PSO mainly focuses on decreasing the norm of the output weights of the SLFN and constraining the input weights and hidden biases within a reasonable range to improve the convergence performance of ELM.

The rest of paper is organized as follows. Section 2 introduces some preliminaries of ELM and PSO. The proposed method is proposed in Section 3. In Section 4, experiment results and discussion for regression and classification problems are given to demonstrate the effectiveness of the proposed algorithm. Finally, the concluding remarks are offered in Section 5.

Section snippets

Extreme learning machine

For N arbitrary distinct samples (xi,ti), where xi=[xi1,xi2,…xin]TRn, ti=[ti1,ti2,,tim]TRm. A SLFN with H hidden neurons and activation function g(·) can approximate these N samples with zero error. This means thatHwo=TwhereH(wh1,,whH,b1,,bH,x1,,xN)=[g(wh1x1+b1)g(whHx1+bH)g(wh1xN+b1)g(whHxN+bH)]N×H,wo=[wo1TwoHT]H×mT=[t1TtNT]N×mwhere whi=[whi1,whi2,,whin]T is the weight vector connecting the i-th hidden neuron and the input neurons, woi=[woi1,woi2,,woim]T is the weight vector

The improved extreme learning machine (IPSO-ELM)

ELM need not spend much time to tune the input weights and hidden biases of the SLFN by randomly choosing these parameters. Since the output weights are computed based on the input weights and hidden biases, there inevitably exists a set of nonoptimal or unnecessary input weights and hidden biases. Thus, two problems may be resulted from randomly choosing parameters in ELM. One is that ELM may require more hidden neurons than conventional gradient-based learning algorithms in some applications,

Experiment results and discussion

In this section, the IPSO-ELM are compared with PSO-ELM [18], E-ELM [14], LM (one of the fastest implementation of BP algorithms and is provided in the neural networks tools box of MATLAB.) and ELM. The parameters in all algorithms in all experiments are determined by trial and error. For IPSO-ELM, PSO-ELM and E-ELM, the maximum optimization epochs are 20, and the population size is 200. For IPSO-ELM and PSO-ELM, the initial inertial weight, wmax, and the final inertial weight, wmin, are

Conclusions

In this paper, an improved evolutionary extreme learning machine based on particle swarm optimization (IPSO-ELM) was proposed. In the new algorithm, an improved PSO was used to optimize the input weights and hidden biases, and minimum norm least-square scheme to analytically determine the output weights. In the process of selecting the input weights and hidden biases, the improved PSO considered not only the RMSE on validation set but also the norm of the output weights as well as constrained

Acknowledgements

This work was supported by the National Natural Science Foundation of China (nos. 61271385, 60702056), Natural Science Foundation of Jiangsu Province (nos. BK2009197, BK2009199) and the Initial Foundation of Science Research of Jiangsu University (no. 07JDG033).

Fei Han received the M.A. degree from Hefei University of Technology in 2003 and the Ph.D. degree from University of Science and Technology of China in 2006. He is currently an Associate Professor of computer science at Jiangsu University. His principal research interests are intelligent computing and intelligent information processing, including neural networks, particle swarm optimization, and bioinformatics.

References (24)

  • D.S. Huang et al.

    A neural root finder of polynomials based on root moments

    Neural Comput.

    (2004)
  • D.S. Huang

    The local minima free condition of feedforward neural networks for outer-supervised learning

    IEEE Trans on Syst. Man Cybern.

    (1998)
  • Cited by (217)

    View all citing articles on Scopus

    Fei Han received the M.A. degree from Hefei University of Technology in 2003 and the Ph.D. degree from University of Science and Technology of China in 2006. He is currently an Associate Professor of computer science at Jiangsu University. His principal research interests are intelligent computing and intelligent information processing, including neural networks, particle swarm optimization, and bioinformatics.

    Hai-Fen Yao is a graduate student in the School of Computer Science and Telecommunication Engineering at Jiangsu University. She received the B.S degree from Suzhou University in 2000. Her research interests include neural networks, particle swarm optimization and intelligent computing.

    Qing-Hua Ling received the B.S. degree from Nanjing Normal University in 2002 and the M.A. degree from Hefei Institute of Intelligent Machines, Chinese Academy of Sciences in 2005. She is currently a lecturer of computer science at Jiangsu University of Science and Technology. Her research interests include neural networks, particle swarm optimization and bioinformatics.

    View full text