Top

Neural Computing and Applications

Published in:

01-04-2012 | Original Article

Training recurrent neural networks using a hybrid algorithm

Authors: Mounir Ben Nasr, Mohamed Chtourou

Published in: Neural Computing and Applications | Issue 3/2012

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This paper proposes a new hybrid approach for recurrent neural networks (RNN). The basic idea of this approach is to train an input layer by unsupervised learning and an output layer by supervised learning. In this method, the Kohonen algorithm is used for unsupervised learning, and dynamic gradient descent method is used for supervised learning. The performances of the proposed algorithm are compared with backpropagation through time (BPTT) on three benchmark problems. Simulation results show that the performances of the new proposed algorithm exceed the standard backpropagation through time in the reduction of the total number of iterations and in the learning time required in the training process.

previous article A novel concept of embedding orthogonal basis function expansion block in a neural equalizer structure for digital communication channel

next article Solution of generalized matrix Riccati differential equation for indefinite stochastic linear quadratic singular fuzzy system with cross-term using neural networks

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Nerrand O, Roussel-Ragot P, Urbani D, Personnaz L, Dreyfus G (1994) Training recurrent neural networks: why and How? An illustration in dynamical process modeling. IEEE Trans Neural Netw 5(2):178–184CrossRef

Narendra KS, Parthasarathy K (1990) Identification and control of dynamical systems using neural networks. IEEE Trans Neural Netw 1(1):4–27CrossRef

Li X, Yu W (2002) Dynamic system identification via recurrent multilayer perception. Inf Sci 147:45–63MathSciNetMATHCrossRef

Guler I, Ubeyli ED (2006) A recurrent neural networks classifier for doppler ultrasound blood flow signal. Pattern Recogn Lett 27:1560–1571CrossRef

Puskorius GV, Feldkamp LA (1998) A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification. Proc IEEE 86:2259–2277CrossRef

Chtourou S, Chtourou M, Hammami O (2008) A hybrid approach for training recurrent neural networks: application to multi-Step-ahead prediction of noisy and large data sets. Neural Comput Appl 17:245–254CrossRef

Connor T, Douglas R (1994) Recurrent neural networks and robust time series prediction. IEEE Trans Neural Netw 5(2):240–254CrossRef

Graves D, Pedrycz W (2009) Fuzzy prediction architecture using recurrent neural networks. Neurocomputing 72:1668–1678CrossRef

Funahashi KI, Nakamura Y (1993) Approximation of dynamical systems by continuous networks. Neural Netw 6:801–806CrossRef

10.

Kosmatopoulos EA, Polycarpou MM, Christodoulou MA, Oannou PI (1995) High-order neural network structures for identification of dynamical systems. IEEE Trans Neural Netw 6:422–431CrossRef

11.

Werbos PJ (1990) Backpropagation through time: What it does and how to do it. IEEE Proc 78(10):1550–1560CrossRef

12.

Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. Parallel distributed processing: explorations in the microstructures of cognition, vol 1. MIT Press, Cambridge, pp 318–362

13.

Williams RJ, Zipser D (1989) A learning algorithm for continually running fully recurrent neural networks. Neural Comput 1:270–280CrossRef

14.

Toomarian N, Barben J (1991) Adjoint-function and temporal learning algorithms in neural networks. Adv Neural Inf Process Syst 3:113–120

15.

Toomarian N, Barben J (1992) Learning a trajectory using adjoint functions and teacher forcing. Neural Netw 5(3):473–484CrossRef

16.

Sim GZ, Chen HH, Lee YC (1992) Green’s function method for fast on-line learning algorithm of recurrent neural networks. Adv Neural Inf Process Syst 4:333–340

17.

Schmidhuber J (1992) A fixed storage O(N³) time complexity learning algorithm for fully recurrent continually running networks. Neural Comput 4(2):243–248CrossRef

18.

Williams R (1989) Complexity of exact gradient computation algorithms for recurrent neural networks. Northeasterm University, College Comp. Sci., Boston, MA, Tech. Rep. NU-CCS-89-27

19.

Feldkamp LA, Prokhorov DV (1998) Phased backpropagation: a hybrid of BPTT and temporal BP. Proc IEEE Int Joint Conf Neural Netw 3:2262–2267

20.

Wan EA (1990) Temporal backpropagation for FIR neural networks. Proc IEEE Int Joint Conf Neural Netw 1:575–580CrossRef

21.

Hochneiter S, Schmidhuber J (1997) Long short term memory. Neural Comput 9(8):1735–1780CrossRef

22.

Atiya AF, Parlos AG (2000) New results on recurrent network training: Unifying the algorithms and accelerating convergence. IEEE Trans Neural Netw 11(3):697–709CrossRef

23.

De Jesus O, Hagan MT (2007) Backpropagation algorithms for a broad class of dynamics networks. IEEE Trans Neural Netw 18(1):14–27CrossRef

24.

Song Q, Wu Y, Soh YC (2008) Robust adaptive gradient descent training algorithm for recurrent neural networks in discrete time domain. IEEE Trans Neural Netw 19(11):1841–1853CrossRef

25.

Mandic DP, Chambers JA (2001) Recurrent neural networks for prediction: learning algorithm, architecture and stability. Wiley, Chichester, UKCrossRef

26.

Williams R (1992) Training recurrent networks using the extended kalman filter, international joint conference on neural networks, Baltimore, ΙV, pp 241–246

27.

Puskorius GV, Feldkamp LA (1992) Recurrent networks training with decoupled extended Kalman Filter algorithm. In: Proceedings of the 1992 SPIE conference on the science of artificial neural networks, Orlando 1710, pp 461–473

28.

Vartak AA, Georgiopoulos M, Anagnostopoulos GC (2005) On line gauss- Newton—based learning for fully recurrent neural networks. Nonlinear Anal 63:867–876CrossRef

29.

Ben Nasr M, Chtourou M (2006) A hybrid training algorithm for feedforward neural networks. Neural Process Lett 24(2):107–117CrossRef

30.

Ben Nasr M, Chtourou M (2009) A fuzzy neighborhood-based training algorithm for feedforward neural networks. Neural Comput Appl 18(2):127–133CrossRef

31.

Kohonen T (1990) The self- organizing map. Proc IEEE 78(9):1464–1480CrossRef

32.

Box GE, Jenkins GM (1970) Time series analysis, forecasting and control, San Francisco, Holden Day

33.

Li M, Mechrotra K, Mohan C, Ranka S (1990) Sunspot numbers forecasting using neural network. In: Proc. IEEE Int. Symp. Intell. Contr., pp 524–528

34.

Kim J (1999) Adaptive neuro-fuzzy inference system and their application to non linear dynamical system. Neural Netw 12:1301–1319CrossRef

Title: Training recurrent neural networks using a hybrid algorithm
Authors: Mounir Ben Nasr
Mohamed Chtourou
Publication date: 01-04-2012
Publisher: Springer-Verlag
Published in: Neural Computing and Applications / Issue 3/2012
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-010-0506-1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 3/2012

Artificial neural networks workflow and its application in the petroleum industry

Adaptive radial basis function networks with kernel shape parameters

Application of seasonal SVR with chaotic immune algorithm in traffic flow forecasting

Examination of the reliability of a technical object after its regeneration in a maintenance system with an artificial neural network

Solution of generalized matrix Riccati differential equation for indefinite stochastic linear quadratic singular fuzzy system with cross-term using neural networks

Triaxial compression behavior of sand and tire wastes using neural networks

Premium Partner