nach oben

Neural Computing and Applications

Erschienen in:

13.02.2020 | S.I. : Brain inspired Computing&Machine Learning Applied Research-BISMLARE

An advanced active set L-BFGS algorithm for training weight-constrained neural networks

verfasst von: Ioannis E. Livieris

Erschienen in: Neural Computing and Applications | Ausgabe 11/2020

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

In this work, a new advanced active set limited memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is proposed for efficiently training weight-constrained neural networks, called AA-L-BFGS. The proposed algorithm possesses the significant property of approximating the curvature of the error function with high-order accuracy by utilizing the theoretically advanced secant condition proposed by Livieris and Pintelas (Appl Math Comput 221:491–502, 2013). Moreover, the global convergence of the proposed algorithm is established provided that the line search satisfies the modified Armijo condition. The presented numerical experiments illustrate the efficiency of the proposed AA-L-BFGS, providing empirical evidence that it significantly accelerates the convergence of the training process.

Vorheriger Artikel Operational neural networks

Nächster Artikel Applying depthwise separable and multi-channel convolutional neural networks of varied kernel size on semantic trajectories

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Anastasiadis AD, Magoulas GD, Vrahatis MN (2005) New globally convergent training scheme based on the resilient propagation algorithm. Neurocomputing 64:253–270CrossRef

Awan SM, Aslam M, Khan ZA, Saeed H (2014) An efficient model based on artificial bee colony optimization algorithm with neural networks for electric load forecasting. Neural Comput Appl 25(7–8):1967–1978CrossRef

Azar AT (2013) Fast neural network learning algorithms for medical applications. Neural Comput Appl 23(3–4):1019–1034CrossRef

Badem H, Basturk A, Caliskan A, Yuksel ME (2017) A new efficient training strategy for deep neural networks by hybridization of artificial bee colony and limited-memory BFGS optimization algorithms. Neurocomputing 266:506–526CrossRef

Bilski K, Smolag J, Galushkin AI (2014) The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: International conference on artificial intelligence and soft computing. Springer, pp 12–21

Demertzis K, Iliadis L (2015) Intelligent bio-inspired detection of food borne pathogen by DNA barcodes: the case of invasive fish species Lagocephalus Sceleratus. In: International conference on engineering applications of neural networks. Springer, pp 89–99

Dolan E, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213MathSciNetCrossRef

Dua D, Taniskidou EK (2017) UCI machine learning repository

Erzin Y, Gul TO (2014) The use of neural networks for the prediction of the settlement of one-way footings on cohesionless soils based on standard penetration test. Neural Comput Appl 24(3–4):891–900CrossRef

10.

Facchinei F, Júdice J, Soares J (1998) An active set Newton algorithm for large-scale nonlinear programs with box constraints. SIAM J Optim 8(1):158–186MathSciNetCrossRef

11.

Gatys LA, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Proceedings of the 28th international conference on neural information processing systems, vol 1, pp 262–270

12.

Hertz J, Krogh A, Palmer R (1991) Introduction to the theory of neural computation. Addison-Wesley, Reading

13.

Horton P, Nakai K (1996) A probabilistic classification system for predicting the cellular localization sites of proteins. Ismb 4:109–115

14.

Iliadis L, Mansfield SD, Avramidis S, El-Kassaby YA (2013) Predicting Douglas-fir wood density by artificial neural networks (ANN) based on progeny testing information. Holzforschung 67(7):771–777CrossRef

15.

Karras DA, Perantonis SJ (1995) An efficient constrained training algorithm for feedforward networks. IEEE Trans Neural Netw 6(6):1420–1434CrossRef

16.

Kayaer K, Yıldırım T (2003) Medical diagnosis on Pima Indian diabetes using general regression neural networks. In: Proceedings of the international conference on artificial neural networks and neural information processing (ICANN/ICONIP), pp 181–184

17.

Khadse CB, Chaudhari MA, Borghate VB (2016) Conjugate gradient back-propagation based artificial neural network for real time power quality assessment. Int J Electr Power Energy Syst 82:197–206CrossRef

18.

Kostić S, Vasović D (2015) Prediction model for compressive strength of basic concrete mixture using artificial neural networks. Neural Comput Appl 26(5):1005–1024CrossRef

19.

Li F, Zhang X, Zhang X, Du C, Xu Y, Tian YC (2018) Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets. Inf Sci 422:242–256CrossRef

20.

Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1–3):503–528MathSciNetCrossRef

21.

Livieris IE (2018) Improving the classification efficiency of an ANN utilizing a new training methodology. Informatics 6(1):1CrossRef

22.

Livieris IE (2019) Forecasting economy-related data utilizing constrained recurrent neural networks. Algorithms 12(85):1MathSciNetMATH

23.

Livieris IE, Pintelas P (2013) A new conjugate gradient algorithm for training neural networks based on a modified secant equation. Appl Math Comput 221:491–502MathSciNetMATH

24.

Livieris IE, Pintelas P (2015) A new class of nonmonotone conjugate gradient training algorithms. Appl Math Comput 266:404–413MathSciNetMATH

25.

Livieris IE, Pintelas P (2019) An adaptive nonmonotone active set -weight constrained- neural network training algorithm. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.06.033 CrossRef

26.

Livieris IE, Pintelas P (2019) An improved weight-constrained neural network training algorithm. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04342-2 CrossRef

27.

Maren AJ, Harston CT, Pap RM (2014) Handbook of neural computing applications. Academic Press, CambridgeMATH

28.

Nguyen D, Widrow B (1990) Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights. Biol Cybern 59:71–113

29.

Noordewier MO, Towell GG, Shavlik JW (1990) Training knowledge-based neural networks to recognize genes in DNA sequences. In: Proceedings of the 3rd international conference on neural information processing systems, pp 530–536

30.

Perantonis SJ, Karras DA (1995) An efficient constrained learning algorithm with momentum acceleration. Neural Netw 8(2):237–249CrossRef

31.

Prechelt L (1994) PROBEN1-A set of benchmarks and benchmarking rules for neural network training algorithms. Technical report 21/94, Fakultt fr Informatik, University of Karlsruhe

32.

Shi ZJ, Wang S (2011) Modified nonmonotone Armijo line search for descent method. Numer Algorithms 57(1):1–25MathSciNetCrossRef

33.

Wan Zhong, Huang Shuai, Zheng Xiao Dong (2012) New cautious BFGS algorithm based on modified Armijo-type line search. J Inequal Appl 2012(1):241MathSciNetCrossRef

34.

Yuan G, Lu X (2011) An active set limited memory BFGS algorithm for bound constrained optimization. Appl Math Model 35:3561–3573MathSciNetCrossRef

Titel: An advanced active set L-BFGS algorithm for training weight-constrained neural networks
verfasst von: Ioannis E. Livieris
Publikationsdatum: 13.02.2020
Verlag: Springer London
Erschienen in: Neural Computing and Applications / Ausgabe 11/2020
Print ISSN: 0941-0643
Elektronische ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-019-04689-6

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 11/2020

Stability property of impulsive inertial neural networks with unbounded time delay and saturating actuators

Generative image completion with image-to-image translation

A novel adaptive model predictive controller for load frequency control of power systems integrated with DFIG wind turbines

Mutual kNN based spectral clustering

Opposition-based antlion optimizer using Cauchy distribution and its application to data clustering problem

Neural networks with block diagonal inner product layers: a look at neural network architecture through the lens of random matrices