Skip to main content
Erschienen in: Neural Processing Letters 3/2020

26.03.2020

Optimizing Deep Feedforward Neural Network Architecture: A Tabu Search Based Approach

verfasst von: Tarun Kumar Gupta, Khalid Raza

Erschienen in: Neural Processing Letters | Ausgabe 3/2020

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The optimal architecture of a deep feedforward neural network (DFNN) is essential for its better accuracy and faster convergence. Also, the training of DFNN becomes tedious as the depth of the network increases. The DFNN can be tweaked using several parameters, such as the number of hidden layers, the number of hidden neurons at each hidden layer, and the number of connections between layers. The optimal architecture of DFNN is usually set using a trial-and-error process, which is an exponential combinatorial problem and a tedious task. To address this problem, we need an algorithm that can automatically design an optimal architecture with improved generalization ability. This work aims to propose a new methodology that can simultaneously optimize the number of hidden layers and their respective neurons for DFNN. This work combines the advantages of Tabu search and Gradient descent with a momentum backpropagation training algorithm. The proposed approach has been tested on four different classification benchmark datasets, which show better generalization ability of the optimized networks.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Williams D, Hinton G (1986) Learning representations by back-propagating errors. Nature 323(6088):533–538MATH Williams D, Hinton G (1986) Learning representations by back-propagating errors. Nature 323(6088):533–538MATH
2.
Zurück zum Zitat Holland JH (1975) Adaptation in natural and artificial systems. An introductory analysis with application to biology, control, and artificial intelligence. Univ. Michigan Press, Ann Arbor, pp 439–444 Holland JH (1975) Adaptation in natural and artificial systems. An introductory analysis with application to biology, control, and artificial intelligence. Univ. Michigan Press, Ann Arbor, pp 439–444
3.
Zurück zum Zitat Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680MathSciNetMATH Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680MathSciNetMATH
4.
Zurück zum Zitat Glover F (1986) Future paths for integer programming and links to artificial intelligence. Comput Oper Res 13(5):533–549MathSciNetMATH Glover F (1986) Future paths for integer programming and links to artificial intelligence. Comput Oper Res 13(5):533–549MathSciNetMATH
5.
Zurück zum Zitat Yang X-S (2010) A new metaheuristic bat-inspired algorithm. In: Nature inspired cooperative strategies for optimization (NICSO 2010), Springer, pp 65–74 Yang X-S (2010) A new metaheuristic bat-inspired algorithm. In: Nature inspired cooperative strategies for optimization (NICSO 2010), Springer, pp 65–74
6.
Zurück zum Zitat Li Y, Fu Y, Li H, Zhang S-W (2009) The improved training algorithm of back propagation neural network with self-adaptive learning rate. In: International conference on computational intelligence and natural computing. vol 1, 2009. CINC’09, pp 73–76 Li Y, Fu Y, Li H, Zhang S-W (2009) The improved training algorithm of back propagation neural network with self-adaptive learning rate. In: International conference on computational intelligence and natural computing. vol 1, 2009. CINC’09, pp 73–76
7.
Zurück zum Zitat Ma DS, Correll J, Wittenbrink B (2015) The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods 47(4):1122–1135 Ma DS, Correll J, Wittenbrink B (2015) The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods 47(4):1122–1135
8.
Zurück zum Zitat Vergara A, Vembu S, Ayhan T, Ryan MA, Homer ML, Huerta R (2012) Chemical gas sensor drift compensation using classifier ensembles. Sens Actuators B Chem 166:320–329 Vergara A, Vembu S, Ayhan T, Ryan MA, Homer ML, Huerta R (2012) Chemical gas sensor drift compensation using classifier ensembles. Sens Actuators B Chem 166:320–329
9.
Zurück zum Zitat Rodriguez-Lujan I, Fonollosa J, Vergara A, Homer M, Huerta R (2014) On the calibration of sensor arrays for pattern recognition using the minimal number of experiments. Chemom Intell Lab Syst 130:123–134 Rodriguez-Lujan I, Fonollosa J, Vergara A, Homer M, Huerta R (2014) On the calibration of sensor arrays for pattern recognition using the minimal number of experiments. Chemom Intell Lab Syst 130:123–134
10.
Zurück zum Zitat LeCun Y, Cortes C (2010) {MNIST} handwritten digit database LeCun Y, Cortes C (2010) {MNIST} handwritten digit database
11.
Zurück zum Zitat Dheeru D, Karra Taniskidou E (2017) {UCI} Machine Learning Repository Dheeru D, Karra Taniskidou E (2017) {UCI} Machine Learning Repository
12.
Zurück zum Zitat Stepniewski SW, Keane AJ (1997) Pruning backpropagation neural networks using modern stochastic optimisation techniques. Neural Comput Appl 5(2):76–98 Stepniewski SW, Keane AJ (1997) Pruning backpropagation neural networks using modern stochastic optimisation techniques. Neural Comput Appl 5(2):76–98
13.
Zurück zum Zitat Ludermir TB, Yamazaki A, Zanchettin C (2006) An optimization methodology for neural network weights and architectures. IEEE Trans Neural Netw 17(6):1452–1459 Ludermir TB, Yamazaki A, Zanchettin C (2006) An optimization methodology for neural network weights and architectures. IEEE Trans Neural Netw 17(6):1452–1459
14.
Zurück zum Zitat Gepperth A, Roth S (2006) Applications of multi-objective structure optimization. Neurocomputing 69(7–9):701–713 Gepperth A, Roth S (2006) Applications of multi-objective structure optimization. Neurocomputing 69(7–9):701–713
15.
Zurück zum Zitat Tsai JT, Chou JH, Liu TK (2006) Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm. IEEE Trans Neural Netw 17(1):69–80 Tsai JT, Chou JH, Liu TK (2006) Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm. IEEE Trans Neural Netw 17(1):69–80
16.
Zurück zum Zitat Huang D-S, Du J-X (2008) A constructive hybrid structure optimization methodology for radial basis probabilistic neural networks. IEEE Trans Neural Netw 19(12):2099–2115 Huang D-S, Du J-X (2008) A constructive hybrid structure optimization methodology for radial basis probabilistic neural networks. IEEE Trans Neural Netw 19(12):2099–2115
17.
Zurück zum Zitat Yu J, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71(4–6):1054–1060 Yu J, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71(4–6):1054–1060
18.
Zurück zum Zitat Islam MM, Amin MF, Ahmmed S, Murase K (2008) An adaptive merging and growing algorithm for designing artificial neural networks. In: IEEE international joint conference on neural networks, 2008. IJCNN 2008 (IEEE world congress on computational intelligence), pp 2003–2008 Islam MM, Amin MF, Ahmmed S, Murase K (2008) An adaptive merging and growing algorithm for designing artificial neural networks. In: IEEE international joint conference on neural networks, 2008. IJCNN 2008 (IEEE world congress on computational intelligence), pp 2003–2008
19.
Zurück zum Zitat Islam MM, Sattar MA, Amin MF, Yao X, Murase K (2009) A new adaptive merging and growing algorithm for designing artificial neural networks. IEEE Trans Syst Man Cybern Part B 39(3):705–722 Islam MM, Sattar MA, Amin MF, Yao X, Murase K (2009) A new adaptive merging and growing algorithm for designing artificial neural networks. IEEE Trans Syst Man Cybern Part B 39(3):705–722
20.
Zurück zum Zitat Goh C-K, Teoh E-J, Tan KC (2008) Hybrid multiobjective evolutionary design for artificial neural networks. IEEE Trans Neural Netw 19(9):1531–1548 Goh C-K, Teoh E-J, Tan KC (2008) Hybrid multiobjective evolutionary design for artificial neural networks. IEEE Trans Neural Netw 19(9):1531–1548
21.
Zurück zum Zitat Hervás-Martínez C, Martínez-Estudillo FJ, Carbonero-Ruz M (2008) Multilogistic regression by means of evolutionary product-unit neural networks. Neural Netw 21(7):951–961MATH Hervás-Martínez C, Martínez-Estudillo FJ, Carbonero-Ruz M (2008) Multilogistic regression by means of evolutionary product-unit neural networks. Neural Netw 21(7):951–961MATH
22.
Zurück zum Zitat Mantzaris D, Anastassopoulos G, Adamopoulos A (2011) Genetic algorithm pruning of probabilistic neural networks in medical disease estimation. Neural Netw 24(8):831–835 Mantzaris D, Anastassopoulos G, Adamopoulos A (2011) Genetic algorithm pruning of probabilistic neural networks in medical disease estimation. Neural Netw 24(8):831–835
23.
Zurück zum Zitat Pelikan M, Goldberg DE, Cantú-Paz E (1999) BOA: the Bayesian optimization algorithm. In: Proceedings of the genetic and evolutionary computation conference GECCO-99, vol. 1, pp 525–532 Pelikan M, Goldberg DE, Cantú-Paz E (1999) BOA: the Bayesian optimization algorithm. In: Proceedings of the genetic and evolutionary computation conference GECCO-99, vol. 1, pp 525–532
24.
Zurück zum Zitat Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, pp 2951–2959 Snoek J, Larochelle H, Adams RP (2012) Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems, pp 2951–2959
25.
Zurück zum Zitat Yu Z, Yu J, Xiang C, Fan J, Tao D (2018) Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering. IEEE Trans Neural Netw Learn Syst 29(12):5947–5959 Yu Z, Yu J, Xiang C, Fan J, Tao D (2018) Beyond bilinear: generalized multimodal factorized high-order pooling for visual question answering. IEEE Trans Neural Netw Learn Syst 29(12):5947–5959
26.
Zurück zum Zitat Yu Z, Yu J, Cui Y, Tao D, Tian Q (2019) Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6281–6290 Yu Z, Yu J, Cui Y, Tao D, Tian Q (2019) Deep modular co-attention networks for visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6281–6290
27.
Zurück zum Zitat Yu Z, Xu D, Yu J, Yu T, Zhao Z, Zhuang Y, Tao D (2019) ActivityNet-QA: a dataset for understanding complex Web videos via question answering. Proc AAAI Conf Artif Intell 33:9127–9134 Yu Z, Xu D, Yu J, Yu T, Zhao Z, Zhuang Y, Tao D (2019) ActivityNet-QA: a dataset for understanding complex Web videos via question answering. Proc AAAI Conf Artif Intell 33:9127–9134
28.
Zurück zum Zitat Zanchettin C, Ludermir TB, Almeida LM (2011) Hybrid training method for MLP: optimization of architecture and training. IEEE Trans Syst Man Cybern Part B Cybern 41(4):1097–1109 Zanchettin C, Ludermir TB, Almeida LM (2011) Hybrid training method for MLP: optimization of architecture and training. IEEE Trans Syst Man Cybern Part B Cybern 41(4):1097–1109
29.
Zurück zum Zitat Baker JE (1987) Reducing bias and inefficiency in the selection algorithm. In: Proceedings of the second international conference on genetic algorithms, pp 14–21 Baker JE (1987) Reducing bias and inefficiency in the selection algorithm. In: Proceedings of the second international conference on genetic algorithms, pp 14–21
31.
Zurück zum Zitat Yu J, Zhu C, Zhang J, Huang Q, Tao D (2019) Spatial pyramid-enhanced NetVLAD with weighted triplet loss for place recognition. IEEE Trans Neural Netw Learn Syst 32(2):661–674 Yu J, Zhu C, Zhang J, Huang Q, Tao D (2019) Spatial pyramid-enhanced NetVLAD with weighted triplet loss for place recognition. IEEE Trans Neural Netw Learn Syst 32(2):661–674
32.
Zurück zum Zitat Yao J, Yu Z, Yu J, Tao D. SPRNet: single pixel reconstruction for one-stage instance segmentation Yao J, Yu Z, Yu J, Tao D. SPRNet: single pixel reconstruction for one-stage instance segmentation
33.
Zurück zum Zitat Zhang J, Yu J, Tao D (2018) Local deep-feature alignment for unsupervised dimension reduction. IEEE Trans Image Process 27(5):2420–2432MathSciNetMATH Zhang J, Yu J, Tao D (2018) Local deep-feature alignment for unsupervised dimension reduction. IEEE Trans Image Process 27(5):2420–2432MathSciNetMATH
34.
Zurück zum Zitat Jaddi NS, Abdullah S, Hamdan AR (2015) Optimization of neural network model using modified bat-inspired algorithm. Appl Soft Comput 37:71–86 Jaddi NS, Abdullah S, Hamdan AR (2015) Optimization of neural network model using modified bat-inspired algorithm. Appl Soft Comput 37:71–86
35.
Zurück zum Zitat Yang WH, Tarng YS (1998) Design optimization of cutting parameters for turning operations based on the Taguchi method. J Mater Process Technol 84(1–3):122–129 Yang WH, Tarng YS (1998) Design optimization of cutting parameters for turning operations based on the Taguchi method. J Mater Process Technol 84(1–3):122–129
36.
Zurück zum Zitat Jaddi NS, Abdullah S, Hamdan AR (2015) Multi-population cooperative bat algorithm-based optimization of artificial neural network model. Inf Sci 294:628–644MathSciNet Jaddi NS, Abdullah S, Hamdan AR (2015) Multi-population cooperative bat algorithm-based optimization of artificial neural network model. Inf Sci 294:628–644MathSciNet
37.
Zurück zum Zitat Jaddi NS, Abdullah S, Hamdan AR (2016) A solution representation of genetic algorithm for neural network weights and structure. Inf Process Lett 116(1):22–25 Jaddi NS, Abdullah S, Hamdan AR (2016) A solution representation of genetic algorithm for neural network weights and structure. Inf Process Lett 116(1):22–25
38.
Zurück zum Zitat Gupta TK, Raza K (2018) Optimization of ANN architecture: a review on nature-inspired techniques. In: Dey N, Borra S, Ashour, Shi F (eds) Machine learning in bio-signal and diagnostic imaging. Elsevier, pp 159–182 Gupta TK, Raza K (2018) Optimization of ANN architecture: a review on nature-inspired techniques. In: Dey N, Borra S, Ashour, Shi F (eds) Machine learning in bio-signal and diagnostic imaging. Elsevier, pp 159–182
39.
Zurück zum Zitat Bengio Y et al (2009) Learning deep architectures for AI. Found Trends® Mach Learn 2(1):1–127MATH Bengio Y et al (2009) Learning deep architectures for AI. Found Trends® Mach Learn 2(1):1–127MATH
Metadaten
Titel
Optimizing Deep Feedforward Neural Network Architecture: A Tabu Search Based Approach
verfasst von
Tarun Kumar Gupta
Khalid Raza
Publikationsdatum
26.03.2020
Verlag
Springer US
Erschienen in
Neural Processing Letters / Ausgabe 3/2020
Print ISSN: 1370-4621
Elektronische ISSN: 1573-773X
DOI
https://doi.org/10.1007/s11063-020-10234-7

Weitere Artikel der Ausgabe 3/2020

Neural Processing Letters 3/2020 Zur Ausgabe

Neuer Inhalt