Skip to main content

2016 | OriginalPaper | Buchkapitel

Evolutionary Multi-task Learning for Modular Training of Feedforward Neural Networks

verfasst von : Rohitash Chandra, Abhishek Gupta, Yew-Soon Ong, Chi-Keong Goh

Erschienen in: Neural Information Processing

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Multi-task learning enables learning algorithms to harness shared knowledge from several tasks in order to provide better performance. In the past, neuro-evolution has shownpromising performance for a number of real-world applications. Recently, evolutionary multi-tasking has been proposed for optimisation problems. In this paper, we present a multi-task learning for neural networks that evolves modular network topologies. In the proposed method, each task is defined by a specific network topology defined with a different number of hidden neurons. The method produces a modular network that could be effective even if some of the neurons and connections are removed from selected trained modules in the network. We demonstrate the effectiveness of the method using feedforward networks to learn selected n-bit parity problems of varying levels of difficulty. The results show better training and generalisation performance when the modules for representing additional knowledge are added by increasing hidden neurons during training.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Angeline, P., Saunders, G., Pollack, J.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Netw. 5(1), 54–65 (1994)CrossRef Angeline, P., Saunders, G., Pollack, J.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Netw. 5(1), 54–65 (1994)CrossRef
2.
Zurück zum Zitat Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive coevolution. Evol. Comput. 5(4), 373–399 (1997)CrossRef Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive coevolution. Evol. Comput. 5(4), 373–399 (1997)CrossRef
3.
Zurück zum Zitat Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRef Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRef
4.
Zurück zum Zitat Sexton, R.S., Dorsey, R.E.: Reliable classification using neural networks: a genetic algorithm and backpropagation comparison. Decis. Support Syst. 30(1), 11–22 (2000)CrossRef Sexton, R.S., Dorsey, R.E.: Reliable classification using neural networks: a genetic algorithm and backpropagation comparison. Decis. Support Syst. 30(1), 11–22 (2000)CrossRef
5.
Zurück zum Zitat Cant-Paz, E., Kamath, C.: An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans. Syst. Man Cybern. B Cybern. 35(5), 915–933 (2005)CrossRef Cant-Paz, E., Kamath, C.: An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans. Syst. Man Cybern. B Cybern. 35(5), 915–933 (2005)CrossRef
6.
Zurück zum Zitat Garcia-Pedrajas, N., Hervas-Martinez, C., Munoz-Perez, J.: COVNET: a cooperative coevolutionary model for evolving artificial neural networks. IEEE Trans. Neural Netw. 14(3), 575–596 (2003)CrossRef Garcia-Pedrajas, N., Hervas-Martinez, C., Munoz-Perez, J.: COVNET: a cooperative coevolutionary model for evolving artificial neural networks. IEEE Trans. Neural Netw. 14(3), 575–596 (2003)CrossRef
7.
Zurück zum Zitat Gomez, F., Schmidhuber, J., Miikkulainen, R.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9, 937–965 (2008)MathSciNetMATH Gomez, F., Schmidhuber, J., Miikkulainen, R.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9, 937–965 (2008)MathSciNetMATH
8.
Zurück zum Zitat Chandra, R.: Competition and collaboration in cooperative coevolution of Elman recurrent neural networks for time-series prediction. IEEE Trans. Neural Netw. Learn. Syst. 26, 3123–3136 (2015)MathSciNetCrossRef Chandra, R.: Competition and collaboration in cooperative coevolution of Elman recurrent neural networks for time-series prediction. IEEE Trans. Neural Netw. Learn. Syst. 26, 3123–3136 (2015)MathSciNetCrossRef
9.
Zurück zum Zitat Heidrich-Meisner, V., Igel, C.: Neuroevolution strategies for episodic reinforcement learning. J. Algorithms 64(4), 152–168 (2009). Reinforcement LearningCrossRefMATH Heidrich-Meisner, V., Igel, C.: Neuroevolution strategies for episodic reinforcement learning. J. Algorithms 64(4), 152–168 (2009). Reinforcement LearningCrossRefMATH
10.
Zurück zum Zitat Happel, B.L., Murre, J.M.: Design and evolution of modular neural network architectures. Neural Networks 7(6–7), 985–1004 (1994). Models of Neurodynamics and BehaviorCrossRef Happel, B.L., Murre, J.M.: Design and evolution of modular neural network architectures. Neural Networks 7(6–7), 985–1004 (1994). Models of Neurodynamics and BehaviorCrossRef
11.
Zurück zum Zitat Clune, J., Mouret, J.-B., Lipson, H.: The evolutionary origins of modularity. Proc. R. Soc. of London B: Biol. Sci. 280(1755) (2013) Clune, J., Mouret, J.-B., Lipson, H.: The evolutionary origins of modularity. Proc. R. Soc. of London B: Biol. Sci. 280(1755) (2013)
12.
Zurück zum Zitat Ellefsen, K.O., Mouret, J.-B., Clune, J.: Neural modularity helps organismsevolve to learn new skills without forgetting old skills. PLoS Comput. Biol. 11(4), 1–24 (2015)CrossRef Ellefsen, K.O., Mouret, J.-B., Clune, J.: Neural modularity helps organismsevolve to learn new skills without forgetting old skills. PLoS Comput. Biol. 11(4), 1–24 (2015)CrossRef
13.
Zurück zum Zitat Misra, J., Saha, I.: Artificial neural networks in hardware: a survey of two decades of progress. Neurocomputing 74(13), 239–255 (2010). Artificial BrainsCrossRef Misra, J., Saha, I.: Artificial neural networks in hardware: a survey of two decades of progress. Neurocomputing 74(13), 239–255 (2010). Artificial BrainsCrossRef
15.
Zurück zum Zitat Gupta, A., Ong, Y.S., Feng, L.: Multifactorial evolution: toward evolutionary multitasking. IEEE Trans. Evol. Comput. 20(3), 343–357 (2016)CrossRef Gupta, A., Ong, Y.S., Feng, L.: Multifactorial evolution: toward evolutionary multitasking. IEEE Trans. Evol. Comput. 20(3), 343–357 (2016)CrossRef
16.
Zurück zum Zitat Gupta, A., Ong, Y.-S., Feng, L., Tan, K.C.: Multiobjective multifactorial optimization in evolutionary multitasking. IEEE Trans, Cybernetics (2016, Accepted) Gupta, A., Ong, Y.-S., Feng, L., Tan, K.C.: Multiobjective multifactorial optimization in evolutionary multitasking. IEEE Trans, Cybernetics (2016, Accepted)
17.
Zurück zum Zitat Ong, Y.-S., Gupta, A.: Evolutionary multitasking: a computer science view of cognitive multitasking. Cognitive Comput., 1–18 (2016) Ong, Y.-S., Gupta, A.: Evolutionary multitasking: a computer science view of cognitive multitasking. Cognitive Comput., 1–18 (2016)
18.
Zurück zum Zitat Chen, X., Ong, Y.-S., Lim, M.-H., Tan, K.C.: A multi-facet survey on memetic computation. IEEE Trans. Evol. Comput. 15(5), 591–607 (2011)CrossRef Chen, X., Ong, Y.-S., Lim, M.-H., Tan, K.C.: A multi-facet survey on memetic computation. IEEE Trans. Evol. Comput. 15(5), 591–607 (2011)CrossRef
19.
Zurück zum Zitat Liu, D., Hohil, M.E., Smith, S.H.: N-bit parity neural networks: new solutions based on linear programming. Neurocomputing 48(14), 477–488 (2002)CrossRefMATH Liu, D., Hohil, M.E., Smith, S.H.: N-bit parity neural networks: new solutions based on linear programming. Neurocomputing 48(14), 477–488 (2002)CrossRefMATH
20.
Zurück zum Zitat Mangal, M., Singh, M.P.: Analysis of pattern classification for the multidimensional parity-bit-checking problem with hybrid evolutionary feed-forward neural network. Neurocomputing 70(79), 1511–1524 (2007). Advances in Computational Intelligence and Learning, 14th European Symposium on Artificial Neural Networks 2006CrossRef Mangal, M., Singh, M.P.: Analysis of pattern classification for the multidimensional parity-bit-checking problem with hybrid evolutionary feed-forward neural network. Neurocomputing 70(79), 1511–1524 (2007). Advances in Computational Intelligence and Learning, 14th European Symposium on Artificial Neural Networks 2006CrossRef
21.
Zurück zum Zitat Mirjalili, S., Hashim, S.Z.M., Sardroudi, H.M.: Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Appl. Math. Comput. 218(22), 11125–11137 (2012)MathSciNetMATH Mirjalili, S., Hashim, S.Z.M., Sardroudi, H.M.: Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Appl. Math. Comput. 218(22), 11125–11137 (2012)MathSciNetMATH
22.
Zurück zum Zitat Chandra, R., Frean, M.R., Zhang, M.: Crossover-based local search in cooperative co-evolutionary feedforward neural networks. Appl. Soft Comput. 12(9), 2924–2932 (2012)CrossRef Chandra, R., Frean, M.R., Zhang, M.: Crossover-based local search in cooperative co-evolutionary feedforward neural networks. Appl. Soft Comput. 12(9), 2924–2932 (2012)CrossRef
23.
Zurück zum Zitat Deb, K., Agrawal, R.B.: Simulated binary crossover for continuous search space. Complex Syst. 9(2), 115–148 (1995)MathSciNetMATH Deb, K., Agrawal, R.B.: Simulated binary crossover for continuous search space. Complex Syst. 9(2), 115–148 (1995)MathSciNetMATH
24.
Zurück zum Zitat Deb, K., Deb, D.: Analysing mutation schemes for real-parameter genetic algorithms. Int. J. Artif. Intell. Soft Comput. 4(1), 1–28 (2014)CrossRef Deb, K., Deb, D.: Analysing mutation schemes for real-parameter genetic algorithms. Int. J. Artif. Intell. Soft Comput. 4(1), 1–28 (2014)CrossRef
Metadaten
Titel
Evolutionary Multi-task Learning for Modular Training of Feedforward Neural Networks
verfasst von
Rohitash Chandra
Abhishek Gupta
Yew-Soon Ong
Chi-Keong Goh
Copyright-Jahr
2016
DOI
https://doi.org/10.1007/978-3-319-46672-9_5