Abstract
Evolutionary design of neural networks has shown a great potential as a powerful optimization tool. However, most evolutionary neural networks have not taken advantage of the fact that they can evolve from modules. This paper presents a hybrid method of modular neural networks and genetic programming as a promising model for evolutionary learning. This paper describes the concepts and methodologies for the evolvable model of modular neural networks, which might not only develop new functionality spontaneously, but also grow and evolve its own structure autonomously. We show the potential of the method by applying an evolved modular network to a visual categorization task with handwritten digits. Sophisticated network architectures as well as functional subsystems emerge from an initial set of randomly-connected networks. Moreover, the evolved neural network has reproduced some of the characteristics of natural visual system, such as the organization of coarse and fine processing of stimuli in separate pathways.
Similar content being viewed by others
References
S.A. Harp, “Towards the genetic synthesis of neural networks,” in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, Morgan Kaufmann: San Mateo, CA, 1989, pp. 360- 369.
D. Whitley and T. Hanson, “Optimizing neural networks using faster, more accurate genetic search,” in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, Morgan Kaufmann: San Mateo, CA, 1989, pp. 391- 396.
H. Kitano, “Designing neural networks using genetic algorithms with graph generation system,” Complex Systems, vol. 4, no.4, pp. 461- 476, 1990.
D.T. Cliff, I. Harvey, and P. Husbands, “Incremental evolution of neural network architectures for adaptive behavior,” Technical Report CSRP 256, University of Sussex School of Cognitive and Computing Science, 1992.
X. Yao, “Evolutionary artificial neural networks,” Int. Journal of Neural Systems, vol. 4, no.3, pp. 203- 222, 1993.
S. Nolfi, O. Miglino, and D. Parisi, “Phenotypic plasticity in evolving neural networks: Evolving the control system for an autonomous agent,” Technical Report PCIA-94-04, Institute of Psychology, C.N.R., Rome, 1994.
S.-B. Cho and K. Shimohara, “Toward evolvable model of modularized neural networks,” in Proc. the 5th Annual Conf. Japanese Neural Network Society, Tsukuba, November 1994, pp. 117- 118.
G.F. Miller, P.M. Todd, and S.U. Hedge, “Designing neural networks using genetic algorithms,” in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, Morgan Kaufmann: San Mateo, CA, 1989, pp. 379- 384.
S.-B. Cho and J.H. Kim, “Combining multiple neural networks by fuzzy integral for robust classification,” IEEE Trans. Systems, Man, and Cybernetics, vol. 25, no.2, pp. 380- 384, 1995.
R.A. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton, “Adaptive mixtures of local experts,” Neural Computation, vol. 3, pp. 79- 87, 1991.
J.B. Hampshire II and A. Waibel, “The meta-pi network: Building distributed knowledge representations for robust multisource pattern recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no.7, pp. 751- 769, 1992.
F. Gruau and D. Whitley, “The cellular development of neural networks: The interaction of learning and evolution,” Research Report 93-04, Laboratoire de l'Informatique du Parallelisme, Ecole Normale Superieure de Lyon, 1993.
S. Nolfi and D. Parisi, “Growing neural networks,” Report PCIA-91-15, Institute of Psychology, C.N.R., Rome, 1991.
E. Mjolsness, D.H. Sharp, and J. Reinitz, “A connectionist model of development,” Journal of Theoretical Biology, vol. 152, pp. 429- 453, 1991.
K. Fleischer and A.H. Barr, “A simulation testbed for the study of multicellular development: The multiple mechanisms of morphogenesis,” in Artificial Life III, edited by C.G. Langton, Addison-Wesley: Reading, MA, pp. 389- 416, 1994.
J.R. Koza, Genetic Programming on the Programming of Computers by Means of Natural Selection, The MIT Press, 1992.
D. Whitley, “The GENITOR algorithm and selective pressure: Why rank-based allocation of reproductive trials is best,” in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, Morgan Kaufmann: San Mateo, CA, 1989, pp. 116- 121
J.M.J. Murre, R.H. Phaf, and G. Wolters, “CALM: Categorizing and learning module,” Neural Networks, vol. 5, pp. 55- 82, 1992.
D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, 1989.
C.Y. Suen, C. Nadal, T. Mai, R. Legault, and L. Lam, “Recognition of handwritten numerals based on the concept of multiple experts,” in Proc. 1st Int. Workshop Frontiers in Handwriting Recognition, Montreal, Canada, 1990, pp. 131- 144.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Cho, SB., Shimohara, K. Evolutionary Learning of Modular Neural Networks with Genetic Programming. Applied Intelligence 9, 191–200 (1998). https://doi.org/10.1023/A:1008388118869
Issue Date:
DOI: https://doi.org/10.1023/A:1008388118869