nach oben

Neural Computing and Applications

Erschienen in:

01.05.2007 | Original Article

An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training

verfasst von: Krzysztof Socha, Christian Blum

Erschienen in: Neural Computing and Applications | Ausgabe 3/2007

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Ant colony optimization (ACO) is an optimization technique that was inspired by the foraging behaviour of real ant colonies. Originally, the method was introduced for the application to discrete optimization problems. Recently we proposed a first ACO variant for continuous optimization. In this work we choose the training of feed-forward neural networks for pattern classification as a test case for this algorithm. In addition, we propose hybrid algorithm variants that incorporate short runs of classical gradient techniques such as backpropagation. For evaluating our algorithms we apply them to classification problems from the medical field, and compare the results to some basic algorithms from the literature. The results show, first, that the best of our algorithms are comparable to gradient-based algorithms for neural network training, and second, that our algorithms compare favorably with a basic genetic algorithm.

Vorheriger Artikel RRS + LS-SVM: a new strategy for “a priori” sample selection

Nächster Artikel Reconfigurable hardware for neural networks: binary versus stochastic

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Note that this paper is an extension of the work published in [7, 32]. The extension consists in a more detailed explanation of the algorithm itself, the conduction of a fourfold cross-validation for all applications to test instances, and the conduction of tests for determining the statistical significance of the obtained results.

Note that k can not be smaller than the number of dimensions of the problem being solved. This is due to the explicit handling of correlation among variables as explained in Sect. 3: In order to be able to rotate the coordinate system properly, the number of solutions available has to be at least equal to the number of dimensions.

Such pseudo-random number generators are routinely available for most programming languages.

At step i, only dimensions i through n are used.

http://www.r-project.org

Due to the limited resources for tuning, the chosen configuration for each race is not necessarily significantly better than all the others. The limit of 100 experiments per race did sometimes not allow reaching that level of assurance. However, the chosen configuration was definitely not significantly worse than any of the others.

Note that Alba and Chicano did not perform a fourfold cross-validation. They only performed the first one of our four cross-validation experiments. Therefore, the results of our ACO algorithms in these tables refer to the results of the first of our four cross-validation experiments.

Alba E, Chicano JF (2004) Training neural networks with GA hybrid algorithms. In: Deb K et al. (ed) Proceedings of the genetic and evolutionary computation conference—GECCO 2004, volume 3102 of Lecture Notes in Computer Science. Springer, Berlin, pp 852–863

Alba E, Marti R (eds) (2006) Metaheuristic procedures for training neural networks. Springer, BerlinMATH

Bilchev B, Parmee IC (1995) The ant colony metaphor for searching continuous design spaces. In: Proceedings of the AISB workshop on evolutionary computation, volume 993 of Lecture Notes in Computer Science, pp 25–39

Birattari M (2005) The problem of tuning metaheuristics as seen from a machine learning perspective. PhD thesis, volume 292 of Dissertationen zur Künstlichen Intelligenz. Akademische Verlagsgesellschaft Aka GmbH, Berlin, Germany

Birattari M, Stützle T, Paquete L, Varrentrapp K (2002) A racing algorithm for configuring metaheuristics. In: Langdon WB et al. (eds) Proceedings of the genetic and evolutionary computation conference. Morgan Kaufman, San Francisco, pp 11–18

Bishop CM (2005) Neural networks for pattern recognition. MIT Press, Cambridge

Blum C, Socha K (2005) Training feed-forward neural networks with ant colony optimization: An application to pattern classification. In: Nedjah N, Mourelle LM, Vellasco MMBR, Abraham A, Köppen M (eds) Proceedings of the Fifth International Conference on Hybrid Intelligent Systems (HIS). IEEE Computer Society, pp 233–238

Bonabeau E, Dorigo M, Theraulaz G (1999) Swarm Intelligence: From Natural to Artificial Systems. Oxford University Press, New YorkMATH

Peter AN (2000) Bosman and Dirk Thierens. Continuous iterated density estimation evolutionary algorithms within the IDEA framework. In: Pelikan M, Mühlenbein H, Rodriguez AO (eds) Proceedings of OBUPM Workshop at GECCO-2000. Morgan-Kaufmann Publishers, San Francisco, pp 197–200

10.

Box GEP, Muller ME (1958) A note on the generation of random normal deviates. Ann Math Stat 29(2):610–611

11.

Cotta C, Alba E, Sagarna R, Larrañaga P (2001) Adjusting weights in artificial neural networks using evolutionary algorithms. In: Larrañaga P, Lozano JA (eds) Estimation of distribution algorithms: a new tool for evolutionary computation. Kluwer Academic Publishers, Boston, pp 361–378

12.

Deneubourg J-L, Aron S, Goss S, Pasteels J-M (1990) The self-organizing exploratory pattern of the argentine ant. J Insect Behav 3:159–168CrossRef

13.

Dorigo M (1992) Optimization, Learning and Natural Algorithms (in Italian). PhD thesis, Dipartimento di Elettronica, Politecnico di Milano, Italy

14.

Dorigo M, Maniezzo V, Colorni A (1996) Ant System: Optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybernetics – Part B 26(1):29–41CrossRef

15.

Dorigo M, Stützle T (2004) Ant Colony Optimization. MIT Press, CambridgeMATH

16.

Dréo J, Siarry P (2002) A new ant colony algorithm using the heterarchical concept aimed at optimization of multiminima continuous functions. In: Dorigo M, Di Caro G, Sampels M (eds) Proceedings of ANTS 2002 – from ant colonies to artificial ants: third international workshop on ant algorithms, vol 2463 of lecture notes in computer science, Springer, Berlin, pp 216–221

17.

Garcia Pedrajas N, Hervás Martinez C, Muñoz Pérez J (2003) COVNET: A cooperative coevolutionary model for evolving artificial neural networks. IEEE Trans Neural Networks 14(3):575–596CrossRef

18.

Golub GH, van Loan CF (1989) Matrix computations, 2nd edn. The John Hopkins University Press, Baltimore

19.

Guntsch M, Middendorf M (2003) Solving multi-objective permutation problems with population based ACO. In: Fonseca CM, Fleming PJ, Zitzler E, Deb K, Thiele L (eds) Proceedings of the second international conference on evolutionary multi-criterion optimization (EMO 2003), vol 2636 of lecture notes in computer science. Springer, Berlin, pp 464–478

20.

Hagan MT, Menhaj MB (1994) Training feedforward networks with the marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993CrossRef

21.

Hansen N, Ostermeier A (2001) Completely derandomized self-adaptation in evolution strategies. Evol Comput 9(2):159–195CrossRef

22.

Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning. Springer, BerlinMATH

23.

Larrañaga P, Lozano JA (eds) (2001) Estimation of distribution algorithms: a new tool for evolutionary computation. Kluwer Academic Publishers, Boston

24.

Mandischer M (2002) A comparison of evolution strategies and backpropagation for neural network training. Neurocomputing 42(1):87–117MATHCrossRef

25.

McGill R, Tukey JW, Larsen WA (1978) Variations of box plots. Am Stat 32:12–16CrossRef

26.

Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of the 2002 international joint conference on neural networks (IJCNN’02), vol 2. IEEE press, pp 1895–1899

27.

Monmarché N, Venturini G, Slimane M (2000) On how pachycondyla apicalis ants suggest a new search algorithm. Future Generation Comput Syst 16:937–946CrossRef

28.

Montana D, Davis L (1989) Training feedforward neural networks using genetic algorithms. In: Proceedings of the eleventh international joint conference on artificial intelligence (IJCAI). Morgan Kaufmann, San Mateo, pp 762–767

29.

Prechelt L (1994) Proben1—a set of neural network benchmark problems and benchmarking rules. Technical Report 21, Fakultät für Informatik, Universität Karlsruhe, Karlsruhe, Germany

30.

Rumelhart D, Hinton G, Williams R (1986) Learning representations by backpropagation errors. Nature 536:323–533

31.

Socha K (2004) Extended ACO for continuous and mixed-variable optimization. In: Dorigo M, Birattari M, Blum C, Gambardella LM, Mondada F, Stützle T (eds) Proceedings of ANTS 2004 – fourth international workshop on ant algorithms and swarm intelligence. Lecture Notes in Computer Science. Springer, Berlin

32.

Socha K, Blum C (2006) Metaheuristic procedures for training neural networks. chapter ant colony optimization. Springer, Berlin (in press)

33.

Socha K, Dorigo M (2006) Ant colony optimization for continuous domains. Eur J Oper Res (in press)

34.

Socha K (2003) The influence of run-time limits on choosing ant system parameters. In: Cantu-Paz E et al. (eds) Proceedings of GECCO 2003—genetic and evolutionary computation conference, vol 2723 of LNCS. Springer, Berlin, pp 49–60

35.

Stanley KO, Miikulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127CrossRef

36.

Stützle T, Hoos HH (2000) \({{\cal MAX}\hbox{-}{\cal MIN}}\) Ant System. Future Generation Computer Systems 16(8):889–914CrossRef

37.

Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82CrossRef

38.

Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447CrossRef

Titel: An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training
verfasst von: Krzysztof Socha
Christian Blum
Publikationsdatum: 01.05.2007
Verlag: Springer-Verlag
Erschienen in: Neural Computing and Applications / Ausgabe 3/2007
Print ISSN: 0941-0643
Elektronische ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-007-0084-z

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 3/2007

Reconfigurable hardware for neural networks: binary versus stochastic

Learning with partly labeled data

RRS + LS-SVM: a new strategy for “a priori” sample selection

Using evolution to improve neural network learning: pitfalls and solutions

A data reduction approach for resolving the imbalanced data issue in functional genomics

Ensemble of hybrid neural network learning approaches for designing pharmaceutical drugs