nach oben

Erschienen in:

2015 | OriginalPaper | Buchkapitel

Evaluating Reward Definitions for Parameter Control

verfasst von : Giorgos Karafotias, Mark Hoogendoorn, A. E. Eiben

Erschienen in: Applications of Evolutionary Computation

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist. Recently, successful parameter control methods based on Reinforcement Learning (RL) have been suggested for one-off applications, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. However, the reward function used was not investigated in depth, though it is a non-trivial factor with an important impact on the performance of a RL mechanism. In this paper, we address this issue by defining and comparing four alternative reward functions for such generic and RL-based EA parameter controllers. We conducted experiments with different EAs, test problems and controllers and results showed that the simplest reward function performs at least as well as the others, making it an ideal choice for generic out-of-the-box parameter control.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Neuro-evolutionary Topology Optimization with Adaptive Improvement Threshold

Nächstes Kapitel Chromatic Selection – An Oversimplified Approach to Multi-objective Optimization

Notice that, for example, as discussed above, the reward definition that Muller et al. [14] found most efficient cannot be generalised.

Any other other information (e.g. diversity) would introduce a bias on the controller’s strategy.

If, on the contrary, we consider a controller for a repetitive application, we could train the controller off-line using multiple training runs, thus being able to take the final best fitness of each run as the reward.

We did not use the more intuitive ratio \(\frac{\varDelta _f}{Ref(n)}\) because preliminary experiments showed the difference \(\varDelta _f - Ref(N)\) to perform better.

http://coco.gforge.inria.fr/doku.php?id=bbob-2013.

The source code was acquired directly from the authors.

http://www3.ntu.edu.sg/home/epnsugan/index_files/CEC11-RWP/CEC11-RWP.htm. The source code of GA MPC is available at the same competition page.

The source code for the (IPOP) CMA-ES was acquired from https://www.lri.fr/hansen/cmaes_inmatlab.html. The 10DDr variation was added by us.

For a comparison of the controllers used in this paper to other benchmarks we refer the reader to [10] and [12].

The source code for this experiment is available for download at http://www.few.vu.nl/~gks290/resources/evostar2015.tar.gz.

Alba, E., Dorronsoro, B.: Cellular Genetic Algorithms, 1st edn. Springer, Heidelberg (2008)MATH

Chen, F., Gao, Y., Chen, Z., Chen, S.: SCGA: controlling genetic algorithms with Sarsa(0). In: International Conference on Computational Intelligence for Modelling, Control and Automation, 2005 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, vol. 1, pp. 1177–1183 (2005)

Eiben, A.E., Horvath, M., Kowalczyk, W., Schut, M.C.: Reinforcement learning for online control of evolutionary algorithms. In: Brueckner, S.A., Hassas, S., Jelasity, M., Yamins, D. (eds.) ESOA 2006. LNCS (LNAI), vol. 4335, pp. 151–160. Springer, Heidelberg (2007) CrossRef

Eiben, A.E., Hinterding, R., Michalewicz, Z.: Parameter control in evolutionary algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999)CrossRef

Eiben, A.E., Smith, J.: Introduction to Evolutionary Computing. Springer, Heidelberg (2003) CrossRefMATH

Elsayed, S., Sarker, R., Essam, D.: GA with a new multi-parent crossover for solving IEEE-CEC2011 competition problems. In: Proceedings of the 2011 IEEE Congress on Evolutionary Computation, pp. 1034–1040. IEEE Press, New Orleans, USA (2011)

Fialho, Á., Da Costa, L., Schoenauer, M., Sebag, M.: Extreme value based adaptive operator selection. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 175–184. Springer, Heidelberg (2008) CrossRef

Gong, W., Fialho, A., Cai, Z.: Adaptive strategy selection in differential evolution. In: Pelikan, M., Branke, J. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2010), pp. 409–416. ACM, Portland (2010)

Holtschulte, N.J., Moses, M.: Benchmarking cellular genetic algorithms on the BBOB noiseless testbed. In: Proceeding of the Fifteenth Annual Conference Companion on Genetic and Evolutionary Computation Conference Companion, GECCO 2013 Companion, pp. 1201–1208. ACM (2013)

10.

Karafotias, G., Eiben, A.E., Hoogendoorn, M.: Generic parameter control with reinforcement learning. In: Arnold, D.V. (ed.) GECCO 2014: Proceedings of the 16th annual conference on Genetic and evolutionary computation, pp. 1319–1326. ACM, New York (2014)CrossRef

11.

Karafotias, G., Hoogendoorn, M., Eiben, A.E.: Parameter control in evolutionary algorithms: trends and challenges. IEEE Transactions on Evolutionary Computation (2014, to appear). doi:10.1109/TEVC.2014.2308294

12.

Karafotias, G., Hoogendoorn, M., Weel, B.: Comparing generic parameter controllers for EAs. In: 2014 IEEE Symposium on Foundations of Computational Intelligence (FOCI), pp. 46–53, December 2014

13.

Liao, T., Stützle, T.:. Bounding the population size of IPOP-CMA-ES on the noiseless BBOB testbed. In: Proceeding of the Fifteenth Annual Conference Companion on Genetic and Evolutionary Computation Conference Companion, pp. 1161–1168. ACM (2013)

14.

Muller, S., Schraudolph, N., Koumoutsakos, P.: Step size adaptation in evolution strategies using reinforcement learning. In: 2002 Congress on Evolutionary Computation (CEC 2002), pp. 151–156. IEEE Press, Piscataway, NJ, Honolulu, USA 12–17 May 2002

15.

Pettinger, J., Everson, R.: Controlling genetic algorithms with reinforcement learning. In: Langdon, W., et al. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2002), page 692, pp. 9–13. Morgan Kaufmann, San Francisco (2002)

16.

Sakurai, Y., Takada, K., Kawabe, T., Tsuruta, S.: A method to control parameters of evolutionary algorithms by using reinforcement learning. In: 2010 Sixth International Conference on Signal-Image Technology and Internet-Based Systems (SITIS), pp. 74–79 (2010)

17.

Schwefel, H.-P.: Evolution and Optimum Seeking. Wiley, New York (1995)

18.

Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)

19.

Uther, W.B., Veloso, M.: Tree based discretization for continuous state space reinforcement learning. In: Proceedings of the Fifteenth National/tenth Conference on Artificial Intelligence/innovative Applications of Artificial Intelligence, AAAI 1998/IAAI 1998, pp. 769–774. American Association for Artificial Intelligence (1998)

Titel: Evaluating Reward Definitions for Parameter Control
verfasst von: Giorgos Karafotias
Mark Hoogendoorn
A. E. Eiben
Verlag: Springer International Publishing
Buch: Applications of Evolutionary Computation
Print ISBN: 978-3-319-16548-6

Electronic ISBN: 978-3-319-16549-3

Copyright-Jahr: 2015
DOI: https://doi.org/10.1007/978-3-319-16549-3_54

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner