Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Hindsight Experience Replay with Evolutionary Decision Trees for Curriculum Goal Generation Read chapter Read first chapter

2024 | OriginalPaper | Chapter

Leveraging More of Biology in Evolutionary Reinforcement Learning

Authors : Bruno Gašperov, Marko Đurasević, Domagoj Jakobovic

Published in: Applications of Evolutionary Computation

Publisher: Springer Nature Switzerland

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this paper, we survey the use of additional biologically inspired mechanisms, principles, and concepts in the area of evolutionary reinforcement learning (ERL). While recent years have witnessed the emergence of a swath of metaphor-laden approaches, many merely echo old algorithms through novel metaphors. Simultaneously, numerous promising ideas from evolutionary biology and related areas, ripe for exploitation within evolutionary machine learning, remain in relative obscurity. To address this gap, we provide a comprehensive analysis of innovative, often unorthodox approaches in ERL that leverage additional bio-inspired elements. Furthermore, we pinpoint research directions in the field with the largest potential to yield impactful outcomes and discuss classes of problems that could benefit the most from such research.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Towards Physical Plausibility in Neuroevolution Systems
next chapter A Hierarchical Dissimilarity Metric for Automated Machine Learning Pipelines, and Visualizing Search Behaviour
Footnotes
1
It should be noted that ERL also includes approaches in which different state-action pairs are directly explored, as well as meta-RL methods.
 
Literature
1.
Aranha, C., et al.: Metaphor-based metaheuristics, a call for action: the elephant in the room. Swarm Intell. 16(1), 1–6 (2022)CrossRef
2.
3.
Kutschera, U., Niklas, K.J.: The modern theory of biological evolution: an expanded synthesis. Naturwissenschaften 91, 255–276 (2004)CrossRef
4.
Barton, N.H.: The “new synthesis’’. Proc. Nat. Acad. Sci. 119(30), e2122147119 (2022)CrossRef
5.
Yuen, S., Ezard, T.H.G., Sobey, A.J.: Epigenetic opportunities for evolutionary computation. R. Soc. Open Sci. 10(5), 221256 (2023)CrossRef
6.
Grudniewski, P.A., Sobey, A.J.: cMLSGA: a co-evolutionary multi-level selection genetic algorithm for multi-objective optimization. arXiv preprint arXiv:​2104.​11072 (2021)
7.
Barton, N., Paixão, T.: Can quantitative and population genetics help us understand evolutionary computation? In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 1573–1580 (2013)
8.
Pontius, J.U., et al.: Initial sequence and comparative analysis of the cat genome. Genome Res. 17(11), 1675–1689 (2007)CrossRef
9.
Vassiliades, V., Mouret, J.-B.: Discovering the elite hypervolume by leveraging interspecies correlation. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 149–156 (2018)
10.
11.
Vie, A., Kleinnijenhuis, A.M., Farmer, D.J.: Qualities, challenges and future of genetic algorithms: a literature review. arXiv preprint arXiv:​2011.​05277 (2020)
12.
Dagdia, Z.C., Avdeyev, P., Bayzid, M.S.: Biological computation and computational biology: survey, challenges, and discussion. Artif. Intell. Rev. 54, 4169–4235 (2021)CrossRef
13.
Miikkulainen, R., Forrest, S.: A biological perspective on evolutionary computation. Nat. Mach. Intell. 3(1), 9–15 (2021)CrossRef
14.
Silver, D., et al.: Mastering the game of go without human knowledge. nature 550(7676), 354–359 (2017)CrossRef
15.
Nguyen, H., La, H.: Review of deep reinforcement learning for robot manipulation. In: 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 590–595. IEEE (2019)
16.
Zhou, S.K., Le, H.N., Luu, K., Nguyen, H.V., Ayache, N.: Deep reinforcement learning in medical imaging: a literature review. Med. Image Anal. 73, 102193 (2021)CrossRef
17.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:​1707.​06347 (2017)
18.
Qian, H., Yang, Yu.: Derivative-free reinforcement learning: a review. Front. Comp. Sci. 15(6), 156336 (2021)CrossRef
19.
Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O., Clune, J.: Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:​1712.​06567 (2017)
20.
Salimans, T., Ho, J., Chen, X., Sidor, S., Sutskever, I.: Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:​1703.​03864 (2017)
21.
22.
Sun, H., Zhang, W., Runxiang, Yu., Zhang, Y.: Motion planning for mobile robots-focusing on deep reinforcement learning: a systematic review. IEEE Access 9, 69061–69081 (2021)CrossRef
23.
Jin, Y., Branke, J.: Evolutionary optimization in uncertain environments - a survey. IEEE Trans. Evol. Comput. 9(3), 303–317 (2005)CrossRef
24.
Jiang, M., Huang, Z., Qiu, L., Huang, W., Yen, G.G.: Transfer learning-based dynamic multiobjective optimization algorithms. IEEE Trans. Evol. Comput. 22(4), 501–514 (2017)CrossRef
25.
Stanley, K.O., Lehman, J., Soros, L.: Open-endedness: the last grand challenge you’ve never heard of (2017)
26.
Mora, C., Tittensor, D.P., Adl, S., Simpson, A.G.B., Worm, B.: How many species are there on earth and in the ocean? PLoS Biol. 9(8), e1001127 (2011)CrossRef
27.
Rasmussen, S., Sibani, P.: Two modes of evolution: optimization and expansion. Artif. Life 25(1), 9–21 (2019)CrossRef
28.
Packard, N., et al.: An overview of open-ended evolution: editorial introduction to the open-ended evolution ii special issue. Artif. Life 25(2), 93–103 (2019)CrossRef
29.
30.
Pugh, J.K., Soros, L.B., Stanley, K.O.: Quality diversity: a new frontier for evolutionary computation. Front. Robot. AI 3, 40 (2016)CrossRef
31.
Pugh, J.K., Soros, L.B., Szerlip, P.A., Stanley, K.O.: Confronting the challenge of quality diversity. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 967–974 (2015)
32.
Earle, S., Snider, J., Fontaine, M.C., Nikolaidis, S., Togelius, J.: Illuminating diverse neural cellular automata for level generation. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 68–76 (2022)
33.
Chand, S., Howard, D.: Path towards multilevel evolution of robots. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1381–1382 (2020)
34.
35.
Riederer, J.M., Tiso, S., van Eldijk, T.J.B., Weissing, F.J.: Capturing the facets of evolvability in a mechanistic framework. Trends Ecol. Evol. 37(5), 430–439 (2022)CrossRef
36.
Dawkins, R.: The evolution of evolvability. In: Artificial Life, pp. 201–220. Routledge (2019)
37.
Watson, R.A., Szathmáry, E.: How can evolution learn? Trends Ecol. Evol. 31(2), 147–157 (2016)CrossRef
38.
Lehman, J., Stanley, K.O.: Evolvability is inevitable: increasing evolvability without the pressure to adapt. PLoS ONE 8(4), e62186 (2013)CrossRef
39.
Mengistu, H., Lehman, J., Clune, J.: Evolvability search: directly selecting for evolvability in order to study and produce it. In: 2016 Proceedings of the Genetic and Evolutionary Computation Conference, pp. 141–148 (2016)
40.
Gajewski, A., Clune, J., Stanley, K.O., Lehman, J.: Evolvability ES: scalable and direct optimization of evolvability. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 107–115 (2019)
41.
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135. PMLR (2017)
42.
Katona, A., Franks, D.W., Walker, J.A.: Quality evolvability ES: evolving individuals with a distribution of well performing and diverse offspring. In: The 2022 Conference on Artificial Life, ALIFE 2022. MIT Press (2021)
43.
Gašperov, B., Đurasević, M.: On evolvability and behavior landscapes in neuroevolutionary divergent search. arXiv preprint arXiv:​2306.​09849 (2023)
44.
Doncieux, S., Paolo, G., Laflaquière, A., Coninx, A.: Novelty search makes evolvability inevitable. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 85–93 (2020)
45.
Shorten, D., Nitschke, G.: How evolvable is novelty search? In: 2014 IEEE International Conference on Evolvable Systems, pp. 125–132. IEEE (2014)
46.
Medvet, E., Daolio, F., Tagliapietra, D.: Evolvability in grammatical evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 977–984 (2017)
47.
Liu, D., Virgolin, M., Alderliesten, T., Bosman, P.A.N.: Evolvability degeneration in multi-objective genetic programming for symbolic regression. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 973–981 (2022)
48.
Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Computat. 11(1), 1–18 (2003)CrossRef
49.
50.
Krause, O., Arbonès, D.R., Igel, C.: CMA-ES with optimal covariance update and storage complexity. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
51.
Heidrich-Meisner, V., Igel, C.: Uncertainty handling CMA-ES for reinforcement learning. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1211–1218 (2009)
52.
Branke, J., Mattfeld, D.C.: Anticipation and flexibility in dynamic scheduling. Int. J. Prod. Res. 43(15), 3103–3129 (2005)CrossRef
53.
Pinto, L., Davidson, J., Sukthankar, R., Gupta, A.: Robust adversarial reinforcement learning. In: International Conference on Machine Learning, pp. 2817–2826. PMLR (2017)
54.
Masel, J., Trotter, M.V.: Robustness and evolvability. Trends Genet. 26(9), 406–414 (2010)CrossRef
55.
Wagner, A.: Robustness and evolvability: a paradox resolved. Proc. R. Soc. B Biol. Sci. 275(1630), 91–100 (2008)CrossRef
56.
Spencer, C.C.A., et al.: The influence of recombination on human genetic diversity. PLoS Genet. 2(9), e148 (2006)CrossRef
57.
Zainuddin, F.A., Samad, Md.F.A., Tunggal, D.: A review of crossover methods and problem representation of genetic algorithm in recent engineering applications. Int. J. Adv. Sci. Technol. 29(6s), 759–769 (2020)
58.
Paixão, T., Barton, N.: A variance decomposition approach to the analysis of genetic algorithms. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 845–852 (2013)
59.
60.
Mitchell, M., Holland, J.H., Forrest, S.: The royal road for genetic algorithms: fitness landscapes and GA performance. Technical report, Los Alamos National Lab., NM (United States) (1991)
61.
Polani, D., Miikkulainen, R.: Fast reinforcement learning through eugenic neuro-evolution, pp. 99–277. The University of Texas at Austin, AI (1999)
62.
Polani, D., Miikkulainen, R.: Eugenic neuro-evolution for reinforcement learning. In: Proceedings of the 2nd Annual Conference on Genetic and Evolutionary Computation, pp. 1041–1046 (2000)
63.
Ventresca, M., Ombuki-Berman, B.: Epistasis in multi-objective evolutionary recurrent neuro-controllers. In: 2007 IEEE Symposium on Artificial Life, pp. 77–84. IEEE (2007)
64.
65.
Huizinga, J., Stanley, K.O., Clune, J.: The emergence of canalization and evolvability in an open-ended, interactive evolutionary system. Artif. Life 24(3), 157–181 (2018)CrossRef
66.
Katona, A., Lourenço, N., Machado, P., Franks, D.W., Walker, J.A.: Utilizing the untapped potential of indirect encoding for neural networks with meta learning. In: Castillo, P.A., Jiménez Laredo, J.L. (eds.) EvoApplications 2021. LNCS, vol. 12694, pp. 537–551. Springer, Cham (2021). https://​doi.​org/​10.​1007/​978-3-030-72699-7_​34CrossRef
67.
Wang, R., et al.: Enhanced poet: open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In: International Conference on Machine Learning, pp. 9940–9951. PMLR (2020)
68.
Karafotias, G., Hoogendoorn, M., Eiben, Á.E.: Parameter control in evolutionary algorithms: trends and challenges. IEEE Trans. Evol. Comput. 19(2), 167–187 (2014)CrossRef
69.
Rand, W.: Genetic Algorithms in Dynamic and Coevolving Environments. Ph.D. thesis. Citeseer
70.
Bedau, M.A., Packard, N.H.: Evolution of evolvability via adaptation of mutation rates. Biosystems 69(2–3), 143–162 (2003)CrossRef
71.
Aleti, A.: An adaptive approach to controlling parameters of evolutionary algorithms. Swinburne University of Technology (2012)
72.
Xu, K., Ma, Y., Li, W.: Dynamics-aware novelty search with behavior repulsion. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1112–1120 (2022)
73.
Weber, M., Schübeler, D.: Genomic patterns of DNA methylation: targets and function of an epigenetic mark. Curr. Opin. Cell Biol. 19(3), 273–280 (2007)CrossRef
74.
Turner, B.M.: Histone acetylation and an epigenetic code. BioEssays 22(9), 836–845 (2000)CrossRef
75.
Hu, T.: Evolvability and rate of evolution in evolutionary computation. Ph.D. thesis, Memorial University of Newfoundland (2010)
76.
Wang, Y., Liu, H., Sun, Z.: Lamarck rises from his grave: parental environment-induced epigenetic inheritance in model organisms and humans. Biol. Rev. 92(4), 2084–2111 (2017)CrossRef
77.
Mukhlish, F., Page, J., Bain, M.: Reward-based epigenetic learning algorithm for a decentralised multi-agent system. Int. J. Intell. Unmanned Syst. 8(3), 201–224 (2020)CrossRef
78.
Mukhlish, F., Page, J., Bain, M.: From reward to histone: combining temporal-difference learning and epigenetic inheritance for swarm’s coevolving decision making. In: 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp. 1–6. IEEE (2020)
79.
Page, J., Armstrong, R., Mukhlish, F.: Simulating search and rescue operations using swarm technology to determine how many searchers are needed to locate missing persons/objects in the shortest time. In: Naweed, A., Bowditch, L., Sprick, C. (eds.) ASC 2019. CCIS, vol. 1067, pp. 106–112. Springer, Singapore (2019). https://​doi.​org/​10.​1007/​978-981-32-9582-7_​8CrossRef
80.
81.
Boyko, A., Kovalchuk, I.: Epigenetic control of plant stress response. Environ. Mol. Mutagen. 49(1), 61–72 (2008)CrossRef
82.
Khetarpal, K., Riemer, M., Rish, I., Precup, D.: Towards continual reinforcement learning: a review and perspectives. J. Artif. Intell. Res. 75, 1401–1476 (2022)MathSciNetCrossRef
83.
Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: zeros, signs, and the supermask. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
84.
Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., Rastegari, M.: What’s hidden in a randomly weighted neural network? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893–11902 (2020)
85.
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:​1803.​03635 (2018)
86.
Kimura, M.: The Neutral Theory of Molecular Evolution. Cambridge University Press, Cambridge (1983)CrossRef
87.
88.
Dal Piccol Sotto, L.F., Mayer, S., Garcke, J.: The pole balancing problem from the viewpoint of system flexibility. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 427–430 (2022)
89.
Galván-López, E., Poli, R., Kattan, A., O’Neill, M., Brabazon, A.: Neutrality in evolutionary algorithms\(\ldots \) what do we know? Evol. Syst. 2, 145–163 (2011)CrossRef
90.
Odling-Smee, F.J., Laland, K.N., Feldman, M.W.: Niche Construction: The Neglected Process in Evolution (MPB-37). Princeton University Press (2013)
91.
Flynn, E.G., Laland, K.N., Kendal, R.L., Kendal, J.R.: Target article with commentaries: developmental niche construction. Dev. Sci. 16(2), 296–313 (2013)CrossRef
92.
Dawkins, R.: The Extended Phenotype: The Long Reach of the Gene. Oxford University Press (2016)
93.
Millhouse, T., Moses, M., Mitchell, M.: Frontiers in evolutionary computation: a workshop report. arXiv preprint arXiv:​2110.​10320 (2021)
94.
Perolat, J., Leibo, J.Z., Zambaldi, V., Beattie, C., Tuyls, K., Graepel, T.: A multi-agent reinforcement learning model of common-pool resource appropriation. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
95.
96.
Hamon, G., Nisioti, E., Moulin-Frier, C.: Eco-evolutionary dynamics of non-episodic neuroevolution in large multi-agent environments. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 143–146 (2023)
97.
Berseth, G., et al.: SMiRL: surprise minimizing reinforcement learning in unstable environments. arXiv preprint arXiv:​1912.​05510 (2019)
98.
Friston, K.: The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. 13(7), 293–301 (2009)CrossRef
99.
Lipson, H., et al.: Principles of modularity, regularity, and hierarchy for scalable systems. J. Biol. Phys. Chem. 7(4), 125 (2007)CrossRef
100.
Mengistu, H., Huizinga, J., Mouret, J.-B., Clune, J.: The evolutionary origins of hierarchy. PLoS Comput. Biol. 12(6), e1004829 (2016)CrossRef
101.
Clune, J., Mouret, J.-B., Lipson, H.: The evolutionary origins of modularity. Proc. R. Soc. B Biol. Sci. 280(1755), 20122863 (2013)CrossRef
102.
Hutsebaut-Buysse, M., Mets, K., Latré, S.: Hierarchical reinforcement learning: a survey and open research challenges. Mach. Learn. Knowl. Extr. 4(1), 172–221 (2022)CrossRef
103.
Abramowitz, S., Nitschke, G.: Scalable evolutionary hierarchical reinforcement learning. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 272–275 (2022)
104.
Hansen, T.F.: The evolution of genetic architecture. Annu. Rev. Ecol. Evol. Syst. 37, 123–157 (2006)CrossRef
105.
106.
Smith, S.D., Pennell, M.W., Dunn, C.W., Edwards, S.V.: Phylogenetics is the new genetics (for most of biodiversity). Trends Ecol. Evol. 35(5), 415–425 (2020)CrossRef
107.
108.
Cussat-Blanc, S., Harrington, K., Pollack, J.: Gene regulatory network evolution through augmenting topologies. IEEE Trans. Evol. Comput. 19(6), 823–837 (2015)CrossRef
109.
Dolson, E., Ofria, C.: Ecological theory provides insights about evolutionary computation. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 105–106 (2018)
110.
Moreno, M.A., Dolson, E., Rodriguez-Papa, S.: Toward phylogenetic inference of evolutionary dynamics at scale. In: Artificial Life Conference Proceedings 35, vol. 2023, p. 79 (2023)
111.
112.
Salehi, A., Coninx, A., Doncieux, S.: Few-shot quality-diversity optimization. IEEE Robot. Autom. Lett. 7(2), 4424–4431 (2022)CrossRef
113.
Rainford, P.F., Porter, B.: Using phylogenetic analysis to enhance genetic improvement. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 849–857 (2022)
114.
Knapp, J.S., Peterson, G.L.: Natural evolution speciation for NEAT. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 1487–1493. IEEE (2019)
115.
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRef
116.
Dixit, G.: Learning to coordinate in sparse asymmetric multiagent systems (2023)
117.
118.
Turney, P., Whitley, D., Anderson, R.W.: Evolution, learning, and instinct: 100 years of the Baldwin effect. Evol. Comput. 4(3), iv–viii (1996)
119.
Abrantes, J.P., Abrantes, A.J., Oliehoek, F.A.: Mimicking evolution with reinforcement learning. arXiv preprint arXiv:​2004.​00048 (2020)
120.
Stanton, C., Clune, J.: Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PLoS ONE 11(9), e0162235 (2016)CrossRef
121.
122.
Schmidgall, S.: Adaptive reinforcement learning through evolving self-modifying neural networks. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 89–90 (2020)
123.
Yaman, A., Iacca, G., Mocanu, D.C., Coler, M., Fletcher, G., Pechenizkiy, M.: Evolving plasticity for autonomous learning under changing environmental conditions. Evol. Comput. 29(3), 391–414 (2021)CrossRef
124.
Davies, A.: On the interaction of function, constraint and complexity in evolutionary systems. Ph.D. thesis, University of Southampton (2014)
125.
Macallum, A.B.: The paleochemistry of the body fluids and tissues. Physiol. Rev. 6(2), 316–357 (1926)CrossRef
126.
127.
Stickland, A.C., Murray, I.: BERT and PALs: projected attention layers for efficient adaptation in multi-task learning. In: International Conference on Machine Learning, pp. 5986–5995. PMLR (2019)
128.
Sunagawa, J., Yamaguchi, R., Nakaoka, S.: Evolving neural networks through bio-inspired parent selection in dynamic environments. Biosystems 218, 104686 (2022)CrossRef
129.
Tang, Y., Nguyen, D., Ha, D.: Neuroevolution of self-interpretable agents. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 414–424 (2020)
130.
Gaier, A., Ha, D.: Weight agnostic neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
131.
Freeman, D., Ha, D., Metz, L.: Learning to predict without looking ahead: world models without forward prediction. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
132.
Fisher, R.A.: XV.-the correlation between relatives on the supposition of mendelian inheritance. Earth Environ. Sci. Trans. R. Soc. Edinburgh 52(2), 399–433 (1919)
133.
Smith, D., Tokarchuk, L., Wiggins, G.: Exploring conflicting objectives with MADNS: multiple assessment directed novelty search. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 23–24 (2016)
134.
Smith, D., Tokarchuk, L., Wiggins, G.: Harnessing phenotypic diversity towards multiple independent objectives. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 961–968 (2016)
135.
Uiterwaal, S.F., Lagerstrom, I.T., Luhring, T.M., Salsbery, M.E., DeLong, J.P.: Trade-offs between morphology and thermal niches mediate adaptation in response to competing selective pressures. Ecol. Evol. 10(3), 1368–1377 (2020)CrossRef
136.
Walsh, B.: Crops can be strong and sensitive. Nat. Plants 3(9), 694–695 (2017)CrossRef
137.
Ofria, C., Adami, C., Collier, T.C.: Selective pressures on genomes in molecular evolution. J. Theoret. Biol. 222(4), 477–483 (2003)MathSciNetCrossRef
138.
Back, T.: Selective pressure in evolutionary algorithms: a characterization of selection mechanisms. In: Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence, pp. 57–62. IEEE (1994)
139.
140.
Gottlieb, J., Oudeyer, P.-Y.: Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19(12), 758–770 (2018)CrossRef
141.
142.
Santucci, V.G., Oudeyer, P.-Y., Barto, A., Baldassarre, G.: Intrinsically motivated open-ended learning in autonomous robots. Front. Neurorobot. 3, 115 (2020) CrossRef
143.
Colas, C., Karch, T., Sigaud, O., Oudeyer, P.-Y.: Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. J. Artif. Intell. Res. 74, 1159–1199 (2022)MathSciNetCrossRef
144.
Georgeon, O.L., Marshall, J.B., Gay, S.: Interactional motivation in artificial systems: between extrinsic and intrinsic motivation. In: 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1–2. IEEE (2012)
145.
Reinitz, J., Vakulenko, S., Grigoriev, D., Weber, A.: Adaptation, fitness landscape learning and fast evolution. F1000Research 8, 358 (2019)
146.
Kouvaris, K.: How evolution learns to evolve: principles of induction in the evolution of adaptive potential. Ph.D. thesis, University of Southampton (2018)
147.
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)CrossRef
148.
Bai, H., Cheng, R., Jin, Y.: Evolutionary reinforcement learning: a survey. Intell. Comput. 2, 0025 (2023)CrossRef
149.
150.
Draghi, J., Wagner, G.P.: Evolution of evolvability in a developmental model. Evolution 62(2), 301–315 (2008)CrossRef
151.
152.
Lehman, J., Stanley, K.O.: Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 211–218 (2011)
153.
154.
Banzhaf, W., et al.: Defining and simulating open-ended novelty: requirements, guidelines, and challenges. Theor. Biosci. 135, 131–161 (2016)CrossRef
155.
Dawkins, R.: The Selfish Gene. Oxford University Press (2016)
156.
Song, X., Gao, W., Yang, Y., Choromanski, K., Pacchiano, A., Tang, Y.: ES-MAML: simple hessian-free meta learning. arXiv preprint arXiv:​1910.​01215 (2019)
157.
Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press (2006)
158.
Finn, C., Rajeswaran, A., Kakade, S., Levine, S.: Online meta-learning. In: International Conference on Machine Learning, pp. 1920–1930. PMLR (2019)
159.
Yao, H., Zhou, Y., Mahdavi, M., Li, Z.J., Socher, R., Xiong, C.: Online structured meta-learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6779–6790 (2020)
160.
161.
Cully, A.: Multi-emitter map-elites: improving quality, diversity and data efficiency with heterogeneous sets of emitters. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 84–92 (2021)
162.
Mercado, R., Munoz-Jimenez, V., Ramos, M., Ramos, F.: Generation of virtual creatures under multidisciplinary biological premises. Artif. Life Robot. 27(3), 495–505 (2022)CrossRef
163.
Stock, M., Gorochowski, T.: Open-endedness in synthetic biology: a route to continual innovation for biological design. Sci. Adv. 10, eadi3621 (2023)
164.
Borg, J.M., Buskell, A., Kapitany, R., Powers, S.T., Reindl, E., Tennie, C.: Evolved open-endedness in cultural evolution: a new dimension in open-ended evolution research. Arti. Life, 1–22 (2023)
165.
Samvelyan, M., et al.: Minihack the planet: a sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:​2109.​13202 (2021)
166.
Menashe, J., Stone, P.: Escape room: a configurable testbed for hierarchical reinforcement learning. arXiv preprint arXiv:​1812.​09521 (2018)
167.
Kaznatcheev, A.: Algorithmic biology of evolution and ecology. Ph.D. thesis, University of Oxford (2020)
168.
Metadata
Title
Leveraging More of Biology in Evolutionary Reinforcement Learning
Authors
Bruno Gašperov
Marko Đurasević
Domagoj Jakobovic
Copyright Year
2024
Book
Applications of Evolutionary Computation

Print ISBN: 978-3-031-56854-1

Electronic ISBN: 978-3-031-56855-8

Copyright Year: 2024

DOI
https://doi.org/10.1007/978-3-031-56855-8_6

Premium Partner

    Image Credits