Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
ATZ-Fachkongress

Chassis.tech plus 2024


4 Kongresse in einer Veranstaltung

Treffen Sie in München die Fachleute von Chassis, Lenkung, Bremsen sowie Reifen/Räder.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
Fundamentals of Evolutionary Machine Learning Kapitel lesen Erstes Kapitel lesen

2024 | OriginalPaper | Buchkapitel

2. Evolutionary Supervised Machine Learning

verfasst von : Risto Miikkulainen

Erschienen in: Handbook of Evolutionary Machine Learning

Verlag: Springer Nature Singapore

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

This chapter provides an overview of evolutionary approaches to supervised learning. It starts with the definition and scope of the opportunity, and then reviews three main areas: evolving general neural network designs, evolving solutions that are explainable, and forming a synergy of evolutionary and gradient-based methods.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Fundamentals of Evolutionary Machine Learning
Nächstes Kapitel EML for Unsupervised Learning
Literatur
1.
Aharonov-Barki, R., Beker, T., Ruppin, E.: Emergence of memory-driven command neurons in evolved artificial agents. Neural Comput. 13, 691–716 (2001)CrossRefMATH
2.
Aitkenhead, M.J.: A co-evolving decision tree classification method. Expert Syst. Appl. 34, 19–25 (2008)CrossRef
3.
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Precup, D., Teh, Y.W. (eds.), Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 214–223 (2017)
4.
Barros, R.C., Basgalupp, M.P., De Carvalho, A.C., Freitas, A.A.: A survey of evolutionary algorithms for decision-tree induction. IEEE Trans. Syst., Man, Cybern., Part C (Appl. Rev.) 42, 291–312 (2012)
5.
6.
Bingham, G., Macke, W., Miikkulainen, R.: Evolutionary optimization of deep learning activation functions. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 289–296 (2020)
7.
Bingham, G., Miikkulainen, R.: Discovering parametric activation functions. Neural Netw. 148, 48–65 (2022)CrossRef
8.
9.
Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. Chapman and Hall/CRC (1984)
10.
Butz, M.V., Lanzi, P.L., Wilson, S.W.: Function approximation with xcs: Hyperellipsoidal conditions, recursive least squares, and compaction. IEEE Trans. Evolut. Comput. 12, 355–376 (2008)CrossRef
11.
Canatar, A., Bordelon, B., Pehlevan, C.: Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nat. Commun. 12, 1914 (2021)CrossRef
12.
13.
Dai, E., Zhao, T., Zhu, H., Xu, J., Guo, Z., Liu, H., Tang, J., Wang, S.: A comprehensive survey on trustworthy graph neural networks: privacy, robustness, fairness, and explainability. arXiv:​2104.​05605, 2020
14.
De Jong, K.: Learning with genetic algorithms: an overview. Mach. Learn. 3, 121–138 10 (1988)
15.
Deb, K., Myburgh, C.: A population-based fast algorithm for a billion-dimensional resource allocation problem with integer variables. Eur. J. Oper. Res. 261, 460–474 (2017)MathSciNetCrossRefMATH
16.
17.
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20, 1–21 (2019)MathSciNetMATH
18.
Gaier, A., Ha, D.: Weight agnostic neural networks. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.), Advances in Neural Information Processing Systems 5364–5378 (2019)
19.
Ganon, Z., Keinan, A., Ruppin, E.: Evolutionary network minimization: adaptive implicit pruning of successful agents. In: Banzhaf, W., Ziegler, J., Christaller, T., Dittrich, P., Kim, J.T. (eds.) Advances in Artificial Life, pp. 319–327. Springer, Berlin (2003)CrossRef
20.
Gao, B., Gouk, H., Hospedales, T.M.: Searching for robustness: loss learning for noisy classification tasks. IEEE/CVF International Conference on Computer Vision, pp. 6650–6659 (2021)
21.
Gomez, F., Miikkulainen, R.: Incremental evolution of complex general behavior. Adapt. Behav. 5, 317–342 (1997)CrossRef
22.
Gomez, F., Schmidhuber, J., Miikkulainen, R. and Mitchell, M.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 937–965 (2008)
23.
Gonzalez, S., Kant, M., Miikkulainen, R.: Evolving GAN formulations for higher quality image synthesis. In: Kozma, R., Alippi, C., Choe, Y., Morabito, F.C. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn. Elsevier, New York (2023)
24.
Gonzalez, S., Landgraf, J., Miikkulainen, R.: Faster training by selecting samples using embeddings. In: Proceedings of the 2019 International Joint Conference on Neural Networks, pp. 1–7 (2019)
25.
Gonzalez, S., Miikkulainen, R.: Improved training speed, accuracy, and data utilization through loss function optimization. In: Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2020)
26.
27.
Gonzalez, S., Miikkulainen, R.: Optimizing loss functions through multivariate Taylor polynomial parameterization. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 305–313 (2021)
28.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y.: Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q., (eds.), Advances in Neural Information Processing Systems 27, pp. 2672–2680 (2014)
29.
Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9, 159–195 (2001)CrossRef
30.
Hanson, S.J., Pratt, L.Y.: Comparing biases for minimal network construction with back-propagation. In: Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 177–185. MIT Press, Cambridge (1988)
31.
32.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
33.
Hemberg, E., Veeramachaneni, K., Wanigarekara, P., Shahrzad, H., Hodjat, B., O’Reilly, U.-M.: Learning decision lists with lagged physiological time series. In: Workshop on Data Mining for Medicine and Healthcare, 14th SIAM International Conference on Data Mining, pp. 82–87 (2014)
34.
Holland, J.H.: Escaping brittleness: the possibilities of general purpose learning algorithms applied to parallel rule-based systems. In: Michalski, R.S., Carbonell, J.G., Mitchell, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, vol. 2, pp. 593–623. Morgan Kaufmann, Los Altos (1986)
35.
Hoos, H.: Programming by optimization. Commun. ACM 55, 70–80 (2012)CrossRef
36.
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)CrossRefMATH
37.
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Min, W., Yi, X.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020)MathSciNetCrossRefMATH
38.
Ijspeert, A.J.: Central pattern generators for locomotion control in animals and robots: A review. Neural Netw. 21, 642–653 (2008)CrossRef
39.
Jankowski, D., Jackowski, K.: Evolutionary algorithm for decision tree induction. In: Saeed, K., Snášel, V. (eds.) Computer Information Systems and Industrial Management, pp. 23–32. Springer, Berlin (2014)CrossRef
40.
Kashtan, N., Alon, U.: Spontaneous evolution of modularity and network motifs. Proc. Natl. Acad. Sci. 102, 13773–13778 (2005)CrossRef
41.
Langdon, W.B., Poli, R., McPhee, N.F., Koza, J.R.: Genetic programming: An introduction and tutorial, with a survey of techniques and applications. In: Fulcher, J., Jain, L.C. (eds.) Computational Intelligence: A Compendium, pp. 927–1028. Springer, Berlin (2008)CrossRef
42.
Liang, J., Meyerson, E. and Miikkulainen, R.: Evolutionary architecture search for deep multitask networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 466–473 (2018)
43.
Liang, J., Gonzalez, S., Shahrzad, H., Miikkulainen, R.: Regularized evolutionary population-based training. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 323–331 (2021)
44.
Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K. and Miikkulainen, R.: Evolutionary neural AutoML for deep learning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2019), pp. 401–409 (2019)
45.
Liang, J.Z., Miikkulainen, R.: Evolutionary bilevel optimization for complex control tasks. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2015), pp. 871–878 (2015)
46.
Light, W.: Ridge functions, sigmoidal functions and neural networks. In: Approximation Theory VII, pp. 158–201. Academic, Boston (1992)
47.
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G.G., Tan, K.C.: A survey on evolutionary neural architecture search. IEEE Trans. Neural Netw. Learn. Syst. 1–21 (2021)
48.
Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.: Evolutionary quantization of neural networks with mixed-precision. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2785–2789 (2021)
49.
Lu, Z., Deb, K., Goodman, E., Banzhaf, W., Boddeti, V.N.: Nsganetv2: evolutionary multi-objective surrogate-assisted neural architecture search. In: European Conference on Computer Vision ECCV-2020, LNCS, vol. 12346, pp. 35–51 (2020)
50.
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2794–2802 (2017)
51.
Meyerson, E., Miikkulainen, R.: Pseudo-task augmentation: from deep multitask learning to intratask sharing—and back. In: Proceedings of the 35th International Conference on Machine Learning, pp. 739–748 (2018)
52.
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. In: Morabito, C.F., Alippi, C., Choe, Y., Kozma, R. (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2nd edn., pp. 293–312. Elsevier, New York (2023)
53.
Miikkulainen, R., Meyerson, E., Qiu, X., Sinha, U., Kumar, R., Hofmann, K., Yan, Y.M., Ye, M., Yang, J., Caiazza, D. and Brown, S.M.: Evaluating medical aesthetics treatments through evolved age-estimation models. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1009–1017 (2021)
54.
Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. In: International Joint Conference on Artificial Intelligene, pp. 762–767 (1989)
55.
Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive co-evolution. Evol. Comput. 5, 373–399 (1997)CrossRef
56.
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814 (2010)
57.
Oymak, S.: Learning compact neural networks with regularization. In: International Conference on Machine Learning, pp. 3963–3972 (2018)
58.
Papavasileiou, E., Cornelis, J., Jansen, B.: A systematic literature review of the successors of “neuroevolution of augmenting topologies.” Evol. Comput. 29, 1–73 (2021)CrossRef
59.
Park, J., Sandberg, I.W.: Universal approximation using radial-basis-function networks. Neural Comput. 3, 246–257 (1991)CrossRef
60.
Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1, 81–106 (1986)CrossRef
61.
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco (1993)
62.
63.
Rawal, A., Miikkulainen, R.: Discovering gated recurrent neural network architectures. In: Iba, H., Noman, N., (eds.), Deep Neural Evolution - Deep Learning with Evolutionary Computation, pp. 233–251. Springer (2020)
64.
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4780–4789 (2019)
65.
Real, E., Liang, C., So, D., Le, Q.: AutoML-Zero: evolving machine learning algorithms from scratch. In: Daumé III, H., Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, pp. 8007–8019 (2020)
66.
Reed, R.: Pruning algorithms-a survey. IEEE Trans. Neural Netw. 4, 740–747 (1993)CrossRef
67.
Routley, N.: Visualizing the trillion-fold increase in computing power. Retrieved 11/17/2022 (2017)
68.
Schaul, T., Schmidhuber, J.: Metalearning. Scholarpedia 5, 4650 (2010)CrossRef
69.
70.
Shahrzad, H., Hodjat, B., Dolle, C., Denissov, A., Lau, S., Goodhew, D., Dyer, J., Miikkulainen, R.: Enhanced optimization with composite objectives and novelty pulsation. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds.), Genetic Programming Theory and Practice XVII, pp. 275–293. Springer, New York (2020)
71.
72.
Sharma, S., Henderson, J., Ghosh, J.: CERTIFAI: a common framework to provide explanations and analyse the fairness and robustness of black-box models. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 166–172, New York 2020. Association for Computing Machinery (2020)
73.
Shayani, H., Bentley, P.J., Tyrrell, A.M.: An fpga-based model suitable for evolution and development of spiking neural networks. In: Proceedings of the European Symposium on Artificial Neural Networks, pp. 197–202 (2008)
74.
Sinha, A., Malo, P., Xu, P. and Deb, K.: A bilevel optimization approach to automated parameter tuning. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 847-854, Vancouver, BC, Canada (2014)
75.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp. 2256–2265 (2015)
76.
Srinivasan, S., Ramakrishnan, S.: Evolutionary multi objective optimization for rule mining: a review. Artif. Intell. Rev. 36, 205–248, 10 (2011)
77.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)MathSciNetMATH
78.
Stanley, K.O.: Efficient Evolution of Neural Networks Through Complexification. PhD thesis, Department of Computer Sciences, The University of Texas at Austin (2004)
79.
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15, 185–212 (2009)CrossRef
80.
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolut. Comput. 10, 99–127 (2002)CrossRef
81.
Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)CrossRef
82.
Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O. and Clune, J.: Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning (2017). arXiv:​1712.​06567
83.
Suganuma, M., Kobayashi, M., Shirakawa, S., Nagao, T.: Evolution of deep convolutional neural networks using cartesian genetic programming. Evol. Comput. 28, 141–163 (2020)CrossRef
84.
Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Evolving deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 24, 394–407 (2020)CrossRef
85.
Sun, Y., Xue, B., Zhang, M., Yen, G.G., Lv, J.: Automatically designing cnn architectures using the genetic algorithm for image classification. IEEE Trans. Cybern. 50, 3840–3854 (2020)CrossRef
86.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)
87.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)
88.
Urbanowicz, R., Moore, J.: Learning classifier systems: a complete introduction, review, and roadmap. J. Artif. Evolut. Appl. 2009, 736398 (2009)
89.
Urbanowicz, R.J., Bertasius, G. and Moore, J.H.: An extended michigan-style learning classifier system for flexible supervised learning, classification, and data mining. In: International Conference on Parallel Problem Solving from Nature, pp. 211–221. Springer (2014)
90.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Guyon, I., Von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R., (eds.), Advances in Neural Information Processing Systems, vol. 30, pp. 6000–6010 (2017)
91.
Vinyals, O., Toshev, A., Bengio, S. and Erhan, D.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)
92.
Wu, X., Jia, L., Zhang, X., Chen, L., Liang, Y., Zhou, Y., Wu, C.: Neural architecture search based on cartesian genetic programming coding method (2021). arXiv:​2103.​07173
Metadaten
Titel
Evolutionary Supervised Machine Learning
verfasst von
Risto Miikkulainen
Copyright-Jahr
2024
Verlag
Springer Nature Singapore
Buch
Handbook of Evolutionary Machine Learning

Print ISBN: 978-981-9938-13-1

Electronic ISBN: 978-981-9938-14-8

Copyright-Jahr: 2024

DOI
https://doi.org/10.1007/978-981-99-3814-8_2

Premium Partner

    Bildnachweise