Top

Published in:

2021 | OriginalPaper | Chapter

Continuous Ant-Based Neural Topology Search

Authors : AbdElRahman ElSaid, Joshua Karns, Zimeng Lyu, Alexander G. Ororbia, Travis Desell

Published in: Applications of Evolutionary Computation

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This work introduces a novel, nature-inspired neural architecture search (NAS) algorithm based on ant colony optimization, Continuous Ant-based Neural Topology Search (CANTS), which utilizes synthetic ants that move over a continuous search space based on the density and distribution of pheromones, strongly inspired by how ants move in the real world. The paths taken by the ant agents through the search space are utilized to construct artificial neural networks (ANNs). This continuous search space allows CANTS to automate the design of ANNs of any size, removing a key limitation inherent to many current NAS algorithms that must operate within structures of a size predetermined by the user. CANTS employs a distributed asynchronous strategy which allows it to scale to large-scale high performance computing resources, works with a variety of recurrent memory cell structures, and uses of a communal weight sharing strategy to reduce training time. The proposed procedure is evaluated on three real-world, time series prediction problems in the field of power systems and compared to two state-of-the-art algorithms. Results show that CANTS is able to provide improved or competitive results on all of these problems while also being easier to use, requiring half the number of user-specified hyper-parameters.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter EDM-DRL: Toward Stable Reinforcement Learning Through Ensembled Directed Mutation

next chapter Playing with Dynamic Systems - Battling Swarms in Virtual Reality

https://github.com/travisdesell/exact/tree/master/datasets/.

https://opendata-renewables.engie.com.

Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)

Erkaymaz, O., Özer, M., Yumuşak, N.: Impact of small-world topology on the performance of a feed-forward artificial neural network based on 2 different real-life problems. Turkish J. Electr. Eng. Comput. Sci. 22(3), 708–718 (2014)CrossRef

Barna, G., Kaski, K.: Choosing optimal network structure. In: International Neural Network Conference, pp. 890–893. Springer, Dordrecht (1990). https://doi.org/10.1007/978-94-009-0643-3_122

Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 (2018)

Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)

Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018)

Xie, S., Zheng, H., Liu, C., Lin, L.: Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 (2018)

Luo, R., Tian, F., Qin, T., Chen, E., Liu, T.Y.: Neural architecture optimization. In: Advances in Neural Information Processing Systems, pp. 7816–7827 (2018)

Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nature Mach. Intell. 1(1), 24–35 (2019)CrossRef

10.

Darwish, A., Hassanien, A.E., Das, S.: A survey of swarm and evolutionary computing approaches for deep learning. Artif. Intell. Rev. 53(3), 1767–1812 (2019). https://doi.org/10.1007/s10462-019-09719-2CrossRef

11.

Horng, M.H.: Fine-tuning parameters of deep belief networks using artificial bee colony algorithm. DEStech Transactions on Computer Science and Engineering (2017)

12.

Yang, X.S.: A new metaheuristic bat-inspired algorithm. In: González, J.R., Pelta, D.A., Cruz, C., Terrazas, G., Krasnogor, N. (eds.) Nature Inspired Cooperative Strategies for Optimization (NICSO 2010). Studies in Computational Intelligence, vol. 284, pp. 65–74. Springer, Berlin (2010) https://doi.org/10.1007/978-3-642-12538-6_6

13.

Yang, X.S.: Nature-Inspired Metaheuristic Algorithms. Luniver Press, London (2010)

14.

Leke, C., Ndjiongue, A.R., Twala, B., Marwala, T.: A deep learning-cuckoo search method for missing data estimation in high-dimensional datasets. In: Tan, Y., Takagi, H., Shi, Y. (eds.) ICSI 2017. LNCS, vol. 10385, pp. 561–572. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61824-1_61CrossRef

15.

Dorigo, M., Maniezzo, V., Colorni, A.: Ant system: optimization by a colony of cooperating agents. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 26(1), 29–41 (1996)

16.

Desell, T., Clachar, S., Higgins, J., Wild, B.: Evolving deep recurrent neural networks using ant colony optimization. In: Ochoa, G., Chicano, F. (eds.) EvoCOP 2015. LNCS, vol. 9026, pp. 86–98. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16468-7_8CrossRef

17.

Mavrovouniotis, M., Yang, S.: Evolving neural networks using ant colony optimization with pheromone trail limits. In: 2013 13th UK Workshop on Computational Intelligence (UKCI), pp. 16–23. IEEE (2013)

18.

ElSaid, A., El Jamiy, F., Higgins, J., Wild, B., Desell, T.: Optimizing long short-term memory recurrent neural networks using ant colony optimization to predict turbine engine vibration. Appl. Soft Comput. 73, 969–991 (2018)CrossRef

19.

ElSaid, A.E.R., Ororbia, A.G., Desell, T.J.: Ant-based neural topology search (ANTS) for optimizing recurrent networks. In: Castillo, P.A., Jiménez Laredo, J.L., Fernández de Vega, F. (eds.) EvoApplications 2020. LNCS, vol. 12104, pp. 626–641. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43722-0_40CrossRef

20.

Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 (2018)

21.

Guo, Z., et al.: Single path one-shot neural architecture search with uniform sampling. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 544–560. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_32CrossRef

22.

Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine Learning, pp. 550–559 (2018)

23.

Dong, X., Yang, Y.: One-shot neural architecture search via self-evaluated template network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3681–3690 (2019)

24.

Zhao, Y., Wang, L., Tian, Y., Fonseca, R., Guo, T.: Few-shot neural architecture search. arXiv preprint arXiv:2006.06863 (2020)

25.

Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRef

26.

Miikkulainen, R., et al.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312. Elsevier (2019)

27.

Ororbia, A., ElSaid, A., Desell, T.: Investigating recurrent neural network memory structures using neuro-evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019, pp. 446–455. ACM, New York, NY, USA (2019). https://doi.org/10.1145/3321707.3321795

28.

Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. life 15(2), 185–212 (2009)CrossRef

29.

Ororbia II, A.G., Mikolov, T., Reitter, D.: Learning simpler language models with the differential state framework. Neural Computation, pp. 1–26 (2017), https://doi.org/10.1162/neco_a_01017, pMID: 28957029

30.

Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014)

31.

Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRef

32.

Zhou, G.-B., Wu, J., Zhang, C.-L., Zhou, Z.-H.: Minimal gated unit for recurrent neural networks. Int. J. Autom. Comput. 13(3), 226–234 (2016). https://doi.org/10.1007/s11633-016-1006-2CrossRef

33.

Collins, J., Sohl-Dickstein, J., Sussillo, D.: Capacity and trainability in recurrent neural networks. arXiv preprint arXiv:1611.09913 (2016)

34.

Socha, K., Dorigo, M.: Ant colony optimization for continuous domains. Eur. J. Oper. Res. 185(3), 1155–1173 (2008)MathSciNetCrossRef

35.

Kuhn, L.D.: Ant colony optimization for continuous spaces. Computer Science and Computer Engineering Undergraduate Honors Theses (35) (2002)

36.

Xiao, J., Li, L.: A hybrid ant colony optimization for continuous domains. Exp. Syst. Appl. 38(9), 11072–11077 (2011)CrossRef

37.

Gupta, H., Ghosh, B.: Transistor size optimization in digital circuits using ant colony optimization for continuous domain. Int. J. Circuit Theor. Appl. 42(6), 642–658 (2014)CrossRef

38.

Bilchev, G., Parmee, I.C.: The ant colony metaphor for searching continuous design spaces. In: Fogarty, T.C. (ed.) AISB EC 1995. LNCS, vol. 993, pp. 25–39. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60469-3_22CrossRef

39.

Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. Kdd. 96, 226–231 (1996)

40.

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

41.

Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)

42.

Mozer, M.C., Kazakov, D., Lindsey, R.V.: Discrete event, continuous time rnns. arXiv preprint arXiv:1710.04110 (2017)

43.

Rochester Institute of Technology: Research computing services (2019). https://www.rit.edu/researchcomputing/

Title: Continuous Ant-Based Neural Topology Search
Authors: AbdElRahman ElSaid
Joshua Karns
Zimeng Lyu
Alexander G. Ororbia
Travis Desell
Publisher: Springer International Publishing
Book: Applications of Evolutionary Computation
Print ISBN: 978-3-030-72698-0

Electronic ISBN: 978-3-030-72699-7

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-3-030-72699-7_19

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner