Top

The Review of Socionetwork Strategies

Published in:

15-10-2023 | Article

Application of a Stochastic Schemata Exploiter for Multi-Objective Hyper-parameter Optimization of Machine Learning

Authors: Hiroya Makino, Eisuke Kita

Published in: The Review of Socionetwork Strategies | Issue 2/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The Stochastic Schemata Exploiter (SSE), one of the Evolutionary Algorithms, is designed to find the optimal solution of a function. SSE extracts common schemata from individual sets with high fitness and generates individuals from the common schemata. For hyper-parameter optimization, the initialization method, the schema extraction method, and the new individual generation method, which are characteristic processes in SSE, are extended. In this paper, an SSE-based multi-objective optimization for AutoML is proposed. AutoML gives good results in terms of model accuracy. However, if only model accuracy is considered, the model may be too complex. Such complex models cannot always be allowed because of the long computation time. The proposed method maximizes the stacking model accuracy and minimizes the model complexity simultaneously. When compared with existing methods, SSE has interesting features such as fewer control parameters and faster convergence properties. The visualization method makes the optimization process transparent and helps users understand the process.

previous article Reciprocity is Different: Experimental Evidence from a Trust Game Between Japanese Domestic and International Students

next article Preface of Special Issue on International Workshop on AI compliance mechanism (WAICOM 2022)

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Makino, H., Feng, X., & Kita, E. (2020). Stochastic schemata exploiter-based optimization of convolutional neural network. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 4365–4371. https://doi.org/10.1109/SMC42975.2020.9283473

Makino, H., & Kita, E. (2021). Stochastic schemata exploiter-based AutoML. In: Proceedings of the 2021 International Conference on Data Mining Workshops (ICDMW), pp. 238–245.

Aizawa, A. N. (1994). Evolving SSE: A stochastic schemata exploiter. In: Proceedings of the First IEEE Conference on evolutionary computation. IEEE World Congress on Computational Intelligence, pp. 525–529. https://doi.org/10.1109/ICEC.1994.349895

Aizawa, A. N. (1996). Evolving SSE: A new population-oriented search scheme based on schemata processing. Systems and Computers in Japan, 27(2), 41–52. https://doi.org/10.1002/scj.4690270204CrossRef

Maruyama, T., & Kita, E. (2007). Extension of stochastic schemata exploiter to real-valued problem. In: Proceedings of Computer Aided Optimum Design in Engineering X, pp. 45–53.

Maruyama, T., & Kita, E. (2007). Investigation of real-valued stochastic schemata exploiter. Information Processing Society of Japan Transactions on Mathematical Modeling and its Applications, 48(SIG19(TOM19)), 10–22.

Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., Leyton-Brown, K. (2019). Auto-WEKA: Automatic model selection and hyperparameter optimization in WEKA. In: Automated machine learning: methods, systems, challenges (pp. 81–95). Springer.

Ledell, E., & Poirier, S. (2020). H2O AutoML: Scalable automatic machine learning. In: 7th ICML Workshop on Automated Machine Learning, pp. 1–16.

Maziarz, K., Tan, M., Khorlin, A., Georgiev, M., & Gesmundo, A. (2018). Evolutionary-neural hybrid agents for architecture search. arXiv preprint arXiv:1811.09828.

10.

Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710.

11.

Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

12.

Holland, J. H. (1992). Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. MIT Press.CrossRef

13.

Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2), 182–197. https://doi.org/10.1109/4235.996017CrossRef

14.

Zhang, Q., & Li, H. (2007). MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6), 712–731. https://doi.org/10.1109/TEVC.2007.892759CrossRef

15.

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(1), 281–305.

16.

Mantovani, R. G., Rossi, A. L. D., Vanschoren, J., Bischl, B., & De Carvalho, A. C. P. L. F. (2015). Effectiveness of random search in SVM hyper-parameter tuning. In: 2015 International Joint Conference on Neural Networks, pp. 1–8. https://doi.org/10.1109/IJCNN.2015.7280664

17.

Hansen, N., Muller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1), 1–18. https://doi.org/10.1162/106365603321828970CrossRef

18.

Xia, Y., Liu, C., Li, Y. Y., & Liu, N. (2017). A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring. Expert Systems with Applications, 78, 225–241. https://doi.org/10.1016/j.eswa.2017.02.017CrossRef

19.

Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. arXiv preprint arXiv:1206.2944.

20.

Feurer, M., & Hutter, F. (2019). Hyperparameter optimization. In: F. Hutter, L. Kotthoff, J. Vanschoren (eds.), Automated machine learning: Methods, systems, challenges, (pp. 3–33). Springer.

21.

Friedrichs, F., & Igel, C. (2005). Evolutionary tuning of multiple SVM parameters. Neurocomputing, 64, 107–117. https://doi.org/10.1016/j.neucom.2004.11.022CrossRef

22.

Loshchilov, I., & Hutter, F. (2016). CMA-ES for hyperparameter optimization of deep neural networks. arXiv preprint arXiv:1604.07269.

23.

Zhang, L. M. (2019). A new compensatory genetic algorithm-based method for effective compressed multi-function convolutional neural network model selection with multi-objective optimization. arXiv preprint arXiv:1906.11912, 1–13.

24.

Laredo, D., Qin, Y., Schütze, O., & Sun, J.-Q. (2019). Automatic model selection for neural networks. arXiv preprint arXiv:1905.06010, 1–31.

25.

Loni, M., Majd, A., Loni, A., Daneshtalab, M., Sjodin, M., & Troubitsyna, E. (2018). Designing compact convolutional neural network for embedded stereo vision systems. https://doi.org/10.1109/MCSoC2018.2018.00049

26.

Vargas, D. V., & Kotyan, S. (2019). Evolving robust neural architectures to defend from adversarial attacks. arXiv preprint arXiv:1906.11667.

27.

Liu, Q., Li, X., Liu, H., & Guo, Z. (2020). Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art. Applied Soft Computing. https://doi.org/10.1016/j.asoc.2020.106382CrossRef

28.

Lu, Z., Whalen, I., Boddeti, V., Dhebar, Y., Deb, K., Goodman, E., & Banzhaf, W. (2019). NSGA-Net: Neural architecture search using multi-objective genetic algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419–427.

29.

Hsu, C.-H., Chang, S.-H., Liang, J.-H., Chou, H.-P., Liu, C.-H., Chang, S.-C., Pan, J.-Y., Chen, Y.-T., Wei, W., & Juan, D.-C. (2018). MONAS: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332.

30.

Liang, J., Meyerson, E., Hodjat, B., Fink, D., Mutch, K., & Miikkulainen, R. (2019). Evolutionary neural AutoML for deep learning. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 401–409.

31.

Stanley, K. O., & Miikkulainen, R. (2002). Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2), 99–127. https://doi.org/10.1162/106365602320169811CrossRef

32.

Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., & Tan, K. C. (2021). A survey on evolutionary neural architecture search. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2021.3100554CrossRef

33.

Dong, J. D., Cheng, A. C., Juan, D. C., Wei, W., & Sun, M. (2018). PPP-Net: Platform-aware progressive search for pareto-optimal neural architectures. In: International Conference on Learning Representations (ICLR) Workshop 2018, pp. 1–4.

34.

Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In: International Conference on learning and intelligent optimization, pp. 507–523.

35.

Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67. https://doi.org/10.1080/00401706.1970.10488634CrossRef

36.

Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.xCrossRef

37.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

38.

Tibshirani, R. (2011). Regression shrinkage and selection via the Lasso: A retrospective. Journal of the Royal Statistical Society: Series B (statistical methodology), 73(3), 273–282. https://doi.org/10.1111/j.1467-9868.2011.00771.xCrossRef

39.

Feng, X., Zhao, J., & Kita, E. (2019). Genetic algorithm based optimization of deep neural network ensemble for personal identification in pedestrians behaviors. In: Proceedings of the 2019 International Conference on Data Mining Workshops (ICDMW), pp. 318–325. https://doi.org/10.1109/ICDMW.2019.00054

40.

Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural architecture search. In: F. Hutter, L. Kotthoff, J. Vanschoren (eds.), Automated machine learning: Methods, systems, challenges, (pp. 63–77). Springer.CrossRef

41.

Zoph, B., & Le, Q. V. (2017). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.

42.

Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019). Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, 33, 4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780

43.

Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, pp. 1–14.

44.

Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1646–1654.

45.

Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017). A survey of deep neural network architectures and their applications. Neurocomputing, 234, 11–26. https://doi.org/10.1016/j.neucom.2016.12.038CrossRef

46.

Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on machine learning, pp. 807–814.

47.

Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1201/9780429469275-8CrossRef

48.

Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on knowledge discovery and data mining, pp. 785–794.

49.

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T. Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 3149–3157.

50.

Blank, J., & Deb, K. (2020). Pymoo: Multi-objective optimization in python. IEEE Access, 8, 89497–89509. https://doi.org/10.1109/ACCESS.2020.2990567CrossRef

51.

Nash, W. J., Sellers, T. L., Talbot, S. R., Cawthorn, A. J., & Ford, W. B. (1994). The population biology of abalone (Haliotis species) in Tasmania. i. Blacklip abalone (H. rubra) from the north coast and islands of bass strait. Sea fisheries division, technical report 48.

52.

Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009). Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4), 547–553.CrossRef

Title: Application of a Stochastic Schemata Exploiter for Multi-Objective Hyper-parameter Optimization of Machine Learning
Authors: Hiroya Makino
Eisuke Kita
Publication date: 15-10-2023
Publisher: Springer Nature Singapore
Published in: The Review of Socionetwork Strategies / Issue 2/2023
Print ISSN: 2523-3173
Electronic ISSN: 1867-3236
DOI: https://doi.org/10.1007/s12626-023-00151-1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 2/2023

Role of Regulatory Sandboxes and MLOps for AI-Enabled Public Sector Services

Revolutionizing Cloud Security: Leveraging Quantum Computing and Key Distribution for Enhanced Protection

Review and Comparison of US, EU, and UK Regulations on Cyber Risk/Security of the Current Blockchain Technologies: Viewpoint from 2023

Reciprocity is Different: Experimental Evidence from a Trust Game Between Japanese Domestic and International Students

Obligation for AI Systems in Healthcare: Prepare for Trouble and Make it Double?

Towards a Risk-Based Continuous Auditing-Based Certification for Machine Learning

Premium Partner