nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

Neural Network Training with Safe Regularization in the Null Space of Batch Activations

verfasst von : Matthias Kissel, Martin Gottwald, Klaus Diepold

Erschienen in: Artificial Neural Networks and Machine Learning – ICANN 2020

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

We propose to formulate the training of neural networks with side optimization goals, such as obtaining structured weight matrices, as lexicographic optimization problem. The lexicographic order can be maintained during training by optimizing the side-optimization goal exclusively in the null space of batch activations. We call the resulting training method Safe Regularization, because the side optimization goal can be safely integrated into the training with limited influence on the main optimization goal. Moreover, this results in a higher robustness regarding the choice of regularization hyperparameters. We validate our training method with multiple real-world regression data sets with the side-optimization goal of obtaining sparse weight matrices.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Prototype-Based Online Learning on Homogeneously Labeled Streaming Data

Nächstes Kapitel The Effect of Batch Normalization in the Symmetric Phase

Belsley, D.A., Kuh, E., Welsch, R.: Regression diagnostics: identifying influential data and sources of collinearity (1980)

Cheng, Y., Felix, X.Y., Feris, R.S., Kumar, S., Choudhary, A., Chang, S.F.: Fast neural networks with circulant projections. arXiv preprint arXiv:1502.03436 (2015)

Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

Duchi, J., Singer, Y.: Efficient online and batch learning using forward backward splitting. J. Mach. Learn. Res. 10, 2899–2934 (2009)MathSciNetMATH

Feppon, F., Allaire, G., Dapogny, C.: Null space gradient flows for constrained optimization with applications to shape optimization (2019)

Gerritsma, J., Onnink, R., Versluis, A.: Geometry, resistance and stability of the delft systematic yacht hull series. Int. Shipbuilding Prog. 28(328), 276–297 (1981)CrossRef

Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (2010)

Goggin, S.D., Gustafson, K.E., Johnson, K.M.: Accessing the null space with nonlinear multilayer neural networks. In: Science of Artificial Neural Networks (1992)

Goldstein, T., Studer, C., Baraniuk, R.: A field guide to forward-backward splitting with a FASTA implementation. arXiv preprint arXiv:1411.3406 (2014)

10.

Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)MATH

11.

Harrison Jr., D., Rubinfeld, D.L.: Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manag. 5(1), 81–102 (1978)CrossRef

12.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

13.

Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2014)

14.

Kukačka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy. arXiv preprint arXiv:1710.10686 (2017)

15.

Lopes, M.E.: Estimating unknown sparsity in compressed sensing. In: Proceedings of the 30th International Conference on Machine Learning, vol. 28 (2013)

16.

Ortigosa, I., Lopez, R., Garcia, J.: A neural networks approach to residuary resistance of sailing yachts prediction. In: Proceedings of the International Conference on Marine Engineering MARINE (2007)

17.

Potlapalli, H., Luo, R.C.: Projection learning for self-organizing neural networks. IEEE Trans. Ind. Electron. 43(4), 485–491 (1996)CrossRef

18.

Redmond, M., Baveja, A.: A data-driven software tool for enabling cooperative information sharing among police departments. Eur. J. Oper. Res. 141(3), 660–678 (2002)CrossRef

19.

Scardapane, S., Comminiello, D., Hussain, A., Uncini, A.: Group sparse regularization for deep neural networks. Neurocomputing 241, 81–89 (2017)CrossRef

20.

Shen, H.: Towards a mathematical understanding of the difficulty in learning with feedforward neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

21.

Sindhwani, V., Sainath, T., Kumar, S.: Structured transforms for small-footprint deep learning. In: Advances in Neural Information Processing Systems (2015)

22.

Wang, H., Langley, R., Kim, S., McCord-Snook, E., Wang, H.: Efficient exploration of gradient space for online learning to rank. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (2018)

23.

Weigend, A.S., Rumelhart, D.E., Huberman, B.A.: Generalization by weight-elimination with application to forecasting. In: Advances in Neural Information Processing Systems (1991)

24.

Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems (2016)

25.

Xie, D., Xiong, J., Pu, S.: All you need is beyond a good init: exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

26.

Xu, K., Cao, T., Shah, S., Maung, C., Schweitzer, H.: Cleaning the null space: a privacy mechanism for predictors. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

27.

Yeh, I.C.: Modeling of strength of high-performance concrete using artificial neural networks. Cem. Concr. Res. 28(12), 1797–1808 (1998)CrossRef

28.

Yoon, S.W., Seo, J., Moon, J.: Meta learner with linear nulling. arXiv preprint arXiv:1806.01010 (2018)

29.

Yuan, Y.: A null space algorithm for constrained optimization. In: Advances in Scientific Computing. Science Press, Beijing (2001)

30.

Zhang, C., Patras, P., Haddadi, H.: Deep learning in mobile and wireless networking: a survey. IEEE Commun. Surv. Tutor. 21(3), 2224–2287 (2019)CrossRef

Titel: Neural Network Training with Safe Regularization in the Null Space of Batch Activations
verfasst von: Matthias Kissel
Martin Gottwald
Klaus Diepold
Verlag: Springer International Publishing
Buch: Artificial Neural Networks and Machine Learning – ICANN 2020
Print ISBN: 978-3-030-61615-1

Electronic ISBN: 978-3-030-61616-8

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-61616-8_18

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"