Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Density Estimation by Monte Carlo and Quasi-Monte Carlo Read chapter Read first chapter

2022 | OriginalPaper | Chapter

Artificial Neural Networks Generated by Low Discrepancy Sequences

Authors : Alexander Keller, Matthijs Van keirsbilck

Published in: Monte Carlo and Quasi-Monte Carlo Methods

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Artificial neural networks can be represented by paths. Generated as random walks on a dense network graph, we find that the resulting sparse networks allow for deterministic initialization and even weights with fixed sign. Such networks can be trained sparse from scratch, avoiding the expensive procedure of training a dense network and compressing it afterwards. Although sparse, weights are accessed as contiguous blocks of memory. In addition, enumerating the paths using deterministic low discrepancy sequences, for example variants of the Sobol’ sequence, amounts to connecting the layers of neural units by progressive permutations, which naturally avoids bank conflicts in parallel computer hardware. We demonstrate that the artificial neural networks generated by low discrepancy sequences can achieve an accuracy within reach of their dense counterparts at a much lower computational complexity.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Applications of Multivariate Quasi-Random Sampling with Neural Networks
Literature
1.
2.
3.
4.
Dey, S., Beerei, P., Chugg, K.: Interleaver design for deep neural networks. In: 51st Asilomar Conference on Signals, Systems, and Computers, pp. 1979–1983. IEEE (2017)
5.
Dey, S., Huang, K.-W., Beerel, P., Chugg, K.: Characterizing sparse connectivity patterns in neural networks. In: 2018 Information Theory and Applications Workshop (ITA), pp. 1–9. IEEE (2018)
6.
7.
Dey, S., Shao, Y., Chugg, K., Beerel, P.: Accelerating training of deep neural networks via sparse edge processing. In: Lintas, A., Rovetta, S., Verschure, P.F., Villa, A.E. (eds.) Artificial Neural Networks and Machine Learning—ICANN 2017, pp. 273–280. Springer International Publishing, Cham (2017)
8.
Dick, J., Pillichshammer, F.: Digital Nets and Sequences. Cambridge University Press, Discrepancy Theory and Quasi-Monte Carlo Integration (2010)
9.
Farhat, N.H., Psaltis, D., Prata, A., Paek, E.: Optical implementation of the Hopfield model. Appl. Opt. 24(10), 1469–1475 (1985)
10.
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: International Conference on Learning Representations (ICLR) (2019)
11.
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323. JMLR Workshop and Conference Proceedings (2011)
12.
13.
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
14.
15.
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
16.
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)
17.
Jayakumar, S., Pascanu, R., Rae, J., Osindero, S., Elsen, E.: Top-KAST: Top-k always sparse training. Adv. Neural Inf. Process. Syst. 33 (2020)
18.
Joe, S., Kuo, F.: Remark on algorithm 659: Implementing Sobol’s quasirandom sequence generator. ACM Trans. Math. Softw. 29(1), 49–57 (2003)
19.
20.
Keller, A.: Myths of computer graphics. In: Niederreiter, H. (ed.) Monte Carlo and Quasi-Monte Carlo Methods 2004, pp. 217–243. Springer, Berlin (2006)
21.
Keller, A.: Quasi-Monte Carlo image synthesis in a nutshell. In: Dick, J., Kuo, F., Peters, G., Sloan, I. (eds.) Monte Carlo and Quasi-Monte Carlo Methods 2012, pp. 203–238. Springer, Berlin (2013)
22.
Keller, A., Grünschloß, L.: Parallel quasi-Monte Carlo integration by partitioning low discrepancy sequences. In: Plaskota, L., Woźniakowski, H. (eds.) Monte Carlo and Quasi-Monte Carlo Methods 2010, pp. 487–498. Springer, Berlin (2012). http://​gruenschloss.​org/​parqmc/​parqmc.​pdf
23.
Kriman, S., Beliaev, S., Ginsburg, B., Huang, J., Kuchaiev, O., Lavrukhin, V., Leary, R., Li, J., Zhang, Y.: Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6124–6128 (2020). https://​arxiv.​org/​abs/​1910.​10261
24.
Kundu, S., Nazemi, M., Pedram, M., Chugg, K., Beerel, P.: Pre-defined sparsity for low-complexity convolutional neural networks (2020)
25.
26.
Kung, H., Leiserson, C.: Systolic arrays (for VLSI). SIAM Sparse Matrix Proc. 1978, 256–282 (1979)
27.
28.
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264–11272 (2019)
29.
Mordido, G., Van keirsbilck, M., Keller, A.: Instant quantization of neural networks using Monte Carlo methods. In: NeurIPS 2019 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (NeurIPS 2019 EMC\(^2\)) (2019)
30.
Mordido, G., Van keirsbilck, M., Keller, A.: Monte Carlo gradient quantization. In: CVPR 2020 Joint Workshop on Efficient Deep Learning in Computer Vision (CVPR 2020 EDLCV) (2020)
31.
Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1992)
32.
Owen, A.: Randomly permuted \((t,m,s)\)-nets and \((t,s)\)-sequences. In: Niederreiter, H., Shiue, P. (eds.) Monte Carlo and Quasi-Monte Carlo Methods in Scientific Computing, Lecture Notes in Statistics, vol. 106, pp. 299–315. Springer, Berlin (1995)
33.
34.
Rui, X., Daquan, H., Zhineng, L.: A perfect shuffle type of interpattern association optical neural network model. Guangzi Xuebao/Acta Photonica Sinica 29(1) (2000)
35.
Rumelhart, D., Hinton, G., Williams, R.: Learning representations by back-propagating errors. In: Anderson, J., Rosenfeld, E. (eds.) Neurocomputing: Foundations of Research, pp. 696–699. MIT Press, Cambridge, MA, USA (1988)
36.
Sobol’, I.: On the Distribution of points in a cube and the approximate evaluation of integrals. Zh. vychisl. Mat. mat. Fiz. 7(4), 784–802 (1967). USSR Comput. Math. Math. Phys. 86–112
37.
de Sousa, C.: An overview on weight initialization methods for feedforward neural networks. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp. 52–59. IEEE (2016)
38.
Stone, H.: Parallel processing with the perfect shuffle. IEEE Trans. Comput. 20(2), 153–161 (1971)
39.
Wächter, C.: Quasi-Monte Carlo Light Transport Simulation by Efficient Ray Tracing. Ph.D. thesis, Universität Ulm (2008)
40.
Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: Zeros, signs, and the supermask. In: NeurIPS 2019 (2019). Arxiv:​1905.​01067
41.
Metadata
Title
Artificial Neural Networks Generated by Low Discrepancy Sequences
Authors
Alexander Keller
Matthijs Van keirsbilck
Copyright Year
2022
Book
Monte Carlo and Quasi-Monte Carlo Methods

Print ISBN: 978-3-030-98318-5

Electronic ISBN: 978-3-030-98319-2

Copyright Year: 2022

DOI
https://doi.org/10.1007/978-3-030-98319-2_15

Premium Partner

    Image Credits