Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Fine-Grained Channel Pruning for Deep Residual Neural Networks Read chapter Read first chapter

2020 | OriginalPaper | Chapter

Neural Network Compression via Learnable Wavelet Transforms

Authors : Moritz Wolter, Shaohui Lin, Angela Yao

Published in: Artificial Neural Networks and Machine Learning – ICANN 2020

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Wavelets are well known for data compression, yet have rarely been applied to the compression of neural networks. This paper shows how the fast wavelet transform can be used to compress linear layers in neural networks. Linear layers still occupy a significant portion of the parameters in recurrent neural networks (RNNs). Through our method, we can learn both the wavelet bases and corresponding coefficients to efficiently represent the linear layers of RNNs. Our wavelet compressed RNNs have significantly fewer parameters yet still perform competitively with the state-of-the-art on synthetic and real-world RNN benchmarks (Source code is available at https://​github.​com/​v0lta/​Wavelet-network-compression). Wavelet optimization adds basis flexibility, without large numbers of extra weights.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Detecting Uncertain BNN Outputs on FPGA Using Monte Carlo Dropout Sampling
next chapter Fast and Robust Compression of Deep Convolutional Neural Networks
Literature
1.
Courbariaux, M., Bengio, Y., David, J.-P.: BinaryConnect: training deep neural networks with binary weights during propagations. In: NIPS (2015)
2.
3.
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural network with pruning, trained quantization and Huffman coding. In 4th: International Conference on Learning Representations, ICLR 2016 (2016)
4.
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural networks. In: NIPS (2015)
5.
Lin, S., Ji, R., Li, Y., Wu, Y., Huang, F., Zhang, B.: Accelerating convolutional networks via global & dynamic filter pruning. In: IJCAI (2018)
6.
Denton, E.L., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: NIPS (2014)
7.
Lin, S., Ji, R., Guo, X., Li, X.: Towards convolutional neural networks compression via global error reconstruction. In: IJCAI (2016)
8.
Novikov, A., Podoprikhin, D., Osokin, A., Vetrov, D.P.: Tensorizing neural networks. In: NIPS (2015)
9.
Yang, Z., et al.: Deep fried convnets. In: ICCV (2015)
10.
Ailon, N., Chazelle, B.: The fast Johnson-Lindenstrauss transform and approximate nearest neighbors. SIAM J. Comput. 39(1), 302–322 (2009)
11.
Le, Q., Sarlós, T., Smola, A.: Fastfood-approximating kernel expansions in loglinear time. In: ICML, vol. 85 (2013)
12.
Strang, G., Nguyen, T.: Wavelets and Filter Banks. SIAM, Philadelphia (1996)
13.
Cheng, Y., Yu, F.X., Feris, R.S., Kumar, S., Choudhary, A., Chang, S.F.: An exploration of parameter redundancy in deep networks with circulant projections. In: ICCV (2015)
14.
15.
Tjandra, A., Sakti, S., Nakamura, S.: Compressing recurrent neural network with tensor train. In: IJCNN. IEEE (2017)
16.
Yang, Y., Krompass, D., Tresp, V.: Tensor-train recurrent neural networks for video classification. In: ICML. JMLR. org (2017)
17.
Denil, M., Shakibi, B., Dinh, L., Ranzato, M.A., De Freitas, N.: Predicting parameters in deep learning. In: NIPS, pp. 2148–2156 (2013)
18.
Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. In: Proceedings of the British Machine Vision Conference. BMVA Press (2014)
19.
Wen, W., et al.: Learning intrinsic sparse structures within long short-term memory. In: International Conference on Learning Representations (2018)
20.
Narang, S., Elsen, E., Diamos, G., Sengupta, S.: Exploring sparsity in recurrent neural networks. In: ICLR (2017)
21.
22.
Wang, Z., Lin, J., Wang, Z.: Accelerating recurrent neural networks: a memory-efficient approach. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 25(10), 2763–2775 (2017)
23.
Pan, Y., et al.: Compressing recurrent neural networks with tensor ring for action recognition. In: AAAI (2019)
24.
Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: CVPR (2018)
25.
Rustamov, R., Guibas, L.J.: Wavelets on graphs via deep learning. In: Advances in Neural Information Processing Systems (2013)
26.
Bruna, J., Zaremba, W., Szlam, A., Lecun, Y.: Spectral networks and locally connected networks on graphs. In: International Conference on Learning Representations (ICLR 2014), CBLS, April 2014 (2014)
27.
Lan-lan Chen, Yu., Zhao, J.Z., Zou, J.: Automatic detection of alertness/drowsiness from physiological signals using wavelet-based nonlinear features and machine learning. Expert Syst. Appl. 42(21), 7344–7355 (2015)CrossRef
28.
29.
Arjovsky, M., Shah, A., Bengio, Y.: Unitary evolution recurrent neural networks. In: ICML (2016)
30.
LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
31.
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
32.
Marcus, M., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: The Penn Treebank (1993)
33.
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:​1803.​01271 (2018)
Metadata
Title
Neural Network Compression via Learnable Wavelet Transforms
Authors
Moritz Wolter
Shaohui Lin
Angela Yao
Copyright Year
2020
Book
Artificial Neural Networks and Machine Learning – ICANN 2020

Print ISBN: 978-3-030-61615-1

Electronic ISBN: 978-3-030-61616-8

Copyright Year: 2020

DOI
https://doi.org/10.1007/978-3-030-61616-8_4

Premium Partner

    Image Credits