Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Application of Image Classification for Fine-Grained Nudity Detection Read chapter Read first chapter

2019 | OriginalPaper | Chapter

Reconstruction Error Aware Pruning for Accelerating Neural Networks

Authors : Koji Kamma, Toshikazu Wada

Published in: Advances in Visual Computing

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

This paper presents a pruning method, Reconstruction Error Aware Pruning (REAP), to reduce the redundancy of convolutional neural network models for accelerating the inference. REAP is an extension of one of the state-of-the-art channel pruning methods. Our method takes 3 steps, (1) evaluating the importance of each channel based on the reconstruction error of the outputs in each convolutional layer, (2) pruning less important channels, (3) updating the remaining weights by the least squares method so as to reconstruct the outputs. By pruning with REAP, one can produce a fast and accurate model out of a large pretrained model. Besides, REAP saves us lots of time and efforts required for retraining the pruned model. As our method requires a large computational cost, we have developed an algorithm based on biorthogonal system to conduct the computation efficiently. In the experiments, we show that REAP can conduct pruning with smaller sacrifice of the model performances than several existing state-of-the-art methods such as CP [9], ThiNet [17], DCP [25], and so on.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter DomainSiam: Domain-Aware Siamese Network for Visual Object Tracking
next chapter Bioinspired Simulation of Knotting Hagfish
Literature
1.
Aghasi, A., Abdi, A., Nguyen, N., Romberg, J.: Net-Trim: convex pruning of deep neural networks with performance guarantee. In: Advances in Neural Information Processing Systems, vol. 30, pp. 3177–3186. Curran Associates Inc. (2017)
2.
akamaster. Proper implementation of resnet-s for cifar10/100 in pytorch that matches description of the original paper (2019)
3.
Courbariaux, M., Bengio, Y., David, J.-P.: BinaryConnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, vol. 28, pp. 3123–3131. Curran Associates Inc. (2015)
4.
Deng, J., Dong, W., Socher, R., Li, J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)
5.
Dong, X., Chen, S., Pan, S.: Learning to prune deep neural networks via layer-wise optimal brain surgeon. In: Advances in Neural Information Processing Systems, vol. 30, pp. 4857–4867. Curran Associates Inc. (2017)
6.
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2016)
7.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778 (2016)
8.
He, T., Fan, Y., Qian, Y., Tan, T., Yu, K.: Reshaping deep neural network for fast decoding by node-pruning, pp. 245–249 (2014)
9.
He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of International Conference on Computer Vision (2017)
10.
11.
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks, pp. 2261–2269 (2017)
12.
Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization. In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011
13.
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (Canadian institute for advanced research)
14.
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, vol. 2, pp. 598–605. Morgan-Kaufmann (1990)
15.
Liu, B., Wang, M., Foroosh, H., Tappen, M.F., Pensky, M.: Sparse convolutional neural networks, pp. 806–814 (2015)
16.
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of International Conference on Computer Vision (2017)
17.
Luo, J.-H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of International Conference on Computer Vision (2017)
18.
Simonyan, K., Zisserman, A.: Very deep convoolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2015)
19.
Wang, H., Zhang, Q., Wang, Y., Hu, H.: Structured probabilistic pruning for convolutional neural network acceleration. In: Proceedings of British Machine Vision Conference (2018)
20.
Xie, G., Wang, J., Zhang, T., Lai, J., Hong, R., Qi, G.-J.: Interleaved structured sparse convolutional neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)
21.
Xue, J., Li, J., Gong, Y.: Restructuring of deep neural network acoustic models with singular value decomposition. In: INTERSPEECH (2013)
22.
Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of Computer Vision and Pattern Recognition (2018)
23.
Yu, X., Liu, T., Wang, X., Tao, D.: On compressing deep models by low rank and sparse decomposition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
24.
Zhou, A., Yao, A., Wang, K., Chen, Y.: Explicit loss-error-aware quantization for low-bit deep neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)
25.
Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Proceedings of Advances in Neural Information Processing Systems (2018)
26.
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cuDNN: efficient Primitives for Deep Learning. Technical report (2011)
27.
Zhao, Q., et al.: M2Det: a single-shot object detector based on multi-level feature pyramid network. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI) (2019)
Metadata
Title
Reconstruction Error Aware Pruning for Accelerating Neural Networks
Authors
Koji Kamma
Toshikazu Wada
Copyright Year
2019
Book
Advances in Visual Computing

Print ISBN: 978-3-030-33719-3

Electronic ISBN: 978-3-030-33720-9

Copyright Year: 2019

DOI
https://doi.org/10.1007/978-3-030-33720-9_5

Premium Partner

    Image Credits