Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
International Journal of Machine Learning and Cybernetics
Published in: International Journal of Machine Learning and Cybernetics 9/2022

08-03-2022 | Original Article

Network pruning via probing the importance of filters

Authors: Jiandong Kuang, Mingwen Shao, Ran Wang, Wangmeng Zuo, Weiping Ding

Published in: International Journal of Machine Learning and Cybernetics | Issue 9/2022

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Filter pruning is one of the most effective approaches to reduce the storage and computational cost of convolutional neural networks. How to measure the importance of each filter is the key problem for filter pruning. In this work, we propose a novel method that can evaluate the importance of each filter and gradually prunes those filters with small scores. Specifically, the importance is obtained via probing the effect of each filter on the task-related loss function by randomly pruning the original network. The smaller the effect on the task-related loss function, the lower the importance of the filter. It’s worth noting that our method is scale consistent across all layers without requiring layer-wise sensitivity analysis, which can be used to prune various networks, including ResNet and DenseNet. Extensive experiments demonstrate the outstanding performance of our method. For example, on ILSVRC-2012, our method can prune 42.74% floating point operations and 39.61% parameters of ResNet-50 with only 0.73% Top-1 accuracy loss and 0.37% Top-5 accuracy loss.
next article Unsupervised image clustering algorithm based on contrastive learning and K-nearest neighbors

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
Show more products
Literature
1.
Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2015) Semantic image segmentation with deep convolutional nets and fully connected crfs. In: Int. conf. on learning representations (ICLR)
2.
Chen P, Liu S, Zhao H, Jia J (2021) Distilling knowledge via knowledge review. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5008–5017
3.
Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proc. IEEE conf. computer vision and pattern recognition, pp 1800–1807
4.
Chu X, Zhang B, Xu R, Li J (2019) Fairnas: rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:​1907.​01845
5.
Courbariaux M, Bengio Y, David JP (2015) Binaryconnect: training deep neural networks with binary weights during propagations. Adv Neural Inf Process Syst 28:3123–3131
6.
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: Proc. IEEE conf. computer vision and pattern recognition, pp 248–255
7.
Ding X, Ding G, Guo Y, Han J, Yan C (2019) Approximated oracle filter pruning for destructive cnn width optimization. In: Int. conf. on machine learning (ICML), pp 1607–1616
8.
Dong X, Yang Y (2019) Network pruning via transformable architecture search. Adv Neural Inf Process Syst 32:760–771
9.
Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proc. IEEE conf. computer vision and pattern recognition, pp 580–587
10.
Guo Y, Yao A, Chen Y (2016) Dynamic network surgery for efficient dnns. Adv Neural Inf Process Syst 29:1387–1395
11.
Guo Z, Zhang X, Mu H, Heng W, Liu Z, Wei Y, Sun J (2020) Single path one-shot neural architecture search with uniform sampling. In: Proc. European conf. computer vision (ECCV), pp 544–560
12.
Han S, Pool J, Tran J, Dally WJ (2015) Learning both weights and connections for efficient neural networks. Adv Neural Inf Process Syst 28:1135–1143
13.
Han S, Mao H, Dally WJ (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Int. conf. on learning representations (ICLR)
14.
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. IEEE conf. computer vision and pattern recognition, pp 770–778
15.
He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: The IEEE int. conf. computer vision (ICCV), pp 1398–1406
16.
He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: Int. joint conf. artificial intelligence (IJCAI), pp 2234–2240
17.
He Y, Lin J, Liu Z, Wang H, Li LJ, Han S (2018) Amc: automl for model compression and acceleration on mobile devices. In: Proc. European conf. computer vision (ECCV), pp 784–800
18.
He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proc. IEEE conf. computer vision and pattern recognition, pp 4340–4349
19.
He Y, Ding Y, Liu P, Zhu L, Zhang H, Yang Y (2020) Learning filter pruning criteria for deep convolutional neural networks acceleration. In: Proc. IEEE conf. computer vision and pattern recognition, pp –2018
20.
21.
Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2261–2269
22.
Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y (2017) Quantized neural networks: training neural networks with low precision weights and activations. J Mach Learn Res 18(1):6869–6898MathSciNetMATH
23.
Hu H, Peng R, Tai YW, Tang CK (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:​1607.​03250
24.
Jaderberg M, Vedaldi A, Zisserman A (2014) Speeding up convolutional neural networks with low rank expansions. In: British machine vision conference (BMVC)
25.
Krizhevsky A (2009) Learning multiple layers of features from tiny images. In: Technical report
26.
Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90CrossRef
27.
Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: Int. conf. on learning representations (ICLR)
28.
Li B, Wu B, Su J, Wang G, Lin L (2020) Eagleeye: fast sub-net evaluation for efficient neural network pruning. In: Proc. European conf. computer vision (ECCV), pp 639–654
29.
Li Y, Gong R, Tan X, Yang Y, Hu P, Zhang Q, Yu F, Wang W, Gu S (2021) Brecq: pushing the limit of post-training quantization by block reconstruction. In: Int. conf. on learning representations (ICLR)
30.
Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2790–2799
31.
Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: Proc. IEEE conf. computer vision and pattern recognition, pp 1529–1538
32.
Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: The IEEE int. conf. computer vision (ICCV), pp 2755–2763
33.
Liu Z, Mu H, Zhang X, Guo Z, Yang X, Cheng KT, Sun J (2019) Metapruning: meta learning for automatic neural network channel pruning. In: The IEEE int. conf. computer vision (ICCV), pp 3296–3305
34.
Liu H, Simonyan K, Yang Y (2019) Darts: differentiable architecture search. In: Int. conf. on learning representations (ICLR)
35.
Luo JH, Wu J, Lin W (2017) Thinet: a filter level pruning method for deep neural network compression. In: The IEEE int. conf. computer vision (ICCV), pp 5068–5076
36.
Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2017) Pruning convolutional neural networks for resource efficient inference. In: Int. conf. on learning representations (ICLR)
37.
Ning X, Zhao T, Li W, Lei P, Wang Y, Yang H (2020) Dsa: more efficient budgeted pruning via differentiable sparsity allocation. In: Proc. European conf. computer vision (ECCV), pp 592–607
38.
Petsiuk V, Das A, Saenko K (2018) Rise: randomized input sampling for explanation of black-box models. In: British machine vision conference (BMVC), p 151
39.
Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149CrossRef
40.
41.
42.
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Int. conf. on learning representations (ICLR)
43.
Tang Y, Wang Y, Xu Y, Deng Y, Xu C, Tao D, Xu C (2021) Manifold regularized dynamic network pruning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5018–5028
44.
Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning structured sparsity in deep neural networks. Adv Neural Inf Process Syst 29:2074–2082
45.
Yamamoto K (2021) Learnable companding quantization for accurate low-bit neural networks. In: Proc. IEEE conf. computer vision and pattern recognition, pp 5029–5038
46.
Ye J, Lu X, Lin ZL, Wang JZ (2018) Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In: Int. conf. on learning representations (ICLR)
47.
You Z, Yan K, Ye J, Ma M, Wang P (2019) Gate decorator: global filter pruning method for accelerating deep convolutional neural networks. Adv Neural Inf Process Syst 32:2133–2144
48.
Yu J, Huang T (2019) Universally slimmable networks and improved training techniques. In: The ieee int. conf. computer vision (ICCV), pp 1803–1811
49.
Zhao C, Ni B, Zhang J, Zhao Q, Zhang W, Tian Q (2019) Variational convolutional neural network pruning. In: Proc. IEEE conf. computer vision and pattern recognition, pp 2780–2789
50.
Zhuang Z, Tan M, Zhuang B, Liu J, Guo Y, Wu Q, Huang J, Zhu J (2018) Discrimination-aware channel pruning for deep neural networks. Adv Neural Inf Process Syst 31:883–894
Metadata
Title
Network pruning via probing the importance of filters
Authors
Jiandong Kuang
Mingwen Shao
Ran Wang
Wangmeng Zuo
Weiping Ding
Publication date
08-03-2022
Publisher
Springer Berlin Heidelberg
Published in
International Journal of Machine Learning and Cybernetics / Issue 9/2022
Print ISSN: 1868-8071
Electronic ISSN: 1868-808X
DOI
https://doi.org/10.1007/s13042-022-01530-w

Other articles of this Issue 9/2022

International Journal of Machine Learning and Cybernetics 9/2022 Go to the issue

Original Article

Clustering mixed type data: a space structure-based approach

Original Article

A hierarchical reasoning graph neural network for the automatic scoring of answer transcriptions in video job interviews

Original Article

An improved multi-population whale optimization algorithm

Original Article

Restricted subgradient descend method for sparse signal learning

Original Article

Video person re-identification using key frame screening with index and feature reorganization based on inter-frame relation

Original Article

CIRAN: extracting crowd interaction with residual attention network for pedestrian trajectory prediction