Top

Neural Computing and Applications

Published in:

09-03-2021 | S.I. : DICTA 2019

Efficient structured pruning based on deep feature stabilization

Authors: Sheng Xu, Hanlin Chen, Xuan Gong, Kexin Liu, Jinhu Lü, Baochang Zhang

Published in: Neural Computing and Applications | Issue 13/2021

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The application of convolutional neural networks (CNNs) in computer vision highly depends on the consumption of computation and memory resources, which affects its development on resource-limited devices. Accordingly, CNN compression has attracted increasing attention. In this paper, we propose an efficient end-to-end pruning method based on feature stabilization (EPFS), which is feasible to be implemented for structured pruning such as filter pruning and block pruning. For block pruning, we introduce a mask to scale the output of structures and the \(\ell _1\)-regularization term to sparsify the mask. For filter pruning, a novel \(\ell _2\)-regularization term is proposed to constraint the mask along with the \(\ell _1\)-regularization. Besides, we introduce the Center Loss to stabilize the deep feature and fast iterative shrinkage-thresholding algorithm (FISTA) to accelerate the convergence of mask. Extensive experiments demonstrate the superiority of our EPFS. On CIFAR-10, EPFS saves \(47.5\%\) FLOPs on VGGNet with \(1.17\%\) Top-1 accuracy increase. Furthermore, on ImageNet ILSVRC2012, EPFS reduces \(55.2\%\) FLOPs on ResNet-18 with o.nly \(1.63\%\) Top-1 accuracy decrease, which promotes the state-of-the-arts.

previous article Data augmentation for patch-based OCT chorio-retinal segmentation using generative adversarial networks

next article Fixed-Lens camera setup and calibrated image registration for multifocus multiview 3D reconstruction

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

The number of floating-point operations.

Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202MathSciNetCrossRef

Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: Advances in neural information processing systems (NIPS), pp 2148–2156

Ding X, Ding G, Guo Y, Han J (2019) Centripetal sgd for pruning very deep convolutional networks with complicated structure. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4943–4953

Dong X, Huang J, Yang Y, Yan S (2017) More is less: A more complicated network with less inference complexity. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 5840–5848

Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338CrossRef

Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 580–587

Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: International conference on artificial intelligence and statistics (AISTATS), pp 315–323

Goldstein T, Studer C, Baraniuk R (2014) A field guide to forward-backward splitting with a fasta implementation. arXiv preprint arXiv:1411.3406

Gu J, Li C, Zhang B, Han J, Cao X, Liu J, Doermann D (2019) Projection convolutional neural networks for 1-bit cnns via discrete back propagation. In: AAAI conference on artificial intelligence (AAAI) vol 33, pp 8344–8351

10.

Gu J, Zhao J, Jiang X, Zhang B, Liu J, Guo G, Ji R (2019) Bayesian optimized 1-bit cnns. In: IEEE international conference on computer vision (ICCV), pp 4909–4917

11.

Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems (NIPS), pp 1135–1143

12.

Hassibi B, Stork DG, Wolff GJ (1993) Optimal brain surgeon and general network pruning. In: IEEE international conference on neural networks (ICNN), pp 293–299

13.

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

14.

He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. In: International joint conference on artificial intelligence (IJCAI), pp 2234–2240

15.

He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4340–4349

16.

He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: IEEE International conference on computer vision (ICCV), pp 1389–1397

17.

Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al (2019) Searching for mobilenetv3. In: IEEE international conference on computer vision (ICCV), pp 1314–1324

18.

Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

19.

Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning (ICML), pp 448–456

20.

Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images. Tech. rep, Citeseer

21.

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1097–1105

22.

LeCun Y, Denker JS, Solla SA (1990) Optimal brain damage. In: Advances in neural information processing systems (NIPS), pp 598–605

23.

Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: International conference on learning representations (ICLR), pp 1–13

24.

Lin S, Ji R, Chen C, Huang F (2017) Espace: Accelerating convolutional neural networks via eliminating spatial and channel redundancy. In: AAAI conference on artificial intelligence (AAAI), pp 1424–1430

25.

Lin S, Ji R, Li Y, Wu Y, Huang F, Zhang B (2018) Accelerating convolutional networks via global & dynamic filter pruning. In: International joint conference on artificial intelligence (IJCAI), pp 2425–2432

26.

Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2790–2799

27.

Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision (ECCV), pp 740–755

28.

Luan S, Chen C, Zhang B, Han J, Liu J (2018) Gabor convolutional networks. IEEE Trans Image Process 27(9):4357–4366MathSciNetCrossRef

29.

Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression. In: IEEE international conference on computer vision (ICCV), pp 5058–5066

30.

Mathieu M, Henaff M, LeCun Y (2014) Fast training of convolutional networks through ffts. In: International conference on learning representations (ICLR), pp 1–9

31.

Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al (2019) Pytorch: An imperative style, high-performance deep learning library. In: Advances in neural information processing systems (NIPS), pp 8026–8037

32.

Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: Imagenet classification using binary convolutional neural networks. In: European conference on computer vision (ECCV), pp 525–542

33.

Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

34.

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252MathSciNetCrossRef

35.

Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4510–4520

36.

Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR), pp 1–15

37.

Singh P, Verma VK, Rai P, Namboodiri V (2020) Leveraging filter correlations for deep model compression. In: The IEEE winter conference on applications of computer vision (WACV), pp 835–844

38.

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

39.

Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol) 58(1):267–288MathSciNetMATH

40.

Wang X, Zhang B, Li C, Ji R, Han J, Cao X, Liu J (2018) Modulated convolutional networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 840–848

41.

Wang Y, Xu C, You S, Tao D, Xu C (2016) Cnnpack: Packing convolutional neural networks in the frequency domain. In: Advances in neural information processing systems (NIPS), pp 253–261

42.

Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision (ECCV), pp 499–515

43.

Xu S, Chen H, Liu K, Lii J, Zhang B (2019) Efficient block pruning based on kernel and feature stablization. In: International conference on digital image computing: techniques and applications (DICTA), pp 1–6

44.

Yu R, Li A, Chen CF, Lai JH, Morariu VI, Han X, Gao M, Lin CY, Davis LS (2018) Nisp: Pruning networks using neuron importance score propagation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 9194–9203

45.

Zhang Z, Saligrama V (2015) Rapid: Rapidly accelerated proximal gradient algorithms for convex minimization. In: International conference on acoustics, speech and signal processing (ICASSP), pp 3796–3800

46.

Zoph B, Le QV (2017) Neural architecture search with reinforcement learning. In: International conference on learning representations (ICLR), pp 1–16

Title: Efficient structured pruning based on deep feature stabilization
Authors: Sheng Xu
Hanlin Chen
Xuan Gong
Kexin Liu
Jinhu Lü
Baochang Zhang
Publication date: 09-03-2021
Publisher: Springer London
Published in: Neural Computing and Applications / Issue 13/2021
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-021-05828-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 13/2021

Unconfined compressive strength (UCS) prediction in real-time while drilling using artificial intelligence tools

Data augmentation for patch-based OCT chorio-retinal segmentation using generative adversarial networks

Memristor-based BAM circuit implementation for image associative memory and filling-in

A bio-inspired spatiotemporal contrast operator for small and low-heat-signature target detection in infrared imagery

Analysis of acceptable additive consistency and consensus of group decision making with interval-valued hesitant fuzzy preference relations

Transformer guided geometry model for flow-based unsupervised visual odometry

Premium Partner