Top

Neural Computing and Applications

Published in:

20-10-2023 | Original Article

Revolutionizing neural network efficiency: introducing FPAC for filter pruning via attention consistency

Authors: Suja Cherukullapurath Mana, Sudha Rajesh, Kalaiarasi Governor, Hemalatha Chandrasekaran, Kanipriya Murugesan

Published in: Neural Computing and Applications | Issue 2/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The process of feature mapping is used to represent features on a map with their corresponding properties. The features are displayed visually and their associated information is made available. Various abstraction approaches, including alignment approaches, are introduced to reduce the computational complexity. However, previous works based on standard treatments of feature maps often suffer from the negative effects of noise and background details. To solve this problem, we present a simple and effective implementation approach for Filter Pruning via Attention Consistency (FPAC), which determines a revolutionary filter pruning mechanism. Feature maps that concentrate on a single layer are inconsistent, which can affect the spatial dimension. The feature map with minimum consistency is less significant and is experimentally demonstrated. This study presents a novel layer-wise pruning technique using the Aphid Ant Mutualism (AAM) algorithm, which considers the sensitivity of various convolutional network layers to model inference and sets the optimal pruning ratio. The accuracy of the compressed model is enhanced by eliminating high redundancy through pruning correlated filters. The performance of FPAC is confirmed through the Caltech 256 image dataset. With VGG-16 on the Caltech 256 image dataset, the classification accuracy is enhanced from 93.96 to 94.03%. With ResNet-50 on the Caltech 256 image dataset, 45% FLOPs are achieved with an accuracy loss of only 0.53%.

previous article Multi-objective reward generalization: improving performance of Deep Reinforcement Learning for applications in single-asset trading

next article DVC-Net: a new dual-view context-aware network for emotion recognition in the wild

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29(9):2352–2449. https://doi.org/10.1162/neco_a_00990MathSciNetCrossRef

Khan A, Sohail A, Zahoora U, Qureshi AS (2020) A survey of the recent architectures of deep convolutional neural networks. Artif Intell Rev 53:5455–5516. https://doi.org/10.1007/s10462-020-09825-6CrossRef

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105. https://doi.org/10.1145/3065386CrossRef

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition 1–9.

Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, https://doi.org/10.48550/arXiv.1409.1556

Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition 1701–1708.

Denil M, Shakibi B, Dinh L, Ranzato MA, De Freitas N (2013) Predicting parameters in deep learning. Advances in neural information processing systems, 26.

Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27.

Lin S, Ji R, Chen C, Tao D, Luo J (2018) Holistic cnn compression via low-rank decomposition with knowledge transfer. IEEE Trans Pattern Anal Mach Intell 41(12):2889–2905. https://doi.org/10.1109/TPAMI.2018.2873305CrossRef

10.

Zhang X, Zou J, Ming X, He K, Sun J (2015) Efficient and accurate approximations of nonlinear convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and pattern Recognition 1984–1992.

11.

Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2016) Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. https://doi.org/10.48550/arXiv.1611.06440

12.

Hu H, Peng R, Tai YW, Tang CK (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250. https://doi.org/10.48550/arXiv.1607.03250.

13.

Qian X, Klabjan, D (2021) A probabilistic approach to neural network pruning. In International Conference on Machine Learning 8640–8649. PMLR.

14.

Rosenfeld JS, Frankle J, Carbin M, Shavit N (2021) On the predictability of pruning across scales. In International Conference on Machine Learning 9075–9083. PMLR.

15.

Lin S, Ji R, Yan C, Zhang B, Cao L, Ye Q, Huang F, Doermann D (2019) Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2790–2799.

16.

He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks[C]//Proceedings of the IEEE international conference on computer vision. 1389–1397.

17.

Chen W, Wilson J, Tyree S, Weinberger K, Chen Y (2015, June) Compressing neural networks with the hashing trick. In: International conference on machine learning 2285–2294. PMLR.

18.

Wu J, Leng C, Wang Y, Hu Q, Cheng J (2016) Quantized convolutional neural networks for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition 4820–4828.

19.

Liu W, Zhou P, Zhao Z, Wang Z, Deng H, Ju Q (2020) Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178. https://doi.org/10.48550/arXiv.2004.02178

20.

Sun S, Cheng Y, Gan Z, Liu J (2019) Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355. https://doi.org/10.48550/arXiv.1908.09355

21.

Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710. https://doi.org/10.48550/arXiv.1608.08710

22.

He Y, Liu P, Wang Z, Hu Z, Yang Y (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 4340–4349.

23.

He Y, Kang G, Dong X, Fu Y, Yang Y (2018) Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866. https://doi.org/10.48550/arXiv.1808.06866.

24.

Zhuo H, Qian X, Fu Y, Yang H, Xue X (2018) Scsp: Spectral clustering filter pruning with soft self-adaption manners. arXiv preprint arXiv:1806.05320. https://doi.org/10.48550/arXiv.1806.05320.

25.

Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning method for deep neural network compression[C]//Proceedings of the IEEE international conference on computer vision. 5058–5066.

26.

Suau X, Zappella L, Palakkode V, Apostoloff N (2018) Principal filter analysis for guided network compression. arXiv preprint arXiv:1807.10585, 2.

27.

Wang D, Zhou L, Zhang X, Bai X, Zhou J (2018) Exploring linear relationship in feature map subspace for convnets compression. arXiv preprint arXiv:1803.05729. https://doi.org/10.48550/arXiv.1803.05729.

28.

Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 1529–1538.

29.

Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,. https://doi.org/10.48550/arXiv.1409.1556.

30.

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition 770–778.

31.

Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition 4700–4708.

32.

Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images (Technical Report). University of Toronto,Canada, https://www.cs.toronto.edu/ kriz/cifar.html.

33.

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115:211–252. https://doi.org/10.1007/s11263-015-0816-yMathSciNetCrossRef

34.

Li G, Ma X, Wang X, Yue H, Li J, Liu L, Feng X, Xue J (2022) Optimizing deep neural networks on intelligent edge accelerators via flexible-rate filter pruning. J Syst Architect 124:102431. https://doi.org/10.1016/j.sysarc.2022.102431CrossRef

35.

Sarvani CH, Ghorai M, Dubey SR, Basha SS (2022) HRel: Filter pruning based on High Relevance between activation maps and class labels. Neural Netw 147:186–197. https://doi.org/10.1016/j.neunet.2021.12.017CrossRef

36.

Chen Y, Wen X, Zhang Y, He Q (2022) FPC: Filter pruning via the contribution of output feature map for deep convolutional neural networks acceleration. Knowl-Based Syst 238:107876. https://doi.org/10.1016/j.knosys.2021.107876CrossRef

37.

Zhang K, Liu G, Lv M (2022) RUFP: reinitializing unimportant filters for soft pruning. Neurocomputing 483:311–321. https://doi.org/10.1016/j.neucom.2022.02.024CrossRef

38.

Wang W, Yu Z, Fu C, Cai D, He X (2021) COP: customized correlation-based Filter level pruning method for deep CNN compression. Neurocomputing 464:533–545. https://doi.org/10.1016/j.neucom.2021.08.098CrossRef

39.

Choudhary T, Mishra V, Goswami A, Sarangapani J (2021) A transfer learning with structured filter pruning approach for improved breast cancer classification on point-of-care devices. Comput Biol Med 134:104432. https://doi.org/10.1016/j.compbiomed.2021.104432CrossRef

40.

Yao K, Cao F, Leung Y, Liang J (2021) Deep neural network compression through interpretability-based filter pruning. Pattern Recogn 119:108056. https://doi.org/10.1016/j.patcog.2021.108056CrossRef

41.

Li D, Chen S, Liu X, Sun Y, Zhang L (2020) Towards Optimal Filter Pruning with Balanced Performance and Pruning Speed. In: Proceedings of the Asian Conference on Computer Vision.

42.

Eslami N, Yazdani S, Mirzaei M, Hadavandi E (2022) Aphid-Ant Mutualism: a novel nature-inspired metaheuristic algorithm for solving optimization problems. Math Comput Simul. https://doi.org/10.1016/j.matcom.2022.05.015MathSciNetCrossRef

43.

Dorigo M, Blum C (2005) Ant colony optimization theory: a survey. Theoret Comput Sci 344(2–3):243–278. https://doi.org/10.1016/j.tcs.2005.05.020MathSciNetCrossRef

44.

Meng Z, Li G, Wang X, Sait SM, Yıldız AR (2021) A comparative study of metaheuristic algorithms for reliability-based design optimization problems. Arch Comput Methods Eng 28(3):1853–1869. https://doi.org/10.1007/s11831-020-09443-z

45.

Li J (2018) Caltech 256 image dataset. Kaggle. Retrieved July 21, 2022, from https://www.kaggle.com/datasets/jessicali9530/caltech256.

46.

He Y, Ding Y, Liu P, Zhu L, Zhang H, Yang Y (2020) Learning filter pruning criteria for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2009–2018.

47.

Yang H, Liang Y, Liu W, Meng F (2023) Filter pruning via attention consistency on feature maps. Appl Sci 13(3):1964. https://doi.org/10.3390/app13031964CrossRef

48.

Shao L, Zuo H, Zhang J, Xu Z, Yao J, Wang Z, Li H (2021) Filter pruning via measuring feature map information. Sensors 21(19):6601. https://doi.org/10.3390/s21196601CrossRef

Title: Revolutionizing neural network efficiency: introducing FPAC for filter pruning via attention consistency
Authors: Suja Cherukullapurath Mana
Sudha Rajesh
Kalaiarasi Governor
Hemalatha Chandrasekaran
Kanipriya Murugesan
Publication date: 20-10-2023
Publisher: Springer London
Published in: Neural Computing and Applications / Issue 2/2024
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-023-09037-3

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 2/2024

A novel reinforcement learning-based reptile search algorithm for solving optimization problems

Capsule network-based disease classification for Vitis Vinifera leaves

Pathological brain classification using multiple kernel-based deep convolutional neural network

DVC-Net: a new dual-view context-aware network for emotion recognition in the wild

Cross-media web video event mining based on multiple semantic-paths embedding

Correction to: Fractional-order chaotic oscillator-based Aquila optimization algorithm for maximization of the chaotic with Lorentz oscillator

Premium Partner