Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
MTZ-Fachkongress

Antriebe und Energiesysteme von morgen


Austausch von Automobil- und Energiewirtschaft

Am 14./15. Mai geht es in Chemnitz um die Mobilitätswende durch Elektro- und Wasserstofffahrzeuge.
Erweiterte Suche
Anmelden
nach oben
International Journal of Machine Learning and Cybernetics
Erschienen in: International Journal of Machine Learning and Cybernetics 2/2024

13.08.2023 | Original Article

AFMPM: adaptive feature map pruning method based on feature distillation

verfasst von: Yufeng Guo, Weiwei Zhang, Junhuang Wang, Ming Ji, Chenghui Zhen, Zhengzheng Guo

Erschienen in: International Journal of Machine Learning and Cybernetics | Ausgabe 2/2024

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Feature distillation is a technology that uses the middle layer feature map of the teacher network as knowledge to transfer to the students. The feature information not only reflects the image information but also covers the feature extraction ability of the teacher network. However, the existing feature distillation methods lack theoretical guidance for feature map evaluation and suffer from the mismatch of sizes between high-dimensional feature maps and low-dimensional feature maps, and poor information utilization. In this paper, we propose an Adaptive Feature Map Pruning Method (AFMPM) for feature distillation, which transforms the problem of feature map pruning into the problem of optimization so that the valid information of the feature map is retained to the maximum extent. AFMPM has achieved significant improvements in feature distillation, and the advanced and generalized nature of the method has been verified by conducting experiments on the teacher-student distillation framework and the self-distillation framework.
Vorheriger Artikel Program synthesis algorithm based on context consistency heuristic
Nächster Artikel An improved deep network-based RGB-D semantic segmentation method for indoor scenes

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Weitere Produktempfehlungen anzeigen
Literatur
1.
Wang C, Zhang S, Song S et al (2022) Learn from the past: experience ensemble knowledge distillation. arXiv preprint https://​arxiv.​org/​abs/​2202.​12488
2.
Yao J, Zhang S, Yao Y et al (2022) Edge-cloud polarization and collaboration: a comprehensive survey for AI. IEEE Trans Knowl Data Eng 35:6866
3.
Liu Z, Sun M, Zhou T et al (2018) Rethinking the value of network pruning. arXiv preprint https://​arxiv.​org/​abs/​1810.​0527
4.
Wang D, Zhang S, Di Z et al (2022) A Novel Architecture Slimming Method for Network Pruning and Knowledge Distillation. arXiv preprint https://​arxiv.​org/​abs/​2202.​10461
5.
Yang C, An Z, Cai L et al (2022) Knowledge distillation using hierarchical self-supervision augmented distribution. IEEE Trans Neural Netw Learn Syst (TNNLS) 12:1–15
6.
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint https://​arxiv.​org/​abs/​1503.​02531
7.
Romero A, Ballas N, Kahou SE et al (2015) FitNets: hints for thin deep nets. arXiv preprint https://​arxiv.​org/​abs/​1412.​6550
8.
Zagoruyko S, Komodakis N (2016) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. arXiv preprint https://​arxiv.​org/​abs/​1612.​03928
9.
Yim J, Joo D, Bae J et al (2017) A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 7130–7138
10.
Kim J, Park SU, Kwak N (2018) Paraphrasing complex network: network compression via factor transfer. Neural Inf Process Syst (NIPS) 31:2765–2774
11.
Heo B, Lee M, Yun S et al (2019) Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In: AAAI Conference on Artificial Intelligence, pp 3779–3787
12.
Song J, Chen Y, Ye J et al (2022) Spot-adaptive knowledge distillation. In: IEEE Trans Image Process, pp 3359–3370
13.
Srinivas S, Fleuret F (2018) Knowledge transfer with jacobian matching. In: International Conference on Machine Learning (PMLR), pp 4723–4731
14.
Zhang L, Song J, Gao A et al (2019) Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp 3713–3722
15.
Lin M, Ji R, Wang Y et al (2020) Hrank: Filter pruning using high-rank feature map. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 1529–1538
16.
Amik FR, Tasin AI, Ahmed S et al (2022) Dynamic Rectification Knowledge Distillation. arXiv preprint https://​arxiv.​org/​abs/​2201.​11319
17.
Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Handbook of systemic autoimmune diseases 1(4)
18.
Park W, Kim D, Lu Y et al (2019) Relational knowledge distillation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 3967–3976
19.
Tian Y, Krishnan D, Isola P (2019) Contrastive representation distillation. arXiv preprint https://​arxiv.​org/​abs/​1910.​10699
20.
Heo B, Kim J, Yun S et al (2019) A comprehensive overhaul of feature distillation. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1921–1930
21.
Huang Z, Wang N (2017) Like what you like: knowledge distill via neuron selectivity transfer. arXiv preprint https://​arxiv.​org/​abs/​1707.​01219
22.
Passalis N, Tefas A (2018) Probabilistic knowledge transfer for deep representation learning. IEEE Trans Neural Netw Learn Syst (TNNLS) 32:2030–2039MathSciNetCrossRef
23.
Ahn S, Hu SX, Damianou A et al (2019) Variational information distillation for knowledge transfer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 9163 – 917
24.
Sun D, Yao A, Zhou A et al (2019) Deeply-supervised knowledge synergy. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 6997–7006
25.
Tung F, Mori G (2019) Similarity-preserving knowledge distillation. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp 1365–1374
26.
Lee CY, Xie S, Gallagher P et al (2015) Deeply-supervised nets. Artif Intell Stat PMLR 21:562–570
27.
Peng B, Jin X, Liu J et al (2019) Correlation congruence for knowledge distillation. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp 5007–5016
28.
Chen D, Mei JP, Zhang Y et al (2021) Cross-layer distillation with semantic calibration. In: AAAI Conference on Artificial Intelligence, pp 7028–7036
29.
Zhang L, Shi Y, Shi Z et al (2020) Task-oriented feature distillation. Neural Inf Process Syst (NIPS) 33:14759–14771
30.
LeCun Y, Denker J, Solla S (1989) Optimal brain damage. Neural Inf Process Syst (NIPS) 598–605
31.
Han S, Pool J, Tran J et al (2015) Learning both weights and connections for efficient neural network. Neural Inf Process Syst (NIPS) 1:1135–1143
32.
Frankle J, Carbin M (2018) The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint https://​arxiv.​org/​abs/​1803.​03635
33.
Frankle J, Dziugaite GK, Roy D et al (2020) Linear mode connectivity and the lottery ticket hypothesis. In: International Conference on Machine Learning (PMLR), pp 3259–3269
34.
Ye J, Lu X, Lin Z et al (2018) Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint https://​arxiv.​org/​abs/​1802.​00124
35.
Zhuang T, Zhang Z, Huang Y et al (2020) Neuron-level structured pruning using polarization regularizer. Neural Inf Process Syst (NIPS) 33:9865–9877
36.
Hu H, Peng R, Tai YW et al (2016) Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint https://​arxiv.​org/​abs/​1607.​03250
37.
Luo JH, Wu J (2017) An entropy-based pruning method for cnn compression. arXiv preprint arXiv:1706.05791
38.
He Y, Liu P, Wang Z et al (2019) Filter pruning via geometric median for deep convolutional neural networks acceleration. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 4340–4349
39.
Wang X, Zheng Z, He Y et al (2023) Progressive local filter pruning for image retrieval acceleration. IEEE Trans Multimed 4:1–11
40.
Li G, Wang J, Shen HW et al (2020) Cnnpruner: pruning convolutional neural networks with visual analytics. IEEE Trans Vis Comput Gr 27:1364–1373CrossRef
41.
Wang X, Zheng Z, He Y et al (2022) Soft person reidentification network pruning via blockwise adjacent filter decaying. IEEE Trans Cybern 52:13293–13307CrossRef
Metadaten
Titel
AFMPM: adaptive feature map pruning method based on feature distillation
verfasst von
Yufeng Guo
Weiwei Zhang
Junhuang Wang
Ming Ji
Chenghui Zhen
Zhengzheng Guo
Publikationsdatum
13.08.2023
Verlag
Springer Berlin Heidelberg
Erschienen in
International Journal of Machine Learning and Cybernetics / Ausgabe 2/2024
Print ISSN: 1868-8071
Elektronische ISSN: 1868-808X
DOI
https://doi.org/10.1007/s13042-023-01926-2

Weitere Artikel der Ausgabe 2/2024

International Journal of Machine Learning and Cybernetics 2/2024 Zur Ausgabe

Original Article

FRS-SIFS: fuzzy rough set session identification and feature selection in web robot detection

Original Article

Optimal airport selection utilizing power Muirhead mean based group decision model with 2-tuple linguistic q-rung orthopair fuzzy information

Original Article

MetaCL: a semi-supervised meta learning architecture via contrastive learning

Original Article

Decision space information driven algorithm for dynamic multiobjective optimization with a changing number of objectives

Original Article

Multi-scale adaptive networks for efficient inference

Original Article

Three-way conflict analysis with similarity degree on an issue set

Neuer Inhalt

    Bildnachweise