Top

Neural Computing and Applications

Published in:

06-10-2022 | Original Article

Fine-grained image retrieval by combining attention mechanism and context information

Authors: Xiaoqing Li, Jinwen Ma

Published in: Neural Computing and Applications | Issue 2/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Recently, fine-grained image retrieval (FGIR) has become a hot topic in computer vision. Most of the advanced retrieval algorithms in this field mainly focus on the construction of loss function and the design of hard sample mining strategy. In this paper, we improve the performance of the FGIR algorithm from another perspective and propose an attention mechanism and context Information constraints-based image retrieval (AMCICIR) method for FGIR. It first applies an attention learning mechanism to gradually refine object location and extracts useful local features from coarse to fine. Then, it uses an improved graph convolutional network (GCN), where the adjacency matrix is dynamically adjusted with the current features and model retrieval performances during the model learning, to model the internal semantic interactions of the learned local features, so as to obtain a more discriminative and fine-grained image representation. Finally, various experiments are conducted on two fine-grained image datasets, CUB-200-2011 and Cars-196, and the experimental results show that the AMCICIR algorithm can outperform pervious state-of-the-art works remarkably.

previous article Solving TSP by using combinatorial Bees algorithm with nearest neighbor method

next article Toward reliable machine learning with Congruity: a quality measure based on formal concept analysis

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Wah C, Branson S, Welinder P, Perona P, Belongie S (2011) The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology

Krause J, Stark M, Deng J, Fei-Fei L (2013) 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops, pp 554–561

Khosla A, Jayadevaprakash N, Yao B, Li F-F (2011) Novel dataset for fine-grained image categorization: Stanford dogs. In: Proc. CVPR workshop on fine-grained visual categorization (FGVC), vol 2. Citeseer

Zhang X, Wang S, Li Z, Ma S (2017) Landmark image retrieval by jointing feature refinement and multimodal classifier learning. IEEE Trans Cybern 48(6):1682–1695CrossRef

D’Innocente A, Garg N, Zhang Y, Bazzani L, Donoser M (2021) Localized triplet loss for fine-grained fashion image retrieval. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3910–3915

Dubey SR, Roy SK, Chakraborty S, Mukherjee S, Chaudhuri BB (2020) Local bit-plane decoded convolutional neural network features for biomedical image retrieval. Neural Comput Appl 32(11):7539–7551CrossRef

Radenovi F, Tolias G, Chum O (2018) Fine-tuning CNN image retrieval with no human annotation. IEEE Trans Pattern Anal Mach Intell 41(7):1655–1668CrossRef

Kim S, Seo M, Laptev I, Cho M, Kwak S (2019) Deep metric learning beyond binary supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2288–2297

Zeng X, Liu S, Wang X, Zhang Y, Chen K, Li D (2021) Hard decorrelated centralized loss for fine-grained image retrieval. Neurocomputing 453:26–37CrossRef

10.

Wang X, Han X, Huang W, Dong D, Scott MR (2019) Multi-similarity loss with general pair weighting for deep metric learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5022–5030

11.

Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

12.

Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

13.

Wang W, Cui Y, Li G, Jiang C, Deng S (2020) A self-attention-based destruction and construction learning fine-grained image classification method for retail product recognition. Neural Comput Appl 32(18):14613–14622CrossRef

14.

Sa L, Yu C, Ma X, Zhao X, Xie T (2022) Attentive fine-grained recognition for cross-domain few-shot classification. Neural Comput Appl 34(6):4733–4746CrossRef

15.

Lin H, Song Y, Zeng Z, Wang W, Wang J (2021) Aggregating object features based on attention weights for fine-grained image retrieval. In: 2020 25th international conference on pattern recognition (ICPR). IEEE, pp 2838–2844

16.

Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision. Springer, pp 213–229

17.

Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al (2020) An image is worth \(16\times 16\) words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

18.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems, pp 6000–6010

19.

Hu T, Qi H, Huang Q, Lu Y (2019) See better before looking closer: weakly supervised data augmentation network for fine-grained visual classification. arXiv preprint arXiv:1901.09891

20.

Ranjan N, Mundada K, Phaltane K, Ahmad S (2016) A survey on techniques in NLP. Int J Comput Appl 134(8):6–9

21.

Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L (2020) Every document owns its structure: inductive text classification via graph neural networks. arXiv preprint arXiv:2004.13826

22.

Liu X, You X, Zhang X, Wu J, Lv P (2020) Tensor graph convolutional networks for text classification. In: Proceedings of the AAAI conference on artificial intelligence, pp 8409–8416

23.

Tu M, Wang G, Huang J, Tang Y, He X, Zhou B (2019) Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. arXiv preprint arXiv:1905.07374

24.

Visin F, Ciccone M, Romero A, Kastner K, Cho K, Bengio Y, Matteucci M, Courville A (2016) Reseg: a recurrent neural network-based model for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 41–48

25.

Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848CrossRef

26.

Yuan Y, Chen X, Wang J (2019) Object-contextual representations for semantic segmentation. arXiv preprint arXiv:1909.11065

27.

Zhou B, Liu X, Liu Y, Huang Y, Liò P, Wang Y (2021) Spectral transform forms scalable transformer. arXiv preprint arXiv:2111.07602

28.

Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In: Proceedings of the 30th international conference on neural information processing systems, pp 3844–3852

29.

Yan S, Xiong Y, Lin D (2018) Spatial temporal graph convolutional networks for skeleton-based action recognition. In: 32nd AAAI conference on artificial intelligence

30.

Chen Z, Li S, Yang B, Li Q, Liu H (2021) Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 1113–1122

31.

Gao J, Zhang T, Xu C (2019) I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 8303–8311

32.

Hu T, Xu J, Huang C, Qi H, Huang Q, Lu Y (2018) Weakly supervised bilinear attention network for fine-grained visual classification. arXiv preprint arXiv:1808.02152

33.

Cao G, Zhu Y, Lu X (2021) Fine-grained image retrieval via multiple part-level feature ensemble. In: 2021 IEEE international conference on multimedia and expo (ICME). IEEE, pp 1–6

34.

Ohsong H, Xiang Y, Jegelka S, Savarese S (2016) Deep metric learning via lifted structured feature embedding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4004–4012

35.

Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

36.

Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

37.

Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252MathSciNetCrossRef

38.

Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction by learning an invariant mapping. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 2. IEEE, pp 1735–1742

39.

Hu J, Lu J, Tan Y-P (2014) Discriminative deep metric learning for face verification in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1875–1882

40.

Sohn K (2016) Improved deep metric learning with multi-class n-pair loss objective. In: Advances in neural information processing systems, pp 1857–1865

41.

Movshovitz-Attias Y, Toshev A, Leung TK, Ioffe S, Singh S (2017) No fuss distance metric learning using proxies. In: Proceedings of the IEEE international conference on computer vision, pp 360–368

42.

Wu C-Y, Manmatha R, Smola AJ, Krahenbuhl P (2017) Sampling matters in deep embedding learning. In: Proceedings of the IEEE international conference on computer vision, pp 2840–2848

43.

Roth K, Brattoli B, Ommer B (2019) Mic: mining interclass characteristics for improved metric learning. In: Proceedings of the IEEE international conference on computer vision, pp 8000–8009

44.

Yuan Y, Yang K, Zhang C (2017) Hard-aware deeply cascaded embedding. In: Proceedings of the IEEE international conference on computer vision, pp 814–823

45.

Opitz M, Waltner G, Possegger H, Bischof H (2018) Deep metric learning with bier: boosting independent embeddings robustly. IEEE Trans Pattern Anal Mach Intell

46.

Kim W, Goyal B, Chawla K, Lee J, Kwon K (2018) Attention-based ensemble for deep metric learning. In: Proceedings of the European conference on computer vision, pp 736–751

47.

Ge W (2018) Deep metric learning with hierarchical triplet loss. In: Proceedings of the European conference on computer vision, pp 269–28

48.

Zheng X, Ji R, Sun X, Zhang B, Wu Y, Huang F. Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scale layer. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9291–9298

49.

Zeng X, Zhang Y, Wang X, Chen K, Li D, Yang W (2020) Fine-grained image retrieval via piecewise cross entropy loss. Image Vis Comput 93:103820CrossRef

50.

Kim S, Kim D, Cho M, Kwak S (2020) Proxy anchor loss for deep metric learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3238–3247

51.

Seidenschwarz JD, Elezi I, Leal-Taixé L (2021) Learning intra-batch connections for deep metric learning. In: International conference on machine learning. PMLR, pp 9410–9421

52.

Wei X-S, Luo J-H, Wu J, Zhou Z-H (2017) Selective convolutional descriptor aggregation for fine-grained image retrieval. IEEE Trans Image Process 26(6):2868–2881MathSciNetCrossRefMATH

53.

Zheng X, Ji R, Sun X, Wu Y, Huang F, Yang Y (2018) Centralized ranking loss with weakly supervised localization for fine-grained object retrieval. In: IJCAI, pp 1226–1233

Title: Fine-grained image retrieval by combining attention mechanism and context information
Authors: Xiaoqing Li
Jinwen Ma
Publication date: 06-10-2022
Publisher: Springer London
Published in: Neural Computing and Applications / Issue 2/2023
Print ISSN: 0941-0643
Electronic ISSN: 1433-3058
DOI: https://doi.org/10.1007/s00521-022-07873-3

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 2/2023

Hypothesis derivation and its verification by a wholly automated many-objective evolutionary optimization system

Adapting deep learning models between regional markets

Deep reinforcement learning for automated search of model parameters: photo-fenton wastewater disinfection case study

Distance-based arranging oversampling technique for imbalanced data

FT-HID: a large-scale RGB-D dataset for first- and third-person human interaction analysis

Evaluating the performance of meta-heuristic algorithms on CEC 2021 benchmark problems

Premium Partner