Skip to main content
SpringerLink
Log in

Hierarchical learning with backtracking algorithm based on the Visual Confusion Label Tree for large-scale image classification

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, a hierarchical learning algorithm based on the Bayesian Neural Network classifier with backtracking is proposed to support large-scale image classification, where a Visual Confusion Label Tree is established for constructing a hierarchical structure for large numbers of categories in image datasets and determining the hierarchical learning tasks automatically. Specifically, the Visual Confusion Label Tree is established based on outputs of convolution neural network models. One parent node on the Visual Confusion Label Tree contains a set of sibling coarse-grained categories, and child nodes have several sets of fine-grained categories which are partitions of categories on the parent node. The proposed Hierarchical Bayesian Neural Network with backtracking algorithm can benefit from the hierarchical structure of the Visual Confusion Label Tree. Focusing on those confusion subsets instead of the entire set of categories makes the classification ability of the tree classifier stronger. The backtracking algorithm can utilize the uncertainty information captured from the Bayesian Neural Network to make a second classification to re-correct samples that were classified incorrectly in the previous classification process. Experiments on four large-scale datasets show that our tree classifier obtains a significant improvement over the state-of-the-art tree classifier, which have demonstrated the discriminative hierarchical structure of our Visual Confusion Label Tree and the effectiveness of our Hierarchical Bayesian Neural Network with backtracking algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embeddings for fine-grained image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2927–2936 (2015)

  2. Berg, A.C., Deng, F.F.L.J., Satheesh, S.: Fast and balanced: efficient label tree learning for large scale object recognition. In: Advances in Neural Information Processing Systems, pp. 567–575 (2011)

  3. Blondel, V.D., Guillaume, J.L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech. Theory Exp. 2008(10), P10008 (2008)

    Article  Google Scholar 

  4. Chen, L., Yang, M.: Semi-supervised dictionary learning with label propagation for image classification. Comput. Vis. Med. 3(1), 83–94 (2017)

    Article  Google Scholar 

  5. Fan, J., He, X., Zhou, N., Peng, J., Jain, R.: Quantitative characterization of semantic gaps for learning complexity estimation and inference model selection. IEEE Trans. Multimed. 14(5), 1414–1428 (2012)

    Article  Google Scholar 

  6. Fan, J., Zhao, T., Kuang, Z., Zheng, Y., Zhang, J., Yu, J., Peng, J.: Hd-mtl: hierarchical deep multi-task learning for large-scale visual recognition. IEEE Trans. Image Process. 26(4), 1923–1938 (2017)

    Article  MathSciNet  Google Scholar 

  7. Gilks, W.R., Richardson, S., Spiegelhalter, D.: Markov Chain Monte Carlo in Practice. CRC Press, Boca Raton (1995)

    Book  Google Scholar 

  8. Grangier, D., Bengio, S., Weston, J.: Label embedding trees for large multi-class tasks. In: Advances in Neural Information Processing Systems, pp. 163–171 (2010)

  9. Gregory, G., Alex, H., Pietro, P.: Caltech-256 object category dataset. In: California Institute of Technology Systems (2007)

  10. Guillaumin, M., Ferrari, V.: Large-scale knowledge transfer for object localization in imagenet. In: Proceedings of the IEEE on Computer Vision and Pattern Recognition, pp. 3202–3209 (2012)

  11. Ji, R., Wen, L., Zhang, L., Du, D., Wu, Y., Zhao, C., Liu, X., Huang, F.: Attention convolutional binary neural tree for fine-grained visual categorization. arXiv preprint arXiv:1909.11378

  12. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678 (2014)

  13. Jia, D.: Hedging your bets: optimizing accuracy-specificity trade-offs in large scale visual recognition. In: Proceedings of the IEEE on Computer Vision and Pattern Recognition, pp. 3450–3457 (2012)

  14. Jin, R., Dou, Y., Wang, Y., Niu, X.: Confusion graph: detecting confusion communities in large scale image classification. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1980–1986 (2017)

  15. Keller, J.M., Gray, M.R., Givens, J.A.: A fuzzy k-nearest neighbor algorithm. IEEE Trans. Syst. Man Cybern. 4, 580–585 (1985)

    Article  Google Scholar 

  16. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  17. Krizhevsky, A.: Learning multiple layers of features from tiny images

  18. Kumar, M.P., Torr, P.H., Zisserman, A.: An invariant large margin nearest neighbour classifier. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1–8 (2007)

  19. LeCun, Y., Jackel, L., Bottou, L., Brunot, A., Cortes, C., Denker, J., Drucker, H., Guyon, I., Muller, U., Sackinger, E., et al.: Comparison of learning algorithms for handwritten digit recognition. In: International Conference on Artificial Neural Networks, vol. 60, Perth, Australia, pp. 53–60 (1995)

  20. Lei, H., Mei, K., Xin, J., Dong, P., Fan, J.: Hierarchical learning of large-margin metrics for large-scale image classification. Neurocomputing 208, 46–58 (2016)

    Article  Google Scholar 

  21. Li, L.J., Wang, C., Lim, Y., Blei, D.M., Li, F.F.: Building and using a semantivisual image hierarchy. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3336–3343 (2010)

  22. Liu, Y., Dou, Y., Jin, R., Li, R.: Visual confusion label tree for image classification. In: Proceedings of the IEEE International Conference on Multimedia and Expo , IEEE, pp. 1–6 (2018)

  23. Neal, R.M.: Bayesian Learning for Neural Networks, vol. 118. Springer Science & Business Media, Cham (2012)

    Google Scholar 

  24. Oh, S.: Top-k hierarchical classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2450–2456 (2017)

  25. Peng, J., Gao, L., Fan, J., Zhou, N.: Hierarchical learning of tree classifiers for large-scale plant species identification. IEEE Trans. Image Process. 24(11), 4172–4184 (2015)

    Article  MathSciNet  Google Scholar 

  26. Ristin, M., Gall, J., Guillaumin, M., Van Gool, L.: From categories to subcategories: large-scale image classification with partial class label refinement. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 231–239 (2015)

  27. Schmid, C.: Constructing category hierarchies for visual recognition. In: European Conference on Computer Vision, pp. 479–491 (2008)

  28. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  29. Socher, R., L, LJ., Li, K., L, FF., Deng, J., Dong, W.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE on Computer Vision and Pattern Recognition pp. 248–255 (2009)

  30. Sun, M., Huang, W., Savarese, S.: Find the best path: an efficient and accurate classifier for image hierarchies. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 265–272 (2013)

  31. Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)

    Article  Google Scholar 

  32. Szegedy, C., Liu, W., Jia, Y.: Going deeper with convolutions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1–9 (2015)

  33. Tappen, M., Shamir, O., Liu, C., Liu, B., Sadeghi, F.: Probabilistic label trees for efficient large scale image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 843–850 (2013)

  34. Verma, N., Mahajan, D., Sellamanickam, S., Nair, V.: Learning hierarchical similarity metrics. In: Proceedings of the IEEE on Computer Vision and Pattern Recognition, pp. 2280–2287 (2012)

  35. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset. In: Technical Report CNS-TR-2010-001, Caltech (2010)

  36. Wainwright, M.J., Jordan, M.I., et al.: Graphical models, exponential families, and variational inference. Found. Trends® Mach. Learn. 1(1–2), 1–305 (2008)

    MATH  Google Scholar 

  37. Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemometr. Intell. Lab. Syst. 2(1–3), 37–52 (1987)

    Article  Google Scholar 

  38. Xie, S., Yang, T., Wang, X., Lin, Y.: Hyper-class augmented and regularized deep learning for fine-grained image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2645–2654 (2015)

  39. Xing, E.P., Zhao, B., Li, F.F.: Large-scale category structure aware image categorization. In: Advances in Neural Information Processing Systems, pp. 1251–1259 (2011)

  40. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision. Springer, pp. 818–833 (2014)

  41. Zhang, W., Stella, X.Y., Teng, S.H.: Power svm: generalization with exemplar classification uncertainty. In: Proceedings of the IEEE on Computer Vision and Pattern Recognition, pp. 2144–2151 (2012)

  42. Zhang, J., Gao, X., Zheng, Y., Fan, J.: Hierarchical learning of multi-task sparse metrics for large-scale image classification. Pattern Recogn. 67, 97–109 (2017)

    Article  Google Scholar 

  43. Zhang, Y., Fan, J., Zhang, J., Gao, X.: Exploiting related and unrelated tasks for hierarchical metric learning and image classification. IEEE Trans. Image Process. 29, 883–896 (2019)

    Article  MathSciNet  Google Scholar 

  44. Zhao, J., Liu, J., Fan, D., Cao, Y., Yang, J., Cheng, M.: EGNet: edge guidance network for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8779–8788 (2019)

  45. Zhou, D., Xiao, L., Wu, M.: Hierarchical classification via orthogonal transfer. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 801–808 (2011)

Download references

Author information

Authors and Affiliations

  1. National Laboratory for Parallel and Distributed Processing, National University of Defense Technology, Hunan, 410073, Changsha, China

    Yuntao Liu, Yong Dou, Ruochun Jin, Rongchun Li & Peng Qiao

Authors
  1. Yuntao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruochun Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rongchun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peng Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuntao Liu.

Ethics declarations

Conflict of Interest

All authors declare that they have no conflict of interest.

Funding

This study was funded by the Ministry of Science and Technology of the People’s Republic of China (Grant Number 2018YFB1003400), National Natural Science Foundation of China (Grant Number 61802419) and National Natural Science Foundation of China (Grant Number 61902415).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Dou, Y., Jin, R. et al. Hierarchical learning with backtracking algorithm based on the Visual Confusion Label Tree for large-scale image classification. Vis Comput 38, 897–917 (2022). https://doi.org/10.1007/s00371-021-02058-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02058-w

Keywords

Search

Navigation