Skip to main content
SpringerLink
Log in

Voting combinations-based ensemble of fine-tuned convolutional neural networks for food image recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Obesity is one of today’s most visible, uncared, and common public health problems worldwide. To manage weight loss, obtain calorie intake and record eating lists, the development of the diverse automatic dietary assessment applications has great importance. Recently, deep learning becomes a popular approach that provides outstanding image recognition results. In this paper, we use ResNet, GoogleNet, VGGNet, and InceptionV3 with fine-tuning based on deep learning for image-based and computer-aided food recognition task. We also apply six voting combination rules (namely, minimum probability, average of probabilities, median, maximum probability, product of probabilities, and weighted probabilities) for ensemble methods. The experimental results demonstrate that our proposed ensemble voting scheme with transfer learning gives promising results compared to the state-of-the-art methods on Food-101, UEC-FOOD100, and UEC-FOOD256 image datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Aguilar E, Bolaños M, Radeva P (2017) Food recognition using fusion of classifiers based on cnns. In: International conference on image analysis and processing. Springer, Cham, pp 213–224

  2. Arora S, Chaware G, Chinchankar D, Dixit E, Jain S (2019) Survey of different approaches used for food recognition. In: Information and communication technology for competitive strategies. Springer, Singapore, pp 551–560

  3. Attokaren DJ, Fernandes IG, Sriram A, Murthy YS, Koolagudi SG (2017) Food classification from images using convolutional neural networks. In: TENCON 2017-2017 IEEE Region 10 conference IEEE, pp 2801–2806

  4. Bossard L, Guillaumin M, Van Gool L (2014) Food-101–mining discriminative components with random forests. In: 2014 European conference on computer vision Springer Cham 8694, pp 446–461

  5. Ciocca G, Napoletano P, Schettini R (2018) CNN-based features for retrieval and classification of food images. Comput Vision Image Understand 176:70–77

    Article  Google Scholar 

  6. Fan DP, Cheng MM, Liu JJ, Gao SH, Hou Q, Borji A (2018) Salient objects in clutter: Bringing salient object detection to the foreground. In: Proceedings of the European conference on computer vision (ECCV), pp 186–202

  7. Fan DP, Wang W, Cheng MM, Shen J (2019) Shifting more attention to video salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8554–8564

  8. Gelbart MA, Snoek J, Adams RP (2014) Bayesian optimization with unknown constraints arXiv:1403.5607

  9. Goodfellow I, Bengio y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  10. Guo T, Dong J, Li H, Gao Y (2017) Simple convolutional neural network on image classification. In: 2017 IEEE 2nd international conference on big data analysis (ICBDA) IEEE , pp 721–724

  11. Hassannejad H, Matrella G, Ciampolini P, De Munari I, Mordonini M, et al. (2016) Food image recognition using very deep convolutional networks. In: Proceedings of the 2nd international workshop on multimedia assisted dietary management ACM, pp 41–49

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  13. Kawano Y, Yanai K (2014) foodcam: A real-time mobile food recognition system employing fisher vector. In: International conference on multimedia modeling. Springer, Cham, pp 369–373

  14. Kawano Y, Yanai K (2014) Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In: European conference on computer vision. Springer, Cham, pp 3–17

  15. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  16. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  17. Liu C, Cao Y, Luo Y, Chen G, Vokkarane V, et al. (2016) Deepfood: Deep learning-based food image recognition for computer-aided dietary assessment. In: International conference on smart homes and health telematics. Springer, Cham, pp 37–48

  18. Liu C, Cao Y, Luo Y, Chen G, Vokkarane V, et al (2017) A new deep learning-based food recognition system for dietary assessment on an edge computing service infrastructure. IEEE Trans Serv Comput 11(2):249–261

    Article  Google Scholar 

  19. Mandal B, Puhan NB, Verma A (2018) Deep convolutional generative adversarial network based food recognition using partially labeled data. IEEE Sens Lett 3(2):1–4. https://doi.org/10.1109/LSENS.2018.2886427

    Article  Google Scholar 

  20. Martinel N, Foresti GL, Micheloni C (2018) Wide-slice residual networks for food recognition. In: IEEE winter conference on applications of computer vision (WACV), pp 567–576

  21. Martinel N, Piciarelli C, Micheloni C (2017) An ensemble feature method for food classification. Mach Grap Vision 26(1/4):13–39

    Google Scholar 

  22. Matsuda Y, Hoashi H, Yanai K (2012) Recognition of multiple-food images by detecting candidate regions. In: IEEE international conference on multimedia and expo, pp 25–30

  23. McAllister P, Zheng H, Bond R (2018) Moorhead a combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets. Comput Bio Med 95:217–233

    Article  Google Scholar 

  24. McGuinness K (2017) Insight@DCU deep learning workshop

  25. National Heart, Lung and Blood Institute (2019) Overweight and obesity

  26. Ponti MP Jr (2011) Combining classifiers: from the creation of ensembles to the decision fusion. In: 2011 24th SIBGRAPI conference on graphics, patterns, and images tutorials IEEE, pp 1–10

  27. Sagi O, Rokach L (2018) Ensemble learning: a survey. Wiley Interdiscip Rev: Data Mining Knowl Discov 8(4):e1249. https://doi.org/10.1002/widm.1249

    Article  Google Scholar 

  28. Shahriari B, Swersky K, Wang Z, Adams RP, De Freitas N (2015) Taking the human out of the loop: A review of Bayesian optimization. Proc IEEE 104(1):148–175

    Article  Google Scholar 

  29. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition arXiv:1409.1556

  30. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, et al. (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  31. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  32. Tasci E (2019) Ensemble of fine-tuned convolutional neural networks for food image recognition. In: International conference on computer technologies and applications in food and agriculture (ICCTAFA19), pp 14–20

  33. Wang W, Shen J, Yang R, Porikli F (2017) Saliency-aware video object segmentation. IEEE Trans Pattern Anal Mach Intell 40(1):20–33

    Article  Google Scholar 

  34. World Health Organization (2016) Fact sheet on obesity and overweight

  35. Yanai K, Kawano Y (2015) Food image recognition using deep convolutional network with pre-training and fine-tuning. In: IEEE Multimedia & Expo Workshops (ICMEW); Turin, Italy, vol 2015, pp 1–5

  36. Zhang J, Fan DP, Dai Y, Anwar S, Saleh FS, Zhang T, Barnes N (2020) UC-Net: uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8582–8591

  37. Zhang C, Ma Y (2012) Ensemble machine learning: methods and applications. Springer Science & Business Media, Springer

    Book  Google Scholar 

  38. Zhang J, Yu X, Li A, Song P, Liu B, Dai Y (2020) Weakly-Supervised Salient object detection via scribble annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12546–12555

  39. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J, Cheng MM (2019) EGNEt: Edge guidance network for salient object detection. In: Proceedings of the IEEE international conference on computer vision, pp 8779–8788

Download references

Acknowledgments

This study is an extended and improved version of the International Conference on Computer Technologies and Applications in Food and Agriculture (ICCTAFA) 2019 where it is one of the selected papers in the conference. The author thanks anonymous reviewers for helpful comments and suggesting substantial improvements. Their suggestions helped improve and clarify this manuscript.

Author information

Authors and Affiliations

  1. Ege University, Computer Engineering Department, Izmir, Turkey

    Erdal Tasci

Authors
  1. Erdal Tasci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erdal Tasci.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tasci, E. Voting combinations-based ensemble of fine-tuned convolutional neural networks for food image recognition. Multimed Tools Appl 79, 30397–30418 (2020). https://doi.org/10.1007/s11042-020-09486-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09486-1

Keywords

Search

Navigation