Brian Y. Lim
Xinni Chng
ABSTRACT
Food logging can help users understand their food choices and encourage healthier eating habits. However, current apps still pose many usability challenges, including tedious manual text entry of food names. Recently, advances in computer vision and deep learning are enabling automatic food recognition for instant and convenient logging. However, as a nascent technology, this suffers from inaccuracy, which may lead to poor adoption or misuse. We investigated the trade-off between accuracy and convenience of automatic photo recognition in comparison to manual search logging. Specifically, we have developed a mobile app prototype that integrates both photo recognition and search logging capabilities, and conducted formative investigations on the usability and usage of automatic photo recognition in food logging in a series of studies: online requirements survey, usability lab study, and 1-week field trial in an Asian country. Participants were interested in convenient, automatic photo logging, but dominantly used manual search logging due to a lack of data coverage and accuracy. We identified reasons for poor accuracy and highlight complications in using inaccurate automatic photo logging. We further discuss opportunities for design and technology to address these challenges.
- Academy of Nutrition and Dietetics. Advertising and Sponsorship. http://www.eatrightpro.org/resources/about-us/advertising-and-sponsorship. Retreived 21 September 2016.Google Scholar
- Frank Bentley and Konrad Tollmar. 2013. The power of mobile notifications to increase wellbeing logging behavior. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1095--1098. ACM. Google ScholarDigital Library
- Vinay Bettadapura, Edison Thomaz, Aman Parnami, Gregory D. Abowd, and Irfan Essa. 2015. Leveraging context to support automated food recognition in restaurants. In IEEE Winter Conference on Applications of Computer Vision (WACV), Hawaii, USA, pages 580--587. Google ScholarDigital Library
- L. Bossard, M. Guillaumin, and L. Van Gool. Food-101: Mining discriminative components with random forests. In ECCV 2014.Google ScholarCross Ref
- Chen, J., and Ngo, C. W. (2016, October). Deep-based ingredient recognition for cooking recipe retrieval. In Proceedings of the 2016 ACM on Multimedia Conference (pp. 32--41). ACM. Google ScholarDigital Library
- Sunny Consolvo. 2012. Designing for Healthy Lifestyles: Design Considerations for Mobile Technologies to Encourage Consumer Health and Wellness. Foundations and Trends in Human--Computer Interaction, 6(3-4), 167--315. Google ScholarDigital Library
- Felicia Cordeiro, Elizabeth Bales, Erin Cherry, and James Fogarty. 2015. Rethinking the Mobile Food Journal: Exploring Opportunities for Lightweight Photo-Based Capture. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3207--3216. Google ScholarDigital Library
- Felicia Cordeiro, Daniel A. Epstein, Edison Thomaz, Elizabeth Bales, Arvind K. Jagannathan, Gregory D. Abowd, and James Fogarty. 2015. Barriers and Negative Nudges: Exploring Challenges in Food Journaling. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1159--1162. Google ScholarDigital Library
- Daniel A. Epstein, Felicia Cordeiro, James Fogarty, Gary Hsieh, and Sean A. Munson. 2016. Crumbs: Lightweight Daily Food Challenges to Promote Engagement and Mindfulness. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5632--5644. Google ScholarDigital Library
- BJ Fogg. 2009. Creating persuasive technologies: an eight-step design process. In Proceedings of the 4th International Conference on Persuasive Technology (Persuasive '09). ACM, New York, NY, USA" Article 44, 6 pages.. Google ScholarDigital Library
- FoodAI. http://foodai.org. Retrieved 21 September 2016.Google Scholar
- Robin Fox. 2003. Food and eating: an anthropological perspective. Social issues research center.Google Scholar
- Joan Gandy. 2014. Manual of Dietetic Practice. John Wiley & Sons.Google Scholar
- GlycoLeap, diabetes management app by Holmusk. https://glycoleap.com/. Retrieved 21 September 20, 2016.Google Scholar
- Health Promotion Board. Energy and Nutrition Composition of Food. Singapore. http://focos.hpb.gov.sg/eservices/ENCF/. Retrieved 21 September 2016.Google Scholar
- Helander E, Kaipainen K, Korhonen I, Wansink B. Factors related to sustained use of a free mobile app for dietary self-monitoring with photography and peer feedback: retrospective cohort study. Journal of medical Internet research. 2014;16(4):e109.Google Scholar
- Mahan, L. Kathleen, and Janice L. Raymond. Krause's food & the nutrition care process. Elsevier Health Sciences, 2016.Google Scholar
- Leswing Kif. 2015. Under Armour buys MyFitnessPal for $475M. https://gigaom.com/2015/02/04/under-armour-buys-myfitnesspal-for-475-million/ 4 February 2015. Retrieved 21 September 2016.Google Scholar
- Yoojung Kim, Sookyoung Ji, Hyunjeong Lee, Jeong-Whun Kim, Sooyoung Yoo, and Joongseek Lee. 2016. "My Doctor is Keeping an Eye on Me!" Exploring the Clinical Applicability of a Mobile Food Logger. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5620--5631. Google ScholarDigital Library
- K. Kitamura, C. de Silva, T. Yamasaki, and K. Aizawa. 2010. Image processing based approach to food balance analysis for personal food logging. In Proceedings of the 2010 IEEE International Conference on Multimedia and Expo: ICME '10, pp. 625--630.Google Scholar
- Kato-Lin, Yi-Chin, Vibhanshu Abhishek, Julie S. Downs, and Rema Padman. "Food for Thought: The Impact of m-Health Enabled Interventions on Eating Behavior." Available at SSRN 2736792 (2016).Google Scholar
- Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. Google ScholarDigital Library
- Jennifer Mankoff, Gary Hsieh, Ho Chak Hung, Sharon Lee, and Elizabeth Nitao. 2002. Using Low-Cost Sensing to Support Nutritional Awareness. In Proceedings of the 4th international conference on Ubiquitous Computing (UbiComp '02), Gaetano Borriello and Lars Erik Holmquist (Eds.). Springer-Verlag, London, UK, UK, 371--376. Google ScholarDigital Library
- Martin, C. K., Han, H., Coulon, S. M., Allen, H. R., Champagne, C. M., & Anton, S. D. (2008, 07). A novel method to remotely measure food intake of free-living individuals in real time: The remote food photography method. BJN British Journal of Nutrition, 101(03), 446.Google ScholarCross Ref
- Y. Matsuda, H. Hoashi, and K. Yanai. Recognition of Multiple-Food images by detecting candidate regions. In ICME, pages 25--30, July 2012. Google ScholarDigital Library
- Myers, A., Johnston, N., Rathod, V., Korattikara, A., Gorban, A., Silberman, N., ... Murphy, K. (2015, 12). Im2Calories: Towards an Automated Mobile Vision Food Diary. 2015 IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
- MyFitnessPal. 2014. Announcing 75 Million MyFitnessPal Users! http://blog.myfitnesspal.com/announcing-75-million-myfitnesspal-users/. 9 December 2014. Retreived 21 September 2016.Google Scholar
- Noronha, J., Hysen, E., Zhang, H., & Gajos, K. Z. (2011). Platemate. Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology - UIST '11.Google Scholar
- Nutritionix food database. http://nutritionix.com. Retrieved 21 September 2016.Google Scholar
- Mashfiqui Rabbi, Jean Costa, Fabian Okeke, Max Schachere, Mi Zhang, and Tanzeem Choudhury. 2015. An intelligent crowd-worker selection approach for reliable content labeling of food images. In Proceedings of the conference on Wireless Health (WH '15). ACM, New York, NY, USA" Article 9, 8 pages. Google ScholarDigital Library
- Rahman, T., Adams, A.T., Zhang, M., Cherry, E., Zhou, B., Peng, H., and Choudhury, T. BodyBeat: A Mobile System for Sensing Non-Speech Body Sounds. MobiSys 2014, 2--13. Google ScholarDigital Library
- Rise.us nutrition coach mobile app. https://www.rise.us/. Retrieved 21 September 2016.Google Scholar
- Settles, B. (2010). Active learning literature survey. University of Wisconsin, Madison, 52(55-66), 11.Google Scholar
- Singapore Nutrition and Dietetics Association. Accredited Dietitians. http://www.snda.org.sg/index.php?r=accreditations/accredited_dietitians. Retrieved 21 September 2016.Google Scholar
- Singapore Nutrition and Dietetics Association. Accredited Nutritionists. http://www.snda.org.sg/index.php?r=accreditations/accredited_nutritionists. Retrieved 21 September 2016.Google Scholar
- Singaporean Cuisine. Wikipedia. https://en.wikipedia.org/wiki/Singaporean_cuisine. Retrieved 21 September 2016.Google Scholar
- Edison Thomaz. 2016. Automatic Eating Detection in Real-World Settings with Commodity Sensing. Ph.D. Dissertation. School of Interactive Computing, Georgia Tech.Google Scholar
- C. C. Tsai, G. Lee, F. Raab, G. J. Norman, T. Sohn, W. Griswold, and K. Patrick, "Usability and feasability of pmeb: A mobile phone application for monitoring real time caloric balance," Mobile Networks and Applications, vol. 12, no. 2--3, pp. 173--184, 2007. Google ScholarDigital Library
- Walter C. Willett. 1998. Nutritional Epidemiology, 2nd ed. Oxford University Press.Google Scholar
- Hope S. Warshaw and Karmeen Kulkarni. 2001. Complete guide to carb counting. American Diabetes Association.Google Scholar
- World Health Organization. 2003. Diet, nutrition and the prevention of chronic diseases. WHO Technical Report Series, No. 916 (TRS 916). http://apps.who.int/iris/bitstream/10665/42665/1/WHO_TRS_916.pdf. Retrieved 21 September 2016.Google Scholar
- World Health Organization. 2015. Q&A on the carcinogenicity of the consumption of red meat and processed meat. http://www.iarc.fr/en/mediacentre/iarcnews/pdf/Monographs-Q&A_Vol114.pdf. Retrieved 21 September 2016.Google Scholar
- World Health Organization. 2016. Obesity and overweight. Fact Sheet. June 2016. http://www.who.int/mediacentre/factsheets/fs311/en/. Retrieved 21 September 2016.Google Scholar
- World Health Organization. 2016. Global report on diabetes. http://apps.who.int/iris/bitstream/10665/204871/1/9789241565257_eng.pdf?ua=1. Retrieved 21 September 2016.Google Scholar
Index Terms
- Trade-off between Automation and Accuracy in Mobile Photo Recognition Food Logging
Recommendations
Rethinking the Mobile Food Journal: Exploring Opportunities for Lightweight Photo-Based Capture
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsFood choices are among the most frequent and important health decisions in everyday life, but remain notoriously difficult to capture. This work examines opportunities for lightweight photo-based capture in mobile food journals. We first report on a ...
Recipe-oriented Food Logging for Nutritional Management
MM '22: Proceedings of the 30th ACM International Conference on MultimediaWe propose a recipe-oriented food logging method that records food by recipe, unlike the ordinary food logging method that records food by name. We also develop an application RecipeLog for this purpose. RecipeLog can create a "skeleton recipe," which ...
FoodLog: Multimedia Food Recording Platform and its Application
MADiMa '19: Proceedings of the 5th International Workshop on Multimedia Assisted Dietary ManagementMost existing food recording tools rely on text-based recording, which makes it difficult for users to continuously record and is an unintuitive approach to understanding food content. Photo based recording is advantageous because it provides image-...
Comments