Skip to main content
Top

2020 | OriginalPaper | Chapter

Utilising Information Foraging Theory for User Interaction with Image Query Auto-Completion

Authors : Amit Kumar Jaiswal, Haiming Liu, Ingo Frommholz

Published in: Advances in Information Retrieval

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Query Auto-completion (QAC) is a prominently used feature in search engines, where user interaction with such explicit feature is facilitated by the possible automatic suggestion of queries based on a prefix typed by the user. Existing QAC models have pursued a little on user interaction and cannot capture a user’s information need (IN) context. In this work, we devise a new task of QAC applied on an image for estimating patch (one of the key components of Information Foraging Theory) probabilities for query suggestion. Our work supports query completion by extending a user query prefix (one or two characters) to a complete query utilising a foraging-based probabilistic patch selection model. We present iBERT, to fine-tune the BERT (Bidirectional Encoder Representations from Transformers) model, which leverages combined textual-image queries for a solution to image QAC by computing probabilities of a large set of image patches. The reflected patch probabilities are used for selection while being agnostic to changing information need or contextual mechanisms. Experimental results show that query auto-completion using both natural language queries and images is more effective than using only language-level queries. Also, our fine-tuned iBERT model allows to efficiently rank patches in the image.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Footnotes
3
The lowercase “i” represents image patch.
 
4
F1 score for the baseline methods shown in Table 1 were not available.
 
Literature
1.
go back to reference Azzopardi, L., Girolami, M., Van Rijsbergen, K.: Investigating the relationship between language model perplexity and IR precision-recall measures (2003) Azzopardi, L., Girolami, M., Van Rijsbergen, K.: Investigating the relationship between language model perplexity and IR precision-recall measures (2003)
3.
go back to reference Bar-Yossef, Z., Kraus, N.: Context-sensitive query auto-completion. In: Proceedings of the 20th International Conference on World Wide Web, pp. 107–116. ACM (2011) Bar-Yossef, Z., Kraus, N.: Context-sensitive query auto-completion. In: Proceedings of the 20th International Conference on World Wide Web, pp. 107–116. ACM (2011)
4.
go back to reference Cai, F., De Rijke, M., et al.: A survey of query auto completion in information retrieval. Found. Trends® Inf. Retrieval 10(4), 273–363 (2016)CrossRef Cai, F., De Rijke, M., et al.: A survey of query auto completion in information retrieval. Found. Trends® Inf. Retrieval 10(4), 273–363 (2016)CrossRef
5.
go back to reference Cao, H., et al.: Context-aware query suggestion by mining click-through and session data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 875–883. ACM (2008) Cao, H., et al.: Context-aware query suggestion by mining click-through and session data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 875–883. ACM (2008)
6.
go back to reference Chi, E.H., Pirolli, P., Chen, K., Pitkow, J.: Using information scent to model user information needs and actions and the web. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 490–497. ACM (2001) Chi, E.H., Pirolli, P., Chen, K., Pitkow, J.: Using information scent to model user information needs and actions and the web. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 490–497. ACM (2001)
7.
go back to reference Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:​1810.​04805 (2018)
8.
go back to reference Hauff, C., Murdock, V., Baeza-Yates, R.: Improved query difficulty prediction for the web. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, pp. 439–448. ACM (2008) Hauff, C., Murdock, V., Baeza-Yates, R.: Improved query difficulty prediction for the web. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, pp. 439–448. ACM (2008)
9.
go back to reference He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017) He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
10.
go back to reference Hu, R., Dollár, P., He, K., Darrell, T., Girshick, R.: Learning to segment every thing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4233–4241 (2018) Hu, R., Dollár, P., He, K., Darrell, T., Girshick, R.: Learning to segment every thing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4233–4241 (2018)
11.
go back to reference Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., Darrell, T.: Natural language object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555–4564 (2016) Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., Darrell, T.: Natural language object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555–4564 (2016)
13.
go back to reference Jaiswal, A.K., Holdack, G., Frommholz, I., Liu, H.: Quantum-like generalization of complex word embedding: a lightweight approach for textual classification. In: Proceedings of the Conference “Lernen, Wissen, Daten, Analysen”, LWDA 2018, Mannheim, Germany, 22–24 August 2018, pp. 159–168 (2018). http://ceur-ws.org/Vol-2191/paper19.pdf Jaiswal, A.K., Holdack, G., Frommholz, I., Liu, H.: Quantum-like generalization of complex word embedding: a lightweight approach for textual classification. In: Proceedings of the Conference “Lernen, Wissen, Daten, Analysen”, LWDA 2018, Mannheim, Germany, 22–24 August 2018, pp. 159–168 (2018). http://​ceur-ws.​org/​Vol-2191/​paper19.​pdf
14.
go back to reference Ji, S., Li, G., Li, C., Feng, J.: Efficient interactive fuzzy keyword search. In: Proceedings of the 18th International Conference on World Wide Web, pp. 371–380. ACM (2009) Ji, S., Li, G., Li, C., Feng, J.: Efficient interactive fuzzy keyword search. In: Proceedings of the 18th International Conference on World Wide Web, pp. 371–380. ACM (2009)
15.
go back to reference Jiang, J.Y., Ke, Y.Y., Chien, P.Y., Cheng, P.J.: Learning user reformulation behavior for query auto-completion. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 445–454. ACM (2014) Jiang, J.Y., Ke, Y.Y., Chien, P.Y., Cheng, P.J.: Learning user reformulation behavior for query auto-completion. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 445–454. ACM (2014)
16.
go back to reference Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: referring to objects in photographs of natural scenes. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787–798 (2014) Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: referring to objects in photographs of natural scenes. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787–798 (2014)
17.
go back to reference Kharitonov, E., Macdonald, C., Serdyukov, P., Ounis, I.: User model-based metrics for offline query suggestion evaluation. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 633–642. ACM (2013) Kharitonov, E., Macdonald, C., Serdyukov, P., Ounis, I.: User model-based metrics for offline query suggestion evaluation. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 633–642. ACM (2013)
18.
go back to reference Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123(1), 32–73 (2017)MathSciNetCrossRef Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123(1), 32–73 (2017)MathSciNetCrossRef
19.
go back to reference Liu, H., Mulholland, P., Song, D., Uren, V., Rüger, S.: Applying information foraging theory to understand user interaction with content-based image retrieval. In: Proceedings of the Third Symposium on Information Interaction in Context, pp. 135–144. ACM (2010) Liu, H., Mulholland, P., Song, D., Uren, V., Rüger, S.: Applying information foraging theory to understand user interaction with content-based image retrieval. In: Proceedings of the Third Symposium on Information Interaction in Context, pp. 135–144. ACM (2010)
21.
go back to reference McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: contextualized word vectors. In: Advances in Neural Information Processing Systems, pp. 6294–6305 (2017) McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: contextualized word vectors. In: Advances in Neural Information Processing Systems, pp. 6294–6305 (2017)
22.
go back to reference Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013) Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
23.
go back to reference Mitra, B.: Exploring session context using distributed representations of queries and reformulations. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3–12. ACM (2015) Mitra, B.: Exploring session context using distributed representations of queries and reformulations. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3–12. ACM (2015)
24.
go back to reference Mitra, B., Craswell, N.: Query auto-completion for rare prefixes. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 1755–1758. ACM (2015) Mitra, B., Craswell, N.: Query auto-completion for rare prefixes. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 1755–1758. ACM (2015)
25.
go back to reference Mitra, B., Rosset, C., Hawking, D., Craswell, N., Diaz, F., Yilmaz, E.: Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. arXiv preprint arXiv:1907.03693 (2019) Mitra, B., Rosset, C., Hawking, D., Craswell, N., Diaz, F., Yilmaz, E.: Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. arXiv preprint arXiv:​1907.​03693 (2019)
26.
go back to reference Park, D.H., Chiba, R.: A neural language model for query auto-completion. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189–1192. ACM (2017) Park, D.H., Chiba, R.: A neural language model for query auto-completion. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1189–1192. ACM (2017)
27.
go back to reference Pirolli, P., Card, S.: Information foraging. Psychol. Rev. 106(4), 643 (1999)CrossRef Pirolli, P., Card, S.: Information foraging. Psychol. Rev. 106(4), 643 (1999)CrossRef
28.
go back to reference Pirolli, P., Card, S.K., Van Der Wege, M.M.: Visual information foraging in a focus+ context visualization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 506–513. ACM (2001) Pirolli, P., Card, S.K., Van Der Wege, M.M.: Visual information foraging in a focus+ context visualization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 506–513. ACM (2001)
29.
go back to reference Shao, T., Chen, H., Chen, W.: Query auto-completion based on word2vec semantic similarity. In: Journal of Physics: Conference Series, vol. 1004, p. 012018. IOP Publishing (2018) Shao, T., Chen, H., Chen, W.: Query auto-completion based on word2vec semantic similarity. In: Journal of Physics: Conference Series, vol. 1004, p. 012018. IOP Publishing (2018)
31.
go back to reference Shokouhi, M.: Learning to personalize query auto-completion. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 103–112. ACM (2013) Shokouhi, M.: Learning to personalize query auto-completion. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 103–112. ACM (2013)
32.
go back to reference Sundar, S.S., Knobloch-Westerwick, S., Hastall, M.R.: News cues: information scent and cognitive heuristics. J. Am. Soc. Inform. Sci. Technol. 58(3), 366–378 (2007)CrossRef Sundar, S.S., Knobloch-Westerwick, S., Hastall, M.R.: News cues: information scent and cognitive heuristics. J. Am. Soc. Inform. Sci. Technol. 58(3), 366–378 (2007)CrossRef
33.
go back to reference Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 1017–1024 (2011) Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 1017–1024 (2011)
34.
go back to reference Vijayakumar, A.K., et al.: Diverse beam search: decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424 (2016) Vijayakumar, A.K., et al.: Diverse beam search: decoding diverse solutions from neural sequence models. arXiv preprint arXiv:​1610.​02424 (2016)
35.
go back to reference Weber, I., Castillo, C.: The demographics of web search. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 523–530. ACM (2010) Weber, I., Castillo, C.: The demographics of web search. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 523–530. ACM (2010)
36.
go back to reference White, R.: Beliefs and biases in web search. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3–12. ACM (2013) White, R.: Beliefs and biases in web search. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3–12. ACM (2013)
37.
go back to reference White, R.W., Marchionini, G.: Examining the effectiveness of real-time query expansion. Inf. Process. Manag. 43(3), 685–704 (2007)CrossRef White, R.W., Marchionini, G.: Examining the effectiveness of real-time query expansion. Inf. Process. Manag. 43(3), 685–704 (2007)CrossRef
38.
go back to reference Wittek, P., Liu, Y.H., Darányi, S., Gedeon, T., Lim, I.S.: Risk and ambiguity in information seeking: eye gaze patterns reveal contextual behavior in dealing with uncertainty. Front. Psychol. 7, 1790 (2016)CrossRef Wittek, P., Liu, Y.H., Darányi, S., Gedeon, T., Lim, I.S.: Risk and ambiguity in information seeking: eye gaze patterns reveal contextual behavior in dealing with uncertainty. Front. Psychol. 7, 1790 (2016)CrossRef
39.
go back to reference Wu, C.C., Mei, T., Hsu, W.H., Rui, Y.: Learning to personalize trending image search suggestion. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 727–736. ACM (2014) Wu, C.C., Mei, T., Hsu, W.H., Rui, Y.: Learning to personalize trending image search suggestion. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 727–736. ACM (2014)
Metadata
Title
Utilising Information Foraging Theory for User Interaction with Image Query Auto-Completion
Authors
Amit Kumar Jaiswal
Haiming Liu
Ingo Frommholz
Copyright Year
2020
DOI
https://doi.org/10.1007/978-3-030-45439-5_44