nach oben

Discover Computing

Erschienen in:

10.03.2022 | Special Issue on ECIR 2021

CoSearcher: studying the effectiveness of conversational search refinement and clarification through user simulation

verfasst von: Alexandre Salle, Shervin Malmasi, Oleg Rokhlenko, Eugene Agichtein

Erschienen in: Discover Computing | Ausgabe 2/2022

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

A key application of conversational search is refining a user’s search intent by asking a series of clarification questions, aiming to improve the relevance of search results. Training and evaluating such conversational systems currently requires human participation, making it infeasible to examine a wide range of user behaviors. To support robust training/evaluation of such systems, we propose a simulation framework called CoSearcher Information about code/resources available at https://github.com/amzn/cosearcher that includes a parameterized user simulator controlling key behavioral factors like cooperativeness and patience. To evaluate our approach, we use both a standard conversational query clarification benchmark and develop an extended dataset using query suggestions from a popular Web search engine as a source of additional refinement candidates. Using these datasets, we investigate the impact of a variety of conditions on search refinement and clarification effectiveness over a wide range of user behaviors, semantic policies, and dynamic facet generation. Our results quantify the effects of user behavior variation, and identify conditions required for conversational search refinement and clarification to be effective. This paper is an extended version of our previous work, and includes new experimental results for comparing semantic similarity ranking strategies for facets, using enhanced representations of facets, learning from negative user responses, among other new results and more detailed experimental descriptions.

Vorheriger Artikel CEQE to SQET: A study of contextualized embeddings for query expansion

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

An alternative would be to consider all facets, not just from a single topic but from all topics, as candidates. In our use case, this does not make sense since our application is one where the user provides a topic. We assume that the topic is accurate and do not consider facets beyond it. In future work we could consider other scenarios, e.g. where the user provides no initial topic, there is uncertainty about the topic, or we begin with a very broad topic which may share facets with other topics.

https://azure.microsoft.com/en-us/services/cognitive-services/autosuggest/

https://github.com/alexandres/lexvec

Implementation distributed by authors at https://github.com/aliannejadi/qulac

Note that since they do not perform explicit intent refinement, they submit the entire dialogue context as a query to the IR system, whereas we submit only the topic and the refined facet.

The authors do not specify what short means, but for discussion we presume it to mean single word: a simple “Yes” or “No”.

Agirre, E., Banea, C., Cardie, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Wiebe, J. (2014, August). SemEval-2014 task 10: Multilingual semantic textual similarity. In: Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) (pp. 81-91). Dublin, Ireland: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/S14-2010 10.3115/v1/S14-2010

Agirre, E., Banea, C., Cardie, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Wiebe, J. (2015, June). SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015) (pp. 252–263). Denver, Colorado: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/S15-2045 10.18653/v1/S15-2045

Agirre, E., Banea, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Mihalcea, R., Wiebe, J. (2016, June). SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016) (pp. 497–511). San Diego, California: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/S16-1081 10.18653/v1/S16-1081

Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W. (2013, June). *SEM 2013 shared task: Semantic textual similarity. Second joint conference on lexical and computational semantics (*SEM), volume 1: Proceedings of the main conference and the shared task: Semantic textual similarity (pp. 32–43). Atlanta, Georgia, USA: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/S13-1004

Agirre, E., Diab, M., Cer, D., Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. Proceedings of the first joint conference on lexical and computational semantics-volume 1: Proceedings of the main conference and the shared task, and volume 2: Proceedings of the sixth international workshop on semantic evaluation (pp. 385–393).

Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., Burtsev, M. (2020). ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ). arXiv preprint. arXiv:2009.11352.

Aliannejadi, M., Zamani, H., Crestani, F., Croft, W.B. (2019). Asking clarifying questions in open-domain information-seeking conversations. Proceedings of the 42nd international acm sigir conference on research and development in information retrieval (pp. 475–484).

Azzopardi, L., & Zuccon, G. (2016). An analysis of the cost and benefit of search interactions. Proceedings of the 2016 acm international conference on the theory of information retrieval (pp. 59–68).

Belkin, N. J., Cool, C., Stein, A., & Thiel, U. (1995). Cases, scripts, and informationseeking strategies: On the design of interactive information retrieval systems. Expert systems with applications, 9(3), 379–395.CrossRef

Bojanowski, P., Grave, E., Joulin, A., Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 135–146. Retrieved from https://www.aclweb.org/anthology/Q17-1010https://doi.org/10.1162/tacl_a_00051

Bordes, A., Boureau, Y.-L., Weston, J. (2016). Learning end-to-end goal-oriented dialog. arXiv preprintarXiv:1605.07683.

Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L. (2017, August). SemEval2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017) (pp. 1–14). Vancouver, Canada: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/S17-2001https://doi.org/10.18653/v1/S17-2001

Chandramohan, S., Geist, M., Lefevre, F., Pietquin, O. (2011). User simulation in dialogue systems using inverse reinforcement learning. Twelfth annual conference of the international speech communication association.

Clarke, C.L., Craswell, N., Soboroff, I. (2009). Overview of the trec 2009 web track (Tech. Rep.). WATERLOO UNIV (ONTARIO).

Clarke, C.L., Craswell, N., Voorhees, E.M. (2012). Overview of the trec 2012 web track (Tech. Rep.). NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD.

Crescenzi, A., Kelly, D., Azzopardi, L. (2016). Impacts of time constraints and system delays on user experience. Proceedings of the 2016 acm on conference on human information interaction and retrieval (pp. 141-150).

Croft, W. B., & Thompson, R. H. (1987). I3r: A new approach to the design of document retrieval systems. Journal of the American Society for Information Science, 38(6), 389–404.CrossRef

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171–4186).

El Asri, L., He, J., & Suleman, K. (2016). A sequence-to-sequence model for user simulation in spoken dialogue systems. Interspeech, 2016, 1151–1155.

Fagan, J. C. (2010). Usability studies of faceted browsing: A literature review. Information Technology and Libraries, 29(2), 58–66.CrossRef

Hashemi, H., Zamani, H., & Croft, W.B. (2020). Guided transformer: Leveraging multiple external sources for representation learning in conversational search. In Proceedings of the 43rd international acm sigir conference on research and development in information retrieval (p. 1131-1140). New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3397271.3401061

Hearst, M. (2006). Design recommendations for hierarchical faceted search interfaces. Acm sigir workshop on faceted search (pp. 1–5).

Hearst, M., Elliott, A., English, J., Sinha, R., Swearingen, K., & Yee, K.-P. (2002). Finding the flow in web site search. Communications of the ACM, 45(9), 42–49.CrossRef

Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422–446.CrossRef

Keizer, S., Gasic, M., Jurcicek, F., Mairesse, F., Thomson, B., Yu, K., Young, S. (2010). Parameter estimation for agenda-based user simulation. In Proceedings of the sigdial 2010 conference (pp. 116–123).

Kiesel, J., Bahrami, A., Stein, B., Anand, A., Hagen, M. (2018). Toward voice query clarification. The 41st international acm sigir conference on research & development in information retrieval (pp. 1257–1260).

Kotov, A., & Zhai, C. (2010). Towards natural question guided search. In Proceedings of the 19th international conference on world wide web (pp. 541–550).

Krasakis, A.M., Aliannejadi, M., Voskarides, N., Kanoulas, E. (2020). Analysing the effect of clarifying questions on document ranking in conversational search. Proceedings of the 2020 acm sigir on international conference on theory of information retrieval (p. 129–132). New York, NY, USA: Association for Computing Machinery. Retrieved from https://doi.org/10.1145/3409256.3409817

Kules, B., Capra, R., Banta, M., Sierra, T. (2009). What do exploratory searchers look at in a faceted search interface? Proceedings of the 9th acm/ieee-cs joint conference on digital libraries (pp. 313–322).

Li, X., Lipton, Z.C., Dhingra, B., Li, L., Gao, J., Chen, Y.-N. (2016). A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688.

Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A. (2018). Advances in pre-training distributed word representations. In Proceedings of the international conference on language resources and evaluation (lrec 2018).

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J. (2013). Distributed representations of words and phrases and their compositionality. InProceedings of the 26th international conference on neural information processing systems - volume 2 (p. 3111–3119). Red Hook, NY, USA: Curran Associates Inc.

Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L. (2016). Generating natural questions about an image. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1802–1813).

Nogueira, R., & Cho, K. (2020). Passage re-ranking with bert.

Papangelis, A., Papadakos, P., Kotti, M., Stylianou, Y., Tzitzikas, Y., Plexousakis, D. (n.d.). Ld-sds: Towards an expressive spoken dialogue system based on linked-data. arXiv:171002973v1

Pennington, J., Socher, R., Manning, C. (2014, October). GloVe: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543). Doha, Qatar: Association for Computational Linguistics. Retrieved from https://doi.org/10.3115/v1/D14-1162

Rao, S., & Daume III, H. (2018). Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 2737–2746).

Rao, S., & Daume III, H. (2019). Answer-based adversarial training for generating clarification questions. In Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 143–155).

Reimers, N., & Gurevych, I. (2019, November). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) (pp. 3982-3992). Hong Kong, China: Association for Computational Linguistics. Retrieved from https://www.aclweb.org/anthology/D19-1410https://doi.org/10.18653/v1/D19-1410

Rudnicky, A., & Xu, W. (1999). An agenda-based dialog management architecture for spoken language systems. Ieee automatic speech recognition and understanding workshop (Vol. 13).

Salle, A., Idiart, M., Villavicencio, A. (2016). Enhancing the lexvec distributed word representation model using positional contexts and external memory.

Salle, A., Malmasi, S., Rokhlenko, O., Agichtein, E. (2021). Studying the effectiveness of conversational search refinement through user simulation. In: D. Hiemstra, M.-F. Moens, J. Mothe, R. Perego, M. Potthast, & F. Sebastiani (Eds) Advances in information retrieval. Springer International Publishing, Cham, pp. 587–602.

Salle, A., & Villavicencio, A. (2018, June). Incorporating subword information into matrix factorization word embeddings. In: Proceedings of the second workshop on subword/character LEvel models (pp. 66–71). New Orleans: Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-1209

Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S. (2007, April). Agenda-based user simulation for bootstrapping a POMDP dialogue system. Human language technologies 2007: The conference of the north American chapter of the association for computational linguistics; companion volume, short papers (pp. 149-152). Rochester, New York: Association for Computational Linguistics. Retrieved from https://aclanthology.org/N07-2038

Shah, P., Hakkani-Tur, D., Heck, L. (2016). Interactive reinforcement learning for task-oriented dialogue management. Workshop on Deep Learning for Action and Interaction.

Smucker, M.D., & Clarke, C.L. (2012). Time-based calibration of effectiveness measures. In Proceedings of the 35th international acm sigir conference on research and development in information retrieval (pp. 95–104).

Sun, Y., & Zhang, Y. (2018). Conversational recommender system. In The 41st international acm sigir conference on research & development in information retrieval (pp. 235–244).

Trippas, J.R., Spina, D., Cavedon, L., Joho, H., Sanderson, M. (2018). Informing the design of spoken conversational search: perspective paper. In Proceedings of the 2018 conference on human information interaction & retrieval (pp. 32-41).

Tunkelang, D. (2009). Faceted search. Synthesis lectures on information concepts, retrieval, and services, 1(1), 1–80.CrossRef

Vandic, D., Aanen, S., Frasincar, F., & Kaymak, U. (2017). Dynamic facet ordering for faceted product search engines. IEEE Transactions on Knowledge and Data Engineering, 29(5), 1004–1016.CrossRef

Weizenbaum, J., et al. (1966). Eliza-a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45.CrossRef

Wen, T., Vandyke, D., Mrksšíic, N., Gasšíic, M., Rojas-Barahona, L., Su, P., Young, S. (2017). A network-based end-to-end trainable task-oriented dialogue system. 15th conference of the european chapter of the association for computational linguistics, eacl 2017-proceedings of conference (Vol. 1, pp. 438-449).

Yee, K.-P., Swearingen, K., Li, K., Hearst, M. (2003). Faceted metadata for image search and browsing. In Proceedings of the sigchi conference on human factors in computing systems (pp. 401-408).

Young, S. J. (2000). Probabilistic methods in spoken-dialogue systems. Philosophical Transactions of the Royal Society of London. Series A Mathematical Physical and Engineering Sciences, 358(1769), 1389–1402.CrossRef

Zamani, H., Lueck, G., Chen, E., Quispe, R., Luu, F., Craswell, N. (2020). Mimics: A large-scale data collection for search clarification. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 3189–3196).

Zhang, S., & Balog, K. (2020). Evaluating conversational recommender systems via user simulation. In Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining (pp. 1512–1520)

Titel: CoSearcher: studying the effectiveness of conversational search refinement and clarification through user simulation
verfasst von: Alexandre Salle
Shervin Malmasi
Oleg Rokhlenko
Eugene Agichtein
Publikationsdatum: 10.03.2022
Verlag: Springer Netherlands
Erschienen in: Discover Computing / Ausgabe 2/2022
Print ISSN: 2948-2984
Elektronische ISSN: 2948-2992
DOI: https://doi.org/10.1007/s10791-022-09404-z

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 2/2022

On cross-lingual retrieval with multilingual text encoders

sMARE: a new paradigm to evaluate and understand query performance prediction methods

Open-domain conversational search assistants: the Transformer is all you need

CEQE to SQET: A study of contextualized embeddings for query expansion

Guest editorial: special issue on ECIR 2021

Premium Partner