Ovidiu Dan
Brian D. Davison
Skip Abstract Section
Abstract
Search satisfaction is defined as the fulfillment of a user’s information need. Characterizing and predicting the satisfaction of search engine users is vital for improving ranking models, increasing user retention rates, and growing market share. This article provides an overview of the research areas related to user satisfaction. First, we show that whenever users choose to defect from one search engine to another they do so mostly due to dissatisfaction with the search results. We also describe several search engine switching prediction methods, which could help search engines retain more users. Second, we discuss research on the difference between good and bad abandonment, which shows that in approximately 30% of all abandoned searches the users are in fact satisfied with the results. Third, we catalog techniques to determine queries and groups of queries that are underperforming in terms of user satisfaction. This can help improve search engines by developing specialized rankers for these query patterns. Fourth, we detail how task difficulty affects user behavior and how task difficulty can be predicted. Fifth, we characterize satisfaction and we compare major satisfaction prediction algorithms.
- Mikhail Ageev, Qi Guo, Dmitry Lagun, and Eugene Agichtein. 2011. Find it if you can. In SIGIR’11. ACM Press, New York, NY, 345.Google Scholar
- Azzah Al-Maskari, Mark Sanderson, and Paul Clough. 2007. The relationship between IR effectiveness measures and user satisfaction. In SIGIR’07. ACM Press, New York, NY, 773. Google ScholarDigital Library
- Rolph E. Anderson and Srini S. Srinivasan. 2003. E-satisfaction and e-loyalty: A contingency framework. Psychol. Market. 20, 2 (Feb. 2003), 123--138.Google ScholarCross Ref
- Jaime Arguello. 2014. Predicting search task difficulty. In Advances in Information Retrieval. Springer, Berlin, 88--99.Google Scholar
- Anne Aula, Rehan M. Khan, and Zhiwei Guan. 2010. How does search behavior change as search becomes more difficult? In CHI’10. ACM Press, New York, NY, 35. Google ScholarDigital Library
- Sharon E. Beatty and Scott M. Smith. 1987. External search effort: An investigation across several product categories. J. Consum. Res. (1987), 83--95.Google Scholar
- Ruth N. Bolton. 1998. A dynamic model of the duration of the customer’s relationship with a continuous service provider: The role of satisfaction. Market. Sci. 17, 1 (1998), 45--65. Google ScholarDigital Library
- David Carmel, Elad Yom-Tov, Adam Darlow, and Dan Pelleg. 2006. What makes a query difficult? In SIGIR’06. ACM Press, New York, NY, 390. Google ScholarDigital Library
- Lydia B. Chilton and Jaime Teevan. 2011. Addressing people’s information needs directly in a web search result page. In WWW’11. ACM Press, New York, NY, 27. Google ScholarDigital Library
- Aleksandr Chuklin and Pavel Serdyukov. 2012a. Good abandonments in factoid queries. In WWW’12 Companion. ACM Press, New York, NY, 483. Google ScholarDigital Library
- Aleksandr Chuklin and Pavel Serdyukov. 2012b. Potential good abandonment prediction. In WWW’12 Companion. ACM Press, New York, NY, 485. Google ScholarDigital Library
- G. A. Churchill Jr. and C. Surprenant. 1982. Churchill, Gilbert A., Jr, an investigation into the determinants of customer satisfaction, 19:4 (1982:Nov.) p.491. J. Market. Res. 4, 19 (1982), 491--504.Google ScholarCross Ref
- Steve Cronen-Townsend, Yun Zhou, and W. Bruce Croft. 2002. Predicting query performance. In SIGIR’02. ACM Press, New York, NY, 299. Google ScholarDigital Library
- Ovidiu Dan, Pavel Dmitriev, and Ryen W. White. 2012. Mining for insights in the search engine query stream. In WWW’12 Companion. ACM Press, New York, NY, 489. Google ScholarDigital Library
- Abdigani Diriye, Ryen White, Georg Buscher, and Susan Dumais. 2012. Leaving so soon? In CIKM’12. ACM Press, New York, NY, 1025.Google Scholar
- Richard O. Duda, Peter E. Hart, and David G. Stork. 2000. Pattern Classification. John Wiley & Sons, New York, NY. Google ScholarDigital Library
- J. F. Engel and R. D. Blackwell. 1982. Consumer Behavior. Dryden Press.Google Scholar
- Deborah Fallows. Search Engine Users. Technical Report. Pew Internet and American Life Project. http://www.pewinternet.org/files/old-media/Files/Reports/2005/PIP_Searche ngine_users.pdf.pdf.Google Scholar
- Henry A. Feild, James Allan, and Rosie Jones. 2010. Predicting searcher frustration. In SIGIR’10. ACM Press, New York, NY, 34. Google ScholarDigital Library
- Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Trans. Inform. Syst. 23, 2 (April 2005), 147--168. Google ScholarDigital Library
- Qi Guo, Ryen W. White, Susan T. Dumais, Jue Wang, and Blake Anderson. 2010. Predicting query performance using query, result, and user interaction features. In Adaptivity, Personalization and Fusion of Heterogeneous Information. LE CENTRE DE HAUTES ETUDES INTERNATIONALES D’INFORMATIQUE DOCUMENTAIRE, 198--201. Google ScholarDigital Library
- Qi Guo, Ryen W. White, Yunqiao Zhang, Blake Anderson, and Susan T. Dumais. 2011. Why searchers switch. In SIGIR’11. ACM Press, New York, NY, 335.Google Scholar
- Ahmed Hassan. 2012. A semi-supervised approach to modeling web search satisfaction. In SIGIR’12. ACM Press, New York, NY, 275. Google ScholarDigital Library
- Ahmed Hassan, Rosie Jones, and Kristina Lisa Klinkner. 2010. Beyond DCG. In WSDM’10. ACM Press, New York, NY, 221. Google ScholarDigital Library
- Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey. 2013. Beyond clicks. In CIKM’13. ACM Press, New York, NY, 2019--2028. Google ScholarDigital Library
- Ahmed Hassan, Yang Song, and Li-wei He. 2011. A task level metric for measuring web search satisfaction and its application on improving relevance estimation. In CIKM’11. ACM Press, New York, NY, 125. Google ScholarDigital Library
- Ahmed Hassan and Ryen W. White. 2013. Personalized models of search satisfaction. In CIKM’13. ACM Press, New York, NY, 2009--2018. Google ScholarDigital Library
- Ahmed Hassan, Ryen W. White, Susan T. Dumais, and Yi-Min Wang. 2014. Struggling or exploring? Disambiguating long search sessions. In WSDM’14. ACM Press, New York, NY, 53--62. Google ScholarDigital Library
- Ahmed Hassan, Ryen W. White, and Yi-Min Wang. 2013. Toward self-correcting search engines. In SIGIR’13. ACM Press, New York, NY, 263. Google ScholarDigital Library
- Claudia Hauff, Vanessa Murdock, and Ricardo Baeza-Yates. 2008. Improved query difficulty prediction for the web. In CIKM’08. ACM Press, New York, NY, 439. Google ScholarDigital Library
- Ben He and Iadh Ounis. 2004. Inferring query performance using pre-retrieval predictors. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 3246. 43--54.Google Scholar
- Ben He and Iadh Ounis. 2006. Query performance prediction. Inform. Syst. 31, 7 (Nov. 2006), 585--594. Google ScholarDigital Library
- Allison P. Heath and Ryen W. White. 2008. Defection detection. In WWW’08. ACM Press, New York, NY, 1173. Google ScholarDigital Library
- Thorsten Hennig-Thurau and Alexander Klee. 1997. The impact of customer satisfaction and relationship quality on customer retention: A critical reassessment and model development. Psychol. Market. 14 (1997), 737--764.Google ScholarCross Ref
- Victor Hu, Maria Stone, Jan Pedersen, and Ryen W. White. 2011. Effects of search success on search engine re-use. In CIKM’11. ACM Press, New York, NY, 1841. Google ScholarDigital Library
- Jeff Huang, Ryen W. White, and Susan Dumais. 2011. No clicks, no problem. In CHI’11. ACM Press, New York, NY, 1225.Google Scholar
- Scott B. Huffman and Michael Hochster. 2007. How well does result relevance predict session satisfaction? In SIGIR’07. ACM Press, New York, NY, 567. Google ScholarDigital Library
- Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inform. Syst. (TOIS) 20, 4 (2002), 422--446. Google ScholarDigital Library
- Jiepu Jiang, Ahmed Hassan Awadallah, Xiaolin Shi, and Ryen W. White. 2015. Understanding and predicting graded search satisfaction. In WSDM’15. ACM, New York, NY, 57--66. Google ScholarDigital Library
- Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In SIGIR’05. ACM Press, New York, NY, 154. Google ScholarDigital Library
- Thomas O. Jones, W. Earl Sasser, and others. 1995. Why satisfied customers defect. Harvard Bus. Rev. 73, 6 (1995), 88.Google Scholar
- Yun-Fang Juan and Chi-Chao Chang. 2005. An analysis of search engine switching behavior using click streams. In Special Interest Tracks and Posters - WWW’05. ACM Press, New York, NY, 1050. Google ScholarDigital Library
- Jeonghyun Kim. 2006. Task difficulty as a predictor and indicator of web searching interaction. In CHI’06 Extended Abstracts on Human Factors in Computing Systems - CHI EA’06. ACM Press, New York, NY, 959. Google ScholarDigital Library
- Youngho Kim, Ahmed Hassan, Ryen W. White, and Yi-Min Wang. 2013. Playing by the rules. In WSDM’13. ACM Press, New York, NY, 133.Google Scholar
- Youngho Kim, Ahmed Hassan, Ryen W. White, and Imed Zitouni. 2014. Modeling dwell time to predict click-level satisfaction. In WSDM’14. ACM, New York, NY, 193--202. Google ScholarDigital Library
- Srivatsan Laxman, Vikram Tankasali, and Ryen W. White. 2008. Stream prediction using a generative model based on frequent episodes in event sequences. In KDD’08. ACM Press, New York, NY, 453. Google ScholarDigital Library
- Jane Li, Scott Huffman, and Akihito Tokuda. 2009. Good abandonment in mobile and PC internet search. In SIGIR’09. 43. Google ScholarDigital Library
- Chang Liu, Jingjing Liu, and Nicholas J. Belkin. 2014. Predicting search task difficulty at different search stages. In CIKM’14. ACM Press, New York, NY, 569--578. Google ScholarDigital Library
- Chang Liu, Jingjing Liu, Michael Cole, Nicholas J. Belkin, and Xiangmin Zhang. 2012a. Task difficulty and domain knowledge effects on information search behaviors. Proc. Am. Soc. Inform. Sci. Technol. 49, 1 (Jan. 2012), 1--10.Google ScholarCross Ref
- Jingjing Liu, Chang Liu, Michael Cole, Nicholas J. Belkin, and Xiangmin Zhang. 2012b. Exploring and predicting search task difficulty. In CIKM’12. ACM Press, New York, NY, 1313. Google ScholarDigital Library
- Jingjing Liu, Chang Liu, Jacek Gwizdka, and Nicholas J. Belkin. 2010a. Can search systems detect users’ task difficulty? In SIGIR’10. ACM Press, New York, NY, 845. Google ScholarDigital Library
- Jingjing Liu, Chang Liu, Jacek Gwizdka, and Nicholas J. Belkin. 2010b. Predicting task difficulty for different task types. Proc. Am. Soc. Inform. Sci. Technol. 47, 1 (Nov. 2010), 1--10.Google Scholar
- Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, and others. 2008. Intro. Inform. Retriev. Vol. 1. Cambridge University Press.Google Scholar
- T. Mukhopadhyay, U. Rajan, and R. Telang. 2004. Competition between internet search engines. In Proceedings of the 37th Annual Hawaii International Conference on System Sciences, 2004. IEEE, Los Alamitos, CA, 10 pp. Google ScholarDigital Library
- R. L. Oliver. 2006. Customer satisfaction research. In The Handbook of Marketing Research. 1--40.Google Scholar
- Denis Savenkov, Dmitry Lagun, and Qiaoling Liu. 2013. Search engine switching detection based on user personal preferences and behavior patterns. In SIGIR’13. ACM Press, New York, NY, 33. Google ScholarDigital Library
- Yang Song, Xiaolin Shi, Ryen White, and Ahmed Hassan Awadallah. 2014. Context-aware web search abandonment prediction. In SIGIR’14. ACM, New York, NY, 93--102. Google ScholarDigital Library
- Sofia Stamou and Efthimis N. Efthimiadis. 2010. Interpreting user inactivity on search results. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 5993 LNCS. 100--113. Google ScholarDigital Library
- David K. Tse and Peter C. Wilton. 1988. Models of consumer satisfaction formation: An extension. J. Market. Rese.arch 25, 2 (1988), pp. 204--212.Google Scholar
- Hongning Wang, Yang Song, Ming-Wei Chang, Xiaodong He, Ahmed Hassan, and Ryen White. 2014. Modeling action-level satisfaction for search task satisfaction prediction. In SIGIR’14. ACM, New York, NY. Google ScholarDigital Library
- Ryen W. White and Steven M. Drucker. 2007. Investigating behavioral variability in web search. In WWW’07. ACM Press, New York, NY, 21. Google ScholarDigital Library
- Ryen W. White and Susan T. Dumais. 2009. Characterizing and predicting search engine switching behavior. In CIKM’09. ACM Press, New York, NY, 87. Google ScholarDigital Library
- Ryen W. White, Ashish Kapoor, and Susan T. Dumais. 2010. Modeling long-term search engine usage. In User Modeling, Adaptation, and Personalization. Springer, Berlin, 28--39. Google ScholarDigital Library
- Ryen W. White and Dan Morris. 2007. Investigating the querying and browsing behavior of advanced search engine users. In SIGIR’07. ACM Press, New York, NY, 255. Google ScholarDigital Library
- Ryen W. White, Matthew Richardson, Mikhail Bilenko, and Allison P. Heath. 2008. Enhancing web search by promoting multiple search engine use. In SIGIR’08. ACM Press, New York, NY, 43. Google ScholarDigital Library
Index Terms
- Measuring and Predicting Search Engine Users’ Satisfaction
Recommendations
"Satisfaction with Failure" or "Unsatisfied Success": Investigating the Relationship between Search Success and User Satisfaction
WWW '18: Proceedings of the 2018 World Wide Web ConferenceUser satisfaction has been paid much attention to in recent Web search evaluation studies. Although satisfaction is often considered as an important symbol of search success, it doesn»t guarantee success in many cases, especially for complex search task ...
A task level metric for measuring web search satisfaction and its application on improving relevance estimation
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementUnderstanding the behavior of satisfied and unsatisfied Web search users is very important for improving users search experience. Collecting labeled data that characterizes search behavior is a very challenging problem. Most of the previous work used a ...
Beyond DCG: user behavior as a predictor of a successful search
WSDM '10: Proceedings of the third ACM international conference on Web search and data miningWeb search engines are traditionally evaluated in terms of the relevance of web pages to individual queries. However, relevance of web pages does not tell the complete picture, since an individual query may represent only a piece of the user's ...
Comments