Jiepu Jiang
Ryen W. White
ABSTRACT
Understanding and estimating satisfaction with search engines is an important aspect of evaluating retrieval performance. Research to date has modeled and predicted search satisfaction on a binary scale, i.e., the searchers are either satisfied or dissatisfied with their search outcome. However, users' search experience is a complex construct and there are different degrees of satisfaction. As such, binary classification of satisfaction may be limiting. To the best of our knowledge, we are the first to study the problem of understanding and predicting graded (multi-level) search satisfaction. We ex-amine sessions mined from search engine logs, where searcher satisfaction was also assessed on multi-point scale by human annotators. Leveraging these search log data, we observe rich and non-monotonous changes in search behavior in sessions with different degrees of satisfaction. The findings suggest that we should predict finer-grained satisfaction levels. To address this issue, we model search satisfaction using features indicating search outcome, search effort, and changes in both outcome and effort during a session. We show that our approach can predict subtle changes in search satisfaction more accurately than state-of-the-art methods, affording greater insight into search satisfaction. The strong performance of our models has implications for search providers seeking to accu-rately measure satisfaction with their services.
- M. Ageev et al. 2011. Find it if you can: A game for modeling different types of web search success using interaction data. In SIGIR'11: 345--354. Google Scholar
Digital Library
- A. Al-Maskari et al. 2007. The relationship between IR effec-tiveness measures and user satisfaction. In SIGIR'07: 773--774. Google ScholarDigital Library
- J. Arguello. 2014. Predicting search task difficulty. In ECIR'14: 88--99.Google ScholarDigital Library
- L. Azzopardi et al. 2013. How query cost affects search be-havior. In SIGIR'13: 23--32. Google ScholarDigital Library
- L. Azzopardi. 2014. Modelling interaction with economic models of search. In SIGIR'14: 3--12. Google ScholarDigital Library
- J. E. Bailey and S. W. Pearson. 1983. Development of a tool for measuring and analyzing computer user satisfaction. Man-agement Science, 29(5): 530--545.Google ScholarDigital Library
- P. N. Bennett et al. 2012. Modeling the impact of short- and long-term behavior on search personalization. In SIGIR'12: 185--194. Google ScholarDigital Library
- H. Feild et al. 2010. Predicting searcher frustration. In SIGIR'10: 34--41. Google ScholarDigital Library
- S. Fox et al. 2005. Evaluating implicit measures to improve web search. ACM TOIS, 23(2): 147--168. Google ScholarDigital Library
- Q. Guo et al. 2011. Why searchers switch: understanding and predicting engine switching rationales. In SIGIR'11: 335--344. Google ScholarDigital Library
- A. Hassan. 2012. A semi-supervised approach to modeling web search satisfaction. In SIGIR'12: 275--284. Google ScholarDigital Library
- A. Hassan et al. 2013. Beyond clicks: Query reformulation as a predictor of search satisfaction. In CIKM'13: 2019--2028. Google ScholarDigital Library
- A. Hassan et al. 2010. Beyond DCG: User behavior as a pre-dictor of a successful search. In WSDM'10: 221--230. Google ScholarDigital Library
- A. Hassan et al. 2014. Struggling or exploring? Disambiguating long search sessions? In WSDM'14: 53--62. Google ScholarDigital Library
- S. B. Huffman and M. Hochster. 2007. How well does result relevance predict session satisfaction? In SIGIR'07: 567--574. Google ScholarDigital Library
- B. Ives et al. 1983. The measurement of user information sat-isfaction. CACM, 26(10): 785--793. Google ScholarDigital Library
- K. Järvelin and J. Kekäläinen. 2000. IR evaluation methods for retrieving highly relevant documents. In SIGIR'00: 41--48. Google ScholarDigital Library
- K. Järvelin et al. 2008. Discounted cumulated gain based eval-uation of multiple-query IR sessions. In ECIR'08: 4--15. Google ScholarDigital Library
- J. Jiang et al. Searching, browsing, and clicking in a search session: changes in user behavior by task and over time. In SIGIR'14: 607--616. Google ScholarDigital Library
- T. Joachims et al. 2005. Accurately interpreting clickthrough data as implicit feedback. In SIGIR'05: 154--161. Google ScholarDigital Library
- R. Jones and K. Klinkner. 2008. Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In CIKM'08: 699--708. Google ScholarDigital Library
- E. Kanoulas et al. 2011. Evaluating multi-query sessions. In SIGIR'11: 1053--1062. Google ScholarDigital Library
- D. Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Foundation and Trends in In-formation Retrieval, 3(1--2): 1--224. Google ScholarDigital Library
- Y. Kim et al. 2014. Modeling dwell time to predict click-level satisfaction. In WSDM'14: 193--202. Google ScholarDigital Library
- R. Kohavi et al. 2009. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Dis-covery. 18(1): 140--181. Google ScholarDigital Library
- J. Liu et al. 2012. Exploring and predicting search task diffi-culty. In CIKM'12: 1313--1322. Google ScholarDigital Library
- J. Liu and N. J. Belkin. 2010. Personalizing information re-trieval for multi-session tasks. In SIGIR'10: 26--33. Google ScholarDigital Library
- L. Lorigo et al. 2006. The influence of task and gender on search and evaluation behavior using Google. IP&M, 42(4): 1123--1131. Google ScholarDigital Library
- G. Mankiw. 2010. Principles of Macroeconomics. South-Western Cengage Learning.Google Scholar
- A. Marshall. 2009. Principles of Economics: Abridged Edi-tion. Cosimo Classics.Google Scholar
- V. McKinney et al. 2002. The measurement of Web-customer satisfaction: an expectation and disconfirmation approach. In-formation Systems Research, 13(3): 296--315. Google ScholarDigital Library
- R. L. Oliver. 1980. A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research. 17(4): 460--470.Google ScholarCross Ref
- C. L. Smith and P. B. Kantor. 2008. User adaptation: Good results from poor systems. In SIGIR'08: 147--154. Google ScholarDigital Library
- L. T. Su. 2003. A comprehensive and systematic model of user evaluation of Web search engines. JASIST, 54(13): 1175--1192. Google ScholarDigital Library
- H. Wang et al. 2014. Modeling action-level satisfaction for search task satisfaction prediction. In SIGIR'14: 123--132. Google ScholarDigital Library
- R. W. White. 2013. Beliefs and biases in web search. In SIGIR'13: 3--12. Google ScholarDigital Library
- E. Yilmaz et al. 2014. Relevance and effort: an analysis of document utility. In CIKM'14: 91--100. Google ScholarDigital Library
Index Terms
- Understanding and Predicting Graded Search Satisfaction
Recommendations
Measuring and Predicting Search Engine Users’ Satisfaction
Search satisfaction is defined as the fulfillment of a user’s information need. Characterizing and predicting the satisfaction of search engine users is vital for improving ranking models, increasing user retention rates, and growing market share. This ...
Beyond Success Rate: Utility as a Search Quality Metric for Online Experiments
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge ManagementUser satisfaction metrics are an integral part of search engine development as they help system developers to understand and evaluate the quality of the user experience. Research to date has mostly focused on predicting success or frustration as a proxy ...
Does Vertical Bring more Satisfaction?: Predicting Search Satisfaction in a Heterogeneous Environment
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge ManagementThe study of search satisfaction is one of the prime concerns in search performance evaluation research. Most existing works on search satisfaction primarily rely on the hypothesis that all results on search engine result pages (SERPs) are homogeneous. ...
Comments