ABSTRACT
Many techniques for improving search result quality have been proposed. Typically, these techniques increase average effectiveness by devising advanced ranking features and/or by developing sophisticated learning to rank algorithms. However, while these approaches typically improve average performance of search results relative to simple baselines, they often ignore the important issue of robustness. That is, although achieving an average gain overall, the new models often hurt performance on many queries. This limits their application in real-world retrieval scenarios. Given that robustness is an important measure that can negatively impact user satisfaction, we present a unified framework for jointly optimizing effectiveness and robustness. We propose an objective that captures the tradeoff between these two competing measures and demonstrate how we can jointly optimize for these two measures in a principled learning framework. Experiments indicate that ranking models learned this way significantly decreased the worst ranking failures while maintaining strong average effectiveness on par with current state-of-the-art models.
- P. N. Bennett, F. Radlinski, R. W. White, and E. Yilmaz. Inferring and using location metadata to personalize web search. In SIGIR, pages 135--144, 2011. Google ScholarDigital Library
- R. Bhattacharjee and A. Goel. Algorithms and incentives for robust ranking. In SODA 2007. Google ScholarDigital Library
- J. Bian, X. Li, F. Li, Z. Zheng, and H. Zha. Ranking specialization for web search: a divide-and-conquer approach by using topical ranksvm. In WWW, pages 131--140, 2010. Google ScholarDigital Library
- J. Bian, T.-Y. Liu, T. Qin, and H. Zha. Ranking with query-dependent loss for web search. In WSDM, pages 141--150, 2010. Google ScholarDigital Library
- C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML, pages 89--96, 2005. Google ScholarDigital Library
- C. J. C. Burges. From RankNet to LambdaRank to LambdaMART: An Overview. Technical report, Microsoft Research, 2010.Google Scholar
- C. J. C. Burges, R. Ragno, and Q. V. Le. Learning to Rank with Nonsmooth Cost Functions. In NIPS, pages 193--200. MIT Press, 2006.Google ScholarDigital Library
- S. Büttcher, C. L. A. Clarke, and B. Lushman. Term proximity scoring for ad-hoc retrieval on very large text collections. In SIGIR, pages 621--622, 2006. Google ScholarDigital Library
- O. Chapelle and Y. Chang. Yahoo! learning to rank challenge overview. Journal of Machine Learning Research - Proceedings Track, 14:1--24, 2011.Google Scholar
- K. Collins-Thompson. Reducing the risk of query expansion via robust constrained optimization. In CIKM 2009, pages 837--846. Google ScholarDigital Library
- K. Collins-Thompson, P. N. Bennett, R. W. White, S. de la Chica, and D. Sontag. Personalizing web search results by reading level. In CIKM, pages 403--412, 2011. Google ScholarDigital Library
- S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Predicting query performance. In SIGIR, pages 299--306. Google ScholarDigital Library
- N. Dai, M. Shokouhi, and B. D. Davison. Learning to rank for freshness and relevance. In SIGIR, pages 95--104, 2011. Google ScholarDigital Library
- P. Donmez, K. M. Svore, and C. J. C. Burges. On the local optimality of lambdarank. In SIGIR, pages 460--467, 2009. Google ScholarDigital Library
- Z. Dou, R. Song, and J.-R. Wen. A large-scale evaluation and analysis of personalized search strategies. In WWW, pages 581--590, 2007. Google ScholarDigital Library
- E. N. Efthimiadis. Chapter: Query expansion. Annual Review of Information Science and Technology, 31:121--187, 1996.Google Scholar
- J. H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1189--1232, 2000.Google ScholarCross Ref
- X. Geng, T.-Y. Liu, T. Qin, A. Arnold, H. Li, and H.-Y. Shum. Query dependent ranking using k-nearest neighbor. In SIGIR, pages 115--122, 2008. Google ScholarDigital Library
- F. Gey. Inferring probability of relevance using the method of logistic regression. In Proc. 17th SIGIR Conference, 1994. Google ScholarDigital Library
- Q. Guo, R. W. White, S. Dumais, J. Wang, and B. Anderson. Predicting query performance using query, result, and user interaction features. In RIAO, 2010. Google ScholarDigital Library
- C. Hauff, D. Hiemstra, and F. de Jong. A survey of pre-retrieval query performance predictors. In CIKM, pages 1419--1420, 2008. Google ScholarDigital Library
- C. Kang, X. Wang, J. Chen, C. Liao, Y. Chang, B. L. Tseng, and Z. Zheng. Learning to re-rank web search results with multiple pairwise features. In WSDM'11, pages 735--744, 2011. Google ScholarDigital Library
- B. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, and C. Newell. Explaining the user experience of recommender systems. J. of User Modeling and User-Adapted Interaction (UMUAI), 22, 2011. Google ScholarDigital Library
- F. Li, X. Li, S. Ji, and Z. Zheng. Comparing both relevance and robustness in selection of web ranking functions. In SIGIR '09, pages 648--649, 2009. Google ScholarDigital Library
- T.-Y. Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3(3), 2009.Google Scholar
- Y. Lv, C. Zhai, and W. Chen. A boosting approach to improving pseudo-relevance feedback. In SIGIR, pages 165--174, 2011. Google ScholarDigital Library
- D. Metzler and W. B. Croft. A Markov Random Field model for term dependencies. In Proc. 28th SIGIR Conference, pages 472--479, 2005. Google ScholarDigital Library
- J. Ponte and W. B. Croft. A language modeling approach to information retrieval. In Proc. 21st SIGIR Conference, pages 275--281, 1998. Google ScholarDigital Library
- S. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Proc. 3rd TREC Conference, pages 109--126, 1994.Google Scholar
- R. Steuer. Multiple Criteria Optimization: Theory, Computation and Application. John Wiley, 1986.Google Scholar
- K. M. Svore, M. N. Volkovs, and C. J. Burges. Learning to rank with multiple objective functions. In WWW, pages 367--376, 2011. Google ScholarDigital Library
- T. Tao and C. Zhai. An exploration of proximity measures in information retrieval. In Proc. 30th SIGIR Conference, 2007. Google ScholarDigital Library
- J. Teevan, S. T. Dumais, and E. Horvitz. Potential for personalization. ACM TOCHI, 17(1), 2010. Google ScholarDigital Library
- E. M. Voorhees. The trec robust retrieval track. SIGIR Forum, 39(1):11--20, June 2005. Google ScholarDigital Library
- J. Wang and J. Zhu. Portfolio theory of information retrieval. In SIGIR. ACM, 2009. Google ScholarDigital Library
- R. W. White, P. N. Bennett, and S. T. Dumais. Predicting short-term interests using activity-based search context. In CIKM, pages 1009--1018, 2010. Google ScholarDigital Library
- Q. Wu, C. J. Burges, K. M. Svore, and J. Gao. Adapting boosting for information retrieval measures. Information Retrieval, 13:254--270, June 2010. Google ScholarDigital Library
- Y. Zhou and W. B. Croft. Query performance prediction in web search environments. In Research and Development in Information Retrieval, pages 543--550, 2007. Google ScholarDigital Library
- J. Zhu, J. Wang, I. J. Cox, and M. J. Taylor. Risky business: modeling and exploiting uncertainty in information retrieval. In SIGIR. ACM, 2009. Google ScholarDigital Library
Index Terms
- Robust ranking models via risk-sensitive optimization
Recommendations
Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge ManagementIn this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...
Video search re-ranking via multi-graph propagation
MM '07: Proceedings of the 15th ACM international conference on MultimediaThis paper1 is concerned with the problem of multimodal fusion in video search. First, we employ an object-sensitive approach to query analysis to improve the baseline result of text-based video search. Then, we propose a PageRank-like graph-based ...
Ranking model selection and fusion for effective microblog search
SoMeRA '14: Proceedings of the first international workshop on Social media retrieval and analysisRe-ranking was shown to have positive impact on the effectiveness for microblog search. Yet existing approaches mostly focused on using a single ranker to learn some better ranking function with respect to various relevance features. Given various ...
Comments