John Guiver
ABSTRACT
In this paper we address the issue of learning to rank for document retrieval using Thurstonian models based on sparse Gaussian processes. Thurstonian models represent each document for a given query as a probability distribution in a score space; these distributions over scores naturally give rise to distributions over document rankings. However, in general we do not have observed rankings with which to train the model; instead, each document in the training set is judged to have a particular relevance level: for example "Bad", "Fair", "Good", or "Excellent". The performance of the model is then evaluated using information retrieval (IR) metrics such as Normalised Discounted Cumulative Gain (NDCG). Recently Taylor et al. presented a method called SoftRank which allows the direct gradient optimisation of a smoothed version of NDCG using a Thurstonian model. In this approach, document scores are represented by the outputs of a neural network, and score distributions are created artificially by adding random noise to the scores. The SoftRank mechanism is a general one; it can be applied to different IR metrics, and make use of different underlying models. In this paper we extend the SoftRank framework to make use of the score uncertainties which are naturally provided by a Gaussian process (GP), which is a probabilistic non-linear regression model. We further develop the model by using sparse Gaussian process techniques, which give improved performance and efficiency, and show competitive results against baseline methods when tested on the publicly available LETOR OHSUMED data set. We also explore how the available uncertainty information can be used in prediction and how it affects model performance.
- C. Burges, R. Ragno, and Q. V. L. Le. Learning to rank with nonsmooth cost functions. In NIPS, 2006.Google Scholar
- C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML, 2005. Google ScholarDigital Library
- W. Chu and Z. Ghahramani. Gaussian processes for ordinal regression. JMLR, 6:1019--1041, 2005. Google ScholarDigital Library
- K. Crammer and Y. Singer. Pranking with ranking. In NIPS 14, 2002.Google Scholar
- R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115--132. MIT Press, 2000.Google Scholar
- K. Järvelin and J. Kekäläinen. IR evaluation methods for retrieving highly relevant documents. In SIGIR, 2000. Google ScholarDigital Library
- T. Joachims. Optimizing search engines using clickthrough data. In KDD, 2002. Google ScholarDigital Library
- N. D. Lawrence. Learning for larger datasets with the Gaussian process latent variable model. In M. Meila and X. Shen, editors, AISTATS 11. Omnipress, 2007.Google Scholar
- T.-Y. Liu. LETOR: Benchmark datasets for learning to rank, 2007. Microsoft Research Asia. http://research.microsoft.com/users/LETOR/.Google Scholar
- R. M. Neal. Bayesian learning for neural networks. In Lecture Notes in Statistics 118. Springer, 1996. Google ScholarDigital Library
- J. Nocedal and S. Wright. Numerical Optimization, Second Edition. Springer, 2006.Google Scholar
- J. Quiñonero Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. JMLR, 6:1939--1959, Dec 2005. Google ScholarDigital Library
- C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT press, 2006. Google ScholarDigital Library
- S. Robertson and H. Zaragoza. On rank-based effectiveness measures and optimization. Information Retrieval, 10(3):321--339, 2007. Google ScholarDigital Library
- S. Robertson, H. Zaragoza, and M. Taylor. A simple BM 25 extension to multiple weighted fields. In CIKM, pages 42--29, 2004. Google ScholarDigital Library
- E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. Platt, editors, NIPS 18, pages 1257--1264. MIT press, Cambridge, MA, 2006.Google Scholar
- M. Taylor, J. Guiver, S. Robertson, and T. Minka. SoftRank: optimizing non-smooth rank metrics. In WSDM '08, pages 77--86. ACM, 2008. Google ScholarDigital Library
- L. L. Thurstone. A law of comparative judgement. Psychological Reviews, 34:273--286, 1927.Google ScholarCross Ref
Index Terms
- Learning to rank with SoftRank and Gaussian processes
Recommendations
SoftRank: optimizing non-smooth rank metrics
WSDM '08: Proceedings of the 2008 International Conference on Web Search and Data MiningWe address the problem of learning large complex ranking functions. Most IR applications use evaluation metrics that depend only upon the ranks of documents. However, most ranking functions generate document scores, which are sorted to produce a ...
Effective rank aggregation for metasearching
Nowadays, mashup services and especially metasearch engines play an increasingly important role on the Web. Most of users use them directly or indirectly to access and aggregate information from more than one data sources. Similarly to the rest of the ...
Rank aggregation model for meta search: an approach using text and rank analysis measures
Intelligent information processing IIOne problem domain of meta search is to combine and improve the precision of ranking results from various search systems. This paper describes a rank aggregation model that incorporates text analysis measure with existing rank-based method, e.g. Best ...
Comments