Hui Fang
Skip Abstract Section
Abstract
Developing effective retrieval models is a long-standing central challenge in information retrieval research. In order to develop more effective models, it is necessary to understand the deficiencies of the current retrieval models and the relative strengths of each of them. In this article, we propose a general methodology to analytically and experimentally diagnose the weaknesses of a retrieval function, which provides guidance on how to further improve its performance. Our methodology is motivated by the empirical observation that good retrieval performance is closely related to the use of various retrieval heuristics. We connect the weaknesses and strengths of a retrieval function with its implementations of these retrieval heuristics, and propose two strategies to check how well a retrieval function implements the desired retrieval heuristics. The first strategy is to formalize heuristics as constraints, and use constraint analysis to analytically check the implementation of retrieval heuristics. The second strategy is to define a set of relevance-preserving perturbations and perform diagnostic tests to empirically evaluate how well a retrieval function implements retrieval heuristics. Experiments show that both strategies are effective to identify the potential problems in implementations of the retrieval heuristics. The performance of retrieval functions can be improved after we fix these problems.
- Amati, G. and Rijsbergen, C. J. V. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans. Inf. Syst. 20, 4, 357--389. Google Scholar
Digital Library
- Carterette, B. and Allan, J. 2005. Incremental test collections. In Proceedings of the 14th International Conference on Information and Knowledge Management (CIKM’05). Google ScholarDigital Library
- Carterette, B., Allan, J., and Sitaraman, R. 2006. Minimal test collections for retrieval evaluation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Cormack, G. V., Palmer, C. R., and Clarke, C. L. 1998. Efficient construction of large test collections. In Proceedings of the ACM-SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Fang, H. 2008. A re-examination of query expansion using lexical resources. In Proceedings of the 46th Annual Meetings of the Association for Computational Linguistics.Google Scholar
- Fang, H., Tao, T., and Zhai, C. 2004. A formal study of information retrieval heuristics. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Fang, H. and Zhai, C. 2005. An exploration of axiomatic approaches to information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Fang, H. and Zhai, C. 2006. Semantic term matching in axiomatic approaches to information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Fuhr, N. 1992. Probabilistic models in information retrieval. Comput. J. 35, 3, 243--255. Google ScholarDigital Library
- Fuhr, N. 2001. Language models and uncertain inference in information retrieval. In Proceedings of the Language Modeling and IR Workshop. 6--11.Google Scholar
- Harman, D. and Buckley, C. 2004. Sigir 2004 workshop: Ria and where can ir go from here. SIGIR Forum 38, 2, 45--49. Google ScholarDigital Library
- He, B. and Ounis, I. 2005. A study of the dirichlet priors for term frequency normalisation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Hiemstra, D. 2000. A probabilistic justification for using tf-idf term wieghting in information retrieval. Int. J. Digital Libraries, 131--139.Google Scholar
- Lafferty, J. and Zhai, C. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling and Information Retrieval, W. B. Croft and J. Lafferty Eds., Kluwer Academic Publishers.Google Scholar
- Lavrenko, V. and Croft, B. 2001. Relevance-Based language models. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 120--127. Google ScholarDigital Library
- Lopresti, D. and Zhou, J. 1996. Retrieval strategy for noisy text. In Proceedings of the Symposium on Document Analysis and Information Retrieval.Google Scholar
- Ponte, J. and Croft, W. B. 1998. A language modeling approach to information retrieval. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 275--281. Google ScholarDigital Library
- Robertson, S. and Sparck Jones, K. 1976. Relevance weighting of search terms. J. Amer. Soc. Inf. Sci. 27, 129--146.Google ScholarCross Ref
- Robertson, S. and Walker, S. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241. Google ScholarDigital Library
- Robertson, S. and Walker, S. 1997. On relevance weights with little relevance information. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 16--24. Google ScholarDigital Library
- Robertson, S. E., Walker, S., Jones, S., M.Hancock-Beaulieu, M., and Gatford, M. 1995. Okapi at TREC-3. In Proceedings of the 3rd Text REtrieval Conference (TREC-3). D. K. Harman Ed., 109--126.Google Scholar
- Salton, G. 1989. Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer. Addison-Wesley. Google ScholarDigital Library
- Salton, G. and Buckley, C. 1988. Term-Weighting approaches in automatic text retrieval. Inf. Process. Manag. 24, 513--523. Google ScholarDigital Library
- Salton, G. and McGill, M. 1983. Introduction to Modern Information Retrieval. McGraw-Hill. Google ScholarDigital Library
- Salton, G., Yang, C. S., and Yu, C. T. 1975. A theory of term importance in automatic text analysis. J. Amer. Soc. Inf. Sci. 26, 1, 33--44.Google ScholarCross Ref
- Sanderson, M. and Joho, H. 2004. Forming test collections with no system pooling. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Shi, S., Wen, J.-R., Yu, Q., Song, R., and Ma, W.-Y. 2005. Gravitation-based model for information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 488--495. Google ScholarDigital Library
- Singhal, A. 2001. Modern information retrieval: A brief overview. Bull. IEEE Comput. Soc. Techn. Committee Data Engin. 24, 4, 35--43.Google Scholar
- Singhal, A., Buckley, C., and Mitra, M. 1996a. Pivoted document length normalization. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 21--29. Google ScholarDigital Library
- Singhal, A., Salton, G., and Buckley, C. 1996b. Length normalization in degraded text collections. In Proceedings of the Symposium on Document Analysis and Information Retrieval. 149--162.Google Scholar
- Singhal, A., Choi, J., Hindle, D., Lewis, D. D., and Pereira, F. C. N. 1998. ATT at TREC-7. In Proceedings of the Text REtrieval Conference. 186--198.Google Scholar
- Soboroff, I., Nicholas, C., and Cahan, P. 2001. Ranking retrieval systems without relevance judgements. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Sparck Jones, K. and Willett, P., Eds. 1997. Readings in Information Retrieval. Morgan Kaufmann Publishers. Google ScholarDigital Library
- Tao, T. and Zhai, C. 2007. An exploration of proximity measures in information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Turtle, H. and Croft, W. B. 1991. Evaluation of an inference network-based retrieval model. ACM Trans. Inf. Syst. 9, 3, 187--222. Google ScholarDigital Library
- van Rijbergen, C. J. 1977. A theoretical basis for theuse of co-occurrence data in information retrieval. J. Document., 106--119.Google Scholar
- van Rijsbergen, C. J. 1986. A non-classical logic for information retrieval. Comput. J. 29, 6.Google ScholarCross Ref
- Voorhees, E. M. 2007. Trec: Continuing information retrieval’s tradition of experimentation. Comm. ACM 50, 11, 51--54. Google ScholarDigital Library
- Wong, K.-F., Song, D., Bruza, P., and Cheng, C.-H. 2001. Application of aboutness to func- tional benchmarking in information retrieval. ACM Trans. Infor. Syst. 19, 4, 337--370. Google ScholarDigital Library
- Wong, S. K. M. and Yao, Y. Y. 1995. On modeling information retrieval with probabilistic inference. ACM Trans. Inf. Syst. 13, 1, 69--99. Google ScholarDigital Library
- Zhai, C. and Lafferty, J. 2001a. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of the 10th International Conference on Information and Knowledge Management (CIKM’01). 403--410. Google ScholarDigital Library
- Zhai, C. and Lafferty, J. 2001b. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 334--342. Google ScholarDigital Library
- Zhou, Y. and Croft, W. B. 2006. Ranking robustness: a novel framework to predict query performance. In Proceedings of the 15th International Conference on Information and Knowledge Management (CIKM’06). 567. Google ScholarDigital Library
- Zobel, J. 1998. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Google ScholarDigital Library
- Zobel, J. and Moffat, A. 1998. Exploring the similarity space. SIGIR Forum 31, 1, 18--34. Google ScholarDigital Library
Index Terms
- Diagnostic Evaluation of Information Retrieval Models
Recommendations
An exploration of proximity measures in information retrieval
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalIn most existing retrieval models, documents are scored primarily based on various kinds of term statistics such as within-document frequencies, inverse document frequencies, and document lengths. Intuitively, the proximity of matched query terms in a ...
A formal study of information retrieval heuristics
SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrievalEmpirical studies of information retrieval methods show that good retrieval performance is closely related to the use of various retrieval heuristics, such as TF-IDF weighting. One basic research question is thus what exactly are these "necessary" ...
An exploration of axiomatic approaches to information retrieval
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrievalExisting retrieval models generally do not offer any guarantee for optimal retrieval performance. Indeed, it is even difficult, if not impossible, to predict a model's empirical performance analytically. This limitation is at least partly caused by the ...
Comments