nach oben

Discover Computing

Erschienen in:

01.02.2011 | The Second International Conference on the Theory of Information Retrieval (ICTIR2009)

Modeling score distributions in information retrieval

verfasst von: Avi Arampatzis, Stephen Robertson

Erschienen in: Discover Computing | Ausgabe 1/2011

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

We review the history of modeling score distributions, focusing on the mixture of normal-exponential by investigating the theoretical as well as the empirical evidence supporting its use. We discuss previously suggested conditions which valid binary mixture models should satisfy, such as the Recall-Fallout Convexity Hypothesis, and formulate two new hypotheses considering the component distributions, individually as well as in pairs, under some limiting conditions of parameter values. From all the mixtures suggested in the past, the current theoretical argument points to the two gamma as the most-likely universal model, with the normal-exponential being a usable approximation. Beyond the theoretical contribution, we provide new experimental evidence showing vector space or geometric models, and BM25, as being ‘friendly’ to the normal-exponential, and that the non-convexity problem that the mixture possesses is practically not severe. Furthermore, we review recent non-binary mixture models, speculate on graded relevance, and consider methods such as logistic regression for score calibration.

Vorheriger Artikel Retrieval constraints and word frequency distributions a log-logistic model for IR

Nächster Artikel Variational bayes for modeling score distributions

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

The full table is not shown here. At its bottom part, there are cases where the fits are a complete failure (median upper probability of practically zero) and the F ₁@K correlation is very weak: 0.07–0.15.

As a proof for this consider kernel density estimation methods with a Gaussian kernel, i.e. methods for approximating an arbitrary density from data points by a non-weighted sum of equal variance Gaussians positioned at each data point. By allowing a weighted sum and unequal variances, a mixture of Gaussians provides even better flexibility.

The Kanoulas et al. (2009) results are arguable, given the use of the K-S goodness-of-fit test in inappropriate ways. In principle, the K-S test cannot be used when the distribution parameters are estimated from the data, as in their study; however, their results can be considered indicative.

Other forms of regression analysis, e.g. linear (van Rijsbergen 1992) or polynomial (Fuhr et al. 1993), have also been tried. In order to consider general linear models, a function which expands to the whole real line is needed. Cox (1970) gives good reasons why the logistic function is the simplest function which does this, and moreover it has some nice properties. A major benefit is that of yielding only values between 0 and 1 so there is no problem with outliers.

Arampatzis, A. (2001). Unbiased s-d threshold optimization, initial query degradation, decay, and incrementality, for adaptive document filtering. In Proceedings TREC 2001, NIST.

Arampatzis, A., & van Hameren, A. (2001). The score-distributional threshold optimization for adaptive binary classification tasks. In Proceedings SIGIR’01 (pp. 285–293). ACM Press.

Arampatzis, A., & Kamps, J. (2008). Where to stop reading a ranked list? In: Proceedings TREC 2008, NIST.

Arampatzis, A., & Kamps, J. (2009). A signal-to-noise approach to score normalization. In Proceedings CIKM (pp. 797–806). ACM Press.

Arampatzis, A., Beney, J., Koster, C. H. A., & van der Weide, T. P. (2000). Incrementality, half-life, and threshold optimization for adaptive document filtering. In Proceedings TREC 2000, NIST.

Arampatzis, A., Kamps, J., & Robertson, S. (2009). Where to stop reading a ranked list? Threshold optimization using truncated score distributions. In: Proceedings SIGIR’09 (pp. 524–531). ACM Press.

Baumgarten, C. (1999). A probabilitstic solution to the selection and fusion problem in distributed information retrieval. In Proceedings SIGIR’99 (pp 246–253). ACM Press

Bookstein, A. (1977). When the most “pertinent” document should not be retrieved—An analysis of the Swets model. Information Processing and Management 13(6), 377–383.MATHCrossRef

Callan, J. (2000). Distributed information retrieval. In Advances information retrieval: Recent research from the CIIR (ir 5, pp. 127–150). Kluwer.

Collins-Thompson, K., Ogilvie, P., Zhang, Y., & Callan, J. (2002). Information filtering, novelty detection, and named-page finding. In Proceedings TREC 2002, NIST.

Cooper, W. S. (1991). Some inconsistencies and misnomers in probabilistic information retrieval. In Proceedings SIGIR’91 (pp. 57–61). ACM Press.

Cooper, W. S., Gey, F. C., & Dabney, D. P. (1992). Probabilistic retrieval based on staged logistic regression. In Proceedings SIGIR’92 (pp. 198–210). ACM Press.

Cooper, W. S., Chen, A., & Gey, F. C. (1994). Experiments in the probabilistic retrieval of full text documents. In Proceedings TREC 1994, NIST.

Cormack, G. V., Lhoták, O., & Palmer, C. R. (1999). Estimating precision by random sampling (poster abstract). In Proceedings SIGIR’99 (pp 273–274). ACM Press.

Cox, D. R. (1970). The analysis of binary data. London: Chapman & Hall.MATH

Craswell, N., Robertson, S., Zaragoza, H., & Taylor, M. (2005). Relevance weighting for query-independent evidence. In Proceedings SIGIR’05 (pp. 416–423). ACM Press.

Fernández, M., Vallet, D., & Castells, P. (2006). Probabilistic score normalization for rank aggregation. In ECIR, Lecture notes in computer science (Vol. 3936, pp. 553–556). Springer.

Fernández, M., Vallet, D., & Castells, P. (2006). Using historical data to enhance rank aggregation. In Proceedings SIGIR’06 (pp. 643–644). ACM Press.

Fuhr, N., Pfeifer, U., Bremkamp, C., Pollmann, M., & Buckley, C. (1993). Probabilistic learning approaches for indexing and retrieval with the trec-2 collection. In Proceedings TREC 1993, NIST.

Hawking, D., & Robertson, S. (2003). On collection size and retrieval effectiveness. Information Retrieval 6(1), 99–105.CrossRef

Kamps, J., de Rijke, M., & Sigurbjörnsson, B. (2005). Combination methods for crosslingual web retrieval. In CLEF, Lecture notes in computer science (Vol. 4022, pp. 856–864). Springer.

Kanoulas, E., Pavlu, V., Dai, K., & Aslam, J. A. (2009). Modeling the score distributions of relevant and non-relevant documents. In ICTIR, Lecture notes in computer science (Vol. 5766, pp. 152–163). Springer.

Lee, J. H. (1997). Analyses of multiple evidence combination. In Proceedings SIGIR’97 (pp. 267–276). ACM Press.

Lewis, D. D. (1995). Evaluating and optimizing autonomous text classification systems. In Proceedings SIGIR’95 (pp. 246–254). ACM Press.

Manmatha, R., Rath, T. M., & Feng, F. (2001). Modeling score distributions for combining the outputs of search engines. In Proceedings SIGIR’01 (pp. 267–275). ACM Press.

Nottelmann, H., & Fuhr, N. (2003). From uncertain inference to probability of relevance for advanced IR applications. In ECIR, Lecture notes in computer science (Vol. 2633, pp. 235–250). Springer.

Oard, D. W., Hedin, B., Tomlinson, S., & Baron, J. R. (2009). Overview of the TREC 2008 legal track. In Proceedings TREC 2008, NIST.

van Rijsbergen, C. J. (1979). Information retrieval. Butterworth

van Rijsbergen, C. J. (1992). Probabilistic retrieval revisited. The Computer Journal 35(3), 291–298.MATHCrossRef

Ripley, B. D., & Hjort N. L. (1995). Pattern recognition and neural networks. New York, NY: Cambridge University Press.

Robertson, S. E. (1969). The parametric description of retrieval tests. Part 1: The basic parameters. Journal of Documentation 25(1), 1–27.CrossRef

Robertson, S. E. (1977). The probabilistic character of relevance. Information Processing Management 13(4), 247–251.CrossRef

Robertson, S. E. (2007). On score distributions and relevance. In ECIR, Lecture notes in computer science (Vol. 4425, pp. 40–51). Springer.

Robertson, S. E., & Bovey, J. D. (1982). Statistical problems in the application of probabilistic models to information retrieval. Technical report, Report No. 5739, BLR&DD

Robertson, S. E., & Walker, S. (2000). Threshold setting in adaptive filtering. Journal of Documentation 56, 312–331.CrossRef

Savoy, J. (2003). Report on CLEF-2003 multilingual tracks. In CLEF, Lecture notes in computer science (Vol. 3237, pp. 64–73). Springer.

Swets, J. A. (1963). Information retrieval systems. Science 141(3577), 245–250.CrossRef

Swets, J. A. (1969). Effectiveness of information retrieval methods. American Documentation 20, 72–89.CrossRef

Zhang, Y., & Callan, J. (2001). Maximum likelihood estimation for filtering thresholds. In Proceedings SIGIR’01 (pp. 294–302). ACM Press.

Titel: Modeling score distributions in information retrieval
verfasst von: Avi Arampatzis
Stephen Robertson
Publikationsdatum: 01.02.2011
Verlag: Springer Netherlands
Erschienen in: Discover Computing / Ausgabe 1/2011
Print ISSN: 2948-2984
Elektronische ISSN: 2948-2992
DOI: https://doi.org/10.1007/s10791-010-9145-5

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 1/2011

Retrieval constraints and word frequency distributions a log-logistic model for IR

Variational bayes for modeling score distributions

Specificity aboutness in XML retrieval

An analysis of NP-completeness in novelty and diversity ranking

Introduction to special issue on the second international conference on the theory of information retrieval