nach oben

Discover Computing

Erschienen in:

25.05.2021

Pseudo relevance feedback optimization

verfasst von: Avi Arampatzis, Georgios Peikos, Symeon Symeonidis

Erschienen in: Discover Computing | Ausgabe 4-5/2021

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

We propose a method for automatic optimization of pseudo relevance feedback (PRF) in information retrieval. Based on the conjecture that the initial query’s contribution to the final query may not be necessary once a good model is built from pseudo relevant documents, we set out to optimize per query only the number of top-retrieved documents to be used for feedback. The optimization is based on several query performance predictors for the initial query, by building a linear regression model discovering the optimal machine learning pipeline via genetic programming. Even by using only 50–100 training queries, the method yields statistically-significant improvements in MAP of 18–35% over the initial query, 7–11% over the feedback model with the best fixed number of pseudo-relevant documents, and up to 10% (5.5% on median) over the standard method of optimizing both the balance coefficient and the number of feedback documents by grid-search in the training set. Compared to state-of-the-art PRF methods from the recent literature, our method outperforms by up to 21% with an average of 10%. Further analysis shows that we are still far from the method’s effectiveness ceiling (in contrast to the standard method), leaving amble room for further improvements.

Nächster Artikel Using word semantic concepts for plagiarism detection in text documents

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nevertheless, this may not be an effective strategy when the set of pseudo-relevant gets too small. When the amount of training data is not sufficient, perhaps a larger but ‘dirtier’ set is preferable to a smaller but ‘cleaner’ one.

For example, PRF in the Terrier search engine defaults at \(\alpha = 0.6\) and \(K=3\).

Even though a perfect \(Q_{r,K}\) could potentially encapsulate all \(Q_0\)’s information, it may also contain additional information that is not necessary to the information need. This may also be the case for \(Q_0\), depending on how well the user has expressed the information need, so a (large) contribution of \(Q_0\) may still not have a desirable effect.

Although it is risky to draw conclusions on non-significant correlations, the widespread negative co-efficients in Table 3 are worrisome. They seem to suggest a flaw in our argument at the beginning of Section 3 (i.e. the better the \(Q_0\)’s effectiveness, the larger the \(K_{\mathrm {opt\_MAP}}\)), but there is no flaw. We will investigate and discuss this in Section 5.1.

http://epistasislab.github.io/tpot

From a theoretical point of view, the MAE is not the right forecast error to be measured/minimized here for the problem at hand. Nevertheless, in practice, MAE has worked better than alternatives. We will elaborate on this in Section 5.2.

https://trec.nist.gov

http://terrier.org

These are approximate (but very close) numbers, not accounting for the excluded topic 368 whenever this is missing from a test set of a split.

We tried a higher resolution for K (see lines at \(\alpha =0\) in Table 8 in Sect. 4.2.2), and \(K=10\) was still the best overall in MAP. While Table 8 seems like it shows MAP in the training sets of splits, we remind that SPLT1’s training set is SPLT4’s test set, and so on.

These percentage improvements are resulting from Table 6, but not shown in the table due to the limited space.

Again, not accounting for the excluded topic 368, whenever this occurs in a training set.

There are 116 out of the 150 queries (77.3%) with \(K_{\mathrm {opt\_MAP}}\le R\) in our dataset. Also note that the Rs are not the real ones but under-estimated—in various degrees—by TREC’s pooling process.

Abdelmgeid Amin, A. (2008). Using a query expansion technique to improve document retrieval. International Journal Information Technologies and Knowledge, 2(4), 343–348.

Amati, G., & van Rijsbergen, C. J. (2002). Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS), 20(4), 357–389.CrossRef

Amati, G., Carpineto, C., & Romano, G. (2004). Query difficulty, robustness, and selective application of query expansion. In Advances in Information Retrieval, 26th European Conference on IR Research, ECIR 2004, Proceedings, Springer, Lecture Notes in Computer Science, 2997, pp 127–137

Arampatzis, A. (2001). Unbiased S-D threshold optimization, initial query degradation, decay, and incrementality, for adaptive document filtering. In Proceedings of The Tenth Text REtrieval Conference, TREC 2001, National Institute of Standards and Technology (NIST), 250, pp 596–603.

Arampatzis, A., & Robertson, S. (2011). Modeling score distributions in information retrieval. Information Retrieval, 14(1), 26–46.CrossRef

Arampatzis, A., Beney, J., Koster, CHA., & van der Weide, TP. (2000). Incrementality, half-life, and threshold optimization for adaptive document filtering. In Proceedings of The Ninth Text REtrieval Conference, TREC 2000, National Institute of Standards and Technology (NIST), NIST Special Publication, 249.

Arampatzis, A., Kamps, J., & Robertson, S. (2009). Where to stop reading a ranked list?: threshold optimization using truncated score distributions. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, ACM, pp 524–531.

Azad, H. K., & Deepak, A. (2019). Query expansion techniques for information retrieval: A survey. Information Processing & Management, 56(5), 1698–1735.CrossRef

Croft, W. B., & Harper, D. J. (1979). Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35(4), 285–295.CrossRef

Cronen-Townsend, S., Zhou, Y., & Croft, WB. (2002). Predicting query performance. In: SIGIR 2002: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 299–306.

Crouch, C. J., Crouch, D. B., Chen, Q., & Holtz, S. J. (2002). Improving the retrieval effectiveness of very short queries. Information Processing & Management, 38(1), 1–36.CrossRef

Hauff, C. (2010). Predicting the effectiveness of queries and retrieval systems. SIGIR Forum, 44(1), 88.CrossRef

Jaleel, NA., Allan, J., Croft, WB., Diaz, F., Larkey, LS., Li, X., Smucker, MD., & Wade, C. (2004). Umass at TREC 2004: Novelty and HARD. In Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, National Institute of Standards and Technology (NIST), NIST Special Publication, vol 500-261.

Karisani, P., Rahgozar, M., & Oroumchian, F. (2016). A query term re-weighting approach using document similarity. Information Processing & Management, 52(3), 478–489.CrossRef

Kekäläinen, J., & Järvelin, K. (1998). The impact of query structure and query expansion on retrieval performance. In SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 130–137.

Lavrenko, V., & Croft, WB. (2001). Relevance-based language models. In SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 120–127.

Lv, Y., & Zhai, C. (2009a). Adaptive relevance feedback in information retrieval. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, ACM, pp 255–264.

Lv, Y., & Zhai, C. (2009b). A comparative study of methods for estimating query language models with pseudo feedback. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, ACM, pp 1895–1898.

Lv, Y., & Zhai, C. (2010). Positional relevance model for pseudo-relevance feedback. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, ACM, pp 579–586.

Lv, Y., & Zhai, C. (2014). Revisiting the divergence minimization feedback model. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, ACM, pp 1863–1866.

Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press.

Markovits, G., Shtok, A., Kurland, O., & Carmel, D. (2012). Predicting query performance for fusion-based retrieval. In 21st ACM International Conference on Information and Knowledge Management, CIKM’12, ACM, pp 813–822.

Mitra, M., Singhal, A., & Buckley, C. (1998). Improving automatic query expansion. In SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 206–214.

Olson, RS., Bartley, N., Urbanowicz, RJ., & Moore, JH. (2016). Evaluation of a tree-based pipeline optimization tool for automating data science. In Proceedings of the 2016 on Genetic and Evolutionary Computation Conference, ACM, pp 485–492.

Parapar, J., & Barreiro, A. (2011). Promoting divergent terms in the estimation of relevance models. In Advances in Information Retrieval Theory - Third International Conference, ICTIR 2011. Proceedings, Springer, Lecture Notes in Computer Science, vol 6931, pp 77–88.

Parapar, J., Quindimil, M. A. P., & Barreiro, A. (2014). Score distributions for pseudo relevance feedback. Information Sciences, 273, 171–181.CrossRef

Raiber, F., & Kurland, O. (2014). Query-performance prediction: setting the expectations straight. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’14, ACM, pp 13–22.

Rocchio, JJ. (1971). Relevance feedback in information retrieval. In The SMART Retrieval System - Experiments in Automatic Document Processing, Prentice Hall, pp 313–323.

Rosipal, R., Girolami, M. A., Trejo, L. J., & Cichocki, A. (2001). Kernel PCA for feature extraction and de-noising in nonlinear regression. Neural Computing & Applications, 10(3), 231–243.CrossRef

Ruthven, I., & Lalmas, M. (2003). A survey on the use of relevance feedback for information access systems. Knowledge Engineering Review, 18(2), 95–145.CrossRef

Sakai, T., Manabe, T., & Koyama, M. (2005). Flexible pseudo-relevance feedback via selective sampling. ACM Transactions on Asian Language Information Processing (TALIP), 4(2), 111–135.CrossRef

Salton, G. (1971). The SMART Retrieval System-Experiments in Automatic Document Processing. USA: Prentice-Hall Inc.

Shtok, A., Kurland, O., Carmel, D., Raiber, F., & Markovits, G. (2012). Predicting query performance by query-drift estimation. ACM Transactions on Information Systems (TOIS) 30(2):11:1–11:35.

Sihvonen, A., & Vakkari, P. (2004). Subject knowledge improves interactive query expansion assisted by a thesaurus. Journal of Documentation, 60(6), 673–690.CrossRef

Singh, J., Prasad, M., Daraghmi, Y., Tiwari, P., Yadav, P., Bharill, N., Pratama, M., & Saxena A. (2017). Fuzzy logic hybrid model with semantic filtering approach for pseudo relevance feedback-based query expansion. In 2017 IEEE Symposium Series on Computational Intelligence, SSCI 2017, IEEE, pp 1–7.

Tao, T., & Zhai, C. (2006). Regularized estimation of mixture models for robust pseudo-relevance feedback. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 162–169.

Tao, Y., & Wu, S. (2014). Query performance prediction by considering score magnitude and variance together. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, ACM, pp 1891–1894.

Tofallis, C. (2015). A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society, 66(8), 1352–1362.CrossRef

Valcarce, D., Parapar, J., & Barreiro, Á. (2018). Lime: linear methods for pseudo-relevance feedback. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018, Pau, ACM, pp 678–687.

Voorhees, EM., & Harman, D. (1999). Overview of the eighth text retrieval conference (TREC-8). In Proceedings of The Eighth Text Retrieval Conference, TREC 1999, National Institute of Standards and Technology (NIST), vol Special Publication 246.

Xu, J., & Croft, WB. (1996). Query expansion using local and global document analysis. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 4–11.

Yom-Tov, E., Fine, S., Carmel, D., & Darlow, A. (2005). Metasearch and federation using query difficulty prediction. In Proceedings of the ACM SIGIR Workshop on Predicting Query Difficulty.

Zhou, Y. (2008). Retrieval performance prediction and document quality. PhD thesis, University of Massachusetts Amherst.

Zhou, Y., & Croft, WB. (2007). Query performance prediction in web search environments. In: SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp 543–550.

Titel: Pseudo relevance feedback optimization
verfasst von: Avi Arampatzis
Georgios Peikos
Symeon Symeonidis
Publikationsdatum: 25.05.2021
Verlag: Springer Netherlands
Erschienen in: Discover Computing / Ausgabe 4-5/2021
Print ISSN: 2948-2984
Elektronische ISSN: 2948-2992
DOI: https://doi.org/10.1007/s10791-021-09393-5

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner