Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
What Happened in CLEF For a While? Read chapter Read first chapter

2019 | OriginalPaper | Chapter

Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm

Authors : Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Ian Soboroff

Published in: Experimental IR Meets Multilinguality, Multimodality, and Interaction

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Reproducibility has become increasingly important for many research areas, among those IR is not an exception and has started to be concerned with reproducibility and its impact on research results. This paper describes our second attempt to propose a lab on reproducibility named CENTRE, held during CLEF 2019. The aim of CENTRE is to run both a replicability and reproducibility challenge across all the major IR evaluation campaigns and to provide the IR community with a venue where previous research results can be explored and discussed. This paper reports the participant results and preliminary considerations on the second edition of CENTRE@CLEF 2019.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter To Check or Not to Check: Syntax, Semantics, and Context in the Language of Check-Worthy Claims
next chapter Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims
Footnotes
Literature
1.
Allan, J., et al.: Research frontiers in information retrieval – report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018). In: SIGIR Forum, vol. 52, no. 1, June 2018
2.
Allan, J., Harman, D.K., Kanoulas, E., Li, D., Van Gysel, C., Voorhees, E.M.: TREC 2017 common core track overview. In: Voorhees and Ellis [28]
3.
Allan, J., Harman, D.K., Kanoulas, E., Li, D., Van Gysel, C., Voorhees, E.M.: TREC 2018 common core track overview. In: Voorhees and Ellis [29]
4.
Arguello, J., Crane, M., Diaz, F., Lin, J., Trotman, A.: Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). In: SIGIR Forum, vol. 49, no. 2, pp. 107–116 (2015)CrossRef
5.
Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Has adhoc retrieval improved since 1994? In: Allan, J., Aslam, J.A., Sanderson, M., Zhai, C., Zobel, J. (eds.) Proceedings of 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), pp. 692–693. ACM Press, New York (2009)
6.
Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Cheung, D.W.L., Song, I.Y., Chu, W.W., Hu, X., Lin, J.J. (eds.) Proceedings of 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)
7.
Benham, R., et al.: RMIT at the 2018 TREC CORE Track. In: Voorhees and Ellis [29]
8.
Breuer, T., Schaer, P.: Replicability and reproducibility of automatic routing runs. In: Cappellato et al. [9]
9.
Cappellato, L., Ferro, N., Losada, D., Müller, H. (eds.): CLEF 2019 Working Notes, CEUR Workshop Proceedings. CEUR-WS.org (2019)
10.
Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): CLEF 2018 Working Notes, CEUR Workshop Proceedings. CEUR-WS.org, ISSN 1613–0073 (2018)
11.
Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., Wu, Z.Z.: The SIGIR 2019 open-source IR replicability challenge (OSIRRC 2019). In: Chevalier, M., Gaussier, É., Piwowarski, B., Maarek, Y., Nie, J.Y., Scholer, F. (eds.) Proceedings of 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019) (2019)
12.
Ferro, N., et al.: The Dagstuhl perspectives workshop on performance modeling and prediction. In: SIGIR Forum, vol. 52, no. 1, June 2018CrossRef
13.
Ferro, N., Fuhr, N., Järvelin, K., Kando, N., Lippold, M., Zobel, J.: Increasing reproducibility in IR: findings from the Dagstuhl seminar on “reproducibility of data-oriented experiments in e-science”. In: SIGIR Forum, vol. 50, no. 1, pp. 68–82 (2016)CrossRef
14.
Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2019: overview of the replicability and reproducibility tasks. In: Cappellato et al. [9]
15.
Ferro, N., Kelly, D.: SIGIR initiative to implement ACM artifact review and badging. In: SIGIR Forum, vol. 52, no. 1, June 2018CrossRef
16.
Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2018: overview of the replicability task. In: Cappellato et al. [10]
17.
18.
Freire, J., Fuhr, N., Rauber, A. (eds.): Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. Dagstuhl Reports, vol. 6, no. 1. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany (2016)
19.
Fuhr, N.: Some common mistakes in IR evaluation, and how they can be avoided. In: SIGIR Forum vol. 51, no. 3, pp. 32–41 (2017)CrossRef
20.
Grossman, M.R., Cormack, G.V.: MRG\(\_\)UWaterloo and WaterlooCormack participation in the TREC 2017 common core track. In: Voorhees and Ellis [28]
21.
Jungwirth, M., Hanbury, A.: Replicating an experiment in cross-lingual information retrieval with explicit semantic analysis. In: Cappellato et al. [10]
22.
Kendall, M.G.: Rank Correlation Methods. Griffin, Oxford (1948)MATH
23.
Kenney, J.F., Keeping, E.S.: Mathematics of Statistics - Part One, 3rd edn. D. Van Nostrand Company, Princeton (1954)MATH
24.
Kharazmi, S., Scholer, F., Vallet, D., Sanderson, M.: Examining additivity and weak baselines. ACM Trans. Inf. Syst. (TOIS) 34(4), 23:1–23:18 (2016)CrossRef
25.
26.
Munafò, M.R., et al.: A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021:1–0021:9 (2017)CrossRef
27.
Sakai, T., Ferro, N., Soboroff, I., Zeng, Z., Xiao, P., Maistro, M.: Overview of the NTCIR-14 CENTRE task. In: Ishita, E., Kando, N., Kato, M.P., Liu, Y. (eds.) Proceedings of 14th NTCIR Conference on Evaluation of Information Access Technologies, pp. 494–509. National Institute of Informatics, Tokyo (2019)
28.
Voorhees, E.M., Ellis, A. (eds.): The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). National Institute of Standards and Technology (NIST), Special Publication 500–324, Washington, USA (2018)
29.
Voorhees, E.M., Ellis, A. (eds.): The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018). National Institute of Standards and Technology (NIST), Washington, USA (2019)
30.
Zobel, J., Webber, W., Sanderson, M., Moffat, A.: Principles for robust evaluation infrastructure. In: Agosti, M., Ferro, N., Thanos, C. (eds.) Proceedings of Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011), pp. 3–6. ACM Press, New York (2011)
Metadata
Title
Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm
Authors
Nicola Ferro
Norbert Fuhr
Maria Maistro
Tetsuya Sakai
Ian Soboroff
Copyright Year
2019
Book
Experimental IR Meets Multilinguality, Multimodality, and Interaction

Print ISBN: 978-3-030-28576-0

Electronic ISBN: 978-3-030-28577-7

Copyright Year: 2019

DOI
https://doi.org/10.1007/978-3-030-28577-7_24

Premium Partner

    Image Credits