Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis Read chapter Read first chapter

2020 | OriginalPaper | Chapter

A Study on a Stopping Strategy for Systematic Reviews Based on a Distributed Effort Approach

Author : Giorgio Maria Di Nunzio

Published in: Experimental IR Meets Multilinguality, Multimodality, and Interaction

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Systematic reviews are scientific investigations that use strategies to include a comprehensive search of all potentially relevant articles and the use of explicit, reproducible criteria in the selection of articles for review. As time and resources are limited for compiling a systematic review, limits to the search are needed. In this paper, we describe the stopping strategy that we have been designed and refined over three years of participation to the CLEF eHealth Technology Assisted Review Task. In particular, we present a comparison of a Continuous Active Learning approach that uses either a fixed amount or a variable amount of resources according to the size of the pool. The results show that our approach performs on average much better than any other participant in the CLEF 2019 eHealth TAR task. Nevertheless, a failure analysis allows to understand the weak points of this approach and possible future directions.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Protest Event Detection: When Task-Specific Models Outperform an Event-Driven Method
next chapter Fact Check-Worthiness Detection with Contrastive Ranking
Footnotes
Literature
1.
Cook, D.J., Mulrow, C.D., Haynes, R.B.: Systematic reviews: synthesis of best evidence for clinical decisions. Ann. Intern. Med. 126(5), 376–380 (1997)CrossRef
2.
Cormack, G.V., Grossman, M.R.: Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. In: Geva, S., Trotman, A., Bruza, P., Clarke, C.L., Järvelin, K. (eds.) The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014, Gold Coast , QLD, Australia, 06–11 July 2014, pp. 153–162. ACM (2014)
3.
4.
Gopalakrishnan, S., Ganeshkumar, P.: Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med. Primary Care 2(1), 9–14 (2013)CrossRef
5.
Grossman, M.R., Cormack, G.V., Roegiest, A.: TREC 2016 total recall track overview. In: Proceedings of The Twenty-Fifth TREC 2016, Gaithersburg, Maryland, USA, 15–18 November 2016 (2016)
6.
7.
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: Cappellato, L., Ferro, N., Goeuriot, L., Mandl, T. (eds.) CEUR Workshop Proceedings of Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, vol. 1866, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017)
8.
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2017 technologically assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings of Working Notes of CLEF 2017 - Conference and Labs of the Evaluation Forum, Dublin, Ireland, 11–14 September 2017. CEUR-WS.org (2017)
9.
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2018 technologically assisted reviews in empirical medicine overview. In: Cappellato, L., Ferro, N., Nie, J.-Y., Soulier, L., eds. CEUR Workshop Proceedings of Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018, vol. 2125. CEUR-WS.org (2018)
10.
Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: CLEF 2019 technology assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings of CLEF 2019 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS.org (2019)
11.
Kastner, M., Straus, S., Goldsmith, C.H.: Estimating the horizon of articles to decide when to stop searching in systematic reviews: an example using a systematic review of RCTS evaluating osteoporosis clinical decision support tools. In: Proceedings of AMIA Annual Symposium. AMIA Symposium, vol. 2007, no. 10, pp. 389–393 (2007)
12.
13.
Li, D., Kanoulas, E.: Automatic thresholding by sampling documents and estimating recall. In: Cappellato, L., Ferro, N., Losada, D.E., Müller, H. (eds.) CEUR Workshop Proceedings of Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, 9–12 September 2019, vol. 2380. CEUR-WS.org (2019)
14.
Nunzio, G.M.D.: A new decision to take for cost-sensitive Naïve Bayes classifiers. Inf. Process. Manag. 50(5), 653–674 (2014)CrossRef
15.
Nunzio, G.M.D.: Interactive text categorisation: the geometry of likelihood spaces. Stud. Comput. Intell. 668, 13–34 (2017)
16.
Di Nunzio, G.M.: A study of an automatic stopping strategy for technologically assisted medical reviews. In: Proceedings of Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Grenoble, France , 26–29 March 2018, pp. 672–677 (2018)
17.
Di Nunzio, G.M., Ciuffreda, G., Vezzani, F.: Interactive sampling for systematic reviews. IMS Unipd at CLEF 2018 ehealth task 2. In: Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, 10–14 September 2018 (2018)
18.
Di Nunzio, G.M., Maistro, M., Vezzani, F.: A gamified approach to Naïve Bayes classification: a case study for newswires and systematic medical reviews. In: Companion of the Web Conference 2018 on the Web Conference 2018, WWW 2018, Lyon, France, 23–27 April 2018, pp. 1139–1146 (2018)
19.
Di Nunzio, G.M., Maistro, M., Zilio, D.: Gamification for machine learning: the classification game. In: Proceedings of the Third International Workshop on Gamification for Information Retrieval co-located with 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016), pp. 45–52, Pisa, Italy, 21 July 2016 (2016)
20.
Di Nunzio, G.M., Maistro, M., Zilio, D.: The University of Padua (IMS) at TREC 2016 total recall track. In: Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, 15–18 November 2016 (2016)
21.
Robertson, S.E.: The Probability Ranking Principle in IR, pp. 281–286. Morgan Kaufmann Publishers Inc., San Francisco (1997)
22.
Roegiest, A., Cormack, G.V., Grossman, M.R., Clarke, C.L.A.: TREC 2015 total recall track overview. In: Proceedings of The Twenty-fourth TREC 2015, Gaithersburg, Maryland, USA, 17–20 November 2015 (2015)
23.
Scells, H., Zuccon, G., Koopman, B., Clark, J.: Automatic boolean query formulation for systematic review literature search. In: Huang, Y., King, I., Liu, T.-Y., van Steen, M. (eds.) WWW 2020: The Web Conference 2020, Taipei, Taiwan, 20–24 April 2020, pp. 1071–1081. ACM/IW3C2 (2020)
24.
Williams, V., Boylan, A.-M., Nunan, D.: Qualitative research as evidence: expanding the paradigm for evidence-based healthcare. BMJ Evid.-Based Med. 24(5), 168–169 (2019)CrossRef
25.
Xiang, Y.-T., et al.: Timely research papers about COVID-19 in China. Lancet 395(10225), 684–685 (2020)CrossRef
26.
Zhang, H., Abualsaud, M., Ghelani, N., Smucker, M.D., Cormack, G.V., Grossman, M.R.: Effective user interaction for high-recall retrieval: less is more. In: Cuzzocrea, A., (eds.) Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, 22–26 October 2018, pp. 187–196. ACM (2018)
27.
Zou, J., Li, D., Kanoulas, E.: Technology assisted reviews: Finding the last few relevant documents by asking yes/no questions to reviewers. In: Thompson, K.C., Mei, Q., Brian, D.D., Liu, Y., Yilmaz, E. (eds.) The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, 08–12 July 2018, pp. 949–952. ACM (2018)
Metadata
Title
A Study on a Stopping Strategy for Systematic Reviews Based on a Distributed Effort Approach
Author
Giorgio Maria Di Nunzio
Copyright Year
2020
Book
Experimental IR Meets Multilinguality, Multimodality, and Interaction

Print ISBN: 978-3-030-58218-0

Electronic ISBN: 978-3-030-58219-7

Copyright Year: 2020

DOI
https://doi.org/10.1007/978-3-030-58219-7_10

Premium Partner

    Image Credits