nach oben

Erschienen in:

03.05.2018

An analysis of evaluation campaigns in ad-hoc medical information retrieval: CLEF eHealth 2013 and 2014

verfasst von: Lorraine Goeuriot, Gareth J. F. Jones, Liadh Kelly, Johannes Leveling, Mihai Lupu, Joao Palotti, Guido Zuccon

Erschienen in: Discover Computing | Ausgabe 6/2018

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Since its inception in 2013, one of the key contributions of the CLEF eHealth evaluation campaign has been the organization of an ad-hoc information retrieval (IR) benchmarking task. This IR task evaluates systems intended to support laypeople searching for and understanding health information. Each year the task provides registered participants with standard IR test collections consisting of a document collection and topic set. Participants then return retrieval results obtained by their IR systems for each query, which are assessed using a pooling procedure. In this article we focus on CLEF eHealth 2013 and 2014s retrieval task, which saw topics created based on patients’ information needs associated with their medical discharge summaries. We overview the task and datasets created, and the results obtained by participating teams over these two years. We then provide a detailed comparative analysis of the results, and conduct an evaluation of the datasets in the light of these results. This twofold study of the evaluation campaign teaches us about technical aspects of medical IR, such as the effectiveness of query expansion; the quality and characteristics of CLEF eHealth IR datasets, such as their reliability; and how to run an IR evaluation campaign in the medical domain.

Vorheriger Artikel Linear feature extraction for ranking

Nächster Artikel (CF)2 architecture: contextual collaborative filtering

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

http://clef-ehealth.org/.

http://www.clef-initiative.eu/.

http://www.trec-cds.org/.

Subjects were asked to describe the picture as if it were their own health issue. See Palotti et al. (2015) for details.

http://trec.nist.gov/data/robust.html.

http://trec.nist.gov/.

http://mimic.physionet.org/.

A query here is text typed in a search engine. A topic is an enriched query.

Runs are described in the section that analyzes participants’ retrieval results.

A single definition is provided here for example purposes. In reality an exhaustive list of definitions is provided for gathered terminologies.

Further details on baselines used are provided in the Task overview papers Goeuriot et al. (2013a, 2014c).

Alsulmi, M. & Carterette, B. (2016). Improving clinical case search using semantic based query reformulations. In 2016 IEEE international conference on bioinformatics and biomedicine (BIBM) (pp. 694–698). IEEE.

Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009). Improvements that don’t add up: Ad-hoc retrieval results since 1998. In CIKM 2009 (pp. 601–610). ACM.

Barajas, K. C., & Akella, R. (2013). Incorporating statistical topic models in the retrieval of health care documents. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Bedrick, S., & Sheikhshabbafghi, G. (2013). Lucene, MetaMap, and language modeling: OHSU at CLEF eHealth 2013. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Benigeri, M., & Pluye, P. (2003). Shortcomings of health information on the internet. Health Promotion International, 18(4), 381386.CrossRef

Chappell, T., & Geva, S. (2013). Working notes for TopSig at ShARe/CLEF eHealth 2013. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Choi, S., & Choi, J. (2013). SNUMedinfo at CLEFeHealth2013 task 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Choi, S., & Choi, J. (2014). Exploring effective information retrieval technique for the medical web documents: SNUMedinfo at CLEFeHealth2014 Task 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Claveau, V. (2012). Unsupervised and semi-supervised morphological analysis for information retrieval in the biomedical domain. In COLING, 2012 (pp. 629–645).

Claveau, V., Hamon, T., Grabar, N., & Le Maguer, S. (2014). RePaLi participation to CLEF eHealth IR challenge 2014: Leveraging term variation. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Demner-Fushman, D., & Lin, J. (2007). Answering clinical questions with knowledge-based and statistical techniques. Computational Linguistics, 33(1), 63–103.CrossRef

Dramé, K., Mougin, F., & Diallo, G. (2014). Query expansion using external resources for improving information retrieval in the biomedical domain. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Goeuriot, L., Chapman, W., Jones, G. J. F., Kelly, L., Leveling, J., & Salanterä, S. (2014a). Building realistic potential patients queries for medical information retrieval evaluation. In Proceedings of the LREC workshop on building and evaluating resources for health and biomedical text processing.

Goeuriot, L., Jones, G. J. F., Kelly, L., Leveling, J., Hanbury, A., Müller, H., et al. (2013a). ShARe/CLEF eHealth Evaluation Lab 2013, task 3: Information retrieval to address patients’ questions when reading clinical reports. In CLEF online working notes.

Goeuriot, L., Kelly, L., & Leveling, J. (2014b). An analysis of query difficulty for information retrieval in the medical domain. In Proceedings of the ACM special interest group on information retrieval conference (SIGIR 2014).

Goeuriot, L., Kelly, L., Hanlen, L., Suominen, H., Névéol, A., Palotti, J., et al. (2015). Overview of the CLEF eHealth Evaluation Lab 2015. In Proceedings of CLEF 2015.

Goeuriot, L., Kelly, L., Jones, G. J. F., Zuccon, G., Suominen, H., Hanbury, A., et al. (2013b). Creation of a new evaluation benchmark for information retrieval targeting patient information needs. In R. Song, W. Webber, N. Kando, & K. Kishida (Eds.), Proceedings of the 5th international workshop on evaluating information access (EVIA), a satellite workshop of the NTCIR-10 conference, Tokyo/Fukuoka, Japan, National Institute of Informatics/Kijima Printing.

Goeuriot, L., Kelly, L., Li, W., Palotti, J., Pecina, P., Zuccon, G., et al. (2014c). ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In CLEF online working notes.

Hanbury, A., & Müller, H. (2012). Khresmoi—Multimodal multilingual medical information search. In Proceedings of medical informatics Europe 2012 (MIE 2012), Village of the Future.

Hansen, D. L., Derry, H. A., Resnick, P. J., & Richardson, C. R. (2003). Adolescents searching for health information on the internet: An observational study. Journal of Medical Internet Research, 5(4), e25. https://doi.org/10.2196/jmir.5.4.e25.CrossRef

Harman, D., & Buckley, C. (2004). The NRRC reliable information access (RIA) workshop. In SIGIR 2004 (pp. 528–529). ACM.

Hauff, C., de Jong, F., Kelly, D., & Azzopardi, L. (2010). Query quality: User ratings and system predictions. In SIGIR ’10 (pp. 743–744). New York, NY: ACM.

He, B., & Ounis, I. (2006). Query performance prediction. Information Systems, 31(7), 585–594.CrossRef

Hersh, W. R., Buckley, C., Leone, T.J., & Hickam, D. H. (1994). OHSUMED: An interactive retrieval evaluation and new large test collection for research. In Proceedings of SIGIR ’94, (pp. 192–201).CrossRef

Kalpathy-Cramer, J., Müller, H., Bedrick, S., Eggel, I., de Herrera, A. G. S., & Tsikrika, T. (2011). The CLEF 2011 medical image retrieval and classification tasks. In Working notes of CLEF 2011 (Cross Language Evaluation Forum).

Kelly, L., Goeuriot, L., Suominen, H., Névéol, A., Palotti, J., & Zuccon, G. (2016). Overview of the CLEF eHealth Evaluation Lab 2016. In Proceedings of CLEF 2016.

Kelly, L., Goeuriot, L., Suominen, H., Schreck, T., Leroy, G., Mowery, D. L., et al. (2014). Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In Proceedings of CLEF 2014.

Koopman, B., & Zuccon, G. (2014). Why assessing relevance in medical IR is demanding. In Proceedings of medical information retrieval (MedIR) workshop (SIGIR).

Koopman, B., & Zuccon, G. (2016). A test collection for matching patients to clinical trials. In Proceedings of the 39th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’16 (pp. 669–672). New York, NY: ACM.

Koopman, B., Zuccon, G., Bruza, P., Sitbon, L., & Lawley, M. (2012). An evaluation of corpus-driven measures of medical concept similarity for information retrieval. In Proceedings of CIKM 2012.

Koopman, B., Zuccon, G., Bruza, P., Sitbon, L., & Lawley, M. (2016). Information retrieval as semantic inference: A graph inference model applied to medical search. Information Retrieval Journal, 19(1–2), 6–37.CrossRef

Ksentini, N., Tmar, M., & Gargouri, F. (2014). Miracl at CLEF 2014: eHealth information retrieval task. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Leveling, J., Goeuriot, L., Kelly, L., & Jones, G. J. F. (2012). DCU@TRECMed 2012: Using ad-hoc baselines for domain-specific retrieval. In Proceedings of TREC 2012. NIST.

Limsopatham, N., Macdonald, C., & Ounis, I. (2013a). Inferring conceptual relationships to improve medical records search. In Proceedings of the 10th conference on open research areas in information retrieval (pp. 1–8).

Limsopatham, N., Macdonald, C., & Ounis, I. (2013b). University of Glasgow at CLEF 2013: Experiments in eHealth Task 3 with Terrier. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Lin, J. (2005). Evaluation of resources for question answering evaluation. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2005) (pp. 392–399).

Liu, X., Nie, J.-Y., & Sordoni, A. (2016). Constraining word embeddings by prior knowledge—Application to medical information retrieval. In Asia information retrieval symposium (pp. 155–167). Springer.

Malagon, J. M. C., & López, M. M. (2014). Laberinto at ShARe/CLEF eHealth Evaluation Lab. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Müller, H., Clough, P., Deselaers, T., & Caputo, B. (Eds.). (2016). ImageCLEF—Experimental evaluation in visual information retrieval (Vol. 32)., The information retrieval series Berlin: Springer.MATH

Oh, H.-S., & Jung, Y. (2014). A multiple-stage approach to re-ranking clinical documents. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Ozturkmenoglu, O., Alpkocak, A., & Kilinc, D. (2014). Demir at CLEF eHealth: The effects of selective query expansion to information retrieval. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Palotti, J., Zuccon, G., Bernhardt, J., Hanbury, A., & Goeuriot, L. (2016). Assessors agreement: A case study across assessor type, payment levels, query variations and relevance dimensions. In International conference of the cross-language evaluation forum for European languages (pp. 40–53). Springer.

Palotti, J., Zuccon, G., Goeuriot, L., Kelly, L., Hanbury, A., Jones, G. J., et al. (2015). CLEF eHealth evaluation lab 2015, task 2: Retrieving information about medical symptoms. In Proceedings of CLEF eHealth Evaluation Lab.

Pecina, P., Dušek, O., Goeuriot, L., Haji, J., Hlaváčová, J., Jones, G. J. F., et al. (2014). Adaptation of machine translation for multilingual information retrieval in the medical domain. Journal of Artificial Intelligence in Medicine, Special Issue on Health Document Text Mining and Information, 61(3), 165–185.

Roberts, K., Simpson, M. S., Voorhees, E., & Hersh, W. R. (2015). Overview of the TREC 2015 clinical decision support track. In Proceedings of TREC.

Roberts, P. M., Cohen, A. M., & Hersh, W. R. (2009). Tasks, topics and relevance judging for the TREC Genomics Track: Five years of experience evaluating biomedical text information retrieval systems. Information Retrieval, 12, 81–97.CrossRef

Robertson, S. E., & Jones, K. S. (1994). Simple, proven approaches to text retrieval. Technical report 356, University of Cambridge.

Sakai, T., & Mitamura, T. (2010). Boiling down information retrieval test collections. In RIAO 2010 (pp. 49–56). CID.

Saleh, S., & Pecina, P. (2014). Cuni at the ShARe/CLEF eHealth Evaluation Lab 2014. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Salton, G., Wong, A., & Yang, C.-S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620.CrossRef

Shen, W., & Nie, J.-Y. (2015). Is concept mapping useful for biomedical information retrieval? In International conference of the cross-language evaluation forum for European languages (pp. 281–286). Springer.

Shen, W., Nie, J.-Y., Liu, X., & Liui, X. (2014). An investigation of the effectiveness of concept-based approach in medical information retrieval GRIUM@CLEF2014eHealthTask 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Simpson, M. S., Voorhees, E. M., & Hersh, W. (2014). Overview of the TREC 2014 clinical decision support track. In Proceedings of TREC.

Soboroff, I. (2009). A guide to the ria workshop data archive. Information Retrieval, 12(6), 642–651.CrossRef

Soldaini, L., Cohan, A., Yates, A., Goharian, N., & Frieder, O. (2015). Retrieving medical literature for clinical decision support. In European conference on information retrieval (pp. 538–549). Springer.

Suominen, H., Salanterä, S., Velupillai, S., Chapman, W. W., Savova, G., Elhadad, N., et al. (2013). ShARe/CLEF eHealth Evaluation Lab 2013: Three shared tasks on natural language processing and machine learning to make clinical reports easier to understand for patients. In CLEF 2013. Lecture notes in computer science (LNCS). Springer.

Thakkar, H., Iyer, G., Shah, K., & Majumder, P. (2014). Team IRLabDAIICT at ShARe/CLEF eHealth 2014 Task 3: User-centered information retrieval system for clinical documents. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Thesprasith, O., & Jaruskulchai, C. (2014). CSKU GPRF-QE for medical topic web retrieval. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Tibi, O., Thuma, E., & Mosweunyane, G. (2017). Selective collection enrichment in user-centred health information retrieval. In 2017 1st international conference on next generation computing applications (NextComp) (pp. 175–181). IEEE.

Urbano, J., Marrero, M., & Martín, D. (2013). On the measurement of test collection reliability. In SIGIR ’13 (pp. 393–402). ACM.

Verberne, S. (2014). A language-modelling approach to user-centred health information retrieval. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Voorhees, E. M., & Hersh, W. (2012). Overview of the TREC 2012 medical records track. In TREC 2012. NIST.

Voorhees, E. M., & Tong, R. M. (2011). Overview of the TREC 2011 medical records track. In Proceedings of TREC. NIST.

Voorhees, E. M. (2005). The TREC robust retrieval track. SIGIR Forum, 39(1), 11–20.CrossRef

White, R., & Horvitz, E. (2008). Cyberchondria: Studies of the escalation of medical concerns in web search. Technical report, Microsoft Research.

Wu, J., & Huang, J. (2014). York University at CLEF eHealth 2014: A learning-to-rank approach for medical document retrieval. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Xia, Y., Xie, Z., Zhang, Q., Wang, H., & Zhao, H. (2014). Cannabis_TREATS_cancer: Incorporating fine-grained ontological relations in medical document ranking. In Natural language processing and Chinese computing (pp. 275–285). Springer.

Yang, C., Bhattacharya, S., & Srinivasan, P. (2014). The University of Iowa at CLEF 2014: eHealth Task 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Zhang, Y., Cohen, T., Jiang, M., Tang, B., & Xu, H. (2013). Evaluation of vector space models for medical disorders information retrieval. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Zhong, X., Xia, Y., Xie, Z., Na, S., Hu, Q., & Huang, Y. (2013). Concept-based medical document retrieval: THCIB at CLEF eHealth lab 2013 task 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Zhu, D., & Carterette, B. (2012). Improving health records search using multiple query expansion collections. In 2012 IEEE international conference on bioinformatics and biomedicine (BIBM) (pp. 1–7). IEEE.

Zhu, D., Wu, S., James, M., Carterette, B., & Liu, H. (2013). Using discharge summaries to improve information retrieval in clinical domain. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Zhu, D., Stephen, W., Carterette, B., & Liu, H. (2014). Using large clinical corpora for query expansion in text-based cohort identification. Journal of Biomedical Informatics, 49, 275–281.CrossRef

Zuccon, G., & Koopman, B. (2018). Choices in knowledge-base retrieval for consumer health search. In ECIR’18.

Zuccon, G., Koopman, B., & Nguyen, A. (2013). Retrieval of health advice on the web: AEHRC at ShARe/CLEF eHealth evaluation lab task 3. In Proceedings of the ShARe/CLEF eHealth Evaluation Lab.

Zuccon, G., Koopman, B., & Palotti, J. (2015). Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis information. In Advances in information retrieval (pp. 562–567).

Zuccon, G., Koopman, B., Bruza, P., & Azzopardi, L. (2015). Integrating and evaluating neural word embeddings in information retrieval. In Proceedings of the 20th Australasian document computing symposium (p. 12). ACM.

Zuccon, G., Koopman, B., Nguyen, A., Vickers, D., & Butt, L. (2012). Exploiting medical hierarchies for concept-based information retrieval. In Proceedings of the seventeenth Australasian document computing symposium (pp. 111–114). ACM.

Zuccon, G., Palotti, J., Goeuriot, L., Kelly, L., Lupu, M., Pecina, P., et al. (2016). The IR task at the CLEF eHealth Evaluation Lab 2016: User-centred health information retrieval. In Proceedings of CLEF.

Titel: An analysis of evaluation campaigns in ad-hoc medical information retrieval: CLEF eHealth 2013 and 2014
verfasst von: Lorraine Goeuriot
Gareth J. F. Jones
Liadh Kelly
Johannes Leveling
Mihai Lupu
Joao Palotti
Guido Zuccon
Publikationsdatum: 03.05.2018
Verlag: Springer Netherlands
Erschienen in: Discover Computing / Ausgabe 6/2018
Print ISSN: 2948-2984
Elektronische ISSN: 2948-2992
DOI: https://doi.org/10.1007/s10791-018-9331-4

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner