Skip to main content

Advertisement

SpringerLink
Log in

Towards a Repository for Standardized Medical Image and Signal Case Data Annotated with Ground Truth

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Validation of medical signal and image processing systems requires quality-assured, representative and generally acknowledged databases accompanied by appropriate reference (ground truth) and clinical metadata, which are composed laboriously for each project and are not shared with the scientific community. In our vision, such data will be stored centrally in an open repository. We propose an architecture for a standardized case data and ground truth information repository supporting the evaluation and analysis of computer-aided diagnosis based on (a) the Reference Model for an Open Archival Information System (OAIS) provided by the NASA Consultative Committee for Space Data Systems (ISO 14721:2003), (b) the Dublin Core Metadata Initiative (DCMI) Element Set (ISO 15836:2009), (c) the Open Archive Initiative (OAI) Protocol for Metadata Harvesting, and (d) the Image Retrieval in Medical Applications (IRMA) framework. In our implementation, a portal bunches all of the functionalities that are needed for data submission and retrieval. The complete life cycle of the data (define, create, store, sustain, share, use, and improve) is managed. Sophisticated search tools make it easier to use the datasets, which may be merged from different providers. An integrated history record guarantees reproducibility. A standardized creation report is generated with a permanent digital object identifier. This creation report must be referenced by all of the data users. Peer-reviewed e-publishing of these reports will create a reputation for the data contributors and will form de-facto standards regarding image and signal datasets. Good practice guidelines for validation methodology complement the concept of the case repository. This procedure will increase the comparability of evaluation studies for medical signal and image processing methods and applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Abbreviations

AIM:

Annotation and Image Markup

AIP:

Archival Information Package

BIRN:

Biomedical Informatics Research Network

CAD:

Computer-Aided Diagnosis

CADe:

Computer-Aided Detection

CADx:

Computer-Aided Diagnostics

CARS:

Computer-Assisted Radiology and Surgery

CAS:

Computer-Aided Surgery

CBIIT:

Center for Biomedical Informatics and Information Technology

CBIR:

Content-Based Image Retrieval

CIP:

Cancer Imaging Program

CLEF:

Cross-Language Evaluation Forum

CMH:

Cambridge Memorial Hospital

CMSP:

Custom Medical Stock Photo

CT:

Computed Tomography

DARE:

Document Analysis Research Engine

DCES:

Dublin Core Element Set

DCMI:

Dublin Core Metadata Initiative

DDSM:

Digital Database for Screening Mammography

DICOM:

Digital Imaging and Communication in Medicine

DIP:

Dissemination Information Package

DOAR:

Directory of Open Access Repositories

DOI:

Digital Object Identifier

DR:

Digital Radiography

DRIVER:

Digital Repository Infrastructure Vision for European Research

EFMI:

European Federation of Medical Informatics

GIF:

Graphics Interchange Format

GNU:

Gnu is Not Unix

GPL:

General Public License

GUI:

Graphical User Interface

HUG:

University Hospitals of Geneva

HTTP:

Hypertext Transfer Protocol

IF:

Impact Factor

IHE:

Integrating the Healthcare Enterprise

IRMA:

Image Retrieval in Medical Applications

ISI:

Institute for Scientific Information

ISO:

International Organization for Standardization

JPG:

Joint Photographic Experts Group

LIDC:

Lung Image Database Consortium

LONI:

Laboratory Of Neuro Imaging

MEDIREC:

Medical Image Reference Center

MRI:

Magnetic Resonance Imaging

MIAS:

Mammographic Image Analysis Society

MIE:

Medical Informatics Europe

MIRC:

Medical Imaging Resource Center

MOD:

Magneto-Optical Disk

NASA:

National Aeronautics and Space Administration

NBIA:

National Biomedical Imaging Archive

NCI:

National Cancer Institute

NIH:

National Institutes of Health

OAI:

Open Archive Initiative

OAIS:

Open Archival Information System

OpenCV:

Open-Source Computer Vision Library

PACS:

Picture Archiving and Communication System

PDI:

Preservation Description Information

PEIR:

Pathology Education Instructional Resource

PET:

Positron Emission Tomography

PMH:

Protocol for Metadata Harvesting

PURL:

Persistent Uniform Resource Locator

QBE:

Query By Example

RepSIS:

Repository for Standardized Medical Image and Signal case data references

RIDER:

Reference Image Database to Evaluate therapy Response

ROI:

Region Of Interest

RSNA:

Radiological Society of North America

SciDR:

Science Digital Repository

SDB:

Simulated Brain Database

SIP:

Submission Information Package

SSD:

Solid State Drive

TCE:

Teaching File and Clinical Trial Export

TCGA:

The Cancer Genome Atlas

URL:

Uniform Resource Locator

WGMIP:

Working Group on Medical Image Processing

XML:

Extensible Markup Language

References

  1. Akgul C, Rubin D, Napel S, et al: Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24(2):208–222, 2011

    Article  PubMed  Google Scholar 

  2. Horsch A, Prinz M, Schneider S, Sipilä O, Spinnler K, Vallée JP, Verdonck-de-Leeuw I, Vogl R, Wittenberg T, Zahlmann G: Establishing an international reference image database for research and development in medical image processing. Method Inform Med 43(4):409–412, 2004

    CAS  Google Scholar 

  3. Horsch A, Blank R, Eigenmann D: The EFMI reference image database initiative: concept, state and related work. Procs CARS 1281:447–452, 2005

    Google Scholar 

  4. Horsch A, Hapfelmeier A, Elter M. Needs assessment for next generation computer-aided mammography reference image databases and evaluation studies. Int J Comput Assist Radiol Surg 6(6): 749–767, 2011

    Google Scholar 

  5. Zheng B: Computer-aided diagnosis in mammography using content-based image retrieval approaches: current status and future perspectives. Algorithms 2:828–849, 2009

    Article  PubMed  Google Scholar 

  6. Andriole KP, Barish MA, Khorasani R: Advanced image processing in the clinical arena: issues to consider. J Am Coll Radiol 3(4):296–298, 2006

    Article  PubMed  Google Scholar 

  7. Dammann F: Image processing in radiology. Rofo 174(5):541–550, 2002

    Article  PubMed  CAS  Google Scholar 

  8. Horsch A, Punys V, Wismüller A, Castro Martinez A, Clarke L. Workshop on Validation of Medical Image Processing Systems at Tromsø Telemedicine Conference TTeC 2006. June 12, 2006, Memorandum, available at www.efmi-wg-mip.net

  9. Lopresti D, Lamiroy B. Document analysis and research in the year 2021. Lect Notes Computer Sci 6703; 264–274, 2011

  10. Müller H, Rosset A, Vallée JP, Terrier F, Geissbuhler A: A reference data set for the evaluation of medical image retrieval systems. Comput Med Imaging Graph 28(6):295–305, 2004

    Article  PubMed  Google Scholar 

  11. Kamauu AWC, Whipple JJ, DuVall SL, Siddiqui KM, Siegel EL, Avrin D: IHE teaching file and clinical trial export integration profile: functional examples. RadioGraphics 28:933–945, 2008

    Article  PubMed  Google Scholar 

  12. Horsch A, Thurmayr R: How to identify and assess tasks and challenges of medical image processing. Procs MIE 2003, 281–5

  13. Consultative Committee for Space Data Systems (ed). Reference Model for an Open Archival Information System (OAIS). Blue Book CCSDS 650.0-B-1, Washington: NASA, 2002

  14. Dublin Core (ed). Dublin Core Metadata Element Set, Version 1.1 2010-10-11, http://dublincore.org/documents/dces/

  15. Lagoze C, van de Sompel H (ed). The Open Archives Initiative Protocol for Metadata Harvesting. Technical Report. Protocol Version 2.0 of 2002-06-14, Document Version 2008-12-07 T20:42:00, 2008. http://www.openarchives.org/OAI/openarchivesprotocol.html

  16. Swan A, Awre C. Linking UK Repositories: Technical and Organisational Models to Support user-oriented Services Across Institutional and other Digital Repositories. Scoping Study Report, University of Hull, 2006, http://www.rsp.ac.uk/usage/harvesters.

  17. ISO Technical Committee 215 (ed). Health informatics—Harmonized data types for information exchange. Technical Standard ISO/DIS 21090:2011.

  18. Hussein R, Engelmann U, Schroeter A, Meinzer HP: DICOM structured reporting. Part 1. Overview and characteristics. RadioGraphics 24:891–896, 2004

    Article  PubMed  Google Scholar 

  19. Channin DS, Mongkolwat P, Kleper V, Sepukar K, Rubin DL: The caBIG annotation and image Markup project. J Digit Imaging 23(2):217–225, 2010

    Article  PubMed  Google Scholar 

  20. Zimmerman SL, Kim W, Boonn WW: Informatics in radiology: automated structured reporting of imaging findings using the AIM Standard and XML. Radiographics 31(3):881–887, 2011

    Article  PubMed  Google Scholar 

  21. The Digital Archiving Consultancy Limited (ed). Towards a European e-Infrastructure for e-Science Digital Repositories: a report for the European Commission. Technical Report 2006 S88-092641. Middlesex, UK, 2008. http://www.e-scidr.eu/wp-content/uploads/2007/03/e-SciDR_DAC_Final_Report.pdf

  22. Deserno TM, Güld MO, Plodowski B, Spitzer K, Wein BB, Schubert H, Ney H, Seidl T: Extended query refinement for medical image retrieval. J Digit Imaging 21(3):280–289, 2008

    Article  PubMed  Google Scholar 

  23. Lehmann TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB: Content-based image retrieval in medical applications. Methods Inf Med 43(4):354–361, 2004

    PubMed  CAS  Google Scholar 

  24. Güld MO, Thies C, Fischer B, Lehmann TM: A generic concept for the implementation of medical image retrieval systems. Int J Med Inform 76(2–3):252–259, 2007

    Article  PubMed  Google Scholar 

  25. Welter P, Riesmeier J, Fischer B, Grouls C, Kuhl C, Deserno TM: Bridging the integration gap from imaging to information systems: a uniform data concept for content-based image retrieval in computer-aided diagnosis. J Am Med Inform Assoc 18:506–510, 2011

    Article  PubMed  Google Scholar 

  26. Lamiroy B, Lopresti D. Supporting experimental research in computer vision. SPIE Newsroom 2001; April 1, doi: 10.1117/2.1201103.003558

  27. Lamiroy B, Lopresti D, Korth H, Heflin J. How carefully designed open resource sharing can help and expand document analysis research. Proc SPIE 7874: 78740O, 2011

  28. Deserno TM, Antani S, Long R: Ontology of gaps in content-based image retrieval. J Digit Imaging 22(2):202–215, 2009

    Article  PubMed  Google Scholar 

  29. Doi K: Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 31(4–5):198–211, 2007

    Article  PubMed  Google Scholar 

  30. Müller H. Medical multimedia retrieval 2.0. Yearb Med Inform 55–63, 2008

Download references

Acknowledgments

This research was partly funded by the German Research Foundation (DFG), grant no. Le 1108/9. The authors would like to thank George Thoma, National Library of Medicine, National Institutes of Health (NIH), USA, for his critical reflections on our approach and for contributing a perspective that improved this research.

Author information

Authors and Affiliations

  1. Dept. of Medical Informatics, RWTH Aachen University, Pauwelsstraße 30, 52074, Aachen, Germany

    Thomas M. Deserno & Petra Welter

  2. Institut für Med. Statistik und Epidemiologie, TU München, Ismaninger Str. 22, 81675, Munich, Germany

    Alexander Horsch

Authors
  1. Thomas M. Deserno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Petra Welter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexander Horsch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas M. Deserno.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Deserno, T.M., Welter, P. & Horsch, A. Towards a Repository for Standardized Medical Image and Signal Case Data Annotated with Ground Truth. J Digit Imaging 25, 213–226 (2012). https://doi.org/10.1007/s10278-011-9428-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-011-9428-4

Keywords

Search

Navigation