Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Discover Computing
Published in: Discover Computing 6/2006

01-12-2006

Evaluating the effectiveness of content-oriented XML retrieval methods

Authors: Norbert Gövert, Norbert Fuhr, Mounia Lalmas, Gabriella Kazai

Published in: Discover Computing | Issue 6/2006

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Content-oriented XML retrieval approaches aim at a more focused retrieval strategy: Instead of retrieving whole documents, document components that are exhaustive to the information need while at the same time being as specific as possible should be retrieved. In this article, we show that the evaluation methods developed for standard retrieval must be modified in order to deal with the structure of XML documents. More precisely, the size and overlap of document components must be taken into account. For this purpose, we propose a new effectiveness metric based on the definition of a concept space defined upon the notions of exhaustiveness and specificity of a search result. We compare the results of this new metric by the results obtained with the official metric used in INEX, the evaluation initiative for content-oriented XML retrieval.
previous article Swedish full text retrieval: Effectiveness of different combinations of indexing strategies with query terms
next article Precision prediction based on ranked list coherence

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
Footnotes
5
For example, the original TREC collection contains both newspaper articles (of the size of one or more kB) and a number of Federal Register documents (up to a few MB large) (Harman, 1993); treating both kinds of documents equally in evaluation is not appropriate from our point of view.
 
6
We make no explicit assumptions about users here, due to the fact that little is known about user behaviour when searching XML documents. However, the ongoing INEX interactive track is addressing this issue (Tombros et al., 2005).
 
7
In this article, the terms elements and components are used interchangeably.
 
8
Readers familiar with the classical IR literature will note that the terms ‘exhaustiveness’ and ‘specificity’ originally were introduced in the context of document indexing, where they referred to properties of the set of indexing terms assigned to a document (Lancaster, 1968); in contrast, we are regarding properties of a document (component) with respect to a query here.
 
9
In this paper, we use the term ‘topical exhaustiveness’ instead of ‘topical relevance’, in order to emphasize the two dimensions of relevance regarded here.
 
10
In INEX 2002, another but comparable definition of relevance was used, also based on two dimensions. The first dimension, topical relevance, corresponds to the exhaustiveness dimension defined in INEX 2003. The second dimension, coverage, is related to specificity. It has four values: no coverage, too small, too big and exact.
 
11
This comes from the fact that exhaustiveness remains or increases when going from a child element to its parent element, whereas specificity usually decreases in such a case—see Section 5.
 
12
In INEX 2003, the HyREX system developed in Duisburg-Essen was made available to participants for the topic creation phase, see http://​www.​is.​informatik.​uni-duisburg.​de/​projects/​hyrex/​.
 
13
We chose these two test sets since they differ in the nature of the assessments and the size of the runs; the INEX 2004 setting was similar to that of 2003.
 
Literature
Baeza-Yates, R., Fuhr, N., & Maarek, Y. S. (Eds.) (2002). Proceedings of the SIGIR 2002 Workshop on XML and Information Retrieval.
Beaulieu, M., & Robertson, S. (1996). Evaluating interactive systems in TREC. Journal of the American Society for Information Science, 47(1), 85–94.
Chiaramella, Y., Mulhem, P., & Fourel, F. (1996). A model for multimedia information retrieval. Technical report, FERMI ESPRIT BRA 8134, University of Glasgow.
Cleverdon, C. W., Mills, J., & Keen, E. M. (1966). Factors determining the performance of indexing systems, vol. 2: Test results. Technical report, Aslib Cranfield Research Project, Cranfield, England.
Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems. Journal of the American Society for Information Science, 19, 30–41.
Cosijn, E., & Ingwersen, P. (2000). Dimensions of relevance. Information Processing and Management, 36(4), 533–550.
Croft, W. B., Moffat, A., van Rijsbergen, C. J., Wilkinson, R., & Zobel, J. (Eds.) (1998). In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM.
Fuhr, N., & Lalmas, M. (2004). Report on the INEX 2003 Workshop. SIGIR Forum, 38(1).
Fuhr, N., Malik, S., & Lalmas, M. (2004b). Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2003. In Fuhr et al. (2004a), (pp. 1–11).
Harman, D. (1993). Overview of the First Text REtrieval Conference. In D. Harman (Ed.) The First Text REtrieval Conference (TREC-1), Gaithersburg, Md. 20899, National Institute of Standards and Technology Special Publication 500-207.
Jones, K. S., & van Rijsbergen, C. J. (1976). Information retrieval test collections. Journal of Documentation, 32(1), 59–75.
Kando, N., & Adachi, J. (2004). Report from the NTCIR workshop 3. SIGIR Forum, 38(1), 10–16.
Kazai, G. (2004). Report of the INEX 2003 Metrics working group. In Fuhr et al. (2004a) pp. 184–190.
Kazai, G., Lalmas, M., & de Vries, A. P. (2004). The overlap problem in content-oriented XML retrieval evaluation. In K. Jäarvelin, J. Allen, P. Bruza, & M. Sanderson (Eds.), Proceedings of the 27st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 72–79) New York, ACM.
Kekäläinen, J., & Järvelin, K. (2002). Using graded relevance sssessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53(13).
Lancaster, F. W. (1968). Evaluation of the MEDLARS demand service. Report, National Library of Medicine, Bethesda, Maryland.
Peters, C., Braschler, M., Gonzalo, J., & Kluck, M. (Eds.) (2002). Evaluation of cross-language information retrieval systems (CLEF 2001). Vol. 2406 of Lecture Notes in Computer Science. Heidelberg et al., Springer.
Piwowarski, B., & Lalmas, M. (2004). Ensuring consistent and exhaustive relevance assessments for XML retrieval evaluation. In L. Gravano (Ed.), Proceedings of the 13th International Conference on Information and Knowledge Management. New York, ACM.
Raghavan, V. V., Bollmann, P., & Jung, G. S. (1989). A critical investigation of recall and precision as measures of retrieval system performance. ACM Transactions on Information Systems, 7(3), 205–229.
Salton, G. (Ed.) (1971). The SMART retrieval system—Experiments in automatic document processing. Englewood, Cliffs, New Jersey: Prentice Hall.
Salton, G., & McGill, M. J. (1983). Introduction to modern information retrieval. New York: McGraw-Hill.
Saracevic, T. (1995). Evaluation of evaluation in information retrieval. In: E. A. Fox, P. Ingwersen, and R. Fidel (Eds.), Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. (pp. 138–146), New York, ACM. ISBN 0-89791-714-6.
Saracevic, T. (1996). Relevance reconsidered. In P. Ingwersen and N. O. Pors (Eds.), In Proceedings of the 2nd International Conference on Conceptions of Library and Information Science (CoLIS 2), Oct. 13–16, 1996, (pp. 201–218).
Tombros, A., Larsen, B., & Malik, S. (2005). The Interactive Track at INEX 2004. In N. Fuhr, M. Lalmas, S. Malik, & Z. Szlavik (Eds.), Advances in XML Information Retrieval: Third International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2004, Dagstuhl Castle, Germany, Dec. 6–8, 2004, Revised Selected Papers, Vol. 3493. Springer-Verlag GmbH. http://​www.​springeronline.​com/​3-540-26166-4.
trec_eval (2002). Evaluation techniques and measures. In Voorhees and Harman, (2002), NIST.
Voorhees, E. M. (1998). Variations in relevance judgements and the measurement of retrieval effectiveness. In Croft et al. (1998), (pp. 315–323), ACM.
Voorhees, E. M., & Harman, D. K. (Eds.) (2002). The Tenth Text Retrieval Conference (TREC 2001). Gaithersburg, MD, USA: NIST.
Wong, S. K. M., & Yao, Y. Y. (1995). On modeling information retrieval with probabilistic inference. ACM Transactions on Information Systems, 13(1), 38–68.
Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments?. In Croft et al. (1998), (pp. 307–314), ACM.
Metadata
Title
Evaluating the effectiveness of content-oriented XML retrieval methods
Authors
Norbert Gövert
Norbert Fuhr
Mounia Lalmas
Gabriella Kazai
Publication date
01-12-2006
Publisher
Springer Netherlands
Published in
Discover Computing / Issue 6/2006
Print ISSN: 2948-2984
Electronic ISSN: 2948-2992
DOI
https://doi.org/10.1007/s10791-006-9008-2

Other articles of this Issue 6/2006

Discover Computing 6/2006 Go to the issue

OriginalPaper

Swedish full text retrieval: Effectiveness of different combinations of indexing strategies with query terms

OriginalPaper

Precision prediction based on ranked list coherence

OriginalPaper

A relatedness analysis of government regulations using domain knowledge and structural organization

OriginalPaper

Δ-distance: A family of dissimilarity metrics between images represented by multi-level feature vectors

Premium Partner

    Image Credits