Towards Semantic Evaluation of Information Retrieval

The paper discuss fundamentals of semantic evaluation of information retrieval systems. Semantic evaluation is understood in two ways. Semantic evaluation

sensu stricto

consists of automatic global methods of information retrieval evaluation which are based on knowledge representation systems. Semantic evaluation

sensu largo

includes also evaluation of retrieved results presented using new methods and comparing them to previously used which evaluated unordered set of documents or lists of ranked documents. Semantic information retrieval methods can be treated as storing meaning of words which are basic building blocks of retrieved texts. In the paper, ontologies are taken as systems which represent knowledge and meaning. Ontologies serve as a basis for semantic modeling of information needs, which are modeled as families of concepts. Semantic modeling depends also on algorithmic methods of assigning concepts to documents. Some algebraic and partially ordered set methods in semantic modeling are proposed leading to different types of semantic modeling. Then semantic value of a document is discussed, it is relativized to a family of concepts and essentially depends on the used ontology. The paper focuses on sematic relevance of documents, both binary and graded, together with semantic ranking of documents. Various types of semantic value and semantic relevance are proposed and also some semantic versions of information retrieval evaluation measures are given.