IR Research: Systems, Interaction, Evaluation and Theories

The ultimate goal of information retrieval (IR) research is to create ways to support humans to better access information in order to better carry out their (work) tasks. Because of this, IR research has a primarily technological interest in knowledge creation – how to find information (better)? IR research therefore has a constructive aspect (to create novel systems) and an evaluative aspect (are they any good?). Evaluation is sometimes referred to as a hallmark and distinctive feature of IR research. No claim on IR system performance is granted any merit unless proven through evaluation. Technological innovation alone is not sufficient. In fact, much research in IR deals with IR evaluation and its methodology.

Evaluation, in general, is the systematic determination of merit and significance of something using criteria against some standards. Evaluation therefore requires some object that is evaluated and some goal that should be achieved or served. In IR, both can be set in many ways. The object usually is an IR system – but what is an IR system? The goal is typically the quality of the retrieved result – but what is the retrieved result and how does one measure quality? These questions can be answered in alternative ways leading to different kinds of IR evaluation.

Practical life with all its variability is difficult and expensive to investigate. Therefore surrogate and more easily measurable goals are employed in IR evaluation, typically the quality of the ranked result list instead of the work task result. The task performance process may also be cut down from a work task to a search task and down to running an individual query in a test collection. This simplification has led to standardization of research designs and tremendous success in IR research. However, as the goals and systems drift farther away from the practical life condition, one needs to ask, whether the findings still best serve the initial goal of evaluation (supporting human performance)? If means (outputs) replace ends (outcomes), one runs the risk of sub-optimization.

It is important to evaluate all subsystems of information retrieval processes, in addition to the search engines. Through a wider perspective one may be able to put the subsystems and their contributions in relation with each other. We will discuss nested IR evaluation frameworks ranging from IR system centered evaluation to work-task based evaluation. We will also point to the Pandora’s box of problems that the enlargement of the scope of research entails. Is science at risk here?

The contributions of a research area, in addition to constructive and evaluative contributions, may be generally empirical, theoretical and methodological. Why should anyone in IR care about anything beyond IR experimentation (i.e. evaluation) using test collections? The Cranfield model seeks to relate texts (documents), queries, their representations and matching to topical relevance in ranked output. Who relates this, and a range of possible other contributing factors, to outcomes in search task performance or work task performance? The talk will outline some possibilities for descriptive, explanatory and theoretical research in IR. As an example of descriptive research, we will discuss information access in task processes. Regarding explanatory and theoretical research, we look at unit theories that connect work task stages and properties to information need properties, information sources, and searching. Such studies do not solve a technical problem, nor evaluate any particular technique, and may therefore be considered unpractical. However, they may identify mechanisms that mediate between IR processes and task outcomes and position factors in the processes of information access into a balanced perspective. Therefore they may help focus research efforts on technical problems or evaluation.