Contributions of Language Modeling to the Theory and Practice of Information Retrieval

This paper presents an analysis of what language modeling (LM) is in the context of information retrieval (IR). We argue that there are two principal contributions of the language modeling approach. First, that it brings the thinking, theory, and practical knowledge of research in related fields to bear on the retrieval problem. Second, that it makes patent that parameter estimation is important for probabilistic IR approaches. In particular, it has brought to the attention of the IR community the idea that explicit consideration needs to be given to variance reduction in the design of statistical estimators. We describe a simulation environment which has been developed for the study of theoretical issues in information retrieval. Results obtained from the simulation are presented, which show quantitatively how variance reduction techniques applied to parameter estimation can improve performance for the ad-hoc retrieval task.