Comparing Google to a digital reference service for answering factual and topical requests by keyword and question queries

This paper seeks to evaluate to what extent Google retrieved correct answers to queries inferred from factual and topical requests in a digital Ask‐a‐Librarian service.

In total, 100 factual and 100 topical questions were picked from a digital reference service run by public libraries. The inferred queries simulated average web queries. They were expressed as separate keywords and as questions. The top ten retrieval results were observed for each answer. The inspection was stopped when the first correct answer was identified.

Google retrieved correct answers to 42 percent of the topical questions and 29 percent of factual questions by keyword queries. The performance of queries in question form was considerably weaker. The results concerning the characteristics of queries and retrieval effectiveness are also presented. Evaluations indicate that the public library reference services answered at least 55 percent of the questions correctly. Thus Google did not outperform the Ask‐a‐Librarian service.

The study introduces a new way of evaluating search engines by comparing their performance with other related services such as an Ask‐a‐Librarian service.