2. t-Tests

This chapter first explains how the following classical significance tests for comparing two means work: the paired t-test for paired data (Sect. 2.2) and (Student’s) two-sample t-test and Welch’s two-sample t-test for unpaired data (Sects. 2.3 and 2.4). You have paired data if, for example, you evaluate two search engines using the same topic set with some evaluation measure such as normalised Discounted Cumulative Gain (nDCG) (Järvelin and Kekäläinen, ACM TOIS 20(4):422–446, 2002). (For a survey on IR evaluation measures, see Sakai (Metrics, statistics, tests. In: PROMISE winter school 2013: bridging between information retrieval and databases. LNCS 8173, pp 116–163, 2014).) You have unpaired data if, for example, you evaluated System 1 with User Group A and System 2 with User Group B; the group sizes may differ. This chapter then discusses the relationship between the aforementioned two two-sample t-tests (Sect. 2.5) and shows how the three t-tests can easily be conducted using Excel (Sect. 2.6) and R (Sect. 2.7). Finally, it describes how confidence intervals for the mean differences can be constructed, based on the assumptions that form the basis of the three t-tests (Sect. 2.8).