Weitere Kapitel dieses Buchs durch Wischen aufrufen
This section describes how power analysis on published papers can be done using a suite of simple R scripts, so that better-designed experiments can be conducted in the future. Here, “better” means “ensuring appropriate statistical power”. First, an overview of the five R scripts is given (Sect. 7.2), followed by a description of each script (Sects. 7.3, 7.4, 7.5, 7.6, and 7.7). The five scripts, which are for paired t-test, two-sample t-test, one-way ANOVA, two-way ANOVA without replication, and two-way ANOVA with replication, respectively, were adapted from the R scripts of Toyoda (Introduction to statistical power analysis: a tutorial with R (in Japanese). Tokyo Tosyo, 2009): his original scripts, which contain Japanese character codes, are available from his book’s website (http://www.tokyo-tosho.co.jp/download/DL02065.zip); Toyoda’s scripts (and therefore mine as well) rely on R libraries called stats and pwr. (The present author is solely responsible for any problems caused by modifying the original scripts of Toyoda.) Finally, it provides summary while touching upon a survey I conducted using these R scripts, with a decade’s worth of IR papers from ACM SIGIR (http://sigir.org/) and TOIS (https://tois.acm.org/) (Sakai Statistical significance, power, and sample sizes: a systematic review of SIGIR and TOIS. In: Proceedings of ACM SIGIR 2016, pp 5–14, 2016), where it was demonstrated that there are highly overpowered and highly underpowered experiments in the results reported in the IR literature. Highly overpowered experiments use a lot more resources than necessary, while highly underpowered experiments are highly likely to miss important differences that exist due to the use of small samples. We can probably do better by learning from previous studies and/or from pilot studies.
Bitte loggen Sie sich ein, um Zugang zu diesem Inhalt zu erhalten
Sie möchten Zugang zu diesem Inhalt erhalten? Dann informieren Sie sich jetzt über unsere Produkte:
Note that you can look up the specification of any standard R function using ? on the R command line, e.g. ? ‘power.t.test’.
If the group sizes are unequal, the average group size over the m groups can be used [ 4].
This is in fact the example we discussed in Sect. 7.3.
For general considerations required for designing user studies, see Kelly [ 2].
J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd edn. (Psychology Press, New York, 1988) MATH
D. Kelly, Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retr. 3(1–2), 1–224 (2009)
T. Sakai, Statistical significance, power, and sample sizes: a systematic review of SIGIR and TOIS, in Proceedings of ACM SIGIR, Pisa, 2016, pp. 5–14
H. Toyoda, Introduction to Statistical Power Analysis: A Tutorial with R (in Japanese) (Tokyo Tosyo, Chiyoda, 2009)
- Power Analysis Using R
- Springer Singapore
- Chapter 7