7. Power Analysis Using R

This section describes how power analysis on published papers can be done using a suite of simple R scripts, so that better-designed experiments can be conducted in the future. Here, “better” means “ensuring appropriate statistical power”. First, an overview of the five R scripts is given (Sect. 7.2), followed by a description of each script (Sects. 7.3, 7.4, 7.5, 7.6, and 7.7). The five scripts, which are for paired t-test, two-sample t-test, one-way ANOVA, two-way ANOVA without replication, and two-way ANOVA with replication, respectively, were adapted from the R scripts of Toyoda (Introduction to statistical power analysis: a tutorial with R (in Japanese). Tokyo Tosyo, 2009): his original scripts, which contain Japanese character codes, are available from his book’s website (http://​www.​tokyo-tosho.​co.​jp/​download/​DL02065.​zip); Toyoda’s scripts (and therefore mine as well) rely on R libraries called stats and pwr. (The present author is solely responsible for any problems caused by modifying the original scripts of Toyoda.) Finally, it provides summary while touching upon a survey I conducted using these R scripts, with a decade’s worth of IR papers from ACM SIGIR (http://​sigir.​org/​) and TOIS (https://​tois.​acm.​org/​) (Sakai Statistical significance, power, and sample sizes: a systematic review of SIGIR and TOIS. In: Proceedings of ACM SIGIR 2016, pp 5–14, 2016), where it was demonstrated that there are highly overpowered and highly underpowered experiments in the results reported in the IR literature. Highly overpowered experiments use a lot more resources than necessary, while highly underpowered experiments are highly likely to miss important differences that exist due to the use of small samples. We can probably do better by learning from previous studies and/or from pilot studies.