The role of replications in empirical software engineering—a word of warning

Excerpt

There are many issues raised by Shull et al. that I would whole-heartedly endorse. For example, replication is a basic component of the scientific method, so it hardly needs to be justified. Furthermore, a full comprehensive report of any empirical study is a requirement of good science. When restrictions on the length of conference or journal paper prevent such reporting, it is an extremely good idea to record all the experimental details somewhere, whether in a “laboratory package” or a technical report. However, is such supporting information essential for subsequent replications? The authors give examples such as insufficient training, or systems not having defects appropriate for the technique, causing replications to wrongly contradict the original experiment. I believe issues such as the criticality of training or the type of relevant defects should be reported in the published study not hidden in ancillary data. It may be preferable to advise empirical researchers to make very clear the critical components of the software engineering methods they are studying in their journal and conference papers. …