Basics of Software Engineering Experimentation

This book addresses experimentation in Software Engineering (SE). The book is aimed at raising readers’ interest in experimentation that has been lacking in the SE field. Borrowing the experimental tradition from other science and engineering areas could serve the discipline well.

Chapter 1 briefly described some ideas about the need for experimentation in SE. This chapter discusses in more detail this question. In particular, section 2.2. analyses what use the process of experimentation is to any scientific and engineering discipline, that is, analyses how scientific experimentation contributes to the development of a science or branch of engineering. We will see how these ideas are also applicable to a scientific discipline to be used for software development.

This chapter aims to put the remainder of the book, that is, Parts II and III, into context. For this purpose, it focuses on describing the steps to be taken to run an experiment, the most important of which will be described in the other parts of the book. Beforehand, section 3.2 examines what sort of relationships among variables can be outputted by an experiment. Having described these relationships, the process of stepwise refinement involved in any experimentation process is described in section 3.3. Each cycle of this process involves running a given experiment, the process to be followed is described in section 3.4. We will see that this process is composed of the phases of goal definition, design, execution and analysis. All these phases are essential for the success of the experiment. However, experimental design and analysis call for special attention. Therefore, Parts II and III of the book focus on these two phases, respectively. Finally, section 3.5 describes what can be deduced at the end of these stages and what role statistics plays in determining the above conclusion.

This chapter focuses on the basic concepts to be handled during experimental design. Before addressing design, we need to study the terminology to be used. This is done in section 4.2. Sections 4.3 and 4.4 focus on the application of this terminology to the particular field of SE. In those sections we suggest possible variables for SE experiments as an aid for novice experimenters. However, the variables proposed here are only a suggestion and SE experimenters can work with an alternative set depending on their particular goals. Additionally, we will also examine some variables used in real SE experiments, going beyond a merely theoretical discussion.

As discussed in Chapter 4, experimental design decides which variables will be examined and their values, which data will be collected, how many experiments have to be run and how many times the experiments have to be repeated. In other words, a decision is made on how the experiment will actually be arranged. This chapter examines the different kinds of experimental design there are and the circumstances under which each one should be used.

Having designed the experimentation, each unitary experiment is run as prescribed by the design. Measurements of the response variable are taken during the experiments. So, after completing the unitary experiments, experimenters have a collection of data, called the results of the experimentation. By examining or analysing these data, experimenters will arrive at conclusions about the relationships between factors, the influence of factors on the response variables and the alternatives that improve the values of the response variables. The data analysis branch of statistics provides the techniques for examining the experimental results. These are statistical methods that are used to analyse the data output by experiments and are explained in this, the third part of the book.

We are now going to address the comparison of k alternatives for any one factor. The method of analysis examined in this chapter can also be applied for k=2 and is, therefore, an alternative procedure to the one discussed in Chapter 7. Again we are looking at one-factor experiments, in which the other parameters would either remain unchanged or, as remarked upon in Chapter 2, have similar values. The underlying philosophy and process is similar to the comparison of two means: the question is whether there are real differences between the results obtained for the different options or whether the differences observed are merely due to chance. Again, a standard distribution that will output the level of significance of the differences found will be used to answer this question. Once again randomisation is essential if this standard distribution is to be used as a reference.

As specified in Chapter 5, there is an experimental design for dealing with variables whose effect on the response variable we are not interested in. Designs of this sort are known as block designs, and the variables whose effect is to be eliminated are known as blocking variables. This chapter discusses the process for analysing data collected from experiments designed thus. Firstly, we will address the case where there is one variable that is not of interest (section 9.2) and then go on to review the analysis process when there are several blocking variables (section 9.3, 9.4 and 9.5). A somewhat special analysis has to be conducted when any of the response variables that should have been gathered are missing. We look at how to do this analysis in section 9.6 Finally, in section, 9.7, we will examine the case where the block size is smaller than the number of factor alternatives, which we referred to as incomplete block designs in Chapter 5.

As discussed in Chapter 5, the design to be used when all the factors involved in the experiment are of interest to the investigation, that is, we want to find out what impact they have on the response variable, is a factorial design. Designs of this sort study the effect of each factor individually, as well as any interactive influence some factors combined with others could have on the response variable.

Nested or hierarchical designs were described in Chapter 5 as particular cases of multiple-factor designs in which not all the alternatives of the factors can be studied together, as is the case in factorial designs.

As the number of factors in a 2^k factorial design increases, the number of experiments required to get a full replication soon exceeds the resources of most experimenters. For example, a full replication of a 2⁶ design calls for 64 experiments. Only 6 of 63 degrees of freedom correspond to principal effects in this design and only 15 to two-factor interactions; the other 42 correspond to interactions of three or more factors.

As mentioned in Chapter 5, it may not be possible to experiment with all the combinations of alternatives required by a factorial design under homogeneous conditions. This situation is managed using blocks. As discussed in preceding chapters, blocks can refer to the subjects running the experiment, the times at which they are run, variations in the projects used as experimental units or any other undesired variation that occurs from one unitary experiment to another.

As mentioned in Chapter 6, non-parametric tests are applicable in two circumstances: (1) when parametric tests cannot be used owing to the scale of the response variable, that is, the scale is nominal or ordinal, and (2) when, even when the scale of the response variable admits the use of parametric tests, that is, it is an interval or ratio scale, the observations gathered do not meet the constraints on normality called for by parametric tests. Remember that the methods of analysis that we discussed in earlier chapters (parametric methods) are subject to a series of constraints. Therefore, one of the steps taken in earlier chapters during the analysis process was to validate the mathematical model that explained the observations gathered from the experiment. An explanation of how to run these tests was given in these chapters. When these constraints are not met, the analysis of the data collected during experiments must switch to non-parametric methods.

An important decision in any problem of experimental design is to determine how many times an experiment should be replicated. Note that we are referring to the internal replication of an experiment. Generally, the more it is replicated, the more accurate the results of the experiment will be. However, resources tend to be limited, which places constraints on the number of replications. In this chapter, we will consider several methods for determining the best number of replications for a given experiment. We will focus on one-factor designs, but the general-purpose methodology can be extended to more complex experimental situations.

This chapter aims to outline the most important ideas discussed throughout the book, mainly focusing on some useful points for correctly running and documenting experiments so that they can be replicated by other people. Section 16.2 groups a series of precautions, which, although they have been addressed in other chapters of the book, should be taken into account by readers whenever they experiment. Section 16.3, on the other hand, provides a set of guidelines to aid novice experimenters to document their empirical work. These guidelines are designed to ease external replication. They can also be used to check that the fundamental design and analysis issues of each experiment have been taken into consideration.