Systematic literature reviews in software engineering – A tertiary study

Abstract

Context

In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007.

Objective

The aim of this on-going research is to provide an annotated catalogue of SLRs available to software engineering researchers and practitioners. This study updates our previous study using a broad automated search.

Method

We performed a broad automated search to find SLRs published in the time period 1st January 2004 to 30th June 2008. We contrast the number, quality and source of these SLRs with SLRs found in the original study.

Results

Our broad search found an additional 35 SLRs corresponding to 33 unique studies. Of these papers, 17 appeared relevant to the undergraduate educational curriculum and 12 appeared of possible interest to practitioners. The number of SLRs being published is increasing. The quality of papers in conferences and workshops has improved as more researchers use SLR guidelines.

Conclusion

SLRs appear to have gone past the stage of being used solely by innovators but cannot yet be considered a main stream software engineering research methodology. They are addressing a wide range of topics but still have limitations, such as often failing to assess primary study quality.

Introduction

In a series of three papers Kitchenham, Dybå and Jørgensen suggested that software engineers in general, and empirical software engineering researchers in particular, should adopt evidence-based practice as pioneered in the fields of medicine and sociology [1], [2], [3]. They proposed a framework for Evidence-based Software Engineering (EBSE), derived from medical standards, that relies on aggregating best available evidence to address engineering questions posed by practitioners and researchers. The most reliable evidence comes from aggregating all empirical studies on a particular topic. The recommended methodology for aggregating empirical studies is a systematic literature review (SLR) (see for example [4], [5], [6]). Kitchenham adapted the medical guidelines for SLRs to software engineering [7], and later updated them to include insights from sociology research [8].

SLRs are a means of aggregating knowledge about a software engineering topic or research question [5], [6], [7], [8]. The SLR methodology aims to be as unbiased as possible by being auditable and repeatable. SLRs are referred to as secondary studies and the studies they analyse are referred to as primary studies. There are two different types of SLRs:

• Conventional SLRs aggregate results related to a specific research question e.g. “Is testing technique a more effective at defect detection than testing technique b?” If there are sufficient comparable primary studies with quantitative estimates of the difference between methods, meta-analysis can be used to undertake a formal statistically-based aggregation. However, we have found that meta-analysis is seldom possible for SLRs in software engineering because there are often insufficient primary studies.

• Mapping studies. These studies aim to find and classify the primary studies in a specific topic area. They have coarser-grained research questions such as “What do we know about topic x?” They may be used to identify available literature prior to undertaking conventional SLRs. They use the same methods for searching and data extraction as conventional SLRs but rely more on tabulating the primary studies in specific categories. An example is the study of software engineering experiments [9] which led to a series of follow-on SLRs including [10], [11]. In addition, some mapping studies are concerned about how academics undertake research in software engineering (e.g. [13]) rather than what we know about a specific software engineering topic. The study reported in this paper is a mapping study.

This distinction between mapping studies and conventional SLRs can be somewhat fuzzy. Some mapping studies (like this one) provide a more detailed review of the topics covered in each primary study including issues such as major outcomes and quality evaluations of primary studies.

We believe secondary studies can play a vital role both in supporting further research efforts and also in providing information about the impact of methods and tools to assist software engineering practitioners and managers [2], [1]. However, these studies need to be readily available to those who would benefit from them. For example, researchers entering a new field would benefit from mapping studies in the area, whereas standards writers would benefit from conventional SLRs evaluating the benefits of specific techniques. Academics would also benefit from mapping studies and conventional SLRs when preparing teaching materials or writing text books. For this reason we believe it is important to catalogue and evaluate all such papers.

We recently published the results of a mapping study aimed at identifying software engineering SLRs [12]. The study is referred to as a tertiary study, because it was a SLR of secondary studies.The goal of the study was to identify how many SLRs had been published, what research topics were being addressed, and the limitations of current SLRs. For that study we used a manual search of a targeted set of 13 conferences and journals during the period January 1st 2004 to 30th June 2007. The sources were selected because they were known to include empirical studies and literature surveys, and had been used as sources for other mapping studies (e.g. [9], [13]). This search identified 20 SLRs of which eight were mapping studies and one a meta-analysis.

In this paper, we report the results of a broad automated search covering the period 1st January 2004 to 30th June 2008, and contrast them with our previous results. In effect we compare three sets of SLRs:

• Those reported in the original study, covering the time period January 2004 to June 30th 2007 [12].

• Those found in the time period January 2004 to June 30th 2007 that were found by the broad automated search and were not included in the original study. We discuss the differences between the results of the manual search and the broad automated search in [14].

• Those found in the time period July 1st 2007 to June 30th 2008.

These are illustrated in Fig. 1. For convenience and to simplify referencing, these sets papers are respectively referred to as T1, T2-1 and T2-2 respectively in the rest of the paper (T for ‘tertiary’). The original study [12] is referred to as T1, this one as T2.

Section 2 reports our methodology. Section 3 reports data we extracted from each SLR. Section 4 answers our research questions. We report the limitations of our study in Section 5 and our conclusions in Section 6.