nach oben

2008 | Buch

Guide to Advanced Empirical Software Engineering

herausgegeben von: Forrest Shull, Janice Singer, Dag I. K. Sjøberg

Verlag: Springer London

Enthalten in: Springer Professional "Wirtschaft+Technik" , Springer Professional "Technik" , Springer Professional "Wirtschaft"

Einloggen, um Zugang zu erhalten

Über dieses Buch

Empirical studies have become an important part of software engineering research and practice. Ten years ago, it was rare to see a conference or journal article about a software development tool or process that had empirical data to back up the claims. Today, in contrast, it is becoming more and more common that software engineering conferences and journals are not only publishing, but eliciting, articles that describe a study or evaluation. Moreover, a very successful conference (International Symposium on Empirical Software Engineering and Measurement), journal (Empirical Software Engineering), and organization (International Software Engineering Research Network) have all evolved in the last 10 years that focus solely on this area. As a further illustration of the growth of empirical software engineering, a search in the articles of 10 software engineering journals showed that the proportion of articles that used the term “empirical software engineering” d- bled from about 6% in 1997 to about 12% in 2006. While empirical software engineering has seen such substantial growth, there is not yet a reference book that describes advanced techniques for running studies and their application. This book aims to fill that gap. The chapters are written by some of the top international empirical software engineering researchers and focus on the practical knowledge necessary for conducting, reporting, and using empirical methods in software engineering. The book is intended to serve as a standard reference.

Inhaltsverzeichnis

Frontmatter

Research Methods and Techniques

Chapter 1. Software Engineering Data Collection for Field Studies

Software engineering is an intensely people-oriented activity, yet little is known about how software engineers perform their work. In order to improve software engineering tools and practice, it is therefore essential to conduct field studies, i.e., to study real practitioners as they solve real problems. To aid this goal, we describe a series of data collection techniques for such studies, organized around a taxonomy based on the degree to which interaction with software engineers is necessary. For each technique, we provide examples from the literature, an analysis of some of its advantages and disadvantages, and a discussion of special reporting requirements. We also talk briefly about recording options and data analysis.

Janice Singer, Susan E. Sim, Timothy C. Lethbridge

Chapter 2. Qualitative Methods

Software engineering involves a blend of non-technical as well as technical issues that often have to be taken into account in the design of empirical studies. In particular, the behavior of people is an integral part of software development and maintenance. This aspect of our subject presents complexities and challenges for the empirical researcher. In many other disciplines, qualitative research methods have been developed and are commonly used to handle the complexity of issues involving people performing tasks in their workplace. This chapter presents several qualitative methods for data collection and analysis and describes them in terms of how they might be incorporated into empirical studies of software engineering, in particular how they might be combined with quantitative methods. To illustrate this use of qualitative methods, examples from real software engineering studies are used throughout.

Carolyn B. Seaman

Chapter 3. Personal Opinion Surveys

Although surveys are an extremely common research method, surveybased research is not an easy option. In this chapter, we use examples of three software engineering surveys to illustrate the advantages and pitfalls of using surveys. We discuss the six most important stages in survey-based research: setting the survey’s objectives; selecting the most appropriate survey design; constructing the survey instrument (concentrating on self-administered questionnaires); assessing the reliability and validity of the survey instrument; administering the instrument; and, finally, analysing the collected data. This chapter provides only an introduction to survey-based research; readers should consult the referenced literature for more detailed advice.

Barbara A. Kitchenham, Shari L. Pfleeger

Chapter 4. The Focus Group Method as an Empirical Tool in Software Engineering

This chapter presents the focus group method and discusses its use for empirical research in the software engineering context. The background, process and main characteristics of the method are presented, as well as guidelines for its use. Moreover, the traditional as well computer-mediated focus group variations are compared to each other. The chapter concludes in with a discussion of the applicability of the method for software engineering research. In summary, the focus group method is a cost-effective and quick empirical research approach for obtaining qualitative insights and feedback from practitioners. It can be used in several phases and types of research. However, a major limitation of the method is that it is useful only in studying concepts that can be understood by knowledgeable participants in a limited time. We also emphasize the importance of empirical rigor when the method is used in scholarly work.

Jyrki Kontio, Johanna Bragge, Laura Lehtola

Chapter 5. Simulation Methods

This chapter aims to raise awareness about the usefulness and importance of simulation in support of software engineering. Simulation is applied in many critical engineering areas and enables one to address issues before they become problems. Simulation – in particular process simulation – is a state of the art technology to analyze process behaviour, risks and complex systems with their inherent uncertainties. Simulation provides insights into the designs of development processes and projects before significant time and cost has been invested, and can be of great benefit in support of training. The systematic combination of simulation methods with empirical research has the potential for becoming a powerful tool in applied software engineering research. The creation of virtual software engineering laboratories helps to allocate available resources of both industry and academia more effectively.

Mark Müller, Dietmar Pfahl

Practical Foundations

Chapter 6. Statistical Methods and Measurement

Useful ways of measuring software engineering phenomena have to address two challenges: defining realistic and valid metrics that can feasibly be collected under the constraints and time pressures of real-world software development contexts, and determining valid and accurate ways of analysing the resulting data to guide decisions. Too often, the difficulties of addressing the first challenge mean that the second is given little attention. The purpose of this chapter is to present different techniques for the definition and analysis of metrics such as product quality data. Specifically, statistical issues in the definition and application of metrics are presented with reference to software engineering examples.

Jarrett Rosenberg

Chapter 7. Missing Data in Software Engineering

The collection of valid software engineering data involves substantial effort and is not a priority in most software production environments. This often leads to missing or otherwise invalid data. This fact tends to be overlooked by most software engineering researchers and may lead to a biased analysis. This chapter reviews missing data methods and applies them on a software engineering data set to illustrate a variety of practical contexts where such techniques are needed and to highlight the pitfalls of ignoring the missing data problem.

Audris Mockus

Chapter 8. Reporting Experiments in Software Engineering

Background: One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) It is difficult to locate relevant information and (2) important information is often missing.

Objective: A guideline for reporting results from controlled experiments is expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results.

Method: The guideline for reporting is based on (1) a survey of the most prominent published proposals for reporting guidelines in software engineering and (2) an iterative development incorporating feedback from members of the research community.

Result: This chapter presents the unification of a set of guidelines for reporting experiments in software engineering.

Limitation: The guideline has not been evaluated broadly yet.

Conclusion: The resulting guideline provides detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical study, i.e., experiments (controlled experiments and quasi-experiments).

Andreas Jedlitschka, Marcus Ciolkowski, Dietmar Pfahl

Chapter 9. A Practical Guide to Ethical Research Involving Humans

The popularity of empirical methods in software engineering research is on the rise. Surveys, experiments, metrics, case studies, and field studies are examples of empirical methods used to investigate both software engineering processes and products. The increased application of such methods has also brought about an increase in discussions about adapting these methods to the particularities of software engineering. In contrast, the ethical issues raised by empirical methods have received little attention in the software engineering literature. In this chapter, we introduce four ethics principles of primary importance for conducting ethical research. We additionally discuss and provide examples of applying these principles in the context of ethics review.

Norman G. Vinson, Janice Singer

Chapter 10. The Management of University–Industry Collaborations Involving Empirical Studies of Software Enginee

In this chapter we will discuss some of the pragmatic considerations that we believe university researchers and companies should consider when establishing collaborative software engineering research projects; in particular, those involving empirical studies of software engineers. The chapter is illustrated using as a case study a research collaboration in which the authors are involved. We enumerate the costs, benefits, risks and risk-reducing factors that can have an impact on all the parties involved in the collaboration (the company, the faculty members and the graduate student researchers). Understanding this information is needed to help justify the research in the first place, and to manage it effectively. We then discuss many of the activities that will be needed to plan and manage the project, including such issues as attracting students, handling intellectual property, obtaining ethical approval and interacting with participants. The main objective of the chapter is to provoke some thoughts in the minds of those planning empirical research projects in software engineering.

Timothy C. Lethbridge, Steve Lyon, Peter Perry

Knowledge Creation

Chapter 11. Selecting Empirical Methods for Software Engineering Research

Selecting a research method for empirical software engineering research is problematic because the benefits and challenges to using each method are not yet well catalogued. Therefore, this chapter describes a number of empirical methods available. It examines the goals of each and analyzes the types of questions each best addresses. Theoretical stances behind the methods, practical considerations in the application of the methods and data collection are also briefly reviewed. Taken together, this information provides a suitable basis for both understanding and selecting from the variety of methods applicable to empirical software engineering.

Steve Easterbrook, Janice Singer, Margaret-Anne Storey, Daniela Damian

Chapter 12. Building Theories in Software Engineering

In mature sciences, building theories is the principal method of acquiring and accumulating knowledge that may be used in a wide range of settings. In software engineering, there is relatively little focus on theories. In particular, there is little use and development of empirically-based theories. We propose, and illustrate with examples, an initial framework for describing software engineering theories, and give advice on how to start proposing, testing, modifying and using theories to support both research and practise in software engineering.

Dag I. K. Sjøberg, Tore Dybå, Bente C. D. Anda, Jo E. Hannay

Chapter 13. Building Theories from Multiple Evidence Sources

As emphasized in other chapters of this book, useful results in empirical software engineering require a variety of data to be collected through different studies – focusing on a single context or single metric rarely tells a useful story. But, in each study, the requirements of the local context are liable to impose different constraints on study design, the metrics to be collected, and other factors. Thus, even when all the studies focus on the same phenomenon (say, software quality), such studies can validly collect a number of different measures that are not at all compatible (say, number of defects required to be fixed during development, number of problem reports received from the customer, total amount of effort that needed to be spent on rework). Can anything be done to build a useful body of knowledge from these disparate pieces? This chapter addresses strategies that have been applied to date to draw conclusions from across such varied but valid data sets. Key approaches are compared and the data to which they are best suited are identified. Our analysis together with associated lessons learned provide decision support for readers interested in choosing and using such approaches to build up useful theories.

Forrest Shull, Raimund L. Feldmann

Chapter 14. Replication's Role in Software Engineering

We provide motivation for researchers to replicate experiments in software engineering. The ideology of replication is discussed. We address the question: Is an experiment worth repeating? The current lack of replication studies is highlighted. We make clear that exact replication is unattainable and we draw on our first experience of performing an external replication. To categorise various kinds of replication, we propose a simple extension to Basili et al.’s framework for experimentation in software engineering. We present guidance as to the level of reported detail required to enable others perform a replication. Our conclusion is that there is only one route for empirical software engineering to follow: to make available laboratory packages of experimental materials to facilitate internal and external replications, especially the latter, which have greater confirming power.

A. Brooks, M. Roper, M. Wood, J. Daly, J. Miller

Backmatter

Titel: Guide to Advanced Empirical Software Engineering
herausgegeben von: Forrest Shull
Janice Singer
Dag I. K. Sjøberg
Verlag: Springer London
Electronic ISBN: 978-1-84800-044-5
Print ISBN: 978-1-84800-043-8
DOI: https://doi.org/10.1007/978-1-84800-044-5

Springer Professional