1 Introduction
2 Related Works
3 Background
-
Scenario
: Pop element
-
Given
a non-empty Stack
-
When
a stack has N elements
-
And
element E is on top of the stack
-
Then
a pop operation returns E
-
And
the new size of the stack is N-1
4 Experiment Definition
4.1 Experiment Participants
4.2 Experiment Material
-
Installation and configuration of Java Enterprise;
-
Installation and configuration of the standalone version of Fit wiki and its dependencies;
-
Installation and configuration of Eclipse IDE with the Cucumber plugin;
-
Quick-start examples to validate all development environments.
4.3 Hypothesis Definition
-
RQ1. Which of these acceptance tests techniques is easier to learn?
-
RQ2. Which of these acceptance tests techniques requires less effort (in time) to specify acceptance test scenarios?
-
RQ3. Which of these acceptance tests techniques is the best one to communicate software requirements, as a form of expressing consistent requirements?
-
H 0a the correctness of acceptance test scenarios specified by participants who attended a three-hour lecture about Fit tables and Gherkin language is the same for both acceptance testing techniques.
-
H 0b the effort to specify acceptance test scenarios is the same for both techniques.
-
H 0c the correctness of software functionalities specified using Fit tables and Gherkin language and implemented by the participants is the same using both acceptance testing techniques.
-
H 1a the correctness of acceptance test scenarios specified by participants who attended a three-hour lecture about Fit tables and Gherkin language is different when both acceptance testing techniques are used.
-
H 1b the effort to specify acceptance test scenarios using Fit tables is not the same when Gherkin language is used.
-
H 1c the correctness of software functionalities implemented by the participants is different when both acceptance testing techniques are used.
-
ATS#. Acceptance tests of requirement # were specified: {correctly, incorrectly};
-
ATST#. The participants need {?} minutes to specify acceptance test scenarios of requirement #;
-
AR#. The delivered source code implemented based on the acceptance test scenarios of requirement # is executable and it was accepted by the business stakeholder: {yes, no};
Id | Requirement |
---|---|
SR1 | Rectification of personnel profile information |
SR2 | Calculation of salary bonus per person |
SR3 | Exclusion of personnel profile information |
SR4 | Calculation the average of salary bonus per position |
4.4 Experiment Design
Participants | Objects and treatments | |||
---|---|---|---|---|
SR1
|
SR2
|
SR3
|
SR4
| |
Group A | (F) | (F) | – | – |
Group B | – | – | (G) | (G) |
-
(F) Software requirements specified as acceptance test scenarios using Fit Tables.
-
(G) Software requirements specified as acceptance test scenarios using Gherkin language.
Participants | Objects and treatments | |||
---|---|---|---|---|
SR1
|
SR2
|
SR3
|
SR4
| |
Group A | – | – | (G) | (G) |
Group B | (F) | (F) | – | – |
4.5 Training
-
a half-hour lecture about acceptance testing, TDD, ATDD, and BDD;
-
one-and-a-half-hour lecture about Fit tables and FitNesse, including how to configure this framework and practice exercises;
-
one-and-a-half-hour lecture about Gherkin language and Cucumber, including how to configure this framework and practice exercises.
4.6 Experiment Procedure
-
(1) Participants had 20 min to check if their environment to specify acceptance tests, which was previously set up in the training section, was working. In this step, they also answered the six first questions of the questionnaire.
-
(2) Participants read an overview of the HRS application and received a document with the description of two new requirements for this application.
-
(3) For each requirement:
-
(3.a) Participants filled the start time in their time sheets.
-
(3.b) Participants had to understand the requirements; if they had any doubt a business stakeholder was available to clarify them.
-
(3.c) Participants had to specify the requirement using the acceptance testing technique assign to them.
-
(3.d) When finished, participants had to mark the stop time on their time sheet.
-
(4) Then, participants had to answer the next eight questions of the questionnaire and to send the produced artifact to a web repository. The artifacts were identified by a random numeric id and only the researchers knew who uploaded them.
-
(5) An expert in acceptance testing evaluated all uploaded artifacts and marked the ones that were inconsistent or incomplete. This mark was visible only to the researchers.
-
(6) In the sixth step, the second part of our experiment was started. Participants had 20 min to check if their environment to develop the next tasks was working. After this, they had to download acceptance tests artifacts produced by a participant of the other group.
-
(7) Then, participants had to answer two questions in the questionnaire related to their view about the downloaded artifacts.
-
(8) The researchers had to verify which participants downloaded acceptance tests that, according to the expert evaluation, were incorrect. Then, they exchanged the incorrect acceptance tests by correct tests that express the same requirements using the same acceptance testing technique.
-
(9) Then, for each acceptance test scenario (requirement):
-
(9.a) Participants had to fill the start time in their time sheets.
-
(9.b) Participants had to understand by themselves the acceptance test.
-
(9.c) Participants had to develop the new requirement of the HRS application expressed by the acceptance tests.
-
(9.d) When finished, participants had to mark the stop time on their time sheet.
-
(10) Then, participants had to answer the next eight questions of the questionnaire and to send the produced source code to a web repository. The artifacts were identified by a random numeric id, and only researchers knew who uploaded them.
5 Results and Data Analysis
5.1 Consistency and Correctness of the Acceptance Test Scenarios
Acceptance test scenarios were specified: | ||
---|---|---|
Treatment | Correctly | Incorrectly |
Fit Tables (T) | 17 | 1 |
Gherkin Language (G) | 13 | 5 |
5.2 Time to Complete Acceptance Test Scenarios Specifications
Treatment | Time list (in minutes) |
---|---|
Fit tables (F) | {20, 21, 21, 23, 35, 36, 40, 45, 50, 65, 66, 77, 79, 88, 108, 120, 126, 135} |
Gherkin Language (G) | {15, 16, 17, 15, 39, 30, 30, 30, 45, 35, 60, 28, 40, 40, 70, 57, 84, 75} |
5.3 Applicability of Acceptance Test Scenarios to Communicate Software Requirements
The delivered source code implemented based on the acceptance test scenarios is executable and it was accepted by the business stakeholder: | ||
---|---|---|
Treatment | Yes | No |
Fit Tables (T) | 13 | 5 |
Gherkin Language (G) | 10 | 8 |
5.4 Experiment Questionnaire
6 Threats to the Validity
-
using the acceptance tests specified by an expert as input for part 2, avoiding that some mistakes in the acceptance test scenarios created by other participants were unnoticed by the expert;
-
using automated acceptance tests, or even JUnit tests, to validate if the code implemented by the participants meets the user needs.