Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Empirical Software Engineering
Published in: Empirical Software Engineering 6/2021

01-11-2021

Empirical evaluation of tools for hairy requirements engineering tasks

Author: Daniel M. Berry

Published in: Empirical Software Engineering | Issue 6/2021

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Context

A hairy requirements engineering (RE) task involving natural language (NL) documents is (1) a non-algorithmic task to find all relevant answers in a set of documents, that is (2) not inherently difficult for NL-understanding humans on a small scale, but is (3) unmanageable in the large scale. In performing a hairy RE task, humans need more help finding all the relevant answers than they do in recognizing that an answer is irrelevant. Therefore, a hairy RE task demands the assistance of a tool that focuses more on achieving high recall, i.e., finding all relevant answers, than on achieving high precision, i.e., finding only relevant answers. As close to 100% recall as possible is needed, particularly when the task is applied to the development of a high-dependability system. In this case, a hairy-RE-task tool that falls short of close to 100% recall may even be useless, because to find the missing information, a human has to do the entire task manually anyway. On the other hand, too much imprecision, too many irrelevant answers in the tool’s output, means that manually vetting the tool’s output to eliminate the irrelevant answers may be too burdensome. The reality is that all that can be realistically expected and validated is that the recall of a hairy-RE-task tool is higher than the recall of a human doing the task manually.

Objective

Therefore, the evaluation of any hairy-RE-task tool must consider the context in which the tool is used, and it must compare the performance of a human applying the tool to do the task with the performance of a human doing the task entirely manually, in the same context. The context of a hairy-RE-task tool includes characteristics of the documents being subjected to the task and the purposes of subjecting the documents to the task. However, traditionally, many a hairy-RE-task tool has been evaluated by considering only (1) how high is its precision, or (2) how high is its F-measure, which weights recall and precision equally, ignoring the context, and possibly leading to incorrect — often underestimated — conclusions about how effective it is.

Method

To evaluate a hairy-RE-task tool, this article offers an empirical procedure that takes into account not only (1) the performance of the tool, but also (2) the context in which the task is being done, (3) the performance of humans doing the task manually, and (4) the performance of those vetting the tool’s output. The empirical procedure uses (I) on one hand, (1) the recall and precision of the tool, (2) a contextually weighted F-measure for the tool, (3) a new measure called summarization of the tool, and (4) the time required for vetting the tool’s output, and (II) on the other hand, (1) the recall and precision achievable by and (2) the time required by a human doing the task.

Results

The use of the procedure is shown for a variety of different contexts, including that of successive attempts to improve the recall of an imagined hairy-RE-task tool. The procedure is shown to be context dependent, in that the actual next step of the procedure followed in any context depends on the values that have emerged in previous steps.

Conclusion

Any recommendation for a hairy-RE-task tool to achieve close to 100% recall comes with caveats and may be required only in specific high-dependability contexts. Appendix C applies parts of this procedure to several hairy-RE-task tools reported in the literature using data published about them. The surprising finding is that some of the previously evaluated tools are actually better than they were thought to be when they were evaluated using mainly precision or an unweighted F-measure.
previous article FACER: An API usage-based code-example recommender for opportunistic reuse
next article What do class comments tell us? An investigation of comment evolution and practices in Pharo Smalltalk

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now
Appendix
Available only for authorised users
Footnotes
1
I chose the word “hairy” to evoke the metaphor of the hairy theorem or proof.
 
2
An algorithmic task is one for which there exists an algorithm whose output is guaranteed logically, as opposed to empirically, to be 100% correct. For example, searching for every occurrence of the word “only” is algorithmic; think grep. However, searching for each occurrence of “only” that is in the wrong position for its intended meaning is non-algorithmic.
 
3
The phrase “this article” is used to refer to only the document that you are reading right now, to distinguish it from the document of any related work, for which the word “paper” is used.
 
4
Later, this article defines “close to” in terms of what humans can actually achieve. Until the background has been laid for this definition, a vague understanding suffices, but “close to 100%” is definitely less than 100%. Nowhere does this article insist on achieving 100% correctness, recall, or precision!
 
5
“Recalling” is to “recall” as “precise” is to “precision”.
 
6
This used to be called the ‘humanly achievable high recall (HAHR)”, expressing the hope that it is close to 100%. However, actual values have proved to be quite low, sometimes as low as 32.95%.
 
7
Of course, one can argue that such a tool is useful as a defense against a human’s less-than-100% recall when the tool is run as a double check after the human has done the tool’s task manually. However, it seems to me, that if the human knows that the tool will be run, he or she might be lazy in carrying out the manual task and not do as well as possible. Empirical studies are needed to see if this effect is is real, and if so, how destructive it is of the human’s recall. Perhaps, the effect is technology-induced inattentional blindness (Mack and Rock 1998), which has been studied in depth elsewhere. Winkler and Vogelsang may have found evidence for this effect (Winkler and Vogelsang 2018).
 
8
The word “I” is used when describing activities that required thinking and a decision on my part to advance the research. The data speak for themselves. “I” is less pretentious than “this author”. Also “this author” can be ambiguous in the context of a discussion about a related work’s authors.
 
9
This sentence says only “probably”. Section 7 explains how to empirically decide how accurate this prediction is.
 
10
If a tool can be shown to always have 100% precision, it is never necessary to vet the tool’s output.
 
11
Of course, any t for which this property does not hold could be considered to be poorly designed!
 
12
Think: “speedup” is roughly “acceleration”; hence “a”.
 
13
The HAR is realistically the best that can be expected. However, see Item 13 in Section 7.2 for an idea about increasing HAR beyond the what is possible by one human being.
 
14
A discussion of which formula for the F-measure is more appropriate to evaluate tools for hairy tasks is outside the scope of this article.
 
15
on the assumption that the time required for a run of t is negligible or other work can be done while t is running on its own.
 
16
Section 7 explains how to determine empirically if this claim is true for any T and t.
 
17
Whenever “X et al” would be ambiguous w.r.t. the bibliography of this article, the full list of author family names starting with X is given. If in addition, there would be several occurrences of that long list in the following text, an acronym is introduced to be used in place of those occurrences. If on the other hand, the intent of “X et al” is to refer to all works, with possibly different author lists, that are referred to by “X et al”, then “X et al” is used. There are occasional local, footnote-marked-and-explained exceptions to these rules.
 
18
The minimum possible precision is λ, which is positive if the document has at least one relevant answer.
 
19
There are many other NLP measures that are called by the same name, but this measure is not any of them.
 
20
As pointed out by an anonymous reviewer, it may not be helpful for the tool to include with a candidate link, information about why its IR method decided that the link is a candidate, particularly if the method, as does LSI (Latent Semantic Indexing), computes the similarity of the link’s source and target in the space of concepts rather than in the more traditional space of terms.
 
21
I learned from reviews of previous versions of this article, that presenting the formula-laden description of the evaluation procedure of Section 7 without prefacing it with example evaluations using the procedure loses too many readers, including me. On the other hand, some readers understand examples applying formulae only after reading and understanding the formulae. If you are one of the latter kind of reader, please read Section 7 before reading Section 6.
 
22
I had to concoct a fictitious analysis, because I have no actual complete analysis. I myself realized what is needed to do a complete context-aware evaluation of a tool for an NL RE task only after the last evaluation in which I participated was completed, and in general, it is impossible to obtain several key data after the evaluation steps are completed. In addition, to date, to my knowledge, no one has published a full analysis of the kind prescribed in this article, probably because doing this kind of analysis is very much not traditional.
 
23
The reason that these recalls are only approximate is that the number of answers in a tool’s output must be a whole integer!
 
24
For a list of all identifications used throughout this article, see Appendix A.
 
25
The comparison of \(R_{\textit {vet},D,T,t_{1}}\) and the HAR, RaveExpert, D, T, should be accompanied by the calculation of a p-value, which allows determining the statistical significance of the whatever difference there is between the two values, each of which is, after all, the average of the R values of the members of a group of domain experts. The same holds for all comparisons like this one. See Section 7.4.
 
26
All the standard issues concerning the selection of a representative document with which to test a tool for at task T and concerning the internal and external validity of an evaluation based on a gold standard are outside the scope of this article, for no other reason than that these issues have been studied thoroughly elsewhere (Manning et al. 2008). Nevertheless, the threats arising from these issues are discussed in Section 7.5.
 
27
“Vel cetera” is to “inclusive or” as “et cetera” is to “and”.
 
28
Here and here alone, “Maalej et al.” is used because Maalej is the one person who is an author in all of the cited work.
 
29
“Nocuous” is the opposite of “innocuous”.
 
30
“Hayes, Dekhtyar, et al.” is used to refer to all the tracing work done by Hayes, Dekhtyar, and various colleagues and students over the years, as a single body of work, while an acronym, e.g., “HDLG”, consisting of the ordered initials of the family names of the authors of a paper, is used to refer to the work of the paper featured in a section, e.g, that cited by the reference “(Hayes et al. 2018)”.
 
31
This observation calls to mind the famous joke: “A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, ‘this is where the light is’.” (Freedman 2010)
 
32
The formula for F1 that they give in their paper lacks the multiplier 2 in the numerator. So, one would expect the reported values to be one half of what they should be, but the reported values lie, correctly, between those of recall and precision. Moreover, they match the values that I computed for building Table 16. Therefore, they probably used the correct formula to compute the F1 values.
 
33
There are not enough data in the paper to estimate either β from the construction of the gold standard.
 
34
In the quotation, “accuracy” seems to mean “average F2”, because average F2 is the only value that is maintained in moving from the fourth to the fifth row. Each has an average F2 of 0.82.
 
Literature
Anish PR, Ghaisas S (2014) Product knowledge configurator for requirements gap analysis and customizations. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 437–443
Antoniol G, Canfora G, Casazza G, De Lucia A (2000) Identifying the starting impact set of a maintenance request: a case study. In: Proceedings of the fourth European conference on software maintenance and reengineering, pp 227–230
Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983CrossRef
Arora C, Sabetzadeh M, Briand L, Zimmer F (2015) Automated checking of conformance to requirements templates using natural language processing. IEEE Trans Softw Eng 41(10):944–968CrossRef
Beard M, Kraft N, Etzkorn L, Lukins S (2011) Measuring the accuracy of information retrieval based bug localization techniques. In: 18th working conference on reverse engineering (WCRE), pp 124–128
Berry DM (2017) Evaluation of tools for hairy requirements and software engineering tasks. In: Proceedings of workshop on empirical requirements engineering (EmpirRE) in IEEE 25th international requirements engineering conference workshops, pp 284–291
Berry DM, Cleland-Huang J, Ferrari A, Maalej W, Mylopoulos J, Zowghi D (2017) Panel: context-dependent evaluation of tools for NL RE tasks: recall vs. precision, and beyond. In: IEEE 25th international requirements engineering conference (RE), pp 570–573
Berry DM, Gacitua R, Sawyer P, Tjong SF (2012) The case for dumb requirements engineering tools. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 211–217
Binkley D, Lawrie D (2010) Information retrieval applications in software maintenance and evolution. In: Laplante PA (ed) Encyclopedia of software engineering, pp 454–463. Taylor & Francis
Breaux TD, Gordon DG (2013) Regulatory requirements traceability and analysis using semi-formal specifications. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 141–157
Breaux TD, Schaub F (2014) Scaling requirements extraction to the crowd: experiments with privacy policies. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 163–172
Bucchiarone A, Gnesi S, Pierini P (2005) Quality analysis of NL requirements: an industrial case study. In: Proc. 13th IEEE international requirements engineering conference (RE), pp 390–394
Casamayor A, Godoy D, Campo M (2012) Functional grouping of natural language requirements for assistance in architectural software design. Knowledge-Based Systs 30:78–86CrossRef
Cavalcanti G, Borba P, Accioly P (2017) Should we replace our merge tools?. In: Proceedings of the 39th international conference on software engineering companion (ICSE-C), pp 325–327
Chantree F, Nuseibeh B, de Roeck A, Willis A (2006) Identifying nocuous ambiguities in natural language requirements. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 56–65
Cleland-Huang J, Czauderna A, Gibiec M, Emenecker J (2010) A machine learning approach for tracing regulatory codes to product specific requirements. In: Proceedings of the international conference on software engineering (ICSE), pp 155–164
Cleland-Huang J, Gotel O, Zisman A (eds) (2012) Software and systems traceability. Springer, London
Cleland-Huang J, Zemont G, Lukasik W (2004) A heterogeneous solution for improving the return on investment of requirements traceability. In: Proceedings of the 12th IEEE international requirements engineering conference (RE), pp 230–239
Cuddeback D, Dekhtyar A, Hayes JH (2010) Automated requirements traceability: the study of human analysts. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 231–240
De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artifact management systems using information retrieval methods. ACM Transactions on Software Engineering Methodology 16 (4):13,1–13,50CrossRef
De Lucia A, Marcus A, Oliveto R, Poshyvanyk D (2012) Information retrieval methods for automated traceability recovery. In: Cleland-Huang J, Gotel O, Zisman A (eds) Software and systems traceability. Springer, pp 71–98
De Lucia A, Oliveto R, Tortora G (2008) IR-based traceability recovery processes: an empirical comparison of “one-shot” and incremental processes. In: 26th IEEE/ACM international conference on automated software engineering (ASE), pp 39–48, Los Alamitos, CA, USA. IEEE Computer Society
De Lucia A, Oliveto R, Tortora G (2009) Assessing IR-based traceability recovery tools through controlled experiments. Empir Softw Eng 14(1):57–92CrossRef
De Lucia A, Oliveto R, Tortora G (2009) The role of the coverage analysis during IR-based traceability recovery: a controlled experiment. In: IEEE international conference on software maintenance (ICSM), pp 371–380
Dekhtyar A, Hayes JH, Smith M (2011) Towards a model of analyst effort for traceability research. In: Proceedings of the 6th international workshop on traceability in emerging forms of software engineering (TEFSE), pp 58–62
Delater A, Paech B (2013) Tracing requirements and source code during software development: an empirical study. In: Proceedings of the international symposium on empirical software engineering and measurement (ESEM), pp 25–34
Dwarakanath A, Ramnani RR, Sengupta S (2013) Automatic extraction of glossary terms from natural language requirements. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 314–319
Fabbrini F, Fusani M, Gnesi S, Lami G (2001) An automatic quality evaluation for natural language requirements. In: Requirements engineering: foundation for software quality (REFSQ), pp 1–18
Ferrari A, Dell’Orletta F, Spagnolo GO, Gnesi S (2014) Measuring and improving the completeness of natural language requirements. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 23–38
Freedman DH (2010) Wrong: why experts keep failing us — and how to know when not to trust them. Little, Brown and Company, New York
Gacitua R, Sawyer P (2008) Ensemble methods for ontology learning - an empirical experiment to evaluate combinations of concept acquisition techniques. In: ICIS’2008, pp 328–333
Gacitua R, Sawyer P, Gervasi V (2010) On the effectiveness of abstraction identification in requirements engineering. In: Proceedings of the 18th IEEE international requirements engineering conference (RE), pp 5–14
Gervasi V, Zowghi D (2014) Supporting traceability through affinity mining. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 143–152
Gleich B, Creighton O, Kof L (2010) Ambiguity detection: towards a tool explaining ambiguity sources. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 218–232
Goldin L, Berry DM (1997) Abstfinder: a prototype abstraction finder for natural language text for use in requirements elicitation. Autom Softw Eng 4:375–412CrossRef
Gotel O, Cleland-Huang J, Hayes JH, Zisman A, Egyed A, Grünbacher P, Antoniol G (2012) The quest for ubiquity: a roadmap for software and systems traceability research. In: 20th IEEE international requirements engineering conference (RE), pp 71–80
Groen EC, Schowalter J, Kopczyńska S, Polst S, Alvani S (2018) Is there really a need for using NLP to elicit requirements? A benchmarking study to assess scalability of manual analysis. In: Schmid K, Spoletini P (eds) Joint proceedings of the REFSQ 2018 Co-located events: the workshop on natural language processing for RE (NLP4RE), pp 1–11. CEUR Workshop Proceedings 2075. http://​ceur-ws.​org/​Vol-2075/​NLP4RE_​paper11.​pdf
Guzman E, Maalej W (2014) How do users like this feature? A fine grained sentiment analysis of app reviews. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 153–162
Hayes JH, Dekhtyar A (2005) Humans in the traceability loop: can’t live with ’em, can’t live without ’em. In: Proceedings of the 3rd international workshop on traceability in emerging forms of software engineering (TEFSE), pp 20–23
Hayes JH, Dekhtyar A, Larsen J, Guéhéneuc Y-G (2018) Effective use of analysts’ effort in automated tracing. Requirements Engineering Journal 23(1):119–143CrossRef
Hayes JH, Dekhtyar A, Osborne J (2003) Improving requirements tracing via information retrieval. In: Proceedings of the 11th IEEE international requirements engineering conference (RE), pp 138–147
Hayes JH, Dekhtyar A, Sundaram SK (2006) Advancing candidate link generation for requirements tracing: the study of methods. IEEE Trans Softw Eng 32(1):4–19CrossRef
Hayes JH, Dekhtyar A, Sundaram SK, Howard S (2004) Helping analysts trace requirements: an objective look. In: Proceedings of the 12th IEEE international requirements engineering conference (RE), pp 249–259
Heindl M, Biffl S (2005) A case study on value-based requirements tracing. In: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on foundation for software engineering ESEC/FSE, pp 60–69
Hübner P, Paech B (2017) Using interaction data for continuous creation of trace links between source code and requirements in issue tracking systems. In: Proceedings of the 23rd international working conference on requirements engineering: foundation for software quality (REFSQ), pp TBD
Ingram C, Riddle S (2012) Cost-benefits of traceability. In: Cleland-Huang J, Gotel O, Zisman A (eds) Software and systems traceability, pp 23–42. Springer
Jha N, Mahmoud A (2017) Mining user requirements from application store reviews using frame semantics. In: Grünbacher P, Perini A (eds) Proceedings of the 23rd international working conference on requirements engineering: foundation for software quality (REFSQ), pp 273–287. Springer
Jha N, Mahmoud A (2018) Using frame semantics for classifying and summarizing application store reviews. Empir Softw Eng 23(6):3734–3767CrossRef
Jha N, Mahmoud A (2019) Mining non-functional requirements from App store reviews. Empir Softw Eng 24(6):3659–3695CrossRef
Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, Norwell
Knauss E, Ott D (2014) Semi-automatic categorization of natural language requirements. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 39–54
Kohavi R, Provost F (1998) Glossary of terms. Mach Learn 30 (2):271–274
Kong W, Hayes JH, Dekhtyar A, Dekhtyar O (2012) Process improvement for traceability: a study of human fallibility. In: 20th IEEE international requirements engineering conference (RE), pp 31–40
Kong W, Hayes JH, Dekhtyar A, Holden J (2011) How do we trace requirements: an initial study of analyst behavior in trace validation tasks. In: Proceedings of the international workshop on cooperative and human aspects of software engineering (CHASE), pp 32–39
Li H (2011) Learning to rank for information retrieval and natural language processing. Morgan & Claypool Publishers
Maalej W (2017) In-person, verbal communication
Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Requirements Engineering Journal 21(3):311–331CrossRef
Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Reqs Engg J 21(3):311–331CrossRef
Mäder P, Egyed A (2015) Do developers benefit from requirements traceability when evolving and maintaining a software system? Empir Softw Eng 20 (2):413–441CrossRef
Mahmoud A, Niu N (2014) Supporting requirements to code traceability through refactoring. Requirements Engineering Journal 19(3):309–329CrossRef
Marcus A, Haiduc S (2013) Text retrieval approaches for concept location in source code. In: De Lucia A, Ferrucci F (eds) Software engineering: international summer schools (ISSSE), 2009–2011, Salerno, Italy. Revised Tutorial Lectures, pp 126–158. Springer, Berlin
Maro S, Steghöfer J, Hayes J, Cleland-Huang J, Staron M (2018) Vetting automatically generated trace links: what information is useful to human analysts?. In: IEEE 26th international requirements engineering conference (RE), pp 52–63
Menzies T, Dekhtyar A, Distefano J, Greenwald J (2007) Problems with precision: a response to comments on data mining static code attributes to learn defect predictors. IEEE Transactions on Software Enginerring 33(9):637–640CrossRef
Merten T, Krämer D, Mager B, Schell P, Bürsner S, Paech B (2016) Do information retrieval algorithms for automated traceability perform effectively on issue tracking system data?. In: Proceedings of the 22nd international working conference on requirements engineering: foundation for software quality (REFSQ), pp 45–62
Montgomery L, Damian D (2017) What do support analysts know about their customers? on the study and prediction of support ticket escalations in large software organizations. In: Proceedings of the 25th IEEE international requirements engineering conference (RE), page to appear
Mori K, Okubo N, Ueda Y, Katahira M, Amagasa T (2020) Joint proceedings of REFSQ-2020 workshops, doctoral symposium, live studies track, and poster track. In: Sabetzadeh M, Vogelsang A, Abualhaija S, Borg M, Dalpiaz F, Daneva M, Fernández N, Franch X, Fucci D, Gervasi V, Groen E, Guizzardi R, Herrmann A, Horkoff J, Mich L, Perini A, Susi A (eds). http://​ceur-ws.​org/​Vol-2584/​NLP4RE-paper2.​pdf
Nagappan M (2018) In-person, verbal communication
Nikora AP, Hayes JH, Holbrook EA (2010) Experiments in automated identification of ambiguous natural-language requirements. In: Proceedings of the international symposium on software reliability engineering, pp 229–238
Oliveto R, Gethers M, Poshyvanyk D, De Lucia A (2010) On the equivalence of information retrieval methods for automated traceability link recovery. In: IEEE 18th international conference on program comprehension (ICPC), pp 68–71
Pagano D, Maalej W (2013) User feedback in the appstore: an empirical study. In: Proceedings of the 21st IEEE international requirements engineering conference (RE), pp 125–134
Pittke F, Leopold H, Mendling J (2015) Automatic detection and resolution of lexical ambiguity in process models. IEEE Trans Softw Eng 41(6):526–544CrossRef
Pollock L, Vijay-Shanker K, Hill E, Sridhara G, Shepherd D (2013) Natural language-based software analyses and tools for software maintenance. In: De Lucia A, Ferrucci F (eds) Software engineering: international summer schools (ISSSE), 2009–2011, Salerno, Italy. Revised Tutorial Lectures, pp 126–158. Springer, Berlin
Poshyvanyk D, Marcus A (2007) Combining formal concept analysis with information retrieval for concept location in source code. In: Proceedings of the 15th IEEE international conference on program comprehension (ICPC), pp 37–48
Quirchmayr T, Paech B, Kohl R, Karey H (2017) Semi-automatic software feature-relevant information extraction from natural language user manuals. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 255–272
Rahimi M, Mirakhorli M, Cleland-Huang J (2014) Automated extraction and visualization of quality concerns from requirements specifications. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 253–262
Rempel P, Mäder P (2017) Preventing defects: the impact of requirements traceability completeness on software quality. IEEE Trans Softw Eng 43 (8):777–797CrossRef
Riaz M, King JT, Slankas J, Williams LA (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 183–192
Robeer M, Lucassen G, van der Werf JME, Dalpiaz F, Brinkkemper S (2016) Automated extraction of conceptual models from user stories via NLP. In: Int. Reqs. Engg. Conf. (RE), pp 196–205
Ryan K (1993) The role of natural language in requirements engineering. In: Proceedings of the IEEE international symposium on requirements engineering, pp 240–242
Saito S, Iimura Y, Tashiro H, Massey AK, Antón AI (2016) Visualizing the effects of requirements evolution. In: Proceedings of the international conference on software engineering (ICSE) companion, pp 152–161
Saracevic T (1995) Evaluation of evaluation in information retrieval. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR), pp 138–146
Sutcliffe A, Rayson P, Bull CN, Sawyer P (2014) Discovering affect-laden requirements to achieve system acceptance. In: Proceedings of the IEEE international requirements engineering conference (RE), pp 173–182
Tjong SF, Berry DM (2013) The design of SREE — a prototype potential ambiguity finder for requirements specifications and lessons learned. In: Proceedings of the international working conference on requirements engineering: foundation of software quality (REFSQ), pp 80–95
Wang L, Nakagawa H, Tsuchiya T (2020) Opinion analysis and organization of mobile application user reviews. In: Sabetzadeh M, Vogelsang A, Abualhaija S, Borg M, Dalpiaz F, Daneva M, Fernández N, Franch X, Fucci D, Gervasi V, Groen E, Guizzardi R, Herrmann A, Horkoff J, Mich L, Perini A, Susi A (eds) Joint proceedings of REFSQ-2020 workshops, doctoral symposium, live studies track, and poster track. http://​ceur-ws.​org/​Vol-2584/​NLP4RE-paper4.​pdf
Wilson WM, Rosenberg LH, Hyatt LE (1997) Automated analysis of requirement specifications. In: Proc. 19th international conference on software engineering (ICSE), pp 161–171
Winkler JP, Grönberg J, Vogelsang A (2019) Optimizing for recall in automatic requirements classification: an empirical study. In: 27th IEEE international requirements engineering conference (RE), pp 40–50
Winkler JP, Vogelsang A (2018) Using tools to assist identification of non-requirements in requirements specifications — a controlled experiment. In: Kamsties E, Horkoff J, Dalpiaz F (eds) Proceedings of requirements engineering: foundation for software quality (REFSQ), pp 57–71
Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Burlington
Wnuk K, Höst M, Regnell B (2012) Replication of an experiment on linguistic tool support for consolidation of requirements from multiple sources. Empirical Softw Engg 17(3):305–344CrossRef
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell
Yang H, Roeck AND, Gervasi V, Willis A, Nuseibeh B (2011) Analysing anaphoric ambiguity in natural language requirements. Requirements Engineering Journal 16(3):163–189CrossRef
Yazdani Seqerloo A, Amiri MJ, Parsa S, Koupaee M (2019) Automatic test cases generation from business process models. Requirements Engineering Journal 24(1):119–132CrossRef
Zeni N, Kiyavitskaya N, Mich L, Cordy JR, Mylopoulos J (2015) Gaiust: supporting the extraction of rights and obligations for regulatory compliance. Requirements Engineering Journal 20(1):1–22CrossRef
Metadata
Title
Empirical evaluation of tools for hairy requirements engineering tasks
Author
Daniel M. Berry
Publication date
01-11-2021
Publisher
Springer US
Published in
Empirical Software Engineering / Issue 6/2021
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI
https://doi.org/10.1007/s10664-021-09986-0

Other articles of this Issue 6/2021

Empirical Software Engineering 6/2021 Go to the issue

OriginalPaper

Applying test case prioritization to software microbenchmarks

OriginalPaper

Maintenance-related concerns for post-deployed Ethereum smart contract development: issues, techniques, and future challenges

OriginalPaper

A study of how Docker Compose is used to compose multi-component systems

OriginalPaper

How to cherry pick the bug report for better summarization?

OriginalPaper

FACER: An API usage-based code-example recommender for opportunistic reuse

Journal First Paper

How are project-specific forums utilized? A study of participation, content, and sentiment in the Eclipse ecosystem

Premium Partner

    Image Credits