Mika V. Mäntylä
ABSTRACT
Time pressure is prevalent in the software industry in which shorter and shorter deadlines and high customer demands lead to increasingly tight deadlines. However, the effects of time pressure have received little attention in software engineering research. We performed a controlled experiment on time pressure with 97 observations from 54 subjects. Using a two-by-two crossover design, our subjects performed requirements review and test case development tasks. We found statistically significant evidence that time pressure increases efficiency in test case development (high effect size Cohen’s d=1.279) and in requirements review (medium effect size Cohen’s d=0.650). However, we found no statistically significant evidence that time pressure would decrease effectiveness or cause adverse effects on motivation, frustration or perceived performance. We also investigated the role of knowledge but found no evidence of the mediating role of knowledge in time pressure as suggested by prior work, possibly due to our subjects. We conclude that applying moderate time pressure for limited periods could be used to increase efficiency in software engineering tasks that are well structured and straight forward.
- Molokken,K. and Jorgensen,M., "A review of software surveys on software effort estimation," Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 International Symposium on, 2003, pp. 223-230. Google Scholar
Digital Library
- Wirth,N., "A plea for lean software," Computer, vol. 28, no. 2, 1995, pp. 64-68. Google ScholarDigital Library
- Austin,R.D., "The effects of time pressure on quality in software development: An agency model," Information Systems Research, vol. 12, no. 2, 2001, pp. 195-207. Google ScholarDigital Library
- Costello,S.H., "Software engineering under deadline pressure," ACM SIGSOFT Software Engineering Notes, vol. 9, no. 5, 1984, pp. 15-19. Google ScholarDigital Library
- Baddoo,N. and Hall,T., "De-motivators for software process improvement: an analysis of practitioners’ views," J.Syst.Software, vol. 66, no. 1, 2003, pp. 23-33. Google ScholarDigital Library
- Glass,R.L., "LOYAL OPPOSITION: Project Retrospectives, and Why They Never Happen," IEEE Software, vol. 19, no. 5, 2002, pp. 112-111. Google ScholarDigital Library
- Juristo,N. and Vegas,S., "Using differences among replications of software engineering experiments to gain knowledge," Empirical Software Engineering and Measurement, 2009. ESEM 2009. 3rd International Symposium on, 2009, pp. 356-366. Google ScholarDigital Library
- Sonnentag,S., Brodbeck,F.C., Heinbokel,T. and Stolte,W., "Stressor‐burnout relationship in software development teams," J.Occup.Organ.Psychol., vol. 67, no. 4, 1994, pp. 327-341.Google ScholarCross Ref
- Mäntylä,M.V., Khomh,F., Adams,B., Engström,E. and Petersen,K., "On Rapid Releases and Software Testing," International Conference on Software Maintenance, 2013, pp. 1- 10.Google Scholar
- Shah,H., Harrold,M.J. and Sinha,S., "Global software testing under deadline pressure: Vendor-side experiences," Information and Software Technology, no. in Press, Google ScholarDigital Library
- Nan,N. and Harter,D.E., "Impact of budget and schedule pressure on software development cycle time and effort," Software Engineering, IEEE Transactions On, vol. 35, no. 5, 2009, pp. 624-637. Google ScholarDigital Library
- Mäntylä,M.V. and Itkonen,J., "More testers – The effect of crowd size and time restriction in software testing," Information and Software Technology, vol. 55, no. 6, 2013, pp. 986-1003. Google ScholarDigital Library
- Jørgensen,M. and Sjøberg,D.I., "Impact of effort estimates on software project work," Information and Software Technology, vol. 43, no. 15, 2001, pp. 939-948.Google ScholarCross Ref
- Glover,S.M., "The influence of time pressure and accountability on auditors' processing of nondiagnostic information," Journal of Accounting Research, vol. 35, no. 2, 1997, pp. 213-226.Google ScholarCross Ref
- Beilock,S.L., Bertenthal,B.I., Hoerger,M. and Carr,T.H., "When does haste make waste? Speed-accuracy tradeoff, skill level, and the tools of the trade." Journal of Experimental Psychology: Applied, vol. 14, no. 4, 2008, pp. 340.Google ScholarCross Ref
- McDaniel,L.S., "The effects of time pressure and audit program structure on audit performance," Journal of Accounting Research, vol. 28, no. 2, 1990, pp. 267-285.Google ScholarCross Ref
- Topi,H., Valacich,J.S. and Hoffer,J.A., "The effects of task complexity and time availability limitations on human performance in database query tasks," International Journal of Human-Computer Studies, vol. 62, no. 3, 2005, pp. 349-379. Google ScholarDigital Library
- Fogelström,N.D. and Gorschek,T., "Test-case Driven versus Checklist-based Inspections of Software Requirements–An Experimental Evaluation," WER07-Workshop em Engenharia de Requisitos, Toronto, Canada, 2007, pp. 116-126.Google Scholar
- Kocher,M.G. and Sutter,M., "Time is money—Time pressure, incentives, and the quality of decision-making," Journal of Economic Behavior & Organization, vol. 61, no. 3, 2006, pp. 375-392.Google ScholarCross Ref
- Calderwood,R., Klein,G.A. and Crandall,B.W., "Time pressure, skill, and move quality in chess," Am.J.Psychol., 1988, pp. 481-493.Google ScholarCross Ref
- Sweller,J., "Cognitive load theory, learning difficulty, and instructional design," Learning and Instruction, vol. 4, no. 4, 1994, pp. 295-312.Google ScholarCross Ref
- Spilker,B.C., "The effects of time pressure and knowledge on key word selection behavior in tax research," Accounting Review, 1995, pp. 49-70.Google Scholar
- Wilson,D.N. and Hall,T., "Perceptions of software quality: a pilot study," Software Quality Journal, vol. 7, no. 1, 1998, pp. 67-75. Google ScholarDigital Library
- Linzer,M., Konrad,T.R., Douglas,J., McMurray,J.E., Pathman,D.E., Williams,E.S., Schwartz,M.D., Gerrity,M., Scheckler,W. and Bigby,J., "Managed care, time pressure, and physician job satisfaction: results from the physician worklife study," Journal of General Internal Medicine, vol. 15, no. 7, 2000, pp. 441-450.Google ScholarCross Ref
- Sjøberg,D., Hannay,J., Hansen,O., Kampenes,V., Karahasanovic,A., Liborg,N.K. and Rekdal,A., "A survey of controlled experiments in software engineering," IEEE Trans.Software Eng., vol. 31, no. 9, 2005, pp. 733-753. Google ScholarDigital Library
- Fry,Z.P. and Weimer,W., "A human study of fault localization accuracy," Software Maintenance (ICSM), 2010 IEEE International Conference on, 2010, pp. 1-10. Google ScholarDigital Library
- Kemerer,C.F. and Paulk,M.C., "The impact of design and code reviews on software quality: An empirical study based on PSP data," IEEE Trans.Software Eng., vol. 35, no. 4, 2009, pp. 534-550. Google ScholarDigital Library
- Walia,G.S., Carver,J.C. and Nagappan,N., "The effect of the number of inspectors on the defect estimates produced by capture-recapture models," Proceedings of the 30th international conference on Software engineering, 2008, pp. 331- 340. Google ScholarDigital Library
- Maldonado,J.C., Carver,J., Shull,F., Fabbri,S., Dória,E., Martimiano,L., Mendonça,M. and Basili,V., "Perspective-Based Reading: A Replicated Experiment Focused on Individual Reviewer Effectiveness," Empirical Software Engineering, vol. 11, no. 1, 2006, pp. 119-142. Google ScholarDigital Library
- Biffl,S. and Halling,M., "Investigating the defect detection effectiveness and cost benefit of nominal inspection teams," Software Engineering, IEEE Transactions On, vol. 29, no. 5, 2003, pp. 385-397. Google ScholarDigital Library
- Klein,B.D., Goodhue,D.L. and Davis,G.B., "Can humans detect errors in data? Impact of base rates, incentives, and goals," MIS Quarterly, 1997, pp. 169-194. Google ScholarDigital Library
- Johnson,J.G. and Raab,M., "Take the first: Option-generation and resulting choices," Organ.Behav.Hum.Decis.Process., vol. 91, no. 2, 2003, pp. 215-229.Google ScholarCross Ref
- Teng,C., Shyu,Y.L., Chiou,W., Fan,H. and Lam,S.M., "Interactive effects of nurse-experienced time pressure and burnout on patient safety: a cross-sectional survey," Int.J.Nurs.Stud., vol. 47, no. 11, 2010, pp. 1442-1450.Google ScholarCross Ref
- Beilock,S.L. and DeCaro,M.S., "From poor performance to success under stress: working memory, strategy selection, and mathematical problem solving under pressure." Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 33, no. 6, 2007, pp. 983.Google ScholarCross Ref
- Thomas,R.W., Fugate,B.S. and Koukova,N.T., "Coping with Time Pressure and Knowledge Sharing in Buyer–Supplier Relationships," Journal of Supply Chain Management, vol. 47, no. 3, 2011, pp. 22-42.Google ScholarCross Ref
- De Dreu,C.K., "Time pressure and closing of the mind in negotiation," Organ.Behav.Hum.Decis.Process., vol. 91, no. 2, 2003, pp. 280-295.Google ScholarCross Ref
- Ben Zur,H. and Breznitz,S.J., "The effect of time pressure on risky choice behavior," Acta Psychol., vol. 47, no. 2, 2. 1981, pp. 89-104.Google Scholar
- Wohlin,C., Höst,M., Runeson,P., Ohlsson,M.C., Regnell,B. and Wesslén,A., Experimentation in software engineering: an introduction, Kluwer Academic Pub, 2000. Google ScholarDigital Library
- M. V. Mäntylä., "Experiment materials," 2013, Accessed 2013 http://users.tkk.fi/~mmantyla/TP/Google Scholar
- Höst,M., Regnell,B. and Wohlin,C., "Using students as subjects—a comparative study of students and professionals in lead-time impact assessment," Empirical Software Engineering, vol. 5, no. 3, 2000, pp. 201-214. Google ScholarDigital Library
- Tichy,W.F., "Hints for reviewing empirical work in software engineering," Empirical Software Engineering, vol. 5, no. 4, 2000, pp. 309-312. Google ScholarDigital Library
- Carver,J., Jaccheri,L., Morasca,S. and Shull,F., "Issues in using students in empirical studies in software engineering education," Software Metrics Symposium, 2003. Proceedings. Ninth International, 2003, pp. 239-249. Google ScholarDigital Library
- Runeson,P., "Using students as experiment subjects–an analysis on graduate and freshmen student data," Proceedings of the 7th International Conference on Empirical Assessment in Software Engineering.–Keele University, UK, 2003, pp. 95-102.Google Scholar
- Svahnberg,M., Aurum,A. and Wohlin,C., "Using students as subjects-an empirical evaluation," Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, 2008, pp. 288-290. Google ScholarDigital Library
- Berander,P., "Using students as subjects in requirements prioritization," Empirical Software Engineering, 2004. ISESE'04. Proceedings. 2004 International Symposium on, 2004, pp. 167-176. Google ScholarDigital Library
- Mortensen,T., Fisher,R. and Wines,G., "Students as surrogates for practicing accountants: Further evidence," Accounting Forum, 2012,Google Scholar
- Anon., "Power Distance Index," 2009, Accessed 2013 9/7 http://www.clearlycultural.com/geert-hofstede-culturaldimensions/power-distance-index/Google Scholar
- Hart,S.G. and Staveland,L.E., "Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research," Human Mental Workload, vol. 1, no. 3, 1988, pp. 139-183.Google ScholarCross Ref
- Bergersen,G.R., Hannay,J.E., Sjoberg,D.I., Dyba,T. and Karahasanovic,A., "Inferring skill from tests of programming performance: Combining time and quality," Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on, 2011, pp. 305-314. Google ScholarDigital Library
- Cohen,J., Statistical power analysis for the behavioral sciences, Lawrence Erlbaum, 1988.Google Scholar
- J. J. Rogmann., "orddom: Ordinal Dominance Statistics," 2013, Accessed 2013 9/9 http://cran.rproject.org/web/packages/orddom/Google Scholar
- Cliff,N., "Dominance statistics: Ordinal analyses to answer ordinal questions." Psychol.Bull., vol. 114, no. 3, 1993, pp. 494.Google ScholarCross Ref
- Beecham,S., Baddoo,N., Hall,T., Robinson,H. and Sharp,H., "Motivation in Software Engineering: A systematic literature review," Information and Software Technology, vol. 50, no. 9-10, 2007, pp. 860-878. Google ScholarDigital Library
- R.M. Yerkes and J.D. Dodson, "The relation of strength of stimulus to rapidity of habit‐formation," Journal of Comparative Neurology and Psychology, vol. 18, no. 5, 1908, pp. 459-482.Google ScholarCross Ref
- Ariely,D., Gneezy,U., Loewenstein,G. and Mazar,N., "Large stakes and big mistakes," Rev.Econ.Stud., vol. 76, no. 2, 2009, pp. 451-469.Google ScholarCross Ref
- I.D. Steiner, Group Process and Productivity, New York, New York, USA: Academic Press, 1972.Google Scholar
- Campbell,D.T. and Stanley,J.C., Experimental and quasiexperimental design for research, Chicago, USA: Rand Mcnally College Publishing Company, 1966.Google Scholar
- Dybå,T., Kampenes,V.B. and Sjøberg,D.I., "A systematic review of statistical power in software engineering experiments," Information and Software Technology, vol. 48, no. 8, 2006, pp. 745-755.Google ScholarCross Ref
- Bonner,S.E. and Lewis,B.L., "Determinants of auditor expertise," Journal of Accounting Research, vol. 28, 1990, pp. 1- 20.Google ScholarCross Ref
- Fogelström,N.D., Barney,S., Aurum,A. and Hederstierna,A., "When product managers gamble with requirements: Attitudes to value and risk," in Requirements engineering: Foundation for software quality, Springer, 2009, pp. 1-15. Google ScholarDigital Library
Index Terms
- Time pressure: a controlled experiment of test case development and requirements review
Recommendations
Effect of time-pressure on perceived and actual performance in functional software testing
ICSSP '18: Proceedings of the 2018 International Conference on Software and System ProcessBackground: Time-pressure is an inevitable reality of software industry that influences the performance of software engineers. It may result in adverse effects on software quality or distort the perception of performance on executed tasks to differ from ...
Comments