Davin McCall
Michael Kölling
Abstract
The types of programming errors that novice programmers make and struggle to resolve have long been of interest to researchers. Various past studies have analyzed the frequency of compiler diagnostic messages. This information, however, does not have a direct correlation to the types of errors students make, due to the inaccuracy and imprecision of diagnostic messages. Furthermore, few attempts have been made to determine the severity of different kinds of errors in terms other than frequency of occurrence. Previously, we developed a method for meaningful categorization of errors, and produced a frequency distribution of these error categories; in this article, we extend the previous method to also make a determination of error difficulty, in order to give a better measurement of the overall severity of different kinds of errors.
An error category hierarchy was developed and validated, and errors in snapshots of students source code were categorized accordingly. The result is a frequency table of logical error categories rather than diagnostic messages. Resolution time for each of the analyzed errors was calculated, and the average resolution time for each category of error was determined; this defines an error difficulty score. The combination of frequency and difficulty allow us to identify the types of error that are most problematic for novice programmers.
The results show that ranking errors by severity—a product of frequency and difficulty—yields a significantly different ordering than ranking them by frequency alone, indicating that error frequency by itself may not be a suitable indicator for which errors are actually the most problematic for students.
- Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An analysis of patterns of debugging among novice computer science students. SIGCSE Bulletin 37, 3 (June 2005), 84--88. Google Scholar
Digital Library
- Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The effects of enhanced compiler error messages on a syntax error debugging test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. ACM, 640--645. Google ScholarDigital Library
- Brett A. Becker and Catherine Mooney. 2016. Categorizing compiler error messages with principal component analysis. In Proceedings of the 12th China-Europe International Symposium on Software Engineering Education (CEISEE’16). 28--29.Google Scholar
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77--101.Google ScholarCross Ref
- Neil C. C. Brown and Amjad Altadmri. 2014. Investigating novice programming mistakes: Educator beliefs vs. student data. In Proceedings of the 10th Annual Conference on International Computing Education Research. ACM, 43--50. Google ScholarDigital Library
- Neil Christopher Charles Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A large scale repository of novice programmers’ activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education. ACM, 223--228. Google ScholarDigital Library
- Matthew Conway, Steve Audia, Tommy Burnette, Dennis Cosgrove, and Kevin Christiansen. 2000. Alice: Lessons learned from building a 3D system for novices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 486--493. Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 Conference on Innovation and Technology in Computer Science Education. ACM, 273--278. Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education. ACM, 75--80. Google ScholarDigital Library
- T. Flowers, C. A. Carver, and J. Jackson. 2004. Empowering students and building confidence in novice programmers through Gauntlet. In Proceedings of the 34th Annual Conference on Frontiers in Education (FIE’04). IEEE, T3H--10.Google Scholar
- Maria Hristova, Ananya Misra, Megan Rutter, and Rebecca Mercuri. 2003. Identifying and correcting Java programming errors for introductory computer science students. In ACM SIGCSE Bulletin, Vol. 35. ACM, 153--156. Google ScholarDigital Library
- James Jackson, Michael Cobb, and Curtis Carver. 2005. Identifying top Java errors for novice programmers. In Proceedings of the 35th Annual Conference on Frontiers in Education (FIE’05). IEEE, T4C--T4C.Google ScholarCross Ref
- Matthew C. Jadud. 2005. A first look at novice compilation behaviour using BlueJ. Computer Science Education 15, 1 (2005), 25--40. arXiv:http://dx.doi.org/10.1080/08993400500056530Google ScholarCross Ref
- Andrew J. Ko and Brad A. Myers. 2005. A framework and methodology for studying the causes of software errors in programming systems. Journal of Visual Languages and Computing 16, 1--2 (2005), 41--84. Google ScholarDigital Library
- Michael Kölling. 2010. The Greenfoot programming environment. ACM Transactions on Computing Education 10, 4 (Nov. 2010), Article 14, 21 pages. Google ScholarDigital Library
- Michael Kölling, Neil C. C. Brown, and Amjad Altadmri. 2015. Frame-based editing: Easing the transition from blocks to text-based programming. In Proceedings of the Workshop in Primary and Secondary Computing Education (WiPSCE’15). ACM, New York, NY, 29--38. Google ScholarDigital Library
- Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ system and its pedagogy. Computer Science Education 13, 4 (2003), 249--268.Google ScholarCross Ref
- John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The scratch programming language and environment. ACM Transactions on Computing Education 10, 4 (Nov. 2010), Article 16, 15 pages. Google ScholarDigital Library
- Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011. Mind your language: On novices’ interactions with error messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2011). ACM, New York, NY, 3--18. Google ScholarDigital Library
- D. McCall and M. Kölling. 2014. Meaningful categorisation of novice programmer errors. In Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE’14). 1--8.Google Scholar
- Dermot Shinners-Kennedy and Sally A. Fincher. 2013. Identifying threshold concepts: From dead end to a new direction. In Proceedings of the 9th Annual International ACM Conference on International Computing Education Research (ICER’13). ACM, New York, NY, 9--18. Google ScholarDigital Library
- Juha Sorva. 2013. Notional machines and introductory programming education. Transactions on Computing Education 13, 2 (July 2013), Article 8, 31 pages. Google ScholarDigital Library
- James C. Spohrer and Elliot Soloway. 1986. Novice mistakes: Are the folk wisdoms correct? Communications of the ACM 29, 7 (1986), 624--632. Google ScholarDigital Library
- Christopher Watson, Frederick W. B. Li, and Jamie L. Godwin. 2013. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Proceedings of the 2013 IEEE 13th International Conference on Advanced Learning Technologies (ICALT’13). IEEE, 319--323. Google ScholarDigital Library
Index Terms
- A New Look at Novice Programmer Errors
Recommendations
Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science EducationDiagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty ...
Common logic errors made by novice programmers
ACE '18: Proceedings of the 20th Australasian Computing Education ConferenceErrors in the logic of a program (sometimes referred to as semantic errors) can be very frustrating for novice programmers to locate and resolve. Developing a better understanding of the kinds of logic error that are most common and problematic for ...
The Future of the Error Message: Comparing Large Language Models and Novice Programmer Effectiveness in Fixing Errors
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2Research on enhancing error message presentation is of great interest to teachers and developers alike because improving Integrated Development Environments (IDEs) increases early student retention and efficiency at all levels with more effective ...
Comments