ABSTRACT
When solving programming problems, novices are often not aware of where they are in the problem-solving process. For instance, students who misinterpret the problem prompt will most likely not form a valid conceptual model of the task and fail to make progress towards a working solution. Avoiding such errors, and recovering from them once they occur, requires metacognitive skills that enable students to reflect on their problem-solving processes. For these reasons, developing metacognitive awareness is crucially important for novice students. Previous research has shown that explicitly teaching key steps of programming problem-solving, and having students reflect on where they are in the problem-solving process, can help students complete future programming assignments. Such metacognitive awareness training can be done through personal tutoring, but can be difficult to implement without a high ratio of instructors to students. We explore a more scalable approach, making use of an automated assessment tool, and conduct a controlled experiment to see whether scaffolding the problem-solving process would increase metacognitive awareness and improve student performance. We collected all code submissions by students in both control and experimental groups, as well as data from direct observation using a think-aloud protocol. We found that students who received the intervention showed a higher degree of understanding of the problem prompt and were more likely to complete the programming task successfully.
- Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education, Vol. 15, 2 (2005), 83--102.Google Scholar
- Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, and Debajyoti Sinha. 1999. Beyond kappa: A review of interrater agreement measures. Canadian journal of statistics, Vol. 27, 1 (1999), 3--23.Google Scholar
- Titus Barik, Jim Witschey, Brittany Johnson, and Emerson Murphy-Hill. 2014. Compiler error notifications revisited: an interaction-first approach for helping developers more effectively comprehend and resolve error notifications. In Companion Proceedings of the 36th International Conference on Software Engineering. ACM, 536--539. Google ScholarDigital Library
- Theresa Beaubouef and John Mason. 2005. Why the high attrition rate for computer science students: some thoughts and observations. ACM SIGCSE Bulletin, Vol. 37, 2 (2005), 103--106. Google ScholarDigital Library
- Brett A Becker. 2016. A new metric to quantify repeated compiler errors for novice programmers. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 296--301. Google ScholarDigital Library
- Brett A Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective compiler error message enhancement for novice programming students. Computer Science Education, Vol. 26, 2--3 (2016), 148--175.Google ScholarCross Ref
- Brett A Becker, Kyle Goslin, and Graham Glanville. 2018a. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. In Proceedings of the 2018 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 640--645. Google ScholarDigital Library
- Brett A Becker and Catherine Mooney. 2016. Categorizing compiler error messages with principal component analysis. In 12th China-Europe Int'l Symposium on Software Engineering Education (CEISEE 2016), Shenyang, China. 28--29.Google Scholar
- Brett A. Becker, Cormac Murray, Tianyi Tao, Changheng Song, Robert McCartney, and Kate Sanders. 2018b. Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. 634--639. Google ScholarDigital Library
- Susan Bergin, Ronan Reilly, and Desmond Traynor. 2005. Examining the role of self-regulated learning on introductory programming performance. In Proceedings of the first international workshop on Computing education research. ACM, 81--86. Google ScholarDigital Library
- Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 conference on Innovation & technology in computer science education. ACM, 273--278. Google ScholarDigital Library
- Christopher Douce, David Livingstone, and James Orwell. 2005. Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing (JERIC), Vol. 5, 3 (2005), 4. Google ScholarDigital Library
- Karl Anders Ericsson and Herbert Alexander Simon. 1993. Protocol analysis .MIT press Cambridge, MA.Google Scholar
- Anneli Etel"apelto. 1993. Metacognition and the expertise of computer program comprehension. Scandinavian Journal of Educational Research, Vol. 37, 3 (1993), 243--254.Google ScholarCross Ref
- Katrina Falkner, Rebecca Vivian, and Nickolas JG Falkner. 2014. Identifying computer science self-regulated learning strategies. In Proceedings of the 2014 conference on Innovation & technology in computer science education. ACM, 291--296. Google ScholarDigital Library
- Thomas Flowers, Curtis A Carver, and James Jackson. 2004. Empowering students and building confidence in novice programmers through Gauntlet. In Frontiers in Education, 2004. FIE 2004. 34th Annual. IEEE, T3H--10.Google ScholarCross Ref
- Xun Ge and Susan M Land. 2003. Scaffolding students' problem-solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research and Development, Vol. 51, 1 (2003), 21--38.Google ScholarCross Ref
- Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R Klemmer. 2010. What would other programmers do: suggesting solutions to error messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1019--1028. Google ScholarDigital Library
- Matthias Hauswirth and Andrea Adamoli. 2017. Metacognitive calibration when learning to program. In Proceedings of the 17th Koli Calling Conference on Computing Education Research. ACM, 50--59. Google ScholarDigital Library
- Jack Hollingsworth. 1960. Automatic graders for programming classes. Commun. ACM, Vol. 3, 10 (1960), 528--529. Google ScholarDigital Library
- Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Sepp"al"a. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research. ACM, 86--93. Google ScholarDigital Library
- Matthew C Jadud. 2006. An exploration of novice compilation behaviour in BlueJ. Ph.D. Dissertation. University of Kent.Google Scholar
- Antti-Juhani Kaijanaho and Ville Tirronen. 2018. Fixed versus Growth Mindset Does not Seem to Matter Much: A Prospective Observational Study in Two Late Bachelor level Computer Science Courses. In Proceedings of the 2018 ACM Conference on International Computing Education Research. ACM, 11--20. Google ScholarDigital Library
- Dastyni Loksa and Andrew J Ko. 2016. The role of self-regulation in programming problem solving process and success. In Proceedings of the 2016 ACM Conference on International Computing Education Research. ACM, 83--91. Google ScholarDigital Library
- Dastyni Loksa, Andrew J Ko, Will Jernigan, Alannah Oleson, Christopher J Mendez, and Margaret M Burnett. 2016. Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1449--1461. Google ScholarDigital Library
- Andrew Luxton-Reilly, Simon, Ibrihim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory Programming: A Systematic Literature Review. In Proceedings of the 2018 ITiCSE Conference on Working Group Reports (ITiCSE-WGR '18). ACM, New York, NY, USA. in press. Google ScholarDigital Library
- Kathleen M MacQueen, Eleanor McLellan, Kelly Kay, and Bobby Milstein. 1998. Codebook development for team-based qualitative analysis. CAM Journal, Vol. 10, 2 (1998), 31--36.Google ScholarCross Ref
- Murali Mani and Quamrul Mazumder. 2013. Incorporating metacognition into learning. In Proceeding of the 44th ACM technical symposium on Computer science education. ACM, 53--58. Google ScholarDigital Library
- Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011a. Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42nd ACM technical symposium on Computer science education. ACM, 499--504. Google ScholarDigital Library
- Guillaume Marceau, Kathi Fisler, and Shriram Krishnamurthi. 2011b. Mind your language: on novices' interactions with error messages. In Proceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software. ACM, 3--18. Google ScholarDigital Library
- Janet Metcalfe and Arthur P Shimamura. 1994. Metacognition: Knowing about knowing .MIT press.Google ScholarCross Ref
- Laurie Murphy, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: the good, the bad, and the quirky--a qualitative analysis of novices' strategies. In ACM SIGCSE Bulletin, Vol. 40. ACM. Google ScholarDigital Library
- Raymond Pettit, John Homer, Roger Gee, Susan Mengel, and Adam Starbuck. 2015. An empirical study of iterative improvement in programming assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, 410--415. Google ScholarDigital Library
- Raymond Pettit and James Prather. 2017. Automated Assessment Tools: Too Many Cooks, Not Enough Collaboration. J. Comput. Sci. Coll., Vol. 32, 4 (April 2017), 113--121. http://dl.acm.org/citation.cfm?id=3055338.3079060 Google ScholarDigital Library
- Raymond S Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, 465--470. Google ScholarDigital Library
- James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research. ACM, 74--82. Google ScholarDigital Library
- James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017. On Novices' Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research . ACM, 74--82. Google ScholarDigital Library
- Ido Roll, Natasha G Holmes, James Day, and Doug Bonn. 2012. Evaluating metacognitive scaffolding in guided invention activities. Instructional science, Vol. 40, 4 (2012), 691--710.Google Scholar
- Jeffrey Rubin and Dana Chisnell. 2008. Handbook of usability testing: how to plan, design and conduct effective tests 2 ed.). John Wiley & Sons. Google ScholarDigital Library
- Johnny Salda na. 2015. The coding manual for qualitative researchers .Sage.Google Scholar
- Tom Schorsch. 1995. CAP: an automated self-assessment tool to check Pascal programs for syntax, logic and style errors. In ACM SIGCSE Bulletin, Vol. 27. ACM, 168--172. Google ScholarDigital Library
- Michael J Scott and Gheorghita Ghinea. 2014. On the domain-specificity of mindsets: The relationship between aptitude beliefs and programming practice. IEEE Transactions on Education, Vol. 57, 3 (2014), 169--174. Google ScholarDigital Library
- Teresa M Shaft. 1995. Helping programmers understand computer programs: the use of metacognition. ACM SIGMIS Database, Vol. 26, 4 (1995), 25--46. Google ScholarDigital Library
- Donna Teague, Malcolm Corney, Alireza Ahadi, and Raymond Lister. 2013. A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers. In Proceedings of the Fifteenth Australasian Computing Education Conference-Volume 136. Australian Computer Society, Inc., 87--95. Google ScholarDigital Library
- Dwayne Towell and Brent Reeves. 2009. From Walls to Steps: Using online automatic homework checking tools to improve learning in introductory programming courses. (2009).Google Scholar
- V Javier Traver. 2010. On compiler error messages: what they say and what they mean. Advances in Human-Computer Interaction, Vol. 2010 (2010). Google ScholarDigital Library
- Christopher Watson and Frederick WB Li. 2014. Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education. ACM, 39--44. Google ScholarDigital Library
- Christopher Watson, Frederick WB Li, and Jamie L Godwin. 2012. Bluefix: Using crowd-sourced feedback to support programming students in error diagnosis and repair. In International Conference on Web-Based Learning . Springer, 228--239. Google ScholarDigital Library
- Jacqueline Whalley and Nadia Kasto. 2014. A qualitative think-aloud study of novice programmers' code writing strategies. In Proceedings of the 2014 conference on Innovation & technology in computer science education. ACM, 279--284. Google ScholarDigital Library
Index Terms
- First Things First: Providing Metacognitive Scaffolding for Interpreting Problem Prompts
Recommendations
Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
ICER '18: Proceedings of the 2018 ACM Conference on International Computing Education ResearchMost novice programmers are not explicitly aware of the problem-solving process used to approach programming problems and cannot articulate to an instructor where they are in that process. Many are now arguing that this skill, called metacognitive ...
What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming
ICER '20: Proceedings of the 2020 ACM Conference on International Computing Education ResearchMetacognition and self-regulation are popular areas of interest in programming education, and they have been extensively researched outside of computing. While computing education researchers should draw upon this prior work, programming education is ...
A Closer Look at Metacognitive Scaffolding: Solving Test Cases Before Programming
Koli Calling '19: Proceedings of the 19th Koli Calling International Conference on Computing Education ResearchPrior research has shown that novices face metacognitive difficulties associated with understanding problem statements. Metacognition refers to the processes that an individual uses to plan, monitor, and assess their own understanding and performance ...
Comments