Ken Goldman
Lisa Kaczmarczyk
Michael C. Loui
Skip Abstract Section
Abstract
A concept inventory is a standardized assessment tool intended to evaluate a student’s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.
- Allen, M. J. and Yen, W. M. 2002. Introduction to Measurement Theory. Long Grove, Ill.: Waveland Press.Google Scholar
- Almstrum, V. L., Henderson, P. B., Harvey, V., Heeren, C., Marion, W., Riedesel, C., Soh, L.-K., and Tew, A. E. 2006. Concept inventories in computer science for the topic discrete mathematics. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE’06). 163--167. Google ScholarDigital Library
- Bransford, J. D., Brown, A. L., and Cocking, R. R. 1999. How People Learn: Brain, Mind, Experience, and School. National Academy Press.Google Scholar
- Buck, J. R., Wage, K. E., Hjalmarson, M. A., and Nelson, J. K. 2007. Comparing student understanding of signals and systems using a concept inventory, a traditional exam, and interviews. In Proceedings of the 37th ASEE/IEEE Frontiers in Education Conference (ASEE’07). S1G1--S1G6.Google Scholar
- Clark, R., Feldon, D., Merrienboer, J. V., Yates, K., and Early, S. 2008. Cognitive task analysis. In Handbook of Research on Educational Communications and Technology 3rd Ed., J. Spector, M. Merrill, J. van Merrienboer, and M. Driscoll, Eds. Lawrence Erlbaum Associates, Mahwah, NJ. 1801--1856.Google Scholar
- Clayton, M. J. 1997. Delphi: A technique to harness expert opinion for critical decision-making tasks in education. Educ. Psychol. 17, 373--386.Google ScholarCross Ref
- Dalkey, N. and Helmer, O. 1963. An experimental application of the Delphi Method to the use of experts. Manage. Sci. 9, 458--467.Google Scholar
- Evans, D. 2006. Personal communication.Google Scholar
- Evans, D. et al. 2003. Progress on concept inventory assessment tools. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).Google Scholar
- Glaser, B. and Strauss, A. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, Chicago.Google Scholar
- Goldman, K., Gross, P., Heeren, C., Herman, G., Kaczmarczyk, L., Loui, M. C., and Zilles, C. 2008. Identifying important and difficult concepts in introductory computing courses using a delphi process. In Proceedings of the 39th Annual ACM Technical Symposium on Computer Science Education (SIGCSE’08). 256--260. Google ScholarDigital Library
- Gray, G. L., Evans, D., Cornwell, P., Costanzo, F., and Self, B. 2003. Toward a nationwide dynamics concept inventory assessment test. In Proceedings of the American Society of Engineering Education Annual Conference (ASEE’03).Google Scholar
- Hake, R. 1998. Interactive-engagement vs traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Amer. J. Physics 66, 64--74.Google ScholarCross Ref
- Heffernan, N. and Koedinger, K. R. 1998. A developmental model for algebra symbolization: The results of a difficulty factors assessment. In Proceedings of the 20th Annual Conference of the Cognitive Science Society (CogSci’98). 484--489.Google Scholar
- Herman, G. L., Kaczmarczyk, L., Loui, M. C., and Zilles, C. 2008. Proof by incomplete enumeration and other logical misconceptions. In Proceedings of the 4th International Computing Education Research Workshop (ICER’08). 59--70. Google ScholarDigital Library
- Herman, G. L., Loui, M. C., and Zilles, C. 2010. Creating the Digital Logic Concept Inventory. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10). (In Press). Google ScholarDigital Library
- Herman, G. L., Zilles, C., and Loui, M. C. 2009. Work in progress: Students’ misconceptions about state in digital systems. In Proceedings of the 39th ASEE/IEEE Frontiers in Education Conference (ASEE’09). Google ScholarDigital Library
- Huettel, L. G., Forbes, J., Franzoni, L., Malkin, R., Nadeau, J., and Ybarra, G. 2007. Using tablet PCs to enhance engineering and computer science education. In The Impact of Tablet PCs and Pen-Based Technology on Education, J. C. Prey, R. H. Reed, and D. A. Berque, Eds. Purdue University Press. 59--66.Google Scholar
- Jacobi, A., Martin, J., Mitchell, J., and Newell, T. 2003. A concept inventory for heat transfer. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).Google Scholar
- Kaczmarczyk, L., Petrick, E., East, J. P., and Herman, G. L. 2010. Identifying student misconceptions of programming. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10). Google ScholarDigital Library
- Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, A. A. R., Moström, J. E., Sanders, K., Seppälä, O., Simon, B., and Thomas, L. 2004. A multinational study of reading and tracing skills in novice programmers. In Proceedings of Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR’04). 119--150. Google ScholarDigital Library
- Marion, B. and Baldwin, D. 2007. SIGCSE commitee report: On the implementation of a discrete mathematics course. SIGCSE Bull. 39, 2, 109--126.Google Scholar
- McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., and Wilusz, T. 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. In Proceedings of Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE-WGR’01). 125--180. Google ScholarDigital Library
- Mestre, J. P. 2005. Facts and myths about pedagogies of engagement in science learning. Peer Rev. 7, 2, 24--27.Google Scholar
- Olds, B. M., Streveler, R. A., Miller, R. L., and Nelson, M. A. 2004. Preliminary results from the development of a concept inventory thermal and transport science. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE’04).Google Scholar
- Pill, J. 1971. The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Econ. Plan. Sci. 5, 1, 57--71.Google ScholarCross Ref
- Rowe, G. and Smaill, C. 2008. Development of an elecromagnetics course concept inventory. In Proceedings of the American Society for Engineering Education Annual Conference and Exposition (ASEE’08).Google Scholar
- Simon, B., Kohanfars, M., Lee, J., Tamayo, K., and Cutts, Q. 2010. Experience report: Peer instruction in introductory computing. In Proceedings of the 41st Annual ACM Technical Symposium on Computer Science Education (SIGCSE’10). Google ScholarDigital Library
- Soloway, E. and Spohrer, J. C. 1988. Studying the Novice Programmer. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. Google ScholarDigital Library
- Stone, A., Allen, K., Rhodes, T. R., Murphy, T. J., Shehab, R. L., and Saha, C. 2003. The statistics concept inventory: A pilot study. In Proceedings of the 33rd ASEE/IEEE Frontiers in Education (ASEE’03).Google Scholar
- Streveler, R., Olds, B. M., Miller, R. L., and Nelson, M. A. 2003. Using a Delphi study to identify the most difficult concepts for studentsto master in thermal and transport science. In Proceddings of the American Society of Engineering Education, Annual Conference and Exposition (ASEE’03).Google Scholar
- Tew, A. E., McCracken, W. M., and Guzdial, M. 2005. Impact of alternative introductory courses on programming concept understanding. In Proceedings of the 1st International Workshop on Computing Education Research (ICER’05). 25--35. Google ScholarDigital Library
- The Computer Society of the Institute for Electrical and Electronic Engineers and Association for Computing Machinery. 2001. Computing Curricula 2001, Computer science volume. http://www.sigcse.org/cc2001/index.html.Google Scholar
- Zendler, A. and Spannagel, C. 2008. Empirical foundation of central concepts for computer science education. J. Educ. Res. Comput. 2, 6, 1--615. Google ScholarDigital Library
Index Terms
- Setting the Scope of Concept Inventories for Introductory Computing Subjects
Recommendations
Identifying student misconceptions of programming
SIGCSE '10: Proceedings of the 41st ACM technical symposium on Computer science educationComputing educators are often baffled by the misconceptions that their CS1 students hold. We need to understand these misconceptions more clearly in order to help students form correct conceptions. This paper describes one stage in the development of a ...
Creating the digital logic concept inventory
SIGCSE '10: Proceedings of the 41st ACM technical symposium on Computer science educationA concept inventory (CI) is a standardized assessment tool that evaluates how well a student's conceptual framework matches the accepted conceptual framework of a discipline. In this paper, we present our process in creating and evaluating the alpha ...
Identifying important and difficult concepts in introductory computing courses using a delphi process
SIGCSE '08: Proceedings of the 39th SIGCSE technical symposium on Computer science educationA Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing ...
Comments