Toward the visual understanding of computing curricula

The pilot assessment used a questionnaire distributed to the entire CC2020 task force. Instead of taking an open-ended approach (asking, e.g., “What are the knowledge, skills, and dispositions in competency statement A?”), we focused on a closed question approach where the respondents can choose from a list of categories for each of knowledge, skills, and dispositions. Hence the first part was to define categories for knowledge, skills, and dispositions, from which the respondents can choose from. The authors characterized and compared the six curricular guidelines to categorize the knowledge, skills, and dispositions associated with each of the competencies. The competencies consist of noun and verb combinations where the nouns represent the knowledge and skills that are the basis of the model curricula. The verbs imply the depth of expertise associated with each of the knowledge and skill areas. The authors grouped the competencies in each curriculum guidelines according to a topic; each group of competencies also implied one or more dispositions. The authors considered a similar list in the IT2017 report as well as Bloom’s taxonomy (Bloom and Krathwohl 1956) (Krathwohl et al. 1973). This resulted in four categories for knowledge, seven for skills, and ten dispositions. Appendix B shows the categories and their definitions.

The authors then took each of the competencies from IT2017 and formulated two questions for each. The first question asked respondents to classify the knowledge associated with the competency into one of the four categories shown in Appendix B. The second question asked them to indicate from a skills table one or more skills associated with that competency. Figure 7 shows an example question pair. Appendix C shows the list of competencies that were used in the pilot. At the end of each category of competencies, an additional question asked the respondent to identify one or more of the dispositions shown in Appendix B associated with that group of competencies. Finally, a series of questions concerning the overall classification scheme was part of the survey, some of which were open-ended questions.

The survey was distributed to the 36 members of the CC2020 task force. Those individuals represent a significant sample of educators and professionals across the computing disciplines. Since there were almost 150 questions, it was unlikely that people would take the time to answer all questions. Thus, the list of questions was divided into two parts. Several questions were put into both groups. The questions were then distributed to the 36 task force members with 18 assigned to each of the two parts.

Eight people responded to one part and ten people responded to the other part. Furthermore, respondents could opt to “skip” questions; thus some questions did not attain full response. The small size of the pilot sample and the way it was obtained prevented us from using inferential statistics. Nevertheless, the pilot survey provided some interesting insights into the competencies and this approach to characterize a curriculum. The generated results show percentages, but the reader should be aware that, for example, “at least 80% of the respondents” may mean four out of five people responded to one question or nine out of eleven people who responded to another question.

There was much variability in the selection of knowledge and skill areas associated with most of the questions. For the knowledge area, at least 80% of the respondents chose the same category in only seven out of the 47 competency statements (See Appendix D.1). There was only one case where all respondents chose the same category; for the competency “SS. Show how to choose among operating system options and install at least an operating system on a computer device” all six respondents chose the category ‘Procedural’. Still, the majority of questions had a strong tendency toward a single knowledge category. In 39 out of the 47 questions, at least 50% of the respondents chose the same knowledge category (See Appendix D.2). This outcome still leaves much variation across the other categories chosen.

Similarly, in the skills questions, 35 of the questions had two skill categories that in total comprised more than 50% of the responses (See Appendix D.3). For example, for the competency “A. Express how the growth of the internet and demands for information have changed data handling and transactional and analytical processing, and led to the creation of special purpose databases,” two out of eight responses were for the ‘Analysis’ category and three were for the ‘Assessment’ category. However, there were still many questions where the skills chosen were distributed relatively equally across the seven categories.

The tendencies with the disposition question showed greater variation. Only 1 out of the 11 questions had two characteristics that totaled more than 50% of the choices available. This is probably because the competency statements rarely make explicit references to dispositions, leaving it to the judgement of the respondent whether they are implied.

The final set of summary questions related to the categorizations underlying the survey. Appendix E shows the results for those questions. Here, one can see that there was a very strong agreement that the category choices for knowledge and skills are appropriate. Almost all the respondents provided a response of yes or mostly yes to those questions. Similarly, the respondents agreed that the categorization could be useful to the description of other computing subdisciplines. Finally, the respondents felt the list of dispositions was mostly complete. There were a series of open-ended questions which asked if there were any other dispositions that were missing from the list. There was only one person who commented on the disposition list and that comment did not offer any additional items.

The survey provides a mixed picture of the efficacy of using competencies to characterize the implied knowledge, skills, and dispositions related to the curriculum. On the positive side, the responses showed that at least 50% of the responses clustered in the same categories. Most respondents agreed that the categories for the knowledge, skills, and dispositions are appropriate. Nevertheless, there was an extensive amount of dispersion from the central tendencies. Thus, it may not be possible to use these results to help “train” a natural language processing tool to help characterize the similarities and differences among computer curricula based on the categorizations given in the pilot.

The most likely explanation for the wide range of responses is that the language and scope of the competencies are prone to different interpretations of the required knowledge and skills. Indeed, the competencies intend to be a general guideline to those preparing a curriculum and thus not constructed in a way that would prescribe a single specific interpretation. Furthermore, there have been reports that reaching agreement on Bloom’s level for even a constrained domain of introductory programming exercises can be challenging (Whalley et al. 2006).

The dispersion of responses with respect to dispositions points to a broader problem. Most of the competencies do not explicitly address dispositions. Dispositions are not only important to industry but are also clearly needed for long-term professional success, and even for personal growth and development of character and citizenship. In that regard, it may be necessary for future curricular documents to indicate explicitly the scope and breadth of dispositions in computing curricular graduates.

Additionally, even though no respondent offered additional dispositions, this does not mean that no other characteristics exist. Indeed, a wide variety of dispositions have been proposed (Perkins et al. 1993) (Clear 2017). Along with knowledge and skills, the CC2020 taskforce is actively discussing what dispositions should be within the scope of the CC2020 project.

Note that the pilot respondents were from the CC2020 project task force. Thus they are more likely to respond similarly compared to the population-at-large. Although the number of total respondents itself was 18, and thus the sample size is limited from a structured population, the 40% response rate provides credibility with statistical confidence. Thus, the results shown in Appendix D are important as they demonstrate that a consensus of categorization is achievable. The authors plan on conducting a formal survey targeting a broad international community after applying the improvements based on the pilot such as limiting the number of questions per respondent to increase the response rate as well as reviewing and revising the current categorizations. We are envisioning the results of the formal survey to help create a training set for a natural language processing program that applies the results across all the curricula. The results of the formal survey may also be used to help define a more consistent set of principles for future curriculum reviews that enable more straightforward interpretations of the curricular guidelines.

Table 1 is an excerpt of Table 3.1 in (ACM et al. 2005). It provided a comparison of comparative weights of computing knowledge areas, giving two numeric evaluations (minimum and maximum) for each of the computing knowledge areas. For example, Table 1 indicated that computer engineering has a minimum weight of 2 and maximum weight of 5 for Operating Systems Principles & Design, whereas information systems has a minimum of 1 and a maximum of 1 for the same knowledge area. For E-business, the corresponding values for CE were 0 and 0 and for IS 4 and 5. Collectively, the weights formed a reasonable mechanism for comparing the relative importance of different knowledge areas for different subdisciplines. Another table (Table 3.2 in (ACM et al. 2005)) provided similar weights for non-computing topics (including organizational/business topics that are particularly important for IS and electronics topics particularly significant for CE).

The knowledge areas included in these tables were created by first collecting all knowledge area level elements from the curriculum documents that existed in 2005, and then resolving potential naming conflicts between the subdisciplines. For the resulting set of knowledge areas, the CC2005 task force collectively determined the comparative weights based on a comprehensive evaluation of the relative emphasis on each knowledge area in each of the curriculum recommendations. This evaluation was not only based on a specific numeric value (e.g., number of core hours) but it also took into account the nature of the coverage during the time dedicated to the knowledge area.

The CC2020 steering committee decided to create a new version of these tables. There were some differences in how they derived the knowledge categories and the values in them compared to the CC2005 process. In CC2005, the work to create these tables took place in face-to-face meeting discussions among the task force members. In CC2020, the process was more elaborate: it had multiple stages and was partially distributed. The stages were as follows.

Table 2 shows the resulting table, consisting of the 36 knowledge areas each of which is grouped into one of six clusters (knowledge categories). The table also shows the minimum and maximum weights for each of the knowledge areas in each of the six computing subdisciplines. For each knowledge area, the highest values are highlighted in yellow and the lowest values in light blue. Note that the table also includes the new discipline for cybersecurity (CY) as a new column compared to the CC2005.

The final set of 36 knowledge areas included seven new or significantly restructured and two with substantive name changes. Seven knowledge areas from CC2005 did not appear any more in CC2020. Table 3 includes the details of the changes.

Some of the new knowledge areas are concerned with topics that did not exist (or just started to exist) in 2005, such as cloud computing and internet of things, and this is to be expected as the computing discipline evolves. What is especially interesting is that there are also areas that existed in 2005 but were not included in CC2005, such as software development fundamentals and systems fundamentals.

It is also interesting to note the areas that were not included in the final set of 36 CC2020 areas. This does not mean that knowledge areas such as digital media development or scientific computing are no longer important. Instead, it indicates a shift of focus within the undergraduate curricula.

The first version of the knowledge area table (after stage 1) was organized in an alphabetical order, and this was used when developing and administering the survey (stages 2 and 3). This sequencing of the knowledge areas was not, however, considered to be helpful for the purposes of visualization or any other use that requires scale properties, and this was the primary reason for stage 4. Table 2 reflects the resulting sequencing, which includes the six knowledge categories that roughly correspond to those of the semiotic ladder. It is important to note that the clustering and further sequencing concluded without access to the values linking the knowledge areas to the computing subdisciplines. Therefore, the way the maximum disciplinary row values align with the clusters provides evidence regarding the robustness of the clustering process.

The administration of the survey to the steering committee members did not reveal any substantive problems with the knowledge area. The steering committee decided that it would be important to obtain a broader perspective on the relative weights. Thus as part of future work, the members of the full task force will complete the survey with one important change: during this new round, each respondent will assign weights only for those subdisciplines that the respondent had at the beginning of the survey declared their own.