Outcomes after the third round
The results of the third round can be found in Tables
2 and
3.
Table 2
Statistics for the list of concepts
1 | Design (as a verb) | 4.83 | 5 | 0.38 | 100.0 | 1.2 |
2 | System | 4.67 | 5 | 0.48 | 100.0 | 6.1 |
3 | Modeling | 4.50 | 5 | 0.57 | 96.7 | 1.9 |
4 | Social interaction | 4.26 | 4 | 0.64 | 90.0 | 0.5 |
5 | Optimization | 4.00 | 4 | 0.74 | 80.0 | 2.1 |
6 | Innovation | 3.85 | 4 | 0.68 | 70.0 | 1.5 |
7 | Specifications | 3.85 | 4 | 0.71 | 73.3 | 3.9 |
8 | Design (as a noun) | 3.83 | 4 | 0.59 | 70.0 | 2.0 |
9 | Sustainability | 3.83 | 4 | 0.59 | 73.3 | 1.4 |
10 | Trade-offs | 3.82 | 4 | 0.82 | 70.0 | 2.6 |
11 | Energy | 3.79 | 4 | 0.62 | 76.7 | 2.2 |
12 | Materials | 3.78 | 4 | 0.74 | 73.3 | 4.0 |
13 | Resource | 3.72 | 4 | 0.67 | 73.3 | 0.6 |
14 | Technology assessment | 3.76 | 4 | 0.69 | 63.3 | 2.0 |
15 | Invention | 3.70 | 4 | 0.73 | 63.3 | 4.6 |
16 | Risk and failure | 3.64 | 4 | 0.85 | 60.0 | 1.4 |
17 | Information | 3.59 | 4 | 0.74 | 60.0 | 0.4 |
18 | Function | 3.54 | 4 | 0.75 | 56.7 | 1.6 |
19 | Structure | 3.43 | 4 | 0.67 | 53.3 | 0.0 |
20 | Product lifecycle | 3.55 | 3 | 0.75 | 50.0 | 5.6 |
21 | Measuring | 3.32 | 3 | 0.77 | 36.7 | 2.5 |
22 | Standards | 3.31 | 3 | 0.69 | 36.7 | 2.3 |
23 | Application of science | 3.28 | 3 | 0.86 | 40.0 | 1.8 |
24 | Efficiency | 3.23 | 3 | 0.72 | 33.3 | 1.6 |
25 | Heuristics | 3.04 | 3 | 0.76 | 30.0 | 4.2 |
26 | Quality assurance | 2.97 | 3 | 0.61 | 16.7 | 3.8 |
27 | Modularity | 2.87 | 3 | 0.90 | 13.3 | 3.4 |
28 | Working principle | 2.82 | 3 | 0.89 | 16.7 | 4.5 |
29 | Algorithms | 2.80 | 3 | 0.85 | 13.3 | 2.5 |
30 | Complexity | 2.72 | 3 | 0.75 | 13.3 | 2.8 |
31 | Intellectual property | 2.66 | 3 | 0.67 | 10.0 | 1.6 |
32 | Tolerance | 2.49 | 3 | 0.64 | 3.3 | 11.0 |
33 | Practical reasoning | 2.49 | 2 | 0.69 | 6.7 | 14.0 |
34 | Technological trajectory | 2.38 | 2 | 0.59 | 0.0 | 6.0 |
Table 3
Statistics for the list of contexts
1 | Energy in society | 4.37 | 4.00 | 0.72 | 93.3 | 3.7 |
2 | Biotechnology | 4.27 | 4.00 | 0.69 | 93.3 | 5.0 |
3 | Sustainable technology | 4.23 | 4.00 | 0.63 | 90.0 | 4.7 |
4 | Transportation (using vehicles, traveling) | 4.14 | 4.00 | 0.62 | 86.7 | 0.1 |
5 | Medical technologies | 4.10 | 4.00 | 0.92 | 86.7 | 1.1 |
6 | Food | 3.94 | 4.00 | 0.58 | 80.0 | 3.8 |
7 | Industrial production | 3.85 | 4.00 | 0.74 | 66.7 | 6.4 |
8 | Water resource management | 3.84 | 4.00 | 0.79 | 73.3 | 3.9 |
9 | Construction | 3.74 | 4.00 | 0.72 | 66.7 | 0.4 |
10 | 2-way communication | 3.68 | 4.00 | 0.80 | 70.0 | 4.0 |
11 | Global warming | 3.62 | 4.00 | 0.97 | 55.2 | 3.8 |
12 | Domestic technologies | 3.60 | 4.00 | 0.79 | 58.6 | 5.4 |
13 | Safety/security | 3.52 | 3.00 | 0.65 | 46.7 | 1.9 |
14 | Nanotechnology | 3.48 | 4.00 | 0.91 | 50.0 | 1.0 |
15 | Scientific research and exploration | 3.31 | 3.00 | 0.90 | 36.7 | 2.8 |
16 | Security/big brother | 3.04 | 3.00 | 1.00 | 33.3 | 1.1 |
17 | Sports and recreation | 3.01 | 3.00 | 0.86 | 30.0 | 0.9 |
18 | 1-way communication | 2.97 | 3.00 | 0.82 | 20.0 | 2.3 |
19 | Virtual reality | 2.96 | 3.00 | 0.88 | 20.0 | 4.7 |
20 | Imagining the future | 2.90 | 3.00 | 0.88 | 26.7 | 4.5 |
21 | Do-it-yourself | 2.70 | 3.00 | 0.92 | 16.7 | 8.0 |
22 | Politics and technology | 2.66 | 3.00 | 0.93 | 16.7 | 5.0 |
23 | Rescue | 2.62 | 3.00 | 0.69 | 0.0 | 9.0 |
24 | Packaging | 2.58 | 3.00 | 0.87 | 10.3 | 6.7 |
25 | Toys | 2.56 | 3.00 | 0.86 | 13.3 | 13.0 |
26 | Robotization of society | 2.55 | 2.00 | 0.82 | 10.3 | 10.8 |
27 | Technology for peace | 2.50 | 3.00 | 0.90 | 6.7 | 15.9 |
28 | Music | 2.50 | 2.00 | 0.90 | 10.0 | 17.4 |
29 | Entertainment | 2.46 | 3.00 | 0.90 | 6.7 | 18.9 |
30 | Education | 2.44 | 2.00 | 0.66 | 3.3 | 2.0 |
31 | Personal care | 2.42 | 2.00 | 0.74 | 6.7 | 9.1 |
32 | Digital photography | 2.38 | 3.00 | 0.74 | 3.3 | 8.8 |
33 | Art and technology | 2.36 | 3.00 | 0.88 | 6.7 | 2.1 |
34 | Crime scene investigation | 2.29 | 3.00 | 0.76 | 0.0 | 18.9 |
35 | Religions & technology | 1.59 | 1.00 | 0.72 | 3.3 | 14.6 |
Literature does not provide unambiguous cut-off points for consensus (indicated by the SD) and stability (calculated as 100 times “new score minus old score” divided by “new score”). As a measure for consensus, for several authors a standard deviation below 1 already signifies consensus. Osborne et al. used the stricter criterion that 66% of respondents should rate the context 4 or above. Osborne et al. (
2003) also used 33% as the maximum for stability (higher percentages mean: no stability). That requirement is easily met in our scores, both for concepts and contexts.
The results of the Delphi study have shown that a number of concepts stand out as possible foundations for an engineering and technology education curriculum.
The concepts “design (as a verb),” “system,” “modeling,” “social interaction,” and “optimization” were given the highest average score by the Delphi experts. Of these, “optimization” gave rise to somewhat more disagreement among the experts than did the other concepts in this top-five list.
“Second-best” concepts were “innovation,” “specifications,” “design (as a noun),” “sustainability,” “energy,” “materials,” “resource,” “trade-offs,” “technology assessment,” and “invention.” Of these ten, “trade-offs,”, “technology assessment,” and “invention” had somewhat less consensus than the other seven.
The concept “function” made an important change from round two to round three. In round three it receives a significantly lower percentage of 4 and 5 ratings. It dropped from 67% in round two to 58% in round three, thus not making the criterion for top concept in the final round. This is combined with a lower standard deviation in round three. As our criterion is a flexible one, we considered including this concept in the final list of “most important concepts”.
All this is followed by a whole list of concepts that get low average scores and high standard deviations. These concepts are apparently at least problematic. At the bottom of the list we find the concepts “technological trajectory,” “practical reasoning,” “tolerance,” “intellectual property,” “complexity,” “algorithms,” “working principle,” “modularity,” and “quality assurance.” For two of these, namely “working principle” and “modularity,” there was substantial lack of consensus. Several experts took the trouble to deviate from the round two average score and account for the deviation. In their opinion these concepts were more important than suggested by the average score. Less disagreement existed about rejecting “practical reasoning,” “complexity,” and “algorithms.” But here too we find experts defending a higher score. One of the experts suggested that the concept “practical reasoning” is very important but could be a sub-concept of “design (as a verb).” Similarly, another expert suggested that “complexity” could belong with “systems.” A third expert suggested that “modularity” could also be put under “systems.” These suggestions seem worth considering. The remaining concepts with low scores (below 3) were rejected by agreement.
Compared to the fairly good agreement on the concepts, it is striking that the contexts gave more rise to disagreement. Standard deviations were generally higher here (0.80 on average) compared to the concepts (0.70 on average). But let us start with what was clearly agreed upon. The contexts “energy in society,” “biotechnology,” “sustainable technology,” “transportation” and “medical technologies” stand out as useful. Of these, “medical technologies” gave rise to the most disagreement. One expert explained that he scored it lower because these technologies seem to draw students to medical schools rather than engineering programs. The context “nanotechnology” scores a mode of four but gives rise to much disagreement. Proponents state that as an emergent technology that has big consequences for our future it is a very important context. Others view it as rather inaccessible and doubt its suitability for teaching technology to young learners. One expert also remarks that this context is much more about science than about technology. Next are “food,” “industrial production,” “water resource management,” “construction,” “two-way communication,” “global warming,” and “domestic technologies.” Of these, “global warming” gives rise to more disagreement; some experts suggest it is close to, and should be integrated with, “sustainable technologies.” Rejected with agreement were the contexts “religions and technology,” “crime scene investigation,” “art and technology,” “digital photography,” “personal care,” and “education.” The experts found most of these too narrow and suggested that they could be subsumed under one of the other contexts. The exception was “religions and technology,” which seems difficult to turn into practical material. Also, it may put the teacher in a difficult position as it can be a loaded subject. For the contexts “entertainment,” “digital photography,” “art and technology” and “religion and technology,” there are nevertheless one or two enthusiastic supporters with arguments for their position. One interesting defense for the last two is that boundary crossing provides rich contexts. Several experts argued to include “digital photography” under “communication” or as a form of “art and technology” under “entertainment.” Entertainment is defended as most relevant as it is so much part of the social, emotional, and physical well-being of students (influencing them both positively and negatively). The remaining contexts got only average scores (2.5–3.5) and often a lack of agreement. Highest standard deviations amongst these were found for “scientific research and exploration,” “politics and technology,” “security/big brother,” “music,” and “technology for peace.” Several experts commented that they ranked “do-it-yourself” higher than the round two average because this context was the route to engineering for many students.
It is striking that the traditional domains of application in the US remain popular, as we see “transportation,” “communication,” “production,” and “construction” all in the list of highly scoring contexts. One of the experts expressed concern about this and wondered if there is a need to take a step forward. Biotechnology was already fairly popular in US technology education curricula and it features strongly here. Perhaps the most interesting outcome is that some new contexts stand out: “energy in society,” “sustainable technology” (with overwhelming support) and “global warming” (with less agreement). These seem to be related to an awareness of the global importance of these contexts. This is reflected by what one of the experts wrote, who could see “making the world a better place” as the umbrella context. Several experts suggested combining the three contexts, and they see global warming as a sub-context or discussion item within these or other contexts. Apart from this, most remarks highly favor the subject.
“Food” is highly supported by 80% of respondents, for different reasons: it touches upon current societal problems on world scale, is heavily influenced by technology and is a basic human need (so familiar to all of us).
Though it gets a lower average rating and higher standard deviation, “water resource management” is highly praised in the comments. Different experts appreciate this context and used similar arguments as those used for food: it is increasingly becoming an issue of high societal relevance, is heavily influenced by technology, and is a basic human need. A single expert commented that it is too narrow. This is probably the reason for the lower ratings. At the same time another expert is taken in by the broadness of it: “from potable water and desalinization to river and flooding control to reservoir building to bottled water to…”
“Medical technology” is another definite winner. “Domestic technologies” gets 4+ from little over half of the respondents (with remarks as “especially automation” and “part of student’s life”) but with much less strong agreement.
Another expert observed that the more practical contexts of lower abstraction level did not survive, in spite of the fact that these are strongly promoted by current educational research. Traditions seem to be strong among the experts.
General issues raised
Some remarks made by the experts give rise to more general considerations. In the first place, there is the issue of the level of abstraction, both in the concepts and in the contexts. Several experts remarked that the Delphi study would result in a list of separate concepts, while actually the list should be structured. Some concepts are at a higher level of abstraction and generality than others. Also there are numerous connections between the concepts that remain hidden in the list. This is clearly one of the limitations of this Delphi study, and it probably could not be avoided, given the limitations of the Delphi method. One could argue that bringing structure to the list is a necessary next step in the process of developing a curriculum. One way of doing this can be to draw a concept map that contains all of the concepts identified by the experts as important. The map should also feature the sub-concepts. The experts saw some of the sub-concepts as important but ranked them low because of their low level in the hierarchy of abstraction. The same problem arises in the list of contexts. Several contexts were seen as very important by the experts but were ranked low because of their specificity.
Another issue for the debate is what to do with the recent insight in educational research that says that contexts should be practices in which students can be involved. That idea clearly was not a priority in the experts’ considerations. Is this a matter of traditionalism or a lack of awareness of the latest educational research studies? Or did the experts consciously reject this new idea, preferring instead the more traditional, broader contexts? Several experts mentioned that the broad and general contexts should be read as umbrella terms that need further concretization and operationalization. In defense of the broader contexts, several experts remarked that in their view engineering and technology education should involve students in the wider global challenges, and this opinion seems to be a valid consideration.
A third issue that several experts noted was whether or not the concepts should be specific for engineering and technology. Sometimes experts remarked that they rejected a concept because it was not specific for engineering and technology. How should we value that consideration? Would it lead to the immediate rejection of one of the highest scoring concepts, “systems,” because it emerged not in engineering but in biology, and is used in many disciplines other than engineering and technology? Why then was this criterion of uniqueness (for engineering and technology) not applied more consistently? Were relevant concepts lost not because they were less important but because they were less specific for engineering and technology? How do we value that? These are questions that were considered in the later expert panel meeting.
A fourth issue concerns the term “engineering and technology.” Some experts suggested that engineering and technology are different and cannot be taken together in one expression that suggests they are almost the same. This raises the question, would separating the two have resulted in two different lists of concepts? If so, how would that be valued? To what extent is this remark related to the perceived difference between general and vocational education, which may suggest that engineering is for the latter and technology for the former? A related remark is that according to some experts the list of concepts was too focused on the design process and did not do justice to the social aspects of technology. Does that suggest a vocational bias in the list? In the list of preferred concepts, this fortunately does not seem to be a problem, because several concepts in this list are directly related to the social aspects of technology. The argument though is: all technologies exist only with human praxis. Therefore, this relationship needs to be embedded in each concept and not treated as something distinct to be considered separately. Looking at the list, we see that the concepts “design (as a verb),” “social interaction,” and “technology assessment” seem to have the human–technology relationship embedded most clearly. Should all concepts clearly reflect the human–technology relationship? These questions, too, were discussed at the expert panel meeting following the Delphi study.
A fifth issue is the relationship between concepts and contexts. Some experts remarked that they had difficulty separating the two. The choice of contexts, according to them, cannot be independent from the preferences for certain concepts. Contexts are not infinitely flexible. Some are more suitable for learning certain concepts than others. The setup of the Delphi rounds did not take that into account. Here too we see a necessary next step in the process leading towards curriculum development.
A sixth issue concerns the different approaches taken by experts in evaluating the contexts and the request for an overarching “umbrella context”. The proposals for new contexts and the comments on existing ones reveal that respondents take very different approaches in suggesting and evaluating contexts. Analyzing the comments, the proposed contexts and the general remarks on the context part, we find roughly nine approaches, each with a different view on what the main criteria for suitable contexts should be. In random order, they state the following:
“The contexts should…”:
1.
Be truly relevant to students’ lives
2.
Exemplify enduring human concerns, being fundamental to human nature and relevant in a variety of cultures and societies
3.
Be situated around societal issues/problems
4.
Encompass the Human-Made World
5.
Be big examples, like the development of the paper clip, as described by Petroski
6.
Be local (culturally, geographically)
7.
Cover the technological domains
8.
Use the “Designed World Standards” in “Standards for Technological Literacy”
9.
Best fit three considerations:
(b)
familiarity to the learner;
(c)
ability for the instructor or curriculum designer to provide more and less complex versions of the contexts that help make salient the critical feathers and relationships.
One respondent noted that before trying to find a set of contexts, we need an overarching “supercontext” or “umbrella context” to work within. “Purpose” could be such an umbrella context. Almost all the approaches listed above could be viewed as overarching umbrella contexts and used to frame the search for a set of suitable contexts. Only the approach that the contexts must be local might be hard to use to this purpose. Is there a best approach, or can the approaches be combined? It seems that two types of contexts received high averages in the final round: the traditional contexts and contexts that fit the suitability criteria of a combination of approaches. An example of such a context is the high scoring “food.” This context is suitable from each of the viewpoints of the first three types of approaches. In the expert panel meeting we have made an effort to find a combination of different approaches.
Finally, several respondents, from the Philosophy, History and Communication as well as from the Technology Education groups posed fervent appeals for a more central place for normative aspects in technology education. Issues relating to ethics, sustainability, and the relationship between humans and technology should be factored into (all) the concepts. Contexts should be used as a discussion arena for these normative issues. In round one a concept was suggested that we did not include in the new list but that is related to this: “unintended or unanticipated consequences”: The idea that all technologies have consequences that are not anticipated by the designers. These consequences may be positive or negative. It seems this is a notion that is still lacking. One of the other respondents added: “Many (most?) of the great challenges on our planet in the 21st century (global warming, world hunger, pandemics, nuclear warfare, etc.) have resulted from technological endeavor and/or will be addressed by technological ‘fixes.’ … I think it’s far more important that kids understand that all technologies have unintended consequences and that we MUST assess them in that light (as well as for intended consequences).” On the other hand, a respondent warns: “Teachers shouldn’t put themselves in the position of seeming to push a political agenda, so sensitive topics must be handled carefully. Some of the contexts could be very tricky to handle (like religion, politics, and technology for peace).” Also one respondent remarked that some of the concepts are too value laden (“robotization of society” and “security/big brother is watching you”) and should be more neutral. The question here seems to be how neutral and “technical” technology education should or can be. Can or should we teach the nature of engineering and technology without involving normative aspects? In the panel meeting this issue has been discussed and a proposal was made that does justice to the appeal to give values a visible place in the lists of concepts and contexts.