Reflection in design education

Departing from this ‘thinking about your own work’, for our students in AUBS it is thinking about your own design. Design is an object here (to think about), similar to experience as an object (to think about). This brings us to the theory of experiential learning as coined by Kolb (1984). Kolb defines experiential learning as ‘the process whereby knowledge is created through the transformation of experience’ (Kolb 1984: p. 38).

Moon (2004) distinguishes experiential learning from reflective learning. Reflection is ‘a form of mental processing—like a form of thinking- that we may use to fulfil a purpose or to achieve some anticipated outcome or we may simply ‘be reflective’ and then an outcome can be unexpected.‘(ibid: 82) and in an academic context ‘reflection/reflective learning or reflective writing (-), is also likely to involve a conscious and stated purpose for the reflection, with an outcome specified in terms of learning, action or clarification. It may be preceded by a description of the purpose and/or subject matter of the reflection. The process and outcome of reflective work are most likely to be in a represented (e.g., written) form, to be seen by others and to be assessed (-).‘(ibid: 83).

In literature several definitions of experiential learning are suggested for educational contexts; unanimity from this range of views is not possible. The root cause of this seems to be lying in the variety of views of experience (ibid: 109). Here, in our cause, the experience was the experience of design(ing). Hence we define experiential learning as ‘a process in which an experience is reflected upon and then translated into concepts (-)’ (ibid: 109), supported by the proposition that experiential learning in academic contexts seems to occur in situations wherein ‘the material of learning is ill-structured and challenging to a learner’ (ibid: 129), designing is exactly such a situation (Simon 1969).

The difference between experiential learning and reflective learning seems to be lying in an immediate reflection in experiential learning and a reflection afterwards in reflective learning in an academic context as Moon defined it. Because students in our course were not asked to record their findings immediately while designing, but to reflect afterwards, the concept of reflective learning is more appropriate in this case.

Similar to this distinction between experiential and reflective learning is the distinction between reflection-in-action (Schön 1982) and reflection-on-action (Schon 1987). Schön’s in the architectural world well received work from 1982 ‘The reflective practitioner’ describes based on only a few cases how professionals in general and architects in particular think while they do. Schön coins this as Reflection in Action (ibid: 54). He clearly distinguishes this from reflection-on-action that relates to the evaluation of the effects of the act according to predetermined goals (Schön 1987). Because we didn’t ask students to record their thoughts while designing but immediately afterwards and taking Schön’s distinction strictly, reflection-in-action is not applicable in our case but reflection-on-action is.

Regarding this reflection-on-action as we applied in our case a distinction can be made between analytical reflection and evaluative reflection (Cowan 2006: p. 66, 81). Analytical reflection reflects on how e.g. things are done and evaluative reflection is ‘a process which leads to the making of a judgement in relation to a set of values or criteria, and one in which the judgement often leads to a consequent decision’ (ibid: 81). In this research paper 3 on the design process is an example of analytical and paper 4 an example of evaluative reflection. Although both types occur, they are not evaluated separately in this research.

In literature on more or less similar courses as ours often no distinction is made between reflection-in-action and reflection-on-action. e.g. ‘becoming a reflective practitioner’ (Fry et al. 2003: p. 215 etc.) assumes that learning how to reflect on action or reflective learning as defined by Moon, will develop reflection-in-action. In literature courses are described that assume to develop reflective practice by letting the student use direct reflective dialogue (Schön 1982) as a start to develop reflection on a higher level. Reflective dialogue is characteristic for e.g. the design process wherein ill-structured problems are contextualised and framed in an ‘inner dialogue’ between the design as designed by the designer and the designers thoughts about this design. e.g. Brockbank et al. (1989) describe the process by which student learners engage in critically reflective learning through reflective dialogue. They lean heavily on the theory of single- and double loop learning as proposed by Argyris (1977), where single loop learning as ‘instrumental learning which leaves underlying values and theories unchanged’ can provoke double loop learning as ‘learning where assumptions are challenged and underlying values are changed’ by questioning these assumptions (Brockbank et al. 1989: p. 43, 45). In our course we asked the students first to register their design decisions in a log called ‘Design and Research-scheme’ and base their afterwards written reflections on that scheme.

Design can be described as reflection in action (Schön 1983). This is confirmed by Roozenburg and Dorst (1998: p. 35) as they write ‘modelling design as reflection in action works particularly well for conceptual stages in a design process’. As indicated above, according to Schön (1983) distinctions can be made between reflection-in- and reflection-on-action; but also between reflection on the design and reflection on the design process. Congruent with it in our course the students were asked in their paper 2 for the benefit of reflecting on the design to generate knowledge over a particular design theme based on scientific literature and in their paper 3 to reflect on their own design process on the basis of five generic elements of Van Dooren et al. (2014). Both kinds of reflection, reflection-in-action and reflection-on-action are believed to contribute to the level of doing while designing (Van Dooren et al. 2017), exactly the reason why we organize the course Academic Design Reflection for our undergraduate students. Also the reason why we would like to know if the level of reflection of the students is rising while following this course. Hence our first research question: Does the level of reflection increases during the course?

As explained above, the responses to 16 questions were collected. The analyses for the first research question are performed on the composite sum scores, i.e., the general level of reflection. For this purpose, the homogeneity of the composite sum scores is tested, using Cronbach’s Alpha. The 16 questions show good homogeneity, as shown in Table 1. Usually, a value for Cronbach’s α of 0.70 or higher is considered to reflect a reliable scale. Because of this result, the scores are summed over the responses to the 16 questions.

The mean sumscores for all three measurement points are provided in Table 2. The dataset of paper 2 includes 293 cases, of which 155 concern the first evaluator and 138 the second evaluator. After deleting cases with only one evaluator, 129 students remain in the dataset, each of which is evaluated by two evaluators (258 cases). The sumscores range from 16 to 64 and the mean score is 41.9 (std = 10.5).

The dataset of paper 3 includes 297 cases, of which 157 concern the first evaluator and 139 the second evaluator (one is unknown). After deleting cases with only one evaluator, 131 students remain in the dataset (262 cases). The scores range from 20 to 64 and the mean score is 45.4 (std = 10.1).

The dataset of paper 4 includes 296 cases, of which 153 concern the first evaluator and 142 the second evaluator (one is unknown). After deleting cases with only one evaluator, 129 students remain in the dataset (258 cases). The scores range from 16 to 64 and the mean score is 46.9 (std = 11.0).

Table 2 shows that the mean sumscores increase between paper 2 and paper 3 and also between paper 3 and paper 4. This points to an increase in the students’ level of reflection. What is also shown is a difference in the mean evaluation of the first and the second evaluator. This difference is observed at all three measurement points.

The increase in the mean level of reflection over time can be statistically tested using a repeated measures analysis of variance. This is a statistical test that compares several means when these means have come from the same participants (Field 2013: p 565). In our analysis, the mean sumscores for the students at the three measurement points are compared. This is called the “within-subjects” factor. We also included a variable in the analysis that indicated whether the sumscore was calcaluted for the first or for the second evaluator. This is called the “between-subjects” factor. For the examination of the scores over time it is important that only students are selected that have observations for both evaluators at each one of the three measurement points. This applies to 100 students.

The results for the 100 students are provided in Table 3 and in Fig. 2. The results resemble the results described above for the total data set. The table shows that the mean sumscore increases over time. This result is supported by the repeated measures anova that shows that the mean sumscore differs statistically significantly over time (p < 0.01). Furthermore, the table shows that the mean sumscore is higher for the first evaluator than for the second evaluator on each of the three measurement points. This effect is also statistically significant (p < 0.01). Finally, there is no interaction effect between time and evaluator (p = 0.48). This means that both first and second evaluator show a similar increase in mean scores over time. This can be clearly seen from Fig. 2.

Based on Table 3 the total average of the level of reflection of the papers on a scale from 16 minimum to 64 maximum is 44.9.

Answering the first research question ‘Does the level of reflection increases during the course Academic Design Reflection?’ we conclude there is a statistically significant but slight increase of the level of reflection. Further we conclude that the scores of the first evaluator, who was also the supervisor of the student, are slightly higher than the more ‘neutral’ second evaluator. A statistical analysis of the differences in the average level of reflection between all groups of students was unjustified because all the groups were too small.

Finally we wanted to show the results of the analysis on the frequency of the scores (1, 2, 3 and 4) related to the classification that we made in four groups of questions: Descriptive Writing, Reflective Writing, Dialogic Reflection and Critical Reflection (see Fig. 1). The frequency of the scores related to the separate questions is statistical normal, meaning that the distribution shows a high frequency of the middle scores 2 and 3 and a low frequency of the extreme scores 1 and 4.

For the second research question we explored whether the four levels of reflection that we adapted from the models by Kember et al. (2000) and Moon (2004: p. 96, 97) could be confirmed in our dataset. This concerns the levels: Descriptive Writing, Reflective Writing, Dialogic Reflection and Critical Reflection (see Fig. 1). We used Cronbach’s Alpha to determine the reliability of the various levels, followed by a Principal Component Analyis (PCA) to check the assumed pattern of four different levels of reflection.

The results of the Cronbach’s Alpha test are presented in Table 4. Taking into account that a scale is judged to be reliable when the coefficient for Cronbach’s Alpha is at least 0.70, all levels of reflection can be considered to have sufficient internal consistency.

To further falsify the validity of the division into four levels we executed an explorative factor analysis, i.e., Principal Components Analysis (PCA). A PCA is a statistical method that can be used to understand the structure of a set of variables (Field 2013, p. 666). It means that we examined which pattern is prevalent in the sixteen responses to the questions. Responses to questions that highly correlate with each other form a “factor”. Multiple factors can be established. We performed the PCA for each of the three papers, including only cases for which two observations were present. The Varimax rotation method was used. This means that we obtained factors that are independent of each other and that are relatively easy to interpret (Field 2013, p. 681).

The results showed that for all three measurement points, two factors could be discerned. Together these factors explain 61 (paper 2), 62 (paper 3) and 66 (paper 4) percent of the variance, which is a good result. The items that primarily and consistently load on one of the two factors are question: 1, 2, 3, 11, 14 and 16. This means that these items correlate relatively high with each other. The items relate to “evidence of reflection”, “understanding of the concepts of lectures, module manual and literature”, “the paper shows a backtrack”, “reflected to see if there can be learned”, “integration of lectures, module manual and literature” and “establishment of an argued individual position”. We read the descriptions of the six items carefully to search for a common underlying theme. However, we did not find such a theme. Furthermore, an inspection of these six items shows that they theoretically belong to all four different levels of reflection.

The items that primarily and consistently load on the other factor are question: 4, 6, 9 and 10. Two of these items (“no discussing reflection” and “most reflection is from one perspective”) theoretically belong to the same level of reflection, i.e., the Understanding/Descriptive reflection level. The other two items (“consciousness that acts and events are localized in and influenced by multiple historical and socio-political contexts” and “follows what is required”) belong to different levels. Again, we do not believe that these four items reflect an underlying common theme.

The other questions did not show consistent results; they sometimes showed the highest loading on the one factor and sometimes on the other factor.

Our results differ from the four factor-structure as hypothesized based on the theory of Moon (2004). First, our results pointed consistently to only two factors instead of four. Besides from that, the questions that made up the two factors differed from what was expected on the basis of the theory by Moon (2004) and the previous study by Kember et al. (2000). Moreover, we were unable to explain the results of the PCA, meaning that we could not find a theoretical or logical explanation for the supposed relationships between the items belonging to each one of the factors. Apparently, in this case, the operationalisations of Moon’s definitions and the operationalisations of the learning objectives of the course, do not match statistically with the classification of Moons and Kember et al.

So answering our second research question ‘Can the operationalisation in the questionnaire of the definitions of reflection derived from theory statistically be confirmed?’ we conclude that our model of classification is not statistically confirmed in the data.

The results from the Cronbach’s Alpha test and the PCA seem contradictory as quite reliable results were found with the first method, but the underlying structure could not be confirmed using PCA. We believe that this finding might be explained by the fact that all items show relatively high intercorrelations. This can also been seen from the high value of Cronbach’s Alpha of 0.94/0.95 that we presented for all sixteen items examined at once. Seen from a qualitative point of view, the setup of the questionnaire seems sensible, however not supported by statistical analysis. We assume that the multi interpretability of the questions plays an important role in this. Further research e.g. a test on the difference in interpretation of the questions by asking the evaluators to group the several questions into each of the four levels of reflection instead of giving it in advance, could provide insight.

Our research object was our Bachelor course in Academic Design Reflection. Two research questions were at hand: (1) does the level of reflection increase during our course and (2) Can the operationalisation in our questionnaire of the definitions of reflection derived from theory statistically be confirmed? The research shows that the average level of reflection of the students papers increases per successively paper and that the total average level of reflection of the papers is sufficient,. Further, we conclude that de scores of the first evaluator, who was also the supervisor of the student, are slightly higher than the more ‘neutral’ second evaluator. Finally the research shows that the classification of the questionnaire into four groups of questions with an ascending reflection level is not supported by statistical analysis, the model doesn’t fit the data, contrary to expectations based on literature. Further research is therefore required, but we already can say a number of things about the implications for our design education.

Many of the TU Delft fields of engineering and design, such as architecture and the built environment, are positioned in the heart of the so called ‘practical sciences’ (Klaasen 2004; Rooij and Frank 2016) or ‘engineering sciences’ (Kamp 2016a, b). Today’s and tomorrow’s complex socio-technical challenges in the built environment, such as sustainable, resilient, fair and healthy spatial development need well considered and integrated (design) solutions. These challenges are the backbone, the rationale and the motivation behind our Architecture &Built Environment academic sub disciplines and our curriculum specialisation tracks, such as architecture, urbanism, landscape architecture, building technology, and management in the built environment. The main knowledge question in our field therefore focuses to a large extent on ‘does it work?’ (e.g. a design, a plan, a solution proposal, a strategy) in contrast to empirical sciences which—via the construction of theories and the formulation of hypotheses—focus on the knowledge question ‘is it true?’.

This specific nature of built environment engineering education requires a specific set of academic design skills in order to (be able to) assess whether or not designs ‘work’. Important umbrella-skills are related to:

There is a call and a need from both inside and outside the faculty to significantly improve the academic skills of design students. At the Delft faculty of Architecture and the Built Environment this is for example apparent from quality assurance programmes (e.g. student questionnaires), dialogues with student bodies (e.g. student council and study association, Stylos) and the Board of Studies.¹ While students learn a lot from the current curricula, they also see room to raise the academic bar substantially. Teaching staff expresses similar views in education evaluation reports. The 2012 QANU visitation committee and the 2017 (self-organised) external audit committee advised the faculty to make better use of the academic design environment (defining it as a missed potential).

Students learn design best by doing—like most skills are learned best by doing them over and over again. But design might be understood better by academic reflection.

We showed that there is significant room for improvement for our bachelor design students. But this also holds for our master students. The faculty is convinced that such an academic reflection skills programme will only be successful, when it is closely related to, or even better, fully integrated into the design projects.

We believe the total average score of reflection can be improved. We assume a higher level of reflection skills is both possible and desirable for our student body. We dedicate an important role to the teachers. The supervisors who take care of the reflection-education now in our course, are design supervisors who have had to learn reflection themselves as in their graduation training only minor attention was paid to their reflection skills. They would, however, be better trained in reflective thinking itself, but also as part of reflection in action. That should give the concept of reflection more depth and make it more clear for them. Instead of the out of educational considerations fairly simple definition of ‘ thinking about your own work ‘ that we have used here, in particular, more attention should be given to ‘(re)considering’ because ‘ reflection on action ‘ after all is about looking back.

Even though the division into groups of level of reflection is not statistically supported, yet, we will still use this division, based on Moon (2004) for educational reasons. It makes clear to students there are different levels of reflection, in a way that seems to appeal to our students. The idea of getting more distance in a metaphorical sense or an ascending abstraction in the contemplation of one’s own work, in this case the own design, seems at least to trigger students to ‘ loosen up ‘ of their work, making them better able to oversee their work, in our opinion a necessity for reflection.

Finally, in the foregoing we ignored the question of the extent to which the skill to reflect leads to better designs. However, at this moment in our faculty a survey is conducted with the question of what is the view of students about the contribution of reflection to their design skills. It goes without saying that this is a very limited question, but it is a start for future research.