This paper presents a mixed methods study in which 77 students and 3 teachers took part, that investigated the practice of Learning by Design (LBD). The study is part of a series of studies, funded by the Netherlands Organisation for Scientific Research, that aims to improve student learning, teaching skills and teacher training. LBD uses the context of design challenges to learn, among other things, science. Previous research showed that this approach to subject integration is quite successful but provides little profit regarding scientific concept learning. Perhaps, when the process of concept learning is better understood, LBD is a suitable method for integration. Through pre- and post-exams we measured, like others, a medium gain in the mastery of scientific concepts. Qualitative data revealed important focus-related issues that impede concept learning. As a result, mainly implicit learning of loose facts and incomplete concepts occurs. More transparency of the learning situation and a stronger focus on underlying concepts should make concept learning more explicit and coherent.
Introduction
Science and technology play an important and increasing role in our modern world. However, international studies, e.g. ROSE (Sjöberg and Schreiner 2010), indicate this is not followed by an increasing interest in and understanding of science and technology among juveniles. To counter that, more meaningful and motivating teaching methods based on interdisciplinary teaching are necessary (Lustig et al. 2009; Osborne and Dillon 2008). In response to this the integration of science, technology, engineering and mathematics (STEM) has become a main topic within educational systems (Rennie et al. 2012) where (designing) technology, due to its wide contexts and informative activities, has the means of becoming the catalyst for integration (Clark and Ernst 2007).
Based on Roth (2001) technology is described as the entire set of activities that leads initial vague ideas through construction and testing of prototypes to a final model. This model, including all the knowledge and skills its creation entails, solves a problem or improves a pre-existing solution. On this basis, the potential of teaching science through designing technology is that the design task provides the context for applying science knowledge and science concepts provide content needed for design realisation. Many attempts to respond to this strong interplay of science and technology appear to be unsuccessful (Lustig et al. 2009; Osborne and Dillon 2008). Nevertheless, reasonably successful approaches like “Learning by Design” (LBD; Kolodner 2002b) show that proper integration can bring significantly better collaboration skills, meta-cognitive skills (e.g. checking work) and science and technology skills (e.g. fair testing). An essential prerequisite for success, however, is a teacher’s deep understanding of the design process. Science teachers, for example, often may not be able to adapt their science teaching methods to activities that involve designing technology (Sidawi 2009; Wendell 2008). In view of this it is obvious that technology and design teachers have to take a leading role. Therefore, they need know that LBD suffers from the fact students do not learn scientific concepts better within LBD (Kolodner 2002b; Kolodner et al. 2003a, b) which is also the main topic of this study that gains insight into why concept learning is limited and how to fortify it.
Anzeige
Importance for technology curricula
The introduction demonstrates technology and design education are directive for LBD and STEM education in general. However, technology education curricula should also benefit from STEM research like, for example, this LBD study. This is based on the Standards for Technological Literacy (International Technology Education Association 2007) where design activities are regarded as a core process of technology education. All too often, however, design is used as instructional strategy where product realisation has the emphasis, often by using trial-and-error as strategy (Burghardt and Hacker 2004). To tackle this problem students have to notice that conceptual knowledge and design processes cannot be divorced (Jones 1997). The goal is to produce students with a more conceptual understanding of design technology (International Technology Education Association 2007). For example, students should focus on concepts behind design realisations such as properties of materials, construction techniques and knowledge of electric circuits, where the latter concerns this study. Therefore, the ITEEA (formerly ITEA) emphasizes design challenges that rely on math and science knowledge to improve design performance. Against this background we need to say that also technology has its own network of conceptual knowledge. It is just because of the focus of this study that this conceptual framework has less attention. In practice this framework is just as important as the scientific knowledge domain. To conclude, knowledge about (the interplay of) concept learning and design processes, both important technological learning objectives, will strengthen technology curricula.
Foundations of LBD
LBD is a project-based inquiry approach where students learn, among skills and practices, scientific content through achieving design challenges (Kolodner 2002b). LBD is based on two educational pedagogies. First, problem-based learning (PBL): a cognitive apprenticeship approach that stimulates learning by collaboration, solving real-world problems and reflection (Norman and Schmidt 1992). Second, case-based reasoning (CBR): a constructivist model of learning that refers to solving new problems by adapting old solutions or interpreting new situations in light of similar situations (Kolodner et al. 1996). Thus, combining both pedagogies learning becomes problem-based, collaborative, reflective, context-related and task-related (doing and knowing). All fundamental elements in solving real-world, design-related problems. Therefore, design is a suitable context for learning, where science provides a part of the content needed for success (Sidawi 2009).
Figure 1 shows LBD is based on two cycles of activities: design and investigation. We will give a short description of how both cycles interplay. A more detailed description can be found in Kolodner et al. (2003a). To achieve a design challenge students have to explore design-related skills and concepts they need to learn/know. By carrying out investigations they learn those things in order to apply them during prototyping and eventually in the final design. Investigation of this application may lead to other things they need to learn and investigation starts again. Thus, students learn concepts and skills (science- and technology-based) that are needed for success by identifying a need to learn them, trying them out, questioning their handling and thinking, and acting again (iteration). That is how the practice of science (investigation) and technology (design) constantly interact. But, students will not necessarily identify all aspects they have to learn and not all insights will be applied properly. Therefore, teacher-guided rituals (poster session, pin-up session and gallery walk) take place for sharing experiences and ideas among design groups. This is more or less similar to how engineers engage with peers and clients (Kolodner et al. 2003b). Complemented by whole-class discussions students are assisted in understanding design-related principles and science. Table 3 shows how LBD activities take place in practice.
Fig. 1
Learning by design’s cycles. Note: Reprinted from Kolodner (2002b, p. 339)
×
Because concept learning has the main focus in this study it is necessary to illuminate how LBD aims for this. Generally, we can say LBD points toward a constructivist mode of education, in which one learns by extracting wisdom from experiences (Kolodner et al. 1996). It fits together with the conceptual change model that learning is a process of personal construction and that students, in an appropriate environment, will construct a more scientific framework of knowledge if they notice scientific conceptions are superior to their pre-task conceptions (Abdul Gafoor and Akhilesh 2013; Cobern 1994). For this, LBD challenges deliberately address cognitive conflicts where students’ existing ideas are no longer sufficient for succeeding. In compliance with Nussbaum and Novick (1982), and Cosgrove and Osborne (1985), LBD contains four main elements for conceptual change: (a) exploration of students’ preconceptions: preliminary phase, (b) sharpening student awareness of own and other’s framework: focus phase, (c) investigation and explanation of the conceptual conflict: challenging phase, (d) accommodation of the new (context-free) conceptual model: application phase. Furthermore, according to literature on learning, e.g. Brandsford, Brown, Donovan, and Pellegrino (2003), LBD contains several elements that (should) promote concept learning: collaboration, reflection, contextual learning, applying what is learned, learning from failures and iteration, and connecting skills, practices and concepts.
Anzeige
Previous LBD research and objective of this study
From 1999 till 2003 over 3500 American middle school students (ages 12–14; grades 6–8) took part in studies that compared achievements of LBD classes to non-LBD classes (Kolodner et al. 2003a, b). Results show that LBD students learn scientific concepts as well or slightly better (not significant) than comparison students with respect to knowledge transfer (mastery outside the design context). However, LBD students performed significantly better at collaboration skills, metacognitive skills (e.g. checking work, reflection) and science skills (e.g. fair testing, using prior knowledge). So it seems LBD makes students more skilful but does not care for better concept learning. This is supported by review of literature on design-based science teaching (Sidawi 2009).
This is notable because, according to the previous section, LDB theoretically provides a sound basis for concept learning. Thus, what factors impede concept learning? The results of previous LBD studies were based on a set of validated performance tasks and multiple choice tests conducted before and after the learning intervention. A detailed analysis of the LBD practice itself had less attention despite the fact it could provide more insight into the process of concept learning. Therefore, this will be the main objective of this study.
To gain insight into a hypothesis that states why concept learning is limited and to provide the study with important points of interest, literature upon design-based learning is helpful. Nearly all design-based science approaches are complex because many objects of integration (e.g. skills, practices, attitudes and content) are combined and remain under-exposed (Berlin and White 1994). Various studies give similar focus-related explanations for this. For example, expert designers focus on content because skills, practices and activities are familiar. Novices mainly focus on process-related issues, needed for success, in which content is largely overlooked (Popovic 2004). Wendell (2008) states that scientific content may not emerge because students focus on doing. For this, they try to avoid unknown content areas, because of complexity and diversity of hands-on activities that mainly dominate the process. Students rather rely on prior knowledge and assumptions (trial and error). Thus, a lack of focus on content and a dominant process focus might cause limitations in concept learning. Therefore, this study investigates where students (senior general education) focus on during LBD and, more specific, how and when scientific content is addressed and what students learn from it. Eventually, implications can be deduced for better concept learning and further research.
Methods
For this study the methodology of design-based mixed methods research was chosen. Beside quantitative data about learned science, qualitative data is necessary to investigate the learning process by a thorough analysis of events. The study took place in the second grade of senior general education (havo). 77 students (age 13–14; 33 female; 44 male), spread over 3 adjacent classrooms, were involved accompanied by 3 teachers. All students and teachers had prior experiences regarding characteristic LBD components, but the students had no specific prior knowledge with respect to the scientific design-related content.
Design of the LBD challenge
The LBD task “Back to the Nineties” was related to the physics domain “direct current electric circuits” and design groups (3 students per group, randomly chosen) were challenged to build a battery-operated dance pad that let them use their feet to sound a buzzer or flash lights. The dance pad had to consist of four operating floor pads and one main power switch. The entire activity took 5–6 class periods of 100 min and was guided by an instructive presentation and a student’s and teacher’s guide. To accomplish the task, design specifications were formulated, shown in Table 1, that stimulated the use of underlying science (A till D) and the process of decision-making and creative thinking (E till G).
Table 1
Design specifications and materials
Design specifications
A. The readily available push button serves as main power switch
B. One self-designed floor pad, recognizable by a circular form, should flash a light by stepping on it
C. One self-designed floor pad, recognizable by a triangular form, should flash a second light by stepping on it
D. To sound the buzzer you have to push two self-designed floor pads, cross and rectangular shaped, simultaneously (with two feet)
E. It is not allowed to use more design materials than available
F. The dance pad consists of one piece and can be used frequently without failure
G. The dance pad has a nice design (an eye candy) and is easy to use
Material
Quantity
Material
Quantity
1.5-V AA battery
2
Push button
1
AA battery holder
1
Light bulb
2
Aluminium foil
1 roll
Light bulb holder
2
Cardboard (4 colours)
1 sheet (50 × 70 cm) per colour
Buzzer
1
Tape (one- and double-sided)
1 roll
Electrical wire
500 cm
Regarding specifications A till D, the most fundamental (scientific) design principles concerned proper wiring (combining series and parallel parts) and a proper use of conducting and insulating materials for floor pad creation. Figure 2 shows an example of a design outcome and wiring. To investigate and design electric circuits students used real experiments and an interactive simulation (PhET™ DC-circuit construction kit). Beside proper circuit creation, the design challenge sets for more scientific objectives. Table 2 shows all objectives and their initial appearance.
Fig. 2
Design outcome and wiring
Table 2
Scientific objectives and initial appearance
DC electric circuits objectives
Appearance
1. Students can describe properties of direct current: (A) Conservation of current: current will not be consumed in a circuit; (B) Current can be seen, based on an educational model, as a substance for energy transportation
The interactive simulation shows current flow and enables current measurement
Real experimentation enables students to measure current flow
2. Applying the fact that a battery is an energy source and the driving force behind current flow. Beside a closed circuit this force is a prerequisite for a functional circuit
The effect of a power supply and circuit switching is explored during experimentation
Dance pad operation is based on circuit switching
3. Knowing the effect of series and parallel components on current flow (through a battery): parallel components increase and series components decrease current flow
Similar to objective 1
4. Recognizing and designing series, parallel and combined circuits and, with respect to this, identifying and describing circuit operation
Operation is based on proper wiring. Students have to meet design specifications A till D
Wiring can be studied by experimentation
5. Students know that conductors and insulators influence current flow: conductors enable current flow while insulators impede current flow
Students have to design floor pads by combining conducting and insulating materials (design specifications B till D)
6. Students know that circuits (in daily life) have a purpose in converting an input in an output (action)
The dance pad is a daily life example of a system based on an electric circuit
Objectives adapted from Kennisbasis natuurwetenschappen en technologie voor de onderbouw (pp. 41–44), by Oorschot et al. (2014)
×
Furthermore, Table 3 shows which LBD stages and activities took place to guide the process and to help students to understand underlying concepts and phenomena. In addition, a few modifications, listed below, were implemented to enrich the original LBD approach. These modifications mainly concern the usage of modern learning resources.
Table 3
Stages and activities
Stages (time)
Activitiesa
Final productsb
1. Introducing the Design Challenge and Context (20–30 min)
Introduction of context, design challenge, activities, organisation, learning sources, time schedules, materials, objectives, etc
2. Understanding the Challenge, Messing About, Whiteboarding (50–60 min)
Exploration of the challenge, context and objectives (G)
Writing down ideas, (research) questions and hypotheses (G): what to do and learn?
Adjustments of the design rules and design solutions (C/G)
Design diary stage 6
• Prototype design (G)
7. Iterative Redesign (60–120 min)
Iteration of previous steps depending on decisions made (C/G)
Improving the design (G)
Final discussion about design solutions and scientific concepts (C)
Design diary stage 7
• Final design solution (G)
• Final reflection (individual)
C = class activity or product; G = design group activity or product
aAvailable resources: ELE, smartphones, laptops, tablets, Microsoft Office® software, interactive simulation, internet access, materials and tools for design realisation, materials for conducting experiments
bDesign diary (ELE-archived): reflections, feedback, process descriptions and pictures/movies. Bulleted lists are stage-specific
A fully equipped (online) electronic learning environment (ELE) with guidance for each design stage, background materials regarding skills and practices and space to collect (requested) writings, pictures, sketches, simulations, etc.
The possibility of using tablets, laptops and smartphones to build a digital design diary and to access digital resources like internet and simulation software.
The obligation to build virtual simulations in addition to real experiments, based on Finkelstein et al. (2005).
Framework for learning
Because the students’ focus and the way they use and learn science from that is the main topic of this study, literature was studied to become more informed about elements related to concept learning. This resulted in three important, closely connected, elements that were helpful in collecting and analysing qualitative data. According to Horton (2006), as shown in Table 4, a learning activity has three essential types of interaction that should contribute learning. Within these interactions five important intertwined activities can be specified. Maybe not surprisingly, all elements in Table 4 are to a greater or lesser extent part of the LBD approach.
Table 4
Learning-related interactions and elements
Type of interaction
Learning-related elements
1. Student (to Student) interaction
(A) Collaboration: sharing information enriches the individual learning process and fortifies knowledge building (Parkinson 2001; Roth 1995). Sketching and drawing helps students to externalise and share their ideas and it allows peers to review ideas (Popovic 2004; Roth 2001). The presence of the construction materials and tools, that are necessary for design creation, stimulates peer discussion about scientific concepts (Murphy and Hennessy 2001; Roth 2001)
(B) Reflection: reflecting on knowledge, skills, practices, attitudes and received feedback makes students more aware of doing and thinking and stimulates to maintain strengths or to make adjustments. Student collaboration provides input for reflection (Roth 1995)
2. Student to teacher interaction
(C) Teacher and peer feedback: providing feedback and receiving peer and expert feedback is invaluable for teaching and learning. Constructive feedback, also important for self-reflection, provides insight into doing and thinking and reveals students’ strengths and weaknesses (Kolodner 2002a; Kolodner et al. 2003b). Constructive feedback is relevant, goal-directed, well timed, behaviour-focused, collaborative, factual and respectful (Wiggins 2012) and focuses on knowledge, skills, practices and attitudes
(D) Explicit teaching: students often solve problems intuitively by using their awareness and foreknowledge (Hennessy and McCormick 1994; Roth 1995). Students rarely solve problems in a strategic way by using (scientific) domain-related knowledge (Parkinson 2001). Also new insights are rarely linked to underlying concepts. All of this results in trial-and-error behaviour (Popovic 2004). To prevent this, teachers should help students making strategic decisions and knowledge domain connections (Kolodner et al. 2003b; McCormick 1997). By doing this, processes and contents become explicit (Hennessy and McCormick 1994)
3. Student to content interaction
(E) Process-related issues: First, mistakes are an important learning source and provide information about students’ (mis)conceptions, so mistakes must not be corrected prematurely, but should be provided by feedback (Kolodner et al. 2003b). Second, experiencing different contexts in which the same concepts occur fortifies learning, because students’ knowledge is always context-related and not directly related to decontextualized knowledge domains. Through de- and recontextualisation, complemented by explication, understanding is supported (Brandsford et al. 2003; Fortus et al. 2004; Johnson 1997; Parkinson 2001). Third, time pressure impedes learning, because students do not take ownership of the learning process (Murphy and Hennessy 2001). Encouraging students using positive and constructive feedback is to be preferred. Fourth, to incentivise the learning process sufficient control of the classroom management and organisation is needed (e.g. through clear instructions and high-quality learning materials) (Bruinsma 2003). However, it is very important teachers know when to intervene and when to hold back: sensitive assistance (Murphy and Hennessy 2001)
Data collection
To get informed about the (advancement of) students’ mastery of content knowledge pre- and post-exams (multiple-choice) were used. The same exam was used for pre- and post-testing and a control group (N = 26), not taught the task-related content, was used to rule out knowledge absorbing from taking the test. Questions were based on validated multiple-choice tests that proved to uncover students’ (mis)conceptions (Engelhardt and Beichner 2004; Licht and Snoek 1986; Niedderer and Goldberg 1993). The exam consisted of 20, objective-linked, questions. Each objective was served by pairs of similar conceptual and contextual questions to investigate differences in de- and recontextualisation (transfer). Figure 3 shows two examples of paired questions.
Fig. 3
Examples of paired conceptual and contextual questions. Note: Questions are paired horizontally. Question numbers and objectives are corresponding to Tables 2 and 6
×
During the challenge direct non-participant observations took place to investigate students’ and teachers’ behaviours and actions. The event- and scan-based observations mainly focused on occurrence, frequency and (indirectly) absence of events. The observations were guided by observation forms to respond to the simultaneous occurrence or close temporal proximity of events. These forms included a list of behaviours and events, grouped by the learning-related key elements mentioned in Table 4, with space for describing the observation in detail.
During the learning task sound recordings were made of teacher instructions, teacher-student interaction, collaboration between students within design groups and class activities. Sound recordings provide authentic data (regarding the content of explication, reflection and feedback) and express students’ thoughts and use of science vocabulary. Especially, because students were encouraged to think aloud.
Questionnaires (mostly open-ended) were used to ask students to reflect on the learning process. Questions were based on the STARR-method that provides a framework for proper reflection (Verhagen 2011). Especially, students were asked to express their opinion on learning outcomes, disturbing elements and activities that stimulated learning. Questioning took place after the learning intervention and included all students.
For deeper understanding of students’ answers, retrospective interviews took place at the end. Stimulated-recall techniques were used to investigate the extent to which students used science consciously; according to literature a rich source of data (Popovic 2004; Rennie et al. 2012; Roth 2001). In preparation for this, student products were studied to become informed of used science and successfulness of design outcomes. Visible scientific elements were noted and served as stimulus during interviews. Sixteen students, the number data occurred to be saturated (Mason 2010), and all teachers were interviewed.
Analysis
The results of the pre- and post-exam scores will be represented by the total number of correct answers among all students and corresponding fractions. This is performed per question, for contextual and conceptual questions separately and for all questions. The fractions will be used to calculate the gain index 〈g〉. The latter is defined as the ratio of the actual average gain (%post − %pre) to the maximum possible average gain (100 − %pre) (Hake 1998). A paired samples t test and a Wilcoxon signed-rank test are used to determine the difference between pre- and post-scores. Both tests were used because frequency analysis showed the data was only approximately normally distributed. The internal consistency of the exam was tested by calculating Cronbach’s alpha for the items within the different objectives, resulting in an average correlation. Finally, a factor analysis was used to test the (amount of) assumed objectives the exam is based on.
For the qualitative data we derived guidelines for analysis from methodological literature (Boeije 2005; Trochim 2006). Table 5 gives an overview of the qualitative data collection and a brief description of the analysis.
Table 5
Overview of qualitative data and analysis
Qualitative data collection
Implementation
Analysis
Observations
Non-participant expert observations (event- and scan-based) guided by observation forms based on learning-related elements in Table 4
Grouping and categorising observations using labels equal to learning-related issues in Table 4
Sound recordings
Recordings of teacher instructions, student–teacher interaction and student–student interaction
Similar to observation analysis
Counting relative usage of scientific, design-related terms during student collaboration
Product analysis
(1) Examining successfulness of design outcomes (two experts) by scoring per design specification based on 3 categories. (2) Examining the underlying science students used for creating their products
(1) Calculating weighted Cohen’s Kappa and the percentage of successes
(2) Simply writing down which science, according to Table 2, is visible in products
Questionnaires
Students had to reflect on the learning process: learning outcomes, disturbing elements and activities that stimulated learning
Categorising and labelling, similar to observation analysis, students’ answers per question
Interviews
Students: retrospective semi-structured interviews to deepen questionnaire answers. Complemented by stimulated-recall techniques to check the extent to which students used science for the design outcome consciously
Teachers: semi-structured interviews to investigate their opinion regarding learning outcomes, disturbing elements and activities that stimulated learning
Similar to questionnaire analysis
Determining, by studying students’ reasoning, if underlying science is understood and used consciously for design realisation
Because observation forms were based on learning-related elements mentioned in Table 4, those elements were also used as label (code) for (re)grouping the observations (A: collaboration, B: reflection, C: feedback, D: explication, E: process-related). A sixth label was added (F: miscellaneous) for observations that seemed hard to define. In addition, also the type of interaction, according to Table 4, was noted. In the context of methodological triangulation we used the same labelling method for analysing sound recordings, questionnaires and student interviews. Where, in the case of questionnaires and student interviews, labelling took place per question. Sound recordings were first broken down into relevant fragments, after whereupon labelling started. Next, per data collection the data was first sorted by type of interaction, in order to specify the initial focus. Second, the data was sorted by learning-related element(s) to gain insight in the learning process. This resulted in sub-categories of common content, where each sub-category was accompanied by a short description. At this stage the two researchers, guaranteeing reliability by peer debriefing, compared their findings until agreement was reached. According to literature inter-rater agreement can be determined by dividing the number of agreements by the sum of agreements and disagreements (Bijou et al. 1968). In our case (NA = 54, ND = 8) inter-rater agreement was 0.87 which is sufficient.
Furthermore, sound recordings and interviews were analysed to get informed about an increase in learned science. First, sound recordings of student collaboration were used to investigate possible changes in verbal use of scientific terms during the process. For this, the usage of 13 predefined scientific, design-related terms was counted for different stages. Second, student products were examined by simply writing down the used science that was visible in products. Then, student interviews made clear, by looking at the quality of scientific reasoning/underpinning, whether this science was understood and used consciously.
Also, design realisations were reviewed per design specification by two experts using three categories (successful, partially successful and unsuccessful). The percentage of successful scores indicates how successful students faced the design challenge. By calculating the linear weighted Cohen’s Kappa inter-rater agreement was established.
For the qualitative part we ensured validity and reliability in several ways (Hake 2004; Niedderer and Goldberg 1993). First, all methods of qualitative data collection and analysis were based on scientific literature in order to guarantee test-validity resulting in well-founded results. Second, due to coding, peer debriefing and member checking a coherent and explicit chain of analysing and reasoning was provided. Third, we used direct investigation techniques in a real-world educational setting to avoid restricted experimental settings that may cause quasi-valid results because important impacts are ruled out. Fourth, methodological and investigator triangulation is used to check results and to interpret findings.
Results
Students’ achievements
Table 6 shows the pre- and post-exam results (experimental and control group) listed by objective. Cronbach’s alpha, for each individual objective, indicates that the questions have sufficient internal consistency. Regarding objective 1–6 (α1 = 0.80, α2 = 0.83, α3 = 0.71, α4 = 0.79, α5 = 0.66, α6 = 0.76) we find an overall alpha of 0.76.
Table 6
Results pre- and post-exam: experimental group (N = 77) and control group (N = 26)
Question information
Pre-exam
Post-exam
Difference
No.
Obj.a
Type
Score
Fraction
Score
Fraction
Abs.
GainInd.b
Gainc
Exp.
Cont.
Exp.
Cont.
Exp.
Cont.
Exp.
Cont.
Exp.
Cont.
Exp.
Cont.
Exp.
1
1
CC
26
9
0.33
0.35
53
10
0.68
0.38
27
1
0.52
0.06
Medium
6
1
CC
11
3
0.14
0.12
39
3
0.50
0.12
28
0
0.42
0.00
Medium
5
1
CT
12
2
0.15
0.08
55
2
0.71
0.08
43
0
0.65
0.00
Medium
9
1
CT
4
2
0.05
0.08
45
4
0.58
0.15
41
2
0.55
0.08
Medium
12
2
CC
21
7
0.27
0.27
53
8
0.68
0.31
32
1
0.56
0.05
Medium
17
2
CC
22
8
0.28
0.31
26
7
0.33
0.27
4
−1
0.07
−0.13
Low
15
2
CT
22
6
0.28
0.23
32
8
0.41
0.31
10
2
0.18
0.10
Low
20
2
CT
23
6
0.30
0.23
41
6
0.53
0.23
18
0
0.33
0.00
Medium
2
3
CC
46
11
0.59
0.42
39
13
0.50
0.50
−7
2
−0.15
0.13
None
7
3
CC
29
10
0.37
0.38
59
9
0.76
0.35
30
−1
0.61
−0.10
Medium
13
3
CT
31
9
0.40
0.35
35
7
0.45
0.27
4
−2
0.09
−0.22
Low
19
3
CT
27
7
0.35
0.27
52
8
0.67
0.31
25
1
0.49
0.05
Medium
10
4
CC
24
10
0.31
0.38
41
12
0.53
0.46
17
2
0.32
0.13
Medium
11
4
CC
22
9
0.28
0.35
40
7
0.51
0.27
18
−2
0.32
−0.22
Medium
3
4
CT
17
7
0.22
0.27
40
7
0.51
0.27
23
0
0.38
0.00
Medium
18
4
CT
48
12
0.62
0.46
57
11
0.73
0.42
9
1
0.30
−0.08
Medium
16
5
CC
22
5
0.28
0.19
23
7
0.30
0.27
1
2
0.02
0.10
Low
8
5
CT
25
7
0.32
0.27
29
6
0.37
0.23
4
−1
0.08
−0.14
Low
4
6
CC
46
11
0.59
0.42
59
10
0.76
0.38
13
−1
0.41
−0.09
Medium
14
6
CT
28
8
0.36
0.31
48
10
0.62
0.38
20
2
0.40
0.11
Medium
CC
269
83
0.35
0.32
432
86
0.56
0.33
163
3
0.33
0.02
Medium
CT
237
66
0.31
0.25
434
69
0.56
0.27
197
3
0.37
0.02
Medium
Total
506
149
0.33
0.29
866
155
0.56
0.30
360
6
0.35
0.02
Medium
CC = conceptual question, CT = contextual question
A principal component analysis suggests, according to Kaiser’s criteria (eigenvalue > 1), 7 factors are present. However, screen plot analysis indicates, based on linear coinciding, the data should be analysed for 6 factors. Studying the (rotated) component matrix 17 test items (questions) across the components match the distribution of the questions across the objectives which gives a 85 % match.
The control group, used to determine a possible learning effect from completing the test, showed now average gain (%pre = 29; %post = 30). For the experimental group, a paired samples t test indicates the overall gain is significant, t(76) = −18.18; p < 0.001. This is confirmed by the Wilcoxon signed-rank test that gives the same p value. Unless, the experimental group made significant progress, substantially more gain could be possible because the overall gain is just enough to be called “medium” (Hake 1998). Compared to the gains found in different physics (-related) course studies, also including LBD, this gain is comparable (Churukian 2002; Coletta and Phillips 2005; Hake 1998; Kolodner 2002b).
The exam results regarding objective 1, 4, 5 and 6 are consistent, where objective 5 barely shows any gain, and for objectives 2–3 an anomaly is shown. Analysing the questions with no or low gain two things are noteworthy. First, question 2 shows a slight decline because students used the concept for parallel current behaviour, that was mainly important, in a series circuit. Second, the other questions also appealed to concepts that were barely exposed during the challenge (potential difference and resistance). For example, objective 5 (resistance and current flow) was addressed by the increasing amount of components students had to add to their design. However, a correlation with changing current was not investigated. Objective 1 and 4 (highest gains) were appealed strongly during the challenge. Thus, unravelling the requested design is important to predict learning outcomes, to set objectives and to notice possible shortcomings. Finally, differences between CT- and CC-questions were not found.
Table 7 shows how students’ final designs were scored by two experts. For these results the linear weighted Kappa κw is 0.70 (lower limit = 0.60; upper limit = 0.79), so inter rater agreement can be specified as good or substantial. The average relative amount of successes (successful) based on all specifications and both experts is 73 %. For the specifications based on proper science (A–D) this percentage is even 84 %, what implies that a medium gain according to learned science was sufficient for proper design realisation.
Despite the fact students performed reasonably well and students’ talking, as illustrated by Fig. 4, showed increasingly more scientific terms, interviews made clear, shown in Table 8, students lacked proper scientific reasoning. This is supported by the observation students continuously tend to apply trial and error to complete tasks. Summarizing, according to the knowledge dimensions of Bloom’s taxonomy, scientific insights are used as isolated facts and explicit interrelationships that enable them to function together remain underexposed (Krathwohl 2002).
Fig. 4
Relative usage of scientific and design-related terms. Note: Sound-recordings of 5 design groups were analysed by counting the usage of 13 predefined design- and objective-related terms. This was done for student discussion (40 min per group) during the early exploration stage and for one of the final stages (stage 6). For stage 3 a total amount of 417 terms were counted compared to 741 terms for stage 6. For both stages the relative distribution of used terms is shown by a chart. Beside the fact the total amount of used terms increased, the initial relatively large difference in usage decreased and certain terms became more favourite (e.g., current, circuit and parallel). The terms resistance, voltage and series stayed less favourite and were never addressed explicitly during the task
Table 8
Two examples of scientific reasoning by different students
Example 1 (interview 3)
Example 2 (interview 8)
Q. Why did you use a parallel circuit to build the dance pad?
Q. How your self-designed floor pad works?
A. Because the laptop told me to do
A. By stepping on it
Q. Why didn’t you choose a series circuit?
Q. Why does it result in, for example, flashing a light?
A. Then the dance pad will not work
A. Because we used aluminium foil
Q. Why?
Q. Why is this foil so special for your design?
A. Everything goes on and off at the same time
A. Because it lets current through
Q. Why is it that parallel circuits don’t do this?
Q. Does a normal switch functions like your floor pad?
A. Because then the parts don’t have the same wire
A. No, a normal switch contains no aluminium foil
Q. Is there a difference in the amount of current that flows through series or parallel circuits?
Q. How do you call things that can easily let current through?
A. No, it’s the same battery and current always comes back to the battery
A. Conducting
Stimulus Example 1: electric circuit
Stimulus Example 2: designed switch
×
Students’ focus
Because the students’ focus is one of the main topics of this study, students were questioned about learning outcomes. Table 9 shows experienced learning outcomes were mainly task- and product-related. Only 9 % of all replies were related to a better mastery of electricity concepts. According to questionnaires and interviews concepts were seen as a tool for designing a dance pad and the latter was, maybe logically, qualified as the ultimate goal of the challenge. This also explains that the virtual simulation was a successful tool for circuit creation, but circuit operation was not sufficiently understood. Overall, as suggested earlier, our novice design students indeed focused on process-related issues needed for success.
Table 9
Learning outcomes
Learning outcomes: what have you learned from the challenge?a
Fractionb
Knowing how to design (a dance pad) and the practice of designing (a dance pad)
0.33
The practice of creating electric circuits (for a dance pad)
0.18
Creating posters for class discussion
0.13
Proper use of construction materials and tools
0.10
(Better) mastery of electric current concepts
0.09
Knowing how to collaborate with students
0.07
Learning outcomes other than mentioned above (e.g. seeking for information, usage of ICT for educational purposes, presentation techniques, engineering concepts)
0.10
aDescriptions are revised to make categorisation possible
bRelative distribution of all replies gathered through questionnaires and interviews
Learning-related interactions
To investigate science-related learning incitements, students were asked to rate activities based on Table 4. Rating took place, as shown by the results in Table 10, using a five point Likert scale (very poor, poor, fair, good, very good).
Table 10
Learning incitements
Activities
To what degree did the activities listed below help you learn about electricity?
− − [1]
− [2]
o [3]
+ [4]
++ [5]
N
Average
Modus
% (>o)
Student (to student) interaction
25
70
83
39
14
231
2.8
3
23
Suggestions/advice from peers
6
22
29
16
4
77
2.9
3
26
Reviewing own thinking/doing
17
29
16
12
3
77
2.4
2
19
Searching for information
2
19
38
11
7
77
3.0
3
23
Student to teacher interaction
11
23
56
86
55
231
3.7
4
61
Suggestions/advices from teacher
3
9
19
26
20
77
3.7
4
60
Teacher-guided class sessions
5
5
20
28
19
77
3.7
4
61
Teacher-guided science talking
3
9
17
32
16
77
3.6
4
62
Student to content interaction
10
22
62
66
71
231
3.7
5
59
Circuit simulation (software)
0
3
21
23
30
77
4.0
5
69
Real circuit experimentation
8
14
19
18
18
77
3.3
3
47
Creating products/design(parts)
2
5
22
25
23
77
3.8
4
62
Student (to student) interaction
Table 10 shows student (to student) interaction was, according to students, least helpful to learn about electricity. Especially, self-reflection is not appreciated as a useful learning activity. Collaboration with peers scored a better rating (fair), where spontaneous collaboration was mainly triggered by the presence of the construction materials/tools and the making of sketches and drawings. This was established by counting different triggers for student–student discussion based on observation forms (Table 11).
Table 11
Observed triggers for student–student discussion
Trigger for student–student discussiona
Fractionb
Design-related activities and the presence of materials and tools
0.26
Prescribed by the learning task
0.21
The making of sketches and drawings
0.19
Teacher stimulated discussions
0.16
Scientific experimentation (real experiments and simulation software)
0.10
Other triggers (e.g., information seeking, spontaneous discussions, non-task-related triggers)
0.08
aDescriptions are revised to make categorisation possible
bThe fraction of triggers for student–student discussion based on relative distribution of observed events (observation forms)
Regarding information seeking, interviews made clear that gathered information is not shared spontaneously among peers. The major reason for this was, also demonstrated by interviews, the inability of students to properly estimate the value of the information. Furthermore, enthusiastic, highly involved students tended to dominate collaboration or, in case of no effect, to act alone in the future. This in order to finish a task as quickly as possible and experience a sense of accomplishment.
Student to teacher and content interaction
These interactions are rated equally, where circuit simulation gets the highest score regarding learning about electricity. However, interviews made clear, as mentioned before, students need considerable assistance to explicate scientific insights and design decisions adequately. For this, the teacher seems to be important. 12 of 16 interviewed students mentioned the teacher as the most reliable and important source for this kind of reasoning and fellow students were seen as incompetent doing this. Nevertheless, all teachers described the (guidance of the) LBD-task as intensive, time-consuming, complex and a real challenge for students and teachers. Especially, the process of sensitive assistance, mentioned before, seemed to be difficult. Reasons for this were mainly time constraints and the tendency to be too helpful. Students’ reactions were more or less similar and included the complexity of the design diary and complete challenge, mainly due to the extent and openness, experienced time constraints and, sometimes, the low intensity of relentless senses of accomplishments. Students often mentioned to find it difficult to stay focused and to make up their mind, but nearly 72 % also mentioned they became more motivated than usual, what also was noted by teachers. Finally, nearly one fifth of the students indicated it would be desirable to enrich the challenge by adding non-dance pad-related tasks or content. Table 12 provides an overview of the most important criticism expressed by students and teachers.
Table 12
Criticism expressed by students (questionnaires) and teachers (interviews)
Students’ criticisma
Fractionb
1. Having lack of time or experiencing time constraints
0.30
2. The complexity of the task due to diversity, complexity and openness
0.20
3. One-sided focus on the dance pad (over a long period)
0.18
4. Difficult to stay focused and concentrated (tumultuous learning environment and task duration)
0.11
5. The dependency on ICT quality (wireless network connection, hardware and software)
0.09
6. The teacher providing advice and guidance instead of answers and confirmation
0.06
Teachers’ criticism: examples of teachers’ pronouncementsc
Corresponding students’ criticism
“We also ran into time constraints and this led to some amount of stress to get everything done”
1
“Some students had problems to keep on track. […] The learning task is quite complex and appeals to many skills. […] I had to appeal to the utmost of my abilities”
2, 4
“Students often lacked concrete input from me […]. They asked for answers and confirmation […]. It was obvious they often hackled the uncertainty about their progress”
2, 6
“Often I found it difficult to give proper feedback or information. […] I want to help students but I don’t want to impede their learning process by giving too less, too much or wrong information”
2, 6
“It was very busy and noisy in the classroom. A few children asked me to create some rest”
4
“Students described administrative operations as time-consuming, disturbing and confusing. This was strengthened by the fact that internet access was often a problem”
1, 2, 5
“Two groups could not finish their design […] and they were not amused […]. They had just too little time and lost a lot of time due to filling in the design diary”
1
“The network access was frustrating. Also some laptops refused to access the simulation software”
5
“[…] so more teaching or learning activities are necessary to cover the learning content”
3
“[…] and then he asked me whether the dance pad was the only topic or something else was coming on. […] Students often have problems to stay focused when a task is complex or time-consuming”
1, 2, 3, 4
aDescriptions are revised to make categorisation possible
bRelative distribution of students’ criticism based on the total amount of criticism mentioned in questionnaires
cPronouncements teachers made during interviews (translated from Dutch to English)
Discussion and implications
By studying the practice of LBD and less emphasis on pre- and post-testing, this study reveals why concept learning has its limitations, despite the fact LBD theoretically provides a rich learning environment. It clarifies why the found average medium gain (0.35 gain-index) stayed relatively low and offers room for improvement. For example, a previous survey of pre-/post-test data for 62 introductory physics courses, based on interactive engagement (IE) methods, showed gain-indices up to 0.60 (Hake 1998). Those IE-methods are, similar to LBD, designed to promote conceptual understanding through heads- and hands-on activities contributed by peer feedback and discussion and intensive teacher guidance (Hake 1998). A main difference between those IE-methods and LBD is the amount and extensiveness of objects of integration, where LBD seems to be more diverse: teachers and students defined the LBD challenge as complex, mainly due to the extent and openness. Where time-constraints, the malfunctioning of the virtual simulation and network connection, and a disturbing emphasis on the (extensive) design diary were additional negative elements. Thus, the complexity and extensiveness forced students to focus on doing the right things and delivering requested products. Therefore, in accordance with the hypothesis stated before, students were indeed strongly product- and process-focused (What to do and what to deliver?) and qualified scientific content (What to learn?) as tools they needed for success.
The science students learned and used for producing their design mainly became available from activities that strongly determined a successful completion of the challenge. First, the virtual simulation that provided insight in electrical wiring and, second, teacher-driven activities (e.g. student–teacher interaction and teacher-driven class discussions) when concepts were discussed explicitly. Therefore, the more concepts directly determined a successful design outcome the better the concepts were understood. An important fact that also was indicated by Jones (1997) for technological concepts. The students’ strong focus on acting and delivering successful products, according to students the main goal, suppresses the fact that those processes (can/must) increase their concept-related knowledge. This resulted in the fact that concepts, certainly when they were poorly design-related, were badly or only partially understood. This lack of focus on scientific objectives and associated concepts caused the learning of isolated facts that stayed, more or less, implicit. Students used more scientific terms and symbols and designed proper electric circuits, but did not achieve a deeper conceptual understanding. Thus, students learned incomplete concepts, just enough for design-implementation, and too little explicit interrelationships that is essential to master the knowledge domain (Brandsford et al. 2003; Wiggins and McTighe 2006).
This problem of incidental, implicit, informal or unintentional learning was also found in other non-LBD studies (Baskett 1993; Kerka 2000; Marsick and Watkins 2001; Rogers 1997). For example, our findings correspond to the important design-related issue stated in the run-up to the presentation of this study: design is seen as an instructional strategy where product realisation has the emphasis and more (underlying) conceptual understanding is necessary to improve design performance and conceptual understanding. Therefore, the results of this study can be more broadly understood. The practice of design offers a rich learning environment but an overall reinforcement of conceptual awareness is required.
Possibilities for improvement and further research
According to the previous, there are mainly two (interrelated) problems for which solutions will need to be found: (1) reducing the complexity of the challenge without diluting the potentially rich learning environment; (2) a stronger focus on domain specific objectives and related (scientific) concepts, where important interrelationships become explicit.
In general, a detailed analysis of concepts (technology and science), crucial for succeeding, is necessary: when they (have to) emerge and how they are related. This also makes clear which concepts are poorly task-related and need to be addressed otherwise (e.g. demonstrations, lectures, further readings, experiments, etc.). Other previous studies give some more important insights. To discuss and explicate concepts students used for their products and during their collaboration the technique of guided discussion may be helpful (Brandsford et al. 2003). This teacher-led discussion technique encourages students to share (scientific) insights and develop a deeper understanding. To emphasize en explicate the important role of concepts for design purposes (elements of) informed design (Burghardt and Hacker 2004) might be interesting. This strategy aims for thoughtful design decisions based on scientific and mathematical concepts without reverting to trial and error, the tendency the students in our study had. Furthermore, applying explicit instruction (Archer and Hughes 2011) and the use of scaffolding strategies (Bamberger and Cahill 2013) are interesting. Both strategies help to facilitate students’ understanding and overseeing of the learning process. Students are guided through the learning process with clear instructions, proceeding in small steps, checking for understanding and achieving active and successful participation by all students. This focus on successful participation could also respond to the problem we encountered: students (within design groups) were sometimes not equally involved.
To conclude, LBD activities are very teacher-dependent, whether due to content-related choices and otherwise, because of sensitive assistance (guidance). Maybe not surprising, because the teacher plays a significant role in the succeeding of learning activities (Bamberger and Cahill 2013; Van der Veen and Van der Wal 2012). Thus, it will be valuable to study (the interplay of) concept learning and teacher handling in detail to distract important clues for appropriate teacher behaviour.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
