2.1 Defining and measuring self-regulated learning
Following Pintrich, in this study SRL is defined as an “active, constructive process whereby learners set goals for their learning and attempt to monitor, regulate and control their cognition, motivation, and behaviour, guided and constrained by their goals and contextual features in the environment” (Pintrich
2000, p. 453). There is considerable agreement about the importance of SRL, but there has been disagreement about how it can be operationalized and measured in a scientifically useful way (Alexander
2008; Boekaerts and Corno
2005; Zimmerman
2000). As the review of Boekaerts and Corno (
2005) showed, the concept of SRL initially has been viewed as a stable individual characteristic resulting in de-contextualised trait-like measurements. In reaction on this static view on SRL as an aptitude, scholars have started to develop new conceptualizations of SRL by using a situated learning approach in which SRL is viewed as a set of dynamic context-dependent activities. Following this situated learning approach, more qualitative and ecologically valid instruments have been developed to measure SRL in real time (Boekaerts and Cascallar
2006; Boekaerts and Corno
2005; Butler
2002; Cascallar et al.
2006; Perry
2002). In current instruments, these two different operationalizations of the concept of SRL can still be recognised. Winne and Perry (
2000) made a distinction between instruments that measure SRL as an aptitude and instruments that measure SRL as an event. An event-instrument describes the regulation activities during a specific task. When SRL is measured as an aptitude, a single measurement is used to identify a relative enduring attribute of a person.
Next to a distinction in instruments based on the
operationalization of SRL, Van Hout Wolters (
2000) showed how instruments are divided into on-line and off-line methods. This distinction is related to the
moment SRL is measured. On-line methods measure SRL during the learning task, off-line methods measure SRL independently from or directly after a learning task. This last distinction is sometimes seen as overlapping with the aptitude-event measurement distinction. Although aptitude instruments are always used off-line, there are also examples of off-line event-measurement. In Table
1, we classified the types of instruments mentioned in several overviews (Boekaerts and Corno
2005; Van Hout Wolters
2000; Van Hout Wolters et al.
2000; Winne and Perry
2000) according to these distinctions.
Table 1
Classification of the different types of instruments to measure SRL
Aptitude | | General self-report questionnaires General oral interviews General teacher judgments |
Event | Think-aloud methods Eye-movement registration Observation and video-registration of behavior Performance assessment through concrete study tasks, situational manipulations or error detection tasks Trace analysis | Stimulated recall interviews Portfolios and diaries/logs Task-based questionnaire or interview Hypothetical task interview |
There has been debate concerning the pros and cons of the different types of instruments mentioned. A review study by Dinsmore et al. (
2008) showed, that from the 75 studies, 59 % measured SRL by means of de-contextualised self-reports. This strong reliance on aptitude instruments has often been criticized, because it remains unclear which situations the learners have in mind and which references they have for comparison when completing these questionnaires (Dinsmore et al.
2008; Van Hout Wolters
2000). This may explain why low predicative values of these instruments for learning outcomes and low correlations with on-line methods were found (Veenman
2005). Many authors, therefore, consider the results of self-reports instruments to be poor indicators of the actual regulation activities that students use while studying (Perry
2002; Perry and Winne
2006; Pintrich
2004; Veenman
2005; Winne and Perry
2000). Despite these comments, self-report instruments such as the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich and Smith
1993), Inventory of Learning Styles (ILS) (Vermunt
1998), Metacognitive Awareness Inventory (MAI) (Schraw and Dennison
1994), and Learning and Study Strategies Inventory (LASSI) (Weinstein et al.
1987) are still seen as valuable tools for measuring what students perceive to be their general learning preferences, as well as their general motivation and capacity for self-regulation (Perry and Winne
2006; Pintrich
2004; Zimmerman
2008).
An alternative approach is to measure SRL as an event, during an experience or task that is marked by a prior and following event (Winne and Perry
2000). An event-instrument is more suitable for finding relations between specific aspects of real time SRL in authentic contexts (Zimmerman
2008). As Table
1 shows, of the available instruments measuring SRL as an event, some are on-line methods. These on-line methods have the advantage that little information about what happens during the task is lost due to the fact that the measurement actually takes place during the executing of the task (Van Hout Wolters
2006). Despite these benefits, on-line methods are also criticized because of the fact that these instruments influence the learning process of students by for example prompting students to think aloud (Greene and Azevedo
2009). Furthermore, on-line methods only take into account the SRL activities that are performed during the observed learning activity. Moreover, to measure SRL on-line it is essential to have the instrument present during the task. Therefore, for contexts of workplace learning in which students do not learn with the help of pre-set tasks and in which learning is often unplanned (Tynjälä
2008), using on-line instruments for measuring SRL seems to be less relevant and useful.
The off-line event measurement of SRL has less frequently been discussed. Compared to on-line event methods, with these instruments more tacit aspects of SRL can be measured for which the students need some time to recollect what exactly happened during an experience (Howard-Rose and Winne
1993). Of the different types of off-line event instruments, researchers consider portfolios and diaries as one of the most potential and useful instruments to measure SRL in a reliable and valid way (Meeus et al.
2009; Zimmerman
2008). Diaries for example have shown to be equal or even more sensitive than pre- and post-test questionnaires when it comes to measuring changes in SRL in ecologically valid contexts (Zimmerman
2008). From studies in the domain of teacher learning, we know that a digital diary or log is a suitable instrument to collect different kinds of learning experiences (Bakkenes et al.
2010; Hoekstra et al.
2009; Meirink et al.
2009; Van Eekelen et al.
2005; Zwart et al.
2008). On the other hand, researchers also have mentioned that when learners report about unique learning experiences which vary a lot from each other, this also causes standardization problems when the results of individuals need to be compared to each other (Van Hout Wolters
2000).
Concluding, both aptitude and event instruments have the potential to contribute to a deeper understanding of students’ SRL (Howard-Rose and Winne
1993; Winne and Perry
2000). Moreover, no single instrument is capable of capturing all aspects of students’ SRL (Cascallar et al.
2006). The choice of instruments thus depends on the nature of the research problem and the context (Boekaerts and Cascallar
2006; Cascallar et al.
2006; Lonka et al.
2004; Pintrich
2004).
2.2 Criteria for selection of an instrument to measure self-regulation in professional education
Next to the
type of instrument, a number of different aspects are important to take into account when selecting an appropriate method for assessing SRL (Van Hout Wolters
2000). These include: the
goal of the assessment, the
type of data to be collected, the
way of data processing, the
financial aspects of the data collection, the
content of the assessment (which skills are assessed), the
participants and
context, the
assessment procedure, and the
psychometric quality of the instrument. Below, these aspects will be discussed in the context of professional education in which student teachers’ learning at the educational institute is combined with workplace learning.
The goal of the instrument is to diagnose and assess student teachers’ quality of the regulation of their learning across different learning experiences in both the institute and the workplace. Eventually, the instrument should be practical enough to be used on a large scale for repeated measures, to be able to diagnose all student teachers’ quality of regulation during various moments of the teacher education programme. This means that the instrument should generate a type of data that is quantitative or easy to quantify, the processing of the data should be doable in a short timeframe. Regarding the financial aspects the instrument should be able to collect and analyse the data with existing resources of a programme. The content of the assessment will be the actual regulation activities that student teachers use when learning to teach. The participants are student teachers who learn in different contexts of professional education (i.e. institute and workplace). Since the curriculum does not consist of fixed tasks, there is a large variability in student teachers’ learning experiences. This means that the instrument should be able to cover different kinds of learning experiences (e.g., planned and unplanned), in different contexts, with varying duration. This has consequences for the assessment procedure: the variation in learning experiences makes it necessary to include multiple learning experiences to give a reliable estimation of student teachers’ quality of regulation. Finally, the psychometric quality of the instrument should be high enough to discriminate between different qualities of student teachers’ regulation of learning in a reliable and valid way. In sum, the following criteria can be set for the instrument: it should measure off-line, in a reliable and valid way, student teachers’ regulation activities during multiple and different kinds of learning events from the two dominant contexts of professional education. Aggregation of these multiple events should make it possible to discriminate between different qualities of student teachers’ regulation of learning.
From the different off-line event measurements listed in Table
1, the hypothetical and stimulated-recall interview do not meet these criteria, since they are too labour intensive to use for multiple-event measurements with a relatively large number of participants. Although the portfolio has been suggested as a valid instrument, the use of portfolios varies a lot among student teachers and teacher educators and is therefore in itself not structured enough to collect data of all aspects of the regulation process (Van Tartwijk et al.
2007). As mentioned previously, the diary or log has been used successfully before to collect different kinds of teachers’ learning experiences and was therefore selected as the most suitable instrument for this context.