Rubrics vs. self-assessment scripts effect on self-regulation, performance and self-efficacy in pre-service teachers

Highlights

• The effects of rubrics vs. self-assessment scripts vs. control were compared.

• The effects on self-regulation, self-efficacy and performance were measured.

• Script use enhanced learning self-regulatory actions.

• Rubric use decreased negative self-regulatory actions (performance/avoidance SR).

• No significant effects were found for performance or self-efficacy.

Abstract

Two approaches to self-assessment are optimal, because they include the assessment criteria: rubrics and scripts. The aim of this study is to compare the effect of rubrics and scripts on self-regulation, performance and self-efficacy. A total of 69 pre-service teachers participated in the study. During a semester the participants were trained to design multimedia material in three experimental conditions (rubrics, scripts and control). Results showed that students using the scripts had higher levels of learning self-regulation after the intervention, whereas rubrics decreased performance/avoidance self-regulation (negative self-regulatory actions detrimental to learning). No significant effects were found for students’ performance or self-efficacy. Students preferred the use of rubrics to the use of scripts.

Introduction

In recent years self-assessment of learning has received a lot of attention, becoming a growing field in educational psychology (e.g. Dochy et al., 1999, Ross, 2006, Taras, 2010). The reason for this is that self-assessment is a necessary process for self-regulation and learning to occur (Andrade and Valtcheva, 2009, Peters and Kitsantas, 2010, Winne and Hadwin, 1998). Consequently, researchers have looked for different ways to promote students’ self-assessment. There are two self-assessment tools being studied to test their potential effects and conditions for effectiveness: rubrics and scripts (Alonso-Tapia & Panadero, 2010). Rubrics are designed to evaluate, mainly but not exclusively, the product of an activity (Andrade, 2010, Jonsson and Svingby, 2007), whereas scripts are designed to help students during an activity to assess whether the process they are following is adequate (Bannert, 2009, Kramarski and Michalsky, 2010, Nückles et al., 2009). These two tools have proved to have positive effects on self-regulation and learning (Alonso-Tapia and Panadero, 2010, Bannert, 2009, Panadero and Jonsson, 2013). Nevertheless, results about script and rubric effectiveness and the conditions for it are far from conclusive, and no prior research has compared their relative effects on self-regulation and performance in a real classroom setting (Panadero, 2011). Hence, this will be the main objective of this study, which was tested in higher education with pre-service teachers.

Our work is based on several theoretical suppositions about self-assessment and self-regulation. These processes – especially self-regulation – have received considerable attention in the past two decades, as they are crucial competences for higher education students to develop in the transition from secondary education (Torenbeek, Jansen, & Hofman, 2010) and in order to be successful during university training (Heikkiläa and Lonka, 2006, Pintrich, 2004). Self-regulation is a process through which self-generated thoughts, emotions and actions are planned and adapted to reach personal goals (Zimmerman, 2000). An important number of self-regulation theories point out that for such adaptation to occur, students must self-assess their ongoing cognitive, emotional, motivational and behavioural processes. By doing this, they can become aware of what needs to be controlled or changed (Kitsantas and Zimmerman, 2006, Winne and Hadwin, 1998, Zimmerman and Moylan, 2009). In their review of self-regulation theories, Puustinen and Pulkkinen (2001) point out that the five major self-regulation theories consider self-assessment a key self-regulation process, even though they refer to it by different names. There are also empirical findings that demonstrate the validity of this connection: when students self-assess their learning using adequate criteria, they self-regulate with success (Andrade and Valtcheva, 2009, Bannert, 2009, Heikkiläa and Lonka, 2006, Panadero et al., 2012).

However, what implies being able to self-assess one's own learning activity adequately? There is actually a list of conditions for an adequate implementation of self-assessment so that students can learn this skill (Andrade & Valtcheva, 2009). From these, it can be extracted that for appropriate self-assessment to occur, two factors are crucial: (a) using adequate assessment criteria, and (b) using them at the right time (Goodrich, 1996, Panadero, 2011). Therefore, the questions to answer are: (a) what promotes the use of adequate assessment criteria? (b) when is it opportune to use that criteria?

Assessment criteria are the standards against which the execution and the final outcome of a task are evaluated. Although people can set their own assessment criteria for a task, students need to internalize the criteria provided by their teachers to carry out an adequate self-assessment of their work. This internalization is difficult, and often external help is necessary (Andrade, 2010). Rubrics and scripts contain these assessment criteria.

As for the appropriate time, self-regulation is usually divided into different phases (e.g. Winne and Hadwin, 1998, Zimmerman and Moylan, 2009). According to the majority of theories (Puustinen & Pulkkinen, 2001), self-assessment takes part in the final phase – the self-reflection phase (Zimmerman & Moylan, 2009) – where the students analyse what they have done and reflect on its consequences. Nevertheless, in line with other researchers (Boud, 1995, Winne and Hadwin, 1998), we consider that students cannot only self-assess the final product; they must also consider the process through which the final product is reached. In fact, it can be concluded from previous self-assessment research (Andrade, 2010, Boud, 1995) that a good implementation of self-assessment should influence all the self-regulatory phases – i.e. forethought, execution and self-reflection (Zimmerman & Moylan, 2009). This theoretical perspective is supported by research on the effects of self-regulation interventions: the biggest effects occur when the interventions focus on planning and monitoring or planning and evaluation (Dignath, Büttner, & Langfeldt, 2008). Therefore, instructional help to learn how to self-assess should be given during the planning and monitoring phases of the self-regulation process, and not only at the end of it.

There are three types of interventions aimed at promoting self-assessment: (a) self-grading/self-evaluation, or self-assessment without the assessment criteria, (b) rubrics and (c) scripts, including cues and prompts (Alonso-Tapia & Panadero, 2010).

First, self-evaluation implies asking the students to evaluate and score their work without the use of a specific tool. Research has shown that this is not an optimal pedagogical approach, as it is flawed (Dochy et al., 1999, Falchikov and Boud, 1989). In this category can be included those interventions that are aimed at enhancing self-assessment, but do not provide students with assessment criteria. As these two approaches do not include assessment criteria, they do not promote precise self-assessment (Andrade & Valtcheva, 2009).

Rubrics are self-assessment tools with three characteristics: a list of criteria for assessing the important goals of a task, a scale for grading the different levels of achievement and a description for each qualitative level (Andrade & Valtcheva, 2009). Rubrics have been shown to enhance student performance and learning if used in combination with metacognitive activities (for a review: Panadero & Jonsson, 2013), to improve reliability among teachers when rating their students and to improve reliability when the same teacher scores different students (for a review: Jonsson & Svingby, 2007). Nevertheless, there is a need for more empirical evidence on their direct effect on self-regulation.

Scripts, including cues and prompts, are specific sets of steps structured accordingly to the expert model of performing a task from beginning to end. Like rubrics, they also have positive effects, promoting self-regulation and learning (e.g. Bannert, 2009, Peters and Kitsantas, 2010). Scripts have been used mainly in experimental settings, with only a small number of studies carried out in real settings (e.g. Kramarski & Michalsky, 2010).

As rubrics and scripts contain assessment criteria, they are more effective methods than self-evaluation or self-assessment without assessment criteria. However, what are the main differences between these two tools?

There are two main differences. First, rubrics have a scoring feature that scripts do not. Thus rubrics can emphasize grades, whereas scripts do not have such a characteristic. Second, rubrics include a set of text samples describing characteristics for every performance level, and thus might centre students’ attention on outcomes and learning products. On the other hand, scripts are formulated as questions pointing to the steps that the students have to follow, and thus might centre the students’ attention on the learning process – in fact, research on scripts shows that they are cognitively demanding (e.g. Kollar, Fischer, & Slotta, 2007). In sum, both tools are oriented towards promoting students’ self-assessment, but they present salient different features that can influence different effects.

A comparison between rubrics and scripts was carried out by Panadero et al. (2012). They found that the participants using a script or a rubric scored higher than the control group on self-regulation and learning, and that the use of the script enhanced self-regulation more than use of the rubric. However, this study was conducted in an experimental setting with secondary education students. Also learning was assessed through a task that was not scored; hence, it remained to test whether intervention effects would be similar in natural classroom settings, with higher education students, and when the task to be performed counted towards the course grades. That is the aim of the present study.

When planning this study, a crucial question was: can different effects be expected from the use of rubrics and scripts in natural classroom settings? Below, we consider this study's expected effects on the dependent variables (self-regulation, performance, self-efficacy and tool's perceived help).

First, it can be expected that self-assessment will positively affect self-regulation. However, this effect will depend on the measurement method used. Boekaerts and Corno (2005) recommended using contextual measures of self-regulation, rather than a general self-regulation questionnaire, to evaluate specific intervention effects. However, it can be difficult and costly to measure individual self-regulation in natural classroom contexts. Nevertheless, this difficulty can be at least partially overcome if self-regulation is measured through a combination of questionnaires: a general questionnaire assessing self-regulation messages and a specific one with items referring to the competence being acquired (Samuelstuen & Bråten, 2007). For this reason, in the present study, both a general and a specific self-regulation questionnaire will be used. The instruments chosen for this study measure two types of regulatory actions: (a) learning self-regulation (regulatory actions oriented to learning goals) and (b) performance/avoidance self-regulation (regulatory actions oriented to performance and avoidance goals). This differentiation is based on Boekaert's top-down and bottom-up model (Boekaerts & Corno, 2005). With regard to learning self-regulation, previous research shows that rubrics and scripts enhance the use of these strategies (Bannert, 2009, Panadero and Jonsson, 2013). Regarding performance/avoidance self-regulation, the self-assessment tools should decrease this type of negative self-regulation, as they give the students clear criteria for performing the task.

Second, though performance does not depend exclusively on self-assessment and self-regulation, these factors are crucial. Therefore, some improvement in performance can be expected as a result of promoting self-assessment.

Third, if self-assessment affects self-regulation and performance in a positive way, it may also produce an improvement in self-efficacy. Why is self-efficacy important for these study goals? There are two reasons. First, pre-service teachers’ self-efficacy has been linked to their later commitment during their adaptation to the workplace, and to performance (Klassen & Chiu, 2011). Second, and more directly related to this study's goals, self-efficacy has an essential role in self-regulation: if students have low self-efficacy expectations for a task, their motivation will decrease and they will activate fewer and less effective self-regulatory processes. On the contrary, if they hold high self-efficacy expectations, they will be more willing to engage in highly demanding activities to overcome problems that they may find (Zimmerman & Moylan, 2009). If this was the case, the self-efficacy level prior to intervention could mediate the effect of self-assessment in this variable; consequently, self-efficacy needs to be measured before the intervention.

Previous research on the effects of rubrics and scripts on self-efficacy is not conclusive. Alonso-Tapia and Panadero (2010) studied whether scripts enhanced self-efficacy in comparison to a control group based in previous research by Andrade, Wang, Du, and Akawi (2009) that studied the same topic using rubrics. Andrade found that rubrics tended to enhance self-efficacy beliefs in interaction with gender (more so for girls than for boys). Alonso-Tapia and Panadero (2010) found no significant effect in self-efficacy for the use of scripts, attributing this result to the short length of the intervention. In conclusion, the question of the potential effects of scripts on self-efficacy remains open, as does the question of which tool might promote a higher increase in self-efficacy.

Fourth, to be motivated to use rubrics and scripts, students need to perceive these tools as useful. Previous research shows that students value the use of rubrics (Andrade and Du, 2005, Reynolds-Keefer, 2010), but there is no empirical evidence about this topic in relation to scripts. Therefore it remains unclear which tool would be perceived as more useful by the students.

In sum, the research questions and hypotheses of this study are thus as follows:

• Do rubrics and scripts enhance self-regulation when compared to a control group? What self-assessment tool is more effective? Both tools will have positive effects on learning self-regulation over the control group (Hypothesis 1a). Scripts would enhance learning self-regulation more in comparison to the rubric group because scripts are more cognitively demanding (H1b). Regarding performance/avoidance self-regulation, both self-assessment tools’ groups would show lower levels of this type of negative self-regulation than the control group, but there is no hypothesis on the comparison between rubric and scripts (H1c).

• Do rubrics and scripts enhance performance over the control group? Both tools will have positive effects on performance compared to the control group as far as they have positive effects on self-regulation (H2).

• Do rubrics and scripts enhance self-efficacy in comparison with the control group? It is expected that the self-assessment tools will increase self-efficacy in comparison to the control group (H3a), but it is unclear which tool will produce a bigger effect (H3b).

• Which self-assessment tool – rubric or script – do students perceive as the better learning tool? There is no hypothesis on the direction of this effect (H4).