Elsevier

Computers & Education

Volume 58, Issue 1, January 2012, Pages 303-320
Computers & Education

A comparative analysis of the consistency and difference among teacher-assessment, student self-assessment and peer-assessment in a Web-based portfolio assessment environment for high school students

https://doi.org/10.1016/j.compedu.2011.08.005Get rights and content

Abstract

This study explored the consistency and difference of teacher-, student self- and peer-assessment in the context of Web-based portfolio assessment. Participants were 72 senior high school students enrolled in a computer application course. Through the assessment system, the students performed portfolio creation, inspection, self- and peer-assessment; three teachers reviewed portfolios and evaluated learning performances. There were significant differences in the results of the three assessment methods, among which teacher-raters adopted the most rigorous scoring standards, while peer-raters tended to use the most lax standards. The results of self- and teacher-assessment were discovered to be consistent; however, consistency was not discovered between self- and peer-assessment as well as peer- and teacher-assessment. In analyzing their consistency with the end-of-course examination, teacher- and self-assessment demonstrated high consistency, whereas peer-assessment showed a low level of consistency.

Highlights

► This study explored the consistency and difference of teacher-, student self- and peer-assessment. ► There were significant differences in the three assessment methods. ► The results of self- and teacher-assessment were consistent. ► Consistency was not discovered between self- and peer-assessment, peer- and teacher-assessment. ► Teacher- and self-assessment demonstrated high consistency with the end-of-course examination.

Introduction

A learning portfolio could be considered as portfolio assessment when its primary purpose involves assessing learning outcomes. A Web-based portfolio assessment refers to a teacher’s or peers’ authentic evaluation of a student’s efforts, progress and achievements through an online portfolio assessment system platform where the learning processes and outcomes are exhibited (Chang and Tseng, 2009a, Chang and Tseng, 2009b, Barbera, 2009). Authentic assessment involves student engagement in the evaluation process by using authentic evidences of learning processes and outcomes (Barbera, 2009, Barrett, 2007). There are three most common methods used in Web-based portfolio assessment (i.e. teacher-assessment, student self-assessment and peer-assessment), with which teachers may have options to select which method(s) to adopt in order to serve pedagogical needs. In their Web-based portfolio assessment research, Chang and Tseng, 2009a, Chang and Tseng, 2009b incorporated the three methods in order to satisfy the nature of authentic assessment. Teacher-assessment is the most used approach, but the sole dependence on which might overlook students’ real thoughts. According to Barbera (2009) and Jenkins (2004), peer feedback to some extent is more influential and powerful than teacher feedback. Therefore, the integration of self- or peer-assessment will effectively increase students’ involvement, motivation and incentives.

A learning portfolio, when serving as assessment tool, is characterized by the following aspects: 1) it is a dynamic, authentic and comprehensive assessment method; 2) it focuses on both the processes and outcomes of learning; 3) scoring participation spurs students to improve their overall knowledge and skills; 4) past performance is considered, which respects individual differences of students; 5) diverse and multi-dimensional evidence of learning helps to ensure reliable assessment results (Barrett, 2010, Barrett and Garrett, 2009). If the advantages of portfolio assessment are to be taken, none of the three assessment methods should be neglected. Although teacher-assessment plays an indispensable role, the authenticity of portfolio assessment is proved to be readily enhanced when using self- and peer-assessment altogether. Self-assessment is particularly beneficial for students, since it not only keeps them involved, interested and highly motivated in the process, but also encourages self-reflection and responsibility (Barbera, 2009, Miller, 2003, Sadler and Good, 2006). As for peer-assessment, Bouzidi and Jaillet (2009), and Chen (2010) indicated that students are likely to obtain more ideas or inspirations from peer comments which are provided by those who have in-depth understandings of them. Furthermore, peer-scoring enables students to observe other people’s work, which brings new ideas and encourages self-reflection and improvement. Therefore, more than self-assessment serves, peer-assessment benefits students by offering opportunity to observe and compare peers’ portfolios, emulate the strengths and avoid the weaknesses of other people.

There are also several constrictions for a Web-based portfolio assessment, for instance, time management, reliability and validity, assessment rubric, student assessment ability, and technology etc (Chang and Tseng, 2009b, Barrett, 2010, Barrett and Garrett, 2009, Tubaishat et al., 2009). Russell and Butcher (1999) noted that students fail to notice the advantages of portfolio assessment, but only perceive substantial amount of time and effort needed for completing it. For example, some students raised their concerns about spending more time participating in self- and peer-scoring which seems to increase their burden. Hence, some researchers called in question the reliability of self- and peer-assessment, while another controversy was concerned with whether or not the assessment results are able to thoroughly reflect learning achievements. In that case, the issues regarding teacher-, self-, and peer-assessment validity and reliability are of great importance. After summarizing previous studies on validity and reliability, Oskay, Schallies, and Morgil (2008) validated portfolio assessments as a powerful and trustworthy approach; however, further discussions were not made concentrating on peer- and self-assessment-related issues. In Lin et al., 2001a, Lin et al., 2001b)’s Web-based portfolio research, it was found that peer-assessment demonstrated a higher degree of validity than did self-assessment. They also concluded that teacher- and peer-assessment generally had better validity and reliability levels than self-assessment had. It is advisable that the teacher-scoring should account for the largest proportion of a student’s overall course grade in terms of the reliability and validity levels of the three assessment methods.

Sadler and Good (2006) and Sung, Chang, Chiou, and Hou (2005) studies confirmed that there was the consistency among self-, peer- and teacher-assessment. A number of studies discovered the consistency between self- and teacher-assessment (Bouzidi and Jaillet, 2009, Cho et al., 2006, Liu, 2002, Sadler and Good, 2006, Sung et al., 2005, Tsai and Liang, 2009, Tseng and Tsai, 2007). However, the study of Knowles, Holton and Swanson (2005) stated that there was the inconsistency between self- and teacher-assessment. As for peer- and teacher-assessment, Chen (2010)’s study argued that there was inconsistency between peer and teacher assessment. The divergent outcomes above were probably due to various educational levels of students, assessment rubrics, or different assessment procedures employed.

When comparing the scoring results, Lin et al., 2001a, Lin et al., 2001b) favored peer-assessment, and indicated that the scores determined by the three assessment methods from high to low were self-assessment, peer-assessment, and teacher-assessment. In other words, teachers are likely to use strict scoring criteria, while self-assessment is based on a more lax scoring standard. A similar finding was reported by Sadler and Good (2006), who discovered that peer-based scores were lower than self-based scores. To date, relevant investigations are concerned with traditional paper formats, because the reliability and validity appear to be subject to a great number of influential factors in the context of computerized assessments, e.g. technological supports (Bouzidi & Jaillet, 2009). The question so far remains unanswered if portfolio assessments will produce similar results in Web-based and paper-based settings.

The present study hereby attempted to compare the three assessment methods (teacher-, peer- and self-assessment) in a Web-based portfolio assessment environment. The research questions dealt with the consistency and difference of the three methods. Also, we made further attempt to examine the extent to which each method was able to reflect end-of-course examination scores. Hopefully, the research findings will provide references for the researchers and instructors who are interested in Web-based portfolio assessments. The research questions are as follows:

  • 1.1

    Are self-assessment results consistent with teacher-assessment results?

  • 1.2

    Are there significant differences between the two assessment results?

  • 2.1

    Are peer-assessment results consistent with teacher-assessment results?

  • 2.2

    Are there significant differences between the two assessment results?

  • 3.1

    Are self-assessment results consistent with peer-assessment results?

  • 3.2

    Are there significant differences between the two assessment results?

  • 4.1

    Are teacher-assessment results consistent with end-of-course examination scores?

  • 4.2

    Are student self-assessment results consistent with end-of-course examination scores?

  • 4.3

    Are peer-assessment results consistent with end-of-course examination scores?

If research question 4.1, 4.2 and 4.3 are confirmed, then it implied that the three assessment methods can reflect student learning achievements.

Section snippets

Portfolio assessment method

A “learning portfolio” refers to the evaluation of a learner’s a endeavors, growth and achievements based on his/her learning portfolios that document, collect, and reflect on learning outcomes over a period of time (Barbera, 2009, Barrett, 2007). Burch (1997) proposed an assessment method in which not only teachers but also students take part in establishing scoring criteria and assigning grades, known as the “two-tailed assessment”. Although teachers are commonly the sole assessor, portfolio

Participants

The participants were 79 students in a computer course at a senior high school in the Taiwan. In this study, 7 portfolios were not yet completed, and 72 portfolios were hence considered suitable for statistical analysis, including 34 males and 38 females. The duration of the study was a 12-week period with 3 h for each week. A Web-based portfolio assessment system was developed where the students were allowed to perform portfolio creation (setting learning goals, writing reflection, online

Are self-assessment results consistent with teacher-assessment results? Are there significant differences between the two assessment results?

Table 4 demonstrated the coefficients of Pearson’s correlation and t-test of the two assessment results. In Artifact and Overall, the two assessment methods were estimated as significantly correlated, which suggested a high degree of consistency. Nevertheless, insignificance was discovered in other aspects; that is to say, self-scoring and teacher-scoring were not consistent in four other aspects. Assume that teacher-assessment is a valid exterior criterion, self-assessment in general can be

Consistency and difference among assessment methods

Table 8 showed the consistency and difference among the results of the three assessment methods. In Portfolio Creation and Other, there was no consistency or significant difference among the three assessment methods; Goal Setting showed inconsistency among the three assessment methods, and two pairs of assessment methods were found significantly different; inconsistency among all assessment methods was also observed in Reflection and Attitude where a pair of assessment methods was significantly

Conclusion and implication

There were significant differences in the evaluation of portfolios involving the three assessment methods, among which teacher-raters generally had the strictest scoring standards, and peer-raters were the most lax. Sadler and Good (2006) discovered that peer-raters tend to undergrade while self-raters tend to overgrade, which was contradictory to our findings. At this study, the results of self-assessment and teacher-assessment were consistent, but inconsistency existed between self- and

References (51)

  • H. Barrett

    Balancing the two faces of ePortfolios

    Educação, Formação & Tecnologias

    (2010)
  • H. Barrett et al.

    Online personal learning environments: structuring electronic portfolios for lifelong and life wide learning

    On the Horizon

    (2009)
  • Blackboard

    Blackboard’s portfolio creation wizard

    (2011)
  • S. Bloxham et al.

    Understanding the rules of the game: marking peer assessment as a medium for developing students’ conceptions of assessment

    Assessment & Evaluation in Higher Education

    (2004)
  • L. Bouzidi et al.

    Can online peer assessment be trusted?

    Educational Technology & Society

    (2009)
  • C.B. Burch

    Creating a two-tired portfolio rubric

    English Journal

    (1997)
  • Chalk et al.

    ePortfolio authoring

    (2011)
  • C.-C. Chang et al.

    Using a Web-based portfolio assessment system to elevate project-based learning performances

    Interactive Learning Environments

    (2009)
  • C.-C. Chang et al.

    Use and performances of web-based portfolio assessment

    British Journal of Educational Technology

    (2009)
  • K. Cho et al.

    Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives

    Journal of Educational Psychology

    (2006)
  • P. Davies

    Computerized peer assessment

    Innovations in Education and Training International

    (2000)
  • C. Derham et al.

    Digital professional portfolios of preservice teaching: an initial study of score reliability and validity

    International Journal of Technology and Teacher Education

    (2007)
  • ePortfolio.org

    The ePortfolio platform

    (2011)
  • N. Falchikov et al.

    Student peer assessment in higher education: a meta-analysis comparing peer and teacher marks

    Review of Educational Research

    (2000)
  • Gadbury-Amyot, C. (2003). Validity and reliability of portfolio assessment of competency in a baccalaureate dental...
  • Cited by (78)

    • Peer feedback to support collaborative knowledge improvement: What kind of feedback feed-forward?

      2022, Computers and Education
      Citation Excerpt :

      It is a reflective engagement that impacts the work performed by both the giver and the receiver (Falchikov, 2003). When peers are engaged as the agent for feedback, students benefit from the process as the opportunity to observe and compare peers' work could lead to work improvement (Chang, Tseng, & Lou, 2012). Providing feedback to peers promotes knowledge improvement (Cheng, Liang, & Tsai, 2015).

    • Integration of the peer assessment approach with a virtual reality design system for learning earth science

      2020, Computers and Education
      Citation Excerpt :

      In addition, students have the power to give feedback and ratings to their peers in the same way as a teacher does. However, the reliability and validity of the scoring in the peer assessment activity needs to be further explored, and teachers need to provide scoring scaffolding to improve the reliability of students’ ratings (Chang et al., 2012; Chen, 2010). In recent years, many scholars have applied peer assessment in education and have demonstrated the effectiveness of peer assessment activities.

    View all citing articles on Scopus
    View full text