Peer evaluation in online anchored discussion for an increased local relevance of replies

https://doi.org/10.1016/j.chb.2007.09.005Get rights and content

Abstract

This study investigates the use of an evaluation function to increase the local relevance of replies in online anchored discussion. Being implemented in a university course on French linguistics, a regular system for anchored discussion is compared with two versions that are enhanced with an integrated evaluation function. The function asks students to evaluate the relevance of each others’ replies. To compare between experimental and control conditions, the collaboration protocols are analysed with a newly developed coding scheme for the local relevance of replies. Statistical processing of the data is done with a multilevel approach and results indicate that an evaluation function can effectively increase the local relevance of students’ replies, but only if it is actually used to a certain extent. Using the evaluation function is hypothesized to increase students’ awareness of the importance of writing relevant replies.

Introduction

Computer supported collaborative learning (CSCL) offers students possibilities for a deep and active processing of their subject matter, especially when complementing face-to-face (F-2-F) interaction (Dietz-Uhler and Bishop-Clark, 2001, Järvelä and Häkkinen, 2002). As stated by Lapadat (2002), asynchronous online discussion in particular facilitates reflection, conceptual change and the collaborative construction of meaning, making it especially suitable for the collaborative processing of academic literature. This study – which aims to facilitate students’ collaborative processing of literature – will use a specialised form of online discussion called “anchored discussion” (Bernheim Brush, Bargeron, Grudin, Borning, & Gupta, 2002), that integrates students’ online discussion with the subject matter that is being discussed. Van der Pol, Admiraal, and Simons (2006) have demonstrated that anchored discussion is better suited for supporting the early stages of collaborative text processing than regular forum discussion, in that it affords a more efficient and meaning-oriented collaboration by relating the discussion more closely with the subject matter.

However, anchored discussion might still have some constraints with regard to the coherence of students’ interaction and the learning potential of their peer-directed replies. In this article, we develop and investigate an enhanced tool for anchored discussion that aims to increase the quality of students’ replies.

Section snippets

Constructive learning conversations and the relevance of replies

While the level of interactional or “cross-turn” coherence of online learning conversations seems to be an important factor in determining the effectiveness of students’ online collaboration (Hoadley and Enyedy, 1999, Hsi, 1997), it is also often identified as a problematic element (Herring, 1999, Reyes and Tchounikine, 2003, Van der Meij et al., 2005). Students’ difficulties with preserving the interactional coherence are understandable, because students – who do not yet fully master the

Obstacles for achieving local relevance in online learning conversations

The importance and difficulty of maintaining the local relevance of replies in online learning conversations may not only be related to students’ limited understanding of the subject matter, but also to some of its basic communicative constraints. Not only do online discussions generally lack non-verbal information, but their delayed nature and the relatively high investments for writing messages (compared to synchronous or F-2-F communication) generally produce less messages, or ‘turns’ than

Increasing local relevance by providing an evaluation function

Having identified the local relevance of replies as a crucial but problematic element in creating constructive online theoretical learning conversations, we may ask ourselves how to facilitate or stimulate the creation of relevant replies. A known approach to improve the quality of students’ online interaction is to develop tools that provide the participants with information about their collaboration, described by Soller, Martinez, Jermann, and Mühlenbrock (2005) as mirroring and awareness

Method

We have set up a comparative study with three types of anchored discussion: two versions with an additional evaluation function and one regular version, without an evaluation function, to function as a control condition. Our hypothesis is that replies in the anchored discussion with an evaluation function will be more locally relevant than in the discussion groups without an evaluation function. Because we do not yet know which specific design of the evaluation function can be expected to

Results

After describing students’ participation in the online learning conversations in general, we will quantitatively compare the local relevance scores (our dependent variable) in the experimental and control conditions. These scores result from coding the messages according to the developed coding scheme and will function as dependent variable in the analyses. Finally, we will give a view on how students used the evaluation functions by looking at the provided argumentations.

Conclusion and discussion

In response to the original research question, we can conclude that the presence of an evaluation function in a tool for anchored discussion can indeed increase the local relevance of students’ replies. However, this effect was only found for the “receiver evaluates”-version of the two evaluation functions that were tested in this study. In the next paragraphs, we will address to which aspect of (the use of) the two evaluation versions this difference in effectiveness can be attributed, why

References (32)

  • H.H. Clark

    Arenas of language use

    (1992)
  • Dennis, A. R., & Valacich, J. S. (1999, January). Rethinking media richness: Towards a theory of media synchronicity....
  • N. Fay et al.

    Group discussion as interactive dialogue or as serial monologue: The influence of group size

    Psychological Science

    (2000)
  • S.C. Herring

    Interactional coherence in CMC

    Journal of Computer-Mediated Communication

    (1999)
  • C.M. Hoadley et al.

    Between information and communication: middle spaces in computer media for learning

  • J.J. Hox

    Multilevel analysis

    (2002)
  • Cited by (0)

    View full text