Catherine M. Hicks
Vineet Pandey
Scott Klemmer
ABSTRACT
Peer assessment is rapidly growing in online learning, as it presents a method to address scalability challenges. However, research suggests that the benefits of peer review are obtained inconsistently. This paper explores why, introducing three ways that framing task goals significantly changes reviews. Three experiments manipulated features in the review environment. First, adding a numeric scale to open text reviews was found to elicit more explanatory, but lower quality reviews. Second, structuring a review task into short, chunked stages elicited more diverse feedback. Finally, showing reviewers a draft along with finished work elicited reviews that focused more on the work's goals than aesthetic details. These findings demonstrate the importance of carefully structuring online learning environments to ensure high quality peer reviews.
- Carole Ames. 1992. Classrooms: Goals, structures, and student motivation. Journal of Ed Psych 84, 3, 261--271.Google Scholar
Cross Ref
- Erik Borg and Mary Deane. 2011. Measuring the outcomes of individualised writing instruction: A multilayered approach to capturing changes in students' texts. Tech in High Educ. 16, 3, 319--331.Google ScholarCross Ref
- David Boud, Ruth Cohen, and Jane Sampson (Eds.). 2014. Peer learning in higher education: Learning from and with each other. Routledge.Google Scholar
- Nicholas A. Bowman and Michael N. Bastedo. 2011. Anchoring effects in world university rankings: Exploring biases in reputation scores. High Educ. 61, 4, 431--444.Google ScholarCross Ref
- Paul C. Burnett. 2002. Teacher praise and feedback and students' perceptions of the classroom environment. Educ Psych. 22, 1, 5--16.Google ScholarCross Ref
- Bill Buxton. 2010. Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. Google ScholarDigital Library
- Karen Chalk, and Lewis A. Bizo. 2004. Specific praise improves on-task behaviour and numeracy enjoyment: A study of year four pupils engaged in the numeracy hour. Educ Psych in Prac. 20, 4, 335--351.Google ScholarCross Ref
- Winnie Cheng and Martin Warren. 1997. Having second thoughts: Student perceptions before and after a peer assessment exercise. Stud in High Educ. 22, 2, 233--239.Google ScholarCross Ref
- Kwangsu Cho and Charles MacArthur. 2010. Student revision with peer and expert reviewing. Learn and Instruc. 20, 4: 328--338.Google ScholarCross Ref
- Kwangsu Cho and Christian D. Schunn. 2007. Scaffolded writing and rewriting in the discipline: A webbased reciprocal peer review system. Comput. Educ. 48, 3 (April 2007), 409--426. Google ScholarDigital Library
- Marcy A. Church, Andrew J. Elliot, and Shelly L. Gable. 2001. Perceptions of classroom environment, achievement goals, and achievement outcomes. Jour of Educ Psych. 93, 1, 43--54.Google ScholarCross Ref
- D. Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A. Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 11391152. http://dx.doi.org/10.1145/2675133.2675251 Google ScholarDigital Library
- Coursera. Interaction Design Specialization {website}. Retrieved September 23, 2015 from https://www.coursera.org/specializations/interactiondesign/Google Scholar
- Phil Davies. 2000. Computerized peer assessment. Inno in Educ and Teach Inter. 37, 4, 346--355.Google ScholarCross Ref
- Elaine DiGiovanni and Girija Nagaswami. 2001. Online peer review: An alternative to face-toface? ELT Jour. 55, 3, 263--272.Google ScholarCross Ref
- Carol S. Dweck. 1986. Motivational processes affecting learning. Am Psych. 41, 10, 1040--1048.Google ScholarCross Ref
- Carol S. Dweck. 2006. Mindset: The new psychology of success. Random House.Google Scholar
- Carol S. Dweck and Ellen L. Leggert. 1988. A socialcognitive approach to motivation and personality. Psych Rev. 95, 2, 256--273.Google ScholarCross Ref
- Nicholas Epley and Thomas Gilovich. 2006. The anchoring-and-adjustment heuristic: Why the adjustments are insufficient. Psych Sci. 17, 4, 311--318.Google ScholarCross Ref
- Nancy Falchikov and Judy Goldfinch. 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Rev of Ed Res. 70, 3, 287--322.Google ScholarCross Ref
- Dana R. Ferris. 1997. The influence of teacher commentary on student revision. TESOL Quarterly. 31, 2, 315--339.Google ScholarCross Ref
- Kristel M. Gallagher and John A. Updegraff. 2012. Health message framing effects on attitudes, intentions, and behavior: a meta-analytic review. Annals of Beh Med. 43, 1, 101--116.Google ScholarCross Ref
- Joo Seng Mark Gan. 2011. The Effects of Prompts and Explicit Coaching on Peer Feedback Quality. Doctoral Dissertation. The University of Auckland, Auckland, New Zealand.Google Scholar
- Tony Gillie and Donald Broadbent. 1989. What makes interruptions disruptive? A study of length, similarity, and complexity. Psych Res 50, 4, 243--250.Google ScholarCross Ref
- John R. Hayes, Linda Flower, Karen A. Schriver, James F. Stratman, and Linda Carey. 1987. Cognitive processes in revision. In Advances in applied psycholinguistics, Volume II: Reading, writing, and language processing, Sheldon Rosenberg (Ed.). Cambridge, England, Cambridge University Press, 176--240.Google Scholar
- Jennifer Henderlong and Mark R. Lepper. 2002. The effects of praise on children's intrinsic motivation: A review and synthesis. Psych Bull. 128, 5, 774--795.Google ScholarCross Ref
- Melissa L. Kamins and Carol S. Dweck. 1999. Person versus process praise and criticism: Implications for contingent self-worth and coping. Dev Psych. 35, 3, 835--847.Google ScholarCross Ref
- James C. Kaufman, John Baer, Jason C. Cole and Janel D. Sexton. 2008. A comparison of expert and nonexpert raters using the consensual assessment technique. Creat Res Jour. 20, 2, 171--178.Google ScholarCross Ref
- Julia H. Kaufman and Christian D. Schunn. 2011. Students' perceptions about peer assessment for writing: Their origin and impact on revision work. Instruc Sci. 39, 3, 387--406.Google ScholarCross Ref
- Juho Kim, Elena L. Glassman, Andrés MonroyHernández and Meredith Ringel Morris. 2015. RIMES: Embedding interactive multimedia exercises in lecture videos. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1535--1544. http://dx.doi.org/10.1145/2702123.2702186 Google ScholarDigital Library
- Chinmay Kulkarni, Michael S. Bernstein and Scott Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings from The Second (2015) ACM Conference on Learning@ Scale (L@S '15). ACM, New York, NY, USA, 75--84. http://dx.doi.org/10.1145/2724660.2724670 Google ScholarDigital Library
- Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller and Scott R. Klemmer. 2013. Peer and self assessment in massive online classes. ACM Trans. Comput.- Hum. Interact. 20, 6, Article 33 (December 2013), 31 pages. http://dx.doi.org/10.1145/2505057 Google ScholarDigital Library
- Yasmine Kotturi, Chinmay Kulkarni, Michael S. Bernstein and Scott R. Klemmer. 2015. Structure and messaging techniques for online peer learning systems that increase stickiness. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale (L@S '15). ACM, New York, NY, USA, 3138. http://dx.doi.org/10.1145/2724660.2724676 Google ScholarDigital Library
- Irwin P. Levin, Gary J. Gaeth, Judy Schreiber and Marco Lauriola. 2002. A new look at framing effects: Distribution of effect sizes, individual differences, and independence of types of effects. Org Beha and Hum Dec Proc. 88, 1, 411--429.Google ScholarCross Ref
- S. S. J. Lin, E. Z. F. Liu and S. M. Yuan. 2001. Webbased peer assessment: Feedback for students with various thinking-styles. Jour of Comp Assi Learn. 17, 4, 420--432.Google ScholarCross Ref
- Melanie McComsey. 2015. 102B: Cognitive Ethnography. (Jan-Mar 2015). University of California, San Diego. Retrieved from https://www.peerstudio.org/courses/19Google Scholar
- Katie McEwan. 2013. Getting to know Coursera: Who is everyone? (20 February 2013). Retrieved November 30, 2015 from https://cft.vanderbilt.edu/2013/02/getting-to-knowcoursera-who-is-everyone/Google Scholar
- Medium. How a Udacity graduate earns $11k a month reviewing code. Retrieved September 21, 2014 from http://www.medium.com/@olivercameron/how-audacity- graduate-earns-11k-a-month-reviewing-codec2a7d295724cGoogle Scholar
- Hui-Tzu Min. 2005. Training students to become successful peer reviewers. System. 33, 2, 293--308.Google ScholarCross Ref
- Thomas E. Nelson, Zoe M. Oxley and Rosalee A. Clawson. 1997. Toward a psychology of framing effects. Pol Beh. 19, 3, 221--246.Google ScholarCross Ref
- Eleanor O'Rourke, Kyla Haimovitz, Christy Ballweber, Carol Dweck and Zoran Popović. 2014. Brain points: A growth mindset incentive structure boosts persistence in an educational game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 33393348. http://dx.doi.org/10.1145/2556288.2557157 Google ScholarDigital Library
- Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng and Daphne Koller. 2013. Tuned models of peer assessment in MOOCs. In International Conference on Educational Data Mining (EDM 2013).Google Scholar
- Robert Pinsky. 2014. ARPO222x: The Art of Poetry. BUx/Edx. (Sept-Nov 2014). Retrieved from https://www.edx.org/course/art-poetry-bux-arpo222xGoogle Scholar
- Jonathan A. Plucker, James C. Kaufman, Jason S. Temple, and Meihua Qian. 2009. Do experts and novices evaluate movies the same way? Psych & Mark. 26, 5, 470--478.Google ScholarCross Ref
- Michael Prince. 2004. Does active learning work? A review of the research. Jour of Eng Educ. 93, 223--232.Google ScholarCross Ref
- Chris Quintana, Brian J. Reiser, Elizabeth A. Davis, Joseph Krajcik, Eric Fretz, Ravit Golan Duncan, Eleni Kyza, Daniel Edelson and Elliot Soloway. 2004. A scaffolding design framework for software to support science inquiry. The Jour of the Learn Sci. 13, 3, 337386.Google ScholarCross Ref
- Sara Schroter, Nick Black, Stephen Evans, Fiona Godlee, Lyda Osorio and Richard Smith. 2008. What errors do peer reviewers detect, and does training improve their ability to detect them? Jour of the Royal Soc of Med. 101, 10, 507--514.Google ScholarCross Ref
- Dale H. Schunk and Carl W. Swartz. 1993. Goals and progress feedback: Effects on self-efficacy and writing achievement. Cont Ed Psych. 18, 3, 337--354.Google ScholarCross Ref
- Keith J. Topping. 2010. Methodological quandaries in studying process and outcomes in peer assessment. Learn and Instruc. 20, 4, 339--343. http://dx.doi.org/10.1016/j.learninstruc.2009.08.003Google ScholarCross Ref
- J. Gregory Trafton, Erik M. Altmann, Derek P. Brock and Farilee E. Mintz. 2003. Preparing to resume an interrupted task: Effects of prospective goal encoding and retrospective rehearsal. Inter Jour of Hu-Comp Stu. 58, 5, 583--603. Google ScholarDigital Library
- Chih-Hsiung Tu and Marina McIsaac. 2002. The relationship of social presence and interaction in online classes. The Am Jour of Dis Educ. 16, 3, 131--150.Google ScholarCross Ref
- Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Jour of Risk and Uncer. 5, 4, 297--323.Google ScholarCross Ref
- Anne Venables and Raymond Summit. 2003. Enhancing scientific essay writing using peer assessment. Inno in Educ and Teach Inter. 40, 3, 281--290.Google ScholarCross Ref
- University of California, Berkeley Career Center. Graduate School -- Statement {website}. Retrieved November 9, 2014 from https://career.berkeley.edu/Grad/GradStatementGoogle Scholar
- Miriam Walker, Leila Takayama and James A. Landay. 2002. High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting 46, 5 (September 2002), 661665. SAGE Publications.Google Scholar
- Mark E. Walvoord, Mariëlle H. Hoefnagels, Douglas D. Gaffin, Matthew M. Chumchal and David A. Long. 2008. An analysis of calibrated peer review (CPR) in a science lecture classroom. Jour of Coll Sci Teach. 37, 4, 66--73.Google Scholar
- Xu A, Huang SW, Bailey B. 2014. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1433--1444. http://dx.doi.org/10.1145/2531602.2531604 Google ScholarDigital Library
- Alvin Yuan, Kurt Luther, Markus Krause, Sophie Vennix, Steven P. Dow, Björn Hartmann. 2016. Almost an expert: The effects of rubrics and expertise on perceived value of crowdsourced design critiques. To appear in Proceedings of CSCW 2016. Google ScholarDigital Library
- Vivian Zamel. 1985. Responding to student writing. Tesol Quarterly. 19, 1, 79--101.Google ScholarCross Ref
Index Terms
- Framing Feedback: Choosing Review Environment Features that Support High Quality Peer Assessment
Recommendations
Adopting MOOCs for afforable quality education
Devgun (International Journal of Information and Computation Technology, 3(7), 641---646, (2013)) and Sarkar et al. (2016) have discussed models those help in overcoming challenges while adopting technology with higher education. But these models need ...
Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives
L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ ScaleMassive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer ...
MOOCs in Higher Education: A Study on the Learning Effectiveness, Quality and Usability of a SPOC on Introduction to Programming
PCI '20: Proceedings of the 24th Pan-Hellenic Conference on InformaticsIn most Universities today, the sheer size of students in a single class and the limited time for on-campus lessons or activities limit the learning experience for both students and professors. Small Private Online Courses (SPOCs), which are Massive ...
Comments