Ken Reily
Loren Terveen
ABSTRACT
Scientific peer review, open source software development, wikis, and other domains use distributed review to improve quality of created content by providing feedback to the work's creator. Distributed review is used to assess or improve the quality of a work (e.g., an article). However, it can also provide learning benefits to the participants in the review process. We developed an online review system for beginning computer programming students; it gathers multiple anonymous peer reviews to give students feedback on their programming work. We deployed the system in an introductory programming class and evaluated it in a controlled study. We find that: peer reviews are accurate compared to an accepted evaluation standard, that students prefer reviews from other students with less experience than themselves, and that participating in a peer review process results in better learning outcomes.
- Anewalt, K. Using peer review as a vehicle for communication skill development and active learning. J. Comput. Small Coll. 21, 2 (2005), 148--155. Google Scholar
Digital Library
- Bloom, B., Englehart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. Taxonomy of Educational Objectives: The Classification of Educational Goals - Handbook 1: Cognitive Domain.Google Scholar
- David McKay Company, Inc., New York, 1956.Google Scholar
- Cho, K., Chung, T. R., King, W. R., and Schunn, C. Peer-based computer-supported knowledge refinement: an empirical investigation. Commun. ACM 51, 3 (2008), 83--88. Google ScholarDigital Library
- Clancey, W. J. From guidon to neomycin and heracles in twenty short lessons. AI Mag. 7, 3 (1986), 40--60. Google ScholarDigital Library
- Collofello, J. S. Teaching technical reviews in a one-semester software engineering course. In SIGCSE '87: Proceedings of the eighteenth SIGCSE technical symposium on Computer science education (New York, NY, USA, 1987), ACM, pp. 222--227. Google ScholarDigital Library
- Cosley, D., Frankowski, D., Terveen, L., and Riedl, J. Using intelligent task routing and contribution review to help communities build artifacts of lasting value. In CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems (New York, NY, USA, 2006), ACM, pp. 1037--1046. Google ScholarDigital Library
- Denning, T., Kelly, M., Lindquist, D., Malani, R., Griswold, W. G., and Simon, B. Lightweight preliminary peer review: does in-class peer review make sense? In SIGCSE '07: Proceedings of the 38th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2007), ACM, pp. 266--270. Google ScholarDigital Library
- Foltz, P. W., Laham, D., and Landauer, T. K. Automated essay scoring: Applications to education technology. In Proceedings of ED-MEDIA (1999), pp. 939--944.Google Scholar
- Gehringer, E. Strategies and mechanisms for electronic peer review. Frontiers in Education Conference, 2000. FIE 2000. 30th Annual 1 (2000), F1B/2-F1B/7 vol.1. Google ScholarDigital Library
- Gehringer, E. F. Electronic peer review and peer grading in computer-science courses. SIGCSE Bull. 33, 1 (2001), 139--143. Google ScholarDigital Library
- Gehringer, E. F. Electronic peer review builds resources for teaching computer architecture. In Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition (2003), American Society for Engineering Education.Google ScholarCross Ref
- Gehringer, E. F., Chinn, D. D., Manuel A. Pérez-Qui n., and Ardis, M. A. Using peer review in teaching computing. In SIGCSE '05: Proceedings of the 36th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2005), ACM, pp. 321--322. Google ScholarDigital Library
- Gehringer, E. F., Ehresman, L. M., and Skrien, D. J. Expertiza: students helping to write an ood text. In OOPSLA '06: Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications (New York, NY, USA, 2006), ACM, pp. 901--906. Google ScholarDigital Library
- Gotel, O., Scharff, C., and Wildenberg, A. Extending and contributing to an open source web-based system for the assessment of programming problems. In PPPJ '07: Proceedings of the 5th international symposium on Principles and practice of programming in Java (New York, NY, USA, 2007), ACM, pp. 3--12. Google ScholarDigital Library
- Gotel, O., Scharff, C., and Wildenberg, A. Teaching software quality assurance by encouraging student contributions to an open source web-based system for the assessment of programming assignments. In ITiCSE '08: Proceedings of the 13th annual conference on Innovation and technology in computer science education (New York, NY, USA, 2008), ACM, pp. 214--218. Google ScholarDigital Library
- Hamer, J., Kell, C., and Spence, F. Peer assessment using aropä. In ACE '07: Proceedings of the ninth Australasian conference on Computing education (Darlinghurst, Australia, Australia, 2007), Australian Computer Society, Inc., pp. 43--54. Google ScholarDigital Library
- Hamer, J., Ma, K. T. K., and Kwong, H. H. F. A method of automatic grade calibration in peer assessment. In ACE '05: Proceedings of the 7th Australasian conference on Computing education (Darlinghurst, Australia, Australia, 2005), Australian Computer Society, Inc., pp. 67--72. Google ScholarDigital Library
- Herlocker, J. L., Konstan, J. A., and Riedl, J. Explaining collaborative filtering recommendations. In CSCW '00: Proceedings of the 2000 ACM conference on Computer supported cooperative work (New York, NY, USA, 2000), ACM, pp. 241--250. Google ScholarDigital Library
- Hinds, P. J. The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance. Journal of Experimental Psychology: Applied 5, 2 (1999), 205--221.Google ScholarCross Ref
- Krathwohl, D. R., Bloom, B. S., and Masia, B. Taxonomy of Educational Objectives: The Classification of Educational Goals - Handbook 2: Affective Domain, 1 ed. Longman, London, UK, July 1964.Google Scholar
- Lampe, C., and Resnick, P. Slash(dot) and burn: distributed moderation in a large online conversation space. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems (New York, NY, USA, 2004), ACM, pp. 543--550. Google ScholarDigital Library
- Liu, E. Z.-F., Lin, S., Chiu, C.-H., and Yuan, S.-M. Web-based peer review: the learner as both adapter and reviewer. Education, IEEE Transactions on 44, 3 (Aug 2001), 246--251. Google ScholarDigital Library
- Resnick, P., Kuwabara, K., Zeckhauser, R., and Friedman, E. Reputation systems. Commun. ACM 43, 12 (2000), 45--48. Google ScholarDigital Library
- Silva, E., and Moreira, D. Webcom: a tool to use peer review to improve student interaction. J. Educ. Resour. Comput. 3, 1 (2003), 3. Google ScholarDigital Library
- Sullivan, S. L. Reciprocal peer reviews. In SIGCSE '94: Proceedings of the twenty-fifth SIGCSE symposium on Computer science education (New York, NY, USA, 1994), ACM, pp. 314--318. Google ScholarDigital Library
- Topping, K. Peer assessment between students in colleges and universities. Review of Educational Research 68, 3 (1998), 249--276.Google ScholarCross Ref
- Trivedi, A., Kar, D. C., and Patterson-McNeill, H. Automatic assignment management and peer evaluation. J. Comput. Small Coll. 18, 4 (2003), 30--37. Google ScholarDigital Library
- Trytten, D. Progressing from small group work to cooperative learning: a case study from computer science. Frontiers in Education Conference, 1999. FIE'99. 29th Annual 2 (1999), 13A4/22--13A4/27 vol.2.Google ScholarCross Ref
- Trytten, D. A. A design for team peer code review. In SIGCSE '05: Proceedings of the 36th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2005), ACM, pp. 455--459. Google ScholarDigital Library
- Wolfe, W. J. Online student peer reviews. In CITC5 '04: Proceedings of the 5th conference on Information technology education (New York, NY, USA, 2004), ACM, pp. 33--37. Google ScholarDigital Library
- Zhang, J., Ackerman, M. S., and Adamic, L. Expertise networks in online communities: structure and algorithms. In WWW '07: Proceedings of the 16th international conference on World Wide Web (New York, NY, USA, 2007), ACM, pp. 221--230. Google ScholarDigital Library
Index Terms
- Two peers are better than one: aggregating peer reviews for computing assignments is surprisingly accurate
Recommendations
Can Students Review Their Peers?: Comparison of Peer and Instructor Reviews
ITiCSE '22: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1Having students peer review each other's exercises is a common task in modern computing classrooms. In large classes, peer review might even partly replace traditional instructor-led review -- and prior work has found some indications that the quality ...
Trustworthiness of Peers in P2P Overlay Networks
NBIS '12: Proceedings of the 2012 15th International Conference on Network-Based Information SystemsIn peer-to-peer (P2P) overlay networks, a group of multiple peers have to cooperate with each other. P2P systems are in nature scalable distributed systems, where there is no centralized coordinator. It is difficult for each peer to communicate with ...
Trustworthy Cooperation among Peers
ICDCSW '09: Proceedings of the 2009 29th IEEE International Conference on Distributed Computing Systems WorkshopsIn a peer-to-peer (P2P) overlay network, a peer process (peer) communicates with other peers and manipulates objects in local and remote peers. In a fully distributed, unstructured P2P network, there is no centralized coordinator like index and super ...
Comments