Top

International Journal of Technology and Design Education

Published in:

01-01-2021

A systematized review of research with adaptive comparative judgment (ACJ) in higher education

Authors: Scott R. Bartholomew, Matthew D. Jones

Published in: International Journal of Technology and Design Education | Issue 2/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Adaptive Comparative Judgment (ACJ), an approach to the assessment of open-ended problems which utilizes a series of comparisons to produce a standardized score, rank order, and a variety of other statistical measures, has demonstrated high levels of reliability and validity and the potential for application in a wide variety of areas. Further, research into using ACJ, both as a formative and summative assessment tool, has been conducted in multiple contexts across higher education. This systematized review of ACJ research outlines our approach to identifying, classifying, and organizing findings from research with ACJ in higher education settings as well as overarching themes and questions that remain. The intent of this work is to provide readers with an understanding of the current state of the field and several areas of potential further inquiry related to ACJ in higher education.

previous article A computational model for subjective evaluation of novelty in descriptive aptitude

next article Learning by evaluating (LbE) through adaptive comparative judgment

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Akister, J., Bannon, A., & Mullender-Lock, H. (2000). Poster presentations in social work education assessment: A case study. Innovations in Education and Training International, 37(3), 229–233.

Baartman, L. K., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. (2007). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational Research Review, 2(2), 114–129.

Barber, J. (2018). Five go marking an exam question: The use of adaptive comparative judgement to manage subjective bias. Practitioner Research in Higher Education, 11(1), 94–100.

Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education Teacher, 76(6), 13–17.

Bartholomew, S. R., Mentzer, N., & Jones, M. (2019a). Learning by evaluating (LbE). In Mississippi Valley Technology Education Conference. Nashville, TN.

Bartholomew, S. R., Strimel, G. J., & Jackson, A. (2018a). A comparison of traditional and adaptive comparative judgment assessment techniques for freshman engineering design projects. International Journal of Engineering Education, 34(1), 20–33.

Bartholomew, S. R., & Yoshikawa, E. (2018). A systematic review of research around adaptive comparative judgment (ACJ) in K-16 education. 2018 CTETE Monograph Series. Retrieved from https://doi.org/10.21061/ctete-rms.v1.c.1.

Bartholomew, S. R., Yoshikawa, E., & Connolly, P. E. (2018b). Exploring the potential for identifying student competencies in design education through adaptive comparative judgment. In PATT35 Athlone Institute of Technology, Athlone, Ireland 18–21 June, 2018 (pp. 187–194).

Bartholomew, S., Zhang, L., Bravo, E. G., & Strimel, G. J. (2019b). A tool for formative assessment and learning in a graphics design course: Adaptive comparative judgement. The Design Journal, 22(1), 73–95.

Bartholomew, S., Zhang, L., Bravo, E. G., & Strimel, G. J. (2019c). First, last, elsewhere… positioning adaptive comparative judgment in the design learning experience. In PATT 37 University of Malta, Malta, June 2–6, 2019 (pp. 65–74).

Bell, S. (2010). Project-based learning for the 21st century: Skills for the future. The Clearing House, 83(2), 39–43.

Benton, T., & Gallagher, T. (2018). Is comparative judgement just a quick form of multiple marking? Research Matters: A Cambridge Assessment Publication, 26, 22–28.

Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic literature reviews in engineering education and other developing interdisciplinary fields. Journal of Engineering Education, 103(1), 45–76.

Boud, D., & Feletti, G. (2013). The challenge of problem-based learning. London: Routledge.

Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment (p. 36). Cambridge: Cambridge Assessment.

Bramley, T., & Vitello, S. (2018). The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(1), 43–58.

Buckley, J., Canty, D., & Seery, N. (2020). An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education. Irish Educational Studies. https://doi.org/10.1080/03323315.2020.1814838.

Canty, D. (2012). The impact of holistic assessment using adaptive comparative judgment of student learning. PhD Thesis, University of Limerick, Ireland.

Canty, D., Buckley, J., & Seery, N. (2019). Inducting ITE students in assessment practices through the use of comparative judgment. In PATT 37 University of Malta, Malta, June 2–6, 2019 (pp. 117–124).

Canty, D., Seery, N., Hartell, E., & Doyle, A. (2017). Integrating peer assessment in technology education through adaptive comparative judgment. In PATT34 Technology & Engineering Education–Fostering the Creativity of Youth Around the Globe, Millersville University, Pennsylvania, USA (pp. 10–14).

Demonacos, C., Ellis, S., & Barber, J. (2019). Student peer assessment using adaptive comparative judgment: Grading accuracy versus quality of feedback. Practitioner Research in Higher Education, 12(1), 50–59.

Dobson, S. (2006). The assessment of student PowerPoint presentations—Attempting the impossible? Assessment & Evaluation in Higher Education, 31(1), 109–119.

Duran, M., & Dökme, İ. (2016). The effect of the inquiry-based learning approach on student’s critical-thinking skills. Eurasia Journal of Mathematics, Science & Technology Education, 12(12), 2887–2908.

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108.

Hannafin, M. J., Hall, C., Land, S., & Hill, J. (1994). Learning in open-ended environments: Assumptions, methods, and implications. Educational Technology, 34(8), 48–55.

Jones, I., & Alcock, L. (2012). Summative peer assessment of undergraduate calculus using adaptive comparative judgement. In P. Iannone & A. Simpson (Eds.), Mapping university mathematics assessment practices. Norwich: University of East Anglia.

Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal, 12(2), 66–76.

Kimbell, R. (2012a). Evolving project e-scape for national assessment. International Journal of Technology & Design Education, 22, 135–155.

Kimbell, R. (2012b). The origins and underpinning principles of e-scape. International Journal of Technology & Design Education, 22, 123–134.

Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86.

Kumar, M., & Natarajan, U. (2007). Alternative assessment in problem-based learning: Strengths, shortcomings and sustainability. i-Manager’s Journal on Educational Psychology, 1(1), 27.

Lazonder, A. W., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 86(3), 681–718.

Martin-Martin, A., Orduña-Malea, E., Harzing, A. W., & López-Cózar, E. D. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163.

Metzgar, M. (2016). Using adaptive comparative judgement to assess student work in an MBA course. International Journal for Infonomics, 9(3), 1217–1219.

Mills, J. E., & Treagust, D. F. (2003). Engineering education—Is problem-based or project-based learning the answer. Australasian Journal of Engineering Education, 3(2), 2–16.

Moskal, B. M., Leydens, J. A., & Pavelich, M. J. (2002). Validity, reliability and the assessment of engineering education. Journal of Engineering Education, 91(3), 351–354.

Munroe, L. (2015). The open-ended approach framework. European Journal of Educational Research, 4(3), 97–104.

Newhouse, P. (2011). Comparative pairs marking supports authentic assessment of practical performance within constructivist learning environments. In Applications of Rasch measurement in learning environments research (pp. 141–180). Rotterdam: Sense Publishers.

Noruzi, A. (2005). Google Scholar: The new generation of citation indexes. Libri, 55(4), 170–180.

Pollitt, A. (2004). Let’s stop marking exams. Retrieved July 23, 2018, from http://www.cambridgeassessment.org.uk/images/109719-let-s-stop-marking-exams.pdf.

Pollitt, A. (2012). The method of adaptive comparative judgment. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.

Pollitt, A. (2015). On ‘reliability’ bias in ACJ. Cambridge Exam Research. Retrieved February 2, 2018, from https://www.researchgate.net/publication/283318012_On_‘Reliability’_bias_in_ACJ.

Pollitt, A., & Murray, N. L. (1996). What raters really pay attention to. Studies in Language Testing, 3, 74–91.

Potter, T., Englund, L., Charbonneau, J., MacLean, M. T., Newell, J., & Roll, I. (2017). ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching & Learning Inquiry, 5(2), 89–113.

Purzer, S., Fila, N., & Nataraja, K. (2016). Evaluation of current assessment methods in engineering entrepreneurship education. Advances in Engineering Education, 5(1), n1.

Rangel-Smith, C., & Lynch, D. (2018). Addressing the issue of bias in the measurement of reliability in the method of adaptive comparative judgment. In PATT35 Athlone Institute of Technology, Athlone, Ireland 18–21 June, 2018 (pp. 378–387).

Rhind, S. M., Hughes, K. J., Yool, D., Shaw, D., Kerr, W., & Reed, N. (2017). Adaptive comparative judgment: A tool to support students’ assessment literacy. Journal of Veterinary Medical Education, 44(4), 686–691.

Rowsome, P., Seery, N., & Lane, D. (2013). The development of pre-service design educator’s capacity to make professional judgments on design capability using adaptive comparative judgment. American Society for Engineering Education.

Seery, N., Buckley, J., Delahunty, T., & Canty, D. (2018). Integrating learners into the assessment process using adaptive comparative judgement with an ipsative approach to identifying competence based gains relative to student ability levels. International Journal of Technology and Design Education, 29, 701–715. https://doi.org/10.1007/s10798-018-9468-x.CrossRef

Seery, N., Buckley, J., Doyle, A., & Canty, D. (2016). The validity and reliability of adaptive comparative judgements in the assessment of graphical capability. In Proceedings of the 71st Mid-Year Conference of the Engineering Design Graphics Division (pp. 104–109).

Seery, N., Canty, D., & Phelan, P. (2012). The validity and value of peer assessment using adaptive comparative judgement in design driven practical education. International Journal of Technology and Design Education, 22(2), 205–226.

Seery, N., Lane, D., & Canty, D. (2011). Exploring the value of democratic assessment in design based activities of graphical education. In 118th Annual American Society of Engineering Education Conference. Vancouver, BC: American Society for Engineering Education.

Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgment as an approach to essay scoring. Applied Measurement in Education, 29(3), 211–223.

Stevens, D. D., & Levi, A. J. (2013). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus Publishing, LLC.

Strimel, G. J., Bartholomew, S. R., Purzer, S., Yoshikawa, E., & Zhang, L. (2020). Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education. https://doi.org/10.1080/03043797.2020.1718614.

Thomas, J. W. (2000). A review of research on project-based learning. Autodesk Foundation. Retrieved from https://www.asec.purdue.edu/lct/HBCU/documents/AReviewofResearchofProject-BasedLearning.pdf.

Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.

Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562.

Viseu, F., & Oliveira, I. B. (2017). Open-ended tasks in the promotion of classroom communication in mathematics. International Electronic Journal of Elementary Education, 4(2), 287–300.

Zhang, L. (2019). Investigating differences in formative critiquing between instructors and students in graphic design. Doctoral dissertation, Purdue University Libraries.

Zientek, L. R., Werner, J. M., Campuzano, M. V., & Nimon, K. (2018). The use of Google Scholar for research and research dissemination. New Horizons in Adult Education and Human Resource Development, 30(1), 39–46.

Title: A systematized review of research with adaptive comparative judgment (ACJ) in higher education
Authors: Scott R. Bartholomew
Matthew D. Jones
Publication date: 01-01-2021
Publisher: Springer Netherlands
Published in: International Journal of Technology and Design Education / Issue 2/2022
Print ISSN: 0957-7572
Electronic ISSN: 1573-1804
DOI: https://doi.org/10.1007/s10798-020-09642-6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Other articles of this Issue 2/2022

Robotics and information technologies in education: four countries from Alpe-Adria-Danube Region survey

Using game design to increase teachers’ familiarity with design thinking

Transcending disciplines in architecture, structural and building services engineering: a new multidisciplinary educational approach

Measuring industrial design self-efficacy and anxiety

Short and long term impact of a high-tech STEM intervention on pupils’ attitudes towards technology

An importance–performance analysis (IPA) of teachers’ core competencies for implementing maker education in primary and secondary schools

Premium Partner