Top

Published in:

21-01-2023 | Original Paper

A Validation of AI-Enabled Discussion Platform Metrics and Relationships to Student Efforts

Authors: Audon Archibald, Cassie Hudson, Tania Heap, Ruthanne “Rudi” Thompson, Lin Lin, Jaqueline DeMeritt, Heather Lucke

Published in: TechTrends | Issue 2/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Asynchronous discussions are a popular feature in online higher education as they enable instructor-student and student–student interactions at the users’ own time and pace. AI-driven discussion platforms are designed to relieve instructors of automatable tasks, e.g., low-stakes grading and post moderation. Our study investigated the validity of an AI-generated score compared to human-driven methods of evaluating student effort and the impact of instructor interaction on students’ discussion post quality. A series of within-subjects MANOVAs was conducted on 14,599 discussion posts among over 800 students across four classes to measure post ‘curiosity score’ (i.e., an AI-generated metric of post quality) and word count. After checking assumptions, one MANOVA was run for each type of instructor interaction: private coaching, public praising, and public featuring. Instructor coaching appears to impact curiosity scores and word count, with later posts being an average of 40 words longer and scoring an average of 15 points higher than the original post that received instructor coaching. AI-driven tools appear to free up time for more creative human interventions, particularly among instructors teaching high-enrollment classes, where a traditional discussion forum is less scalable.

previous article What’s in a Name? The Meaning of “Learning, Design, and Technology”

next article The use of the Video Platform FlipGrid for Practicing Science Oral Communication

Abe, J. A. A. (2020). Big five, linguistic styles, and successful online learning. The Internet and Higher Education, 45. https://doi.org/10.1016/j.iheduc.2019.100724

Aloni, M., & Harrington, C. (2018). Research based practices for improving the effectiveness of asynchronous online discussion boards. Scholarship of Teaching and Learning in Psychology, 4(4), 271–289. https://doi.org/10.1037/stl0000121CrossRef

Arbaugh, J. B. (2010). Sage, guide, both, or even more? An examination of instructor activity in online MBA courses. Computers & Education, 55(3), 1234–1244. https://doi.org/10.1016/j.compedu.2010.05.020CrossRef

Bryant, J., Heitz, C., Sanghvi, S., & Waple, D. (2020). Artificial intelligence in education: How will it impact K-12 teachers | McKinsey. Retrieved April 10, 2020 from https://www.mckinsey.com/industries/education/our-insights/how-artificial-intelligence-will-impact-k-12-teachers

Bartolic, S., Matzat, U., Tai, J., Burgess, J. L., Boud, D., Craig, H., Archibald, A., De Jaeger, A., Kaplan-Rakowski, R., Lutze-Mann, L, Polly, P., Roth, M., Heap, T., Agapito, J., & Guppy, N. (2022). Student vulnerabilities and confidence in learning in the context of the COVID-19 pandemic. Studies in Higher Education, 47(12), 2460–2472. https://doi.org/10.1080/03075079.2022.2081679

Cheng, C. K., Paré, D. E., Collimore, L.-M., & Joordens, S. (2011). Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performance. Computers & Education, 56(1), 253–261. https://doi.org/10.1016/j.compedu.2010.07.024CrossRef

Dahlstrom-Hakki, I., Alstad, Z., & Banerjee, M. (2020). Comparing synchronous and asynchronous online discussions for students with disabilities: The impact of social presence. Computers & Education, 150. https://doi.org/10.1016/j.compedu.2020.103842

Dennen, V. P. (2008). Looking for evidence of learning: Assessment and analysis methods for online discourse. Computers in Human Behavior, 24(2), 205–219. https://doi.org/10.1016/j.chb.2007.01.010CrossRef

Ding, L., Er, E., & Orey, M. (2018). An exploratory study of student engagement in gamified online discussions. Computers & Education, 120, 213–226. https://doi.org/10.1016/j.compedu.2018.02.007CrossRef

Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find engaging? M. D., 10(2), 13.

Domingue, B. W., Hough, H. J., Lang, D., & Yeatman, J. (2021). Changing Patterns of Growth in Oral Reading Fluency during the COVID-19 Pandemic. Policy Analysis for California Education, PACE.

E. Gilbert, J. (2021). Equitable AI. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–2). Association for Computing Machinery. https://doi.org/10.1145/3411763.3457780

Elder, L., & Paul, R. (1998). The role of socratic questioning in thinking, teaching, and learning. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 71(5), 297–301. https://doi.org/10.1080/00098659809602729CrossRef

Garrison, D. R., & Anderson, T. (2003). E-Learning in the 21st Century: A Framework for Research and Practice (0 ed.). Routledge. https://doi.org/10.4324/9780203166093

Heap, T., Hudson, C., & Archibald, A. (2020). Investigating the impact of an AI-driven discussion platform on educator perceptions and feedback. EDEN Conference Proceedings, 1, 127–136. https://doi.org/10.38069/edenconf-2020-ac0010CrossRef

Hudson, C., Archibald, A., & Heap, T. (2020). Integrating an ai-driven discussion platform: The impact of platform on engagement and quality. EDEN Conference Proceedings, 1, 117–126. https://doi.org/10.38069/edenconf-2020-ac0009CrossRef

Hurt, N. E., Moss, G., Bradley, C., Larson, L., Lovelace, M., Prevost, L., Riley, N., Domizi, D., & Camus, M. (2012). The ‘facebook’ effect: College students’ perceptions of online discussions in the age of social networking. International Journal for the Scholarship of Teaching and Learning, 6(2). https://doi.org/10.20429/ijsotl.2012.060210

Indrajit, R. E., & Wibawa, B. (2020). Portrait of higher education in the covid-19 period in a digital literacy perspective: a reflection on the online lecture process experience. In 2020 Fifth International Conference on Informatics and Computing (ICIC) (pp. 1–5). IEEE.

Knowlton, D. S. (2005). A taxonomy of learning through asynchronous discussion. Journal of Interactive Learning Research, 16(2), 155–177.

Lantz, J. L., Liu, J. C., & Basnyat, I. (2022). Piloting Artificial Intelligence (AI) to Facilitate Online Discussion in Large Online Classes: A Case Study [Chapter]. Cases on Innovative and Successful Uses of Digital Resources for Online Learning; IGI Global. https://doi.org/10.4018/978-1-7998-9004-1.ch009

Lee, J.-E., & Recker, M. (2021). The effects of instructors’ use of online discussions strategies on student participation and performance in university online introductory mathematics courses. Computers & Education, 162, 104084. https://doi.org/10.1016/j.compedu.2020.104084CrossRef

Nedelkoska, L., & Quintini, G. (2018). Automation, skills use and training. OECD. https://doi.org/10.1787/2e2f4eea-enCrossRef

O’Brien, S., & Baugh, V. P. H. (2013). Effects of minimum word counts on writing tasks. Modern Psychological Studies 7, 11–15.

Park, J. H., Schallert, D. L., Sanders, A. J. Z., Williams, K. M., Seo, E., Yu, L.-T., Vogler, J. S., Song, K., Williamson, Z. H., & Knox, M. C. (2015). Does it matter if the teacher is there?: A teacher’s contribution to emerging patterns of interactions in online classroom discussions. Computers & Education, 82, 315–328. https://doi.org/10.1016/j.compedu.2014.11.019CrossRef

Peters, V. L., & Hewitt, J. (2010). An investigation of student practices in asynchronous computer conferencing courses. Computers & Education, 54(4), 951–961.CrossRef

Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1). https://doi.org/10.1186/s41039-017-0062-8

Rochera, M. J., Engel, A., & Coll, C. (2021). The effects of teacher feedback: a case study of an online discussion forum in Higher Education. Revista de Educación a Distancia (RED), 21(67). https://doi.org/10.6018/red.476901

Salter, N. P., & Conneely, M. R. (2015). Structured and unstructured discussion forums as tools for student engagement. Computers in Human Behavior, 46, 18–25. https://doi.org/10.1016/j.chb.2014.12.037CrossRef

Schartel, S. A. (2012). Giving feedback–An integral part of education. Best Practice & Research Clinical Anaesthesiology, 26(1), 77–87.CrossRef

Shum, S. B., Sándor, Á., Goldsmith, R., Bass, R., & McWilliams, M. (2017). Towards reflective writing analytics: rationale, methodology and preliminary results. Journal of Learning Analytics, 4(1), 58–84. https://doi.org/10.18608/jla.2017.41.5CrossRef

Skowronek, M., Gilberti, R. M., Petro, M., Sancomb, C., Maddern, S., & Jankovic, J. (2022). Inclusive STEAM education in diverse disciplines of sustainable energy and AI. Energy and AI, 7, 100124. https://doi.org/10.1016/j.egyai.2021.100124CrossRef

Smith, T.W. (2019). Making the Most of Online Discussion: A Retrospective Analysis. The International Journal of Teaching and Learning in Higher Education, 31, 21–31.

Stein, D. S., & Wanstreet, C. E. (2020). E-coaching success strategies for synchronous discussions. Distance Learning, 17(4), 113–118.

Stein, D. S., Wanstreet, C. E., Slagle, P., Trinko, L. A., & Lutz, M. (2013). From ‘hello’ to higher-order thinking: The effect of coaching and feedback on online chats. The Internet and Higher Education, 16, 78–84. https://doi.org/10.1016/j.iheduc.2012.03.001CrossRef

Wang, Q., & Woo, H. L. (2007). Comparing asynchronous online discussions and face-to-face discussions in a classroom setting. British Journal of Educational Technology, 38(2), 272–286. https://doi.org/10.1111/j.1467-8535.2006.00621.xCrossRef

Wuttikietpaiboon, K. (2013). Engaging graduate students in rich asynchronous online discussions (2013–99090–524; Issues 11-A(E)) [ProQuest Information & Learning]. Retrieved January, 2021 from https://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=2013-99090-524&site=ehost-live

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0CrossRef

Zydney, J. M., deNoyelles, A., & Kyeong-Ju Seo, K. (2012). Creating a community of inquiry in online environments: An exploratory study on the effect of a protocol on interactions within asynchronous discussions. Computers & Education, 58(1), 77–87. https://doi.org/10.1016/j.compedu.2011.07.009CrossRef

Title: A Validation of AI-Enabled Discussion Platform Metrics and Relationships to Student Efforts
Authors: Audon Archibald
Cassie Hudson
Tania Heap
Ruthanne “Rudi” Thompson
Lin Lin
Jaqueline DeMeritt
Heather Lucke
Publication date: 21-01-2023
Publisher: Springer US
Published in: TechTrends / Issue 2/2023
Print ISSN: 8756-3894
Electronic ISSN: 1559-7075
DOI: https://doi.org/10.1007/s11528-022-00825-7

Springer Professional

A Validation of AI-Enabled Discussion Platform Metrics and Relationships to Student Efforts

Abstract

Premium Partner

Springer Professional

Abstract

Please log in to get access to your license.

Other articles of this Issue 2/2023

Introducing the March 2023 Issue

Peer Assessment Collaboration Evaluation: An Innovative Assessment Tool for Online Learning Environments

Supporting Dynamic Instructional Design Decisions Within a Bounded Rationality

The Charm or Chasm of Digital Personalized Learning in Education: Teachers’ Reported Use, Perceptions and Expectations

Correction: Review of Kirschner, P.A., Hendrick, C., and Heal, J. (2022). How Teaching Happens: Seminal Works in Teaching and Teacher Effectiveness and What they Mean in Practice. Routledge. ISBN: 978-1-032-13208-2

The Flipped Classroom Approach in High School Psychology: An Action Research Study