Skip to main content
SpringerLink
Log in

Assessing Collaborative Problem Solving Through Conversational Agents

  • Chapter
  • First Online:
Innovative Assessment of Collaboration

Part of the book series: Methodology of Educational Measurement and Assessment ((MEMA))

Abstract

Communication is a core component of collaborative problem solving and its assessment. Advances in computational linguistics and discourse science have made it possible to analyze conversation on multiple levels of language and discourse in different educational settings. Most of these advances have focused on tutoring contexts in which a student and a tutor collaboratively solve problems, but there has also been some progress in analyzing conversations in small groups. Naturalistic patterns of collaboration in one-on-one tutoring and in small groups have also been compared with theoretically ideal patterns. Conversation-based assessment is currently being applied to measure various competencies, such as literacy, mathematics, science, reasoning, and collaborative problem solving. One conversation-based assessment approach is to design computerized conversational agents that interact with the human in natural language. This chapter reports research that uses one or more agents to assess human competencies while the humans and agents collaboratively solve problems or answer difficult questions. AutoTutor holds a collaborative dialogue in natural language and concurrently assesses student performance. The agent converses through a variety of dialogue moves: questions, short feedback, pumps for information, hints, prompts for specific words, corrections, assertions, summaries, and requests for summaries. Trialogues are conversations between the human and two computer agents that play different roles (e.g., peer, tutor, expert). Trialogues are being applied in both training and assessment contexts on particular skills and competencies. Agents are currently being developed at Educational Testing Service for assessments of individuals on various competencies, including the Programme for International Student Assessment 2015 assessment of collaborative problem solving.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Biswas, G., Jeong, H., Kinnebrew, J., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology-Enhanced Learning, 5, 123–152.

    Article  Google Scholar 

  • Cade, W., Copeland, J., Person, N., & D’Mello, S. K. (2008). Dialogue modes in expert tutoring. In B. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the ninth international conference on intelligent tutoring systems (pp. 470–479). Berlin: Springer.

    Google Scholar 

  • Cai, Z., Feng, S., Baer, W., & Graesser, A. (2014). Instructional strategies in trialog-based intelligent tutoring systems. In R. Sottilare, A. C. Graesser, X. Hu & B. Goldberg (Eds.), Design recommendations for intelligent tutoring systems: Adaptive instructional strategies (Vol. 2, pp. 225–235). Orlando, FL: Army Research Laboratory.

    Google Scholar 

  • Cai, Z., Graesser, A. C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., & Butler, H. (2011). Trialog in ARIES: User input assessment in an intelligent tutoring system. In W. Chen & S. Li (Eds.), Proceedings of the 3rd IEEE international conference on intelligent computing and intelligent systems (pp. 429–433). Guangzhou, China: IEEE Press.

    Google Scholar 

  • Cai, Z., Graesser, A. C., & Hu, X. (2015). ASAT: AutoTutor script authoring tool. In. R. Sottilare, A. C. Graesser, X. Hu & K. Brawner (Eds.), Design recommendations for intelligent tutoring systems: Authoring tools (Vol. 3, pp. 199–210). Orlando, FL: Army Research Laboratory.

    Google Scholar 

  • Care, E., & Griffin, P. (2014). An approach to assessment of collaborative problem solving. Research and Practice in Technology Enhanced Learning, 9, 367–388.

    Google Scholar 

  • Chi, M. T. H., Siler, S., Yamauchi, T., Jeong, H., & Hausmann, R. (2001). Learning from human tutoring. Cognitive Science, 25, 471–534.

    Article  Google Scholar 

  • Cohen, P. A., Kulik, J. A., & Kulik, C. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19, 237–248.

    Article  Google Scholar 

  • D’Mello, S., Dowell, N., & Graesser, A. C. (2011). Does it really matter whether students’ contributions are spoken versus typed in an intelligent tutoring system with natural language? Journal of Experimental Psychology: Applied, 17, 1–17.

    Google Scholar 

  • D’Mello, S., & Graesser, A. C. (2012). Language and discourse are powerful signals of student emotions during tutoring. IEEE Transactions on Learning Technologies, 5, 304–317.

    Article  Google Scholar 

  • D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. C. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170.

    Article  Google Scholar 

  • Dede, C. (2015). Data-intensive research in education: Current work and next steps. Retrieved from http://cra.org/cra-releases-report-on-data-intensive-research-in-education/

  • Dowell, N. M., Graesser, A. C., & Cai, Z. (in press). Language and discourse analysis with Coh-Metrix: Applications from educational material to learning environments at scale. Journal of Learning Analytics., XX, XX–XX.

    Google Scholar 

  • Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M., & Warner, N. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52, 203–224.

    Article  Google Scholar 

  • Forsyth, C. M., Graesser, A. C., Pavlik, P., Cai, Z., Butler, H., Halpern, D. F., et al. (2013). OperationARIES! Methods, mystery and mixed models: Discourse features predict affect in a serious game. Journal of Educational Data Mining, 5, 147–189.

    Google Scholar 

  • Graesser, A. C., Baer, W., Feng, S., Walker, B., Clewley, D., Hayes, D. P., & Greenberg, D. (2015). Emotions in adaptive computer technologies for adults improving learning. In S. Tettegah & M. Gartmeier (Eds.), Emotions, technology, design and learning: Communication, for with and through digital media (pp. 1–35). San Diego, CA: Elsevier.

    Google Scholar 

  • Graesser, A. C., D’Mello, S. K., & Cade, W. (2011). Instruction based on tutoring. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 408–426). New York: Routledge Press.

    Google Scholar 

  • Graesser, A. C., D’Mello, S. K., Hu, X., Cai, Z., Olney, A., & Morgan, B. (2012). AutoTutor. In P. McCarthy & C. Boonthum-Denecke (Eds.), Applied natural language processing: Identification, investigation, and resolution (pp. 169–187). Hershey, PA: IGI Global.

    Google Scholar 

  • Graesser, A. C., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. (in press). Challenges of assessing collaborative problem solving. In E. Care, P. Griffin & M. Wilson (Eds.), Assessment and teaching of 21st century skills. Heidelberg, Germany: Springer.

    Google Scholar 

  • Graesser, A. C., Forsyth, C. M., & Foltz, P. (2016). Assessing conversation quality, reasoning, and problem solving performance with computer agents. In B. Csapo, J. Funke & A. Schleicher (Eds.), On the nature of problem solving: A look behind PISA 2012 problem solving assessment (pp. 275–297). Heidelberg, Germany: OECD Series.

    Google Scholar 

  • Graesser, A. C., Forsyth, C., & Lehman, B. (in press). Two heads are better than one: Learning from computer agents in conversational trialogues. Teachers College Record.

    Google Scholar 

  • Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45, 298–322.

    Article  Google Scholar 

  • Graesser, A. C., Li, H., & Forsyth, C. (2014). Learning by communicating in natural language with conversational agents. Current Directions in Psychological Science, 23, 374–380.

    Article  Google Scholar 

  • Graesser, A. C., McNamara, D. S., Cai, Z., Conley, M., Li, H., & Pennebaker, J. (2014). Coh-Metrix measures text characteristics at multiple levels of language and discourse. Elementary School Journal, 115, 210–229.

    Article  Google Scholar 

  • Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., et al. (2004). AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, and Computers, 36, 180–193.

    Article  Google Scholar 

  • Graesser, A. C., & McNamara, D. S. (2012). Automated analysis of essays and open-ended verbal responses. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf & K. J. Sher (Eds.), APA handbook of research methods in psychology: Vol. 1. Foundations, planning, measures, and psychometrics (pp. 307–325). Washington, DC: American Psychological Association.

    Google Scholar 

  • Graesser, A. C., McNamara, D. S., & VanLehn, K. (2005). Scaffolding deep comprehension strategies through Point & Query, AutoTutor, and iSTART. Educational Psychologist, 40, 225–234.

    Article  Google Scholar 

  • Graesser, A. C., Person, N. K., & Magliano, J. P. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9, 495–522.

    Article  Google Scholar 

  • Griffin, P., & Care, E. (Eds.). (2015). Assessment and teaching of 21st century skills: Methods and approach. Dordrecht, Netherlands: Springer.

    Google Scholar 

  • Halpern, D. F., Millis, K., Graesser, A. C., Butler, H., Forsyth, C., & Cai, Z. (2012). Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity, 7, 93–100.

    Article  Google Scholar 

  • Jackson, G. T., & Graesser, A. C. (2006). Applications of human tutorial dialog in AutoTutor: An intelligent tutoring system. Revista Signos, 39, 31–48.

    Google Scholar 

  • Jackson, G. T., & McNamara, D. S. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105, 1036–1049.

    Article  Google Scholar 

  • Jurafsky, D., & Martin, J. H. (2008). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  • Lehman, B., D’Mello, S. K., Strain, A., Mills, C., Gross, M., Dobbins, A., & Graesser, A. C. (2013). Inducing and tracking confusion with contradictions during complex learning. International Journal of Artificial Intelligence in Education, 22, 85–105.

    Google Scholar 

  • McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge, MA: Cambridge University Press.

    Book  Google Scholar 

  • Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., & Halpern, D. (2011). Operation ARIES! A serious game for teaching scientific inquiry. In M. Ma, A. Oikonomou & J. Lakhmi (Eds.), Serious games and edutainment applications (pp. 169–196). London, England: Springer.

    Google Scholar 

  • Morgan, B., Keshtkar, F., Duan, Y., & Graesser, A. C. (2012). Using state transition networks to analyze multi-party conversations in a serious game. In S. A. Cerri & B. Clancey (Eds.), Proceedings of the 11th international conference on intelligent tutoring systems (ITS 2012) (pp. 162–167). Berlin, Germany: Springer.

    Google Scholar 

  • Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24, 427–469.

    Article  Google Scholar 

  • OECD. (2013). PISA 2015 collaborative problem solving framework. (Paris, France: OECD) Retrieved from http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf

  • Olney, A., D’Mello, S. K., Person, N., Cade, W., Hays, P., Williams, C., & Graesser, A. C. (2012). Guru: A computer tutor that models expert human tutors. In S. Cerri, W. Clancey, G. Papadourakis & K. Panourgia (Eds.), Proceedings of intelligent tutoring systems (ITS) 2012 (pp. 256–261). Berlin, Germany: Springer.

    Google Scholar 

  • Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). LIWC2007: Linguistic inquiry and word count. Austin, TX: LIWC.net.

    Google Scholar 

  • Rosé, C., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., et al. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3, 237–271.

    Article  Google Scholar 

  • Rowe, J., Shores, L. R., Mott, B., & Lester, J. (2010). Integrating learning, problem solving, and engagement in narrative-centered learning environments. International Journal of Artificial Intelligence in Education, 20, 166–177.

    Google Scholar 

  • Rus, V., D’Mello, S., Hu, X., & Graesser, A. C. (2013). Recent advances in intelligent systems with conversational dialogue. AI Magazine, 34, 42–54.

    Google Scholar 

  • Rus, V., McCarthy, P. M., McNamara, D. S., & Graesser, A. C. (2008). A study of textual entailment. International Journal on Artificial Intelligence Tools, 17, 659–685.

    Article  Google Scholar 

  • Shute, V. J., & Ventura, M. (2013). Measuring and supporting learning in games: Stealth assessment. Cambridge, MA: MIT Press.

    Google Scholar 

  • Sottilare, R., Graesser, A., Hu, X., & Holden, H. (Eds.). (2013). Design recommendations for intelligent tutoring systems: Learner modeling (Vol. 1). Orlando, FL: Army Research Laboratory. Sottilare, Robert Graesser, Art Hu, Xiangen Holden, Heather.

    Google Scholar 

  • VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring systems. Educational Psychologist, 46, 197–221.

    Article  Google Scholar 

  • VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.

    Article  Google Scholar 

  • Ward, W., Cole, R., Bolaños, D., Buchenroth-Martin, C., Svirsky, E., & Weston, T. (2013). My science tutor: A conversational multimedia virtual tutor. Journal of Educational Psychology, 105, 1115–1125.

    Article  Google Scholar 

  • Zapata-Rivera, D., Jackson, G. T., & Katz, I. (2015). Authoring conversation-based assessment scenarios. In R. Sottilare, A. C. Graesser, X. Hu & K. Brawner (Eds.), Design recommendations for intelligent tutoring systems (Vol. 3, pp. 169–178). Orlando, FL: Army Research Laboratory.

    Google Scholar 

Download references

Acknowledgements

The research was supported by the National Science Foundation (grants DRK-12-0918409, DRK-12 1418288, and DIBBS-1443068), the Institute of Education Sciences (grants R305A 130030, R305A100875, and R305C120001), the Army Research Laboratory (contract W911NF-12-2-0030), and the Office of Naval Research (contracts ONR N00014-12-C-0643). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF, IES, or DoD.

Author information

Authors and Affiliations

  1. University of Memphis, Memphis, TN, USA

    Arthur C. Graesser, Nia Dowell & Danielle Clewley

Authors
  1. Arthur C. Graesser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nia Dowell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Danielle Clewley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arthur C. Graesser .

Editor information

Editors and Affiliations

  1. ACT , Iowa City, Iowa, USA

    Alina A. von Davier

  2. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Mengxiao Zhu

  3. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Patrick C. Kyllonen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Graesser, A.C., Dowell, N., Clewley, D. (2017). Assessing Collaborative Problem Solving Through Conversational Agents. In: von Davier, A., Zhu, M., Kyllonen, P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33261-1_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33259-8

  • Online ISBN: 978-3-319-33261-1

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation