Hana Vrzakova
Angela Stewart
ABSTRACT
Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.
- Mary Jean Amon, Hana Vrzakova, and Sidney K D'Mello. 2019. Beyond Dyadic Coordination: Multimodal Behavioral Irregularity in Triads Predicts Facets of Collaborative Problem Solving. Cogn. Sci. 43, 10 (2019).Google Scholar
- Paulo Blikstein. 2013. Multimodal learning analytics. In ACM International Conference Proceeding Series.Google ScholarDigital Library
- Dan Calacci, Oren Lederman, David Shrier, and Alex "Sandy" Pentland. 2016. Breakout: An Open Measurement and Intervention Tool for Distributed Peer Learning Groups. CoRR abs/1607.0, (2016).Google Scholar
- Prerna Chikersal, Maria Tomprou, Young Ji Kim, Anita Williams Woolley, and Laura Dabbish. 2017. Deep Structures of Collaboration: Physiological Correlates of Collective Intelligence and Group Satisfaction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 873--888.Google ScholarDigital Library
- Michael Diehl and Wolfgang Stroebe. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle. J. Pers. Soc. Psychol. 53, 3 (1987), 497.Google ScholarCross Ref
- Pierre Dillenbourg. 1999. What do you mean by collaborative learning?Google Scholar
- Muhterem Dindar, Iman Alikhani, Jonna Malmberg, Sanna Järvelä, and Tapio Seppänen. 2019. Examining shared monitoring in collaborative learning: A case of a recurrence quantification analysis approach. Comput. Human Behav. April 2018 (2019).Google Scholar
- Lucca Eloy, Angela E.B. Stewart, Mary Jean Amon, Caroline Reinhardt, Amanda Michaels, Chen Sun, Valerie Shute, Nicholas D Duran, and Sidney D'Mello. 2019. Modeling Team-level Multimodal Dynamics During Multiparty Collaboration. In 2019 International Conference on Multimodal Interaction (ICMI '19), 244--258. Retrieved from Google ScholarDigital Library
- Abigail C Evans, Jacob O Wobbrock, and Katie Davis. 2016. Modeling Collaboration Patterns on an Interactive Tabletop in a Classroom Setting. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16), 860--871.Google ScholarDigital Library
- Michael Flor, Su-Youn Yoon, Jiangang Hao, Lei Liu, and Alina von Davier. 2016. Automated classification of collaborative problem solving interactions in simulated science tasks. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, 31--41.Google ScholarCross Ref
- Neil Fraser. 2015. Ten things we've learned from Blockly. In Proceedings of the 2015 IEEE Blocks and Beyond Workshop, 49--50.Google ScholarDigital Library
- Donald G Gardner and Jon L Pierce. 2016. Organization-based self-esteem in work teams. Gr. Process. Intergr. Relations 19, 3 (2016), 394--408.Google ScholarCross Ref
- Daniel Gigone and Reid Hastie. 1993. The common knowledge effect: Information sharing and group judgment. J. Pers. Soc. Psychol. 65, 5 (1993), 959.Google ScholarCross Ref
- Carl Gutwin, Scott Bateman, Gaurav Arora, and Ashley Coveney. 2017. Looking Away and Catching Up: Dealing with Brief Attentional Disconnection in Synchronous Groupware. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 2221--2235.Google ScholarDigital Library
- Jiangang Hao, Lei Chen, Michael Flor, Lei Liu, and Alina A von Davier. 2017. CPS-Rater: Automated Sequential Annotation for Conversations in Collaborative Problem-Solving Activities. ETS Res. Rep. Ser. 2017, 1 (2017), 1--9.Google ScholarCross Ref
- Tobias Hecking, Dorian Doberstein, and H Ulrich Hoppe. 2019. Predicting the Well-functioning of Learning Groups Under Privacy Restrictions. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 245--249. Retrieved from Google ScholarDigital Library
- Tiffany Herder, Zachari Swiecki, Simon Skov Fougt, Andreas Lindenskov Tamborg, Benjamin Brink Allsopp, David Williamson Shaffer, and Morten Misfeldt. 2018. Supporting Teachers' Intervention in Students' Virtual Collaboration Using a Network Based Model. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK '18), 21--25. Retrieved from Google ScholarDigital Library
- Cindy E. Hmelo-Silver and Howard S. Barrows. 2008. Facilitating collaborative knowledge building. Cogn. Instr. (2008).Google Scholar
- Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. Conf. Hum. Factors Comput. Syst. - Proc. (2019), 1--13.Google Scholar
- Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, and Junji Tomita. 2018. Analyzing Gaze Behavior and Dialogue Act During Turn-taking for Estimating Empathy Skill Level. In Proceedings of the 2018 on International Conference on Multimodal Interaction (ICMI '18), 31--39. Google ScholarDigital Library
- Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Houghton Mifflin Boston.Google Scholar
- Patrick Jermann and Kshitij Sharma. 2018. Gaze as a Proxy for Cognition and Communication. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), 152--154.Google ScholarCross Ref
- Steven J Karau and Kipling D Williams. 1993. Social loafing: A meta-analytic review and theoretical integration. J. Pers. Soc. Psychol. 65, 4 (1993), 681.Google ScholarCross Ref
- Norbert L Kerr and Steven E Bruun. 1983. Dispensability of member effort and group motivation losses: Free-rider effects. J. Pers. Soc. Psychol. 44, 1 (1983), 78.Google ScholarCross Ref
- Norbert L Kerr and R Scott Tindale. 2004. Group performance and decision making. Annu. Rev. Psychol. 55, (2004), 623--655.Google ScholarCross Ref
- Charlotte Larmuseau, Pieter Vanneste, Piet Desmet, and Fien Depaepe. 2019. Multichannel Data for Understanding Cognitive Affordances During Complex Problem Solving. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 61--70. Google ScholarDigital Library
- Yiwen Lin, Nia Dowell, Andrew Godfrey, Heeryung Choi, and Christopher Brooks. 2019. Modeling Gender Dynamics in Intra and Interpersonal Interactions During Online Collaborative Learning. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 431--435. Google ScholarDigital Library
- Emma M. Mercier, Steven E. Higgins, and Laura da Costa. 2014. Different leaders: Emergent organizational and intellectual leadership in children's collaborative learning groups. Int. J. Comput. Collab. Learn. (2014). Google ScholarCross Ref
- Daniele Di Mitri, Jan Schneider, Roland Klemke, Marcus Specht, and Hendrik Drachsler. 2019. Read between the lines: An annotation tool for multimodal data for learning. In ACM International Conference Proceeding Series.Google ScholarDigital Library
- Gabriel Murray and Catharine Oertel. 2018. Predicting Group Performance in Task-Based Interaction. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 14--20. Google ScholarDigital Library
- Bernard A Nijstad, Wolfgang Stroebe, and Hein F M Lodewijkx. 2003. Production blocking and idea generation: Does blocking interfere with cognitive processes? J. Exp. Soc. Psychol. 39, 6 (2003), 531--548.Google ScholarCross Ref
- Xavier Ochoa, Nadir Weibel, Marcelo Worsley, and Sharon Oviatt. 2016. Multimodal learning analytics data challenges. In ACM International Conference Proceeding Series. Google ScholarDigital Library
- Kazuhiro Otsuka, Keisuke Kasuga, and Martina Köhler. 2018. Estimating Visual Focus of Attention in Multiparty Meetings Using Deep Convolutional Neural Networks. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 191--199. Google ScholarDigital Library
- Richard V Palumbo, Marisa E Marraccini, Lisa L Weyandt, Oliver Wilder-Smith, Heather A McGee, Siwei Liu, and Matthew S Goodwin. 2017. Interpersonal autonomic physiology: A systematic review of the literature. Personal. Soc. Psychol. Rev. 21, 2 (2017), 99--141.Google ScholarCross Ref
- Joseph M Reilly and Chris Dede. 2019. Differences in Student Trajectories via Filtered Time Series Analysis in an Immersive Virtual World. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), 130--134. Google ScholarDigital Library
- Joseph M Reilly, Milan Ravenell, and Bertrand Schneider. 2018. Exploring Collaboration Using Motion Sensors and Multi-Modal Learning Analytics. Int. Educ. Data Min. Soc. (2018).Google Scholar
- Jeremy Roschelle and Stephanie D Teasley. 1995. The construction of shared knowledge in collaborative problem solving. In Computer supported collaborative learning, 69--97.Google Scholar
- Bertrand Schneider and Paulo Blikstein. 2018. Tangible User Interfaces and Contrasting Cases as a Preparation for Future Learning. J. Sci. Educ. Technol. (2018). Google ScholarCross Ref
- Jan Schneider, Dirk Börner, Peter Van Rosmalen, and Marcus Specht. 2015. Augmenting the senses: A review on sensor-based learning support. Sensors (Switzerland) (2015). Google ScholarCross Ref
- Julian Schulze and Stefan Krumm. 2017. The "virtual team player": A review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7, 1 (2017), 66--95. Google ScholarCross Ref
- Daniel Spikol, Marcelo Worsley, Luis P. Prieto, Xavier Ochoa, M. J. Rodríguez-Triana, and Mutlu Cukurova. 2017. Current and future multimodal learning analytics data challenges. In ACM International Conference Proceeding Series. Google ScholarDigital Library
- Emma L Starr, Joseph M Reilly, and Bertrand Schneider. 2018. Toward Using Multi-Modal Learning Analytics to Support and Measure Collaboration in Co-Located Dyads. . International Society of the Learning Sciences, Inc.[ISLS].Google Scholar
- Angela Stewart and Sidney K D'Mello. 2018. Connecting the Dots Towards Collaborative AIED: Linking Group Makeup to Process to Learning. In International Conference on Artificial Intelligence in Education, 545--556.Google ScholarCross Ref
- Angela E. B. Stewart, Zachary A. Keirn, and Sidney K D'Mello. 2018. Multimodal Modeling of Coordination and Coregulation Patterns in Speech Rate During Triadic Collaborative Problem Solving. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 21--30. Google ScholarDigital Library
- Angela E.B. Stewart, Hana Vrzakova, Chen Sun, Jade; Yonehiro, Cathlyn Adele Stone, D Duran, Nicholas, Valerie Shute, and Sidney K. D'Mello. 2019. I Say, You Say, We Say: Using Spoken Language to Model Socio-Cognitive Processes during Computer-Supported Collaborative Problem Solving. Proc. ACM Hum. Comput. Interact. 3, November (2019), 19.Google Scholar
- Angela Stewart and Sidney K D Mello. Connecting the Dots towards Collaborative AIED : Linking Group Makeup to Process to Learning.Google Scholar
- Code Studio. Code Studio.Google Scholar
- Chen Sun, Valerie J Shute, Angela Stewart, Jade Yonehiro, Nicholas Duran, and Sidney D'Mello. 2020. Towards a generalized competency model of collaborative problem solving. Comput. Educ. 143, (2020), 103672.Google Scholar
- Hana Vrzakova, Mary Jean Amon, Angela E B Stewart, and Sidney K D'Mello. 2019. Dynamics of Visual Attention in Multiparty Collaborative Problem Solving Using Multidimensional Recurrence Quantification Analysis. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19), 342:1--342:14.Google ScholarDigital Library
- Sebastian Wallot, Andreas Roepstorff, and Dan Mønster. 2016. Multidimensional Recurrence Quantification Analysis (MdRQA) for the analysis of multidimensional time-series: A software implementation in MATLAB and its application to group-level data in joint action. Front. Psychol. 7, (2016), 1835.Google Scholar
- Jacqueline Kory Westlund, Sidney K D'Mello, and Andrew M Olney. 2015. Motion Tracker: camera-based monitoring of bodily movements using motion silhouettes. PLoS One 10, 6 (2015), e0130293.Google ScholarCross Ref
- M Worsley. 2018. (Dis)Engagement matters: Identifying efficacious learning practices with multimodal learning analytics. (2018), 365--369.Google Scholar
- Marcelo Worsley. 2014. Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. In MLA 2014 - Proceedings of the 2014 ACM Multimodal Learning Analytics Workshop and Grand Challenge, Co-located with ICMI 2014.Google ScholarDigital Library
- Bin Xie, Joseph M Reilly, Yong Li Dich, and Bertrand Schneider. 2018. Augmenting Qualitative Analyses of Collaborative Learning Groups Through Multi-Modal Sensing. . International Society of the Learning Sciences, Inc.[ISLS].Google Scholar
- Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating how experienced UX designers effectively work with machine learning. DIS 2018 - Proc. 2018 Des. Interact. Syst. Conf. (2018), 585--596.Google ScholarDigital Library
- Jaebong Yoo and Jihie Kim. 2014. Can online discussion participation predict group project performance? Investigating the roles of linguistic features and participation patterns. Int. J. Artif. Intell. Educ. 24, 1 (2014), 8--32.Google ScholarCross Ref
- Guang Yong Zou. 2007. Toward using confidence intervals to compare correlations. Psychol. Methods 12, 4 (2007), 399.Google ScholarCross Ref
- IBM. Retrieved May 2, 2018 from https://www.ibm.com/watson/services/speech-to-text/Google Scholar
- 2015. PISA 2015 Collaborative Problem Solving Framework.Google Scholar
Index Terms
- Focused or stuck together: multimodal patterns reveal triads' performance in collaborative problem solving
Recommendations
Editable Chat Logs: A Concept for Seamless Integration of Chat Conversations and Documents in Shared Workspaces
Proceedings of the 2006 conference on Cooperative Systems Design: Seamless Integration of Artifacts and Conversations -- Enhanced Concepts of Infrastructure for CommunicationThis paper introduces the concept of editable chat logs for shared workspace systems. In shared workspaces offering a chat for synchronous communication, editable chat logs allow to keep and archive transcripts of chat conversations as documents in the ...
Multimodal Analytics for Real-Time Feedback in Co-located Collaboration
Lifelong Technology-Enhanced LearningAbstractCollaboration is an important 21st century skill; it can take place in a remote or co-located setting. Co-located collaboration (CC) is a very complex process which involves subtle human interactions that can be described with multimodal ...
Exploring Indicators for Collaboration Quality and Its Dimensions in Classroom Settings Using Multimodal Learning Analytics
Responsive and Sustainable Educational FuturesAbstractMultimodal Learning Analytics researchers have explored relationships between collaboration quality and multimodal data. However, the current state-of-art research works have scarcely investigated authentic settings and seldom used video data that ...
Comments