NLP-Based Heuristics for Assessing Participants in CSCL Chats

In this paper, we present an application that can be used for assessing the participants’ contributions to multiple chat conversations that debate the same topics according to different criteria (involvement, knowledge and innovation), along with the ranking of the conversations considering a list of important concepts to be debated. As several factors have been used for determining each participant’s score, we needed to determine their quality for providing an answer that correlates well with the judgment of human evaluators for the same conversations. Thus, we also propose a methodology for testing the values of different factors that may be used for assessing participants in collaborative conversations in order to identify which of them are better or worse suited for providing automatic assessment. Our analysis showed that the heuristics used to assess participants’ innovation and involvement were the best correlated with the human judgment, while at the opposite end was the heuristic used for assessing participants’ knowledge.