Introduction
The problem
The significance
Purpose and the research questions
RQ1: To what extent can our automatic classifier accurately identify the phases of cognitive presence in the online discussion messages from the target MOOC?RQ2: Which classification features can be the most important to identify each phase of cognitive presence according to the automatic classifier training results?RQ3: Can the automatic classifier trained on the target MOOC potentially identify cognitive presence in MOOCs of the other disciplines?
Related studies
The Community of Inquiry (CoI) framework and cognitive presence
Automated classifiers of cognitive presence in online discussion transcripts
Studies by | Algorithm | Main features | Best outcome metrics | |
---|---|---|---|---|
Accuracy (%) | Cohen’s κ | |||
McKlin et al. (2001) | Simple neural networks | Dictionary-based words and phrases | 68 | 0.31 |
Corich et al. (2006) | Bayesian network | Dictionary-based words and phrases | 71 | – |
Kovanović et al. (2014) | Support vector machine | Bag-of-words, n-grams, and structural features | 58.4 | 0.41 |
Waters et al. (2015) | Conditional random fields | Bag-of-words, n-grams, and more structural features | 64.2 | 0.48 |
Kovanović et al. (2016) | Random forest | LIWC, Coh-Metrix, LSA, structural features | 70.3 | 0.63 |
Neto et al. (2018) | Random forest | LIWC, Coh-Metrix, word embeddings, structural features | 83 | 0.72 |
Farrow et al. (2019) | Random forest | Same as Kovanović et al. (2016) | 61.7 | 0.46 |
Barbosa et al. (2020) | Random forest | Same as Kovanović et al. (2016) | 67 | 0.32 |
Neto et al. (2021) | Random forest | Same as Kovanović et al. (2016) | 76 67a 57b | 0.55 0.2 0.38 |
Differences between the current and the prior work
Methods
Description of the data sets
Id | Cognitive phase | Philosophy set | Medicine set | Education set | Humanities set | ||||
---|---|---|---|---|---|---|---|---|---|
Count | % | Count | % | Count | % | Count | % | ||
0 | Other | 85 | 5.75 | 3 | 3.03 | 3 | 3.09 | 12 | 12.2 |
1 | Triggering event | 279 | 18.86 | 36 | 36.4 | 34 | 35.1 | 27 | 27.6 |
2 | Exploration | 835 | 56.46 | 43 | 43.4 | 44 | 45.4 | 41 | 41.8 |
3 | Integration | 244 | 16.50 | 16 | 16.2 | 10 | 10.3 | 16 | 16.3 |
4 | Resolution | 36 | 2.43 | 1 | 1.01 | 6 | 6.19 | 2 | 2.04 |
Fine-tuning process | ntree | mtry | Accuracy (SD) | Cohen’s κ (SD) | |
---|---|---|---|---|---|
With the SMOTE exact method | Min | 500 | 196 | 0.654 (0.034) | 0.414 (0.057) |
Max | 1100 | 54 | 0.689 (0.043) | 0.465 (0.068) | |
Difference | 0.035 | 0.051 | |||
Without the SMOTE exact method | Min | 500 | 2 | 0.659 (0.018) | 0.334 (0.040) |
Max | 1100 | 94 | 0.694 (0.035) | 0.437 (0.069) | |
Difference | 0.035 | 0.103 |
Feature extraction
Discussion contextual features
Linguistic features
Semantic similarity
Name-entity words
Data processing and model training
Optimal parameters
The unbalanced class problem
Classifier performance metrics
Validation of the automatic classifier on the MOOC discussion data sets of other disciplines
Results
Model evaluation when training and testing on the Philosophy MOOC data—RQ1
Classifiers | Accuracy (SD) | Cohen’s κ (SD) | Macro F1 (SD) | Weighed F1 (SD) | ntree | mtry |
---|---|---|---|---|---|---|
Classifier with the SMOTE exact method | 0.730 (0.046) | 0.542 (0.071) | 0.509 (0.069) | 0.742 (0.056) | 1100 | 54 |
Classifier without the SMOTE exact method | 0.736 (0.032) | 0.516 (0.063) | 0.472 (0.054) | 0.771 (0.061) | 1100 | 94 |
Predicted labels | Manual labels | ||||
---|---|---|---|---|---|
Other | Triggering | Exploration | Integration | Resolution | |
Other | 1 | 2 | 1 | 0 | 0 |
Triggering | 7 | 19 | 3 | 0 | 0 |
Exploration | 1 | 7 | 74 | 8 | 1 |
Integration | 0 | 0 | 6 | 13 | 1 |
Resolution | 0 | 0 | 0 | 3 | 1 |
Error rate | 0.889 | 0.321 | 0.119 | 0.458 | 0.667 |
Predicted labels | Manual labels | ||||
---|---|---|---|---|---|
Other | Triggering | Exploration | Integration | Resolution | |
Other | 2 | 0 | 1 | 0 | 0 |
Triggering | 4 | 21 | 3 | 0 | 0 |
Exploration | 3 | 7 | 78 | 16 | 2 |
Integration | 0 | 0 | 2 | 8 | 1 |
Resolution | 0 | 0 | 0 | 0 | 0 |
Error rate | 0.778 | 0.250 | 0.071 | 0.667 | 1.000 |
Feature importance analysis—RQ2
Summary of important features for cognitive presence phases in the philosophy MOOC discussions
-
The other: fewer words, less similar verbs, more readable for second-language readers.
-
Triggering event: more ‘I’, children’s words, non-repeated words, abstract words, and verbs, less similar to the previous message.
-
Exploration: more nouns, more concrete and specific words, more cognitive processing relevant words, more often in the middle of a conversation.
-
Integration: more lexically diverse words, more ‘it’.
-
Resolution: more words, more often at a deeper position of a conversation.
Cross-domain validation of our classifier—RQ3
Disciplines | Messages | % Agreement | Cohen’s κ |
---|---|---|---|
Medicine | 99 | 47.5 | 0.195 |
Education | 97 | 57.7 | 0.371 |
Humanities | 98 | 41.8 | 0.158 |
All | 294 | 49.0 | 0.241 |