The present meta-analysis of studies comparing collaborative learning supported by a CSCL script with unstructured collaborative learning did not yield evidence for a strong negative effect of learning with CSCL scripts on motivation. This finding contradicts the repeatedly formulated hypothesis that learning with CSCL scripts might be too coercive, reducing learners’ autonomy and thus leading to a loss of motivation compared to externally less structured collaborative learning (e.g., Dillenbourg
2002; Wise and Schwarz
2017). Moreover, the meta-analysis shows that CSCL scripts have a small positive effect on domain learning and a medium to large positive effect on learning collaboration skills. Here, the inclusion of more recent primary studies confirms the results of a previous meta-analysis on CSCL script studies (Vogel et al.
2017). Within domain learning, CSCL scripts have a stronger effect on recall tests than on knowledge application tests. These findings are in line with theoretical assumptions and empirical findings on learning and transfer in CSCL contexts (Jeong et al.
2019). Within collaboration skills, the differences between negotiation and information sharing skills were rather small. Remarkably, no studies investigated the effect of CSCL scripts on coordination skills.
To analyze the mechanisms that are assumed to be responsible for the effectiveness of CSCL scripts, we compared CSCL scripts that prompted one, two, or three different collaborative activities as well as specific combinations of these activities based on a collaborative problem solving framework (Liu et al.
2015). Most studies prompted either one or all three collaborative activities. Generally, the results show that prompting only one specific activity tends to result in higher effect sizes compared to prompting a combination of two or three collaborative activities. However, this was more strictly the case for collaboration skills than for domain learning. For domain learning, combining two collaborative activities yielded even smaller effects than combining all three collaborative activities. When conducting more differentiated analyses comparing different combinations of particular collaborative activities, we found that particularly the combination of information sharing and coordination led to a small effect size. Among CSCL scripts that prompted only one collaborative activity, coordination scripting yielded the largest effect on domain learning. Among CSCL scripts prompting two collaborative activities, CSCL scripts combining negotiation and information sharing scripting were most effective. For collaboration skills, detailed analyses on the effect of scripting different combinations of collaborative activities were not possible due to the small number of studies. The results show, however, that scripting a greater number of different collaborative activities reduces the effectiveness of CSCL scripts for learning collaboration skills. However, the moderator analyses did not lead to a substantial reduction in heterogeneity between the CSCL script studies. Therefore, it is still necessary to identify other mechanisms that are relevant for the effectiveness of learning with CSCL scripts beyond the mechanisms proposed and analyzed in the present meta-analysis.
Overall effect of learning with CSCL scripts on motivation (RQ 1)
The first research question addressed the effect of collaborative learning with CSCL scripts compared to unstructured collaborative learning on motivation. In the light of the frequent critique that CSCL scripts might be too coercive and thus reduce learners’ autonomy and motivation, it could be hypothesized that learning with CSCL scripts should have a negative effect on motivation (Dillenbourg
2002; Wise and Schwarz
2017). In contrast, the results showed a small positive but non-significant effect of learning with CSCL scripts on motivation with all studies either reporting non-significant or significant positive effects. Taking a closer look at the primary studies reporting effects of CSCL scripts on motivation enabled us to identify some patterns within the data. Interestingly, most studies considering effects on motivation used CSCL scripts that distributed roles among participants. There were, however, striking differences between the studies yielding positive and null effects. Most studies reporting non-significant effects used CSCL scripts that distributed roles among participants such as tutor and tutee or note-taker roles (G.-Y. Lin
2020; Peterson and Roseth
2016; Weinberger et al.
2005). In these cases, the participants were all undergraduate students. However, some CSCL scripts could even positively affect learners’ motivation. For instance, a CSCL script that did not affect motivation in synchronous learning settings, had positive effects on motivation in in asynchronous learning settings (Peterson and Roseth
2016). Other motivating CSCL scripts applied a “natural” role distribution resulting from true knowledge interdependency (e.g. collaboration between psychologists and physicians, Rummel et al.
2009), or distributed roles with different responsibilities among school pupils (Taylor and Baek
2019). In the latter case, the pupils were more motivated when the roles rotated between learning sessions than when fixed roles were used. It seems plausible that undergraduate students have higher prior knowledge regarding collaboration than school students and therefore perceive such artificial roles as more constraining and disruptive of their natural collaboration than fourth and fifth grade students (Fischer et al.
2013). Nevertheless, the CSCL scripts included in this meta-analysis did not have negative effects on motivation, but rather were comparable to unstructured collaboration. Hence, these CSCL scripts were not detrimental, as suggested by critics, but seemed not to exploit their full potential for increasing motivation.
The duration of the intervention could be an alternative factor explaining differences in the effects of CSCL scripts on motivation. It seems plausible that during an intervention with CSCL script collaboration skills develop. Hence, learners could perceive a CSCL script as more coercive after having received support for some time. Therefore, due to the small number of effects we qualitatively compared studies with respect to their intervention duration. The intervention of two studies lasted for several weeks (Peterson and Roseth
2016; Taylor and Baek
2019). Two other studies used interventions that lasted for 60 to 80 min (Rummel et al.
2009; Weinberger et al.
2005). One study did not provide any information on intervention duration (G.-Y. Lin
2020). Although one would expect learners to become less motivated the longer they learn with a CSCL script, the included studies do not support such pattern. Studies using CSCL scripts in long term interventions yielded significant positive effects (Taylor and Baek
2019) on motivation as well as null effects (Peterson and Roseth
2016). Studies that used CSCL scripts in short term interventions resulted in null effects (Rummel et al.
2009; Weinberger et al.
2005). Therefore, the existing data does not allow for conclusions on how motivation is affected by CSCL scripts over time. Other factors in the design of CSCL scripts might affect learners’ motivation such as fading or adapting the CSCL scripts to individual needs. However, to our knowledge no study on CSCL scripts addresses these aspects together with the intervention duration. Thus, it is necessary to address systematically the question of how different CSCL scripts affect motivation, how this effect changes over time, and how technology can help to exploit the full potential of CSCL scripts.
Another explanation for the non-significant effect on motivation might be that learning with CSCL scripts has ambivalent effects on different factors influencing learners’ motivation. CSCL scripts might help learners easily achieve strong feelings of competence and relatedness because they provide a structure for collaborative learning processes and learners’ involvement in a social context. Such feelings of relatedness and competence are connected to higher levels of motivation (Rienties et al.
2012; Ryan and Deci
2000). On the other hand, the coercive nature of CSCL scripts, which strictly define the activities learners are expected to engage in, could lead to a lower degree of autonomy, which in turn reduces motivation (Wise and Schwarz
2017). This combination of positive and negative effects of learning with CSCL scripts on motivation might balance out, leading to a non-significant effect size close to zero. Unfortunately, the small number of studies empirically examining the effect of CSCL scripts on motivation precludes more nuanced quantitative analyses. Thus, more primary studies addressing hypotheses that take a more differentiated view on motivational factors, for example with respect to the basic psychological needs, are needed. Critique of CSCL scripts rarely address the different aspects of motivation. If we regard motivation as a holistic construct, there is no meta-analytical evidence for an overscripting effect. Thus, this criticism remains a postulate without corresponding empirical evidence.
Overall effects of learning with CSCL scripts on domain learning and learning collaboration skills (RQ 2)
Based on the theoretical assumptions of the script theory of guidance, we further hypothesized that learning with CSCL scripts should have a positive effect on domain learning and learning collaboration skills (Fischer et al.
2013). The results support this hypothesis. Thus, we can conclude that CSCL scripts do indeed support the learning of beneficial collaboration processes that eventually lead to better elaboration of the learning content and ultimately to better collaboration skills and domain learning outcomes (King
2007).
As already detected in the previous meta-analysis by Vogel et al. (
2017), the effect of learning with CSCL scripts on collaboration skills was substantially higher than the effect on domain learning. In the script theory of guidance, it is assumed that the guidance provided by an external script helps learners to participate in a specific CSCL practice, building and reconfiguring internal scripts they can then recall in other situations (Fischer et al.
2013). In doing so, the CSCL script helps learners engage in collaborative activities that are beneficial for domain learning. Therefore, substantially higher effects on collaboration skills might be due to a more direct link between CSCL scripts and the development of collaboration skills, while domain learning is more indirectly supported by accomplishing what the script suggests learners to do. The wide range in magnitude of the effects of CSCL scripts on collaboration skills, however, raises questions about how effective such scaffolding can be. Although the average effect size of CSCL scripts on collaboration skills is comparable to the effect sizes other scaffolds such as example-based learning have on learning (J. Chen et al.
2018; Jeong et al.
2019; Wittwer and Renkl
2010), some studies report effect sizes far beyond the usual effects of scaffolding (Noroozi et al.
2013c). These studies reveal a need for closer examination and challenge future CSCL script designs to increase the effectiveness for domain learning to a similar size. One reason for the extraordinary effectiveness of these CSCL scripts might be that they do not only provide support during collaborative phases but also ask learners to individually prepare for the joint learning phases. Prior studies have found the combination of individual and collaborative phases during collaborative learning to be more beneficial than individual learning or collaborative learning alone (Olsen et al.
2017). Individual phases allow learners to prepare for collaboration and give them time to think about and prepare their contributions before being engaging in communication with the learning partner, when answers are expected to be formulated immediately and little time for individual thinking is available. Another reason might be that these studies measure the internal collaboration script that was addressed by the CSCL script particularly well, whereas other studies with lower effects use broader measures.
Explaining the effectiveness of CSCL scripts prompting different combinations of collaborative activities (RQ 3 and 4)
For domain learning, we assumed that combining prompts for different types of collaborative activities would increase domain learning through synergistic scaffolding. Prior research has shown that learners who engage in all types of collaborative activities have the highest domain learning outcomes (Andrews-Todd and Forsyth
2018). Hence, successful synergistic scaffolding should lead to effects on learning outcomes when the CSCL script combines prompts for different collaborative activities above and beyond the effects achieved through separate prompts for each collaborative activity (Tabak
2004). To investigate this issue, we compared CSCL scripts that prompted one, two, or three different collaborative activities. In contrast to our expectations, CSCL scripts were descriptively most effective when prompting only one collaborative activity and least effective when prompting a combination of two collaborative activities. Notably, on all levels CSCL scripts were more effective in fostering domain learning as measured by application tests compared to recall tests. This is particularly surprising given that lower overall effect sizes were found for application measures compared to recall measures. This may indicate that exclusively prompting one collaborative activity is most effective for enhancing application-oriented knowledge and skills. One reason for the larger effect might be that performing these collaborative activities evokes higher-order cognitive processes that allow learners to connect the new information with prior knowledge and apply new information to a problem (Chi
2009). Why this is only valid for CSCL scripts that prompted only one or three collaborative activities remains a subject for further research.
The finding that CSCL scripts prompting one collaborative activity outperform CSCL scripts prompting all three types of collaborative activities indicates that CSCL scripts did not successfully induce synergistic scaffolding. One plausible reason for this lack of synergistic scaffolding might be connected to the fact that CSCL scripts that prompt different types of collaborative activities are increasingly demanding. It is possible that these pose an additional load on the learner; in particular, scripts for several collaborative activities pose an even higher cognitive load on learners (e.g., F. Kirschner et al.
2009). Possible solutions to take some load off learners when working with highly complex CSCL scripts might be to offer the scaffolds for the different types of collaborative activities independently of each other or in a specific sequence (Schwaighofer et al.
2017). However, this explanation would be in conflict with the finding that scripting three collaborative activities is more effective than scripting two collaborative activities. Primary research on how to combine prompts for different types of collaborative activities in one CSCL script can lead to synergistic scaffolding (Tabak
2004) is still at a nascent stage, and more research is needed to find the most beneficial design for such scaffolding.
To explore the effectiveness of specific combinations of collaborative activities, we compared the different combinations of collaborative activities prompted by the CSCL script in more detail. CSCL scripts addressing solely coordination or a combination of negotiation and information sharing were most effective, followed by scripting information sharing only and the three-way combination. Notably, the effect of CSCL scripts that only prompted negotiation was very variable and non-significant. Upon closer examination, the variability of this effect is reflected in the variability of the CSCL scripts used in this sample. The CSCL scripts range from very elaborate, highly structured discussion scripts (Noroozi et al.
2013c) to argumentation scripts sequencing the order of arguments and counter-arguments (Stegmann et al.
2007) and CSCL scripts that solely prompt to discuss a specific topic (Rau et al.
2017). Comparing these studies, it seems that CSCL scripts offering a higher degree of structure have larger effects on domain learning. Comparing the different combinations of collaborative activities also provided more detailed insights into the question of how synergistic scaffolding might be achieved (Tabak
2004). Specifically, combining the negotiation and information sharing prompts yielded a higher effect size than offering scaffolding for one of the two types of collaborative activities alone. Analogously to the interpretation of the positive effect of combining of individual and collaborative activities (Olsen et al.
2017), this could be seen as a successful combination of two activities that can lead to synergistic effects. CSCL scripts that only prompted negotiation yielded a non-significant effect, which was dramatically improved by combining negotiation with information sharing. Conversely, scripting information sharing alone already had a significant positive effect, yet the effect was even higher when combined with negotiation. Examining the CSCL scripts used in these studies in detail, it stands out that scripting negotiation only means that students are specifically prompted to engage in discussion; however, an information exchange phase is missing (e.g., Puhl et al.
2015; Wu et al.
2019). This phase might help students better engage in beneficial negotiation activities by encouraging prior listening and thinking about the information their learning partners share with them. This effect seems to be in line with the importance of individual phases in which students can first think about and establish their viewpoint before engaging in collaboration (Olsen et al.
2017). Conversely, the effectiveness of information sharing prompts when learning with CSCL scripts does not seem to be comparably dependent on negotiation prompts, since CSCL scripts scaffolding information sharing only already achieved a substantial effect. It seems that the information exchange is one of the most beneficial activities for learning with CSCL scripts. However, its effectiveness can be diminished by additionally scaffolding coordination but further boosted by including negotiation prompts.
When combining all three types of collaborative activities, any benefits resulting from two-way combinations of scaffolding in CSCL scripts seem to vanish. This could be a consequence of over-loading students with too many different activities to focus on (F. Kirschner et al.
2009). Nevertheless, since each type of collaborative activity investigated seemed to work successfully at least when offered separately or in combination with one other activity, it remains an important avenue for further research to determine how the scaffolding of different collaborative activities should best be combined to increase domain learning. Particularly, it would be interesting to gain more insight into how scaffolding coordination can remain beneficial when combined with other activities. Here, the results of the meta-analysis showed that although CSCL scripts scaffolding coordination alone were quite effective, combining coordination activities with other prompts led to far smaller or even non-significant effects. Overall, the results indicate that it is more important to focus on which specific combination of collaborative activities is prompted by the CSCL script than the number of different collaborative activities prompted by the script. Unfortunately, the low number of studies preclude more detailed analyses of whether the three collaborative activities differ in their potential to advance application-oriented and recall-oriented domain learning. In particular, there is still a lack of primary research on the combination of two different collaborative activities.
For the learning of collaboration skills, the three-way combination also led to the smallest effect size. This means that the CSCL scripts included in this meta-analysis failed to induce differentiated scaffolding of collaboration skills in a beneficial way (Tabak
2004). Ideally, prompting all of the different types of collaborative activities would lead to strong effects of the specific prompts on the corresponding collaboration skills. Thus, the differentiated scaffolding of different collaborative activities within a single CSCL script should not reduce their effectiveness. Consequently, the results of this meta-analysis could be interpreted either as suggesting that differentiated scaffolding is not possible in the way proposed by Tabak (
2004), or that it is necessary to further study how to induce differentiated scaffolding when developing CSCL scripts combining scaffolds for different types of collaborative activities in order to support collaboration skills. In addition, the intervention period of most primary studies included in this meta-analysis was rather short. Therefore, it is possible that scripting several different collaborative activities over a short period of time overwhelms the learners by inducing a high cognitive load (e.g., F. Kirschner et al.
2009). It seems plausible that repeated practice of a single collaborative activity falls short if combined with other activity prompts. This might also be a question of measurement. Although CSCL scripts often address more than one collaborative activity, only a few studies measured a mixture of collaboration skills. Therefore, it seems plausible that those scripting only one or two collaborative activities were better able to measure the learning of collaboration skills as their measures better aligned with the specific skill scaffolded by prompting the respective collaborative activity. Also, the prompting of specific types of collaborative activities might not have addressed the optimal scripting level (Fischer et al.
2013). Designing CSCL scripts with a combination of prompts for different types of collaborative activities might often result in prompting only higher scripting levels to avoid scripting that is too extensive and overwhelming scripting. For example, learners might be prompted to discuss the most plausible solution (Rummel et al.
2009) or exchange information (Ertl et al.
2006). Here, particularly inexperienced learners could require more detailed scripting (i.e., on a lower scripting level) for these activities.
Overall, the results of the meta-analysis show that CSCL scripts are beneficial for domain learning and for enhancing collaboration skills. However, the proposed mechanisms, that generally prompting a combination of negotiation, information sharing, or coordination might explain the effectiveness of learning with CSCL scripts, were not supported by the results of this meta-analysis. In particular, combining prompts for two or more types of collaborative activities led to lower effect sizes than only prompting one of these activities. Thus, neither differentiated nor synergistic scaffolding could be successfully achieved by generally combining prompts for different types of collaborative activities in one CSCL script. Thus, for the effectiveness of CSCL scripts, the studies included in this meta-analysis indicate that in some cases less is more when it comes to scaffolding different collaborative activities. This might be due to the additional cognitive load posed by CSCL scripts incorporating scripting for different types of collaborative activities. Thus, how to achieve differentiated and synergistic scaffolding (Tabak
2004) when combining different scaffolds in one CSCL script remains an open question. Sequencing and fading scaffolds throughout the application of a CSCL script might be a promising approach, and thus should be examined in future experimental studies on learning with CSCL scripts.
Implications for the critique that CSCL scripts decrease motivation by “overscripting”
Although motivation is vividly discussed as a factor responsible for the small or negative effects of learning with CSCL scripts (Dillenbourg
2002; Wise and Schwarz
2017), the number of CSCL script studies measuring motivation is rather small. Nevertheless, the non-significant overall effect of CSCL scripts on motivation reported in this meta-analysis does not support the repeatedly asserted critique that the strict structuring of collaborative learning through CSCL scripts leads to a reduction of autonomy, which in turn negatively influences motivation and ultimately impedes learning (Dillenbourg
2002). If such an effect exists at all, it might be reduced by an opposing effect of CSCL scripts enhancing learners’ feelings of competence and social relatedness, which should lead to higher motivation (Järvelä et al.
2010). The empirical studies included in this meta-analysis suggest that CSCL scripts are not detrimental for motivation, but rather can have positive effects. This strongly indicates that the overscripting effect, which is originally based on a conceptual article (Dillenbourg
2002), has been overblown by the research community without being based on empirical evidence. Nevertheless, theories about supporting collaborative learning scenarios could be further developed by reflecting in more detail on the effect of structuring CSCL on motivation and integrating different factors that might have positive or negative effects on motivation, such as those proposed by self-determination theory (Deci and Ryan
1985). Future research on learning with CSCL scripts should measure aspects of motivation by default in order to achieve a more robust sample of effect sizes for motivation. Moreover, given the hypothesized detrimental effect of learning with CSCL scripts on motivation, future CSCL script designs should try to increase the positive effect of CSCL scripts on feelings of competence or social relatedness. Increasing the freedom afforded by CSCL scripts, for example by fading them out (e.g., Wecker and Fischer
2011) or adapting them to learners’ needs (Rau et al.
2017), could presumably decrease a possible negative effect of CSCL scripts on learners’ autonomy.
Implications for the script theory of guidance
The positive effects of CSCL scripts on both domain learning and learning collaboration skills are in line with the theoretical assumptions that learning with CSCL scripts induces beneficial collaborative processes, ultimately leading to better learning compared to unstructured collaborative learning (King
2007). These findings justify the principles formulated in the script theory of guidance that providing external scripts enables learners to engage in collaborative practice in a way that leads to learning of knowledge and skills (Fischer et al.
2013). Although CSCL scripts are mainly considered useful for learning to collaborate (Wise and Schwarz
2017), there are far more studies investigating their effect on domain learning. The small positive effect on domain learning is clearly stable, whereas the large positive effect on collaboration skills is deeply heterogeneous. Additionally, the relatively small number of studies analyzing effects on collaboration skills precludes more comprehensive moderator analyses that could help identify factors explaining the heterogeneity in effect sizes. Thus, in future research, fewer studies on the general effects of CSCL scripts on domain learning are needed, but more studies explicitly analyzing differential effects of CSCL scripts using different designs, CSCL scripts inducing different types of activities, or CSCL scripts implemented in various contexts. In contrast, more studies examining the general effect of CSCL scripts on collaboration skills are still needed. There is a particular lack of studies concerning the effect of CSCL scripts on coordination, despite the fact that studies facilitating coordination were most effective for domain learning. Moreover, given the various measures used to assess collaboration skills, the field would be strengthened by developing instruments for assessing particular collaboration skills such as information sharing or negotiating and widely applying them in studies on computer-supported collaborative learning.
Limitations
Of course, this meta-analysis is not without limitations that must be considered when interpreting the results and drawing conclusions and implications. Although there is a large body of research and strong theoretical foundation concerning the effectiveness of CSCL scripts on learning, the design and context of CSCL scripts in empirical research varies greatly. The targeted collaboration skills range from the construction of individual arguments in short one-hour trainings to engaging in argumentative discourse, exchanging peer feedback and preparing mutual individual introductions, and even adhering to assigned roles for several weeks of collaborative learning activities. This leads to difficulties in finding comparable CSCL script studies and summarizing them in a reasonable way. In addition, there is wide variety in what constitutes a CSCL script. However, by applying a consistent definition of CSCL scripts and searching for studies on CSCL scripts without solely using the term script, but rather focusing on the mechanisms of CSCL scripts, we tried to identify an appropriate sample of primary studies representing what has been discovered empirically about learning with CSCL scripts. It was, however, necessary to constrain the literature search to specific search terms. Although we also used other terms than the term “script” itself, it is possible that we systematically missed studies from research areas using a different nomenclature. One such area might be the research on dynamic support such as conversational agents. We included two studies using conversational agents for the present meta-analysis (Adamson et al.
2014; Ulicsak
2004). Our decision to not use the rather broad search term “support” has led to missing out studies from this field of research (e.g., Wang et al.
2011). However, the respective analyses did not indicate any substantial publication bias. This leads us to conclude that we found a comprehensive sample of primary studies for the phenomenon under investigation. Nevertheless, it is possible that specific research areas are underrepresented.
Regarding the coding of the primary studies, a major problem is that most studies do not provide direct indicators for the factors that are theoretically and empirically assumed to positively affect the effectiveness of learning with CSCL scripts. The assumption is that the more learners engage in specific activities in the learning process (e.g., negotiation), the more they should benefit from collaborative learning (Chi and Wylie
2014; Liu et al.
2015). To analyze the effectiveness of learning with CSCL scripts based on this assumption, the most direct indicator primary studies could provide would be measures of the activities learners actually engage in throughout the collaborative learning process. However, most studies do not report such measures; thus, the most proximal information provided by the primary studies is the description of activities prompted by the CSCL script. What actually happened during the collaboration process in the different studies can only be estimated using this information, resulting in several sources of uncertainty. First, the accuracy of this information varies across studies. While some studies report the activities required in the CSCL script in detail (e.g., Rummel et al.
2009; Ulicsak
2004), others are less detailed, which might have led to inaccurate estimations of the learning activities that were actually used in the studies’ learning processes (e.g., Hsu et al.
2015; Tsovaltzi et al.
2015). A second source of uncertainty is that, even when the activities required by the CSCL scripts are described in detail, we do not know to what extent the learners adhered to what they were asked to do during the learning process. Sometimes learners only complete the required activities on a very superficial level. Nevertheless, the description of activities required by the CSCL scripts is the most proximal estimation for the collaboration process available. Moreover, we assume that the presumed inaccuracy of this measure is well distributed across studies and thus should not substantially bias the results of the comparisons between different types of CSCL scripts.
Another limitation of this meta-analysis is that, although motivation features prominently in criticisms of learning with CSCL scripts, there were only a few studies measuring motivation (e.g., Peterson and Roseth
2016). Moreover, motivation is conceptualized in CSCL research in highly diverse ways, such as approaches building on expectancy value conceptualizations, distinguishing between extrinsic and intrinsic motivation or conceptualizations such as self-determination theory integrating the needs for competence, autonomy, and social relatedness (Deci and Ryan
1985). The broad variety of conceptualizations and the rather small number of independent effect sizes weaken our ability to interpret them coherently and do not allow for examinations of further moderating effects.
More generally, the number of CSCL script studies included in this meta-analysis is relatively low, which reduces the possibility of comparing between different studies on a more fine-grained level. The number of studies at each level of the examined factors was sometimes too low or too unevenly distributed to conduct a comparison and interpret the results in a reasonable manner. This was particularly true for studies analyzing the effects of CSCL scripts on collaboration skills and on motivation. Additionally, even within a moderator level the reported effect sizes are often very heterogeneous. This indicates that studies within moderator levels vary systematically due to unknown covariates. Therefore, such results should be interpreted carefully. Unfortunately, we cannot solve this issue until more primary studies are conducted that would allow more fine-grained analyses.
The final limitation that must be considered when interpreting the results of the meta-analysis at hand is that comparisons of different levels of the moderators are only comparisons between studies. Almost none of the studies compared the moderator levels as a within-study effect (e.g., Peterson and Roseth
2016). Thus, the differences between different moderator levels cannot be interpreted causally. They only suggest a direction for the empirical relationship, which might have been confounded by the specific study designs.