Multilevel Analysis in CSCL Research

Abstract

The aim of this chapter is to explain why multilevel analysis (MLA) is often necessary to correctly answer the questions CSCL researchers address. Although CSCL researchers continue to use statistical techniques such as ­analysis of variance or regression analysis, their datasets are often not suited for these ­techniques. The first reason is that CSCL research deals with individuals ­collaborating in groups, often creating hierarchically nested datasets. This means that such datasets for example contain variables measured at the level of the individual (e.g., learning performance) and variables measured at the level of the group (e.g., group composition or group performance). The number of unique observations at the lowest level, the individual, is higher than at the highest level, the group. Related to this, CSCL datasets often contain differing units of analysis. Some variables that CSCL researchers are interested in are measured at the individual level (e.g., gender, interactive behavior, familiarity with other group members), whereas other variables are ­measured at the group level (e.g., gender group composition, group performance, group consensus). Finally, because group members interact with each other in CSCL environments, this leads to nonindependence of the dependent variable(s) in the dataset. Because of their common experience during the collaboration, students’ scores on the dependent variables will likely correlate (e.g., in a group with a relatively long history of successful collaboration, group members will report similar, high levels of trust, while in groups with a negative collaboration history, group members will likely report low levels of trust). Whether nonindependence is ­present in a dataset can be established by calculating the intraclass correlation coefficient. Whenever researchers encounter datasets with hierarchically nested data, differing units of analysis, and nonindependence, MLA is needed to appropriately model this data structure since it can appropriately disentangle the effects of the different ­levels on the dependent variable(s) of interest. Researchers however also employ other strategies to deal with nonindependence and hierarchy in their ­datasets (e.g., ignoring nonindependence and hierarchy, or aggregating or disaggregating their data). We will highlight the dangers of these strategies using examples from our own research (e.g., increasing the chance of committing a Type I error). The chapter ends with a discussion of the advantages and disadvantages of using MLA for CSCL research. For example, although MLA is a powerful technique to address the data analytical problems CSCL researchers encounter, relatively large sample sizes are necessary.