The use and application of learning theory in learning analytics: a scoping review

Determining the theoretical influences in learning analytics research is important since those theories set forth a number of propositions that contribute to understanding and explaining phenomena or make visible causal relationships between variables, as well as allowing us to make rational inferences about future events (Biesta et al., 2011; Carr, 2006). As such, theories attempt to counter and replace practitioners’ contextually dependent, subjective beliefs with “objective knowledge generated by theory”’ that apply irrespective of context or “parochial practical beliefs” (Carr, 2006, p. 144). The claim to objective, context-less universal principles was later replaced by an acknowledgement that there is no epistemological position which makes it possible to escape or transcend the specificities of context, tradition and culture. In Biesta et al. (2011), the authors refer to Gaston Bachelard’s overview of theory as “a science of the hidden”, making “things visible or intelligible that are not immediately observable” (p. 277). Our position here is that the choice and use of a particular theory suggests a specific “ontology (nature of reality), epistemology (nature of the relationship between knower and known), and method of inquiry itself” (Sandelowski, 1993, p. 214).

In education research, Biesta et al. (2011) point to a lack of capacity building in theory, and specifically, the poor quality of theory use. The role of theory in making sense of phenomena is further impacted by the dominance of evidence in the form of data, and an emphasis on ‘what works’ without considering whether ‘what works’ is also appropriate (and ethical). While traditionally the role of theory has been seen as to aid understanding, Biesta et al. (2011) suggest instead that “theory rather has the task of generating more and different understandings” (p. 233).

Aligned to our objectives here, they moot the need for a “systematic exploration of the forms of theory and ways of theorising that play a role in educational practices, again in order to map usage and generate understanding of what theory and theorising are ‘doing’” (Biesta et al., 2011, p. 234).

Mapping the use and role of learning theories in learning analytics should not only consider the range of learning theories, but also their evolution (Illeris, 2018) and the emergence of the learning sciences. Until the 1950s, learning theory was developing independently across four approaches, namely within German Gestalt psychology; American behaviorism; Russian cultural-historical theory; and Piaget’s constructivism (Illeris, 2018). Since the 1980’s there have been various attempts to cover the field of learning with a single coherent model or framework. Examples include the experiential learning cycle by Kolb, Engeström’s activity theoretical reconceptualisation, Kegan’s constructive-developmental approach and Jarvis’ approach to lifelong learning (Illeris, 2018).

Complicating any study of the theories informing learning analytics research is the issue that there is no single, accepted definition of what constitutes learning theory, i.e., which theories are included and which do not ‘qualify’ as learning theory. Schunk (2019), for example proposes five major categories of learning theory—behaviorism, social cognitive theory, information processing theory, cognitive learning processes, and constructivism, while Harasim (2012) suggests adding connectivism to this list. The list suggested by Bates (2020), excludes information processing theory but also adds connectivism resulting in the following five learning theories, namely objectivism, behaviorism, cognitivism, constructivism and connectivism. Zhou and Brown (2015) list 12 theories that include, for example, the Theory of Moral Development, as an educational learning theory, while Hean et al. (2009) differentiate between two major approaches to understanding learning, i.e., behaviorism (focusing on the outcomes of the learning) and constructivism (focusing on the processes involved in learning). In yet another approach, Woolfolk (2014) discusses behavioral, cognitive and complex cognitive views of learning and suggests that these “focus on the individual and what is happening in his or her ‘head’” (p. 318).

A “relatively recent” development in our understanding of learning is an interdisciplinary approach, namely the learning sciences (Woolfolk, 2014, p. 318). One of the foundations of learning sciences is constructivism foregrounding the social and cultural factors in learning. What differentiates the learning sciences from learning theory, according to Woolfolk (2014), is firstly its interdisciplinary foundations, including computer science, educational psychology, neuroscience and anthropology. Interestingly, in discussing the learning sciences, Woolfolk (2014) then discusses cognitive and social constructivism. Separate from these, but linked, are social cognitive views of learning focusing on concepts such as self-efficacy and SRL. Due to the interdisciplinary basis of the learning sciences, it seems a natural fit with learning analytics (Luckin & Cukurova, 2019; Wise & Cui, 2018).

One of the founding and prominent scholars in learning analytics, George Siemens, acknowledges three major learning theories—behaviorism, cognitivism, and constructivism—but argues that these were developed in periods not impacted through technology (Siemens, 2005). In proposing a new theory, namely connectivism, Siemens opines that, while it is natural that existing theories are revised and adapted to reflect changing environments, there is a limit when “the underlying conditions have altered so significantly, (so) that further modification is no longer sensible” (p. 3). Establishing clear theoretical foundations is not simple, however. The claim of connectivism as learning theory has also attracted criticism. While it falls outside the scope of this review to discuss this debate, connectivism as learning theory has been increasingly accepted (e.g. Bates, 2020; Haythornthwaite, 2011).

Given that there is no single accepted definition of what constitutes a learning theory, the question of which learning theories inform learning analytics remains problematic.

In providing context for this scoping review, we briefly point to some examples of learning analytics’ connection to theory. Consideration of theory as integral to learning analytics research and practice has been evident from the first keynote at the first LAK conference when Haythornthwaite (2011) situated learning analytics in the context of social network research which “provides a groundwork … in terms of both graph theory and studies of social behavior” (p. 18) as well as the social network perspective as “a strong suite of social theory and analytic techniques that can illuminate interaction processes” At the second LAK conference (LAK’12), Clow (2012) proposed a Learning Analytics Cycle “as a development of previous theorisations of learning analytics. However, it is also more fundamentally, a development of much older learning theory” (p. 135; emphasis added) including Kolb’s Experiential Learning Cycle, Schön’s work on reflective practice and Laurillard’s Conversational Framework, and their implications for the Learning Analytics Cycle. He also referred to “other educational literature” that “identifies qualitatively different approaches to study” (p. 136). He observed that these educational theories “take inspiration from the cybernetic conception of control theory, and in particular, the closed-loop control system used widely in engineering of all sorts” (p. 136). In conclusion, Clow (2012) again referred to the “theoretical grounding” of the Learning Analytics Cycle.

4 years into the evolution of learning analytics, the Journal of Learning Analytics included a special section dedicated to learning analytics and learning theory. An introductory article to this by Wise and Shaffer (2015) frame their intention to “provoke a critical dialogue in the field about the ways in which learning analytics research draws on and contributes to theory” (p. 5). Contra to the belief that bigger data removes the need for theory, Wise and Shaffer (2015) claimed that “with larger amounts of data, theory plays an ever-more critical role in analysis” (p. 5). Wise and Shaffer (2015) furthermore list several roles of theory in learning analytics research, including but not limited to, providing guidance to the researcher about the variables to include or exclude in a model; guidance about “what potential confounds, subgroups, or covariates in the data to account for’; “which results to attend to”; and providing a framework for making sense of the results and making those results actionable; as well as assisting the researcher to generalize results to different populations and contexts (p. 9).

The need to explore learning theories informing learning analytics appears even more important in the light of recent research by Guzmán-Valenzuela et al (2021). They found that more than half of all learning analytics papers are published in journals in the applied sciences (engineering and technology) and are predominantly quantitative in nature. The authors conclude that their analysis points to “two LA communities—a data driven, practical and management-oriented community focused on interventions, and an academic community more focused on theories and their development—that tend not to work together” (p. 14) and that “educational and learning theories are insufficiently present in LA research” (p.15).

The remaining (n = 69) LAK papers and (n = 5) JLA articles were further reviewed to map the use of learning theories in learning analytics research (Fig. 2) and to develop a trend analysis (Fig. 3). In total, 32 different learning theories were found across the 74 texts, with 94 instances, i.e., some papers included multiple theories.

Figure 2 Clearly shows the dominance of SRL in informing learning analytics research, with Cognitive Load Theory and Constructivism as second most cited theory, and Cognitive Theory, Connectivism, Meta-cognition, and Situated Learning as third. Flow Theory, Motivation Theory, Networked Learning, Self-Determination Theory and Social Learning were each referenced three times

Of these, theories arising from the broader family of Cognitivism are Cognitive Load Theory, Cognitive Theory, Meta-cognition, and Distributed Cognition. Of interest is the number of ‘self’ theories—possibly from a range of broader learning theory categories such as behaviorism, cognitivism, and constructivism. The ‘self’ theories include—SRL, Self-Determination Theory, Self-Concept and Self-Efficacy. The only learning theories matching those in the previously discussed lists of Bates (2020), Harasim (2012) and Schunk (2019) are social cognitive theory, cognitive learning processes, constructivism, and connectivism.

To establish whether any theory had become more or less dominant over time, we examined the trend over the 10-year period for theories appearing more than three times. As Figs. 3 and 4 illustrate, there is no single pattern of growth or decline. Some theories show growth in their use over the early years of learning analytics research followed by decline, e.g., Meta-cognition, Social Learning, Constructivism and Networked Learning. Four theories; Cognitive Load Theory, Cognitive Theory, Motivation Theory and SRL seem to be on the rise, with SRL showing exponential growth in recent years.

Figure 4 depicts the proportion status of theories in use in the included paper. As noticed in the graph, theories such as constructivism and meta-cognition possessed a large percentage in particular years. However, SRL and cognitive load theory appears consistently across recent years.

Analysis of the various learning theories found in the literature has highlighted how some particularly impact on learning analytics research, and raises questions with regard to the lack of explicit mentions of behaviorism. We first discuss selected examples of how theory has impacted on research.

Recognising their embrace of theoretical pluralism, Ferguson and Buckingham Shum (2012) make clear that they do not ground their work “in a single theory of social learning, nor do we think that a techno-centric taxonomy is helpful.” Rather they are “drawing on diverse pedagogical and technological underpinnings” (p. 25). Conversely, there are examples of other researchers who do link to particular learning theories, such as Worsley and Blikstein (2013) who state that their work “fundamentally builds on Piaget’s notion that knowledge is actively and dynamically constructed by the learner based on resources that she already has, and Papert’s Constructionism” (p. 94).

The role of theory in learning analytics research is emphasized by Wise (2014) who states that learning analytics research needs a “theoretical component that explains what concept or construct the analytic represents and what its relevance and relationship to other concepts and constructs is hypothesized to be” (p. 204). Theory allows us to know how to ask the right questions and know whether the answers are relevant. “Practically this means that an analytics user must have an understanding of the pedagogical context in which the data was generated, knowledge of what particular analytics are meant to indicate, and an appreciation of how these relate to the learning goals of the situation” (p. 204). For example, SRL appeared in a number of studies based on observation of learning strategies with a view to either better understand how students were approaching their learning and/or to inform analytics approaches for detecting learning strategies (see, e.g. Matcha et al, 2020). Similarly, cognitive load theory was often applied in seeking additional insight into learning outcomes (see, e.g., Srivastava et al, 2020).

Given the apparent dominance of a handful of theories, we look briefly at the question of how a learning theory might impact learning analytics research. Of particular interest is to consider the claim that behaviorism underpins much of learning analytics (e.g., Rogers, 2015) given the paucity of explicit references to behaviorism in the reviewed papers and articles.. For Wilson et al. (2017), learning analytics as behaviorism is based on their assumption that using digital traces as proxies for learning is behaviorist. Rogers (2015) bases his claim that learning analytics understands learning as a new form of behaviourism on the importance of and emphasis on “the manipulation of antecedent conditions to control behaviour” (p. 227). It is interesting that in building his argument of learning analytics as behaviorist, Rogers (2015) refers to three articles – Atkisson (2011); Dietrichson (2013), and Lodge and Lewis (2012). Atkisson (2011) sees very little difference between learning analytics and the assumptions underpinning the 'radical behaviorism' philosophy of American psychologist B.F. Skinner, while Dietrichson (2013) refers to the potential of learning analytics to be nothing more than “clickometry” of value in courses founded on behaviorist pedagogies. Lodge and Lewis (2012) state that although there are “notable exceptions”, most of the current approach in learning analytics is “akin to a behavioural theory of learning” and that though “there is some use for a behavioural approach to learning […], these approaches to learning do not provide a complete account of what happens in higher education and through technology-mediated, networked learning environments”.

The only other reference to behaviorism in the context of learning analytics in either LAK papers or JLA articles was from Joksimović et al. (2015) although it is mentioned only as part of their findings on topics discussed by MOOC participants.

Inherent in the claims of Rogers (2015) and others that learning analytics is, essentially, behaviorist and therefore ‘reductionist’ and ‘positivist’, is the implicit reference to the critiques of the work of Skinner and other behavioral theorists (e.g. Breger & McGaugh, 1965) as underestimating context and the inherent interdependencies and inter-relationships in learning. While it falls outside the scope of this scoping review to evaluate whether all SRL in the context of learning analytics behaviorist, this scoping review can provide evidence of how SRL functions in learning analytics research.

The analysis has shown links between learning theory and learning analytics research since its emergence as a field, although we recognize also that the omission of any explicit mention of theory does not mean that the research was not informed by theory.

With increasing access to bigger data sets; a greater variety, granularity and velocity of data; the increasing use and prominence of multimodal learning analytics; and the potential and risks of Artificial Intelligence and Machine Learning, we caution against approaches that devalue learning theory. More data may mean more noise and with underdeveloped theory, there is a real risk that the noise is confused for the signal (Silver, 2012).

A recent paper by Wiley et al. (2020) petitions for a grounding of learning analytics in theoretical understandings of learning. Without theory, data can be selected haphazardly, or learning analytics biased towards “data that is simply proximal to rather than consequential for learning”, (p. 569), and it is theory that distinguishes learning analytics from data analytics. These authors argue for theory-informed or theory-grounded learning analytics on the basis that “theory informs the decision of which data are most appropriate for measuring a particular aspect of learning as well as facilitating the explanation of analytics-identified student outcomes and lighting the path for responsive action” (p. 570). Learning analytics as grounded in theory is also supported by Papamitsiou et al. (2020) who claim that “Failing to include theory and practice (e.g., pedagogical perspective, learning theories) is likely to slow progress, fail to achieve cohesiveness and universality, and might threaten validity” (pp. 567–568).

The analysis also provides evidence of the richness of the field with a wide range of learning theories being used, referred to and applied in learning analytics research. The theoretical plurality of learning analytics research, however, offers both risk and potential. Theoretical pluralism provides more than one heuristic lens with which to engage with and understand a phenomenon and offers flexibility. Midgley (2011) provides a rationale in support of theoretical pluralism by stating that (a) all knowing is inevitably bounded; (b) greater insight can be generated by exploring the boundaries of knowledge than by taking boundaries for granted; (c) different theories assume different boundaries; and (d) exploring multiple boundaries can usefully involve drawing upon multiple theories (p. 6). Multiple theories, each with its own boundary judgment, are a rich resource for undertaking research in complex environments. Acknowledging that paradigmatic pluralism “is a largely unaddressed reality” in educational technology, Kimmons and Johnstun (2019) suggest that professionals have not always been provided with sufficient guidance on how to navigate it. While our analysis pointed to a rich array of educational learning theories being used in learning analytics research, it is not clear to what extent researchers recognize the implications of their choice of learning theory, and the implications for their research designs, methods and analyses should another learning theory have informed the research. This is a pertinent point when one theoretical orientation begins to dominate a research focus and practice as it may result in an incommensurability with other approaches or designs (e.g. Kuhn, 1996), or a monist approach (Kimmons & Johnstun, 2019).

Considering the inherent multi and interdisciplinary nature of the field of learning analytics, pluralism (whether paradigmatic, theoretical, methodological) is much more than a given, but is a characteristic that should be intentionally nurtured. No discipline, theory, or methodology has all the answers nor the definitive answer to the complexities of student learning. This means that once a particular discipline, paradigm or methodology begins to dominate the field of learning analytics, it may lose some of its richness, and potentially one of its defining characteristics (Gašević et al., 2017; Selwyn, 2020). There is also the concern by (McPherson et al., 2016) that “Although learning analytics itself is an interdisciplinary field, it tends to take a ‘one-size-fits-all’ approach to the collection, measurement, and reporting of data, overlooking disciplinary knowledge practices” (p. 158). We should, however, not underestimate the inherent challenge in adopting a pluralist orientation as it means “suspending values, beliefs, or foundational assumptions (e.g., epistemological, ontological) to work within potentially contradictory paradigms at different points in time” (Kimmons & Johnstun, 2019, p. 636). Working within the context of the inherently multi and interdisciplinary field of learning analytics, require, in the words of Kimmons and Johnstun (2019), a commensurabilist orientation “that various methods can be used to achieve the same goals or to uncover the same underlying phenomena, wherein practical day-to-day activities, such as what data we collect and how we collect them, operate semi-autonomously from our underlying assumptions” (p. 636).

Navigating the multi and interdisciplinary nature of learning analytics, between monist and pluralist, or commensurablish or incommensurabilist orientations, will require a commitment to openness and epistemological honesty, recognising one’s own positions and a willingness to engage with other positions—“ situating [ourselves] deeply in disparate paradigmatic communities but struggle with the complexity that such a self-contradicting approach requires” (Kimmons and Jonstun, 2019, p. 640). [See the techniques suggested by Kimmons and Jonstun, 2019].