Identifying significant indicators using LMS data to predict course achievement in online learning

Highlights

• Significant indicators from the LMS data on course achievement were identified.

• Measures of self-regulated learning were significant to course achievement.

• An elaborated time-based measure was useful to capture self-regulated learning.

• The potential use of early prediction by employing LMS data was revealed.

Abstract

This study sought to identify significant behavioral indicators of learning using learning management system (LMS) data regarding online course achievement. Because self-regulated learning is critical to success in online learning, measures reflecting self-regulated learning were included to examine the relationship between LMS data measures and course achievement. Data were collected from 530 college students who took an online course. The results demonstrated that students' regular study, late submissions of assignments, number of sessions (the frequency of course logins), and proof of reading the course information packets significantly predicted their course achievement. These findings verify the importance of self-regulated learning and reveal the advantages of using measures related to meaningful learning behaviors rather than simple frequency measures. Furthermore, the measures collected in the middle of the course significantly predicted course achievement, and the findings support the potential for early prediction using learning performance data. Several implications of these findings are discussed.

Introduction

Online learning has become a conventional mode of learning in higher education. Not only has the number of online educational institutions increased, but an increasing number of traditional universities are offering online courses to meet students' needs. Furthermore, massive open online courses (MOOCs) are now being offered to the public. Thus, online learning has attracted many students and provides additional learning opportunities.

Several researchers have studied the factors that are important to improving online learning and have found self-regulation to be a crucial factor in this regard (Rakes and Dunn, 2010, Sun et al., 2008, You and Kang, 2014, Yukselturk and Bulut, 2007). Online learners are responsible for initiating, planning, and conducting their studies, but many online learners have expressed how difficult it is to maintain their motivation and persistence throughout a course (Elvers et al., 2003, Levy and Ramin, 2012, Michinov et al., 2011). Previous research has shown that failure to study regularly leads to poor academic achievement, and procrastination and withdrawals have proven to be persistent problems in online learning. Therefore, the ways in which strategic support and the self-regulation of online learners influence learning should be investigated to keep students motivated, regulated, and participating in their courses.

In many studies of self-regulated learning, a self-report questionnaire is generally used to measure the level of self-regulation (Hadwin, Nesbit, Jamieson-Noel, Code, & Winne, 2007), which raises concerns regarding whether self-reported data properly represent actual self-regulated learning behaviors in authentic learning contexts. However, self-regulated learning in an online learning environment can be traced because students' learning behaviors are automatically recorded by learning management systems (LMSs). At present, LMS use has become common in most institutions, and LMSs provide new opportunities to monitor students' learning participation and progress (You, 2015). Furthermore, analyzing LMS data allows instructors to discover meaningful patterns (Gašević, Dawson, & Siemens, 2015), to identify at-risk students, to provide proactive feedback, and to adjust instructional strategies (Dietz-Uhler & Hurn, 2013). This approach is called learning analytics, and it enables data-driven decision making while improving institutional productivity. Several researchers have predicted that educational data mining will be extensively employed to optimize institutional decision making, to resolve academic problems, and to enhance students' performances in higher education within a few years (Johnson et al., 2014, Reyes, 2015).

Although the field of learning analytics is still in its infancy, prior research regarding online learning has attempted to use log or LMS data to examine online learning success. According to studies that have utilized students' log data, frequency measures, such as the number of content views, the frequency of logins, and the time spent reading pages, are the most typical measures used to explain individual differences in online learning (Morris et al., 2005, Qu and Johnson, 2005). Numerous studies (Campbell et al., 2006, Johnson, 2005, Morris et al., 2005, Wang and Newlin, 2002) have reported a significant relationship between active participation in online courses and academic performance.

However, several studies (Hadwin et al., 2007, Misanchuk and Schwier, 1992) have claimed that frequency counts of events are minimally relevant to engaged learning and that such measures are limited to suggesting instructional interventions and providing practical learning guidance. From this perspective, researchers need to use LMS data to identify more meaningful measures that are congruent with learning and instructional theories. Hadwin et al. (2007) suggested that the use of elaborated time-based indicators from students' log data, rather than the simple time spent on a specific issue, enables descriptions of students' self-regulated learning. However, notably few attempts have been made to identify appropriate measures of self-regulated learning and to examine the effects on course success.

In this context, the present study aims to identify significant LMS data indicators, including self-regulated learning indicators, to predict course achievement. Additionally, this study examines whether the data collected in the middle of the course can successfully predict final course achievement, which would contribute to the possibility of early prediction based on the learning analytics approach.