Introduction
The online learning environment has recently become the preferred method of sharing learning content, which enables participant’s learning activities to be analyzed, in order to maximize learning activity and optimize the overall system. For example, typical large scale systems, which are called massive open online courses (MOOCs), are used for delivering learning content (Seaton, Bergner, Chuang, Mitros, & Pritchard,
2014). In order to evaluate both participant’s learning activity and learning systems, participant’s access logs and learning performance were analyzed (Seaton, Nesterko, Mullaney, Reich, & Ho,
2014). When the current learning environment, which uses information communication technology, was developed, the effectiveness of the system was evaluated using participant’s learning behavioral data (Nakayama, Kanazawa, & Yamamoto,
2007; Nakayama, Kanazawa, & Yamamoto,
2009). Using a conventional learning assessment approach, such as a summative evaluation, learning performance during the online course was discussed. Also, the cost benefit of an online learning environment has often been discussed (Bates,
2000). Another approach, known as an authentic assessment (Gulikers, Bastiaens, & Kirschner,
2004), focuses on the student’s learning progress. Also, a conventional formative assessment can improve the learning progress of students and the instructive activities of teachers (Bloom, Hastings, & Madaus,
1971). Since student’s learning activity may elicit learning achievement, the formative assessment focuses on activities which improve student’s learning progress in comparison with a summative assessment that is based on their final results. In addition to this, the formative assessment can be conducted using regular learning activities such as note taking and scores of various non-test work results. Though participants are evaluated using final test scores during most university courses, a formative assessment may reflect their overall learning performance. Therefore, a formative assessment can provide various kinds of information about participant’s learning activity and performance. This approach has been applied to educational improvement of the learning progress (Bell et al.,
2015; Bennett,
2015).
Using conventional approaches, the varying effectiveness of student’s aptitudes (Cronbach & Snow
1977) and learning behavior, such as note taking activity (Kiewra,
1985,
1989; Kiewra, Benton, Kim, Risch, & Christensen,
1995; Kobayashi,
2005; Piolat, Olive, & Kellogg,
2005) was studied. The relationships between these factors and learning performance have also been widely analyzed and discussed (Cronbach & Snow,
1977; Nye, Crooks, Powley, & Tripp,
1984; Weener,
1974). These survey and analytical techniques were introduced to the study of the online learning environment, and the learning effectiveness of note taking, and some of the causal relationships between learning activities and note taking behavior were analyzed (Nakayama, Mutsuura, & Yamamoto,
2014a,
2015a,
b,
2017). Previous studies concerning note taking activity, which examined these overall activities, can be used as a summative assessment as student’s achievements. This suggests that metrics of note taking activity can be used as one of the indices of the formative assessment when note contents for every session are assessed.
While some of these measurements affect student’s learning, detailed factors and the timing of their effectiveness may be key issues. The analysis of participant’s formative leaning process and the revision of metrics in response to student’s behavior provide the means to resolve the problem.
This study extracts some of the features of student’s attitudes and the contents of notes taken during learning activities, and tracks their contributions to learning performance as the course progress. During online learning courses, behavioral events are organized as learning activities which are supported by information communication technology. Therefore, the research questions this paper is concerned with are: examining the possibility of predicting scores of final exams as a measure of learning performance, using information collected during the learning progress, identifying the learning activity metrics necessary for the prediction, and evaluating the effectiveness of note taking instruction.
The following topics are also addressed in this paper:
-
A procedure for the estimation of the final exam scores is proposed, using selected features of the contents of notes taken and participant’s characteristics.
-
The effectiveness of note taking instructions is evaluated, in order to examine the performance of the estimations.
-
Possible time periods for the measurement of the progress of the course are discussed, regarding the performance of the predictions.
Discussion
As mentioned in the introduction, student’s characteristics, including note taking scores (NTS), have been confirmed to have an affect on the scores of final exams. Also, the effectiveness of note taking instruction was introduced, though the detailed relationships between these variables were not specified. This paper tries to examine the relationships mathematically in order to improve the learning process.
Fundamentally, the relationships between the scores of final exams and features of note taking activities were analyzed. Though the metrics for note taking activity for courses with and without instruction are comparable in Table
1, the correlation relationships between the scores of final exams and features of note taking activities have changed due to the note taking instruction that was given. All correlation coefficients between variables were significant during the course with instruction, while the coefficients were not significant for the course without instruction. In comparing correlation relationships across metrics of note taking activity between Tables
2 and
3, the contents of notes participants took may have changed. Generally, students replaced the terms presented with their own words in their notes. When note taking instruction was given, students recorded their own words in addition to recording the words presented. Therefore, the metric of additional words written down increased during the course with instruction, as is shown in Table
1.
In the next step, the process of increasing the effectiveness of note taking metrics was measured as a formative assessment, using two approaches. First, the relationship between the scores of final exams and variables of student’s individual behaviors were evaluated using multiple regression analysis and a step wise method of selecting variables. In the results, the overall contribution of most note taking activity metrics which were selected was high when instruction was given. However, the contribution of some variables of student’s characteristics which were selected is small for the no instruction condition. The contribution of a set of metrics of note taking activity increased with the number of sessions as the course progressed. The contribution, when measured as an R-square, remained at a high level between the 4th and 12th sessions of the course, as is shown in Fig.
2. However, the contribution decreased in the last two sessions. In regards to the change in the number of terms the lecturer presented in Fig.
1, the number of terms in the last two session was the smallest of all sessions during the course. The lecturer explained the contents using mainly a textbook, and so the number of terms presented was small. Therefore, this information may influence the metrics of cumulative values.
When the relationships were validated using multiple regression analysis, the possibility of prediction of final exam scores during the progress of the course was confirmed. To do this, SVR was introduced as a robust prediction procedure to test note taking activity metrics. Also, the contribution to predicting final exam scores remained at around 0.6 between the 4th and 12th sessions of the course, as is shown in Fig.
3. These results suggest that it is possible to improve participant’s final exam scores during the course. Since the prediction function is based on metrics of note taking activity, it may be possible to provide each participant with appropriate instruction regarding their individual note taking abilities.
However, the results are based on the case of one course at a single Japanese university, and the number of participants was not large. The validity of this approach should be investigated with care, and the validation for these points will be the subject of our further study.