Effects of embedded questions in recorded lectures

Participants engaged significantly and substantially longer with the video-recorded lecture that included embedded questions. The presence of quizzing resulted in higher overall scores for total time and replays. The findings are aligned with the results in two empirical studies that also reported that more time is spent on lectures with embedded questions compared to non-quizzed lectures (Kovacs 2016; Vural 2013).

Total time is a general signal of participant interactions with the video. In automated data analyses of video usage, total time is considered one of the most important signals of active processing (Guo et al. 2014). The data on replays complement these data. The finding that these replays were both more frequent and longer in duration with the presence of questions matches our expectations, but even so, merely attests to the effectiveness of questions as a stimulus for video processing.

Logging systems enable researchers to mine a variety of user interactions with the video. In this study, these measures were restricted to general records of video processing (i.e., basic play and total time), and a specific record of video processing that is a probable signal of a remediating action (i.e., replays). Future research might want to use more refined data mining techniques to probe more deeply into the effects of embedded questions on video interaction events. For instance, records could be examined for the number of users who attempt to answer the embedded questions, the correctness of the answers, whether replays occur before or after giving an answer, and the time spent on answering (e.g., Kovacs 2016; Li et al. 2015; Li and Baker 2018). Such information can provide answers to questions such as whether embedded questions affect video navigation, and whether there are interaction event peaks around these questions.

The mean scores for usefulness, ease of use and satisfaction were about 5.5 on a 7-point scale in each condition. Participants believed that video-recorded lectures generally provide a useful resource for learning and are easy to process. In addition, they felt that the specific lecture yielded a satisfying experience. These findings are in line with a large number of studies that have reported positive student appraisals of video-recorded lectures (e.g., Baepler et al. 2014; Burgoyne and Eaton 2018; Kim et al. 2014).

There was no effect of condition on usability perceptions. The findings thus did not support the positive contribution of quizzing to satisfaction reported by Spanjers et al. (2015). The absence of such a contribution could have been due to the positive overall appraisals of the lecture itself. Voluntary comments from participants after the experiment indicated that they enjoyed the topic and how it was presented. These positive comments may have overshadowed any perceived benefits of quizzing within the lecture.

In both conditions, the self-efficacy score was considerably above the neutral midscale value. This suggests that participants were fairly confident about their knowledge development. They believed that they remembered and understood the video-recorded lecture well. The positive appraisals for self-efficacy hold considerable promise for students’ future engagement with video-recorded lectures. That is, current self-efficacy has been found to be a predictor of future persistence and effort expenditure in comparable settings (Bandura 2012; Bandura and Locke 2003).

There was no effect of condition on self-efficacy. This finding thus did not corroborate the outcome of a recent meta-analysis on clickers (Hunsu et al. 2016), nor did it replicate the laboratory study by Tweissi (2016) that found a significant effect of embedded questions on self-efficacy. One possible explanation is that, unlike in the present study, in most clicker studies, as well as in Tweissi’s research, feedback was given for the responses to the questions. Another explanation is that the intervention may have been too short to influence the students’ self-efficacy. It takes time to hone ones’ skills, and increase confidence in grasping the content of a video-recorded lecture; a single, 30 min lecture is unlikely to effect a major change in these facets. Future research might therefore want to investigate whether repeated exposure to video-recorded lectures with embedded questions benefits students’ self-efficacy development more from than repeated exposure to such lectures without questions.

The presence of the embedded questions had a significant, medium-sized effect on what was learned from the lecture. By and large, this concurs with the findings on adjunct questions and on quizzing (e.g., Jing et al. 2016; Smith et al. 2010; Uner and Roediger 2018; Vural 2013). To our knowledge, the present study is one of the few controlled experiments on open-ended questions in video-recorded lectures, and it is the only study in which no feedback was given for responses to these questions (see Cummins et al. 2016; Szpunar et al. 2013, 2014; Thomas et al. 2018). The absence of feedback, in combination with asking open-ended instead of multiple-choice questions, was considered to be a strong stimulus for students to engage in information retrieval, which would thereby enhance their learning.

Dunlosky et al. (2013) have argued that open-ended questions are more likely to trigger elaborate retrieval processes than multiple-choice questions do. That is, their review of learning techniques indicated that practice tests that require more generative responses (such as recall or short-answer) are more effective than tests that require less generative responses (such as recognition). They also mentioned, however, that this conclusion is tentative and that further work is needed. Karpicke’s (2017) more recent review indicated that comparative studies from the last 10 years have yielded mixed outcomes and that finding proof for this claim is more complex than initially thought. Among other things, he pointed out that initial retrieval success can play a mediating role, because it is often higher for multiple-choice questions. In addition, he mentioned that feedback plays a mediating role and that the presence of feedback seems especially relevant for open-ended questions.

The present study found that even without feedback, the presence of open-ended questions enhanced learning. An important reason for the absence of feedback was that we wanted to prevent the risk that students might engage less in constructing their own answers when feedback was present. That is, research on adjunct questions warns that feedback can forestall the students’ attempt to retrieve or reconstruct the answer from memory, as students are inclined to depend more on the feedback as a way to obtain the correct answer (Hamaker 1986; Roelle et al. 2017). In addition, a recent meta-analysis on the testing effect provided a similar explanation for the surprising finding that feedback did not moderate learning (Adesope et al. 2017). In view of the substantial evidence in favor of feedback (e.g., Fiorella and Mayer 2018; Shute 2008), Adesope et al. (2017) suggested that more research is needed to reveal when feedback does or does not enhance learning. One feature of feedback that the meta-analysis did not have enough research on to analyze was timing (i.e., immediate versus delayed). To enhance the effectiveness of quizzing, future studies might want to investigate the contribution of delayed feedback, because this design can serve the dual goal of both stimulating and supporting the students’ thought processes, using a different time point for each. That is, the absence of immediate feedback for responses to the open-ended questions may keep the students challenged to construct their own answers, while information about the correct response that is available after a delay may help substantiate, enrich or challenge their own constructed answers in a productive way.

The logged records of the students’ actions involving video play revealed that the presence of questions led to more extensive playing, and hence potential viewing, of the videos. Unfortunately, these data could not be linked to the students’ answers to the embedded questions, because they were not recorded in the present study. This seems like an important issue for further research. If student answers to the embedded questions are known and (delayed) feedback is given, then plausible effects of both of these on additional video play can be evaluated (e.g., correct responses yield little or no extra engagement, incorrect responses stimulate repeated video play). A simpler set-up that such a study could use would be to pre-assess the difficulty level of the questions and then correlate that with video engagement.

Video engagement was not related to test performance. This outcome was unexpected, because empirical studies generally report that more video engagement leads to higher learning outcomes (e.g., Morris et al. 2005; Wei et al. 2015).

The finding stresses the point that video engagement is a proxy for video processing. It is a valuable, unobtrusive record that is a necessary but not sufficient prerequisite for comprehension and learning. As mentioned earlier, more refined records of video interaction events can provide a more detailed view on the effects of embedded questions on video engagement. Future research might want to complement these records with interview data to gather information on the reasons why users do, or do not, engage with embedded questions (e.g., Shin et al. 2018). Such studies could also considered recording verbal protocols or use other observational methods to obtain insights into how users process embedded questions. Such data could reveal whether the questions prompt users to reflect on the lecture and whether they connect the new information with prior knowledge, among others.

The experimental condition had a significantly higher score for the single repeated question, but did not differ from the control condition on the remaining composite test score with that question removed. This finding indicates that while embedded questions have a moderate effect on learning of questioned content, effects on learning of non-questioned content may be more limited. A similar cautionary note has been voiced for quizzing in applied settings using authentic educational materials (Agarwal et al. 2012; Nguyen and McDaniel 2015; Wooldrige et al. 2014). Future research might therefore want to test this by systematically varying the number and type of previously questioned or non-questioned items. Such research might also contribute to gathering information on the learning strategies involved in video replays. In addition, it might want to measure students’ self-regulated learning skills, as this appears to be an ignored moderator in research on quizzing (see Shapiro et al. 2017).

Some limitations of the study have already been mentioned, such as incomplete data on total time, and the absence of information about the answers to the embedded questions. One other limitation has not yet been mentioned, namely, the topic of segmentation. Embedded questions obviously need to be positioned somewhere during a video-recorded lecture. The issue is where such breaks can best be created. By their very nature, embedded questions break down a lecture into parts. This can induce a segmenting effect (see Mayer and Pilegard 2014). To disentangle the two factors (i.e., questions and segmentation), in the present study, both conditions received separate parts of the complete lecture. The video-recorded lecture was split into four segments (videos) and each new segment required a user action to start it playing, as recommended in the multimedia literature (e.g., Biard et al. 2018). In the control condition, each video simply ended when an event was completed; in the experimental condition, there was an open-ended question. Given this parallel construction, it is possible to draw conclusions about the effect of embedded questions. Since embedded question automatically split up a complete lecture into sections, future research on their effects might want to turn to the multimedia literature for a principle-based approach to creating meaningful segments (e.g., Khacharem et al. 2013; Mura et al. 2013; Spanjers et al. 2012b).