How Cross-Representational Signaling Affects Learning from Text and Picture: An Eye-Tracking Study

Diagrams and pictorial representations are often used to support comprehension of instructional documents. Multimedia learning research showed that learning with multiple representations (particularly written text and pictures) can be beneficial to comprehension provided that learners can identify links between representations through cross-references [3, 10]. The most widely accepted models of multimedia learning (CTML from Mayer [6]; ITPC from Schnotz and Bannert [10]) claim that information from verbal and pictorial representations are first processed by media specific (verbal or pictorial) channels before being integrated in a coherent model of the situation relying on both those representations and previous knowledge. The latest version of the ITPC model from Schnotz [9] includes a coherence principle which predicts that “students learn better from words and pictures than from words alone if the words and pictures are semantically related to each other” ([9] p. 23), especially for students with poor reading skills or little prior knowledge.

Effectively guiding learners’ integration processes may be channelled through the insertion of visual or verbal cues in either one or both verbal and pictorial representations [11]. A meta-analysis by Richter et al. [8] found an overall significant beneficial effect of signaling text-picture relations on comprehension that was more profitable to low to medium prior-knowledge than to high prior-knowledge learners. These results confirm the ITPC [9] claim that supporting text-picture semantic links facilitates the construction of a coherent mental representation. A possible moderating effect of reading abilities was however not investigated.

Kalyuga et al. ([4] Exp. 2) used interactive colour coding in both representations to facilitate search of corresponding verbal and pictorial elements. The cueing group performed significantly better than the no-cueing group. Using eye-tracking to compare the use of verbal cues (labelling) in the pictorial representation Mason et al. [5] found that more integrative processing, measured through eye-fixations, occurred with labelled pictures. This research shows that eye-tracking data can give interesting insights in on-line processes of text-picture integration.

In the present study, we implemented Cross-Representational Signaling (CRS) through colour coding cues and picture labelling to highlight semantic links between written texts and visual pictures. Two versions (with or without CRS) of the same multimedia document (a 5-page text and picture instruction explaining the risks of being caught in a rapid) were designed. After completing reading skills tests, participants learned with one of the two versions of the multimedia document and answered comprehension (text-based and inference) questions.

We assume that CRS facilitates the construction of a coherent mental model, which should lead to better comprehension scores, especially for students with lower reading skills. Eye-tracking data will provide insights on the way CRS affects the processing of instructional diagrams. In particular, following Mason et al. [5], we expect that signaling in the text will prompt exploration of the pictures and increase the total time spent on the pictures.

The experimental material was a 5-page expository document including text and static representational pictures on how to escape the Maytag effect when being caught in a rapid. The material was carefully selected and designed to ensure that both media were necessary for no prior knowledge learners to comprehend the document. The pictures were designed to be representational in the sense of Carney and Levin [1]. CRS encompassed the following mutually referencing verbal and visual cues: colours, symbols and labels (see Figs. 1 and 2). The material was presented on a 23” screen, and participants’ eye-movements are recorded with a Tobii TX300.

Participants’ prior knowledge was evaluated online, before the experiment, with a self-assessed multiple-choice knowledge questionnaire. Because the ITPC advocates that semantic links are helpful only to low prior knowledge readers, only participants with low or no prior knowledge on the topic were recruited. During the experiment, participants completed two reading skills assessments (a vocabulary test from Deltour [2] and an inference generation test adapted from Meteyard et al. [7]). Then they studied the multimedia document in one of two experimental conditions (with vs. without CRS). After reading, participants completed a 7 items Likert scale questionnaire on motivation, perceived difficulty and perceived effort. They ended with the comprehension test, with 13 open-ended questions at three levels: text-base comprehension, local bridging and global-bridging inference. A drawing task was also included, in which participants had to draw and name the different currents involved in the formation of a whitewater.

The experiment was still running when we wrote this paper. A random sample of 40 to 50 undergraduate university students in education sciences or psychology will be recruited overall.

Following previous research in multimedia learning using eye-tracking as an online measure of comprehension [3, 5], we will analyze the collected data with first-pass and second-pass fixations. Specifically, we will consider fixations as gazes and focus on look from text to picture, both in general and with targeted AOI.

First, following the ITPC model [9] we expect that multimedia comprehension will be higher in the CRS than in the control condition, especially for students with low reading skills. Regarding on-line processing, we expect that participants reading the multimedia document with CRS will look at the picture during first-pass and second-pass reading more often than participants without CRS. Indeed, research by Mason et al. [5] pointed out that a picture with verbal cues elicited more integration with the text than a picture without verbal cues. Further exploratory analyses of eye-tracking data will provide insights on how text-picture integration processes differ with and without CRL. Participants reading skills will be inserted in the analyses as a potential moderator.