Exploring crossmodal perceptual enhancement and integration in a sequence-reproducing task with cognitive priming

Our perception of certain physical features or attributes are naturally associated with each other, such as the auditory perception of an increase and decrease in pitch is associated with the visual or haptic perception of rising and falling in vertical position [14, 35, 42]. These kinds of associations can be found in other cross-sensory features as well, including the sound-brightness association, sound-quantity association etc. [14, 18, 39, 46]. As such, the term crossmodal correspondence (CC) in this paper refers to ‘a compatibility effect between attributes or dimensions of a stimulus (i.e., an object or event) in different sensory modalities (be they redundant or not)’ [41]. Accumulating behavioural studies show that when values of these associated attributes are combined congruently, for example, increased pitch combined with increased vertical elevation, the accuracy and efficiency of people’s sensory-motor response will be improved (for a detailed review, see [41]). However, if the attributes are combined incongruently (i.e. the direction of change of one of the involved attributes is reversed, for instance, when an increased pitch is associated with downward visual stimuli), sensory-motor responses will be disturbed.

Some research studies have pointed out that many CCs may occur due to the repeated exposure to the natural environment, from which people learned perceptual regularities of associated crossmodal physical features [41, 43]. Praise et al. empirically investigated the statistical mapping between sound frequency and the perceived elevation of the sound source [35]. With a wearable two-directional headphone, a large sample of natural sound recordings was collected by participants who move freely indoors and outdoors. The analysis of recordings reveals a consistent mapping between the sound frequency and the physical elevation of the source of the sound. Furthermore, when participants were asked to do a sound localisation task, researchers obtained a consistent observation concerning the frequency-elevation correspondence. Even though the correlation between the statistical environmental mapping and the perceptual judgement performance cannot be fully explained for now, the above investigation provides experimental ground for an ecological linkage between CCs and daily experience [2, 43].

Recent research also suggests that perception of crossmodal information can be influenced by task-irrelevant contextual features [9, 47]. Walker and Walker conducted an experiment to address the relative perception of crossmodal feature values. In their experiment, participants were presented with six circles with different luminance. Three of the circles were brighter than the background colour, and the other three appeared darker. Participants were asked to classify whether each of the circles was brighter or darker than the background, and confirm their answer by pressing one of two differently-sized keys, though participants neither visually nor haptically aware of the difference in size. Results showed that participants classify brighter circles more quickly with the smaller key in hand, and classified darker circles more rapidly with the larger key in hand. This finding suggests that CCs are not always absolute, and that sensory-motor responses based on them could be influenced subliminally by contextual stimuli, which in this case, the haptic perception of the size of the keys [47].

Moreover, the relativity of crossmodal perception points to further investigation, with a focus on intermediate crossmodal values that sit between pairs of polarised stimuli [9, 42]. Without fully understanding the sequential perception of intermediate values, i.e. graded crossmodal stimuli, the implementation of CCs in real-world scenarios remains limited. After all, a variety of crossmodal interaction cases involve consistent perception-action loops, either for spatial localization [19], graph sonification [3], or motor skill learning [17].

From an HCI perspective, we are aware of two studies that have directly investigated the design possibilities with graded crossmodal stimuli. Metatla et al. [29] evaluated interaction performance with different crossmodal congruency levels between shape, size and elevation. They adopted a game mechanism, using a tablet to display a sequence of visual, auditory, or visual–auditory stimuli. Participants were asked to tap in the perceived sequence as quickly and accurately as possible. Results showed that the visual condition produced better performance than both the auditory and visual–auditory conditions. However, in the bimodal condition where two CCs co-existed, participants tended to rely on pitch–elevation rather than visual stimuli as the primary CC. This result implies that different CCs may have a different level of influence on interactive performance.

In another experiment, Feng and Stockman tried to replicate the crossmodal effect through augmented physical features of a tangible object on an interactive tabletop [16]. Participants were required to discover a hidden target by moving the object around the table, and by observing the concurrent crossmodal feedback to determine the distance to the target. Visual, auditory and haptic modalities were combined into unimodal, bimodal and trimodal feedback. Results revealed that more accurate movement and efficient corrections were achieved with bimodal and trimodal feedback. However, in the crossmodal condition which implemented the mapping between colour and euclidean distance, i.e. an increase in R value (RGB scheme) associated with a decrease in the distance towards the target, one-sixth participants appeared to have opposite mapping polarity. This was due to their familiarity with a different crossmodal association, the colour signifier and motion correctness, i.e. red indicated wrong moves, therefore associated with moving away from the target. These two associations happened to have a mutually exclusive mapping polarity. This result suggested that people’s sensory-motor reaction may not solely be determined by crossmodal stimuli in a bottom-up manner, but sometimes rather influenced by previous experience and current interaction goals [2, 22].

From a human-centred design perspective, researchers have shown that CCs can be implicitly activated in a natural interactive scenario. Bakker et al. applied a human-centred iterative design approach during several music education workshops, where groups of children were encouraged to use body movements to express sound features. Several auditory-haptic crossmodal associations have been identified from self-generated movements, such as increased volume associated with increased speed of movements, as well as rising postures [5]. When implemented these identified CCs on hand-held musical interfaces and applied to a subsequent music learning session, children showed improved music reproduction performance. However, we do not have enough empirical evidence to determine whether the improved performance was due to self-activated crossmodal mappings or due to an improved familiarity produced by a series of iterative design activities.

Priming, in the fields of cognitive and social psychology, generally refer to an experimental technique whereby exposure to one stimulus influences a response to a subsequent stimulus, without conscious guidance or intention [6]. It includes different priming mechanisms: the conceptual priming activate an internal mental representation in one context in such a way that the participant does not realise the relation between that activation and the later influence in an unrelated task [6, 15]; the perceptual priming shares the same activation mechanism, but the contextual features of priming have direct indications for later tasks [6, 37].

In the field of HCI, researchers applied different priming techniques to facilitate interaction [1, 25] and enhance cognitive performance [12, 21, 26]. Two major categories are implemented with empirical investigation. The technique of using subliminal stimuli to trigger fast and automatic responses is referred to as subliminal cueing, which adopts strict experimental control over the factors of cueing time and content. The cueing time is less than 50 ms, within which the perception is assumed to be subliminal, and the cueing is commonly an external stimulus presented as a visual image or geometric shape, which exposes information that would appear during subsequent tasks. This technique was inherited from the masking paradigm from experimental psychology, for the purpose of investigating the effect of priming on selection behaviour [4, 37]. Thus subliminal cueing is also known as subliminal visual masking. For example, Aranyi et al. conducted an experiment to investigate the effect of masked indices on subsequent item selection behaviour in a virtual environment. Results showed that a short-lived impact (within one second) of masked cues was indeed influence participants selections subliminally [4]. Recently, a similar masking paradigm with three types of visual cues has been applied in a mobile application. Evidence showed that the priming stimuli presented with a time window of 17 ms were not fully subliminal [37]. In summary, investigations of cueing effects using the masking paradigm that employing different time windows lead to inconsistent observations. The reliability of the results needs to be further tested. Therefore, this cueing technique is not suitable for tackling the present research question: understanding whether crossmodal perception can be enhanced subliminally for a specific interaction task.

The technique of cognitive priming has been applied with priming material either in the form of visual images, video, or textual stories [21, 25, 26]. The material does not always have a direct relationship with the interaction tasks, and it can be presented either before or during interaction without restriction on time windows. The purpose of this technique is to enhance cognitive function or render affective states for specific interaction tasks. Harrison et al. employed text-based stories as the priming material to investigate the influence on subsequent visual judgment performance on different types of charts. Results showed that positive priming improved participants’ visual judgment accuracy [21]. In another case, Lewis et al. applied affective computational priming in the form of background pictures on a creativity-support design tool [26]. With the background priming, participants showed an improvement in the quality but not the quantity of their design. The effect of cognitive priming was observed even after the priming has been removed in the latter design tasks.

In conclusion, in the existing HCI literature, the reliability of applying the subliminal cueing needs to be further tested. In comparison, cognitive priming has been used to invoke previous experience and mental states for goal-oriented tasks. Following this line of investigation, in the present study, we used a cognitive priming technique to invoke crossmodal experience and measured its effect on subsequent sensory-motor responses.

The experiments have to be a between-subjects design to eliminate the learning effect of priming. There were two independent variables. The first was the type of priming: the perceptual priming (P-prime) and the conceptual priming (C-prime). The second variable was the two crossmodal mappings, the pitch–brightness correspondence and the pitch–elevation correspondence. The two experiments have six conditions with four manipulation groups and two control groups. The manipulation groups were: P-prime and C-prime on pitch–brightness mapping respectively (condition 1, 2), the P-prime and C-prime on another crossmodal mapping, pitch–elevation respectively (condition 3, 4). The control groups were pitch–brightness mapping without priming (condition 5) and the pitch–elevation mapping without priming (condition 6). The manipulation factors and experimental conditions are listed in Table 1.

The task in both experiments is learning and reproducing crossmodal sequences. The sample sequence was made up of two parts, the auditory melody with five-pitch values equal to musical notes C, D, E, G, A; and the visual counterpart with either five levels of brightness or vertical position displayed in five circles on the screen (Fig. 1 Experiment 1a). The detailed information about crossmodal mappings is presented in the experimental platform section. During each trial, a randomly generated sample sequence was displayed once, which has melody matched with the concurrent visual stimuli. Participants are required to reproduce the melody by clicking the circles on the screen to reproduce the melodic sequence.

The general procedure is as follows. First, participants were given a consent form with details of the experiment to read and sign. Then they were introduced to the priming session based on which group they were assigned. Participants who were in manipulation groups were told to watch a short video or an animation, while control groups had no priming session and simply moved to the next session directly (Fig. 2). After priming, participants move to the warm-up session to get familiar with the task procedure and the hardware setup. To ensure everyone received practice without overtraining, they were instructed to practice no less than twice and no more than six times. The crossmodal stimuli used in the warm-up session were different from those in the task session, for the purpose of minimising learning effects. Specifically, the sequences in the warm-up session used three-note sequences, while the task session used five-notes sequences. After practice, participants moved to the task session, which contained 16 trials (Fig. 2). All the sessions were programmed in a single piece of software, participants moved to the following section by pressing a “next” button on the screen.

After participants finished all the trials, they were asked to complete a post-experiment questionnaire. The first section of this questionnaire collected participants’ demographic information, music training history, and whether they had any visual or auditory disorder recently. The second section collected subjective evaluation data, including their interpretation of the priming material, and the interaction strategy they used during the game. The entire experiment lasted for 7 to 13 minutes.

Two depended variables were time intervals between each input and task error rate. Time intervals were counted as the length of time between adjacent clicks. The difference in time intervals between the manipulation groups and the baseline groups was calculated as an indicator of the priming effect on sensory-motor reaction efficiency. The interaction error rate was calculated by dividing the number of wrongly produced notes by the total number of sample notes.

In addition, the qualitative data that was included in the analysis was as follows: (1) the subjective interpretations of priming material collected from the post-experiment questionnaire, and (2) Plots of the overall task accuracy which was calculated not by summing the number of wrongly produced notes, but by the number of incorrectly aligned CCs during reproduction. For example, if one step in a sample sequence change from note C to note E with the brightness going up 2 levels, both the reproduced move from C to E with brightness going up 2 levels and D to E with brightness going up 1 level would be counted as correct, since the increased pitch value corresponded with brightened visual display. However, the step from A to E with brightness level going down would be counted as an incorrect step, as the decreased pitch sound should not be aligned with the increased brightness level.

Since the pitch–brightness and pitch–elevation correspondences have been repeatedly evaluated in the literature as working for most people [14, 18, 39], we assumed that most participants would be able to do the task without perceptual discrepancy on those crossmodal stimuli. Thus we hypothesized that:

H1: Following previous studies [25, 26, 32], we predict that people’s crossmodal perception can be enhanced by cognitive priming. Specifically, that both the P-prime and the C-prime for the pitch–brightness mapping and the pitch–elevation mapping will support faster sensory-motor responses and produce better task accuracy than would be seen in the two baseline groups.

H2: The type of priming will have different sensory-motor modulation effects on task performance. However, we chose not to predict the direction of the difference.

A Kolmogorov-Smirnov test showed that the time intervals data can be assumed to be normally distributed. We ran a one-way ANOVA to compare the results of P-prime, C-prime and baseline conditions on pitch–brightness mapping, as well as the P-prime, C-prime and the baseline condition on pitch–elevation mapping. Fisher’s LSD test was used for posthoc tests of main effects. We used a confidence level of \(\alpha \) = 0.05 for the tests.

The hypothesis H1 that crossmodal perceptual preference can be induced by cognitive priming has been confirmed for the pitch–brightness correspondence. With the pitch–elevation correspondence however, only the C-prime group showed a positive effect on task performance, while the P-prime group had slower motor responses (Fig. 4 (up-right chart)) and poorer accuracy (Fig. 5a (pitch–elevation)).

These inconsistent outcomes between the two crossmodal correspondences may result from different interaction strategies afforded by the task paradigm. Previous research mainly used a speeded classification paradigm [41], which required participants to react to stimuli with a one-shot key press. This process requires little working memory or cognitive resource. In the current experiment, however, participants had a clear interaction goal and engaged with a sequence of input actions. In this regard, the P-prime, which primed a direct association between the auditory and visual stimuli, possibly functioned as explicit instruction, and encouraged participants to recall and check during the interaction constantly. This extra cognitive load could explain the increase in response time. Subjective evaluation support this explanation, which will be discussed in the latter part of the section.

Surprisingly, such degraded performance was less evident in the P-prime group with the pitch–brightness correspondence. According to self-report, the interaction strategy used by participants might compensate the inferior effect. In responses to a question about interaction strategies for reproducing the sequences, 67\(\%\) of participants in the pitch–brightness condition reflected that they used visual tracking strategy to follow the flash pattern of the sequence. In contrast, none of the participants in the pitch–elevation mapping group recalled they had used any strategy. Indeed, the visual pattern for the pitch–elevation correspondence has a vertical arrangement, and the moving path of the sequence is overlapped. While the visual pattern for the pitch–brightness correspondence has a planar arrangement, and the moving path was more likely to be perceived as a trackable trajectory [20]. Since visual perception is more sensitive to spatial arrangement, while auditory perception is more sensitive to temporal information, the combination of a 2-D visual stimulus with a 1-D auditory stimulus can produce a multisensory enhancement effect [24]. As a result, the time delay and the error were less salient in the conditions with a pitch–brightness mapping.

The hypothesis H2 that priming types will have different effects on sensory-motor modulation has been confirmed. Results showed that the C-prime condition did improve participant’s crossmodal perception, in both the pitch–brightness and the pitch–elevation correspondences, which in turn supported faster motor response as well as improved accuracy of crossmodal sequence reproduction. In comparison, the P-prime condition had a positive effect only on the pitch–brightness crossmodal mapping. This effect may be due to the compensatory interaction strategy afforded by the task paradigm. Previous studies have shown that the priming process without conscious awareness can have a positive effect on people’s affective reaction and decision making [25, 32], thus we deduce that the perceptual enhancement with C-priming in our case could also have functioned in a subliminal way. It enabled participants to have a stronger perceptual alignment with a specific CC, which facilitated fast and instantaneous motor response. While the P-prime material may rise to the level of conscious awareness during the priming process, thus involving working memory allocation while doing the task, which lowered the overall performance to some extent.

A supportive finding of the above assumption can be gleaned from the post-experimental questionnaire. Most of the participants in the C-priming groups were not aware of the CC presented during priming, and could not consciously recall the correlation between the priming and the interaction task. In all of the 40 subjective interpretations from the C-prime groups (2 C-prime groups with 20 participants in each), only 2 participants reported that ‘it was about how the sound changes with visual information’, and that ‘they are there to focus your concentration on the screen and the sound’. The other 28 participants’ answers were exclusively focused on the detailed contents of the prime context. Such as ‘It’s showing the sunrise’ for the pitch–brightness priming and ‘flying birds, running animal’ for the pitch–elevation priming. Meanwhile, only 4 of the 40 participants rated that the C-prime was helpful for the interaction task. In contrast, 36 of 40 participants in the P-priming groups were fully aware of the purpose of the priming. One of the typical answers, for instance, was that ‘information that helps you prepare for the game’ for the pitch–brightness priming, and that ‘different notes of music vertically spaced’ for the pitch–elevation priming. 36 of the 40 participants rated that the priming was helpful. From the usability perspective, this evaluation can explain the assumption of extra cognitive load that was discussed for the hypothesis H1.

Combining participants’ behavioural data with their subjective rating, it can be confirmed that the C-prime subliminally enhanced crossmodal perception and led to better task performance. This fact also explains why the C-priming was not considered helpful by most of the participants. In comparison, the purpose of the P-priming was recognised in most cases and thus tended to be acknowledged to be helpful for task completion, however, the actual behavioural data of the P-primed participants pointed to the opposite.

To better understand the quantitative effects of the priming techniques on subsequent goal-oriented tasks, participants’ performance accuracy based on crossmodal alinement, as explained in Sect. 4.4, was plotted on two temporal scales: the within-trial performance and between-trial performance (Fig. 6 Experiment 1). Along the horizontal axis, which represents the steps within trials, the P-prime groups produced more accurate moves in the first two steps and were more prone to error in the last two steps. In comparison, the C-prime group and the control group do not show this pattern. These observations reflect that participants in the P-prime group relied more on working memory capacity than on crossmodal congruency. This observation is consistent with objective behavioural data as discussed previously. Along the vertical axis, which represents the between-trial performance, no improvement can be observed in the later trials in all groups. In other words, the data did not reflect learning effect between trials.

A Kolmogorov-Smirnov test showed that the data set can be assumed to be normally distributed. The one-way ANOVA was used as the statistical method with a confidence level of \(\alpha \) = 0.05.

Hypothesis H3 has been confirmed for priming on the pitch–elevation mapping but rejected for the priming on pitch–brightness mapping. The two priming groups and the control group for the pitch–brightness correspondence have no significant difference in terms of motor response speed (Fig. 4 (lower-left)), and there was no obvious difference between the primed groups and the control group in terms of the accuracy (Fig. 5b). The other two priming groups for the pitch–elevation correspondence had significantly better performance than the control group in terms of both the motor response speed and sequence re-producing accuracy.

One explanation for this fact may be that the pitch–elevation correspondence may have stronger perceptual weight [11] than the pitch–brightness correspondence. The perceptual experience of the pitch–elevation correspondence may be encountered more frequently in daily interactions than the pitch–brightness correspondence, thus the neural response to pitch–elevation stimuli may be stronger than that for the pitch–brightness correspondence [7, 44]. This crossmodal perceptual weighting may not be easy to observe in situations where crossmodal stimuli are isolated from one another, such as in the case of experiment 1 and in many classification-based tasks employed in previous studies [10, 14, 41]. While in the situation of experiment 2, where crossmodal stimuli were overlapped and incongruent, priming on the relatively stronger CC, pitch–elevation correspondence in this case, enabled participants weighting the dominant stimulus selectively in the subsequent activity. However, priming on the less dominant CC, the pitch–brightness correspondence in this case, seems to have little or no effect on modulating subsequent activity. As a result, only those groups primed on the pitch–elevation correspondence produced improved performance. Thus we postulate that crossmodal stimuli integration may occur in a selective manner rather than an additive manner in the presence of priming [23].

Following the discussion in experiment 1, to observe how the priming technique influenced performance in an incongruent crossmodal situation, the crossmodal alignment accuracy of participants has been plotted in Fig. 6 (right). The within-trial comparison shows that the P-priming groups produced more accurate moves at the first few steps in the sequences. Compared with the P-priming groups in experiment 1 (Fig. 6 left), a similar pattern appeared regardless of how crossmodal stimuli were combined. Based on this observation, we can deduce that participants presented with explicit P-priming material tend to do the task in a way that involved more working memory capacity. Different from the situation in experiment 1, the extra attention allocation made participants less susceptible to perceptual distractions in experiment 2. The between-trial comparison shows that participants in the C-priming groups produced better performance in the latter half of total trials than the earlier half. Comparing with the C-priming groups in experiment 1, which shows better accuracy in the earlier trials, it is plausible to attribute the improvement produced in experiment 2 to the practice effect more than to the priming effect. However, the plot for the control groups for experiment 2 does not have a salient observable difference between earlier trials and later trials due to practice. This observation could indicate two things. Firstly, the C-priming do play a part in modulating crossmodal perception, but the effect is not immediate. Secondly, without priming, participants are susceptible to the crossmodal distractor that happened to share the same crossmodal perceptual channel. Therefore, in the absence of priming, the integration of mutually exclusive crossmodal stimuli more likely happened in an additive manner than a selective manner [11].

In terms of potential applications, the present findings could contribute to the field of mindless technology [1, 36] and multisensory interaction [31]. Mindless computing emphasises the subconscious mental process, and is characterized as a fast, automatic interaction that requires little or no effort. Mindless technology makes use of subliminal mental state, aiming to guide people towards intended interaction behaviour without annoying or distracting them. Previous research showed that both environmental features and the interactive system itself could impose a subtle influence on people’s choice and judgement in interactive tasks [1, 4, 36]. The present study, built on theoretical accounts of CCs, contributes to the field by introducing cognitive priming as an approach to enhance crossmodal perception, and consequently improves interactive sensory-motor performance. The implications for designing multisensory, mindless interaction are first, implicit conceptual priming can be used to improve interaction efficiency when information shares only one CC feature. In addition, when there are two or more streams of crossmodal information which have mutually exclusive congruency, explicit perceptual priming could be used for improving interactive performance.

The present study also provides insights for the design of multisensory interactive systems [31]. As is so often discussed, one of the advantages of multisensory interaction is that it can expand peoples’ cognitive capacity by presenting several streams of information through different sensory channels [24, 34, 48]. In order to make good use of this advantage, we need to consider the situation in which two or more streams of information may occupy the same pair of crossmodal channels. To make things worse, the streams of information may contain mutually exclusive CCs, as demonstrated in experiment 2. Both cases could inhibit information processing capacity, and as a result, negate the advantages of multisensory interaction. The design implication for handling this situation could be isolating information streams either spatially or temporally to avoid confusion. Another solution could be shunting information flow through different pairs of crossmodal channels, e.g. the visual–auditory channels and the visual–haptic channels, to reduce cross-sensory distractions.