4.1 General findings
Figure
8 illustrates the statistics over all events, genders and age groups. The indicators
Xn refer to the questions
n of sections
X of the questionnaire shown in Table
2. Answers are encoded as cardinal numbers 0–3 where 0 signifies “no answer”, 1 the first option, and so on.
The statistics on participant behavior show that most jumpers prefer a conservative jump style at their first jump, which is not surprising. Of the 10% daring jumpers, 65% were male and only 5% were youngsters compared to almost 8% in the total sample: young subjects were more risk-averse than elder subjects. It appears noteworthy that 65% of the daring jumpers had prior virtual reality experience compared to only 40% in the entire sample. That seems to imply that users with virtual reality experience trusted the installation more than users without that knowledge—though the risk of injury is not related to the virtual reality part of the installation. Concerning age vs. experience, 47% of the young participants had prior experience compared to 40% in the adults and 33% in the elder subjects; 32% of the female participants had virtual reality experience compared to 46% of the males.
Content statistics: the preferred content was the skydiving application (58% compared to 27 and 15% for space flight and air race, respectively). Of the female participants, even 63% of the subjects chose the skydiving application. Only 6% of the skydivers showed a daring jump, whereas 26% of the air racers performed an extraordinary dive. The correlation is understandable since the air race was presented to the clients as potentially causing nausea, and therefore, mostly chosen by risk-taking individuals. Over all contents, 65% of the participants reported very high immersion and 79% very high excitement. The values for the skydivers lie exactly in the global average. Interestingly, though air racers experience even higher levels of excitement (83%) their chance of very high immersion is slightly lower than the average (63%). It might be concluded that excitement is not just a function of immersion but might also be triggered without perfect virtuality. Finally, the average risk of experiencing nausea was only 8%. For skydiving, it was even lower (5%) and for the air race it was 17%. Hence, for the evaluation of the hypotheses on motion sickness, we mostly depend on the 30 subjects that chose the air race content.
Next, we investigated the general feedback on the Vragrancer. 85% of the participants perceived the smells present in the multimedia content. The non-perceiving 15% are twice the number of subjects that declared (partial) anosmia. After filling out the questionnaire some participants asked whether smells had actually been present and were surprised to hear that this had permanently been the case. Independent of the perception of smells, 86% of the participants replied that smells enrich the virtual reality. Of those subjects that perceived smells even 93% consider olfactory stimuli an interesting enrichment. This positive result appears to justify our approach to use smells only for selected events, thus causing a higher perceptual effect than through constant application.
Of the smell-perceiving participants, 32% considered the smell of bypassing flying objects (a synthetic combustion engine smell used for airplanes and space ships) as positive—though it is fair to say that objectively this smell is not pleasant. Even so, the combustion scent seems to emphasize the multimedia event “flying object” which causes a subjective rating of the smell that differs significantly from the objective judgment. In total, 63% of the smell-perceiving subjects found the scents very well synchronized with their multimedia events. Only 7% could not match them. 55% of the subjects found the smells adequate for their multimedia events which shows that synthetic smells in combination with other channels are able to form a complex, rich-in-detail media object.
The last group of descriptive statistics concerns the touch stimuli employed in the Jumpcube. For example, together with visual stimuli the suspension system was employed to simulate turbulences and crashing through the window of a building. Whereas 96% of the participants found the actual jumping exciting, 91% found the landing (with turbulences) exciting. Of those participants who had a crash through the window in their content, again 91% found it exciting. There appears to be no gender aspect in experienced excitement: though in situ it was expressed very differently by members of the two genders, the feedback is about the same for both.
An interesting result concerns clouds and spray—only present in the skydiving application. Of those participants that perceived spray (94%) 97% found the flight through the spray-enriched clouds exciting. Of the others, only 43% found the clouds exciting and 29% did not even notice them (compared to just 2% of the spray-perceiving subjects). These results show clearly how important a simple haptic stimuli can be for raising content awareness. 99% of the spray-perceiving subjects found their virtual skydive exciting, compared to only 86% of those who did not perceive spray.
The presented findings are only a small sample from the patterns present in the data. A long sequence of other examples could be given. Still, after drawing a sketch of the results it appears beneficial to move on to the formal analysis of the data.
4.2 Hypothesis testing
Due to the nature of the answer categories in the questionnaire, the hypotheses had to be tested with binomial tests. For the hypotheses listed in Table
1, Table
4 lists the subsets that were used to calculate the test variable
p and the default value
\(p_0\) of the binomial tests. Again,
Xn refers to question
n of section
X of the questionnaire in Table
2. The denominator of
\(p_0\) is not given as it is always the total of the participants (
\(n=196\)). The “rule” column refers to the testing method: two-sided (
\(p=p_0\)) or one-sided—depending on the formulation of the hypothesis.
Table 4
Operationalization: data sets employed for hypothesis testing
H.O1 | A5 = yes & C5 = yes | A5 = yes | C5 = yes |
\(p\ge p_0\)
|
H.O2 | A2 = yes & A3 = ok | A3 = ok | A2 = yes |
\(p\ge p_0\)
|
H.O3 | A1 = yes & C6 = yes | A1 = yes | C6 = yes |
\(p\le p_0\)
|
H.O4 | A1 = yes & C4 = very | A1 = yes | C4 = very |
\(p\ge p_0\)
|
H.H1 | B1 = yes & C1 = 50+ | C1 = 50+ | B1 = yes |
\(p = p_0\)
|
H.H2 | B4 = yes & C6 = yes | B4 = yes | C6 = yes |
\(p\le p_0\)
|
H.H3 | C5 = yes & D2 = Sky | D2 = Sky | C5 = yes |
\(p = p_0\)
|
H.H4 | B2 = yes & B3 = yes | B2 = yes | B3 = yes |
\(p\ge p_0\)
|
H.G1 | A1 = yes & B2 = yes & B4 = yes & C5 = yes | A1 = yes & B2 = yes & B4 = yes | C5 = yes |
\(p\ge p_0\)
|
H.G2 | C4 = yes & C5 = yes | C5 = yes | C4 = yes |
\(p\ge p_0\)
|
H.G3 | A1 = yes & B2 = yes & B4 = yes & C3 = yes & C4 = yes | A1 = yes & B2 = yes & B4 = yes & C3 = yes | C4 = yes |
\(p\ge p_0\)
|
In Table
4, “yes” summarizes the answer categories “yes, very” and “rather yes”. One exception is hypothesis H.O4 where only those subjects were considered that found the experience very exciting. All categories not mentioned in the table are unconstrained (
\(Xn=*\)). Furthermore, we did not employ category A4 for the measurement of interestingness, because during the evaluation it became clear that most participants were overtaxed with answering this question objectively.
Table 5
Counts, ratios and results of hypothesis testing
H.O1 |
\(102/107=0.953\)
|
\(182/196=0.929\)
| 0.887 | No |
H.O2 |
\(92/120=0.767\)
|
\(130/196=0.663\)
| 0.995 | Yes |
H.O3 |
\(13/167=0.078\)
|
\(15/196=0.077\)
| 0.516 | No |
H.O4 |
\(132/167=0.790\)
|
\(155/196=791\)
| 0.526 | No |
H.H1 |
\(28/30=0.933\)
|
\(189/196=0.959\)
| 0.350 | No |
H.H2 |
\(5/85=0.059\)
|
\(15/196=0.077\)
| 0.788 | No |
H.H3 |
\(105/114=0.921\)
|
\(182/196=0.929\)
| 0.715 | No |
H.H4 |
\(132/140=0.943\)
|
\(155/196=0.791\)
| 1.000 | Yes |
H.G1 |
\(52/55=0.945\)
|
\(182/196=0.929\)
| 0.762 | No |
H.G2 |
\(179/182=0.984\)
|
\(193/196=0.985\)
| 0.529 | No |
H.G3 |
\(24/24=1.000\)
|
\(193/196=0.985\)
| 1.000 | Yes |
Table
5 lists the results of the testing process. For
p and
\(p_0\), the numbers of hits (set sizes) are given as well as the ratios. The test statistics should be larger than
\(1-\alpha\) to express significance. Since the binomial test is not able to test on a particular
\(\alpha\), we base the discussion of the results on the test values.
The semantics of the results are discussed in the next section. Formally, H.H4 and H.G3 are the hypotheses with the highest significance. That is, spray is an important tactile stimulus and experienced subjects perceive smell and touch more strongly. For hypothesis H.G3, however, (as for H.H2 and H.O3) the numbers on which these results are based are rather low. Hence, conclusions can only be drawn cautiously. Most other hypotheses stand on a solid numerical basis. Some are expressive: H.H2/3, H.O1/2, H.G1. Hypothesis H.H1 does clearly not hold. The other hypotheses are neither here nor there: proof for their correctness could not be found.
4.3 Discussion
Below, we discuss the major results of the evaluation, starting with olfactory stimuli, followed by tactile stimuli and, eventually, general findings. The results of hypothesis testing are augmented with the descriptive statistics discussed in the Sect.
4.1.
4.3.1 The role of smell in virtual reality
Now, does the usage of smells in the Virtual Jumpcube increase the levels of immersion and excitement while mitigating potential nausea? Some results support our theory while no results seem to contradict it. The application of few olfactory stimuli carefully synchronized with the other media channels appears to have a positive impact on the quality of perception.
One fact that supports this conclusion with a significance level of 99.5% is hypothesis H.O2: olfactory stimuli with adequate intensity are rather perceived as pleasant independent of their actual smell. That is, if a smell fits the other dimensions of a multimedia event, participants rated it as adequate—thus making the virtual world more realistic and rich in details which should rather have a positive than a negative effect on the level of immersion.
A striking example for this type of judgment is the perception of the engines of flying objects. Though the smell is clearly unpleasant (it stinks) one third of the participants were willing to rate it as positive. Our explanation is that the smell is considered as augmenting the sensation of the bypassing plane or spaceship, thus contributing to the expressiveness of the virtual world.
Furthermore, the conclusion is directly supported by hypothesis H.O1 which reaches a significance level of 88.7%: subjects that perceive olfactory stimuli as synchronous with the virtual reality content reach a rather higher level of immersion. Since this result is based on more than one hundred participants, it must be considered a major indicator for the usefulness of smells in virtual reality. We found no indication that smells hinder immersion while often, they contribute significantly to reaching a high level of immersion.
Subjectively, 86% of the participants replied that smells enrich the virtual world. Of those subjects that perceived smells even 93% found smells an interesting media channel. There is a limitation in the fact that olfactory stimuli seem to have no particular effect on the levels of excitement in the Virtual Jumpcube: 98% of the smell-perceiving participants found the experience exciting, which is exactly the same result as for the total test group. Of course, at such a high level of excitement it is probably not realistic to expect a significant improvement from one new media channel. It would, therefore, be interesting to evaluate the Vragrancer system in a less exciting virtual reality application (e.g., a training situation). It appears not implausible to us that at lower base levels, olfactory stimuli might contribute to raising the level of excitement.
Finally, we could not identify a link of smell perception and nausea. This might be due to the low number of cases of (slight) nausea in the Virtual Jumpcube. Of the 15 subjects that declared motion sickness, 13 perceived smells and of these 79% found them pleasant which is an average very similar to the one of the entire test groups. All of these 13 participants consider smells an interesting media channel and all 13 experienced immersion.
In summary, smells increase the level of immersion in the virtual reality applications of the Jumpcube. They might bring excitement into sober business applications of virtual reality, but there is no indication that smells help to reduce the risk of nausea—which might be due to the mostly unpleasant nature of the smells used in the Jumpcube applications (explosions, smoke, plane engines, etc.)
4.3.2 The influence of tactile stimuli
The question whether haptic stimuli cause excitement and reduce nausea can be answered with yes. Only the level of immersion need not necessarily be increased by haptic stimulation.
A first result of the evaluation is that physical jumping is exciting (for 96% of the test group). Elderly subjects have slightly less fun at jumping but are in average as excited as the others after the jump. That justifies the effort for building the complex suspension system of the Virtual Jumpcube: free jumping is significantly more than just flying through a virtual environment.
This finding is supported by the results for the other applications of the suspension system. Of those participants who had a crash through the façade of a building in their content, 91% found it exciting. Turbulences were perceived by 66% of the participants and landing was exciting for 91% of all test subjects. All of these experiences are made possible by the suspension system.
The tactile killer feature is spray caused by water added to the wind. Hypothesis H.H4 reached a significance of 100%: spray makes clouds rather more exciting. Spray perceivers found in 97% of the cases flying through the clouds exciting, whereas only 43% of those who did not perceive the spray founds the clouds exciting. These results show clearly how important this simple tactile stimuli is. Properly synchronized, it turns a simple visual cue into an impressive multimedia event. After the jump, 99% of the spray-perceiving subjects found their skydive exciting compared to 86% of the non-perceiving subjects. The strong impact of spray might be explained by its tactile effect on the human skin. Spray causes a sensation similar to brushing. Through the recently discovered C tactile afferents [
15], these sensations seem to go directly into amygdala bypassing the hypothalamus, thus causing a guaranteed emotional reaction in the subject. Whatever the neurological explanation is, it is certain that in this case, haptic stimulation causes a strong improvement in virtual reality perception.
On the other hand, we could not identify a link of immersion and excitement for tactile stimulation. Independent of the level of excitement the level of immersion remains on the same level. Contrary to our expectation, excitement and immersion appear to be dimensions of virtual reality that are not almost perfectly correlated with each other. This question will be investigated in detail in the last part of this section.
Finally, hypothesis H.H2 with a significance level of 78.8% states that the perception of centrifugal forces will rather reduce the risk of nausea. H.H3 (71.5%) states that centrifugal forces in curves are experienced as realistic no matter if the content is realistic or not. Hence, we have a strong indication that the force simulation system has an effect on motion sickness. It appears to contribute to avoiding nausea. This is the case in realistic scenarios (skydive) but as well in fantastic ones such as the space flight and the air race. Hence, it appears reasonable to include force simulation into advanced virtual reality applications.
4.3.3 The influence of perception and experience on immersion
We assume that smell and touch support high levels of immersion if those media channels are in sync with the audiovisual virtual reality content. One fact that supports this idea is hypothesis H.G1 with test statistics of 76.2%: subjects that perceive olfactory and haptic stimuli more often reach a high level of immersion. That is, the ability to perceive these media channels will per se have a positive effect on the level of immersion.
Second, hypothesis H.G3 with 100% significance states that subjects with virtual reality experience react stronger on olfactory and haptic stimuli than subjects without experience. We conclude that the non-standard media channels might in particular be means to raise immersion and excitement for the experienced users that are no longer overwhelmed by audiovisual media cues. This finding appears to be of a general nature: in the test data we could not identify a gender or age bias.
Also linked to experience might be the trust in virtual reality applications that appear to be risky. Above we pointed out that 65% of the bravest jumpers hat prior virtual reality experience. Interestingly, 95% of these subjects experienced immersion compared to a global average of 92.6%—a slight improvement. Despite that only 95% of them are excited while 99% of the non-daring jumpers are. This fact is another illustration for the finding that immersion and excitement are not necessarily linked. In summary, experience could be a key factor in the development and testing of virtual reality applications. If experience influences the behavior and judgment of subjects, it has to be taken into account in virtual reality user studies.
In summary, it appears safe to state that immersion is not necessarily excitement. Overall, 65% of the participants reported very high immersion but 79% reported very high excitement, whereas air racers who experience higher levels of excitement (83%) have slightly lower levels of immersion (63%). These results may be explained by the physical activity required from the participant in the air race. It could be that the haptic feedback from physical exercise might hinder reaching a high level of immersion. This explanation appears to be supported by the findings on tactile stimuli above. For a definite answer though, a more detailed investigation would be required.