Smell and touch in the Virtual Jumpcube

Two major design goals of virtual reality applications are the maximization of immersion and the avoidance of nausea (or motion sickness) in the users [3, 19]. The immersion in the virtual world can be seen as the degree to which we forget the outside world during the experience. In entertainment applications, the maximization of excitement is one further goal of virtual reality as a media system [6, 14]. Often, the same information could be conveyed over other media channels as well though virtual reality allows a more lively information process with potentially higher impact in the receivers. It appears to be obvious that immersion, nausea and excitement are linked phenomena: nausea destroys immersion and excitement, immersion increases excitement and vice versa. Consequently, the optimization of a virtual reality application is a multidimensional problem that requires careful design of the solution.

In this paper, we describe how additional media streams have been employed to maximize immersion, minimize nausea and maximize excitement in one particular virtual reality project, the Virtual Jumpcube [4, 20]. The media under consideration are olfactory and tactile¹ stimuli. The scientific contribution of the paper is the description of the stimulation hardware and software system for these non-standard media, extended testing with almost 200 subjects and a detailed analysis of the results. The underlying assumption is that smell and touch are efficient means to maximize immersion and excitement while minimizing motion sickness. We perceive virtual reality as a multimedia application where audiovisual channels are augmented by the olfactory and tactile stimulation streams and assume that—like for audiovisual media streams—precise synchronization is the key to the efficient usage of these media types [6, 14]. Virtual reality in general and the Virtual Jumpcube setup in particular appear highly suitable for the investigation of smell and touch since these environments are rather static in comparison to, for example, digital video that can be consumed almost anytime anywhere nowadays.

The present investigation is a first step in our endeavor to integrate non-audiovisual stimulations into hybrid virtual reality experiences. It is afflicted with several limitations that are in their majority caused by the business model of this research (see Sect. 3.2 for details). First, the subjects that use the Virtual Jumpcube are guests of some event (e.g., a congress) that pays for the participation of the cube. Hence, we cannot switch off certain stimuli for some of the participants thus creating a comparison group. Under these circumstances, the questionnaires handed out to the subjects have to be relatively short and the usage of technology has to be conservative: every component employed has to be well tested. Furthermore, the present setup was of course not made for the evaluation of nausea but the opposite: we endeavored to include as many promising solutions science offers for the avoidance of motion sickness as possible. Likewise, all audiovisual contents presented in the Virtual Jumpcube are designed for maximal excitement. Therefore, the level for the new olfactory-tactile stimuli is already rather high. There are, however, also mitigating factors and benefits from the application scenario. For example, since a considerable part of the population suffer from partial or total anosmia (up to 20% according to [12]) these subjects can be employed as a comparison group for olfactory stimulation. Moreover, the paramount advantage of this form of testing is that the results reflect the true situation of virtual reality and its defining parameters immersion, nausea and excitement: every subject used the Virtual Jumpcube out of her/his own curiosity in a real-world environment, thus guaranteeing the practical representativeness of the results.

The Virtual Jumpcube setup was first described in [4]. In brief, it is a complex system of hardware components that allows one subject to perform a free jump, flight and landing sequence in the real world while experiencing a virtual world through virtual reality gear. Figure 1 shows the cube in action at the Hannover industrial fair. The development of the majority of the hardware components was necessary for making sure that the unique aspect of the cube—free full-force jumping forward—can be performed without the risk of injury. This aspect also constitutes the unique selling factor of the setup and was approved by almost two thousand jumpers since April 2015: jumping is fun for the acting person but as well for the watching audience, an aspect that made this research prototype attractive for the presentation at tech-shows and related events.

Below, we describe how the Virtual Jumpcube was extended by hardware and software for the presentation of olfactory and tactile stimuli (Sect. 2). Then, in Sect. 3, we specify the experiment that was performed on the setup, explaining all relevant parameters and constraints. At the heart of Sect. 3 are eleven hypotheses that are evaluated on the basis of the results presented in Sect. 4. Next to hypothesis testing, we also perform a general evaluation of the results based on descriptive statistics and conclude on the results in the last section of the paper where we also give an outlook on future scientific investigations in the context of the Virtual Jumpcube project. But first we sketch the present situation in virtual reality smell and touch research in the last section of the introduction.

The Virtual Jumpcube is an example for what is sometimes called the interface challenge in virtual reality research [3, 19]. Many more creative, complex, intriguing hardware prototypes have been built in the past decades. Recently, two further flying (though not jumping) setups have gained attention in the region of the author: Birdly [17] is a well-known system for bird-like flight through virtual worlds. In addition to audiovisual media streams it contains two major haptic channels: first, the user has to flap his/her wings to build up speed, and second, wind is blown into the face of the user. Another flight application is Icaros [10]. Icaros is—in comparison to Birdly and the Virtual Jumpcube—a rather simple hardware setup that allows the user to balance herself and thus experience a flight-like virtual environment. Birdly and Icaros are two beautiful examples for virtual reality flight gear. We assume that further sophisticated flight applications do exist though, to the knowledge of the author, none of these systems would allow the user to perform a free jump, which is a unique experience in itself.

Next to flight applications, a considerable number of walking, swimming and other movement applications were proposed in recent years. We would like to point out [5] for virtual swimming, [13] for gravity reduced stepping and [1] for walking, running and vertical jumping in static environments. Since movement is essential in these setups they must be considered as integrating to a certain degree haptic aspects. Yet to the author’s knowledge they do not explicitly provide tactile stimuli of the skin nor olfactory stimulation. Smells in particular have seen only little application in virtual reality recently. Since the spectacular failures of Smell-o-Vision and Aroma-Rama in the 1960s [3], only little has been done to augment multimedia and virtual reality systems by such stimuli. One notable exception are the works by the group of Alan Chalmers at the University of Warwick [2, 16]. In the area of touch, the situation is significantly more diverse [11]. Numerous systems have been proposed for skin stimulation through gloves, vibration motors, heat and cold. Setups have been built for force simulation and feedback (e.g., [13]) as well as for the encoding of cognitive information in rhythm patterns (for example, the well-known HTC Vive archery game).

Below, where necessary we refer to (recent) discoveries in neuroscience and psychology on the operating mode of our sensor systems for olfactory and tactile perception. This knowledge sources mostly from [12] and—where psychophysical aspects are concerned—from [8]. Knowledge on touch perception, in particular large-field stimulation of the skin, were taken from [7, 15]. For example, we employ the fascinating findings of neuroscience for the design of appropriate olfactory stimulation streams (see Sect. 2.3) that endeavor to contribute to the overall goal of maximal immersion as efficiently as possible.

The Virtual Jumpcube [20] was built as a demonstration object for the potentials of virtual reality in the context of the 200th anniversary of TU Wien in the winter 2014/2015. In [4], we give an introduction of the original system comprising of the frame, suspension system and virtual reality gear and software. Since then, the system has been extended by modules for olfactory and haptic stimulation as well as various contents and scenarios (originally, skydiving, now as well diving, space travel and air racing). Still, the fundamental goal remains the same: it is providing an infrastructure for free jumping and flying in virtual environments. The suspension is based on a system of ropes, pulleys and counterweights balanced by eccentric discs that map the non-linear force curve of the jump onto the static counterweights.

Free jumping while being locked into a virtual environment constitutes a loss of control situation that should influence the perception of the virtual content—hopefully making it more exciting. Major aspects of the installation are the usage of rich media streams, demonstratively visible safety and portability/modularity of the entire setup. Rich media usage means that while the audio tracks of some content contain as well sound effects, background music and—where necessary—the voice of an instructor, the visual layer is always composed of as complex as possible 3D models, special effects and a layer for game play thus combating the fundamental disadvantage of virtual reality (in comparison to 3D movies—the other 3D media type familiar to most users) that the scene has to be rendered ad hoc.

Since the frame of the cube is relatively big (320 cm in each dimension), it can accommodate the hardware modules for new stimuli comfortably. We therefore found it a suitable platform for evaluating the above-mentioned ideas that smell and touch might increase immersion and excitement while reducing the risk of nausea. The only difficulty we encountered was the fact that we could not find any readymades for the stimulations we had in mind. Hence, we had to construct our own prototypes. They are described in the Sects. 2.3 and 2.4. Before that the next section introduces the general architecture of the Jumpcube setup.

The Jumpcube system consists of input, processing and output components. Figure 2 illustrates the interaction of the components. The central element is the application controller which manages input and output components as well as the virtual reality applications. The application controller exists in two implementations: one is based on the Unity game engine, the other on Unreal. Both consume content in the form of 3D models from the attached content database. Sensory input is acquired from connected sensors, in particular the hand-helds of the employed HTC Vive head-mounted display and a self-developed acceleration-based motion sensor for flying applications. Audiovisual output is propagated from the application controller to the head-mounted display and connected headphones.

All other output components—those that provide the accompanying touch and smell stimuli—comprise of a production unit and a managing controller. Production units include fans, heaters, a cooling device, a smell propagation system and a motor for force simulation. All managing controllers were developed during the project. Each consists of an embedded processor (mostly, Raspberry Pi and Arduino boards) and individual control software with a common software interface. The common interface is based on a self-defined control protocol. The protocol defines a simple proprietary JSON dialect for controlling production units (start/stop) and the setting of parameters (attributes and ranges of values). Messages are propagated over a local area network through a socket interface. Furthermore, for calibration and configuration purposes, we provide a browser-based console.

The system architecture includes one notable exception. The suspension system is independent from all other components and purely mechanical. For users with a weight of ca. 30–150 kg and a height of 130–196 cm, it requires no configuration nor calibration. Being made from high-quality sailing and mountaineering equipment, it provides suspension without any notable time delay and the decoupling from the rest of the system guarantees the meeting of legal safety requirements.

The development of the system architecture progressed in an evolutionary bottom-up design process in the following steps: first, we defined the component architecture of dedicated controllers for specific stimulus production units. In the second step, we formulated a TCP-based protocol that is sufficiently general for managing a variety of different production units. This protocol was implemented in a software library that is accessible from both considered game engines and provides a web-based console interface. Based on production, controller and network hardware the two application controllers were implemented next. In this process, particular emphasis was given to the important task of 3D content management. Finally, the entire system was iteratively refined during the development of the three first apps for skydiving, space travel (both unity-based) and air racing (unreal-based).

The present system architecture proofed successful in its ability to integrate new stimulation components. Controllers make use of the implemented control protocol. The present software library can be accessed from most state-of-the-art embedded processors and boards. Since controllers encapsulate the production units, almost any thinkable stimulation hardware can easily be integrated into the environment by just plugging the controller into the local network and registering it via the console. The next two sections describe the most important of these components for smell and touch stimulation in the Virtual Jumpcube.

Figure 3 depicts the various elements of the Vragrancer, the smell provider in the Virtual Jumpcube. It is based on a product named smell controller developed by a German company in the 1990s for usage in smell cinemas. Yet it was never launched and we only found it by coincidence. The smell controller is loaded with cartridges of six smells, each encapsulated in a sealed vial where the liquid smell is held by crystals. On request, the developing company (now operating in the marketing industry) provided us with a stock of smell cartridges for three virtual experiences: skydiving, space travel and air racing. Smells include combustion engine, smoke, coffee, and flowers. All smells are synthetic, sometimes not particularly pleasant yet thematically recognizable. Most importantly, they are approved as harmless by European authorities.

For usage of the smell controller in our system, we built a controller, based on a Raspberry Pi microprocessor and an Arduino for valve control. Communication between game engine and controller is performed over a TCP socket based on the before-mentioned JSON protocol. Smells are propagated by sending compressed air through valves and then over pipes (bundled with the cables of the head-mounted display) to the nose where an opening in the pipe delivers the smell-enriched air directly under the nose. The entire system with a pressure of just one Bar and a pipe length of approximately five meters has a surprisingly low latency of about 100 ms. Embedded in a compact, mobile case the Vragrancer system can be configured with arbitrary smell combinations and can be employed in arbitrary (semi-)static virtual reality applications.

Essential to the Vragrancer is the usage pattern. From the above-mentioned failure of Aroma-Rama and related projects—where smells were used constantly as themes to encode situations and actors—we derived the conclusion that smell is a stimulus essentially different from the audiovisual ones. While we are used to holistic permanent application of the latter stimulus type, a constant stream of smells might overload the consumer. Rather, smells should be used selectively and only in situations where they are well-motivated and recognizable. For such application cartridges of six smells each appeared sufficient for the five minutes experiences of the Jumpcube.

Three haptic modules were employed in the present evaluation: the jump suspension system, the weather system and a force simulation system. They are described below. A fourth haptic component available in the Jumpcube, the vibration system that effects on hands and hips, is not described as it was not part of the evaluation. Such vibration patterns are, for example, employed for making explosions more realistic and for simulating water resistance in a reef diving application.

Table 1 lists the hypotheses that were selected from a pool of ideas for the evaluation of the impact of smell and touch on immersion, excitement and motion sickness in virtual reality applications. Four hypotheses refer to olfactory stimuli (H.O1–4), four to haptic stimuli (H.H1–4) and three concern both types of stimuli. Of course, with the data available other hypotheses could be formulated and tested. We chose this set out of representativeness, clarity and compactness considerations. For further analysis, the curious reader may download the raw data of the evaluation from [9].

The underlying assumption of the hypotheses is that smell and touch improve immersion and excitement while reducing the effect of nausea. The majority of the statements expresses this belief. Some additional hypotheses (e.g. H.H1) were included for clarifying the effect of certain characteristics of the test subjects on the generality of the results. Please note that the hypotheses are formulated independent of the employed audiovisual content. In the next section, we will outline that three different types of audiovisual content were employed in the present experiment. Since our interest is to derive general conclusions on the usage of touch and smell stimuli, these content variants employ exactly the same types of olfactory and haptic stimuli with exactly the same parametrization (though at different locations of the content, of course). In consequence, the results should as far as possible be independent of the employed content semantics.

Below, in addition to hypothesis testing, we provide descriptive statistics of the evaluation results—one the one hand, for broadening the picture drawn by the evaluation, and on the other hand, for deepening the discussion of the results of hypothesis testing.

This section sketches the test environment briefly, gives an idea of the 3D contents used in the evaluation, and finally, discusses the questionnaire we formulated for providing the data basis of the hypothesis testing process.

Three paid events were chosen for the present evaluation of smell and touch in the Virtual Jumpcube:

Table 3 lists statistical information on the participants of the evaluation. Altogether 196 subjects participated. At ESR17, 84 subjects filled out the questionnaire after the jump. Of these 42% were females and 86% were between 19 and 49 years old. Surprisingly, only 36% declared previous virtual reality experience though the expo part of the congress showed a significant number of imaging systems that use virtual reality both in 2016 and 2017. Only 7% of the participants rated their sense of smell as “bad”. The NextM statistics are substantially different. Here, hardly any elderly people participated and almost two thirds of the participants were woman. An outstanding 55% of the participants had already had virtual reality experience and a surprisingly small number of just 3% declared (partial) anosmia. If this were true it would be significantly below the estimated average of the population of 10–15% [12]. Eventually, the VCM had a more diverse audience. Only 61% were neither youngsters nor elderly citizens, 38% had virtual reality experience and 10% admitted a bad sense of smell. In summary, the vast majority of the participants were between 19 and 49 years old, the genders were almost balanced, 40% had experience with the technology and approximately 92% declared themselves as able to perceive smells.

Figure 8 illustrates the statistics over all events, genders and age groups. The indicators Xn refer to the questions n of sections X of the questionnaire shown in Table 2. Answers are encoded as cardinal numbers 0–3 where 0 signifies “no answer”, 1 the first option, and so on.

The statistics on participant behavior show that most jumpers prefer a conservative jump style at their first jump, which is not surprising. Of the 10% daring jumpers, 65% were male and only 5% were youngsters compared to almost 8% in the total sample: young subjects were more risk-averse than elder subjects. It appears noteworthy that 65% of the daring jumpers had prior virtual reality experience compared to only 40% in the entire sample. That seems to imply that users with virtual reality experience trusted the installation more than users without that knowledge—though the risk of injury is not related to the virtual reality part of the installation. Concerning age vs. experience, 47% of the young participants had prior experience compared to 40% in the adults and 33% in the elder subjects; 32% of the female participants had virtual reality experience compared to 46% of the males.

Content statistics: the preferred content was the skydiving application (58% compared to 27 and 15% for space flight and air race, respectively). Of the female participants, even 63% of the subjects chose the skydiving application. Only 6% of the skydivers showed a daring jump, whereas 26% of the air racers performed an extraordinary dive. The correlation is understandable since the air race was presented to the clients as potentially causing nausea, and therefore, mostly chosen by risk-taking individuals. Over all contents, 65% of the participants reported very high immersion and 79% very high excitement. The values for the skydivers lie exactly in the global average. Interestingly, though air racers experience even higher levels of excitement (83%) their chance of very high immersion is slightly lower than the average (63%). It might be concluded that excitement is not just a function of immersion but might also be triggered without perfect virtuality. Finally, the average risk of experiencing nausea was only 8%. For skydiving, it was even lower (5%) and for the air race it was 17%. Hence, for the evaluation of the hypotheses on motion sickness, we mostly depend on the 30 subjects that chose the air race content.

Next, we investigated the general feedback on the Vragrancer. 85% of the participants perceived the smells present in the multimedia content. The non-perceiving 15% are twice the number of subjects that declared (partial) anosmia. After filling out the questionnaire some participants asked whether smells had actually been present and were surprised to hear that this had permanently been the case. Independent of the perception of smells, 86% of the participants replied that smells enrich the virtual reality. Of those subjects that perceived smells even 93% consider olfactory stimuli an interesting enrichment. This positive result appears to justify our approach to use smells only for selected events, thus causing a higher perceptual effect than through constant application.

Of the smell-perceiving participants, 32% considered the smell of bypassing flying objects (a synthetic combustion engine smell used for airplanes and space ships) as positive—though it is fair to say that objectively this smell is not pleasant. Even so, the combustion scent seems to emphasize the multimedia event “flying object” which causes a subjective rating of the smell that differs significantly from the objective judgment. In total, 63% of the smell-perceiving subjects found the scents very well synchronized with their multimedia events. Only 7% could not match them. 55% of the subjects found the smells adequate for their multimedia events which shows that synthetic smells in combination with other channels are able to form a complex, rich-in-detail media object.

The last group of descriptive statistics concerns the touch stimuli employed in the Jumpcube. For example, together with visual stimuli the suspension system was employed to simulate turbulences and crashing through the window of a building. Whereas 96% of the participants found the actual jumping exciting, 91% found the landing (with turbulences) exciting. Of those participants who had a crash through the window in their content, again 91% found it exciting. There appears to be no gender aspect in experienced excitement: though in situ it was expressed very differently by members of the two genders, the feedback is about the same for both.

An interesting result concerns clouds and spray—only present in the skydiving application. Of those participants that perceived spray (94%) 97% found the flight through the spray-enriched clouds exciting. Of the others, only 43% found the clouds exciting and 29% did not even notice them (compared to just 2% of the spray-perceiving subjects). These results show clearly how important a simple haptic stimuli can be for raising content awareness. 99% of the spray-perceiving subjects found their virtual skydive exciting, compared to only 86% of those who did not perceive spray.

The presented findings are only a small sample from the patterns present in the data. A long sequence of other examples could be given. Still, after drawing a sketch of the results it appears beneficial to move on to the formal analysis of the data.

Due to the nature of the answer categories in the questionnaire, the hypotheses had to be tested with binomial tests. For the hypotheses listed in Table 1, Table 4 lists the subsets that were used to calculate the test variable p and the default value \(p_0\) of the binomial tests. Again, Xn refers to question n of section X of the questionnaire in Table 2. The denominator of \(p_0\) is not given as it is always the total of the participants (\(n=196\)). The “rule” column refers to the testing method: two-sided (\(p=p_0\)) or one-sided—depending on the formulation of the hypothesis.

In Table 4, “yes” summarizes the answer categories “yes, very” and “rather yes”. One exception is hypothesis H.O4 where only those subjects were considered that found the experience very exciting. All categories not mentioned in the table are unconstrained (\(Xn=*\)). Furthermore, we did not employ category A4 for the measurement of interestingness, because during the evaluation it became clear that most participants were overtaxed with answering this question objectively.

Table 5 lists the results of the testing process. For p and \(p_0\), the numbers of hits (set sizes) are given as well as the ratios. The test statistics should be larger than \(1-\alpha\) to express significance. Since the binomial test is not able to test on a particular \(\alpha\), we base the discussion of the results on the test values.

The semantics of the results are discussed in the next section. Formally, H.H4 and H.G3 are the hypotheses with the highest significance. That is, spray is an important tactile stimulus and experienced subjects perceive smell and touch more strongly. For hypothesis H.G3, however, (as for H.H2 and H.O3) the numbers on which these results are based are rather low. Hence, conclusions can only be drawn cautiously. Most other hypotheses stand on a solid numerical basis. Some are expressive: H.H2/3, H.O1/2, H.G1. Hypothesis H.H1 does clearly not hold. The other hypotheses are neither here nor there: proof for their correctness could not be found.

Below, we discuss the major results of the evaluation, starting with olfactory stimuli, followed by tactile stimuli and, eventually, general findings. The results of hypothesis testing are augmented with the descriptive statistics discussed in the Sect. 4.1.