Multisensory Softness

Softness can be physically expressed in different ways, for example as a stiffness value (spring constant), compliance value (the inverse of stiffness), or Young’s modulus. The relation between these quantities is discussed, as well as the psychophysical techniques for characterising the perception of these quantities. For perception of softness, multiple cues are available: visual cues include the deformation of the material around the fingers, while haptic cues include the ratio between applied force and resulting displacement of the material, and the force distribution over the contact area. Furthermore, to register these haptic cues, both cutaneous and kinaesthetic information can be used. The role of these different cues and types of information is discussed and also the interaction between the hand and the deformable material.

This chapter deals with the perception of compliance of objects with rigid surfaces when vision is present. Compliance (or its inverse, stiffness) is one of a number of properties that can be called “higher-order,” in the sense that it is computed as a combination of components that are physically independent. For objects having rigid surfaces, the components of compliance are position and force. We consider how each component is conveyed by different sensory modalities used for the interaction, vision and touch. This analysis highlights in particular that vision predominantly contributes to the sensing of position, whereas haptics (active touch) contributes to force sensing. We will further discuss integration of information across the senses; in particular, when such integration occurs in relation to the combination of the components of compliance. Finally, we describe applications of research on multi-modal compliance perception.

Soft or deformable objects, be they rubber ducks or running shoe inserts, are rarely thought of as sources of mechanical vibrations. For similar reasons, it is often overlooked that material softness can be communicated through the vibrotactile sensory channel—that is, through the subset of the haptic perceptual system that is sensitive to mechanical vibration. In this chapter we review current knowledge about the relation between vibrotactile sensation and softness perception. It is possible to distinguish between two main types of softness perception—one pertaining to the surface material qualities of a palpated object and the other linked to volumetric compliance. This information can be obtained through four types of interactions, which will be analysed separately: direct skin contact, indirect skin contact, transient contact, frictional sliding. We review contemporary research on softness perception in these four scenarios. This research has shed light on the perceptual salience of vibrotactile stimuli and on the action-phase dependence of vibrotactile cues for softness. We also highlight the importance of the physiological and mechanical aspects of the interactions for softness perception. In most cases, vibrotactile cues have a comparatively weaker influence on perception than the cues described in other chapters produced by directly manipulating a compliant object with deformable surfaces. Nonetheless, vibrations lead to an appreciable change on perceived compliance that can be exploited in addition to other cues or when such cues are not available.

The auditory perception of materials is a popular topic in the study of non-vocal sound-source perception. In this chapter, we review the empirical evidence on the mechanical and acoustical correlates of the perception of impacted stiff materials, and of the state of matter of sound-generating substances (solids, liquids, gases). As a whole, these studies suggest that recognition abilities are only highly accurate when differentiating between widely diverse materials (e.g. liquids vs. solids or plastics vs. metals) and that limitations in the auditory system, along with the possible internalization of biased statistics in the acoustical environment (e.g. clinking-glass sounds tend to be produced by small objects), might account for the less-than-perfect ability to differentiate between mechanically similar materials. This review is complemented by a summary of studies concerning the perception of deformable materials (fabrics and liquids) and the perceptual and motor-behaviour effects of auditory material-related information in audio-haptic contexts. The results of perceptual studies are the starting point for the development of interactive sound synthesis techniques for rendering the main auditory correlates of material properties, starting from physical models of the involved mechanical interactions. We review the recent literature dealing with contact sound synthesis in such fields as sonic interaction design and virtual reality. Special emphasis is given to softness/hardness correlates in impact sounds, associated with solid object resonances excited through impulsive contact. Synthesis methods for less studied sound-generating systems such as deformable objects (e.g. fabrics and liquids) and aggregate materials are also described.

We perceive compliance of deformable objects using several sources of sensory information obtained during the manual interaction. Some signals are inherently informative about how soft an object is. For example, softness of objects with deformable surfaces can be estimated directly from the pattern of skin deformation over time. On the other hand, other signals that are not inherently informative about compliance can be combined with other sensory signals. This is the case of force applied to the object and the amount of indentation that alone are not informative about softness, but combined they can provide an estimate of compliance. To obtain a unified sense of how soft the object is, the brain needs to appropriately combine all available information into one overall softness percept that accounts for the different contribution of all sensory signals, their time-course, and the precision of the information available. This chapter identifies some of the contributions of sensory information to softness perception and sketches the requirements of a computational model for their combination. This analysis is based on the redundancy and complementarity between the sources of information. Furthermore, it accounts for the dynamic aspects of the combination process by considering the integration of a priori knowledge, expectations, time-evolving sensory signals, and the movement strategies.

Perception during active touch essentially depends on the executed exploratory movements. Humans use different movement schemes to perceive different haptic properties, the so-called exploratory procedures (EPs). The stereotypically used EPs are normally superior to other EPs in perceiving the associated property and it has been speculated that the EPs are a means of maximising pickup of the relevant sensory information. However, EPs are not always executed identically as they vary in a number of ways. For instance, the peak force and the number of fingers used during exploration are not fixed. This chapter reviews existing findings on the exploratory movement strategies that humans use in softness perception and gives an overview on how different manners of exploration affect the performance in softness tasks. It is shown that observers adapt their movement strategies depending on variations of the stimulus value and the exact conditions of the exploratory task, and that different movement parameters, e.g. the peak exploratory forces, considerably affect performance. Overall, results suggest that humans adjust their exploratory strategies to achieve the highest levels of performance in softness discrimination.

Softness perception offers a good opportunity to study how position and force information are integrated during exploratory movements. This chapter presents a model of how this integration might happen when identifying the central position of a very weak elastic force field rendered with a haptic device. In this task, the force near the central position of the force field is below the perceptual threshold. The participant needs to explore the force field actively to identify its central position. The model predicts both the systematic and variable errors observed in this task. It also yields an estimate of the lowest force that can be sensed and suggests that the underlying processes are affected by at least two noise sources that reflect limits in our force and position sensing abilities. An implication of the existence of a force threshold for softness perception is that the perceived stiffness of very soft material might be overestimated.

Softness is an important source of information when interacting with real, remote, or virtual environments (VE) via a haptic human-machine-interface. Humans have no dedicated sense for perceiving softness; instead, inferring an object’s compliance haptically requires the combination of movement and force cues. A telepresence or VE system can alter an object’s mechanical impedance by artefacts such as time delay in the communication channel. Determining the limits for distortions caused by the technical system that do not affect the operator’s softness percept is crucial to ensure a realistic interaction experience. Characterising the perceptual system with a single performance measure such as the just noticeable difference (JND) neglects the role of active movement, which is known to influence perceptual performance. Overcoming this drawback, we propose the usage of dynamic models for haptic perception. On the example of an interaction with a soft virtual environment with time delay in the force feedback, we compare the prediction accuracy of different softness perception model candidates. Experimental data from three psychophysical experiments indicates that a dynamic state observer model captures the perceptual characteristics better than a time delay JND measure and an predictor base on an inverse model representation of the environment.

This book is focused on understanding how the human sensorimotor system integrates various sources of information to form a representation of stiffness—the linear relation between position and force. In this chapter, we will examine attempts to answer this question when users interact with artificially changed environment in which the force resulting from an interaction with the object is delayed, such as in the case of remote bilateral teleoperation.

In order to perform activities such as pressing keys on a keyboard, checking whether a piece of fruit is ripe, determining whether a bike tire is low on air, and shaking another person’s hand, it is necessary to have an understanding of the object’s compliance, or the relationship between one’s applied force and the resulting change in position of one’s hand.

In this chapter we describe a softness display based on the contact area spread rate (CASR) paradigm. This device uses a stretchable fabric as a substrate that can be touched by users, while contact area is directly measured via an optical system. By varying the stretching state of the fabric, different stiffness values can be conveyed to users. We describe a first technological implementation of the display and compare its performance in rendering various levels of stiffness with the one exhibited by a pneumatic CASR-based device. Psychophysical experiments are reported and discussed. Afterwards, we present a new technological implementation for the fabric-based display, with reduced dimensions and faster actuation, which enables rapid changes in the fabric stretching state. These changes are mandatory to properly track typical force/area curves of real materials. System performance in mimicking force-area curves obtained from real objects exhibits a high degree of reliability, also in eliciting overall discriminable levels of softness.

Haptic augmented reality is a new paradigm in human—computer interaction. Akin to traditional visual augmented reality, the technique strives to combine real and virtual sensory stimuli to alter perception during object manipulation. In the context of soft tissue interaction the stimuli felt during contact and indentation of a deformable object are overlaid with forces generated with a haptic device. Such a combined rendering can provide a user with an altered percept of object properties and/or shape. This chapter first outlines the general concept of integrating haptics into augmented reality. Thereafter, we will introduce two heuristic algorithms for haptic augmentation—covering stiffness modulation at either one or two contact points. This will address topics of parameter estimation, contact detection and augmentation computation. Finally, an application example is given in the context of tissue palpation. A method for augmenting virtual stiffer inclusions in physical soft tissue samples is presented.