5. Cross-Modal Signal Recovery

In order to handle issues about the lost, damaged, or delayed arrival of different modality signals after transmission, the corresponding signal recovery schemes should be designed. Firstly, for haptic-dominant multi-modal services, an audio-visual-aided haptic signal recovery scheme is proposed, while for audio-visual-dominant multi-modal services, a haptic-aided visual signal recovery scheme is presented. The core ideas are to utilize the one modality signal (audio-visual or haptic) to recover the other desired modality signal (haptic or visual) by leveraging their potential correlations. Then, for equally important multi-modal services, a visual-haptic mutual signal recovery scheme is provided, which can be seen as the general case for the two schemes mentioned above. Subsequently, besides these three representative cross-modal signal recovery solutions, information retrieval is another method with fast response and high-reliability characteristics. Under this circumstance, an information retrieval-based signal recovery scheme is designed. Finally, in order to enhance the display quality of the receiver and guarantee human experience, a cross-modal super-resolution reconstruction strategy is proposed, which achieves high-resolution visual signals from the received low-resolution visual signals and the related haptic signals.