3. Cross-Modal Coding

This chapter delves into the innovative realm of cross-modal coding, focusing on techniques to enhance the efficiency of multi-modal data transmission. It explores two primary approaches: redundancy elimination and coding schemes. The first approach involves visual-haptic mutual redundancy elimination and audio-haptic joint redundancy elimination, which aim to compress the content of one modality's signals with the aid of others, leveraging correlations among modalities. The second approach includes visual-aided haptic coding and visual-haptic mutual coding, which support both signal compression at the sender and signal reconstruction at the receiver. The chapter defines essential variables and derives important theorems using Shannon's information theory, emphasizing semantic correlations among modalities. Practical implementations, such as a cross-modal codec and a general cross-modal codec, are designed and evaluated through experimental results. The findings demonstrate significant improvements in coding efficiency, latency reduction, and reliability enhancement, highlighting the potential of these techniques for various applications beyond visual and haptic signals.