Speech2Image: Generating Images from Speech Using Pix2Pix Model

Generating images from speech is a fundamental problem that has numerous applications which include art generation, computer-aided design, enhancing learning capabilities in children among others. We present an audio-conditioned image generation model that transfers features from the speech descriptions (source) to respective image (target) domain. To accomplish this, we have used the Caltech-UCSD Birds-200-2011 (CUB-200-2011) dataset as target images and then we generated high-quality speech descriptions of birds to prepare custom dataset for training. We further verified the generated images using the images of birds provided in the CUB-200-2011 dataset. The model is trained and tested on three types of speech representations, i.e Spectrograms, ConstantQ Transforms and Short Time Fourier Transforms the results of which are discussed in subsequent sections. Unlike conventional approaches for speech to image conversions, which rely on a different intermediary domain such as text to realize this transition, our novel approach relies on an intermediate image transition, effectively restricting the number of domains involved in this process.