NeuroGAN: image reconstruction from EEG signals via an attention-based GAN

In this work, we propose an approach to process electroencephalogram (EEG) for a visual perception task for synthesizing the visual stimulus that was shown during the acquisition of EEG (images of objects, digits, and characters). We demonstrate that a cross-modality-based encoder–decoder network shows good performance for image synthesis tasks on simplistic images like digits and characters but fails on complex natural object images. To address this issue, we propose a novel attention & auxiliary classifier-based GAN architecture where the generator itself is a cross-modality-based encoder–decoder network. It generates images along with producing class-specific EEG encoding as a latent representation. In addition to the traditional adversarial loss, we also propose to use perceptual loss and attention modules to generate good-quality images. The performance of the proposed network is measured using two metrics— diversity score and inception score, which quantify the relevance and quality of the reconstructed images, respectively. Experimentation results show that our approach performs better compared to the state-of-the-art for both metrics.