Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Efficient Lightweight Network with Transformer-Based Distillation for Micro-crack Detection of Solar Cells Read chapter Read first chapter

2024 | OriginalPaper | Chapter

RoBrain: Towards Robust Brain-to-Image Reconstruction via Cross-Domain Contrastive Learning

Authors : Che Liu, Changde Du, Huiguang He

Published in: Neural Information Processing

Publisher: Springer Nature Singapore

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

With the development of neuroimaging technology and deep learning methods, neural decoding with functional Magnetic Resonance Imaging (fMRI) of human brain has attracted more and more attention. Neural reconstruction task, which intends to reconstruct stimulus images from fMRI, is one of the most challenging tasks in neural decoding. Due to the instability of neural signals, trials of fMRI collected under the same stimulus prove to be very different, which leads to the poor robustness and generalization ability of the existing models. In this work, we propose a robust brain-to-image model based on cross-domain contrastive learning. With deep neural network (DNN) features as paradigms, our model can extract features of stimulus stably and generate reconstructed images via DCGAN. Experiments on the benchmark Deep Image Reconstruction dataset show that our method can enhance the robustness of reconstruction significantly.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter A Causality-Based Interpretable Cognitive Diagnosis Model
next chapter High-Dimensional Multi-objective PSO Based on Radial Projection
Footnotes
Literature
1.
Bandettini, P.A.: Twenty years of functional MRI: the science and the stories. Neuroimage 62(2), 575–588 (2012)CrossRef
2.
Beauchamp, M.S., et al.: Dynamic stimulation of visual cortex produces form vision in sighted and blind humans. Cell 181(4), 774–783 (2020)CrossRef
3.
Beliy, R., Gaziv, G., Hoogi, A., Strappini, F., Golan, T., Irani, M.: From voxels to pixels and back: self-supervision in natural-image reconstruction from fMRI. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
4.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
5.
Fang, T., Qi, Y., Pan, G.: Reconstructing perceptive images from brain activity by shape-semantic GAN. Adv. Neural. Inf. Process. Syst. 33, 13038–13048 (2020)
6.
Fernández, E., et al.: Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. J. Clin. Invest. 131(23) (2021)
7.
Güçlütürk, Y., Güçlü, U., Seeliger, K., Bosch, S., van Lier, R., van Gerven, M.A.: Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
8.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
9.
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
10.
Mozafari, M., Reddy, L., VanRullen, R.: Reconstructing natural scenes from fMRI patterns using BigBiGAN. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
11.
Naselaris, T., Kay, K.N., Nishimoto, S., Gallant, J.L.: Encoding and decoding in fMRI. Neuroimage 56(2), 400–410 (2011)CrossRef
12.
Nestor, A., Lee, A.C., Plaut, D.C., Behrmann, M.: The face of image reconstruction: progress, pitfalls, prospects. Trends Cogn. Sci. 24(9), 747–759 (2020)CrossRef
13.
Poldrack, R.A., Farah, M.J.: Progress and challenges in probing the human brain. Nature 526(7573), 371–379 (2015)CrossRef
14.
Pulvermüller, F., Tomasello, R., Henningsen-Schomers, M.R., Wennekers, T.: Biological constraints on neural network models of cognitive function. Nat. Rev. Neurosci. 22(8), 488–502 (2021)CrossRef
15.
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:​1511.​06434 (2015)
16.
Rakhimberdina, Z., Jodelet, Q., Liu, X., Murata, T.: Natural image reconstruction from fMRI using deep learning: a survey. Front. Neurosci. 15, 795488 (2021)CrossRef
17.
Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., van Gerven, M.A.: Generative adversarial networks for reconstructing natural images from brain activity. Neuroimage 181, 775–785 (2018)CrossRef
18.
Shen, G., Dwivedi, K., Majima, K., Horikawa, T., Kamitani, Y.: End-to-end deep image reconstruction from human brain activity. Frontiers in Computational Neuroscience 13, 21 (2019)CrossRef
19.
Shen, G., Horikawa, T., Majima, K., Kamitani, Y.: Deep image reconstruction from human brain activity. PLoS Comput. Biol. 15(1), e1006633 (2019)CrossRef
20.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
21.
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRef
22.
Ye, H., Yang, X., Takac, M., Sunderraman, R., Ji, S.: Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:​2107.​02423 (2021)
23.
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Metadata
Title
RoBrain: Towards Robust Brain-to-Image Reconstruction via Cross-Domain Contrastive Learning
Authors
Che Liu
Changde Du
Huiguang He
Copyright Year
2024
Publisher
Springer Nature Singapore
Book
Neural Information Processing

Print ISBN: 978-981-9980-66-6

Electronic ISBN: 978-981-9980-67-3

Copyright Year: 2024

DOI
https://doi.org/10.1007/978-981-99-8067-3_17

Premium Partner

    Image Credits