Skip to main content
SpringerLink
Log in

Improving object segmentation by using EEG signals and rapid serial visual presentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper extends our previous work on the potential of EEG-based brain computer interfaces to segment salient objects in images. The proposed system analyzes the Event Related Potentials (ERP) generated by the rapid serial visual presentation of windows on the image. The detection of the P300 signal allows estimating a saliency map of the image, which is used to seed a semi-supervised object segmentation algorithm. Thanks to the new contributions presented in this work, the average Jaccard index was improved from 0.47 to 0.66 when processed in our publicly available dataset of images, object masks and captured EEG signals. This work also studies alternative architectures to the original one, the impact of object occupation in each image window, and a more robust evaluation based on statistical analysis and a weighted F-score.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://imatge.upc.edu/web/resources/eeg-signals-object-segmentation

References

  1. Bauer G, Gerstenbrand F, Rumpl E (1979) Varieties of the locked-in syndrome. J Neurol 221(2):77–91

    Article  Google Scholar 

  2. Bell CJ, Shenoy P, Chalodhorn R, Rao R (2008) Control of a humanoid robot by a noninvasive brain computer interface in humans. J Neural Eng 16(5):432–441

    Google Scholar 

  3. Bigdely-Shamlo N, Vankov A, Ramirez R, Makeig S (2008) Brain activity-based image classification from rapid serial visual presentation. IEEE Trans Neural Syst Rehabil Eng 16(5):432–441

    Article  Google Scholar 

  4. Bradski G (2000) Dr. Dobb’s Journal of Software Tools

  5. Cruse D, Chennu S, Chatelle C, Bekinschtein TA, Fernández-Espejo D, Pickard JD, Laureys S, Owen AM (2012) Bedside detection of awareness in the vegetative state: a cohort study. Lancet 378(9809):2088–2094

    Article  Google Scholar 

  6. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338

    Article  Google Scholar 

  7. Fernandez-Canellas D (2013) Modeling the temporal dependency of brain responses to rapidly presented stimuli in erp based bci. Master’s thesis, Northeastern University

  8. Healy G, Smeaton A (2011) Eye fixation related potentials in a target search task. In: Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pp 4203–4206

  9. Healy G, Smeaton AF (2011) Optimising the number of channels in eeg-augmented image search. In: Proceedings of the 25th BCS conference on human-computer interaction, BCS-HCI, pp 157–162

  10. Hebbalaguppe R, McGuinness K, Kuklyte J, Healy G, Connor NO, Smeaton A (2013) How Interaction Methods Affect Image Segmentation : User Experience in the Task. In: Proc. The 1st IEEE workshop on user-centred computer vision (UCCV)

  11. Hu X, Li K, Han J, Hua X, Guo L, Liu T (2012) Bridging the semantic gap via functional brain imaging. IEEE Trans Multimed 14(2):314–325

    Article  Google Scholar 

  12. Huang Y, Erdogmus D, Pavel M, Mathan S, Hild II KE (2011) A framework for rapid visual image search using single-trial brain evoked responses. Neurocomputing 74(12-13):2041–2051

    Article  Google Scholar 

  13. Kapoor A, Shenoy P, Tan D (2008) Combining brain computer interfaces with vision for object categorization. In: Computer vision and pattern recognition (CVPR), pp 1–8

  14. Luck SJ (2005) An introduction to the event-related potential technique. MIT Press

  15. Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?. In: CVPR

  16. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol 2, pp 416–423

  17. Mohedano E, Healy G, McGuinness K, Giró-i Nieto X, O’Connor NE, Smeaton AF (2014) Object segmentation in images using eeg signals. In: Proceedings of the ACM international conference on multimedia, MM’14. ACM, New York, NY, USA, pp 417–426

  18. Motomura S, Ojima Y, Zhong N (2009) Eeg/erp meets act-r: A case study for investigating human computation mechanism. In: Zhong N, Li K, Lu S, Chen L (eds) Brain Informatics, volume 5819 of Lecture Notes in Computer Science, pp 63–73

  19. Pathirage I, Khokar K, Klay E, Alqasemi R, Dubey R (2013) A vision based p300 brain computer interface for grasping using a wheelchair-mounted robotic arm. In: 2013 IEEE/ASME international conference on advanced intelligent mechatronics (AIM), pp 188–193

  20. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine learning in Python. J Mach Learn Res 12:2825–2830

    MATH  MathSciNet  Google Scholar 

  21. Roark B, Oken B, M F-O, Orhan U, Erdogmus D (2013) Offline analysis of context contribution to erp-based typing bci performance. J Neural Eng 10(6):432–441

    Google Scholar 

  22. Rother C, Kolmogorov V, Blake A (2004) “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314

    Article  Google Scholar 

  23. Sajda P, Pohlmeyer E, Wang J, Parra LC, Christoforou C, Dmochowski J, Hanna B, Bahlmann C, Singh MK, Chang S-F (2010) In a blink of an eye and a switch of a transistor: cortically coupled computer vision. Proc IEEE 98(3):462–478

    Article  Google Scholar 

  24. Spence R (2002) Rapid, Serial and Visual: a presentation technique with potential. Inf Vis 1(1):13–19

    Article  Google Scholar 

  25. Wang J, Pohlmeyer E, Hanna B, Jiang Y-G, Sajda P, Chang S-F (2009) Brain state decoding for rapid image retrieval. In: Proceedings of the 17th ACM international conference on multimedia MM ’09, pp 945–954

  26. Yazdani A, Vesin J-M, Izzo D, Ampatzis C, Ebrahimi T (2010) Implicit retrieval of salient images using brain computer interface. In: ICIP, pp 3169–3172

Download references

Acknowledgments

This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289 and partially funded by the Project TEC2013-43935-R BigGraph of the Spanish Government.

Author information

Authors and Affiliations

  1. Insight Center for data Analytics, Dublin City University, Dublin, Ireland

    Eva Mohedano, Graham Healy, Kevin McGuinness, Noel E. O’Connor & Alan F. Smeaton

  2. Image Processing Group, Universitat Politcnica de Catalunya, Catalunya, Spain

    Xavier Giró-i-Nieto

Authors
  1. Eva Mohedano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Graham Healy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin McGuinness
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xavier Giró-i-Nieto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Noel E. O’Connor
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alan F. Smeaton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eva Mohedano.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohedano, E., Healy, G., McGuinness, K. et al. Improving object segmentation by using EEG signals and rapid serial visual presentation. Multimed Tools Appl 74, 10137–10159 (2015). https://doi.org/10.1007/s11042-015-2805-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2805-0

Keywords

Search

Navigation