ABSTRACT
Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity.
- I. Arapakis, K. Athanasakos, and J. M. Jose. A comparison of general vs personalised affective models for the prediction of topical relevance. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, SIGIR '10, pages 371--378, New York, NY, USA, 2010. ACM. Google ScholarDigital Library
- I. Arapakis, I. Konstas, and J. M. Jose. Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In Proceedings of the 17th ACM International Conference on Multimedia, MM '09, pages 461--470, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- H. Berger. Über das Elektrenkephalogramm des Menschen. Archiv für Psychiatrie und Nervenkrankheiten, 87(1):527--570, 1929.Google ScholarCross Ref
- F. Crick and C. Koch. A framework for consciousness. Nature Neuroscience, 6:119--126, 2003.Google ScholarCross Ref
- L. Farwell and E. Donchin. Talking off the top of your head: Toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology, 70(6):510--523, 1988.Google ScholarCross Ref
- B. Figner and R. O. Murphy. Using Skin Conductance in Judgment and Decision Making Research. A Handbook of process tracing methods for decision research: A critical review and user's guide, pages 163--184, 2010.Google Scholar
- J. R. Folstein and C. Van Petten. In uence of cognitive control and mismatch on the N2 component of the ERP: A review. Psychophysiology, 45(1):152--170, 2008.Google Scholar
- G. Gomez-Herrero, W. De Clercq, H. Anwar, O. Kara, K. Egiazarian, S. Van Huffel, and W. Van Paesschen. Automatic removal of ocular artifacts in the EEG without an EOG reference channel. In Signal Processing Symposium, 2006. NORSIG 2006. Proceedings of the 7th Nordic, pages 130--133, 2006.Google ScholarCross Ref
- M. Gönen. Bayesian efficient multiple kernel learning. In J. Langford and J. Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML-12) , pages 1--8, New York, NY, USA, 2012. ACM.Google Scholar
- C. J. Gonsalvez and J. Polich. P300 amplitude is determined by target-to-target interval. Psychophysiology, 39(3):388--396, 2002.Google ScholarCross Ref
- A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13:723--773, 2012. Google ScholarDigital Library
- T. Gruber, M. M. Müller, and A. Keil. Modulation of induced gamma band responses in a perceptual learning task in the human EEG. Journal of Cognitive Neuroscience, 14(5):732--744, July 2002. Google ScholarDigital Library
- T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR '05, pages 154--161, New York, NY, USA, 2005. ACM. Google ScholarDigital Library
- D. Kelly and N. J. Belkin. Display time as implicit feedback: Understanding task effects. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'04, pages 377--384, New York, NY, USA, 2004. ACM. Google ScholarDigital Library
- W. Klimesch. EEG alpha and theta oscillations re ect cognitive and memory performance: a review and analysis. Brain Research Reviews, 29(2,3):169--195, 1999.Google ScholarCross Ref
- J. Koenemann and N. J. Belkin. A case for interaction: A study of interactive information retrieval behavior and effectiveness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '96, pages 205--212, New York, NY, USA, 1996. ACM. Google ScholarDigital Library
- Z. Koldovsky, P. Tichavsky, and E. Oja. Efficient variant of algorithm FastICA for independent component analysis attaining the Cram--er-Rao lower bound. Neural Networks, IEEE Transactions on, 17(5):1265--1277, 2006. Google ScholarDigital Library
- A. Lagioia, S. Eliez, M. Schneider, J. S. Simons, M. Van der Linden, and M. Debban. Neural correlates of reality monitoring during adolescence. NeuroImage, 55(3):1393--1400, 2011.Google ScholarCross Ref
- F. Lotte, M. Congedo, A. L--ecuyer, F. Lamarche, and B. Arnaldi. A Review of Classification Algorithms for EEG-based Brain-Computer Interfaces. Journal of Neural Engineering, 4(2), June 2007.Google ScholarCross Ref
- S. J. Luck. An Introduction to the Event-Related Potential Technique (Cognitive Neuroscience). MIT Press, 1 edition, 2005.Google Scholar
- D. J. McFarland, L. A. Miner, T. M. Vaughan, and J. R. Wolpaw. Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topography, 12(3):177--186, 2000.Google ScholarCross Ref
- P. Missonnier, M.-P. Deiber, G. Gold, P. Millet, M. Gex-Fabry Pun, L. Fazio-Costa, P. Giannakopoulos, and V. Ibãnez. Frontal theta event-related synchronization: Comparison of directed attention and working memory load effects. Journal of Neural Transmission, 113(10):1477--1486, 2006.Google ScholarCross Ref
- T. M. Mitchell, S. V. Shinkareva, A. Carlson, K. M. Chang, V. L. Malave, R. A. Mason, and M. A. Just. Predicting human brain activity associated with the meanings of nouns. Science, 320(4880):1191--1195, 2013.Google Scholar
- Y. Moshfeghi and J. M. Jose. An effective implicit relevance feedback technique using affective, physiological and behavioural features. In\ Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval , SIGIR '13, pages 133--142, New York, NY, USA, 2013. ACM. Google ScholarDigital Library
- Y. Moshfeghi, L. R. Pinto, F. E. Pollick, and J. M. Jose. Understanding Relevance: An fMRI Study. In P. Serdyukov, P. Braslavski, S. Kuznetsov, J. Kamps, S. Rüger, E. Agichtein, I. Segalovich, and E. Yilmaz, editors, Advances in Information Retrieval, volume 7814 of Lecture Notes in Computer Science, pages 14--25. Springer Berlin Heidelberg, 2013. Google ScholarDigital Library
- T. E. Nichols and A. P. Holmes. Nonparametric permutation tests for functional neuroimaging: A primer with examples. Human Brain Mapping, 15(1):1--25, 2002.Google ScholarCross Ref
- M. Pantic and L. J. M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Transactions in Pattern Analysis Machine Intelligence, 22(12):1424--1445, Dec. 2000. Google ScholarDigital Library
- M. Pantic and L. J. M. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370--1390, 2003.Google ScholarCross Ref
- R. D. Pascual-Marqui, C. M. Michel, and D. Lehmann. Low resolution electromagnetic tomography: A new method for localizing electrical activity in the brain. International Journal of Psychophysiology, 18(1):49--65, 1994.Google ScholarCross Ref
- J. Polich. Updating p300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10):2128--2148, 2007.Google ScholarCross Ref
- A. T. Pope, E. H. Bogart, and D. S. Bartolome. Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology, 40(1--2):187--195, 1995. EEG in Basic and Applied Settings.Google Scholar
- C. Ranganath, M. K. Johnson, and M. Desposito. Prefrontal activity associated with working memory and episodic long-term memory. Neuropsychologia, 41(3):378--389, 2003. Functional Neuroimaging of Memory.Google ScholarCross Ref
- M. D. Rugg, P. C. Fletcher, C. D. Frith, R. S. J. Frackowiak, and R. J. Dolan. Differential activation of the prefrontal cortex in successful and unsuccessful memory retrieval. Brain, 119:2073--2083, 1996.Google ScholarCross Ref
- T. Ruotsalo, J. Peltonen, M. Eugster, D. Glowacka, K. Konyushkova, K. Athukorala, I. Kosunen, A. Reijonen, P. Myllymáki, G. Jacucci, and S. Kaski. Directing exploratory search with interactive intent modeling. In Proceedings of the 22nd ACM international conference on Conference on information and knowledge management, CIKM '13, pages 1759--1764, New York, NY, USA, 2013. ACM. Google ScholarDigital Library
- Y. Saeys, I. Inza, and P. Larra naga. A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19):2507--2517, 2007. Google ScholarDigital Library
- T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13):2126--2144, Nov. 2007. Google ScholarDigital Library
- S. Sonnenburg, G. Rátsch, C. Scháfer, and B. Schölkopf. Large scale multiple kernel learning. Journal of Machine Learning Research, 7:1531--1565, 2006. Google ScholarDigital Library
- M. M. Spap--e and D. J. Serrien. Interregional synchrony of visuomotor tracking: Perturbation effects and individual differences. Behavioural Brain Research, 213(2):313--318, 2010.Google ScholarCross Ref
- C. Vidaurre and B. Blankertz. Towards a cure for BCI illiteracy. Brain Topography, 23(2):194--198, 2010.Google ScholarCross Ref
- R. W. White and D. Kelly. A study on the effects of personalization and task information on implicit feedback performance. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management, CIKM '06, pages 297--306, New York, NY, USA, 2006. ACM. Google ScholarDigital Library
- J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan. Brain-computer interfaces for communication and control. nClinical Neurophysiology, 113(6):767--791, 2002.Google Scholar
- J. X. Zhfang, H.-C. Leung, and M. K. Johnson. Frontal activations associated with accessing and evaluating information in working memory: an fMRI study. NeuroImage, 20(3):1531--1539, 2003.Google ScholarCross Ref
Index Terms
- Predicting term-relevance from brain signals
Recommendations
The Cortical Activity of Graded Relevance
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information RetrievalRelevance is an essential concept in Information Retrieval (IR). Recent studies using brain imaging have significantly contributed towards the understanding of this concept, but only as a binary notion, i.e. a document being judged as relevant or non-...
When Relevance Judgement is Happening?: An EEG-based Study
SIGIR '15: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information RetrievalRelevance is a central notion in Information Retrieval, but it is considered to be a difficult concept to define. We analyse brain signals for the first 800 milliseconds (ms) of a relevance assessment process to answer the question "when relevance is ...
Towards Predicting a Realisation of an Information Need based on Brain Signals
WWW '19: The World Wide Web ConferenceThe goal of Information Retrieval (IR) systems is to satisfy searchers' Information Need (IN). Our research focuses on next-generation IR engines, which can proactively detect, identify, and serve INs without receiving explicit queries. It is essential, ...
Comments