Jun Wang
ABSTRACT
Human visual perception is able to recognize a wide range of targets under challenging conditions, but has limited throughput. Machine vision and automatic content analytics can process images at a high speed, but suffers from inadequate recognition accuracy for general target classes. In this paper, we propose a new paradigm to explore and combine the strengths of both systems. A single trial EEG-based brain machine interface (BCI) subsystem is used to detect objects of interest of arbitrary classes from an initial subset of images. The EEG detection outcomes are used as input to a graph-based pattern mining subsystem to identify, refine, and propagate the labels to retrieve relevant images from a much larger pool. The combined strategy is unique in its generality, robustness, and high throughput. It has great potential for advancing the state of the art in media retrieval applications. We have evaluated and demonstrated significant performance gains of the proposed system with multiple and diverse image classes over several data sets, including those from Internet (Caltech 101) and remote sensing images. In this paper, we will also present insights learned from the experiments and discuss future research directions.
- N. Bigdely-Shamlo, A. Vankov, R. Ramirez, and S. Makeig. Brain Activity-Based Image Classification From Rapid Serial Visual Presentation. IEEE Trans. on NSRE, 16(5):432--441, 2008.Google Scholar
- N. Bigdely-Shamlo, A. Vankov, R. R. Ramirez, and S. Makeig. Brain activity-based image classification from rapid serial visual presentation. IEEE Trans. on NSRE, 16(5):432--441, Oct. 2008.Google Scholar
- J. Donoghue. Bridging the brain to the world: a perspective on neural interface systems. Neuron, 60(3):511--521, 2008.Google ScholarCross Ref
- M. Dyrholm, C. Christoforou, and L. Parra. Bilinear discriminant component analysis. JMLR, 8:1097--1111, 2007. Google ScholarDigital Library
- M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results. http://www.pascalnetwork.org/challenges/VOC/voc2008/workshop/index.html.Google Scholar
- L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVIU, 106(1):59--70, 2007. Google ScholarDigital Library
- R. Fergus and P. Perona. A Visual Category Filter for Google Images. In Proc. ECCV, 2004.Google ScholarCross Ref
- A. Gerson, L. Parra, and P. Sajda. Cortically coupled computer vision for rapid image search. IEEE Trans. on NSRE, 14(2):174--179, 2006.Google Scholar
- M. Gladwell. Blink: The power of thinking without thinking. Little, brown and company: Time warner book group, New York, 2005.Google Scholar
- M. Hein and M. Maier. Manifold denoising. Proc. NIPS, 19, 2006.Google Scholar
- T. Huang, C. Dagli, S. Rajaram, E. Chang, M. Mandel, G. Poliner, and D. Ellis. Active Learning for Interactive Multimedia Retrieval. Proc. of the IEEE, 96(4):648--667, 2008.Google Scholar
- T. Jebara, J. Wang, and S.-F. Chang. Graph construction and b-matching for semi-supervised learning. In Proc. ICML, 2009. Google ScholarDigital Library
- Y. Jiang, C. Ngo, and J. Yang. Towards optimal bag-of-features for object categorization and semantic video retrieval. In Proc. of CIVR, pages 494--501, 2007. Google ScholarDigital Library
- Y. Jing and S. Baluja. VisualRank: Applying PageRank to Large-Scale Image Search. IEEE Trans. on PAMI, 12, 2008. Google ScholarDigital Library
- A. Kapoor, P. Shenoy, and D. Tan. Combining brain computer interfaces with vision for object categorization. In Proc. CVPR, 2008.Google ScholarCross Ref
- K. Kay, T. Naselaris, R. Prenger, and J. Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352--355, 2008.Google ScholarCross Ref
- C. Keysers, D. Xiao, P. Foldiak, and D. Perrett. The speed of sight. Journal of Cognitive Neuroscience, 13(1):90--101, 2001. Google ScholarDigital Library
- J. Langford, L. Li, and T. Zhang. Sparse Online Learning via Truncated Gradient. JMLR, 10:777--801, 2009. Google ScholarDigital Library
- D. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91--110, 2004. Google ScholarDigital Library
- C. Micchelli and M. Pontil. Learning the kernel function via regularization. JMLR, 6(2):1099--1125, 2006. Google ScholarDigital Library
- K. Mikolajczyk and C. Schmid. Scale&affine invariant interest point detectors. IJCV, 60(1):63--86, 2004. Google ScholarDigital Library
- T. Mitchell, R. Hutchinson, R. Niculescu, F. Pereira, X. Wang, M. Just, and S. Newman. Learning to decode cognitive states from brain images. Machine Learning, 57(1):145--175, 2004. Google ScholarDigital Library
- Y. Miyawaki, H. Uchida, O. Yamashita, M. Sato, Y. Morito, H. Tanabe, N. Sadato, and Y. Kamitani. Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron, 60(5):915--929, 2008.Google ScholarCross Ref
- A. Oliva. Gist of the scene. In Encyclopedia of Neurobiology of Attention, pages 251--256, San Diego, CA, 2005. Elsevier.Google Scholar
- L. Parra, C. Christoforou, A. Gerson, M. Dyrholm, A. Luo, M. Wagner, M. Philiastides, and P. Sajda. Spatiotemporal linear decoding of brain state: Application to performance augmentation in high-throughput tasks. IEEE Signal Processing Magazine, 25(1):95--115, January 2008.Google ScholarCross Ref
- L. Parra, C. Christoforou, A. Gerson, M. Dyrholm, A. Luo, M. Wagner, M. Philiastides, and P. Sajda. Spatiotemporal linear decoding of brain state: Application to performance augmentation in high-throughput tasks. IEEE Signal Processing Magazine, 25(1):95--115, January 2008.Google ScholarCross Ref
- P. Poolman, R. Frank, P. Luu, S. Pederson, and D. Tucker. A single-trial analytic framework for EEG analysis and its application to target detection and classification. NeuroImage, 42(2):787--798, 2008.Google ScholarCross Ref
- M. Potter and E. Levy. Recognition memory for a rapid sequence of pictures. Journal of Experimental Psychology, 81(1):10, 1969.Google ScholarCross Ref
- Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. on CSVT, 8(5):644--655, 1998. Google ScholarDigital Library
- M. Sanderson and P. Clough. cross-language image retrieval track. http://imageclef.org/.Google Scholar
- P. Shenoy and D. Tan. Human-Aided Computing: Utilizing Implicit Human Processing to Classify Images. In Proc. CHI. Google ScholarDigital Library
- A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR '06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pages 321--330, 2006. Google ScholarDigital Library
- S. Thorpe, D. Fize, and C. Marlot. Speed of processing in the human visual system. Nature, 381(6582):520--522, 1996.Google ScholarCross Ref
- J. Wang, S. F. Chang, X. Zhou, and S. T. C. Wong. Active microscopic cellular image annotation by superposable graph transduction with imbalanced labels. In Proc. CVPR, 2008.Google Scholar
- J. Wang, T. Jebara, and S.-F. Chang. Graph transduction via alternating minimization. In Proc. ICML, 2008. Google ScholarDigital Library
- J. Wang, Y.-G. Jiang, and S.-F. Chang. Label diagnosis through self tuning for web image search. In Proc. CVPR, 2009.Google ScholarCross Ref
- D. Zhou, J. Weston, A. Gretton, O. Bousquet, and B. Scholkopf. Ranking on data manifolds. In Proc. NIPS, 2004.Google Scholar
Index Terms
- Brain state decoding for rapid image retrieval
Recommendations
Design of Wearable Brain Computer Interface Based on Motor Imagery
IIH-MSP '14: Proceedings of the 2014 Tenth International Conference on Intelligent Information Hiding and Multimedia Signal ProcessingBrain computer interface (BCI) is a communication system to establish a communication interface between the brain and the device. Motor imagery-based brain computer interface is one of asynchronous BCIs, and have been widely developed in recent years. ...
Development of a Wearable Motor-Imagery-Based Brain---Computer Interface
A motor-imagery-based brain---computer interface (BCI) is a translator that converts the motor intention of the brain into a control command to control external machines without muscles. Numerous motor-imagery-based BCIs have been successfully proposed ...
Mental tasks-based brain-robot interface
This paper describes a Brain Computer Interface (BCI) based on electroencephalography (EEG) that allows control of a robot arm. This interface will enable people with severe disabilities to control a robot arm to assist them in a variety of tasks in ...
Comments