Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
International Journal of Computer Vision

07-03-2024

Open Set Recognition in Real World

Authors: Zhen Yang, Jun Yue, Pedram Ghamisi, Shiliang Zhang, Jiayi Ma, Leyuan Fang

Published in: International Journal of Computer Vision

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Open set recognition (OSR) constitutes a critical endeavor within the domain of computer vision, frequently deployed in applications, such as autonomous driving and medical imaging recognition. Existing OSR methodologies predominantly center on the acquisition of a profound association between image data and corresponding labels, facilitating the extraction of discriminative features instrumental for distinguishing novel categories. Nevertheless, real-world scenarios often introduce not only novel classes (referred to semantic shift) but also intricate environmental modifications that engender alterations in the distribution of established classes (termed as covariate shift). The latter phenomenon has the potential to undermine the robust correlation between images and labels established by conventional statistical correlation modeling approaches, consequently resulting in significant degradation of OSR performance. Causal correlation stands as the fundamental linkage between entities, routinely harnessed by humans to enhance their cognitive capacities for a more profound comprehension of the intricate world. With inspiration drawn from this perspective, our work herein introduces the causal inference-inspired open set recognition (CISOR) approach tailored for real-world OSR (RWOSR). CISOR represents the pioneering initiative to leverage the stability inherent in causal correlation to construct two pivotal modules: the covariate causal independence (CCI) module and the semantic causal uniqueness (SCU) module, both instrumental in addressing the RWOSR problem. The CCI module adeptly confronts the challenge of covariate shift by imposing constraints on the correlations between inter-class causal features. This strategy effectively mitigates the impact of spurious correlations between distinct categories on the generalization capacity of discriminative features. Furthermore, in order to counteract the issue of semantic shift, the SCU module harnesses correlations between causal features within the same class as constraints, thereby facilitating the extraction of resilient causal features endowed with superior discriminative capabilities. Empirical findings substantiate the superior efficacy of the proposed CIOSR method when compared to state-of-the-art approaches across diverse RWOSR benchmark datasets. The source code of this article will be available at https://​github.​com/​yangzhen1252/​RWOSR1.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now
Literature
Bendale, A., & Boult, T. (2015). Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1893–1902).
Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563–1572).
Busto, P. P., & Gall, J. (2017). Open set domain adaptation. In Proceedings of IEEE/CVF international conference on computer vision (pp. 754–763).
Caesar, H., et al. (2020). nuScenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11621–11631).
Cen, J., Luan, D., Zhang, S., et al. (2023). The devil is in the wrongly-classified samples: Towards unified open-set recognition. In Proceedings of ICLR.
Chen, G., Qiao, L., Shi, Y., et al. (2020). Learning open set network with discriminative reciprocal points. In Proceedings of European conference on computer vision (pp. 507–522).
Chen, G., Peng, P., Wang, X., & Tian, Y. (2022). Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8065–8081.PubMed
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297.CrossRef
Fang, L., Yan, Y., Yue, J., & Deng, Y. (2023). Toward the vectorization of hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 61, 1–14.
Fukumizu, K., Bach, F., & Jordan, M. (2004). Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5, 73–99.MathSciNet
Ge, Z. Y., Demyanov, S., Chen, Z., & Garnavi, R. (2017). Generative OpenMax for multi-class open set classification [Online]. Available: arXiv:​1707.​07418.
Geng, C., Huang, S. J., & Chen, S. (2021). Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10), 3614–3631.CrossRefPubMed
Geng, X., Dong, G., Xia, Z., & Liu, H. (2022). SAR target recognition via random sampling combination in open-world environments. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 331–343.ADSCrossRef
Giusti, E., Ghio, S., Oveis, A. H., & Martorella, M. (2022). Proportional similarity-based openmax classiffer for open set recognition in SAR images. Remote Sensing, 14(18), 4665.ADSCrossRef
Gretton, A., Fukumizu, K., Teo, C., et al. (2008). A kernel statistical test of independence. In Proceedings of NeuralPS (pp. 585–592).
Huang, M., et al. (2022). SwinTextSpotter: Scene text spotting via better synergy between text detection and text recognition. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 4583–4593).
Jain, L. P., Scheirer, W. J., & Boult, T. E. (2014). Multi-class open set recognition using probability of inclusion. In Proceedings of the European Conference on Computer Vision (pp. 393–409).
Jo, I., Kim, J., Kang, H., Kim, Y. D., & Choi, S. (2018). Open set recognition by regularising classifier with fake data generated by generative adversarial networks. In Proceedings of international conference on acoustics, speech, and signal processing (pp. 2686–2690).
Kerner, H. R., Wellington, D. F., Wagstaff, K. L., Bell, J. F., Kwan, C., & Amor, H. B. (2019). Novelty detection for multispectral images with application to planetary exploration. In Proceedings of AAAI.
Kotz, S., & Nadarajah, S. (2000). Extreme value distributions: Theory and applications. World Scientific.CrossRef
Kuang, K., Cui, P., Athey, S., et al. (2018). Stable prediction across unknown environments. In Proceedings of KDD.
Kuang, K., Li, L., Geng, Z., et al. (2020). Causal inference. Engineering, 6(3), 253–263.CrossRef
Li, D., Yang, Y., Song, Y., & Hospedales, T. (2017). Deeper, broader and artier domain generalization. In Proceedings IEEE/CVF international conference on computer vision (pp. 5542–5550).
Li, H., Pan, S. J., Wang, S., & Kot, A. C. (2018). Domain generalization with adversarial feature learning. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 5400–5409).
Liu, B., Kang, H., Li, H., Hua, G., & Vasconcelos, N. (2020). Few-shot open-set recognition using meta-learning. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition (pp. 8795–8804).
Liu, J., Tian, J., Han, W., et al. (2023). Learning multiple gaussian prototypes for open-set recognition. Information Sciences, 626, 738–753.CrossRef
Liu, L., Ouyang, W., Wang, X., et al. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128, 261–318.CrossRef
Liu, Z., Fu, Y., Pan, Q., & Zhang, Z. (2023). Orientational distribution learning with hierarchical spatial attention for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 8757–8772.PubMed
Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B. & Yu, S. X. (2019). Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2532–2541).
Lopez-Paz, D., Nishihara, R., Chintala, S., Scholkopf, B., & Bottou, L. (2017). Discovering causal signals in images. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 58–66).
Mensink, T., Verbeek, J., Perronnin, F., & Csurka, G. (2013). Distancebased image classification: Generalizing to new classes at nearzero cost. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(11), 2624–2637.CrossRefPubMed
Moon, W., Park, J., Seong, H. S., et al. (2022). Difficulty-aware simulator for open set recognition. In Proceedings of the European conference on computer vision.
Mousavian, A., Anguelov, D., Flynn, J., & Kosecka, J. (2017). 3D bounding box estimation using deep learning and geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7074–7082).
Neal, L., Olson, M., Fern, X., Wong, W., & Li, F. (2018). Open set learning with counterfactual images. In Proceedings of European conference on computer vision (pp. 613–628).
Nirmal, S., Sowmya, V., & Soman, K. P. (2020). Open set domain adaptation for hyperspectral image classiffcation using generative adversarial network. In Proceedings of ICICCT (pp. 819–827).
Oliveira, H., Silva, C., Machado, G. L., Nogueira, K., & Dos Santos, J. A. (2021). Fully convolutional open set segmentation. Machine Learning 1–52.
Oza, P., & Patel, V. M. (2019). C2AE: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2307–2316).
Pal, D., Bose, S., Banerjee, B., & Jeppu, Y. (2023). MORGAN: Meta-learning based few-shot open-set recognition via generative adversarial network. In Proceedings of WACV (pp. 6295–6304).
Pang, S., et al. (2021). SpineParseNet: Spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation. IEEE Transactions on Medical Imaging, 40(1), 262–273.CrossRefPubMed
Peng, J., Li, L., & Tang, Y. Y. (2019). Maximum likelihood estimation based joint sparse representation for the classification of hyperspectral remote sensing images. IEEE Transactions on Neural Networks and Learning Systems, 30(6), 1790–1802.MathSciNetCrossRefPubMed
Prakhya, S., Venkataram, V., & Kalita, J. (2017). Open set text classification using convolutional neural networks. In Proceedings of the international conference recent advances in natural language processing (pp. 466–475).
Rahimi, A., & Recht, B. (2008). Random features for large scale kernel machines. In Proceedings of NeuralPS (pp. 1177–1184).
Rahman, S., Khan, S. H., & Porikli, F. (2020). Zero-shot object detection: Joint recognition and localization of novel concepts. International Journal of Computer Vision, 128, 2979–2999.CrossRef
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149.CrossRefPubMed
Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., & Boult, T. E. (2013). Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757–1772.CrossRefPubMed
Scheirer, W. J., Jain, L. P., & Boult, T. E. (2014). Probability models for OSR. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2317–2324.CrossRefPubMed
Shen, Z., Cui, P., et al. (2020). Stable learning via sample reweighting. In Proceedings of AAAI conference on artificial intelligence (pp. 5692–5699).
Song, Q., Sun, B., & Li, S. (2023). Multimodal sparse transformer network for audio-visual speech recognition. IEEE Transactions on Neural Networks and Learning Systems, 34(12), 10028–10038.CrossRefPubMed
Strobl, E., Zhang, K., & Visweswaran, S. (2019). Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference7(1).
Torralba, A., & Efros, A. (2011). Unbiased look at dataset bias. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1521–1528).
Wang, L., et al. (2023). Multi-Modal 3D object detection in autonomous driving: A survey and taxonomy. IEEE Transactions on Intelligent Vehicles, 8(7), 3781–3798.CrossRef
Wang, R., Guo, J., Zhao, R., Su, L., Ye, Y., Zhang, X., Zhang, Y., & Feng, R. (2023). Class-aware variational auto-encoder for open set recognition. In Proceedings of IEEE ICME.
Wang, Z., Xu, Q., & Yang, Z. (2022). OpenAUC: Towards AUC-oriented open-set recognition. In Proceedings of NeurIPS.
Wu, L., Fang, L., He, X., He, M., Ma, J., & Zhong, Z. (2023a). Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 8827–8844.
Wu, J., Zhu, D., Fang, L., Deng, Y., & Zhong, Z. (2023b). Efficient layer compression without pruning. IEEE Transactions on Image Processing, 32, 4689–4700.
Xu, Y., Qin, L., Liu, X., Xie, J., & Zhu, S.-C. (2018). A causal and-or graph model for visibility fluent reasoning in tracking interacting objects. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2178–2187).
Yang, X., Feng, F., Ji, W., Wang, M., & Chua, T.-S. (2021). Deconfounded video moment retrieval with causal intervention. In Proceedings of SIGIR (pp. 1–10).
Yang, Y., Hou, C., Lang, Y., Guan, D., Huang, D., & Xu, J. (2019). Open-set human activity recognition based on micro-doppler signatures. Pattern Recognition, 85, 60–69.ADSCrossRef
Yao, L., Chu, Z., & Li, S. (2021). A survey on causal inference. ACM Transactions on Knowledge Discovery from Data, 15(5), 1556–4681.CrossRef
Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., & Naemura, T. (2019). Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4016–4025).
Yu, H., Cui, P., et al. (2023). Stable learning via sparse variable independence. In Proceedings of AAAI conference on artificial intelligence
Yu, Y., Qu, W. Y., Li, N., & Guo, Z. (2017). Open-category classification by adversarial sample generation. In Proceedings of the international joint conferences on artificial intelligence
Yue, J., Fang, L., & He, M. (2022). Spectral-spatial latent reconstruction for open-set hyperspectral image classification. IEEE Transactions on Image Processing, 31, 5227–5241.ADSCrossRefPubMed
Zhang, H., & Patel, V. (2017). Sparse representation-based open set brecognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8), 1690–1696.CrossRefPubMed
Zhang, J., Liu, J., Shi, L., Pan, B., & Xu, X. (2020). An open set domain adaptation network based on adversarial learning for remote sensing image scene classiffcation. In Proceedings of IEEE international geoscience and remote sensing symposium (pp. 1365–1368).
Zhang, S., Feng, X., Fan, W., et al. (2023). Video-audio domain generalization via confounder disentanglement. In Proceedings of AAAI conference on artificial intelligence.
Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y., & Shen, Z. (2021). Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE conference on computer vision and pattern Recognition (pp. 5368–5378).
Zhou, X., Tang, T., Cui, Y., & Kuang, G. (2022). SAR open set recognition based on counterfactual framework. In Proceedings of PIERS (pp. 249–253).
Zhou, Y., Shang, S., Song, X., Zhang, S., You, T., & Zhang, L. (2022). Intelligent radar jamming recognition in open set environment based on deep learning networks. Remote Sensing, 14(24), 6220.ADSCrossRef
Metadata
Title
Open Set Recognition in Real World
Authors
Zhen Yang
Jun Yue
Pedram Ghamisi
Shiliang Zhang
Jiayi Ma
Leyuan Fang
Publication date
07-03-2024
Publisher
Springer US
Published in
International Journal of Computer Vision
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI
https://doi.org/10.1007/s11263-024-02015-9

Premium Partner

    Image Credits