Top

Published in:

2025 | OriginalPaper | Chapter

Can OOD Object Detectors Learn from Foundation Models?

Authors : Jiahui Liu, Xin Wen, Shizhen Zhao, Yingxian Chen, Xiaojuan Qi

Published in: Computer Vision – ECCV 2024

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Out-of-distribution (OOD) object detection is a challenging task due to the absence of open-set OOD data. Inspired by recent advancements in text-to-image generative models, such as Stable Diffusion, we study the potential of generative models trained on large-scale open-set data to synthesize OOD samples, thereby enhancing OOD object detection. We introduce SyncOOD, a simple data curation method that capitalizes on the capabilities of large foundation models to automatically extract meaningful OOD data from text-to-image generative models. This offers the model access to open-world knowledge encapsulated within off-the-shelf foundation models. The synthetic OOD samples are then employed to augment the training of a lightweight, plug-and-play OOD detector, thus effectively optimizing the in-distribution (ID)/OOD decision boundaries. Extensive experiments across multiple benchmarks demonstrate that SyncOOD significantly outperforms existing methods, establishing new state-of-the-art performance with minimal synthetic data usage. The project is available at https://github.com/CVMI-Lab/SyncOOD.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis

next chapter VIDEOSHOP: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion

Available only for authorised users

Concepts overlapping with the test data are removed to avoid information leakage.

Achiam, J., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

Bendale, A., Boult, T.E.: Towards open set deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1563–1572 (2016)

Bishop, C.M.: Novelty detection and neural network validation. IEE Proc.-Vis. Image Sig. Process. 141(4), 217–222 (1994)CrossRef

Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems 33, pp. 1877–1901 (2020)

Chang, C., Liu, Z., Lyu, X., Qi, X.: What matters in detecting AI-generated videos like Sora? arXiv preprint arXiv:2406.19568 (2024)

Chowdhery, A., et al.: PaLM: scaling language modeling with pathways. J. Mach. Learn. Res. 24(240), 1–113 (2023)

Deepshikha, K., Yelleni, S.H., Srijith, P., Mohan, C.K.: Monte Carlo dropblock for modelling uncertainty in object detection. arXiv preprint arXiv:2108.03614 (2021)

Deng, W., et al.: Can 3D vision-language models truly understand natural language? arXiv preprint arXiv:2403.14760 (2024)

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

10.

Dhamija, A., Gunther, M., Ventura, J., Boult, T.: The overlooked elephant of object detection: open set. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1021–1030 (2020)

11.

Du, X., Fang, Z., Diakonikolas, I., Li, Y.: How does unlabeled data provably help out-of-distribution detection? arXiv preprint arXiv:2402.03502 (2024)

12.

Du, X., Gozum, G., Ming, Y., Li, Y.: SIREN: shaping representations for detecting out-of-distribution objects. In: Advances in Neural Information Processing Systems 35, pp. 20434–20449 (2022)

13.

Du, X., Sun, Y., Zhu, J., Li, Y.: Dream the impossible: outlier imagination with diffusion models. In: Advances in Neural Information Processing Systems 36 (2024)

14.

Du, X., Wang, X., Gozum, G., Li, Y.: Unknown-aware object detection: learning what you don’t know from videos in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13678–13688 (2022)

15.

Du, X., Wang, Z., Cai, M., Li, Y.: VOS: learning what you don’t know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197 (2022)

16.

Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The Pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2010)CrossRef

17.

Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: International Conference on Learning Representations (2022)

18.

Gupta, A., Narayan, S., Joseph, K., Khan, S., Khan, F.S., Shah, M.: OW-DETR: open-world detection transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9235–9244 (2022)

19.

Hall, D., et al.: Probabilistic object detection: definition and evaluation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1031–1040 (2020)

20.

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

21.

He, R., et al.: Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574 (2022)

22.

Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (2017)

23.

Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 (2018)

24.

Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems 33, pp. 6840–6851 (2020)

25.

Hsu, Y.C., Shen, Y., Jin, H., Kira, Z.: Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951–10960 (2020)

26.

Huang, R., Geng, A., Li, Y.: On the importance of gradients for detecting distributional shifts in the wild. In: Advances in Neural Information Processing Systems 34, pp. 677–689 (2021)

27.

Joseph, K., Khan, S., Khan, F.S., Balasubramanian, V.N.: Towards open world object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5830–5840 (2021)

28.

Katz-Samuels, J., Nakhleh, J.B., Nowak, R., Li, Y.: Training OOD detectors in their natural habitats. In: International Conference on Machine Learning, pp. 10848–10865. PMLR (2022)

29.

Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)

30.

Kong, S., Ramanan, D.: OpenGAN: open-set recognition via open data generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 813–822 (2021)

31.

Kuznetsova, A., et al.: The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis. 128(7), 1956–1981 (2020)CrossRef

32.

Lee, K., Lee, H., Lee, K., Shin, J.: Training confidence-calibrated classifiers for detecting out-of-distribution samples. In: International Conference on Learning Representations (2018)

33.

Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Advances in Neural Information Processing Systems 31 (2018)

34.

Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: International Conference on Learning Representations, ICLR 2018 (2018)

35.

Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48CrossRef

36.

Liu, J., Chang, C., Liu, J., Wu, X., Ma, L., Qi, X.: MarS3D: a plug-and-play motion-aware model for semantic segmentation on multi-scan 3D point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9372–9381 (2023)

37.

Liu, W., Wang, X., Owens, J., Li, Y.: Energy-based out-of-distribution detection. In: Advances in Neural Information Processing Systems 33, pp. 21464–21475 (2020)

38.

Liu, X., Yang, H., Ravichandran, A., Bhotika, R., Soatto, S.: Continual universal object detection. arXiv preprint arXiv:2002.05347 (2020)

39.

Miller, D., Dayoub, F., Milford, M., Sünderhauf, N.: Evaluating merging strategies for sampling-based uncertainty techniques in object detection. In: International Conference on Robotics and Automation (ICRA), pp. 2348–2354 (2019)

40.

Miller, D., Nicholson, L., Dayoub, F., Sünderhauf, N.: Dropout sampling for robust object detection in open-set conditions. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3243–3249 (2018)

41.

Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427–436 (2015)

42.

Perez-Rua, J.M., Zhu, X., Hospedales, T.M., Xiang, T.: Incremental few-shot object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13846–13855 (2020)

43.

Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

44.

Radford, A., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)

45.

Rahman, S., Khan, S.H., Porikli, F.: Zero-shot object detection: joint recognition and localization of novel concepts. Int. J. Comput. Vis. 128, 2979–2999 (2020)CrossRef

46.

Ren, J., Fort, S., Liu, J., Roy, A.G., Padhy, S., Lakshminarayanan, B.: A simple fix to Mahalanobis distance for improving near-OOD detection. arXiv preprint arXiv:2106.09022 (2021)

47.

Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems 28 (2015)

48.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

49.

Sastry, C.S., Oore, S.: Detecting out-of-distribution examples with gram matrices. In: International Conference on Machine Learning, pp. 8491–8501. PMLR (2020)

50.

Sun, Y., Ming, Y., Zhu, X., Li, Y.: Out-of-distribution detection with deep nearest neighbors. In: International Conference on Machine Learning, pp. 20827–20840. PMLR (2022)

51.

Tack, J., Mo, S., Jeong, J., Shin, J.: CSI: novelty detection via contrastive learning on distributionally shifted instances. In: Advances in Neural Information Processing Systems 33, pp. 11839–11852 (2020)

52.

Tan, H., et al.: Data pruning via moving-one-sample-out. In: Advances in Neural Information Processing Systems (2023)

53.

Tao, L., Du, X., Zhu, X., Li, Y.: Non-parametric outlier synthesis. arXiv preprint arXiv:2303.02966 (2023)

54.

Tian, Y., Fan, L., Chen, K., Katabi, D., Krishnan, D., Isola, P.: Learning vision from models rivals learning vision from data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15887–15898 (2024)

55.

Wang, H., Liu, W., Bocchieri, A., Li, Y.: Can multi-label classification networks know what they don’t know? In: Advances in Neural Information Processing Systems 34, pp. 29074–29087 (2021)

56.

Wang, X., et al.: Robust object detection via instance-level temporal cycle confusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9143–9152 (2021)

57.

Wen, X., Zhao, B., Chen, Y., Pang, J., Qi, X.: Generalization beyond data imbalance: a controlled study on CLIP for transferable insights. arXiv preprint arXiv:2405.21070 (2024)

58.

Wilson, S., Fischer, T., Dayoub, F., Miller, D., Sünderhauf, N.: SAFE: sensitivity-aware features for out-of-distribution object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23565–23576 (2023)

59.

Wu, A., Chen, D., Deng, C.: Deep feature deblurring diffusion for detecting out-of-distribution objects. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13381–13391 (2023)

60.

Wu, A., Deng, C.: Discriminating known from unknown objects via structure-enhanced recurrent variational autoencoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23956–23965 (2023)

61.

Wu, Q., Chen, Y., Yang, C., Yan, J.: Energy-based out-of-distribution detection for graph neural networks. arXiv preprint arXiv:2302.02914 (2023)

62.

Wu, S., Tan, H., Tian, Z., Chen, Y., Qi, X., Jia, J.: Saco loss: sample-wise affinity consistency for vision-language pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27358–27369 (2024)

63.

Wu, X., et al.: CL-NeRF: continual learning of neural radiance fields for evolving scene representation. In: Advances in Neural Information Processing Systems 36 (2024)

64.

Yu, F., et al.: BDD100K: a diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2636–2645 (2020)

65.

Zheng, H., et al.: Out-of-distribution detection learning with unreliable out-of-distribution sources. In: Advances in Neural Information Processing Systems 36, pp. 72110–72123 (2023)

Title: Can OOD Object Detectors Learn from Foundation Models?
Authors: Jiahui Liu
Xin Wen
Shizhen Zhao
Yingxian Chen
Xiaojuan Qi
Publisher: Springer Nature Switzerland
Book: Computer Vision – ECCV 2024
Print ISBN: 978-3-031-73253-9

Electronic ISBN: 978-3-031-73254-6

Copyright Year: 2025
DOI: https://doi.org/10.1007/978-3-031-73254-6_13

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"