nach oben

Erschienen in:

2019 | OriginalPaper | Buchkapitel

Referring Expression Comprehension via Co-attention and Visual Context

verfasst von : Youming Gao, Yi Ji, Ting Xu, Yunlong Xu, Chunping Liu

Erschienen in: Artificial Neural Networks and Machine Learning – ICANN 2019: Image Processing

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

As a research hotspot of multimodal media analysis, referring expression comprehension locates the referred object region in an image by mapping a natural language. Though the localizing accuracy of similar objects is often distorted by the presence or absence of supporting objects in the referring expression, we propose a referring expression comprehension method via co-attention and visual context. For lacking supporting objects in referring expression, we propose co-attention to enhance the attention on attributes for the subject module. For existing supporting objects, we introduce visual context to explore the latent link between the candidate object and its supporters. Experiments on three datasets RefCOCO, RefCOCO+, and RefCOCOg, show that our approach outperforms published approaches by a considerable margin.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Dense Receptive Field Network: A Backbone Network for Object Detection

Nächstes Kapitel Comparison Between U-Net and U-ReNet Models in OCR Tasks

Ben-Younes, H., Cadene, R., Cord, M., Thome, N.: Mutan: multimodal tucker fusion for visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 3 (2017). https://doi.org/10.1109/iccv.2017.285

Chen, K., Kovvuri, R., Nevatia, R.: Query-guided regression network with context policy for phrase grounding. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017). https://doi.org/10.1109/iccv.2017.95

Corbetta, M., Shulman, G.L.: Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3(3), 201 (2002). https://doi.org/10.1038/nrn755CrossRef

Das, A., et al.: Visual dialog. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2 (2017)

He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988. IEEE (2017). https://doi.org/10.1109/iccv.2017.322

Hu, R., Rohrbach, M., Andreas, J., Darrell, T., Saenko, K.: Modeling relationships in referential expressions with compositional modular networks. In: CVPR, pp. 4418–4427 (2017). https://doi.org/10.1109/cvpr.2017.470

Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., Darrell, T.: Natural language object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555–4564 (2016). https://doi.org/10.1109/cvpr.2016.493

Johnson, J., et al.: Inferring and executing programs for visual reasoning. In: ICCV, pp. 3008–3017 (2017). https://doi.org/10.1109/iccv.2017.325

Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48CrossRef

10.

Lindh, A., Ross, R.J., Mahalunkar, A., Salton, G., Kelleher, J.D.: Generating diverse and meaningful captions. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 176–187. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01418-6_18CrossRef

11.

Liu, J., Wang, L., Yang, M.H., et al.: Referring expression generation and comprehension via attributes. In: Proceedings of CVPR (2017). https://doi.org/10.1109/iccv.2017.520

12.

Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via a visual sentinel for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 6, p. 2 (2017). https://doi.org/10.1109/cvpr.2017.345

13.

Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual question answering. In: Advances In Neural Information Processing Systems, pp. 289–297 (2016)

14.

Luo, R., Shakhnarovich, G.: Comprehension-guided referring expressions. In: Computer Vision and Pattern Recognition (CVPR), vol. 2 (2017). https://doi.org/10.1109/cvpr.2017.333

15.

Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11–20 (2016). https://doi.org/10.1109/cvpr.2016.9

16.

Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 792–807. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_48CrossRef

17.

Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). https://doi.org/10.3115/v1/d14-1162

18.

Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015). https://doi.org/10.1109/tpami.2016.2577031CrossRef

19.

Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., Schiele, B.: Grounding of textual phrases in images by reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 817–834. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_49CrossRef

20.

Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., et al.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: ICCV, pp. 618–626 (2017). https://doi.org/10.1109/iccv.2017.74

21.

Wei, W.L., et al: Seethevoice: learning from music to visual storytelling of shots. In: 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2018). https://doi.org/10.1109/icme.2018.8486496

22.

Xiong, C., Zhong, V., Socher, R.: Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604 (2016)

23.

Yu, L., et al.: MAttNet: modular attention network for referring expression comprehension. In: CVPR (2018). https://doi.org/10.1109/CVPR.2018.00142

24.

Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 69–85. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_5CrossRef

25.

Yu, L., Tan, H., Bansal, M., Berg, T.L.: A joint speaker-listener-reinforcer model for referring expressions. In: Computer Vision and Pattern Recognition (CVPR), vol. 2 (2017). https://doi.org/10.1109/cvpr.2017.375

26.

Zhang, Y., Gu, Y., Gu, X.: Two-stream convolutional neural network for multimodal matching. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 14–21. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01418-6_2CrossRef

Titel: Referring Expression Comprehension via Co-attention and Visual Context
verfasst von: Youming Gao
Yi Ji
Ting Xu
Yunlong Xu
Chunping Liu
Verlag: Springer International Publishing
Buch: Artificial Neural Networks and Machine Learning – ICANN 2019: Image Processing
Print ISBN: 978-3-030-30507-9

Electronic ISBN: 978-3-030-30508-6

Copyright-Jahr: 2019
DOI: https://doi.org/10.1007/978-3-030-30508-6_10

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner