nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

Dense Captioning Using Abstract Meaning Representation

verfasst von : Antonio M. S. Almeida Neto, Helena M. Caseli, Tiago A. Almeida

Erschienen in: Intelligent Systems

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The world around us is composed of images that often need to be translated into words. This translation can take place in parts, converting regions of the image into textual descriptions what is also known as dense captioning. By doing so, the information present in this region is converted into words expressing the way objects relate to each other. Computational models have been proposed to perform this task in a similar way to human beings, mainly using deep neural networks. As the same region of the image can be described in several different forms, this study proposes to use the Abstract Meaning Representation (AMR) in the generation of descriptions for a given region. We hypothesize that by using AMR it would be possible to extract the meaning of the text and, as a consequence, improve the quality of the sentences produced by the models. AMR was investigated as a semantic representation formalism evolving the so far proposed models that are based only on purely natural language. The results show that the models trained with sentences in the form of AMR led to better descriptions and the performance achieved was superior in almost all evaluations.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel DeepBT and NLP Data Augmentation Techniques: A New Proposal and a Comprehensive Study

Nächstes Kapitel Can Twitter Data Estimate Reality Show Outcomes?

Image from Visual Genome Dataset [10].

The success of syntactic banks is due to the fact that unifying the various tasks in a single process allowed the use of a single tool. An example of a classic syntactic bank is the Penn Treebank.

The value before the slash indicates the total of unique tokens (types) and the value after the slash, the total number of occurrences of those tokens.

Abend, O., Rappoport, A.: The state of the art in semantic representation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 77–89 (2017)

Banarescu, L., et al.: Abstract meaning representation for sembanking. In: Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 178–186 (2013)

Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72. Association for Computational Linguistics, Ann Arbor, June 2005. https://www.aclweb.org/anthology/W05-0909

Cai, S., Knight, K.: Smatch: an evaluation metric for semantic feature structures. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 748–752. Association for Computational Linguistics, Sofia, August 2013. https://www.aclweb.org/anthology/P13-2131

Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2014, pp. 580–587. IEEE Computer Society, Washington, DC (2014). https://doi.org/10.1109/CVPR.2014.81

Johnson, J., Karpathy, A., Fei-Fei, L.: DenseCap: fully convolutional localization networks for dense captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015

Karpathy, A., Joulin, A., Fei-Fei, L.: Deep fragment embeddings for bidirectional image sentence mapping. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. NIPS 2014, pp. 1889–1897. MIT Press, Cambridge (2014). http://dl.acm.org/citation.cfm?id=2969033.2969038

Konstas, I., Iyer, S., Yatskar, M., Choi, Y., Zettlemoyer, L.: Neural AMR: sequence-to-sequence models for parsing and generation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 146–157. Association for Computational Linguistics, Vancouver, July 2017. https://doi.org/10.18653/v1/P17-1014, https://www.aclweb.org/anthology/P17-1014

10.

Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123(1), 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7MathSciNetCrossRef

11.

Liao, K., Lebanoff, L., Liu, F.: Abstract Meaning Representation for multi-document summarization. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1178–1190. Association for Computational Linguistics, Santa Fe, August 2018. https://www.aclweb.org/anthology/C18-1101

12.

Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48CrossRef

13.

Liu, F., Flanigan, J., Thomson, S., Sadeh, N., Smith, N.A.: Toward abstractive summarization using semantic representations. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1077–1086. Association for Computational Linguistics, Denver, May–Jun 2015. https://doi.org/10.3115/v1/N15-1114, https://www.aclweb.org/anthology/N15-1114

14.

Lyu, C., Titov, I.: AMR parsing as graph prediction with latent alignment. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (2018)

15.

Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. ACL 2002, pp. 311–318. Association for Computational Linguistics, Stroudsburg (2002). https://doi.org/10.3115/1073083.1073135

16.

Reiter, E., Dale, R.: Building applied natural language generation systems. Nat. Lang. Eng. 3(1), 57–87 (1997)CrossRef

17.

Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031CrossRef

18.

Song, L., Zhang, Y., Wang, Z., Gildea, D.: A graph-to-sequence model for AMR-to-text generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL-18), Melbourne, Australia (2018)

19.

Tamchyna, A., Quirk, C., Galley, M.: A discriminative model for semantics-to-string translation. In: Proceedings of the 1st Workshop on Semantics-Driven Statistical Machine Translation (S2MT 2015), pp. 30–36. Association for Computational Linguistics, Beijing, July 2015. https://doi.org/10.18653/v1/W15-3504, https://www.aclweb.org/anthology/W15-3504

20.

Thomee, B., et al.: YFCC100M: the new data in multimedia research. Commun. ACM 59(2), 64–73 (2016). https://doi.org/10.1145/2812802CrossRef

21.

Yang, L., Tang, K., Yang, J., Li, L.J.: Dense captioning with joint inference and visual context. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

22.

Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J.: Context and attribute grounded dense captioning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

23.

Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 69–85. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_5CrossRef

Titel: Dense Captioning Using Abstract Meaning Representation
verfasst von: Antonio M. S. Almeida Neto
Helena M. Caseli
Tiago A. Almeida
Verlag: Springer International Publishing
Buch: Intelligent Systems
Print ISBN: 978-3-030-61376-1

Electronic ISBN: 978-3-030-61377-8

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-61377-8_31

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"