Skip to main content
SpringerLink
Log in

RMAN: Relational multi-head attention neural network for joint extraction of entities and relations

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The task of extracting entities and relations has evolved from distributed extraction to joint extraction. The joint model overcomes the disadvantages of distributed extraction method and strengthens the information interaction between entities and relations. However, the existing methods of the joint model rarely pay attention to the semantic information between words, which have limitations in solving the problem of overlapping relations. In this paper, we propose an RMAN model for joint extraction of entities and relations, which includes multi-feature fusion encoder sentence representation and decoder sequence annotation. We first add a multi-head attention layer after Bi-LSTM to obtain sentence representations, and leverage the attention mechanism to capture relation-based sentence representations. Then, we perform sequence annotation on the sentence representation to obtain entity pairs. Experiments on NYT-single, NYT-multi and WebNLG datasets demonstrate that our model can efficiently extract overlapping triples, which outperforms other baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Adel H, Schütze H (2017) Global normalization of convolutional neural networks for joint entity and relation classification. arXiv:170707719

  2. Akkasi A, Varoğlu E, Dimililer N (2018) Balanced undersampling: a novel sentence-based undersampling method to improve recognition of named entities in chemical and biomedical text. Appl Intell 48 (8):1965–1978

    Article  Google Scholar 

  3. Bunescu R, Mooney R (2005) A shortest path dependency kernel for relation extraction. In: Proceedings of human language technology conference and conference on empirical methods in natural language processing, pp 724–731

  4. Cao P, Chen Y, Liu K, Zhao J, Liu S (2018) Adversarial transfer learning for chinese named entity recognition with self-attention mechanism. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 182–192

  5. Christopoulou F, Tran T T, Sahu S K, Miwa M, Ananiadou S (2020) Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods. J Am Med Inform Assoc 27(1):39–46

    Article  Google Scholar 

  6. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:14123555

  7. Fei H, Ren Y, Ji D (2020) Boundaries and edges rethinking: an end-to-end neural model for overlapping entity relation extraction. Inform Process Manag 57(6):102311

    Article  Google Scholar 

  8. Fu TJ, Li PH, Ma WY (2019) Graphrel: Modeling text as relational graphs for joint entity and relation extraction. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 1409–1418

  9. Geng Z, Chen G, Han Y, Lu G, Li F (2020) Semantic relation extraction using sequential and tree-structured lstm with attention. Inf Sci 509:183–192

    Article  Google Scholar 

  10. Gupta P, Schütze H, Andrassy B (2016) Table filling multi-task recurrent neural network for joint entity and relation extraction. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: technical papers, pp 2537–2547

  11. Hong Y, Liu Y, Yang S, Zhang K, Wen A, Hu J (2020) Improving graph convolutional networks based on relation-aware attention for end-to-end relation extraction. IEEE Access 8:51315–51323

    Article  Google Scholar 

  12. Hu S, Zou L, Zhang X (2018) A state-transition framework to answer complex questions over knowledge base. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 2098–2108

  13. Jiang J, Zhai C (2007) A systematic exploration of the feature space for relation extraction. In: Human language technologies 2007: The conference of the north american chapter of the association for computational linguistics; proceedings of the main conference, pp 113–120

  14. Kambhatla N (2004) Combining lexical, syntactic, and semantic features with maximum entropy models for information extraction. In: Proceedings of the ACL interactive poster and demonstration sessions, pp 178–181

  15. Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C (2016a) Neural architectures for named entity recognition. arXiv:160301360

  16. Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C (2016b) Neural architectures for named entity recognition. arXiv:160301360

  17. Li H, Wu X, Li Z, Wu G (2013) A relation extraction method of chinese named entities based on location and semantic features. Appl Intell 38(1):1–15

    Article  MathSciNet  Google Scholar 

  18. Li Q, Ji H (2014) Incremental joint extraction of entity mentions and relations. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 402–412

  19. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv:13013781

  20. Miwa M, Bansal M (2016) End-to-end relation extraction using lstms on sequences and tree structures. arXiv:160100770

  21. Miwa M, Sasaki Y (2014) Modeling joint entity and relation extraction with table representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1858–1869

  22. Nayak T, Ng H T (2020) Effective modeling of encoder-decoder architecture for joint entity and relation extraction. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 8528–8535

  23. Nguyen DQ, Nguyen TD, Nguyen DQ, Phung D (2017) A novel embedding model for knowledge base completion based on convolutional neural network. arXiv:171202121

  24. Rosaci D (2007) Cilios: Connectionist inductive learning and inter-ontology similarities for recommending information agents. Inform Syst 32(6):793–825

    Article  Google Scholar 

  25. Rosaci D (2015) Finding semantic associations in hierarchically structured groups of web data. Form Asp Comput 27(5):867– 884

    Article  Google Scholar 

  26. Takanobu R, Zhang T, Liu J, Huang M (2019) A hierarchical framework for relation extraction with reinforcement learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 7072–7079

  27. Wang H, Jiang S, Yu Z (2020) Modeling of complex internal logic for knowledge base completion. Appl Intell 50:3336–3349

    Article  Google Scholar 

  28. Yu B, Zhang Z, Shu X, Wang Y, Liu T, Wang B, Li S (2019) Joint extraction of entities and relations based on a novel decomposition strategy. arXiv:190904273

  29. Yuan Y, Zhou X, Pan S, Zhu Q, Song Z, Guo L (2020) A relation-specific attention network for joint entity and relation extraction. In: International joint conference on artificial intelligence 2020, association, for the advancement of artificial intelligence (AAAI), pp 4054–4060

  30. Zeng D, Liu K, Lai S, Zhou G, Zhao J (2014a) Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers, pp 2335–2344

  31. Zeng D, Liu K, Lai S, Zhou G, Zhao J (2014b) Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers, pp 2335–2344

  32. Zeng X, Zeng D, He S, Liu K, Zhao J (2018) Extracting relational facts by an end-to-end neural model with copy mechanism. In: Proceedings of the 56th annual meeting of the association for computational linguistics (volu 1: Long Papers), pp 506–514

  33. Zeng X, He S, Zeng D, Liu K, Liu S, Zhao J (2019) Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 367–377

  34. Zhang S, Zheng D, Hu X, Yang M (2015) Bidirectional long short-term memory networks for relation classification. In: Proceedings of the 29th Pacific Asia conference on language, information and computation, pp 73–78

  35. Zheng S, Wang F, Bao H, Hao Y, Zhou P, Xu B (2017) Joint extraction of entities and relations based on a novel tagging scheme. arXiv:170605075

  36. Zhou G, Zhang M (2007) Extracting relation information from text documents by exploring various types of knowledge. Inform Process Manag 43(4):969–982

    Article  Google Scholar 

  37. Zhou P, Shi W, Tian J, Qi Z, Li B, Hao H, Xu B (2016) Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th annual meeting of the association for computational linguistics (vol 2: Short papers), pp 207–212

Download references

Acknowledgements

This research is supported by Key-Area Research and Development Program of Guangdong Province under Grant 2019B010153002, Key Program of NSFC-Guangdong Joint Funds under Grant U1801263 and U1701262, Science and Technology Projects of Guangzhou under Grant 202007040006, Program of Marine Economy Development (Six Marine Industries) Special Foundation of Department of Natural Resources of Guangdong Province under Grant GDNRC [2020]056, National Natural Science Foundation of China under Grant 62002071, Top Youth Talent Project of Zhujiang Talent Program under Grant 2019QN01X516, National Key R & D project under Grant 2019YFB1705503, R & D projects in key areas of Guangdong Province under Grant 2018B010109007 and Guangdong Provincial Key Laboratory of Cyber-Physical System under Grant 2020B1212060069.

Author information

Authors and Affiliations

  1. School of Automation, Guangdong University of Technology, Guangzhou, China

    Taiqu Lai & Depei Wang

  2. School of Computers, Guangdong University of Technology, Guangzhou, China

    Lianglun Cheng, Haiming Ye & Weiwen Zhang

Authors
  1. Taiqu Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lianglun Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Depei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haiming Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weiwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiwen Zhang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lai, T., Cheng, L., Wang, D. et al. RMAN: Relational multi-head attention neural network for joint extraction of entities and relations. Appl Intell 52, 3132–3142 (2022). https://doi.org/10.1007/s10489-021-02600-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02600-2

Keywords

Search

Navigation