Top

Neural Processing Letters

Published in:

11-05-2022

Implicit Relation Inference with Deep Path Extraction for Commonsense Question Answering

Authors: Peng Yang, Zijian Liu, Bing Li, Penghui Zhang

Published in: Neural Processing Letters | Issue 6/2022

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Natural language inference plays an essential role in Commonsense Question Answering. Conventional models usually adopt keywords in questions and choices as queries to retrieve static and explicit evidence that is used to obtain final answers, where dynamic interaction between different keywords and implicit relations inference of deeper information are often neglected. In this paper, we propose a novel joint model, the Graph Relation retrieval Reasoning Network (GRRN), to explicitly introduce the dynamic interaction among different keywords and generate informative features that contribute to representation updating. In addition, to pursue in-depth relations between different keywords, we develop an optimised Path Evidence Fusion in the GRRN to obtain evidence based on deep paths and implicit relations with comprehensive knowledge by making full use of the original paths in external knowledge graphs. The experimental results show that compared with the baselines, our approach achieves remarkable improvement of 1.74\(\%\) for precision on the CommonsenseQA dataset, thereby demonstrating the superiority of our state-of-the-art approach on implicit relation inference with deep paths.

previous article COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization

next article Color Image Enhancement: A Metaheuristic Chimp Optimization Algorithm

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

https://dumps.wikimedia.org/enwiki/.

https://www.elastic.co/.

https://dictionary.cambridge.org/.

https://pytorch.org/.

Battaglia PW, Hamrick JB, Bapst V, Sanchez-Gonzalez A, Zambaldi V, Malinowski M, Tacchetti A, Raposo D, Santoro A, Faulkner R, et al. (2018) Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261

Bordes A, Usunier N, Garcia-Durán A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, vol 2, pp 2787–2795

Bosselut A, Rashkin H, Sap M, Malaviya C, Celikyilmaz A, Choi Y (2019) Comet: Commonsense transformers for automatic knowledge graph construction. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 4762–4779

Bosselut A, Le Bras R, Choi Y (2021) Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 35:4923–4931CrossRef

Chen D, Fisch A, Weston J, Bordes A (2017) Reading wikipedia to answer open-domain questions. In: Proceedings of the 55th annual meeting of the association for computational linguistics, vol 1, pp 1870–1879

Chen Q, Ji F, Chen H, Zhang Y (2020) Improving commonsense question answering by graph-based iterative retrieval over multiple knowledge sources. arXiv preprint arXiv:2011.02705

Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 1724–1734

Clark K, Luong MT, Le QV, Manning CD (2020) Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555

Cui W, Xiao Y, Wang H, Song Y, Hwang Sw, Wang W (2019) Kbqa: learning question answering over qa corpora and knowledge bases. arXiv preprint arXiv:1903.02419

10.

Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

11.

Doxolodeo K, Mahendra R (2020) Ui at semeval-2020 task 4: Commonsense validation and explanation by exploiting contradiction. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp 614–619

12.

Fadel A, Al-Ayyoub M, Cambria E (2020) Justers at semeval-2020 task 4: Evaluating transformer models against commonsense validation and explanation. In: Proceedings of the fourteenth workshop on semantic evaluation, pp 535–542

13.

Feng Y, Chen X, Lin BY, Wang P, Yan J, Ren X (2020) Scalable multi-hop relational reasoning for knowledge-aware question answering. arXiv preprint arXiv:2005.00646

14.

Gretz S, Bilu Y, Cohen-Karlik E, Slonim N (2020) The workweek is the best time to start a family–a study of gpt-2 based claim generation. arXiv preprint arXiv:2010.06185

15.

Huang Y, Fang M, Zhan X, Cao Q, Liang X, Lin L (2021) Rem-net: Recursive erasure memory network for commonsense evidence refinement. Proceedings of the AAAI Conference on Artificial Intelligence 35:6375–6383CrossRef

16.

He X, Liu Q, Yang Y (2020) Mv-gnn: multi-view graph neural network for compression artifacts reduction. IEEE Trans Image Process 29:6829–6840CrossRefMATH

17.

Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRef

18.

Huang X, Zhang J, Li D, Li P (2019) Knowledge graph embedding based question answering. In: Proceedings of the twelfth ACM international conference on web search and data mining, pp 105–113

19.

Ji S, Pan S, Cambria E, Marttinen P, Yu PS (2022) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans Neural Networks Learn Syst 33(2):494–514MathSciNetCrossRef

20.

Joshi M, Chen D, Liu Y, Weld DS, Zettlemoyer L, Levy O (2020) Spanbert: improving pre-training by representing and predicting spans. Trans Assoc Comput Linguist 8:64–77CrossRef

21.

Khashabi D, Khot T, Sabharwal A, Tafjord O, Clark P, Hajishirzi H (2020) Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700

22.

Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

23.

Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942

24.

Li L, Gan Z, Cheng Y, Liu J (2019a) Relation-aware graph attention network for visual question answering. In: Proceedings of the IEEE/cvf international conference on computer vision, pp 10313–10322

25.

Li S, Chen J, Yu D (2019b) Teaching pretrained models with commonsense reasoning: a preliminary kb-based approach. arXiv preprint arXiv:1909.09743

26.

Lin BY, Chen X, Chen J, Ren X (2019) Kagnet: Knowledge-aware graph networks for commonsense reasoning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2822–2832

27.

Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

28.

Liu Y, Wan Y, He L, Peng H, Yu PS (2021) Kg-bart: knowledge graph augmented bart for generative commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence 35:6418–6425CrossRef

29.

Loshchilov I, Hutter F (2017) Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101

30.

Lukovnikov D, Fischer A, Lehmann J, Auer S (2017) Neural network-based question answering over knowledge graphs on word and character level. In: Proceedings of the 26th international conference on World Wide Web, pp 1211–1220

31.

Lv S, Guo D, Xu J, Tang D, Duan N, Gong M, Shou L, Jiang D, Cao G, Hu S (2020) Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 34:8449–8456CrossRef

32.

Ma K, Francis J, Lu Q, Nyberg E, Oltramari A (2019) Towards generalizable neuro-symbolic systems for commonsense question answering. arXiv preprint arXiv:1910.14087

33.

Ma K, Ilievski F, Francis J, Bisk Y, Nyberg E, Oltramari A (2021) Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 35:13507–13515CrossRef

34.

Mihaylov T, Clark P, Khot T, Sabharwal A (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 2381–2391

35.

Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

36.

Mishra BD, Tandon N, Clark P (2017) Domain-targeted, high precision knowledge extraction. Trans Assoc Comput Linguist 5:233–246CrossRef

37.

Mitra A, Banerjee P, Pal KK, Mishra S, Baral C (2019) How additional knowledge can improve natural language commonsense question answering? arXiv preprint arXiv:1909.08855

38.

Na SH, Lee JH (2020) Jbnu at semeval-2020 task 4: Bert and unilm for commonsense validation and explanation. In: Proceedings of the fourteenth workshop on semantic evaluation, pp 527–534

39.

Nickel M, Murphy K, Tresp V, Gabrilovich E (2015) A review of relational machine learning for knowledge graphs. Proc IEEE 104(1):11–33CrossRef

40.

Nickel M, Rosasco L, Poggio T (2016) Holographic embeddings of knowledge graphs. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp 1955–1961

41.

Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol 1, pp 2227–2237

42.

Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2019) Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683

43.

Romero J, Razniewski S, Pal K, Z Pan J, Sakhadeo A, Weikum G (2019) Commonsense properties from query logs and question answering forums. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp 1411–1420

44.

Saeedi S, Panahi A, Saeedi S, Fong AC (2020) Cs-nlp team at semeval-2020 task 4: evaluation of state-of-the-art nlp deep learning architectures on commonsense reasoning task. arXiv preprint arXiv:2006.01205

45.

Saha A, Pahuja V, Khapra MM, Sankaranarayanan K, Chandar S (2018) Complex sequential question answering: towards learning to converse over linked question answer pairs with a knowledge graph. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 705–713

46.

Santoro A, Raposo D, Barrett DG, Malinowski M, Pascanu R, Battaglia P, Lillicrap T (2017) A simple neural network module for relational reasoning. In: Advances in neural information processing systems, pp 4967–4976

47.

Sap M, Le Bras R, Allaway E, Bhagavatula C, Lourie N, Rashkin H, Roof B, Smith NA, Choi Y (2019) Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence 33:3027–3035CrossRef

48.

Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp 4444–4451

49.

Schlichtkrull M, Kipf TN, Bloem P, Van Den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks. In: European semantic web conference, Springer, pp 593–607

50.

Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Proceedings of the AAAI conference on artificial intelligence, vol 31

51.

Strathearn C, Gkatzia D (2021) Chefbot: A novel framework for the generation of commonsense-enhanced responses for task-based dialogue systems. In: Proceedings of the 14th International Conference on Natural Language Generation, pp 46–47

52.

Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen W (2018) Open domain question answering using early fusion of knowledge bases and text. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 4231–4242

53.

Talmor A, Herzig J, Lourie N, Berant J (2019) Commonsenseqa: A question answering challenge targeting commonsense knowledge. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1, pp 4149–4158

54.

Tandon N, De Melo G, Weikum G (2017) Webchild 2.0: fine-grained commonsense knowledge distillation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics-System Demonstration, pp 115–120

55.

Wan G, Pan S, Gong C, Zhou C, Haffari G (2021) Reasoning like human: Hierar-chical reinforcement learning for knowledge graph reasoning. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp 1926–1932

56.

Wang C, Liang S, Jin Y, Wang Y, Zhu X, Zhang Y (2020a) Semeval-2020 task 4: commonsense validation and explanation. arXiv preprint arXiv:2007.00236

57.

Wang G, Hou X, Yang D, McKeown K, Huang J (2021) Semantic categorization of social knowledge for commonsense question answering. arXiv preprint arXiv:2109.05168

58.

Wang H, Tang X, Lai S, Leung KS, Zhu J, Fung GPC, Wong KF (2020b) Cuhk at semeval-2020 task 4: Commonsense explanation, reasoning and prediction with multi-task learning. arXiv preprint arXiv:2006.09161

59.

Wang P, Peng N, Szekely P, Ren X (2020c) Connecting the dots: a knowledgeable path generator for commonsense question answering. arXiv preprint arXiv:2005.00691

60.

Wang X, Kapanipathi P, Musa R, Yu M, Talamadupula K, Abdelaziz I, Chang M, Fokoue A, Makni B, Mattei N et al (2019) Improving natural language inference using external knowledge in the science questions domain. Proceedings of the AAAI Conference on Artificial Intelligence 33:7208–7215CrossRef

61.

Wu S, Li Y, Zhang D, Zhou Y, Wu Z (2020) Topicka: generating commonsense knowledge-aware dialogue responses towards the recommended topic fact. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp 3766–3772

62.

Yan J, Raman M, Zhang T, Rossi R, Zhao H, Kim S, Lipka N, Ren X (2020) Learning contextualized knowledge structures for commonsense reasoning. arXiv preprint arXiv:2010.12873

63.

Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp 5753–5763

64.

Yasunaga M, Ren H, Bosselut A, Liang P, Leskovec J (2021) Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2104.06378

65.

Ye ZX, Chen Q, Wang W, Ling ZH (2019) Align, mask and select: a simple method for incorporating commonsense knowledge into language representation models. arXiv preprint arXiv:1908.06725

66.

Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M (2018) Augmenting end-to-end dialogue systems with commonsense knowledge. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 4970–4977

67.

Zhan X, Huang Y, Dong X, Cao Q, Liang X (2022) Pathreasoner: Explainable reasoning paths for commonsense question answering. Knowl Based Syst 235:107612CrossRef

68.

Zhang Y, Dai H, Kozareva Z, Smola A, Song L (2018) Variational reasoning for question answering with knowledge graph. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 6069–6076

69.

Zhu C, Cheng Y, Gan Z, Sun S, Goldstein T, Liu J (2019) Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764

70.

Zhan X, Huang Y, Dong X, Cao Q, Liang X (2022) Pathreasoner: Explainable reasoning paths for commonsense question answering. Knowl Based Syst 235:107612CrossRef

71.

Title: Implicit Relation Inference with Deep Path Extraction for Commonsense Question Answering
Authors: Peng Yang
Zijian Liu
Bing Li
Penghui Zhang
Publication date: 11-05-2022
Publisher: Springer US
Published in: Neural Processing Letters / Issue 6/2022
Print ISSN: 1370-4621
Electronic ISSN: 1573-773X
DOI: https://doi.org/10.1007/s11063-022-10831-8

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 6/2022

Adaptive Graph Regularized Deep Semi-nonnegative Matrix Factorization for Data Representation

Generating Optimal Test Case Generation Using Shuffled Shepherd Flamingo Search Model

Conditional Embedding Pre-Training Language Model for Image Captioning

Color Image Enhancement: A Metaheuristic Chimp Optimization Algorithm

New Results on Finite/Fixed-Time Stabilization of Stochastic Second-Order Neutral-Type Neural Networks with Mixed Delays

FuseNet: 3D Object Detection Network with Fused Information for Lidar Point Clouds