Top

Published in:

2019 | OriginalPaper | Chapter

Many vs. Many Query Matching with Hierarchical BERT and Transformer

Authors : Yang Xu, Qiyuan Liu, Dong Zhang, Shoushan Li, Guodong Zhou

Published in: Natural Language Processing and Chinese Computing

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Query matching is a fundamental task in the Natural Language Processing community. In this paper, we focus on an informal scenario where the query may consist of multiple sentences, namely query matching with informal text. On the basis, we first construct two datasets towards different domains. Then, we propose a novel query matching approach for informal text, namely Many vs. Many Matching with hierarchical BERT and transformer. First, we employ fine-tuned BERT (bidirectional encoder representation from transformers) to capture the pair-wise sentence matching representations. Second, we adopt the transformer to accept above all matching representations, which aims to enhance the pair-wise sentence matching vector. Third, we utilize soft attention to get the importance of each matching vector for final matching prediction. Empirical studies demonstrate the effectiveness of the proposed model to query matching with informal text.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Evaluating and Enhancing the Robustness of Retrieval-Based Dialogue Systems with Adversarial Examples

next chapter A Knowledge Selective Adversarial Network for Link Prediction in Knowledge Graph

http://www.ccks2018.cn.

https://dc.cloud.alipay.com.

http://icrc.hitsz.edu.cn/info/1037/1146.htm.

https://pypi.python.org/pypi/jieba/.

https://radimrehurek.com/gensim/models/word2vec.html.

https://stanfordnlp.github.io/CoreNLP/.

Zhou, X., et al.: Multi-turn response selection for chatbots with deep attention matching network. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1118–1127 (2018)

Qiao, Y., Xiong, C., Liu, Z., Liu, Z.: Understanding the behaviors of BERT in ranking. arXiv preprint arXiv:1904.07531 (2019)

Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional deep neural networks. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 373–382. ACM (2015)

Wang, L., et al.: One vs. many QA matching with both word-level and sentence-level attention network. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2540–2550 (2018)

Shen, C., et al.: Sentiment classification towards question-answering with hierarchical matching network. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3654–3663 (2018)

Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)

Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018). https://s3-us-west-2.ama-zonaws.com/openai-assets/research-covers/languageunsupervised/languageunderstandingpaper.pdf

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

Sun, C., Huang, L., Qiu, X.: Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.09588 (2019)

10.

Liu, X., et al.: LCQMC: a large-scale chinese question matching corpus. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1952–1962 (2018)

11.

Feng, M., Xiang, B., Glass, M.R., Wang, L., Zhou, B.: Applying deep learning to answer selection: a study and an open task. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 813–820. IEEE (2015)

12.

Yin, W., Schütze, H., Xiang, B., Zhou, B.: ABCNN: attention-based convolutional neural network for modeling sentence pairs. Trans. Assoc. Comput. Linguist. 4, 259–272 (2016)CrossRef

13.

He, H., Lin, J.: Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 937–948 (2016)

14.

Zhang, H., et al.: Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243 (2019)

15.

Kondratyuk, D.: 75 languages, 1 model: parsing universal dependencies universally. arXiv preprint arXiv:1904.02099 (2019)

16.

Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 (2015)

17.

Tan, M., Dos Santos, C., Xiang, B., Zhou, B.: Improved representation learning for question answer matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 464–473 (2016)

18.

Wang, S., Jiang, J.: A compare-aggregate model for matching text sequences. arXiv preprint arXiv:1611.01747 (2016)

Title: Many vs. Many Query Matching with Hierarchical BERT and Transformer
Authors: Yang Xu
Qiyuan Liu
Dong Zhang
Shoushan Li
Guodong Zhou
Publisher: Springer International Publishing
Book: Natural Language Processing and Chinese Computing
Print ISBN: 978-3-030-32232-8

Electronic ISBN: 978-3-030-32233-5

Copyright Year: 2019
DOI: https://doi.org/10.1007/978-3-030-32233-5_13

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner