Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
A Hybrid Representation of Word Images for Keyword Spotting Read chapter Read first chapter

2020 | OriginalPaper | Chapter

Neural Machine Translation with Soft Reordering Knowledge

Authors : Leiying Zhou, Jie Zhou, Wenjie Lu, Kui Meng, Gongshen Liu

Published in: Neural Information Processing

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The Transformer architecture has been widely used in sequence to sequence tasks since it was proposed. However, it only adds the representations of absolute positions to its inputs to make use of the order information of the sequence. It lacks explicit structures to exploit the reordering knowledge of words. In this paper, we propose a simple but effective method to incorporate the reordering knowledge into the Transformer translation system. The reordering knowledge of each word is obtained by an additional reordering-aware attention sublayer based on its semantic and contextual information. The proposed approach can be easily integrated into the existing framework of the Transformer. Experimental results on two public translation tasks demonstrate that our proposed method can achieve significant translation improvements over the basic Transformer model and also outperforms the existing competitive systems.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Multitask Learning Based on Constrained Hierarchical Attention Network for Multi-aspect Sentiment Classification
next chapter Open Event Trigger Recognition Using Distant Supervision with Hierarchical Self-attentive Neural Network
Footnotes
Literature
1.
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)
2.
Brown, P.F., Pietra, S.D., Pietra, V.J.D., Mercer, R.L.: The mathematics of statistical machine translation: parameter estimation. Comput. Linguistics 19(2), 263–311 (1993)
3.
Cettolo, M., Girardi, C., Federico, M.: Wit3: Web inventory of transcribed and translated talks. In: Conference of European Association for Machine Translation, pp. 261–268 (2012)
4.
Chen, K., Wang, R., Utiyama, M., Sumita, E.: Neural machine translation with reordering embeddings. In: ACL (1), pp. 1787–1799. Association for Computational Linguistics (2019)
5.
Chiang, D.: A hierarchical phrase-based model for statistical machine translation. In: ACL, pp. 263–270. The Association for Computer Linguistics (2005)
6.
Gehring, J., Auli, M., Grangier, D., Dauphin, Y.N.: A convolutional encoder model for neural machine translation. In: ACL (1), pp. 123–135. Association for Computational Linguistics (2017)
7.
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 1243–1252. PMLR (2017)
8.
Kawara, Y., Chu, C., Arase, Y.: Recursive neural network based preordering for English-to-Japanese machine translation. In: ACL (3), pp. 21–27. Association for Computational Linguistics (2018)
9.
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (Poster) (2015)
10.
Meng, F., Zhang, J.: DTMT: a novel deep transition architecture for neural machine translation. In: AAAI, pp. 224–231. AAAI Press (2019)
11.
Nakagawa, T.: Efficient top-down BTG parsing for machine translation preordering. In: ACL (1), pp. 208–218. The Association for Computer Linguistics (2015)
12.
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL (1). The Association for Computer Linguistics (2016)
13.
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: NAACL-HLT (2), pp. 464–468. Association for Computational Linguistics (2018)
14.
So, D.R., Le, Q.V., Liang, C.: The evolved transformer. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5877–5886. PMLR (2019)
15.
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)
16.
Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)
17.
Zhang, J., Wang, M., Liu, Q., Zhou, J.: Incorporating word reordering knowledge into attention-based neural machine translation. In: ACL (1), pp. 1524–1534. Association for Computational Linguistics (2017)
18.
Zhu, Z.: Evaluating neural machine translation in English-Japanese task. In: WAT, pp. 61–68. Workshop on Asian Translation (2015)
Metadata
Title
Neural Machine Translation with Soft Reordering Knowledge
Authors
Leiying Zhou
Jie Zhou
Wenjie Lu
Kui Meng
Gongshen Liu
Copyright Year
2020
Book
Neural Information Processing

Print ISBN: 978-3-030-63819-1

Electronic ISBN: 978-3-030-63820-7

Copyright Year: 2020

DOI
https://doi.org/10.1007/978-3-030-63820-7_79

Premium Partner

    Image Credits