Top

Published in:

2020 | OriginalPaper | Chapter

Domain Adaptation of Transformers for English Word Segmentation

Authors : Ruan Chaves Rodrigues, Acquila Santos Rocha, Marcelo Akira Inuzuka, Hugo Alexandre Dantas do Nascimento

Published in: Intelligent Systems

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Word segmentation can contribute to improve the results of natural language processing tasks on several problem domains, including social media sentiment analysis, source code summarization and neural machine translation. Taking the English language as a case study, we fine-tune a Transformer architecture which has been trained through the Pre-trained Distillation (PD) algorithm, while comparing it to previous experiments with recurrent neural networks. We organize datasets and resources from multiple application domains under a unified format, and demonstrate that our proposed architecture has competitive performance and superior cross-domain generalization in comparison with previous approaches for word segmentation in Western languages.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Can Twitter Data Estimate Reality Show Outcomes?

next chapter Entropy-Based Filter Selection in CNNs Applied to Text Classification

https://github.com/google-research/bert.

https://github.com/jiangpinglei/BERT_ChineseWordSegment.

https://www.grupocole.org/software/VCS/segmnt/models/.

Standard Generalized Markup Language is a metalanguage through which it is possible to define markup languages for documents.

https://github.com/attardi/wikiextractor.

http://www.statmt.org/wmt17/.

https://github.com/afmdnf/hashtag-lstm.

https://www.nltk.org/_modules/nltk/tokenize/punkt.html.

Sentiment140 - a Twitter sentiment analysis tool (2009). http://help.sentiment140.com/for-students

EMNLP 2017 - second conference on machine translation (WMT17) (2017). http://www.statmt.org/wmt17/. Accessed 17 May 2020

Celebi, A., Ozgur, A.: Segmenting hashtags and analyzing their grammatical structure. J. Assoc. Inf. Sci. Technol. 69 (2017). https://doi.org/10.1002/asi.23989

Chen, X., Shi, Z., Qiu, X., Huang, X.: Adversarial multi-criteria learning for Chinese word segmentation (2017)

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

Doval, Y., Gómez-Rodríguez, C.: Comparing neural- and N-gram-based language models for word segmentation. J. Assoc. Inf. Sci. Technol. 70(2), 187–197 (2018). 10/gfs6rd, https://doi.org/10.1002/asi.24082, 105

Huang, W., Cheng, X., Chen, K., Wang, T., Chu, W.: Toward fast and accurate neural Chinese word segmentation with multi-criteria learning. arXiv:1903.04190 [cs], March 2019. http://arxiv.org/abs/1903.04190, 97.9

Inuzuka, M.A., Rocha, A.S., Nascimento, H.A.D.: Segmentation of words written in the Latin alphabet: a systematic review. In: Quaresma, P., Vieira, R., Aluísio, S., Moniz, H., Batista, F., Gonçalves, T. (eds.) PROPOR 2020. LNCS (LNAI), vol. 12037, pp. 291–302. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-41505-1_28CrossRef

Lei, J.: Bert

https://static-content.springer.com/image/chp%3A10.1007%2F978-3-030-61377-8_33/MediaObjects/503050_1_En_33_Figb_HTML.gif

https://static-content.springer.com/image/chp%3A10.1007%2F978-3-030-61377-8_33/MediaObjects/503050_1_En_33_Figc_HTML.gif

(2019). Accessed 17 May 2020

10.

Li, J., Du, Q., Shi, K., He, Y., Wang, X., Xu, J.: Helpful or not? An investigation on the feasibility of identifier splitting via CNN-BiLSTM-CRF (2018)

11.

Li, X., Meng, Y., Sun, X., Han, Q., Yuan, A., Li, J.: Is word segmentation necessary for deep learning of Chinese representations? arXiv preprint arXiv:1905.05526 (2019)

12.

Ma, J., Ganchev, K., Weiss, D.: State-of-the-art Chinese word segmentation with Bi-LSTMs. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4902–4908. Association for Computational Linguistics, Brussels, October–November 2018. https://doi.org/10.18653/v1/D18-1529, https://www.aclweb.org/anthology/D18-1529

13.

Maddela, M., Xu, W., Preoţiuc-Pietro, D.: Multi-task pairwise neural ranking for hashtag segmentation (2019)

14.

Morabia, K., Bhanu Murthy, N.L., Malapati, A., Samant, S.: SEDTWik: segmentation-based event detection from tweets using Wikipedia. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 77–85. Association for Computational Linguistics, Minneapolis, June 2019. https://doi.org/10.18653/v1/N19-3011, https://www.aclweb.org/anthology/N19-3011

15.

Pan, Y., Li, X., Yang, Y., Dong, R.: Morphological word segmentation on agglutinative languages for neural machine translation (2020)

16.

Rhodes, D.: Conditional random field Latin word segmenter (2013)

17.

Saenger, P.: Space Between Words: The Origins of Silent Reading. Stanford University Press, Palo Alto (1997)

18.

Shi, X., Huang, H., Jian, P., Guo, Y., Wei, X., Tang, Y.-K.: Neural Chinese word segmentation as sequence to sequence translation. In: Cheng, X., Ma, W., Liu, H., Shen, H., Feng, S., Xie, X. (eds.) SMP 2017. CCIS, vol. 774, pp. 91–103. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-6805-8_8CrossRef

19.

Souza, F., Nogueira, R., Lotufo, R.: Portuguese named entity recognition using BERT-CRF. arXiv:1909.10649 [cs], February 2020. zSCC: NoCitationData[s0]

20.

Sproat, R.W., Shih, C., Gale, W., Chang, N.: A stochastic finite-state word-segmentation algorithm for Chinese. Comput. Linguist. 22(3), 377–404 (1996). https://www.aclweb.org/anthology/J96-3004

21.

Turc, I., Chang, M.W., Lee, K., Toutanova, K.: Well-read students learn better: on the importance of pre-training compact models (2019)

22.

Wang, B., Pan, B., Li, X., Li, B.: Towards evaluating the robustness of Chinese BERT classifiers (2020)

23.

Yang, H.: BERT meets Chinese word segmentation (2019)

24.

Ye, Y., Zhang, Y., Li, W., Qiu, L., Sun, J.: Improving cross-domain Chinese word segmentation with word embeddings. In: Proceedings of the 2019 Conference of the North, pp. 2726–2735 (2019). https://doi.org/10.18653/v1/N19-1279, http://arxiv.org/abs/1903.01698, arXiv: 1903.01698

25.

Zhang, Y., Clark, S.: Syntactic processing using the generalized perceptron and beam search. Comput. Linguist. 37(1), 105–151 (2011). https://doi.org/10.1162/coli_a_00037, https://www.aclweb.org/anthology/J11-1005

Title: Domain Adaptation of Transformers for English Word Segmentation
Authors: Ruan Chaves Rodrigues
Acquila Santos Rocha
Marcelo Akira Inuzuka
Hugo Alexandre Dantas do Nascimento
Publisher: Springer International Publishing
Book: Intelligent Systems
Print ISBN: 978-3-030-61376-1

Electronic ISBN: 978-3-030-61377-8

Copyright Year: 2020
DOI: https://doi.org/10.1007/978-3-030-61377-8_33

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner