nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

LM-Based Word Embeddings Improve Biomedical Named Entity Recognition: A Detailed Analysis

verfasst von : Liliya Akhtyamova, John Cardiff

Erschienen in: Bioinformatics and Biomedical Engineering

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Recent studies have shown that contextualized word embeddings outperform other types of embeddings on a variety of tasks. However, there is little research done to evaluate their effectiveness in the biomedical domain under multi-task settings.

We derive the contextualized word embeddings from the Flair framework and apply them to the task of biomedical NER on 5 benchmark datasets, yielding major improvements over the baseline and achieving competitive results over the current best systems. We analyze the sources of these improvements, reporting model performances over different combinations of word embeddings, and fine-tuning and casing modes.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel DiS-TSS: An Annotation Agnostic Algorithm for TSS Identification

Nächstes Kapitel Evaluating Mutual Information and Chi-Square Metrics in Text Features Selection Process: A Study Case Applied to the Text Classification in PubMed

https://github.com/zalandoresearch/flair.

http://evexdb.org/pmresources/language-models/.

https://github.com/flairNLP/flair/blob/master/resources/docs/embeddings/CLASSIC_WORD_EMBEDDINGS.md.

http://evexdb.org/pmresources/vec-space-models/wikipedia-pubmed-and-PMC-w2v.bin.

https://github.com/cambridgeltl/MTL-Bioinformatics-2016.

Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: an easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 Conference of the North, pp. 54–59. Association for Computational Linguistics, Stroudsburg (2019). https://doi.org/10.18653/v1/N19-4010, http://aclweb.org/anthology/N19-4010

Akbik, A., Bergmann, T., Vollgraf, R.: Pooled contextualized embeddings for named entity recognition. In: NAACL (2019). https://github.com/zalandoresearch/flair

Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings for sequence labeling. In: COLING (2018). https://github.com/zalandoresearch/flair

Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5(2307–387X), 135–146 (2017). http://arxiv.org/abs/1607.04606

Collobert, R., Weston, J.: A unified architecture for natural language processing, pp. 160–167. Association for Computing Machinery (ACM) (2008). https://doi.org/10.1145/1390156.1390177

Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, ICASSP, pp. 8599–8603, October 2013. https://doi.org/10.1109/ICASSP.2013.6639344, ISBN 9781479903566, ISSN 15206149

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, October 2018. http://arxiv.org/abs/1810.04805

Doǧan, R.I., Leaman, R., Lu, Z.: NCBI disease corpus: a resource for disease name recognition and concept normalization. J. Biomed. Inform. 47, 1–10 (2014). https://doi.org/10.1016/j.jbi.2013.12.006. ISSN 15320464 CrossRefPubMedPubMedCentral

Giorgi, J.M., Bader, G.D.: Towards reliable named entity recognition in the biomedical domain. Bioinformatics 36(1), 280–286 (2020). https://doi.org/10.1093/bioinformatics/btz504. https://academic.oup.com/bioinformatics/article/36/1/280/5520946, ISSN 1367–4803CrossRefPubMed

10.

Gupta, A., Goyal, P., Sarkar, S., Gattu, M.: Fully contextualized biomedical NER. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019. LNCS, vol. 11438, pp. 117–124. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15719-7_15CrossRef

11.

Heinzerling, B., Strube, M.: BPEmb: Tokenization-free pre-trained subword embeddings in 275 languages. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pp. 18–1473 (2018). https://aclweb.org/anthology/papers/L/L18/L18-1473/

12.

Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF Models for Sequence Tagging, August 2015. http://arxiv.org/abs/1508.01991

13.

Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: PubMedQA: a dataset for biomedical research question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 2567–2577, September 2019. http://arxiv.org/abs/1909.06146

14.

Kim, D., et al.: A neural named entity recognition and multi-type normalization tool for biomedical text mining. IEEE Access 7, 73729–73740 (2019). https://doi.org/10.1109/ACCESS.2019.2920708. https://ieeexplore.ieee.org/document/8730332/, ISSN 2169-3536CrossRef

15.

Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260–270. Association for Computational Linguistics, Stroudsburg (2016). https://doi.org/10.18653/v1/N16-1030, http://aclweb.org/anthology/N16-1030

16.

Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (btz682) (2019). https://doi.org/10.1093/bioinformatics/xxxxxx, https://github.com/dmis-lab/biobert

17.

Liu, L., et al.: Empower sequence labeling with task-aware neural language model, September 2017. http://arxiv.org/abs/1709.04109

18.

Liu, S., Johns, E., Davison, A.J.: End-to-End Multi-Task Learning with Attention, March 2018. http://arxiv.org/abs/1803.10704

19.

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp. 3111–3119 (2013). https://arxiv.org/pdf/1310.4546.pdf

20.

Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014). https://nlp.stanford.edu/pubs/glove.pdf

21.

Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365, February 2018. http://arxiv.org/abs/1802.05365

22.

Ruder, S.: An overview of multi-task learning in deep neural networks, June 2017. http://arxiv.org/abs/1706.05098

23.

Sarkar, P., Ross, K., Ruberto, A.J., Rodenburg, D., Hungler, P., Etemad, A.: Classification of cognitive load and expertise for adaptive simulation using deep multitask learning, July 2019. http://arxiv.org/abs/1908.00385

24.

Sharma, S., Daniel, R.: BioFLAIR: pretrained pooled contextualized embeddings for biomedical sequence labeling tasks. arXiv preprint arXiv:1908.05760, August 2019. http://arxiv.org/abs/1908.05760

25.

Sachan, D.S., Xie, P., Sachan, M., Xing, E.P.: Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. Technical report (2018). https://arxiv.org/pdf/1711.07908.pdf

26.

Wang, X., Lyu, J., Dong, L., Xu, K.: Multitask learning for biomedical named entity recognition with cross-sharing structure. BMC Bioinformatics 20(1), 427 (2019). https://doi.org/10.1186/s12859-019-3000-5. ISSN 14712105CrossRefPubMedPubMedCentral

27.

Wang, X., et al.: Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics 35(10), 1745–1752 (2018). https://doi.org/10.1093/bioinformatics/xxxxxx. https://github.com/yuzhimanhua/lm-lstm-crfCrossRef

28.

Watanabe, T., Tamura, A., Ninomiya, T., Makino, T., Iwakura, T.: Multi-task learning for chemical named entity recognition with chemical compound paraphrasing. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 6243–6248 (2019). https://pubchem.ncbi.nlm.nih.gov/

29.

Yoon, W., So, C.H., Lee, J., Kang, J.: CollaboNet: collaboration of deep neural networks for biomedical named entity recognition (2019). https://doi.org/10.1186/s12859-019-2813-6

30.

Zhao, S., Liu, T., Zhao, S., Wang, F.: A Neural multi-task learning framework to jointly model medical named entity recognition and normalization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 817–824 (2019). https://doi.org/10.1609/aaai.v33i01.3301817, https://aaai.org/ojs/index.php/AAAI/article/view/3861, ISSN 2374–3468

Titel: LM-Based Word Embeddings Improve Biomedical Named Entity Recognition: A Detailed Analysis
verfasst von: Liliya Akhtyamova
John Cardiff
Verlag: Springer International Publishing
Buch: Bioinformatics and Biomedical Engineering
Print ISBN: 978-3-030-45384-8

Electronic ISBN: 978-3-030-45385-5

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-45385-5_56

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"