Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
FAIR Principles and Digital Objects: Accelerating Convergence on a Data Infrastructure Read chapter Read first chapter

2019 | OriginalPaper | Chapter

Scope and Challenges of Language Modelling - An Interrogative Survey on Context and Embeddings

Authors : Matthias Nitsche, Marina Tropmann-Frick

Published in: Data Analytics and Management in Data Intensive Domains

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this work we explore the domain of Language Modelling. We focus here on different context selection strategies, data augmentation techniques, and word embedding models. Many of the existing approaches are difficult to understand without specific expertise in this domain. Therefore, we concentrate on appropriate explanations and representations that enable us to compare several approaches.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Proximity Full-Text Search by Means of Additional Indexes with Multi-component Keys: In Pursuit of Optimal Performance
next chapter Scalable Algorithm for Subsequence Similarity Search in Very Large Time Series Data on Cluster of Phi KNL
Literature
1.
Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings for sequence labeling. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1638–1649 (2018)
2.
Athiwaratkun, B., Wilson, A.: Multimodal word distributions. In: Conference of the Association for Computational Linguistics (ACL) (2017)
3.
Bjerva, J., Östling, R., Han Veiga, M., Tiedemann, J., Augenstein, I.: What do language representations really represent? Comput. Linguist. 1–8 (2019, Just Accepted)
4.
Blei, D., Ng, A., Jordan, M.: Latent dirichlet allocation. J. Mach. Learn. Res
5.
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. CoRR
6.
Bolukbasi, T., Chang, K., Zou, J., Saligrama, V., Kalai, A.: Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. CoRR
7.
Council, N.R., Committee, A.L.P.A.: Language and machines: computers in translation and linguistics, a report. In: National Academy of Sciences, National Research Council (1966)
8.
Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inform. Sci. 41(6), 391–407 (1990)CrossRef
9.
Dhingra, B., Liu, H., Salakhutdinov, R., Cohen, W.: A comparative study of word embeddings for reading comprehension. CoRR
10.
Dyer, C.: Notes on noise contrastive estimation and negative sampling. CoRR
11.
Gittens, A., Achlioptas, D., Mahoney, M.: Skip-gram - zipf + uniform = vector additivity. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Canada, vol. 1. pp. 69–76 (2017)
12.
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 5–6 (2005)
13.
Herbelot, A., Baroni, M.: High-risk learning: acquiring new word vectors from tiny data. CoRR
14.
Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. CoRR
15.
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y.: Exploring the limits of language modeling. CoRR
16.
Kim, Y., Jernite, Y., Sontag, D., Rush, A.: Character-aware neural language models. CoRR
17.
Kneser, R., Ney, H.: Improved clustering techniques for class-based statistical language modelling. In: Third European Conference on Speech Communication and Technology (1993)
18.
Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Advances in Neural Information Processing Systems, vol. 27
19.
20.
Manning, C.D., Schuetze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (1999)
21.
McCann, B., Bradbury, J., Xiong, C., Socher, R.: Learned in translation: contextualized word vectors. CoRR
22.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. CoRR abs/1310.4546 (2013)
23.
Mimno, D., Thompson, L.: The strange geometry of skip-gram with negative sampling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, September 2017
24.
Mnih, A., Hinton, G.: Three new graphical models for statistical language modelling. In: Proceedings of the 24th International Conference on Machine Learning, pp. 641–648. ACM (2007)
25.
Ney, H., Essen, U., Kneser, R.: On structuring probabilistic dependences in stochastic language modelling. Comput. Speech Lang. 8(1), 1–38 (1994)CrossRef
26.
Nickel, M., Kiela, D.: Poincaré embeddings for learning hierarchical representations. CoRR
27.
Nitsche, M., Tropmann-Frick, M.: Context and embeddings in language modelling - an exploration. In: Selected Papers of the XX International Conference on Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2018), Moscow, Russia, 9–12 October 2018, pp. 131–138 (2018). http://​ceur-ws.​org/​Vol-2277/​paper24.​pdf
28.
29.
30.
Pinter, Y., Guthrie, R., Eisenstein, J.: Mimicking word embeddings using subword RNNs. CoRR
31.
32.
33.
Srivastava, R., Greff, K., Schmidhuber, J.: Highway networks. CoRR
34.
Tissier, J., Gravier, C., Habrard, A.: Dict2vec: learning word embeddings using lexical dictionaries. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 9–11 September 2017, pp. 254–263 (2017)
35.
36.
Vilnis, L., McCallum, A.: Word representations via gaussian embedding. CoRR
37.
Wieting, J., Bansal, M., Gimpel, K., Livescu, K.: Charagram: Embedding words and sentences via character n-grams. CoRR
38.
Winograd, T.: Understanding natural language. Cogn. Psychol. 3(1), 1–191 (1972)CrossRef
Metadata
Title
Scope and Challenges of Language Modelling - An Interrogative Survey on Context and Embeddings
Authors
Matthias Nitsche
Marina Tropmann-Frick
Copyright Year
2019
Book
Data Analytics and Management in Data Intensive Domains

Print ISBN: 978-3-030-23583-3

Electronic ISBN: 978-3-030-23584-0

Copyright Year: 2019

DOI
https://doi.org/10.1007/978-3-030-23584-0_8

Premium Partner

    Image Credits