Top

Published in:

2021 | OriginalPaper | Chapter

You Can’t Learn What’s Not There: Self Supervised Learning and the Poverty of the Stimulus

Authors : Csaba Veres, Jennifer Sampson

Published in: Natural Language Processing and Information Systems

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Diathesis alternation describes the property of language that individual verbs can be used in different subcategorization frames. However, seemingly similar verbs such as drizzle and spray can behave differently in terms of the alternations they can participate in (drizzle/spray water on the plant; *drizzle/spray the plant with water). By hypothesis, primary linguistic data is not sufficient to learn which verbs alternate and which do not. We tested two state-of-the-art machine learning models trained by self supervision, and found little evidence that they could learn the correct pattern of acceptability judgement in the locative alternation. This is consistent with a poverty of stimulus argument that primary linguistic data does not provide sufficient information to learn aspects of linguistic knowledge. The finding has important consequences for machine learning models trained by self supervision, since they depend on the evidence present in the raw training input.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

next chapter Scaling Federated Learning for Fine-Tuning of Large Language Models

https://www.kaggle.com/c/cola-in-domain-open-evaluation/leaderboard.

Baker, C.: Syntactic theory and the projection problem. Linguist. Inquiry 10, 533–581 (1979)

Bender, E.M., Koller, A.: Climbing towards NLU: on meaning, form, and understanding in the age of data. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185–5198. Association for Computational Linguistics, July 2020. https://www.aclweb.org/anthology/2020.acl-main.463

Berwick, R.C., Pietroski, P., Yankama, B., Chomsky, N.: Poverty of the stimulus revisited. Cogn. Sci. 35(7), 1207–1242 (2011). https://doi.org/10.1111/j.1551-6709.2011.01189.x. ISSN 1551-6709CrossRef

Chmomsky, N.: Aspects of the Theory of Syntax. MIT Press, Cambridge (1965)

Chomsky, N.: Rules and Representations. Columbia Classics in Philosophy. Columbia University Press (1980). https://books.google.no/books?id=KdYOYJwjFo0C. ISBN 9780231048279

Cowie, F.: Innateness and language. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2017 edn (2017)

Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) NAACL-HLT (1), pp. 4171–4186. Association for Computational Linguistics (2019). http://dblp.uni-trier.de/db/conf/naacl/naacl2019-1.html#DevlinCLT19. ISBN 978-1-950737-13-0

Ettinger, A.: What BERT is not: lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Linguist. 8, 34–48 (2020). https://doi.org/10.1162/tacl_a_00298CrossRef

Harris, Z.S.: Distributional structure. WORD 10(2–3), 146–162 (1954). https://doi.org/10.1080/00437956.1954.11659520. ISSN 0043-7956CrossRef

10.

Kassner, N., Schütze, H.: Negated and misprimed probes for pretrained language models: birds can talk, but cannot fly. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7811–7818. Association for Computational Linguistics, July 2020. https://www.aclweb.org/anthology/2020.acl-main.698

11.

Levin, B.: English Verb Classes and Alternations: A Preliminary Investigation. The University of Chicago Press, The University of Chicago (1993). ISBN 0-226-47532-8

12.

Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692

13.

Matthews, B.W.: Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Struct. 405(2), 442–451 (1975). https://doi.org/10.1016/0005-2795(75)90109-9. http://www.sciencedirect.com/science/article/pii/0005279575901099. ISSN 0005-2795CrossRef

14.

Perfors, A., Tenenbaum, J.B., Wonnacott, E.: Variability, negative evidence, and the acquisition of verb argument constructions. J. Child Lang. 37(3), 607–642 (2010). https://doi.org/10.1017/S0305000910000012CrossRef

15.

Pinker, S.: Learnability and Cognition: The Acquisition of Argument Structure (1989/2013), New edn. MIT Press, Cambridge (2013)

16.

Pinker, S.: The Stuff of Thought : Language as a Window Into Human Nature. Viking, New York (2007)

17.

Sahlgren, M.: The distributional hypothesis. Italian J. Linguist. 20, 33–53 (2008)

18.

Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, Red Hook, NY, USA, pp. 6000–6010. Curran Associates Inc. (2017). ISBN 9781510860964

19.

Veres, C., Sandblåst, B.H.: A machine learning benchmark with meaning: learnability and verb semantics. In: Liu, J., Bailey, J. (eds.) AI 2019. LNCS (LNAI), vol. 11919, pp. 369–380. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-35288-2_30CrossRef

20.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: The Proceedings of ICLR (2019)

21.

Wang, A., et al.: SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019)

22.

Wang, W., et al.: Structbert: Incorporating language structures into pre-training for deep language understanding (2019)

23.

Warstadt, A., Singh, A., Bowman, S.R.: Neural Network Acceptability Judgments (2018)

24.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237 (2019). http://arxiv.org/abs/1906.08237

Title: You Can’t Learn What’s Not There: Self Supervised Learning and the Poverty of the Stimulus
Authors: Csaba Veres
Jennifer Sampson
Publisher: Springer International Publishing
Book: Natural Language Processing and Information Systems
Print ISBN: 978-3-030-80598-2

Electronic ISBN: 978-3-030-80599-9

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-3-030-80599-9_1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner