nach oben

Neuroinformatics

Erschienen in:

01.01.2014 | Original Article

Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism

verfasst von: Victor Barrès, Jinyong Lee

Erschienen in: Neuroinformatics | Ausgabe 1/2014

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

How does the language system coordinate with our visual system to yield flexible integration of linguistic, perceptual, and world-knowledge information when we communicate about the world we perceive? Schema theory is a computational framework that allows the simulation of perceptuo-motor coordination programs on the basis of known brain operating principles such as cooperative computation and distributed processing. We present first its application to a model of language production, SemRep/TCG, which combines a semantic representation of visual scenes (SemRep) with Template Construction Grammar (TCG) as a means to generate verbal descriptions of a scene from its associated SemRep graph. SemRep/TCG combines the neurocomputational framework of schema theory with the representational format of construction grammar in a model linking eye-tracking data to visual scene descriptions. We then offer a conceptual extension of TCG to include language comprehension and address data on the role of both world knowledge and grammatical semantics in the comprehension performances of agrammatic aphasic patients. This extension introduces a distinction between heavy and light semantics. The TCG model of language comprehension offers a computational framework to quantitatively analyze the distributed dynamics of language processes, focusing on the interactions between grammatical, world knowledge, and visual information. In particular, it reveals interesting implications for the understanding of the various patterns of comprehension performances of agrammatic aphasics measured using sentence-picture matching tasks. This new step in the life cycle of the model serves as a basis for exploring the specific challenges that neurolinguistic computational modeling poses to the neuroinformatics community.

Vorheriger Artikel Towards a Computational Model of Actor-Based Language Comprehension

Nächster Artikel Action and Language Mechanisms in the Brain: Data, Models and Neuroinformatics

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Aine, C. J., Sanfratello, L., Ranken, D., Best, E., MacArthur, J. A., et al. (2012). MEG-SIM: a web portal for testing MEG analysis methods using realistic simulated and empirical data. Neuroinformatics, 10, 141–158.PubMedCentralPubMedCrossRef

Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition, 73, 247–264.PubMedCrossRef

Ansell, B. J., & Flowers, C. R. (1982). Aphasic adults’ use of heuristic and structural linguistic cues for sentence analysis. Brain and Language, 16, 61–72.PubMedCrossRef

Arbib, M. A. (1981). Perceptual structures and distributed motor control. In V. B. Brooks (Ed.), Handbook of physiology — The nervous system II. Motor control (pp. 1449–1480). Bethesda: American Physiological Society.

Arbib, M. A. (2012). How the brain got language: the mirror system hypothesis. New York & Oxford: Oxford University Press.

Arbib, M. A., & Bota, M. (2003). Language evolution: neural homologies and neuroinformatics. Neural Networks: The Official Journal of the International Neural Network Society, 16, 1237–1260.CrossRef

Arbib, M. A., & Caplan, D. (1979). Neurolinguistics must be computational. The Behavioral and Brain Sciences, 2, 449–483.CrossRef

Arbib, M. A., & Lee, J. Y. (2008). Describing visual scenes: towards a neurolinguistics based on construction grammar. Brain Research, 1225, 146–162.PubMedCrossRef

Arbib, M. A., Conklin, E. J., & Hill, J. C. (1987). From schema theory to language. New York: Oxford University Press. x + 253 pp.

Arbib, M. A., Billard, A., Iacoboni, M., & Oztop, E. (2000). Synthetic brain imaging: grasping, mirror neurons and imitation. Neural Networks: The Official Journal of the International Neural Network Society, 13, 975–997.CrossRef

Arbib, M. A., Plangprasopchok, A., Bonaiuto, J. J., Schuler, R. E. (2013). A neuroinformatics of brain modeling and its implementation in the Brain Operation Database BODB. Neuroinformatics, in press.

Baker, C., Fillmore, C. J., Lowe, J. B. (1998). The {B}erkeley {F}rame{N}et project.

Bakker, R., Wachtler, T., Diesmann, M. (2012). CoCoMac 2.0 and the future of tract-tracing databases. Frontiers in Neuroinformatics, 6.

Barrès, V., Simons, A., & Arbib, M. A. (2013). Synthetic event-related potentials: a computational bridge between neurolinguistic models and experiments. Neural Networks, 37, 66–92.PubMedCrossRef

Bergen B. K, Chang N. (2005a). Embodied construction grammar in simulation-based language understanding. In Construction Grammars: cognitive grounding and theoretical extensions.

Bergen, B. K., & Chang, N. (2005b). Embodied construction grammar in simulation-based language understanding. In J.-O. OÖstman & M. Fried (Eds.), Construction grammar(s): cognitive and cross-language dimensions (pp. 147–190). Amsterdam: John Benjamins.

Berndt, R. S., & Caramazza, A. (1999). How “regular” is sentence comprehension in Broca's aphasia? It depends on how you select the patients. Brain and Language, 67, 242–247.PubMedCrossRef

Berndt, R. S., Mitchum, C. C., & Haendiges, A. N. (1996). Comprehension of reversible sentences in “agrammatism”: a meta-analysis. Cognition, 58, 289–308.PubMedCrossRef

Bornkessel, I., & Schlesewsky, M. (2006). The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. Psychological Review, 113, 787–821.PubMedCrossRef

Caramazza, A., & Zurif, E. B. (1976). Dissociation of algorithmic and heuristic processes in language comprehension: evidence from aphasia. Brain and Language, 3, 572–582.PubMedCrossRef

Caramazza, A., Capasso, R., Capitani, E., & Miceli, G. (2005). Patterns of comprehension performance in agrammatic Broca’s aphasia: a test of the trace deletion hypothesis. Brain and Language, 94, 43–53.PubMedCrossRef

Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology Learning, Memory, and Cognition, 30, 687–696.PubMedCrossRef

Chen, R., Hillis, A. E., Pawlak, M., & Herskovits, E. H. (2008). Voxelwise Bayesian lesion-deficit analysis. NeuroImage, 40, 1633–1642.PubMedCentralPubMedCrossRef

Christianson, K., & Luke, S. G. (2011). Context strengthens initial misinterpretations of text. Scientific Studies of Reading, 15, 136–166.CrossRef

Christianson, K., Hollingworth, A., Halliwell, J. F., & Ferreira, F. (2001). Thematic roles assigned along the garden path linger. Cognitive Psychology, 42, 368–407.PubMedCrossRef

Croft, W. (2001). Radical construction grammar: syntactic theory in typological perspective. Oxford: Oxford University Press.CrossRef

Croft, W., & Cruse, D. A. (2005). Cognitive linguistics. Cambridge: Cambridge University Press.

David, O., Kiebel, S. J., Harrison, L. M., Mattout, J., Kilner, J. M., & Friston, K. J. (2006). Dynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30, 1255–1272.PubMedCrossRef

De Beule, J., & Steels, L. (2005). Hierarchy in fluid construction grammar. In U. Furbach (Ed.), Proceedings of the 28th annual German conference on AI, KI 2005, lecture notes in artificial intelligence (vol. 3698) (pp. 1–15). Berlin, Heidelberg: Springer.

Dominey, P. F., & Boucher, J.-D. (2005). Learning to talk about events from narrated video in a construction grammar framework. Artificial Intelligence, 167, 31–61.CrossRef

Dominey, P. F., Hoen, M., & Inui, T. (2006a). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.PubMedCrossRef

Dominey, P. F., Hoen, M., & Inui, T. (2006b). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.PubMedCrossRef

Dou, D., Frishkoff, G., Rong, J., Frank, R., Malony, A., & Tucker, D. (2007). Development of NeuroElectroMagnetic Ontologies(NEMO): a framework for mining brainwave ontologies. New York: Assoc Computing Machinery.

Draper, B. A., Collins, R. T., Brolio, J., Hanson, A. R., & Riseman, E. M. (1989). The schema system. International Journal of Computer Vision, 2, 209–250.CrossRef

Evans. A. C., Collins, D. L., Mills, S. R., Brown, E. D., Kelly, R. L., Peters, T. M. (1993). 1813–17 vol.3-13 -17 vol.3.

Fellbaum, C. (2010). WordNet. In R. Poli, M. Healy, & A. Kameas (Eds.), Theory and applications of ontology: computer applications (pp. 231–43). Springer: Netherlands.CrossRef

Ferreira, F. (2003). The misinterpretation of noncanonical sentences. Cognitive Psychology, 47, 164–203.PubMedCrossRef

Ferreira, F., & Patson, N. D. (2007). The ‘good enough’ approach to language comprehension. Language and Linguistics Compass, 1, 71–83.CrossRef

Fox, P. T., & Lancaster, J. L. (2002). Mapping context and content: the BrainMap model. Nature Reviews Neuroscience, 3, 319–321.PubMedCrossRef

Fox, P. T., Laird, A. R., Fox, S. P., Fox, P. M., Uecker, A. M., et al. (2005). Brainmap taxonomy of experimental design: description and evaluation. Human Brain Mapping, 25, 185–198.PubMedCrossRef

Frazier, L., & Fodor, J. D. (1978). The sausage machine: a new two-stage parsing model. Cognition, 6, 291–325.CrossRef

Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78–84.PubMedCrossRef

Friederici, A. D. (2009). Pathways to language: fiber tracts in the human brain. Trends in Cognitive Sciences, 13, 175–181.PubMedCrossRef

Gleitman, L. R., January, D., Nappa, R., & Trueswell, J. C. (2007). On the give and take between even apprehension and utterance formulation. Journal of Memory and Language, 57, 544–569.PubMedCentralPubMedCrossRef

Goldberg, A. E. (1995). Constructions: a construction grammar approach to argument structure. Chicago: The University of Chicago Press.

Goodglass, H. (1976). Agrammatism. Studies in Neurolinguistics, 1, 237–260.

Grodzinsky, Y. (2000). The neurology of syntax: language use without Broca’s area. The Behavioral and Brain Sciences, 23, 1–21.PubMedCrossRef

Grodzinsky, Y., Piñango, M. M., Zurif, E., & Drai, D. (1999). The critical role of group studies in neuropsychology: comprehension regularities in Broca’s Aphasia. Brain and Language, 67, 134–147.PubMedCrossRef

Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sciences, 9, 416–423.PubMedCrossRef

Hanson, A. R., & Riseman, E. M. (1978). VISIONS: a computer system for interpreting scenes. In A. R. Hanson & E. M. Riseman (Eds.), Computer vision systems (pp. 129–163). New York: Academic.

Hawkins, J. A. (1999). Processing complexity and Filler-Gap dependencies across grammars. Language & Cognitive Processes, 75, 244–285.

Hickok, G., & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67–99.PubMedCrossRef

Hurford, J. R. (2011). The origins of grammar II: language in the light of evolution. Oxford, New York: Oxford University Press.

Kay, P. (2002). An informal sketch of the formal architecture of construction grammar. Grammars, 5, 1–19.CrossRef

Kay, P., Fillmore, C. J. (1999). Grammatical constructions and linguistic generalizations: the what’s X doing Y? Construction.

Kemmerer, D. (2000a). Grammatically relevant and grammatically irrelevant features of verb meaning can be independently impaired. Aphasiology, 14, 997–1020.CrossRef

Kemmerer, D. (2000b). Selective impairment of knowledge underlying prenominal adjective order: evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 13, 57–82.CrossRef

Kemmerer, D. (2003). Why can you hit someone on the arm but not break someone on the arm?—a neuropsychological investigation of the English body-part possessor ascension construction. Journal of Neurolinguistics, 16, 13–36.CrossRef

Kemmerer, D., & Wright, S. K. (2002). Selective impairment of knowledge underlying un- prefixation: further evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 15, 403–432.CrossRef

Kemmerer, D., Tranel, D., & Zdanczyk, C. (2009). Knowledge of the semantic constraints on adjective order can be selectively impaired. Journal of Neurolinguistics, 22, 91–108.PubMedCentralPubMedCrossRef

Kempen, G., Olsthoorn, N., & Sprenger, S. (2012). Grammatical workspace sharing during language production and language comprehension: evidence from grammatical multitasking. Language & Cognitive Processes, 27, 345–380.CrossRef

Kim, & Osterhout. (2005). The independence of combinatory semantic processing: evidence from event-related potentials. Journal of Memory and Language, 52, 205–225.CrossRef

Kos, M., Vosse, T., Dvd, B., & Hagoort, P. (2010). About edible restaurants: conflicts between syntax and semantics as revealed by ERPs. Frontiers in Language Sciences, 1, 222–22.

Kudo, T. (1984). The effect of semantic plausibility on sentence comprehension in aphasia. Brain and Language, 21, 208–218.PubMedCrossRef

Kuperberg, G. R. (2007). Neural mechanisms of language comprehension: challenges to syntax. Brain Research, 1146, 23–49.PubMedCrossRef

Laird, A. R., Eickhoff, S. B., Kurth, F., Fox, P. M., Uecker, A. M., et al. (2009). ALE meta-analysis workflows via the brainmap database: progress towards a probabilistic functional brain atlas. Frontiers in Neuroinformatics, 3, 23–23.PubMedCentralPubMedCrossRef

Lee J. (2012). Linking eyes to mouth: a schema-based computational model for describing visual scenes. Ph.D. Thesis, Computer Science, University of Southern California, Los Angeles, CA.

Lee J. (In preparation-a). Implementing Template Construction Grammar (TCG) for visual scene description.

Lee J. (In preparation-b). The temporal unfolding of eye movements and utterance formulation.

Lesser, V. R., Fennel, R. D., Erman, L. D., & Reddy, D. R. (1975). Organization of the HEARSAY-II speech understanding system. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23, 11–23.CrossRef

Letovsky, S. I., Whitehead, S. H., Paik, C. H., Miller, G. A., Gerber, J., et al. (1998). A brain image database for structure/function analysis. AJNR. American Journal of Neuroradiology, 19, 1869–1877.PubMed

Levin B. (1993). English verb classes and alternations: a preliminary investigation. University of Chicago Press.

Luria, A. R. (1973). The working brain. Harmondsworth: Penguin.

MacWhinney, B. (2007). The TalkBank Project.

MacWhinney, B., Fromm, D., Forbes, M., & Holland, A. (2011). AphasiaBank: methods for studying discourse. Aphasiology, 25, 1286–1307.PubMedCentralPubMedCrossRef

Makkai, A. (1972). Idiom structure in English

Marcus, D. S., Harwell, J., Olsen, T., Hodge, M., Glasser, M. F., et al. (2011). Informatics and data mining tools and strategies for the human connectome project. Frontiers in Neuroinformatics, 5.

Mayberry, M., Crocker, M. W., Knoeferle, P. (2006). A connectionist model of the coordinated interplay of scene, utterance, and world knowledge.

Menenti, L., Gierhan, S. M. E., Segaert, K., & Hagoort, P. (2011). Shared language overlap and segregation of the neuronal infrastructure for speaking and listening revealed by functional MRI. Psychological Science, 22, 1173–1182.PubMedCrossRef

Miyake, A., Carpenter, P. A., & Just, M. A. (1994). A capacity approach to syntactic comprehension disorders: making normal adults perform like aphasic patients. Cognitive Neuropsychology, 11, 671–717.CrossRef

Miyake, A., Carpenter, P. A., & Just, M. A. (1995). Reduced resources and specific impairments in normal and aphasic sentence comprehension. Cognitive Neuropsychology, 12, 651–679.CrossRef

Mohanan, T., Wee, L. (1999). Grammatical semantics: evidence for structure in meaning. CSLI.

Osterhout, L., Albert, K., Kuperberg, G. (2007). The neurobiology of sentence comprehension, CiteSeerX.

Pinker, S. (1989). Learnability and cognition: the acquisition of argument structure. Cambridge: The MIT Press. 411 pp.

Poeppel, D., Emmorey, K., Hickok, G., & Pylkkänen, L. (2012). Towards a new neurobiology of language. The Journal of Neuroscience, 32, 14125–14131.PubMedCentralPubMedCrossRef

Saffran, E. M., Schwartz, M. F., & Linebarger, M. C. (1998). Semantic influences on thematic role assignment: evidence from normals and aphasics. Brain and Language, 62, 255–297.PubMedCrossRef

Schuler, K. K. (2005). VerbNet: a broad-coverage, comprehensive verb lexicon. University of Pennsylvania

Schwartz, M. F., Linebarger, M. C., Saffran, E. M., & Pate, D. S. (1987). Syntactic transparency and sentence interpretation in aphasia. Language & Cognitive Processes, 2, 85–113.CrossRef

Segaert, K., Menenti, L., Weber, K., Petersson, K. M., & Hagoort, P. (2012). Shared syntax in language production and language comprehension—an fMRI study. Cerebral Cortex, 22, 1662–1670.PubMedCrossRef

Sherman, J. C., & Schweickert, J. (1989). Syntactic and semantic contributions to sentence comprehension in agrammatism. Brain and Language, 37, 419–439.PubMedCrossRef

Spivey, M. J., Richardson, D. C., & Fitneva, S. A. (2005). Thinking outside the brain: spatial indices to visual and linguistic information. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: eye movements and the visual world (pp. 161–190). New York, Hove: Psychology Press.

Sporns, O., Tononi, G., & Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Computational Biology, 1, e42–e42.PubMedCentralPubMedCrossRef

Steels, L. (1999). The talking heads experiment.

Steels, L., & De Beule, J. (2006). Unify and merge in fluid construction grammar. In P. Vogt, Y. Sugita, E. Tuci, & C. Nehaniv (Eds.), Symbol grounding and beyond, proceedings (pp. 197–223). Berlin: Springer-Verlag Berlin.CrossRef

Stefanowitsch, A., & Gries, S. T. (2003). Collostructions: investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8, 209–243.CrossRef

Stephan, K. E., Kamper, L., Bozkurt, A., Burns, G. A. P. C., Young, M. P., & Kotter, R. (2001). Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). Philosophical Transactions of the Royal Society B: Biological Sciences, 356, 1159–1186.CrossRef

Talairach, Tournoux P. (1988). Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: an approach to cerebral imaging. Thieme.

Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.PubMedCrossRef

Van Essen, D. C. (2009). Lost in localization–but found with foci?! NeuroImage, 48, 14–17.PubMedCentralPubMedCrossRef

Vigneau, M., Beaucousin, V., Hervé, P. Y., Duffau, H., Crivello, F., et al. (2006). Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. NeuroImage, 30, 1414–1432.PubMedCrossRef

Vosse, T., & Kempen, G. (2009). In defense of competition during syntactic ambiguity resolution. Journal of Psycholinguistic Research, 38, 1–9.PubMedCentralPubMedCrossRef

Wendel, K., Väisänen, O., Malmivuo. J., Gencer, N. G., Vanrumste. B., et al. (2009). EEG/MEG source imaging: methods, challenges, and open issues. Intelligence and Neuroscience, 2009: 13:1–13:12–13:1–13:12.

Wilbur, R., Kak, A. (2006). Purdue RVL-SLLL American Sign Language Database. ECE Technical Reports.

Zurif, E. B., & Piñango, M. M. (1999). The existence of comprehension patterns in Broca’s Aphasia. Brain and Language, 70, 133–138.PubMedCrossRef

Titel: Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism
verfasst von: Victor Barrès
Jinyong Lee
Publikationsdatum: 01.01.2014
Verlag: Springer US
Erschienen in: Neuroinformatics / Ausgabe 1/2014
Print ISSN: 1539-2791
Elektronische ISSN: 1559-0089
DOI: https://doi.org/10.1007/s12021-013-9197-y

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 1/2014

An Informatics Approach to Integrating Genetic and Neurological Data in Speech and Language Neuroscience

Action, Language and Neuroinformatics: An Introduction

Information Processing in the Mirror Neuron System in Primates and Machines

Action and Language Mechanisms in the Brain: Data, Models and Neuroinformatics

A Neuroinformatics of Brain Modeling and its Implementation in the Brain Operation Database BODB

Ontogenetic Ritualization of Primate Gesture as a Case Study in Dyadic Brain Modeling