Where do Phrases Come from: Some Preliminary Experiments in Connectionist Phrase Generation

Some of the assumptions that were inherent in many of our early models of language generation (assumptions such as the unidirectional flow of information from conceptualization to verbalization, the serial processing of semantic, lexical, syntactic and phonetic information, and the importance of the word as the lexical unit) are coming under closer scrutiny within the symbolic modeling paradigm. At the same time, interest in another paradigm of cognitive modeling, one that has been variously referred to as neural network modeling, parallel distributed processing, and connectionism, has been growing. Certain characteristics of connectionist models, in particular their broad bandwidth information channels which permit parallel, multidirectional information flow, seem especially relevant to current language generation problems. Other useful properties of these models, such as their content addressable memory capacity, their ability to respond to novelty, their capacity to degrade gracefully and the existence of automatic learning procedures, would contribute to the solutions of other language generation problems. This paper gives an overview of the connectionist paradigm and a brief summary of some of the language processing research being done under it. It then describes the results of some experiments aimed at training connectionist networks to perform two language generation tasks. In one study a network was trained to produce appropriate words given a set of semantic attributes, and in another study a network was trained to produce syntactically well-formed phrases given the unordered words. The networks finally did achieve their intended behaviors, and in doing so they exhibited evidence of having acquired linguistic knowledge of synonymy relations and collocation relations in addition to proper semantic associations. Many of the errors made by the networks suggested revisions to network configurations and encoding schemes, and the effects of those revisions are also briefly described.