Language-Picture Question-Answering Through Common Semantic Representation And its Application to the World of Weather Report

This paper presents some methodologies for constructing systems which can understand both natural languages and pictorial patterns, and describes an application system of the weather report world.The methodologies concern the preliminary analyses of linguistic and pictorial data, and the configuration of the understanding system. The data analyses are performed systematically by corresponding pictorial data to linguistic ones from the viewpoints of syntax and semantics. The understanding system which we intend consists of six divisions: Sentence Analysis Division, Picture Analysis Division, Meaning Analysis Division, Meaning Synthesis Division, Sentence Synthesis Division, and Picture Synthesis Division. All these divisions are connected through the medium of meaning representation common to language and picture.The application system is titled as ISOBAR(an Information understanding System Of BAsic weather Report). It can accept both linguistic and pictorial inputs. Also it produces either linguistic or pictorial, or both, output according to the commands which it receives. The most remarkable point of the system is that the performance of the system is based on the semantic processing of the input. As ISOBAR’s world is limited within the weather report of Japan and Far East Asian areas with associated weather charts, the semantic background of the system is very narrow, and is quite a specific one. But the methodologies and algorithms in the system may be valid for more complicated and general systems. ISOBAR has two operating modes in principle. One is the accumulation of meteorological information, and the other is its retrieval. According to the experimental outcomes, the performance of the system is almost good except for the computing cost of the picture processing procedures. Finally, the problems for the future research are remarked.