Visual Language Theory: Towards a Human-Computer Interaction Perspective

The main reason for using visual languages is that they are often far more convenient to the user than traditional textual languages. Therefore, visual languages intended for use by both computers and humans ought to be designed and analyzed not only from the perspective of computational resource requirements, but also from the perspective of languages that are cognitively usable and useful. Theoretical and practical research on visual languages needs to take into account the full context of a coupled human-computer system in which the visual language facilitates interactions between the computational and the cognitive parts. This implies that theoretical analyses ought to address issues of comprehension, reasoning, and interaction in the cognitive realm as well as issues of visual program parsing, execution, and feedback in the computational realm. The human aspect is crucial to visual languages, and therefore we advocate a correspondingly broadened scope of inquiry for visual language research. In this chapter we describe aspects of human use of visual languages that ought to be important considerations in visual language research and design, and summarize research from related fields such as software visualization and diagrammatic reasoning that addresses these issues. A framework consistent with the broadened scope of visual language research is proposed and used to categorize and discuss several formalizations and implemented systems. In the course of showing how a sample of current work fits into this framework, open issues and fruitful directions for future research are also identified.