Diagrammatic Representation and Inference

One might have thought that the issue of use of mental images in reasoning has been put to rest after years of debate, but one would be wrong. The journal Behavioral and Brain Sciences will soon publish a paper by Zenon Pylyshyn, which restates his earlier thesis - this time as a null hypothesis that cannot yet be rejected - that images do not play any role in reasoning, and that information which people often think they get from images when reasoning is really tacit knowledge that they already have. He does not compare the case of use of mental images with the case when the reasoner uses an external diagram for the same problem, so it is not clear if in those cases also he would claim that the diagram does not play a role. In any case, one might think that this is an issue for psychology, but not for artificial intelligence. However, conceptually, unless one has a robot with a vision system and the robot draws a diagram on a surface and uses its visual system to extract information as it reasons about a situation, any claim that a computer program performs diagrammatic reasoning in some situation has a status similar to that of the claims of human use of mental imagery during reasoning. In parallel with skeptics such as Pylyshyn of the role of images in reasoning, there are those in artificial intelligence who claim that the idea of diagrammatic reasoning by a computer is incoherent: what it means for the internal representation to be an image or a depiction is not clear, and the objections to “seeing” or perceiving information in such an image apply equally to the human and the computer cases. I do not think that the idea of computers performing diagrammatic reasoning is incoherent. I will argue my position on this by characterizing diagrammatic reasoning functionally. Then, I will argue that a computer program can be said to be performing this kind of reasoning if there exists a coherent and non-vacuous level of description of the operations of the program that satisfies the functional properties. Along the way, I will identify diagrammatic reasoning as an instance of a larger class of reasoning in which a model of some kind is used to get some information about a situation. Psychology has many more constraints about what sorts of models are available than artificial intelligence, which may be, at least in theory, freer in this regard.

Graphical representations such as maps and diagrams play an important role in everyday communication settings by serving as an effective means of exchanging information. In such communication, graphical representations work not only as “windows” through which we can see the target situations the graphics describe, but also serve as information processing “sites” because we can take advantage of their handiness. Though the final aim of speakers is to exchange information about the target situation, graphical representations are so deeply a part of information processing that they may affect the way people grasp the target situations. This paper presents an empirical investigation of language usage in graphical communication. Drawing on the HCRC Map Task Corpus data, we demonstrate that the existence of graphics affects linguistic expressions of motion when people collaboratively work on a task. This effect demonstrates that the use of graphical representations has an influence on movement event conceptualizations.

Research on graph comprehension has been concerned with relatively low-level information extraction. However, laboratory studies often produce conflicting findings because real-world graph interpretation requires going beyond the data presentation to make inferences and solve problems. Furthermore, in real-world settings, graphical information is presented in the context of relevant prior knowledge. According to our model, knowledge-based graph comprehension involves an interaction of top-down and bottom up processes. Several types of knowledge are brought to bear on graphs: domain knowledge, graphical skills, and explanatory skills. During the initial processing, people chunk the visual features in the graphs. Nevertheless, prior knowledge guides the processing of visual features. We outline the key assumptions of this model and show how this model explains the extant data and generates testable predictions.

Diagrams play an important role in human problem solving. In response to a challenging assignment, three students produced diagrams and subsequent verbal protocols that offer insight into human cognition. The diversity and richness of their response, and their ability to address the task via diagrams, provide an incisive look at the role diagrams play in the development of expertise. This paper recounts how their diagrams led and misled them, and how the diagrams both explained and drove explanation. It also considers how this process might be adapted for a computer program.

Diagrams have many uses in mathematics, one of the most ambitious of which is as a form of proof. The domain we consider is real analysis, where quantification issues are subtle but crucial. Computers offer new possibilities in diagrammatic reasoning, one of which is animation. Here we develop animated rules as a solution to problems of quantification. We show a simple application of this to constraint diagrams, and also how it can deal with the more complex questions of quantification and generalisation in diagrams that use more specific representations. This allows us to tackle difficult theorems that previously could only be proved algebraically.

This article describes an algorithm for the automated generation of any Euler diagram starting with an abstract description of the diagram. An automated generation mechanism for Euler diagrams forms the foundations of a generation algorithm for notations such as Harel’s higraphs, constraint diagrams and some of the UML notation. An algorithm to generate diagrams is an essential component of a diagram tool for users to generate, edit and reason with diagrams.

Euler diagrams use topological properties to represent settheoretical concepts and thus are ‘intuitive’ to some people. When reasoning with Euler diagrams, it is essential to have a notion of correspondence among the regions in different diagrams. At the semantic level, two regions correspond when they represent the same set. However, we wish to construct a purely syntactic definition of corresponding regions, so that reasoning can take place entirely at the diagrammatic level. This task is interesting in Euler diagrams because some regions of one diagram may be missing from another. We construct the correspondence relation from ‘zones’ or minimal regions, introducing the concept of ‘zonal regions’ for the case in which labels may differ between diagrams. We show that the relation is an equivalence relation and that it is a generalization of the counterpart relations introduced by Shin and Hammer.

CDEG (Computerized Diagrammatic Euclidean Geometry) is a computer proof system in which diagrammatic proofs of theorems of Euclidean Geometry can be given formally. The computer system manipulates geometric diagrams using an internal representation that is based on the idea that all the significant information in a geometric diagram is captured by its underlying topology. The proof system that CDEG implements is that of the author’s diagrammatic formal system for geometry, FG. CDEG and FG are strong enough to be able to duplicate most, if not all, of the proofs in the first several books of Euclid’s Elements. This paper explains CDEG and gives a brief example of how it works.

We present a novel approach to the compositional semantics for a large family of diagrams. Examples include engineering drawings such as circuit diagrams, trusses, etc., process control diagrams such as traditional program flowcharts, data-flow diagrams depicting traffic flow or the dependency relationships between operators, etc. Providing a compositional semantics involves defining the meaning of the whole diagram in terms of the meanings of its parts. A common characteristic of these diagrams is that we need to depict both structure and behavior of some artifact, procedure, function, etc. We view these diagrams as graphs that are made up of nodes and edges. The meaning of each diagrammatic element (e.g., node) as well as the connections between elements (e.g., edge) are given in terms of a constrained object [3]. A constrained object is an object whose attributes may be subject to one or more constraints, i.e., relations among the attributes. While the concept of an object and its attributes captures the structural aspects of a diagrammatic element, the concept of constraint captures its behavioral properties. Constrained objects may be thought of as declarative counterparts of the traditional objects found in object-oriented languages.

We present a two-phase retrieval process for finding a target query within a set of 2-D line drawings, where the query is itself a line drawing. We also describe a logic-based matching method that uses unification and resolution to compare the semantic networks representing the spatial structure of two 2-D line drawings.Retrieval of diagrams from an external memory (e.g., a computer-based library of diagrams) is a key problem in reasoning about diagrams. The task in our work assumes a computer-based library of 2-D line drawings, takes as input a representation of a 2-D line drawing as a query, and its desired output is a set of drawings retrieved from the library that are similar to the query (meaning that the target can be found within these sources based on a qualitative highlevel comparison). The goal of the work described in this paper is to develop a retrieval process and a matching procedure for enabling querying by example.

Graphical communication by students can provide important clues to teachers about misconceptions in fields like mathematics and computing. We have built a facility called INFACT-SKETCH that permits conducting experiments in the elicitation and analysis of studentdrawn diagrams. A goal of our project is to achieve the integration of a construction-oriented image processing and programming system with a combination of automatic and manual assessment tools in order to have a highly effective learning environment for information technology concepts. Some of the diagrams we elicit from students represent image processing algorithms, and others represent predictions of what particular mathematical formulas will do to images. In order to serve its educational assessment and research purposes, INFACT-SKETCH provides an unusual combination of features: tight integration with a web-based textual communication system called INFACT-FORUM, administrative control of which drawing tools students will be permitted to use (freehand, rectangles, ovals, text labels, lines, etc.), complete event capture for timed playback of the drawing process for any sketch in the system, graphical quoting option in message replies, and structured annotations for educational assessment. We describe the rationale, intended use, design, and our experience so far with INFACT-SKETCH.

Drawing is a basic but often overlooked mode of human communication. This paper presents a shared whiteboard environment, written in Java, that was designed to be used to collect and analyse data gathered in interactive graphical communication experiments. Users of the software are presented with a ‘virtual’ whiteboard that is connected to another user’s whiteboard to create a shared graphical communication space. In addition to logging all drawing activity between the pair and providing tools for the analysis of this data, the software can manipulate the layout and the degree of interactivity of the drawing being exchanged. The program can also be used to setup and manage multiple simultaneous shared whiteboard connections and subject groupings.

This paper presents a global layout approach used in a generalpurpose visual language generation system system. Our approach is grammarbased graph drawing, in which layout rules are embedded in the productions of reserved graph grammars. Thus, the grammar formalism serves both the visual language grammar and the layout grammar. An example visual language is demonstrated.

We show how Diagrammatic SQL, a query language that can be used to query information represented as diagrams, can be combined with SQL to permit queries on combinations of diagrammatic and symbolic data producing both diagrammatic and symbolic responses.

In the world of diagrammatic notations, the Unified Modeling Language (UML) should be of special interest for cognitive studies. On one hand, UML integrates a host of different diagrammatic languages in real engineering use, and thus studying it should be itself extremely interesting. On the other hand, UML is adopted as a standard in software industry, and has already become a standard de facto in object-oriented analysis and design. This property provides a real practical value for UML studies. However, despite its dramatically increasing popularity, UML’s drawbacks are well known and widely criticized. Currently, users and vendors associate their hopes on a better UML with the next version of the standard, UML 2.0, but it is commonly recognized that the problems standing in front of UML 2.0 are extremely hard [3]. So, careful cognitive analysis of UML appears to be an interesting, beneficial and urgent issue in the diagrammatic world.

Many graphical systems (e.g., Euler diagrams, maps, pictorial images, and even tables) support efficient inferences or rich presentation of information apparently at the expense of expressive flexibility. This association of inferential efficiency, expressive richness, and expressive inflexibility in a graphical system has been pointed out by various researchers (e.g., Sloman [1], Stenning and Oberlander [2]). This paper investigates the semantic mechanism of the association by closely examining a particular system of tabular representations, which, despite its simplicity, clearly exhibits all those opposing functional traits. Using a semantic framework of channel theory (Barwise and Seligman [3]), we will show that the common mechanism is a parallelism between abstraction relations in represented properties and in representing properties.

Reasoning practices and decision making often require information from many different sources, which can be both sentential and diagrammatic. In such situations, there are many advantages to reasoning with the diagrams themselves, as opposed to re-expressing the information content of the diagram in sentential form and reasoning in an abstract sentential language. Thus for these practices, being able to extract and re-express pieces of information from one kind of representation into another is essential. The main goal of this paper is to propose a general framework for the modeling of heterogeneous reasoning systems and, most importantly, heterogeneous rules of inference in those systems. Unlike some other work in designing heterogeneous systems, our purpose will not be to define just one notion of heterogeneous inference, but rather to provide a framework in which many different kinds of heterogeneous rules of inference can be defined. After proposing this framework, we will then show how it can be applied to a sample heterogeneous system to define a number of different heterogeneous rules of inference. We will also discuss how the framework can be used to define rules of inference similar to the Observe Rule in Barwise and Etchemendy’s Hyperproof system.

Rejecting the temptation to make up a list of necessary and sufficient conditions for diagrammatic and sentential systems, we present an important distinction which arises from sentential and diagrammatic features of systems. Importantly, the distinction we will explore in the paper lies at a meta-level. That is, we argue for a major difference in metatheory between diagrammatic and sentential systems, by showing the necessity of a more fine-grained syntax for a diagrammatic system than for a sentential system. Unlike with sentential systems, a diagrammatic system requires two levels of syntax—token and type. Token-syntax is about particular diagrams instantiated on some physical medium, and type-syntax provides a formal definition with which a concrete representation of a diagram must comply. While these two levels of syntax are closely related, the domains of type-syntax and token-syntax are distinct from each other. Euler diagrams are chosen as a case study to illustrate the following major points of the paper: (i) What kinds of diagrammatic features (as opposed to sentential features) require two different levels of syntax? (ii) What is the relation between these two levels of syntax? (iii) What is the advantage of having a two-tiered syntax?

This paper questions how we could and should present diagrams to blind people using computer-generated sound. Using systems that present information about one part of the diagram at a time, rather than the whole, leads to two problems. The first problem is how to present information so that users can integrate the information into a coherent overall picture. The second is how to select the area to be presented. This is looked at by using a system that presents graphs representing central heating system schematics. The system presents information by user choice through either a hierarchical split of information and navigation system, or a connection oriented split of information and navigation system. Further, we have a split as to whether a simple system of presenting location of nodes is used, or not. Tasks, classified as being based on hierarchical information or connection-based information, were set using the system and the effect of the different models was recorded. It was found that there was a match of task to navigation system, but that presentation of position had no discernable effect.

AλgoVista is a web-based search engine that assists programmers to find algorithms and implementations that solve specific problems. AλgoVista is not keyword based but rather requires users to provide — in a very simple textual language — input⇒output samples that describe the behavior of their needed algorithm. Unfortunately, even this simple language has proven too challenging for casual users.

Most software applications present information to the user in a WYSIWYG form, where the main representation on the screen is made as close as possible to a visual facsimile of the final work product. Wherever possible, users specify required transformations of the product by directly selecting and manipulating areas of this visual facsimile. This brings great usability advantages, but is not adequate for the specification of abstract operations such as generalization and inference commands, which are usually represented linguistically in menus and dialogues. We report a series of experimental implementations exploring alternatives to menus and dialogues. In these six systems, abstract functionality is integrated into the work context through two techniques: diagrammatic interpretation of user’s actions (gestures) and diagrammatic overlays superimposed as semi-transparent layers over the visual presentation of the work product. We discuss the diagrammatic justifications and consequences of these alternatives, and present results of preliminary user studies suggesting that both forms of interaction may in future be valuable techniques for exploiting diagrammatic formalisms in software interaction.

How do experienced users extract information from a complex visualization? We examine this question by presenting experienced weather forecasters with visualizations that did not show the needed information explicitly and examining their eye movements. We replicated Carpenter & Shah (1998) when the information was explicitly available on the visualization. However, when the information was not explicitly available, we found that forecasters used spatial reasoning in the form of spatial transformations. We also found a strong imagerial component for constructing meteorological information.

Java program debugging was investigated in programmers who used a software debugging environment (SDE) that provided concurrently displayed, adjacent, multiple and linked representations consisting of the program code, a functional visualisation of the program, and its output.A modified version of the Restricted Focus Viewer (RFV)[3] — a visual attention tracking system - was employed to measure the degree to which each of the representations was used, and to record switches between representations. Other measures included debugging performance (number of bugs identified, the order in which they were identified, bug discovery latencies, etc.).The aim of this investigation was to address questions such as ‘To what extent do programmers use each type of representation?’ and ‘Are particular patterns of representational use associated with superior debugging performance?’.A within-subject design, and comparison of performance under (matched) RFV/no-RFV task conditions, allowed the use of the RFV as an attention-tracking tool to be validated in the programming domain. The results also provide tentative evidence that superior debugging using multiple-representation SDE’s tends to be associated with a) the predominant use of the program code representation, and b) frequent switches between the code representation and the visualisation of the program execution.

Many eye-tracking studies have shown that visual attention patterns during diagram-based problem solving, measured by eye movements, reveal critical aspects of the problem solving process that traditional measures like solution time and accuracy cannot address. In our first experiment (n = 14), we use this method during the solution of a widely-studied high level reasoning problem, Duncker’s (1945) radiation problem, to show that differences in visual attention to a particular diagram feature corresponds with correctly solving the problem. We then extend these findings in a second experiment (n = 81) to evaluate cognitive sensitivity to perceptual changes in the diagram. We show that problem solvers are highly sensitive to the diagram structure, and that the shifts in attention that result from subtle perceptual changes in the diagram appear to have a dramatic positive effect on reasoning.

Information Visualization produces a visual representation of abstract data in order to facilitate a deeper level of understanding of the data under investigation. This paper introduces ViCo, a metric for assessing Information Visualization complexity. The proposed metric allows for the measurement of Information Visualization complexity with respect to tasks and users. The algorithm for developing such a metric for any chosen collection of visualizations is described in general and then applied to two examples for purposes of illustration.

This paper addresses the design of representational systems for complex knowledge rich problems, focussing on scheduling in particular. Multiple tables are ubiquitous in representations of schedule information, but they impose large cognitive demands and inhibit the comprehension of highlevel patterns. The application and evaluation of representational design principles in the development of STARK diagrams, a novel system for scheduling problems, is reported. STARK diagrams integrate conceptual dimensions, principal relations and individual cases into a single diagrammatic structure. An experiment compared performance on STARK diagrams and a conventional representation with features typical of current commercial scheduling software interfaces. Subjects using the STARK diagram performed better at improving an examination schedule by minimising constraint violations. This provides support for the validity and utility of the design principles.

In developing a visual language for mobile robots, it is necessary to represent the uncertainty present in robot position, obstacle position, and even obstacle presence. In developing a visualization of the robot’s modelof its environment, this uncertainty should be presented to the experimenter, in order to be able to evaluate the extent to which the robot’s sensors and sensor fusion rules are providing consistent and reliable information.In Isaac, a project developing a rule-based visual language for mobile robots, a time-varying diagram is used to represent the robot’s current world model. Hue is used to represent object classes, and brightness is used to represent the degree of belief of an object’s presence. A region in which there is confidence that no object is present is shown as white, while a region with high confidence in the presence of an object is represented with color. Saturation is used to represent confidence in the assessment of object presence (or absence): a totally unsaturated (i.e. grey) area represents an area in which there is no belief at all either in favor of or against the presence of any object; a fully saturated area represents an area in which there is high confidence in the region’s classification. The combination of hue to distinguish between object classes with brightness and saturation for belief and confidence results in a three-dimensional color space for model visualization.Sensor characteristics are encoded in belief functions; upon receiving sensor information, both belief functions and confidence levels can be modified. Belief functions in the presence and absence of obstacles in the modelare maintained through Dempster-Shafer evidentialreasoning.

Editors for visual languages should be as simple and convenient to use as possible; at the same time, programmers should be able to create such editors without prohibitive effort. We discuss the benefits that can be gained from combining the following aspects in an editorgenerator approach: direct-manipulation editing (as in drawing programs)structure-based editing (as in common diagram tools)structural analysis and a common formal model As a major practical example, we present an editor for UML class diagrams. We show that direct-manipulation editing capabilities can enhance the usability of such an editor in comparison to standard tools. A further improvement is obtained by including selective abstraction features similar to the well-known “fisheye-viewing” and “semantic zooming” paradigms. We show that the proposed generator architecture provides an excellent base for implementing such features. The resulting technique can be applied to a wide range of different diagram languages; in contrast to other general solutions, it takes into account the abstract structure and specific abstraction features of the individual languages.

Different diagrammatic languages are concrete variants of a core metamodel which specifies the way in which to express relations, and which is the basis for a semantic interpretation. In this paper, we identify families of diagrammatic languages exploiting the notion of metamodel as introduced in UML, i.e. through an abstract syntax, given as a class diagram, and a set of constraints in a logical language. The abstract syntax constrains the types of expressable relations and the types and multiplicities of the participating entities. The constraints express contextual and global properties of the relations and their participants. We propose a set of metamodels describing common types of diagrammatic languages. The advantages of this proposal are manifold: the analysis of constraints in the metamodel can be used to assess the adequacy of a type of language to a domain semantics and it is possible to check whether a concrete notation or syntax complies with the metamodel or introduces unforeseen constraints. Finally, we discuss how this characterisation allows the definition of flexible editors for concrete diagrammatic languages, where a specific editor results from the specialisation of some high-level construction primitives for the relevant family of languages.

In this paper UML statechart diagrams are used as an example of a generic approach to integrating a visual language in a heterogeneous modelling and simulation environment.Asystem represented in a visual language is syntactically defined as an attributed graph, with well-formedness rules specified by a set of firstorder predicates over the abstract syntax of the graph. The language semantics are specified by an Abstract State Machine (ASM) parameterized with syntacticallycorrect attributed graphs. In this paper the key issues in the definition of UML statechart semantics are highlighted.

Conveying information through a diagram depends to some extent on how well it is designed as an input to our visual system. Results of previous work by the author (and collaborators) show that diagrams based on a perceptual syntax (Geon diagrams) can improve the legibility of the semantic content in a diagram. The present work evaluates the learnability of semantic information using a perceptual notation. Results of one experiment are reported. These show that Geon diagrams are easier to learn and remember in comparison to equivalent diagrams using conventional line and box drawings.

This study assesses how the mode of presentation affects the way in which people structure their mental models of a mechanical system, namely a flushing cistern. Subjects were assigned to one of three learning conditions: diagram only, 3-phases diagram, or animation. After learning the material, subjects generated written descriptions of the workings of the cistern. An analysis of temporal conjunctions used and the number of causal events mentioned indicates that for understanding simultaneity and causality, animation does not provide any benefit over seeing the same information in static diagrams.

External representations such as diagrams, sketches, charts, graphs and scribbles on napkins play facilitatory roles in inference, problem-solving and understanding (e.g. [1],[2],[3],[4],[5],[6],[7],[8],[9]). How does the externality and visibility of representations facilitate inference and problem-solving? One benefit of external representations is on memory. They reduce working memory load by providing external tokens for the elements that must otherwise be kept in mind. This frees working memory to perform mental calculations on the elements rather than both keeping elements in mind and operating on them [2],[9]. External representations also serve as visuo-spatial retrieval cues for long term memory, evoking relevant information that might not otherwise be retrieved. Another benefit of external representations is to promote discovery and inference, both visuo-spatial and metaphorical. Perceptual judgements about size, distance, and direction are easily made from external representations (e.g.[4]). In a Venn diagram, set relations such as inclusion are abstractly mapped onto visuo-spatial diagrammatic features, enabling direct perceptual calculation. Visuospatial features such as proximity, connectivity, and alignment provide useful hints to selection of appropriate inference paths (e.g.[1],[6],[8]) and to proper understanding of the structure of a target system (e.g.[5]). Calculations requiring counting, sorting, or ordering are easily made by rearranging external spaces (e.g. [7]).To serve these functions of memory, inference, and calculation, the interpretation of the external representation is static; it must stay the same in order not to introduce error in the operations performed from the external representation. External representations, however, are visuo-spatial displays, and it is known from research on perception that such displays, especially vague and ambiguous ones, can be interpreted and reinterpreted. Are there situations where the very instability of visuo-spatial displays can be used to advantage?One situation where the instability of interpretations of external representations can be beneficial is design. Among the earliest of commentators on this was Schon [10], who proposed that freehand sketches serve as a medium for the dynamic generation of new design ideas. In developing ideas for new projects, designers do not draw sketches to externally represent ideas that are already consolidated in their minds. Rather, they draw sketches to try out ideas, usually vague and uncertain ones. By examining the externalizations, designers can spot problems they may not have anticipated. More than that, they can see new features and relations among elements that they have drawn, ones not intended in the original sketch. These unintended discoveries promote new ideas and refine current ones. This process is iterative as design progresses. Seeing unintended relations and features in sketches requires release from previous interpretations. Previous interpretations can have a strong hold on observers, so preventing fixation and encouraging new interpretations are perceptual processes desirable for designers to acquire.In recent years we have explored ways that designers use external representations to discover and develop design ideas. The project has used both naturalistic and experimental methods. We present data from both these projects that are relevant to the current analysis. In a large naturalistic study, novice and experienced architects were filmed as they designed a museum. Later, they watched their design session, and commented on every stroke of the pen. These protocols have been analyzed in detail [11],[12],[13]. In the experimental study, designers and novices were shown ambiguous sketches and asked to produce as many interpretations of them as possible.The protocols from the design sessions showed clearly that new design ideas were likely to be generated immediately after discovering new relations and features in one's own sketches. Notably these new relations and features were unintentional byproducts of the aspects of the sketches drawn for other reasons. We call this process detection of unintended relations and features [12]. Detection of unintended relations and features is a significant impetus for the generation of new ideas. The generation of new ideas, in turn, was likely to become an impetus for further detection of unintended relations and features, so that each component process drives the other [12]. The joint occurrence of generation of new ideas and detection of unintended relations and features constituted the core cognitive processes of designers as they worked. As is common in discovery problems, the designers themselves were not able to predict what kind of unintended relations and features they were going to detect and what kinds of ideas they were going to generate. Productive design is situated in the physical setting of sketches. They are not merely a static medium for externalizing internal visions, but rather a physical environment from which ideas are generated on the fly.The key to creative design, then, is the cyclic pattern of generating new ideas and detecting unintended relations and features in external representations. More generally, it can be regarded as a coordinated co-generation of new conceptual thoughts and perceptual discoveries in external representations. This appears to be a general phenomenon occurring in a broader context involving any kind of creation, not just in design processes [13]. An important issue is how to facilitate the co-generation of reinterpretations and novel ideas in inspecting external representations, given that it is by no means automatic. We argued that the cognitive skill we have called constructive perception promotes the discovery of new interpretations in external representations [14]. By constructive perception, we mean self-awareness of the ways that perception underlies interpretations of external representations. The self-awareness allows searching for other ways to perceive, enabling reorganization of the external representation to promote novel interpretations. Experienced designers are superior to laypeople in this skill [14]. This finding raises two issues, one cognitive, the other didactic. What constitutes the ability of constructive perception? How can people be trained to use it? Research on these will promote successful use of external representations for creative purposes.

Concept-Mapping was used to create models of the knowledge of expert weather forecasters. STORM-LK (System To Organize Representations in Meteorology-Local Knowledge) demonstrates the use of CMap diagrams to generate large-scale multi-media knowledge models. This merger of interactive graphical communication with knowledge management provides a counterpoint for discussion of logical and psychological issues in diagrammatic reasoning.

Data from expert forecasters making weather reports (using the talk aloud method) were coded for dynamic comments as well as whether the visualization itself was static or dynamic. Preliminary results strongly suggest that meteorologists build dynamic mental models from static images.

An approach to the teaching of electricity is described which uses AVOW diagrams, a novel diagrammatic representation to visualize the laws of electricity. AVOW diagrams can help learners develop useful concepts and a more integrated understanding of electric circuit behavior than alternative teaching methods. In this study the practical potential of the approach to teach trainee science teachers was examined. In addition to the overall learning effect, trainee teachers with little previous experience of electric circuit behavior advanced after brief instruction to a level of understanding approaching the more knowledgeable trainees before instruction. For those trainees with prior knowledge the approach consolidated their understanding and they were able to quickly adopt the technique in the solution to more complex electric circuit problems.

Studies in cognition [1],[2] have investigated the role of external visual representations in different domains in supporting reasoning, problem solving, and communication. These studies often are confined to domains that pose relatively well-defined problems [3], such as geometry [4] and physics [2], with fewer studies in domains where the problems are ill-defined [3], such as meteorology [5] and architecture [6].

This paper presents a small survey of the use of drawing in cross-linguistic communication. The findings indicate that drawing is an infrequent but valuable element of cross-linguistic interactions. It is used opportunistically, for a wide variety of functions, predominantly in dyadic interactions. It is used more frequently to support the drawer in contributing to an interaction conducted in their ‘second’ language, than to support an addressee in understanding a contribution in the drawer’s ‘first’ language. Also, two broad categories of drawing, ‘cross-cultural’ topics and language-use are more frequent in cross-linguistic interactions than in other contexts.

We are now entering the era of pervasive computing, an era where people will access information and services anywhere, anytime, and from a wide variety of devices. The challenge for researchers and practitioners is how to support the design of user interfaces that will empower people to engage in these interactions easily and efficiently. Our work has been in creating design tools that support the best practices of user-centered design. Such practices include the informal techniques used during the early stages of design, such as sketching and “faking” interactions using Wizard of O_z techniques to test early designs. In this talk we will argue that tools with informal user interfaces best support these practices. Informal user interfaces support natural human input, such as speech and writing, while minimizing recognition and transformation of the in put. These interfaces that document, rather than transform, better support a designer’s flow state. Unrecognized input embraces nuanced expression and suggests a malleability of form that is critical for activities such as early-stage design. We will illustrate this by examining informal tools we have created for designing information architectures and web sites, speech-based user interfaces, and eventually anytime, anywhere user interfaces that take advantage of a variety of modes of input and output on a range of devices.