Data Structures for Computational Statistics

This chapter first explains what data structures are and why they are important for statistical software. Then we take a look at why we need interactive environments for our work and what the appropriate tools should be. We do not discuss the requirements for the graphical user interface (GUI) in detail. The last section will present the actual state of soft- and hardware and which future developments we expect.

The first step is to analyze the statistical graphics in use, so we examine the descriptive statistics. Next is the boxplot, the Q-Q plot, the histogram, the regressogram, the barchart, the dotchart, the piechart, the 2D-scatterplot, the 2D-contour plot, the scatterplot matrix, the 3D-scatterplot, the 3D-contour plot, the Chernoff faces and the parallel coordinate plot. We use some of the tools of XploRe to examine the Berlin housing dataset which use the plots mentioned above. Finally we state that two kinds of windows are necessary, one which can draw points, lines and areas, another which can draw glyphs windows, i.e. the Chernoff faces, star diagrams and so on.

Here some applications are discussed (cluster analysis, teachware, regression methods). The cluster analysis will serve as an example for the use of graphics. Teachware needs to be highly interactive and we shortly discuss the approach of Proenca (1995). The section about regression methods shows how detailed a programming language should be. The trade-off between speed and understanding in a statistical routine still plays an important role.

In this chapter we will discuss in detail one statistical technique. We will cover possible extensions (multivariate projections, inclusion of discrete variables) and show which graphics are used for representing results. The danger in EPP is that we interpret a random structure as a real structure in the data. We describe tests for detecting a structure which is not a multivariate gaussian distribution. Pictures are presented from tools (macros in XploRe and XGobi) which are used to execute exploratory projection pursuit. At the end we will specify the requirements necessary for a tool to do EPP, but neiter my implementation nor XGobi can satisfy all of them.

In the first section we will show that graphical objects can be generated in three steps. Then we will develop a hierarchy for the graphical data structures (datapart, windows, displays). In the next section we will give reasons why matrices are no sufficient structure to store statistical data, so we need multidimensional arrays. Then we will discuss their impact on mathematical and statistical operations. The second section will close with a description why we need hierarchical objects. In the third section several forms of linking will be discussed. First we will give examples of linking plots in the thesis, then we will show further examples of linking, i.e. asking data themselves or linked data, the link of events with subroutines and at last the linking between different datasets. The fourth section will describe in short some statistical packages and indicate which features concerning data structures are available in these programs.

Here we will show the implementation of the data structures developed in the chapter before in XploRe. Nevertheless not everything which is explained before will be part of XploRe 3.2. First we will describe how graphical data structures are implemented in XploRe 3.2 and how they can be used interactively. Then we will show the data structures for the data being implemented in XploRe 3.2. The basic data structure is a matrix and no hierarchical lists are possible. Then we describe which possibilities of linking are offered in XploRe 3.2. Then we will describe some selected commands in Xplore 3.2 for producing interactivity, for reading and writing data, reading and storing macros, libraries and for binned kernel estimators. These commands will show solutions to some problems we mentioned before or which will be used to show extensions based on the extension from matrices to multivariate arrays. The third section will describe how some selected tools work (random number generator, PCA, grand tour, multidimensional scaling, clustering, multivariate kernel regression, PPR, wavelet regression, interactive contouring). It will show that we are able to implement a variety of (interactive) tools efficiently with the proposed data structures. The fourth section will describe the implementation of arrays in XploRe 4.0, and the fifth will show how the commands BINDATA and CONV are extended for the use with arrays.

We have presented two versions of the software XploRe which implement our ideas about data structures in statistical software. The data structures for graphics and linking are implemented in XploRe 3.2 while the data structures for statistical data are implemented in XploRe 4.0. But we have not implemented all ideas.