Stochastic Models, Information Theory, and Lie Groups, Volume 2

The concept of a group was described briefly in Chapter 1. This chapter serves as an introduction to a special class of groups, the Lie groups, which are named after Norwegian mathematician Sophus Lie.1 Furthermore, when referring to Lie groups, what will be meant in the context of this book is matrix Lie groups, where each element of the group is a square invertible matrix. Other books focusing specifically on matrix groups include [3, 9, 19].

Functions on Lie groups can be integrated almost as easily as functions on Rn. In many applications, a special kind of Lie group arises. This is the unimodular Lie group. The integral of functions on unimodular Lie groups has the nice property that it is invariant under shifts of the argument of the function, both from the left and the right. The integral on a Lie group can be decomposed into integrals over a subgroup and a coset space.

The purpose of this chapter is to tie together a number of concepts that have been presented earlier. Stochastic models and information-theoretic quantities such as entropy are not disjoint concepts. They overlap nicely in the context of statistical mechanics, where stochastic models describe general classes of equations of motion of physical systems.

Automated (robotic) assembly systems that are able to function in the presence of uncertainties in the positions and orientations of feed parts are, by definition, more robust than those that are not able to do so. This can be quantified with the concept of “parts entropy,” which is a statistical measure of the ensemble of all possible positions and orientations of a single part confined to move in a finite domain. In this chapter the concept of parts entropy is extended to the case of multiple interacting parts. Various issues associated with computing the entropy of ensembles of configurations of parts with excluded-volume constraints are explored. The rapid computation of excluded-volume effects using the “principal kinematic formula” from the field of Integral Geometry is illustrated as a way to potentially avoid the massive computations associated with brute-force calculation of parts entropy when many interacting parts are present.

This chapter is concerned with a number of issues in statistics that have a grouptheoretic flavor. Large books can be found that are dedicated to each of the topics that are discussed briefly in the sections that follow. Although the coverage here is meant to be broad and introductory for the most part, effort is taken to elucidate connections between statistics and the theory of Lie groups, perhaps at the expense of completeness. The main topics covered are numerical sampling techniques, multivariate statistical analysis, and theory/numerical procedures associated with random orthogonal, positive definite, unitary, and Hermitian matrices. Integration on Lie groups and their homogeneous spaces ties these topics together. Pointers to classical applications in particle physics as well as more recent applications in complicated wireless communication networks are provided.

Information theory, as it is known today, resulted from the confluence of two very different roots that had their origins in the first half of the 20th century. On the one hand, information theory originated from electrical engineers such as Hartley, Nyquist, and Shannon [49, 86, 104], who worked on the analysis of systems and strategies to communicate messages from one location to another. On the other hand, mathematicians such as de Bruijn, Cramér, Fisher, Kullbach, and Rao were inventing ideas in probability and statistics that have direct relevance to the study of information transmission. In this chapter the “communications” aspect of information theory is emphasized, whereas in Chapter 3 the “probability and statistics” side was reviewed. In recent years, the theory of finite groups has been connected with equalities in information theory. Lie groups enter as symmetry operations associated with continuous physical models of information transmission such as the linear telegraph equation and nonlinear soliton equations. Lie groups also appear as a domain in which stochastic trajectories evolve in the analysis of noise in optical communication systems that transmit information over fiber optic cables. In addition, some of the basic concepts and definitions in the theory of communication have interesting properties that are enriched by merging them with concepts from group theory. Some of this recent work will be explored here.

Coding theory is concerned with methods for “packaging” and “unpackaging” messages in order that the most information can be reliably send over a communication channel. In this chapter, a greater emphasis is given to the roles of geometry and group theory in communication problems than is usually the case in presentations of this subject. Geometry and group theory enter in problems of communication in a surprising number of different ways. These include the use of finite groups and sphere packings in highdimensional spaces for the design of error-correcting codes (such as those due to Golay and Hamming). These codes facilitate the efficient and robust transmission of information. Additionally, Lie groups enter in certain decoding problems related to determining the state of various motion sensors.

Classical inequalities used in information theory such as those of de Bruijn, Fisher, Cramér, Rao, and Kullback carry over in a natural way from Euclidean space to unimodular Lie groups. The extension of core information-theoretic inequalities defined in the setting of Euclidean space to this broad class of Lie groups is potentially relevant to a number of problems relating to information-gathering in mobile robotics, satellite attitude control, tomographic image reconstruction, biomolecular structure determination, and quantum information theory. In this chapter, several definitions are extended from the Euclidean setting to that of Lie groups (including entropy and the Fisher information matrix), and inequalities analogous to those in classical information theory are derived and stated in the form of more than a dozen theorems. In all such inequalities, addition of random variables is replaced with the group product, and the appropriate generalization of convolution of probability densities is employed.

As has been discussed in earlier chapters, it is possible to define probability densities on Lie groups and to compute convolutions. Since Lie groups are by definition also analytic manifolds, the methodology from Chapter 8 of Volume 1 can be used to define SDEs and Fokker–Planck equations. However, the added structure provided by Lie groups means that these equations can be derived in completely Lie-theoretic terms without ever referring to coordinates or charts. In addition, the natural embedding of Lie groups into matrices means that SDEs can be written extrinsically as well. These topics are discussed here, along with related topics from the field of probability and statistics on Lie groups. These include answering the questions: “How can the concepts of mean and covariance of a pdf on a Lie group be defined?” “If I only care how the mean and covariance behave as a function of time, can I obtain these without solving the Fokker–Planck equation?”

Volume 1 was on establishing terminology and review of fundamental definitions from information theory, geometry, and probability theory on Euclidiean space, Volume 2 has focused on analogous concepts in the setting of Lie groups. A survey of problems that simultaneously involve Lie groups and information theory was provided, including the encoding/decoding of spatial pose (position and orientation). The physics that govern different kinds of communication systems gives rise to SDEs and their corresponding Fokker–Planck equations. In some instances, such as laser phase noise, these can be viewed as a probability flows on a group manifold. In other instances, such as the telegraph equation, Lie groups describe the symmetries of a PDE on Euclidean space. Stochastic models of phenomena such as the conformational fluctuations of DNA and the motions of robotic systems were examined. These lead to probability densities on the group of rigid-body motions, and properties of the corresponding conformational and parts entropy were studied. Numerical tools for solving Fokker–Planck equations on Lie groups such as the rotation group and group of rigid-body motions were reviewed.