36. Directed Information on Abstract Spaces: Properties and Extremum Problems

Directed information is an information theoretic measure which accounts for the direction of information flow over causal systems with feedback, such as network communication and communication for control problems. In this chapter, we discuss several functional and topological properties of directed information for general Polish spaces (complete separable metric spaces) using the topology of weak convergence of probability measures. These include, convexity/concavity of directed information, weak compactness of families of causally conditioned convolutional distributions, lower semicontinuity of directed information, continuity of directed information, and extremum problems of directed information, including variational equalities [utilized in Blahut–Arimoto algorithm (BAA)], which are important in nonanticipative or real-time joint source-channel coding (JSCC). These basic functional and topological properties of directed information are analogous to those of mutual information. Throughout the chapter, the importance of the properties of directed information is discussed in the context of extremum problems of directed information, such as point to point and network applications.