The Multiinformation Function as a Tool for Measuring Stochastic Dependence

Abstract

Given a collection of random variables [ξ _i ] _{i ∈ N} where N is a finite nonempty set, the corresponding multiinformation function assigns to each subset A ⊂ N the relative entropy of the joint distribution of [ξ _i ] _{i ∈ A} with respect to the product of distributions of individual random variables ξ _i for i ∈ A. We argue that it is a useful tool for problems concerning stochastic (conditional) dependence and independence (at least in the discrete case).

First, the multiinformation function makes it possible to express the conditional mutual information between [ξ _i ] _{i ∈ A} and [ξ _i ] _{i ∈ В} given [ξ _i ] _{i ∈ C} (for every disjoint A,B, C ⊂ N), which can be considered as a good measure of conditional stochastic dependence. Second, one can introduce reasonable measures of dependence of level r among variables [ξ _i ] _{i ∈ A} (where A ⊂ N, 1 ≤ r < card A) which are expressible by means of the multiinformation function. Third, it enables one to derive theoretical results on (nonexistence of an) axiomatic characterization of stochastic conditional independence models.