The Maximum Entropy Principle

In Section 1.1, we alluded to the principle that titles this chapter. The idea of this principle, as we said there, is to assign probabilities in such a way that the resulting distribution contains no more information than is inherent in the data. The first attempt to do this was by Laplace and was called the “Principle of Insufficient Reason.” This principle said that two events should be assigned the same probability if there is no reason to do otherwise. This appears reasonable and is fine as far as it goes. However, Jaynes [12] said, “except in cases where there is an evident element of symmetry that clearly renders the events ‘equally possible,’ this assumption may appear just as arbitrary as any other that might be made.” What is needed, of course, is a way to uniquely quantify the “amount of uncertainty” represented by a probability distribution. As mentioned in Section 1.1, this was done in 1948 by C. E. Shannon in a paper entitled A Mathematical Theory of Communication [25]. Shannon was specifically interested in the problem of sen ding data through a noisy, discrete transmission channel. However, his results, most notably a rigorous treatment of entropy, have had far-reaching consequences in the theory of probability and statistics. We are not going to even attempt to summarize the broad effects his work has had nor reference the many papers that have since been written on the subject; our purpose is to quickly develop the notion of entropy and get on with the estimation problem of the last chapter. For more information on the subject of entropy, the reader is referred to the text by Ellis [4].