Skip to main content
Top

2018 | OriginalPaper | Chapter

2. Information Measures for Discrete Systems

Authors : Fady Alajaji, Po-Ning Chen

Published in: An Introduction to Single-User Information Theory

Publisher: Springer Singapore

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this chapter, we define Shannon’s information measures for discrete-time discrete-alphabet systems from a probabilistic standpoint and develop their properties. Elucidating the operational significance of probabilistically defined information measures vis-a-vis the fundamental limits of coding constitutes a main objective of this book; this will be seen in the subsequent chapters.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Footnotes
1
More specifically, Shannon introduced the entropy, conditional entropy, and mutual information measures [340], while divergence is due to Kullback and Leibler [236, 237].
 
2
By discrete alphabets, one usually means finite or countably infinite alphabets. We however focus mostly on finite-alphabet systems, although the presented information measures allow for countable alphabets (when they exist).
 
3
We will interchangeably use the notations \(\{X_n\}_{n=1}^\infty \) and \(\{X_n\}\) to denote discrete-time random processes.
 
4
Note that \(\log |\mathcal{X}|\) is also known as Hartley’s function or entropy; Hartley was the first to suggest measuring information regardless of its content [180].
 
5
Note that \(P_{XY}(\cdot ,\cdot )\) is another common notation for the joint distribution\(P_{X, Y}(\cdot ,\cdot )\).
 
6
Equivocation is an ambiguous statement one uses deliberately in order to deceive or avoid speaking the truth.
 
7
Prevarication is the deliberate act of deviating from the truth (it is a synonym of “equivocation”).
 
8
This condition is equivalent to requiring that \(X_i\) be independent of \((X_{i-1},\ldots , X_1)\) for all i. The equivalence can be directly proved using the chain rule for joint probabilities, i.e., \(P_{X^n}(x^n)=\prod _{i=1}^nP_{X_i|X_1^{i-1}}(x_i|x_1^{i-1})\); it is left as an exercise.
 
9
Definition. A bound is said to be sharp if the bound is achievable for some specific cases. A bound is said to be tight if the bound is achievable for all cases.
 
10
As noted in Footnote 1, this measure was originally introduced by Kullback and Leibler [236, 237].
 
11
In order to be consistent with the units (in bits) adopted for entropy and mutual information, we will also use the base-2 logarithm for divergence unless otherwise specified.
 
12
Given a non-empty set A, the function \(d : A\times A\rightarrow [0,\infty )\) is called a distance or metric if it satisfies the following properties.
1.
Nonnegativity: \(d(a, b)\ge 0\) for every \(a, b\in A\) with equality holding iff \(a=b\).
 
2.
Symmetry: \(d(a,b)=d(b, a)\) for every \(a, b\in A\).
 
3.
Triangle inequality: \(d(a,b)+d(b,c)\ge d(a, c)\) for every \(a,b, c\in A\).
 
 
Metadata
Title
Information Measures for Discrete Systems
Authors
Fady Alajaji
Po-Ning Chen
Copyright Year
2018
Publisher
Springer Singapore
DOI
https://doi.org/10.1007/978-981-10-8001-2_2

Premium Partner