Some definitions in Information Theory

• Entropy can intuitively be thought of as a measure of information, for any probability distribution or in other words, uncertainty of a random variable.
• Mutual information is the measure of the amount of information one random variable contains about another.
• Relative Entropy is the measure of the distance between two probability distributions.

Note

Entropy can be thought of as the self-information of a random variable and mutual information is a special case of relative entropy.

Entropy

Definition

The entropy $\text{H}(X)$ of a discrete random variable $X$ is defined by

$$\text{H}(X)=-\sum _{x\in \mathcal{X}}p(x)\log p(x)$$

Joint Entropy

Definition

The joint entropy $\text{H}(X,Y)$ of a pair of discrete random variables $(X,Y)$ with a joint distribution $p(x,y)$ is defined as

$$\text{H}(X,Y)=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log p(x,y)$$

also,

$$\text{H}(X,Y)=-\text{E}\log p(X,Y)$$

Relative Entropy or Kullback-Leibler Distance

Definition

The relative entropy or Kullback-Leibler distance between two probability mass functions $p(x)$ and $q(x)$ is defined as

$$\begin{array}{rl}\text{D}(p||q)& =\sum _{x\in \mathcal{X}}p(x)\log \frac{p(x)}{q(x)}\\ & \\ & ={\text{E}}_p\log \frac{p(X)}{q(X)}\end{array}$$

Mutual Information

Definition

Consider two random variables $X$ and $Y$ with a joint probability mass function $p(x,y)$ and marginal probability mass functions $p(x)$ and $p(y)$. The mutual information $\text{I}(X,Y)$ is the relative entropy between the joint distribution and the product distribution $p(x)p(y)$:

$$\begin{array}{rl}\text{I}(X;Y)& =\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}\\ & \\ & =\text{D}(p(x,y)||p(x)p(y))\\ & \\ & ={\mathbb{E}}_{p(x,y)}\log \frac{p(X,Y)}{p(X)p(Y)}\end{array}$$