Intuitive Explanation of the Kullback-Leibler Divergence
information-theoryprobabilitystatisticsmachine-learning-mathkl-divergence
Abstraction: KL divergence measures extra bits lost approximating one probability distribution with another
Key points:
- Self-information (Shannon-Information) quantifies surprise of a single event: I(x) = -log P(x); rare events carry more information than likely ones
- Entropy is the expected self-information — the minimum average bits needed to encode a distribution; balanced binary datasets have maximum entropy of 1 bit
- KL divergence D_KL(P||Q) equals the extra bits needed to encode samples from P using a code optimized for Q; it is zero when P and Q are identical
- Derived intuitively via ratio of sequence probabilities from two distributions; in the infinite-sample limit this converges to the formal KL formula
- KL divergence is asymmetric: D_KL(P||Q) is not equal to D_KL(Q||P), so it is not a true metric; Jensen-Shannon divergence is the symmetric, bounded (0-1) alternative
- Binary cross-entropy loss in ML is equivalent to the entropy of a Bernoulli variable parameterized by class frequency
Connections: Kullback Leibler Divergence · Information Theory · Entropy
Source: https://johfischer.com/2021/12/31/intuitive-explanation-of-the-kullback-leibler-divergence/