Kullback-Leibler divergence - Wikipedia
information-theorystatisticsdivergenceentropymachine-learning
Abstraction: Asymmetric measure of difference between two probability distributions
Key points:
- KL divergence D(P||Q) measures expected excess bits when coding P-distributed data using a Q-optimized code
- Always non-negative (Gibbs' inequality); equals zero iff P = Q; not a metric (asymmetric, no triangle inequality)
- For multivariate Gaussians has a closed-form involving trace, determinant, and mean difference terms
- Arises naturally in variational inference (ELBO), EM algorithm (reversed direction), and Bayesian updating
- Infinitesimal Hessian of KL divergence equals the Fisher information metric (information geometry)
- Special case of both f-divergences and Bregman divergences; unique in belonging to both classes
Connections: Information Theory · Kl Divergence · Bayesian Inference
Source: http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence