f-divergence - Wikipedia
information-theorystatisticsdivergenceprobability-theory
Abstraction: General family of probability distribution divergences via convex function
Key points:
- f-divergence D_f(P||Q) is defined via a convex function f with f(1)=0; generalizes KL, Hellinger, total variation, chi-squared, and Jensen-Shannon divergences
- Introduced by Alfréd Rényi; independently studied by Csiszár (1963), Morimoto (1963), Ali and Silvey (1966)
- Key property: all f-divergences satisfy the data-processing inequality and decrease monotonically in Markov processes (Lyapunov functions for Kolmogorov forward equations)
- KL divergence is the only f-divergence that is also a Bregman divergence; total variation is the only one that is also an integral probability metric
- Symmetrization via convex inversion turns KL-divergence into Jeffreys divergence
- Variational representations (Donsker-Varadhan and extensions) relate f-divergences to expectations over test functions
Connections: Information Theory · Statistical Divergence · Kl Divergence