Taxonomy of principal distances

Hamming distance
Euclidean geometry (|{i : pi 6= qi }|) Statistical geometry
Physics entropy JK −1 Additive entropy
Euclidean distance Manhattan Pdistance
R
pP −k p log p cross-entropy
d2 (p, q) = (p − qi )2 (Pythagoras’
i i
d1 (p, q) = i |pi − qi | conditional entropy
(Boltzmann-Gibbs 1878) mutual information
theorem circa 500 BC) (city block-taxi cab) (chain rules)

Minkowski distance InformationR entropy

pP (Lk -norm)
dk (p, q) = k |pi − qi |k Mahalanobis metric (1936) H(p) = − p log p
i
(C. Shannon 1948)
p
(H. Minkowski 1864-1909) dΣ = (p − q)T Σ−1 (p − q)
Space-time geometry
I-projection Life
H(p) = KL(p||u)
negative entropy

Kullback-Leibler
R divergence
KL(p||q) = p log pq = Ep [log Q
P
]
Quadratic
p distance (relative entropy, 1951)
dQ = (p − q)T Q(p − q)
Non-Euclidean geometries Jeffrey divergence
(Jensen-Shannon)
Fisher information (local entropy)
∂
I(θ) = E[ ∂θ

2
ln p(X|θ) ] Bhattacharya q distance (1967)
R √ √
(R. A. Fisher 1890-1962) d(p, q) = − ln ( p − q)2
Kolmogorov
R
K(p||q) = |q − p|
α = −1 (Kolmogorov-Smirnoff max |p − q|)
Riemannian
q metric tensor
dx
gij dx
R i j exponential families
ds ds
ds Matsushita distance (1956)
q
(B. Riemann 1826-1866,) Mα (p, q) = α
R 1 1
|q α − p α |

Itakura-Saito divergence Chernoff divergence

R (1952) Hellinger
IS(p|q) = i ( pqii − log pqii − 1) Cα (p||q) = − ln pα q 1−α √ √
P qR
H(p||q) = ( p − q)2
(Burg entropy) p √
×α(1 − α) = 2(1 − f g
Rényi divergence (1961) χ2 test
2
χ (p||q) = (q−p)
R 2
R
1
Hα = α(1−α) log f α α = 0
p
1
Rα (p|q) = α(α−1)
R
ln pα q 1−α (K. Pearson, 1857-1936 )
(additive entropy)

Csiszár’ f -divergence
T
R
Df (p||q) = pf ( pq ) Neyman
Kullback-Leibler

(Ali& Silvey 1966, Csiszár 1967)

Bregman divergences (1967) Dual div.∗-conjugate (f ∗ (y) = yf (1/y))
BF (p||q) = F (p) − F (q)− < p − q, ∇F (q) > Df ∗ (p||q) = Df (q||p) Information geometries
Bregman-Csiszár
 divergence (1991) Amari α-divergence (1985)
x log x α = 1
∇
x − log x − 1 α = 0

x log x − x + 1 α = 1 − log x α = −1
Fα (x) = fα (x) = 1+α
1 (−xα + αx − α + 1)
∇∗
0 < α < 1 4 (1 − x 2 ) −1 < α < 1
α(1−α)
1−α2
Dual div. (Legendre) DF ∗ (∇F (p)||∇F (q)) = DF (q||p) Generalized
f -means
Generalized Pythagoras’ theorem
(Generalized projection)
duality... Quantum geometry
Quantum entropy
S(ρ) = −kTr(ρ log ρ)
Permissible Bregman divergences Burbea-Rao (Von Neumann 1927)
(Nock & Nielsen, 2007) (incl. Jensen-Shannon)


f (p)+f (q) p+q
JF (p; q) = −f
2 2

Non-additive entropy
Log Det divergence
Tsallis entropy (1998)
D(P||Q) =< P, Q−1 > − log det PQ−1 − dimP
(Non-additivePentropy)
Tα (p) = 1−α1
( i pαi − 1) Von Neumann divergence
pα
1
R
Tα (p||q) = 1−α
(1 − q α−1
) D(P||Q) = Tr(P(log P − log Q) − P + Q)
Earth mover distance
(EMD 1998)

Algorithmic geometry? Kolmogorov complexity

Distance between two algorithms ?

c Frank Nielsen, 2007. Version 0.2
Last updated, May 2008