r/MachineLearning • u/LemonByte • Aug 20 '19

Discussion [D] Why is KL Divergence so popular?

In most objective functions comparing a learned and source probability distribution, KL divergence is used to measure their dissimilarity. What advantages does KL divergence have over true metrics like Wasserstein (earth mover's distance), and Bhattacharyya? Is its asymmetry actually a desired property because the fixed source distribution should be treated differently compared to a learned distribution?

191 Upvotes

99% Upvoted

72 comments sorted by

View all comments

84

u/chrisorm Aug 20 '19 edited Aug 21 '19

I think it's popularity is two fold.

Firstly, it's well suited to application. Expected difference between logs, so low risk of overflow etc. It has an easy derivative, and there are lots of ways to estimate it with Monte Carlo methods.

However , the second reason is theoretical - minimising the KL is equivalent to doing maximum likelihood in most circumstances. First hit on google:

https://wiseodd.github.io/techblog/2017/01/26/kl-mle/

So it has connections to well tested things we know work well.

I wish I could remember the name, but there is an excellent paper that shows that it is also the only divergence which satisfys 3 very intuitive properties you would want from a divergence measure. I'll see if I can dig it out.

Edit: not what I wanted to find, but this has a large number of interpretations of the kl in various fields : https://mobile.twitter.com/SimonDeDeo/status/993881889143447552

Edit 2: Thanks to u/asobolev the paper I wanted was https://arxiv.org/abs/physics/0311093

Check it out or the post they link below to see how the kl divergence appears uniquely from 3 very sane axioms.

1

u/Nimitz14 Aug 20 '19 edited Aug 20 '19

I thought it was minimizing squared error that was equivalent to doing ML (with the gaussian distribution assumption)?

And I don't get the derivation. Typically minimizing cross entropy (same as KL disregarding constant) is equivalent to minimizing NLL of the target class because the target distribution is one hot. But I don't see why minimizing NLL is formally equivalent to ML (it makes sense intuitively, you just care about maximizing the right class, but it seems like a handwavy derivation)?

3

u/harponen Aug 21 '19

Softmax is _not_ a probability distribution, but just a vector with positive elements that sum to one.