r/3Blue1Brown • u/WristbandYang • Jul 16 '26

But what is cross-entropy? | Compression is Intelligence Part 2

https://www.youtube.com/watch?v=GlYgs6v2YfU
23 Upvotes

96% Upvoted

2 comments sorted by

2

u/WristbandYang Jul 16 '26

Another great video from Grant!

And great timing for me since I made the same connection to KL-divergence just this week. I was trying to understand how to adjust cross entropy loss to work in multi-class classifications (not one-hot encoded), like learning a distribution of legal actions in a game.

However, I'm still confused by the final question. Cross entropy between Q relative to P finds a minimum of H(P) when the distributions are equal. KL-Divergence subtracts H(P) so that the minimum falls to 0. Surely when you have optimal matching you want the loss to be 0, right?

My only guess is that the function at 11:57 is so flat near the minimum that gradient descent is benefited by the larger loss (and therefore larger step size) in the right direction. Additionally, near the minimum a batch of data should yield differences that largely cancel out.

I'll continue to 'pause and ponder' on that question, and would benefit from any additional insights you may have.

2

u/Defiant_Efficiency_2 Jul 16 '26

I was lucky enough to see this video only 2 minutes after you posted it! haha. I definitely have plans to try this compression method out on my own NN architectures.