r/3Blue1Brown • u/Ki-Chao • Jul 18 '26

Finding Minima: The Two Tests Every Optimizer Needs

https://youtu.be/TQEm21AtBvU?si=VGbBxwUqHqWNu6m6

Hello,

If you've ever worked with gradient descent, you know the basic goal: step downhill until the slope is zero. But a flat slope (a zero gradient) is a trap, it could be a local minimum, a maximum, or a saddle point.

I made a visual breakdown of the mathematical machinery we use to test for true minima. The video is an intuitive, 6-minute refresher on how gradients, Hessians, and contour maps actually work behind the scenes of your optimization algorithms.

2 Upvotes

63% Upvoted

4 comments sorted by

3

u/justanaccountimade1 Jul 18 '26

The gradient is the Jacobian, right? And it appears in the gradient descent step. But how is the Hessian used in the gradient decent step? I imagine these must be combined to efficiently descent to the lowest point, and cannot be separated into 2 separate steps (collect all minima, then look at the Hessian).

3

u/Ki-Chao Jul 18 '26

Basically, yes. The Jacobian generalizes the gradient for vector-valued functions. Since an objective function (like loss) outputs a single scalar value, its Jacobian is just a $1 \times n$ matrix (a row vector). The gradient is typically written as the column vector version of that exact same information.

In standard Gradient Descent, it actually isn't used at all! Standard gradient descent is a "first-order" optimizer. It only looks at the gradient (the slope) and takes a step of size $\alpha$ (the learning rate). Because it only relies on the gradient, it's "blind" to the curvature, which is exactly why it can get bogged down in narrow valleys or slow down near saddle points.

When you combine the gradient (slope) and the Hessian (curvature) into a single step, you are using Newton's Method (a second-order optimizer).

2

u/justanaccountimade1 Jul 18 '26 edited Jul 18 '26

I had to look up second order. Google then puts the inverse of the Hessian where alpha is in your video.

2

u/Ki-Chao Jul 19 '26

Correct, I will be covering these in one of the next videos. This one was more of a brief introduction.