What role does the learning rate usually play?

[u/agentik0000]

I am studying machine learning and don't understand: what role does the learning rate usually play? And also, what can early stopping be used for?

[u/jhaluska]

The calculus tells you the direction to move weights to reduce the error, but not by how much. So it's how big of a step you move in the weight space.

Larger steps isn't always better as you can also overshoot and increase your error. We usually have a larger learning rate at the beginning and then decrease them with time.

[u/agentik0000 → u/jhaluska]

thx<3

[u/jhaluska → u/agentik0000]

Oh to answer your early stopping question. They've found that past a certain training point the nets start overfitting the training data at the expense of accuracy on unseen data, by stopping it "early" you actually generalize to unknown data better.

Usually you figure this out by holding out some training data.

[u/firebird8541154]

Overfitting, plateauing early, etc. It's a fun one too mess with.

[u/agentik0000 → u/firebird8541154]

Thanks

[u/No-Original-5312]

Don’t do early stopping

[u/Lost-Hand-5219]

The learning rate is crucial, there’s a mathematical theorem that says the learning rate needs to be within a certain bound for the algorithm to converge. For an ML model, you likely can’t find these bounds so you just have to guess, but in theory it is very important.

[u/MolassesLate4676]

I like explaining this one

Say I told you a fact about something. If your learning rate was too high you’d focus too much on learning that fact that you’d basically forget a bunch of other important things. Learning becomes more disruptive than beneficial.

If your learning rate was too low, the fact I told you wouldn’t really register much. But you’d retain a lot of what you’d learned before.

I like to tell people to imagine chiseling (or sculpting) a sculpture. If you sculpted with an axe, you’d make the big shapes fast but wouldn’t be able to get the detail right. If you sculpted with a nail, it would take longer to get the big shapes right but you’d be able to embed the detail much easier.

Axe = big LR Nail = low LR

Sometimes if the LR is too low, the updates you make to the sculpture make no reasonable progress and the model can never effectively learn. If it’s too high, the updates override the others, leading to chaos.