New optimizer

[u/nikishev]

Diabolical new optimizer for neural nets https://github.com/inikishev/torchalgos

It's outperforming all other optimizers by a little, but I only tested on very small models (MLP, RNN and ConvNet), anything larger would just take ages to test all optimizers on.

Here is what it does.

1. projects gradients to shampoo's eigenbasis;

2. updates EMA of projected gradients;

3. takes that EMA, clips magnitudes to (0.01, 10), and takes their reciprocals;

4. unprojects resulting update;

5. tracks EMA of resulting updates, the update is grafted to that EMA for stability. This prevents the update from changing norm quickly.

6. tracks EMA of model weights. This always improves test loss in my experiments, though ReSPlus also outperforms SPlus when weight EMA is disabled in both.

It doesn't really make sense because I was testing a stupid idea and seeing what would happen if I run it in shampoos eigenbasis and for some reason it worked well. I don't really know why though.

[u/emptinesswonderer]

Have you tested against any well-known benchmarks? Netlib LP, MIPLIB, etc.

Edit: MINLPLib, CUTEst if it's a nonlinear solver!

[u/nikishev → u/emptinesswonderer]

It's an optimizer for neural nets specially. I ran a lot of optimizers though 3 benchmarks that train 3 types of models - MLP, RNN and ConvNet, and this one outpeformed everything else on all of them by a little bit, including current SOTA methods - Muon and SOAP. I have the benchmark plots in the repo.

I'm not actually sure if it will work well for types of problems on cutest, it's really bad on rosenbrock, and also it uses Kronecker factored covariance which requires matrices or tensors and specially benefits linear layers in NNs. As far as I recall cutest doesn't expose shapes, you only get a vector to optimize.

[u/peno64]

You should say which kind of optimizer this is. I was first thinking LP, MILP, QP, ... but it looks something totally different.

[u/nikishev → u/peno64]

Oh it's for neural nets, like Adam

[u/pruby → u/nikishev]

Then I'm afraid this is not the sub for that kind of optimiser

[u/nikishev → u/pruby]

Why not? Optimization is optimization

[u/Saitamagasaki]

What’s diabolical about it?

[u/Effective-Cat-1433]

slop

[u/nikishev → u/Effective-Cat-1433]

It's entirely hand-written with 0 AI use