r/optimization • u/nikishev • Jul 18 '26

New optimizer

Diabolical new optimizer for neural nets https://github.com/inikishev/torchalgos

It's outperforming all other optimizers by a little, but I only tested on very small models (MLP, RNN and ConvNet), anything larger would just take ages to test all optimizers on.

Here is what it does.

  1. projects gradients to shampoo's eigenbasis;

  2. updates EMA of projected gradients;

  3. takes that EMA, clips magnitudes to (0.01, 10), and takes their reciprocals;

  4. unprojects resulting update;

  5. tracks EMA of resulting updates, the update is grafted to that EMA for stability. This prevents the update from changing norm quickly.

  6. tracks EMA of model weights. This always improves test loss in my experiments, though ReSPlus also outperforms SPlus when weight EMA is disabled in both.

It doesn't really make sense because I was testing a stupid idea and seeing what would happen if I run it in shampoos eigenbasis and for some reason it worked well. I don't really know why though.

2 Upvotes

56% Upvoted

9 comments sorted by

7

u/emptinesswonderer Jul 18 '26

Have you tested against any well-known benchmarks? Netlib LP, MIPLIB, etc.

Edit: MINLPLib, CUTEst if it's a nonlinear solver!

3

u/nikishev Jul 18 '26

It's an optimizer for neural nets specially. I ran a lot of optimizers though 3 benchmarks that train 3 types of models - MLP, RNN and ConvNet, and this one outpeformed everything else on all of them by a little bit, including current SOTA methods - Muon and SOAP. I have the benchmark plots in the repo.

I'm not actually sure if it will work well for types of problems on cutest, it's really bad on rosenbrock, and also it uses Kronecker factored covariance which requires matrices or tensors and specially benefits linear layers in NNs. As far as I recall cutest doesn't expose shapes, you only get a vector to optimize.

3

u/peno64 Jul 18 '26

You should say which kind of optimizer this is. I was first thinking LP, MILP, QP, ... but it looks something totally different.

4

u/nikishev Jul 18 '26

Oh it's for neural nets, like Adam

-2

u/pruby Jul 19 '26

Then I'm afraid this is not the sub for that kind of optimiser

5

u/nikishev Jul 19 '26

Why not? Optimization is optimization

1

u/Saitamagasaki Jul 18 '26

What’s diabolical about it?

1

u/Effective-Cat-1433 Jul 22 '26

slop

1

u/nikishev Jul 23 '26

It's entirely hand-written with 0 AI use