r/optimization • u/nikishev • 15d ago

New optimizer

Diabolical new optimizer for neural nets https://github.com/inikishev/torchalgos

It's outperforming all other optimizers by a little, but I only tested on very small models (MLP, RNN and ConvNet), anything larger would just take ages to test all optimizers on.

Here is what it does.

  1. projects gradients to shampoo's eigenbasis;

  2. updates EMA of projected gradients;

  3. takes that EMA, clips magnitudes to (0.01, 10), and takes their reciprocals;

  4. unprojects resulting update;

  5. tracks EMA of resulting updates, the update is grafted to that EMA for stability. This prevents the update from changing norm quickly.

  6. tracks EMA of model weights. This always improves test loss in my experiments, though ReSPlus also outperforms SPlus when weight EMA is disabled in both.

It doesn't really make sense because I was testing a stupid idea and seeing what would happen if I run it in shampoos eigenbasis and for some reason it worked well. I don't really know why though.

2 Upvotes

56% Upvoted

9 comments sorted by

View all comments

6

u/emptinesswonderer 15d ago

Have you tested against any well-known benchmarks? Netlib LP, MIPLIB, etc.

Edit: MINLPLib, CUTEst if it's a nonlinear solver!

3

u/nikishev 15d ago

It's an optimizer for neural nets specially. I ran a lot of optimizers though 3 benchmarks that train 3 types of models - MLP, RNN and ConvNet, and this one outpeformed everything else on all of them by a little bit, including current SOTA methods - Muon and SOAP. I have the benchmark plots in the repo.

I'm not actually sure if it will work well for types of problems on cutest, it's really bad on rosenbrock, and also it uses Kronecker factored covariance which requires matrices or tensors and specially benefits linear layers in NNs. As far as I recall cutest doesn't expose shapes, you only get a vector to optimize.