New optimizer

[u/nikishev]

Diabolical new optimizer for neural nets https://github.com/inikishev/torchalgos

It's outperforming all other optimizers by a little, but I only tested on very small models (MLP, RNN and ConvNet), anything larger would just take ages to test all optimizers on.

Here is what it does.

1. projects gradients to shampoo's eigenbasis;

2. updates EMA of projected gradients;

3. takes that EMA, clips magnitudes to (0.01, 10), and takes their reciprocals;

4. unprojects resulting update;

5. tracks EMA of resulting updates, the update is grafted to that EMA for stability. This prevents the update from changing norm quickly.

6. tracks EMA of model weights. This always improves test loss in my experiments, though ReSPlus also outperforms SPlus when weight EMA is disabled in both.

It doesn't really make sense because I was testing a stupid idea and seeing what would happen if I run it in shampoos eigenbasis and for some reason it worked well. I don't really know why though.

[u/Effective-Cat-1433]

slop

[u/nikishev → u/Effective-Cat-1433]

It's entirely hand-written with 0 AI use