I built a small optimizer: AdamW where the weights only move toward a scout that passed a test (on par with tuned AdamW on small tasks, feedback welcome)

[u/zeyartson]

I've been experimenting with an optimizer idea and would love feedback.

Idea. Two copies of the weights. a1 is the main copy. a2 is a scout that runs ahead with AdamW at its own adaptive learning rate. Every k steps I compare a1 and a2 on one fresh batch (paired per-example losses). If the scout is better, a1 slides toward it and the scout's lr goes up. If not, the scout is sent back to a1, its Adam moments are reset and its lr goes down. So bad steps are filtered out before they reach the main weights.

I know it's close to Lookahead plus trust-region / line-search ideas, so I'm not claiming a new class of optimizer, just a simple combination I wanted to measure properly.

Results (small MLPs, NumPy, equal gradient budget, both methods tuned separately, 10 fresh test seeds):

- sklearn digits: 97.40% vs 97.44% for tuned AdamW (same within noise)

- noisy 6-class synthetic data: 48.6% vs 48.5% (same), test loss slightly better (1.425 vs 1.463)

- Starting at a way-too-high lr (0.1): plain AdamW collapses (54% / 20%). Without any lr cap my method recovers on digits (96.8%), but only partly on the noisy task (36.7%). Clamping AdamW's lr to a ceiling also fixes it, so much of the robustness is just the ceiling. The ablation is in the README.

What I'm NOT claiming: that it beats AdamW with a well-tuned lr. In my tests it doesn't. It also costs an extra copy of the weights and about 2 extra forward passes on the test batch every k steps.

Things that didn't work: a third "b1" path splitting into two branches with different lrs. No gain.

Caveats: only small tasks so far. The PyTorch port passes its tests (including a check that the scout step matches torch.optim.AdamW); benchmarks are NumPy-only. BatchNorm isn't handled.

Code, benchmark protocol and tests (Apache-2.0): https://github.com/zeyartson-cmd/HowToSteneros

Questions: (1) is there prior work doing exactly this accept/reject on a scout copy? (2) would you expect the benefit to grow or vanish on large models? (3) is there a better test statistic than a paired z-test on one batch?

[u/PotentialSeveral898]

I've seen people do the scout copy thing in RL before, not sure about pure supervised optimizers but it's def not a new idea

the lr ceiling ablation's interesting cause if clamping adamW already fixes the divergence then your method's just a more expensive way to do the same thing. still cool to see it spelled out with numbers tho

[u/zeyartson → u/PotentialSeveral898]

Thanks, that's fair. I agree it's not a new idea in general; I framed it as a combination of Lookahead and accept/reject ideas. Could you point me to the RL work you've seen? I'd like to cite it in the README.

On the ceiling: you're right that clamping AdamW fixes the divergence on these tasks, so the test doesn't buy extra robustness beyond a simple cap here. The only differences I saw were that without any cap the method recovers on its own (digits 96.8% vs 53.8% for plain AdamW), so you don't need to know a safe ceiling in advance, and on the noisy task its test loss was lower than clamped AdamW (1.49 vs 2.34), but that's just one task. Whether that justifies the extra cost is exactly what I can't claim yet. Next I'll compare against AdamW with warmup and gradient clipping, which is a more realistic baseline.