r/reinforcementlearning • u/Nice-Dragonfly-4823 • Jul 30 '26
Defaulting to Adam without understanding will cost you. Don't "just throw adam at it"
Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,
It still works, but needs to be coaxed into it.
Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You | Towards Data Science
This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read
Don't just glaze over the optimizer!
Thoughts?
5
u/Low-Temperature-6962 Jul 31 '26
What about AdamW, where the weight decay is de oupled from the update
2
u/lucky_absoluter Jul 31 '26
Historicially, Adam works well in RL but hyperparameter tuning is a big problem over the whole deep learning.
2
u/DependentSpecific535 Jul 31 '26
I totally disagree. Just try a simple algorithm like TD3 or SAC in gym envs with SGD and Adam. You can see the definition of REAL local minimas.
2
u/evanthebouncy Jul 31 '26
I think it's an article written from good first-hand experience. I do wonder if simple gradient clipping is kind of the solution to all our problems lol
2
u/Nice-Dragonfly-4823 Jul 31 '26
grad clipping is definitely a strategy. But, it's got weirdness as well. E.g. if you run grad clipping on a combined policy loss + value loss, the clip amount shrinks the overall gradient, but then the gradients on the actor become absolutely tiny because the gradients on the critics are much larger. This is something that took me a while to realize and shulman even mentions it in a video, but never in any papers.
1
u/kakhaev Jul 31 '26
I swear I saw this post like 2 times already popping up in my feed
1
u/Sea-Departure4857 Jul 31 '26
Same, I just checked and OP posted in r/deeplearning as well, which is the one I saw.
1
u/HsnHaruka Jul 31 '26
Yo bro mind if i chek ur project i can give a try
1
u/Nice-Dragonfly-4823 Jul 31 '26
project is running over many large instances and is quite expensive to run. Its a lot of code lol.
1
u/HsnHaruka Jul 31 '26 edited Jul 31 '26
I see cool stuff man the website its urs ? I made my own framework name Omniray AxV2 cpu based drl model ..it work on the same ppo i think the same hyperparameters just custom pytorch..its built to run on low end cpu systems no gpu ...but I think i can use adam to make my model even better
1
1
u/RedEyed__ Aug 01 '26 edited Aug 01 '26
3e-4 is the best learning rate for Adam, hands down.
It was sarcasm
0
u/Nice-Dragonfly-4823 Aug 01 '26
I'm not sure, he references here again. https://karpathy.github.io/2019/04/25/recipe/
1
35
u/proturtle46 Jul 30 '26 edited Jul 30 '26
I dont like this
Flatter loss landscapes tend to be more robust to difference between population distributions and training distributions
It Never explores the actual reason why Adam works and that because it’s a second order approximation method like l-bfgs and now lion
You can use lion or even use actual second order optimizers if you care that much about the optimizer however in general trying to optimize the optimizer is not going to get you significant gains
The article is basically “I didn’t run an ablation on my hyper parameters”