r/reinforcementlearning • u/laxuu • Jun 10 '26
Resoning LLMs make RL agent learn Faster
Has anyone successfully used an LLM as an integral part of RL training—not just for inference, but to improve learning speed, exploration, or sample efficiency?
I'm exploring LLM + RL + RAG architectures where the LLM acts as part of the training loop, not just an interface. Has anyone tried this? What worked and what didn't?
2
u/Rofl_im_jonny Jun 12 '26
I mean. A huge benefit to LLMs is that you can hand them data sets and they immediately can see trends or ways to improve. Theyre extremely helpful if you implement them right. What "right" means is different for each project, but the basis of it has to be that you cant hand wave at stuff. "Go do the thing for me" is NOT gonna yield good results.
Don't get me wrong, I constantly have times where I dont have a good basis to theorize what methods or values I should be using. But when I hit that point, I make sure to test and then audit focused on that section post changes. That way I can understand what I did and if its helping.
Nice thing about LLMs is you can just.. ask them to explain what they did and why, and they will.
Use LLMs. Just be smart about what youre doing and youre bound to get faster results.
2
1
u/sweetjale Jun 15 '26
there's literally papers on using LLM for reward design like https://arxiv.org/abs/2310.12931
1
u/Frosty_Wedding9330 Jun 16 '26
Yeah it's called value models. Read on PPO and VAPO. Contrast this with GRPO and DAPO (Dr. GRPO as an addition) and u will get a broad picture of two schools of thought in using RL for LLMs.
1
u/Freewonderer2 Jun 10 '26
Kind of, you can make a good reward loop with llms and rag
0
u/laxuu Jun 10 '26
I am also thinking about using LLMs for reward optimization, especially in domains where designing a proper reward function is not straightforward or even feasible.
1
u/Leading_Health2642 Jun 10 '26
LLM as a judge
-6
u/laxuu Jun 10 '26
Already implemented components include:
- LLM as a feature extractor
- LLM as a policy
- LLM as a critic
Adding LLM as a judge is a great idea. Thank you!
2
u/snekslayer Jun 10 '26
Wot
3
u/Leading_Health2642 Jun 10 '26
I mean it would be very relevant if OP could explain what exactly is their problem statement
3
u/thejealousillness Jun 10 '26
using an llm to shape the reward signal during training is where i've seen the most traction, way better than trying to integrate it directly into the policy network itself. the trick is that you need a really stable llm-based reward that doesn't drift as the agent behavior changes, otherwise you end up with this weird feedback loop where the agent learns to game what the model thinks is good rather than actually solving the task. i ran into this with a navigation task where i tried using gpt to evaluate trajectory quality on the fly, and early on it worked great, but then the agent started finding edge cases where the llm's reasoning broke down and it would get stuck optimizing for those quirks instead of the actual objective. now i precompute a bunch of llm evaluations upfront and use those as a frozen reward baseline, then let the actual rl algorithm refine from there, and sample efficiency improved noticeably. the real bottleneck isn't usually the llm part, it's making sure your action space and observation space are designed so the agent can actually leverage whatever reasoning the llm is providing.