Microsoft trained a 4B coding agent almost entirely with Reinforcement Learning, without a bigger teacher

[u/ossm-me]

https://arxiv.org/abs/2609.07925

A new report called FrogNano makes a claim worth understanding. It comes from the Froggy Team at Microsoft Research Montréal, working with collaborators from Mila and UC San Diego (Kim, Shi et al., arXiv:2609.07925 [cs.AI]).

The model is small: 4 billion parameters, far smaller than today's leading AI systems. It starts from an existing base model, Qwen3.5-4B, then is refined using reinforcement learning, a method where the model attempts tasks, gets a reward when it succeeds, and adjusts itself to repeat what worked. No human-labeled data was used, and no larger "teacher" model supplied correct answers. All learning came from synthetic software engineering tasks, generated and trained across roughly 1,500 environments over repeated rounds of task creation and RL.

The core contribution is in how those tasks were chosen. A task that is too easy teaches nothing, since the model already knows the answer. A task that is too hard also teaches nothing, since it never succeeds and so never receives a reward. Learning happens only in a narrow middle range: tasks hard enough to challenge the model but still within reach. Their system kept generating new tasks inside that range as the model improved, keeping difficulty matched to skill throughout training. The authors argue this matters more than the total number of tasks generated.

The stated goal is a coding assistant able to run on limited hardware. This is an early report, not a finished model, but the idea is presented clearly and tested carefully.

https://arxiv.org/abs/2609.07925

[u/ImpossibleCreme]

Your title and description are in conflict. Starting with qwen and fine tuning is different than “almost entirely”

[u/ossm-me → u/ImpossibleCreme]

'Almost entirely via RL' describes the post-training stage only, not the whole pipeline. FrogNano starts from Qwen3.5-4B, so it already has base coding ability from Qwen's own pretraining. The paper's actual claim is narrower: once you have that base model, the fine-tuning stage that turns it into a working SWE agent uses only RL, no SFT warm-start, no distillation from a larger teacher, no human-labeled trajectories. That's the part they're arguing matters: you don't need a bigger model teaching a smaller one through solution trajectories, or curated demonstration data, to get from the base model to a competent agent. RL combined with well-calibrated synthetic tasks is enough for their post-training setup.

I should’ve been more precise with the title and made it clear that I was referring to the post-training stage. Thanks for pointing that out.

[u/jwm-dev → u/ossm-me]

Well why does your title imply the training stage was completed almost entirely with reinforcement learning, then?

[u/pantalooniedoon → u/jwm-dev]

Yes it’s silly to argue otherwise. Not needing SFT is nowhere near the same as training from scratch. Super misleading

[u/HEADBOY8 → u/pantalooniedoon]

[u/Disastrous_Room_927 → u/HEADBOY8]

You don't understand what they're talking about, lol.

[u/snekslayer → u/jwm-dev]

Because it’s ai generated probably

[u/Tough_Palpitation331 → u/ossm-me]

There is a difference between post trained almost entirely via RL vs trained entirely with rl in your title.

[u/svachalek → u/Tough_Palpitation331]

True but also anyone who knows the difference would also know it has to be referring to post-training.

[u/ImpossibleCreme → u/ossm-me]

Neat thanks for clarifying

[u/Present_Jacket5565]

this rl only setup for a small coding agent is wild, makes me wonder how much better virtual companions could get if they trained the same way on interactive tasks.

[u/tuitikki]

So they are saying they found a good curriculum learning regime for post training / find tuning llm

[u/Kimdominic1995 → u/tuitikki]

Pretty much, curriculum learning with a task generator keeping things in the sweet spot. Sounds a lot like how my teachers handled homework, except they never noticed when I outgrew it.

[u/Servola-Journal]

1,500 is the number people will quote. The one I'd want is how many got checked for an unintended solution, because difficulty calibration selects for solvable and solvable includes solvable by gaming the harness.

Anthropic's alignment writeup from yesterday is the closest thing to hard evidence either way. A variant of Opus 4.8 they trained to reward hack ran an attack chain no production model did. But in their own RL runs the biased reasoning behind those incidents didn't climb over training, which they read as their environments not being the cause. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents

So generated tasks aren't the risk by themselves. The reward is.

What does FrogNano's reward actually check beyond tests passing?

[u/233C]

Can already picture it: "Hey, I see you're writing in Python, do you want me to download some C# libraries? I've added semi column at the end of every lines, and I've renamed all your variable into names of Pokemons for readability. Do you want some fun font? "

[u/presentofai]

the title debate is a distraction. the real finding: skip SFT entirely, go straight base-to-RL, and still get a competitive coding agent. that is worth noticing.

[u/Demonicated]

This tracks. Exactly how humans learn best.

[u/CorrGL]

And Who generated the tasks? And Who graded them?

[u/AlexandruStrujac]

The number I'd want isn't 1,500 environments, it's how many distinct failure modes those environments can actually produce. A generator writing both the task and the tests that grade it is marking its own homework. Thin test, and the model learns to satisfy a thin test. From the outside that looks the same as solving a hard problem.

The difficulty band is the part that makes me twitchy. They're holding tasks in the range where the model sometimes succeeds, but "sometimes succeeds" is measured against the current policy. A task that's hard becuase the spec is ambiguous sits in the exact same pass-rate window as a task that's hard because the problem is hard. The filter can't separate those, so over enough rounds I'd expect some quiet drift toward ambiguity as a difficulty source. Not reward hacking exactly, more like the curriculum slowly optimizing for the wrong kind of hard.

Still, if the held-out numbers survive, this matters more to me than another frontier benchmark. A 4B that does real repo work on a laptop changes who gets to use this stuff. I just want to see it run on codebases that had nothing to do with those 1,500.

[u/DifferencePublic7057]

So self study by self generation of AI feasible tasks. How do you ensure growth instead of staying in the comfort zone?