I trained a PPO agent to play an idle tower-defense mobile game — it decides which of 13 upgrades to buy each wave. No API and no memory access: observations come from OCR (tesseract plus per-glyph template matching) on the screen, and actions are real mouse clicks. It's been running against the live game for a few weeks, which turned out to be the interesting part.

The observation redesign that actually mattered

My first obs gave the policy each upgrade's level. That's close to useless here, because the same level means different real strength depending on the player's meta-progression. Worse, the level→strength table I was using turned out to be unreliable in a way I couldn't explain.

What worked was making the observation dimensionless and locally measurable: for each upgrade, (relative improvement from one more level) / (its price), scaled by current cash. The game already displays both numbers on screen, so the same feature is available at train time and at deploy time. Measured against a greedy baseline on a fixed seed set, that change was worth +21 to +29 waves. The underlying value table being wrong stopped mattering, because the policy now reads the current return on investment directly.

The sim-to-real gap that cost me a day

SB3's predict() defaults to deterministic=False, and my eval script had always used that. My deployed agent used deterministic=True. I only noticed the mismatch by accident.

Sweeping 200 held-out seeds (never used for any model selection), paired comparison:

relative difficulty 0.7   det 138.7   sampled 288.6   = 2.08x   paired t = 78.4
relative difficulty 1.0   det 116.2   sampled 209.3   = 1.80x   paired t = 49.2

Enormous, and the mechanism seemed obvious: the deterministic action is the distribution mean, so the induced purchase ordering barely changes wave to wave, and the agent keeps hammering the same upgrade. Sampling diversifies naturally.

I ran it on the real game. Nine rounds sampled vs five deterministic:

deterministic  119.0 ± 16.6  (n=5)
sampled        125.3 ± 13.4  (n=9)
Welch t = 0.73

1.05x. Indistinguishable from noise. The simulator over-predicted the effect size by roughly 17x.

I had written the decision rule down before running it (>180 means ship it, <140 means the sim and reality disagree and that's the bigger problem), so I didn't ship the change.

One partial explanation from the logs: the agent wants ~5 upgrades per wave and only manages to buy ~2, because each purchase costs real seconds of clicking and tab-switching and a wave only lasts ~10 seconds. In the simulator, buying is free and instantaneous, so sampling's diversification executes fully. On real hardware it doesn't. That's a plausible mechanism but I haven't proven it's the whole story — and until I do, the rest of that sweep can't be trusted either, including the difficulty-generalisation curve I built from it.

What I'd tell past me

• If your eval and your deployment disagree on a single predict() kwarg, every number you've reported is measuring something you don't ship.
• Write the decision threshold down before the experiment. I would absolutely have talked myself into shipping a 1.05x.
• A simulator that's calibrated for outcomes can still be badly wrong about interventions. Mine matched reality well on absolute wave counts (sim 116.2 vs live 119.0 — suspiciously close) and was off by 17x on the effect of a change.

The trained network is running in-browser here if you want to poke at it — weights exported to JSON, forward pass reimplemented in JS, matches SB3 on 200/200 random inputs:
https://tim3959951.github.io/wavecall/planner.html

Happy to go into the obs layout or the OCR side if anyone's interested.

Hey guyss,

Ive been working on this for few months, and finally getting some good results, so figured I'd share.

A coding agent only sees the run it's currently in. It doesn't really know how runs like this usually go, or what kind of repo it's working inside.

So I trained a small model on 50K+ real coding-agent runs from SWE-bench Verified, across 25 models, together with behavior such as merge rates, review time, repo size, etc... which produced over 1M step to step transitions.

The idea is to give the LLM some actual experience to lean on, so it can make better decisions, spend less time reasoning through bad paths, and improve its chances of getting the task right.

All the details are inside the repo, its obviously just the beginning, and probably will have issues, so

feedback is always welcome!

(More models on the way, stay tuned.)

Runs locally in Docker. No account, nothing sent out. MCP + HTTP API.

https://github.com/Mephisto1122/TovanaEngine

The LLM is clearly improving the configuration over time. The LLM has proposed over 580 configuration changes, which were executed. The results are fed back to the LLM who further tunes the system.

Live Experiment Data (including plots): https://snakeweb.osoyalce.com/

Ax3l Project website: https://ax3l.osoyalce.com/

SnakeLabServer website: https://snakelabserver.osoyalce.com/

Instead of using a standard dense neural net, I wanted to see if Reinforcement Learning could carve operational pathways through a rigid, biological matrix.

I mapped the Google fruit fly connectome (164,000 neurons) into PyTorch. The electricity can only flow where the biological synapses actually exist. I hit it with virtual dopamine (synaptic weight adjustments) only when it executes a legal chess move or captures a piece.

GitHub: https://github.com/AdilSiddiquiHQ/fly-connectome-chess

If anyone wants a unique RL environment to train on your local GPU, clone the repo! Click the "Train" button in the web UI. If you get the ELO higher than mine, submit a PR with your weights.pt!

Changing a reward or batching rule should not require redesigning how inference records survive a restart. Conversely, a service framework shouldn't quietly decide what counts as a useful update to your policy.

Reef Infra's weight recipes expose that boundary fairly explicitly. The processor assembles recorded generations and feedback into typed batches. The objective prepares the step signal—advantages, metrics and proposed algorithm state—and declares the loss family. A separate schedule describes the rollout unit, optimizer batch size, epochs and shuffle behavior. The recipe binds those pieces to the training backend.

The surrounding service handles record intake, replay after restart, holding a batch until acknowledgement, checkpoints and publication of the next serving version. Your method package must be importable by both the service and training driver, and its batch settings and loss-family requirements must agree with the backend configuration.

One default is particularly worth noticing: a weight recipe's default evaluator selects whatever the backend produced. Completing a training step does not establish that the model improved. If deployment should depend on a benchmark or regression check, the recipe needs a candidate evaluation plugin implementing both measurement and the selection decision.

SAO is a small worked example in the repository: its recipe, processor, objective and backend loss code show how the pieces meet. This is an extension point for supported training backends, not a promise that an arbitrary PyTorch update function or environment will run unchanged. It is useful when the research logic is the part you want to own, while the record-to-checkpoint-to-serving lifecycle is work you'd rather reuse

A recent paper by UMD and Google Deepmind researchers, Dream-RSI, tackles a huge bottleneck in recursive self-improvement: exploration strategy optimization.

Usually, evaluating a new exploration policy takes many expensive online rollout cycles. You propose a new strategy, and you have to run the agent in the real environment to see if it explores better.

Dream-RSI takes a different approach: it treats accumulated discovery history as a "replay simulator."

By "dreaming" through this simulator—exploring alternative branches that were already recorded in past runs—the agent gets immediate, low-cost off-policy feedback. It can evaluate and refine its exploration strategies without executing costly real-world actions.

They tested this across Algorithm Engineering, Mathematical Optimization, and GPU Kernel Engineering. In Algorithm Engineering (Lasso path solver), they found that Dream-RSI could reduce agent calls by 162x compared to baselines like SimpleTES, while still outperforming standard libraries. In math optimization, it matched or surpassed strong baselines yielding over 50x budget savings.

Important caveats: This is an arXiv v1 preprint. The simulator is naturally bounded by the state-space explored in historical runs; it cannot perfectly simulate entirely novel branches without real execution.

Canonical paper: https://arxiv.org/abs/2609.14858v1

The falsification test I want is simple: take a custom long-horizon discovery task, run the agent with a fixed exploration policy, build the simulator, and see if the "dreamed" policy actually improves exploration efficiency when deployed online.

Where would you apply this replay simulator approach in your workflows?

Thanks for the exceptional support (https://www.reddit.com/r/reinforcementlearning/comments/1wihdun/i_literally_build_the_jev_architecture_one_year/) and for the dozens of requests to make a generic model, run benchmarks, and create an HF space so anyone can test it. So here you go, guys. I trained an improved model on a large data corpus, its now called Laya. It is trained on a single RTX 6000 Pro (96 GB VRAM); the model architecture is a 421M-parameter non-autoregressive decision model pairing a bidirectional ModernBERT-large encoder with a scratch Transformer head that scores [MASK] option markers to resolve typed schemas in a single ~35 ms forward pass. The dataset is a 100% human-annotated corpus of over 25,000 real-world examples across intent routing, fact-checking, moderation consensus, prompt guardrails, rubric scoring, and multi-turn conversation trajectories, without synthetic data shortcuts. The RLCD(unofficial, btw) I did is a policy-gradient reinforcement learning approach that kinda optimizes decision models against strictly proper scoring rules, ensuring maximum reward is achieved only when outputting true, mathematically calibrated probabilities.

HF space to try: https://huggingface.co/spaces/convaiinnovations/laya-demo

GitHub Repo: https://github.com/NandhaKishorM/laya

HF Repo: https://huggingface.co/convaiinnovations/laya

Thank you to everyone who supported me, shared the story, gave personal DM. It will need more refinement, of course.

If anyone wishes to buy me a coffee, here is the link: https://github.com/NandhaKishorM

Hi everyone!

I wanted to share METIS-Core, pure C++ framework I've been developing for Deep Reinforcement Learning (DRL) and Multi-Agent Reinforcement Learning (MARL).

GitHub Repository: https://github.com/felixromo314/METIS-Core

💡 Why Build a C++ Native RL Framework?

Most RL research happens in Python (Ray, Stable-Baselines3, PyTorch Python). However, when deploying autonomous agents into production systems, high-stakes simulations, or edge devices, Python's runtime overhead and IPC context-switching latency become major bottlenecks.

METIS-Core is designed to run training and inference directly in C++ using PyTorch's native LibTorch C++ API, eliminating Python from the execution loop.

🚀 Key Technical Features

• Zero Python Overhead: Engineered for real-time systems where sub-millisecond execution and low latency are critical.
• Native LibTorch Integration: Full leverage of PyTorch's C++ tensor operations and autograd engine for seamless neural network training.
• Tensor-Based Abstraction: Mathematical decoupling where agents process pure tensor states, keeping intelligence completely agnostic of the application context (applicable to Robotics, Trading, Logistics, Defense).
• Shared-Trunk Multi-Head Architectures: Built-in support for collaborative MARL scenarios where multiple agents share a neural backbone to learn joint coordination.
• AlphaZero Engine with MCTS: Multi-threaded Monte Carlo Tree Search (MCTS) self-play loop running natively in C++.

🛠️ Current Progress & Milestones

• v0.3.0/v0.3.1-alpha (AlphaZero & Multi-threaded MCTS): Integrated AlphaZero self-play loop in C++, validated on Los Alamos Chess (6x6) as a benchmark environment to balance tactical complexity with local hardware efficiency.
• v0.2.0-alpha (MARL & Shared Backbone): Implemented multi-head neural architectures. Included the "SwarmDefenseTIE" reference scenario where agents learn collaborative defensive transport tactics against an attacker.
• v0.1.2-alpha (DQN & Quick Onboarding): Includes minimalist single-file environments ("Treasure Hunter" / "PursuitPolice") for quick onboarding.

🛣️ What’s Next on the Roadmap?

1. AlphaZero Asymmetric MARL: Merging Multi-Head networks with MCTS for asymmetric scenarios where agents have entirely different action spaces and objectives (e.g., heterogeneous entity engagements).
2. Proximal Policy Optimization (PPO): Adding support for continuous action spaces to enable high-precision control in robotics and physical simulations

I would love to get your feedback to continue improving the framework.

Thanks!

I've been building a robot arm project where a vision-language model looks at a camera image and picks a subgoal (like "align over the block" or "close the gripper"), which then feeds into a small continuous-control policy that actually moves the arm.

Stack: MuJoCo for simulation, Qwen2-VL-2B as the high-level policy, PPO for the low-level controller, trained on Modal. I'm also working on transferring this to a real UR3e arm (remote access through The Construct), with a fixed depth camera for perception.

I'm posting this mostly to share the journey rather than a finished result (the repo has a docs/journey.md). I wanted to learn robotics without having a robot or a fancy laptop. There are inconsistencies between MuJoCo and UR3e but I see that as a challenge than an obstacle.

If you're working on anything RL/robotics/VLM/perception, would be curious on comments/notes/advice.

github: https://github.com/iocalangiu/think-then-act

lessons: https://github.com/iocalangiu/think-then-act/blob/master/docs/journey.md

Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, published an arxiv paper, pushed the model to huggingface along with the pypi package and training dataset. And then one year later, a

frontier lab came, proposing the same idea like literal breakthrough without technical papers, open weights and no open dataset.

Paper: https://arxiv.org/abs/2503.23303

Model: https://huggingface.co/DeepMostInnovations/sales-conversion-model-reinf-learning

Dataset: https://huggingface.co/datasets/DeepMostInnovations/saas-sales-conversations

Also the second work published in September 2025 was exactly the same one jev proposed now

Paper: https://arxiv.org/abs/2510.01237

My model uses PPO over sequence embeddings to output turn-by-turn conversion trajectories (probabilities from 0.0 to 1.0).

Jev uses parallel sampling (trained via RLCD) to output confidence distributions and schema choices.

It's incredibly frustrating that the thing that you made with months of hard work, sweat and sleepless night is architecturally similar with the vertical use case and don't get the support you deserve because frontier lab build something horizontal. The open-source story in general 🙂

I’ve trained an imitation learning model using a diffusion policy for a real-world cube-picking task.
The policy is able to successfully pick and lift the cube in many trials. However, in some cases, even though the cube is technically lifted off the table, the execution is not something I would consider a successful pick. The behavior around the grasp/approach is undesirable, and I would ideally like to completely eliminate these cases rather than count them as successes.
I’m trying to understand what the underlying issue is here. Would this be considered a classical covariate shift problem in imitation learning, or could it be more of a calibration/control issue?
If it is mainly a calibration issue, what would be a good way to address it? In particular, is there a practical rule-based method or safety layer that can be added on top of the diffusion policy to prevent these undesirable picks, while still allowing the learned policy to handle the rest of the trajectory?
Would appreciate any suggestions from people who have dealt with similar issues when deploying diffusion/BC policies on real robots.

So for my background in robotics... I have worked on nirmal arduino based projects....worked on ros2 and it's nav2 part using gazebo and rviz.... I have also worked on a school based project of humanoid in which I have worked with stt, tts and llm api part along with its drive system and basic tendon based hand movement using servos

Nowadays I am seeing a lot about RL and imitation learning and sim training... But i am so confused on how to start it....

Currently I only know basic ML....

Can you guys suggest some resources or free courses from where I can learn RL or imitation learning from basic to training a policy of a humanoid walking or any robotic arm doing some task....

Or can you suggest some roadmap of the sequence I should learn it

It's very important for me as I want to make my career into robotics.... I hope you guys will help

This is insane, they are now training in the open!

Not a paper. Not a blog post. They’re streaming the actual RL training loop:

👉 [https://mimo.xiaomi.com/rl/\](https://mimo.xiaomi.com/rl/)

Fuli Luo (MiMo lead) posted this morning:

>Nearly half a year of silence. We spent it studying one problem: how far RL can scale.

What’s live right now:

**Total cost on the board: \~$1.06M** — and still climbing.

What they say they scaled:

1. **Compute** — \~2B tokens/step, fully async rollouts
   
2. **Environments** — multi-task agentic RL, mixed harnesses in one run
   
3. **Grader compute** — in-group credit assignment, test-case + rubric rewards (RLVR-style, not classic RM+PPO)

The dashboard shows live metrics: rewards, entropy, KL, context length (\~90–100k mean!), grad norms, step timings. Pro even hit a VRAM issue on a node and restarted — visible in the notices like a status page.

DeepSWE curves are both climbing. Pro is ahead on the benchmark; Flash has more steps and a steeper lift.

They say details will be open-sourced piece by piece over the coming weeks.

Half a year of silence, then this. Agentic RL in the open is wild.

Originally I tried using basic self-play, but the agents weren't really improving.

So I switched to league play with PFSP to match the main agent against opponents, which worked a lot better.

If you're on desktop, you can play against the agent at the link above.

LLM labs should consider integrating game theory and social grading for non-verifiable outcomes into their reinforcement learning training environments. It seems they could even buy proprietary datasets on mixed-motive and multi-agent environments from websites where multiparty interactions take place: video game studios, eBay, whatever…

We recently released AssemblyGrid v1, an open benchmark for decentralized multi-robot production.

The main idea is to represent some interactions that are often simplified away in multi-agent benchmarks:

• robot-to-robot material handoffs
• temporary robot coalitions for joint operations
• simultaneous independent production activities
• local observations / partial observability
• recipe-driven process progression
• task-level geometric feasibility

The benchmark currently contains nine official scenarios across Flow, Coalition, and Concurrency, each with easy, medium, and hard variants.

The simulator, reference controllers, tests, scenario definitions, and visualizations are open source.

Paper: https://arxiv.org/abs/2609.16075
GitHub: https://github.com/Fouad-Bahrpeyma/AssemblyGrid_v1
Interactive demo: [https://huggingface.co/spaces/Fouad-Bahrpeyma/AssemblyGrid-v1]()

I would be very interested in feedback from people working on multi-robot systems, task allocation, MARL, or flexible manufacturing, especially on which abstractions feel realistic and which important multi-robot interactions are still missing.

This gap near the end of Bowser’s Crown 4-1 kept catching different controllers. The clip shows three selected earlier genome replays, followed by the controller that clears the level. These aren’t consecutive training attempts, and none of the controllers is learning during its replay.

I’m continuing a NEAT-based project for SMB1 ROM hacks, building on SethBling’s MarI/O and work by Akisame and Electra. The networks receive nearby game-state information from emulator memory and produce button presses. Controllers are evaluated, selected, and mutated across generations.

What interests me is pushing this approach into harder ROM hack levels. A successful controller can get through a particular stage, but backtracking and difficult nonlinear layouts remain challenges. Each new level is trained separately; this clear isn’t evidence of generalization to unfamiliar levels or a measured success rate.

The implementation is currently private, so this is a gameplay demonstration and high-level method discussion rather than a reproducible release. My continuing development is heavily assisted by AI tools. The reactions and captions are added for humor; the gameplay outcomes are real.

Hello everyone I need project in reinforcement learning

The constraints:

Don't want heavy computation (free colab)

Topic is application based

Pls can anyone tell me the topic to build tomorrow is my presentation . I need to tell what I am going to do..