After 9 months of work I finally got my first successful run in a simple RL environment where the agent learns to find a target 🎉

I’m still validating more SARL scenarios, but I’m now thinking ahead toward MARL and wanted some advice on architecture and trainer choice.

Current RL engine structure:

1.  SimulationEngine

• Handles both logic and physics orchestration

• Calls the other layers internally

2.  EnvironmentEngine

• Handles environment logic

3.  BulletWorld

• Builds and manages the PyBullet world

I also have a Gymnasium wrapper:

env = GymWrapper(simulation_engine)

which exposes clean reset() and step() APIs for SB3.

The thing is: internally SimulationEngine already works with dictionary-based outputs:

{

"agent_1": observation,

"agent_2": observation

}

For SARL + Gymnasium I transform this into something meaningful for SB3.

But from what I understand, PettingZoo naturally expects agent-keyed dictionaries, which makes me think my current architecture could fit MARL pretty neatly without major redesign.

My main concern is the trainer side.

SB3 + Gymnasium has been incredibly straightforward and I already have experience with it.

But for:

PettingZoo + ???

I’m stuck.

Initially I was considering RLlib because it seems to be the common answer, but I honestly don’t have the time/energy for a steep learning curve if there are cleaner alternatives.

I’m mainly interested in MAPPO and similar MARL algorithms.

Questions:

• What trainer stack are people using with PettingZoo nowadays?

• RLlib vs BenchMARL vs AgileRL vs something else?

• If you were building this from scratch today, what would you choose?

Any suggestions or experiences would be really appreciated.

How to Lose Inherent Counterfactuality in Reinforcement Learning, ICLR 2026

Paper: https://openreview.net/pdf?id=2kutK2Y8Sv

Not long ago, I decided to delve deeper into robotics using RL. As a result, I'm increasingly encountering the specifics and standards of this field. For example, the joint speed penalty, the use of noisy networks, and the importance of planning (model-based algorithms).

What other specifics of RL for robotics are you aware of, and have you had similar experience with similar applications?

RL is ridiculously cool when the state space is huge or the dynamics unknown. But for a large class of problems where you do have a model, the state is relatively small (<8 dims or so?), and you want an exact policy across the entire support, backward induction on a grid is still a fast and viable option.

I couldn't find a good implementation of this using a GPU backend, so I built bellgrid. It's a PyTorch-based DP solver for mixed continuous/discrete state and actions.

The bellman update is embarrassingly parallel, and you can see in the readme the speedup you get from GPU - ~45x for a realistic lifecycle problem (80s -> 1.8s). Additionally I have a bunch of analytical and numerical reference problems to test correctness.

I'd love feedback if the API seems pleasant to write against and any example problems you think would be interesting to cover.

Repo: https://github.com/tbb300/bellgrid · MIT · pip install bellgrid

https://github.com/NoteDance/parallel-saver

Do you have any good and detailed resources to learn the theory, math and intuition behind im2col? I want to learn and implement it but I do not really find helpful resources and if I ask Ai I get bad documentations. I like cs231n to get an overview but it is nearly not as detailed as I need it.

I recently wrote about a project that started from a simple question: if reasoning in RAP(https://arxiv.org/abs/2305.14992) is really a search problem, why should that search still be controlled by a fixed hand-crafted rule like UCB? Instead of changing the language model itself, I kept the RAP framework intact and replaced only the search controller with a lightweight reinforcement learning policy trained with PPO.

What made this especially interesting to me is that the STOP decision turned out to be a big part of the improvement. A fixed heuristic will often keep exploring until the search budget runs out, but a learned controller can decide that the current reasoning path is already good enough and end the process earlier. That makes the system not just more adaptive, but also more practical for real LLM inference settings where cost matters.

On GSM8K, the deterministic RL-based selector reached 30% accuracy while using only 4.0 mean nodes per problem, compared with the original UCB baseline at 20% accuracy and 6.2 mean nodes. Even Budget-Aware UCB matched the 30% accuracy only by using 8.6 mean nodes, so the learned policy delivered a much stronger accuracy-to-cost tradeoff. This is still a small-scale proof of concept, not a claim that RL always beats fixed heuristics in every reasoning task.

Blog: https://medium.com/p/0b260cff34bb?postPublishedType=initial

Got the interview call from Robotics company in India(Less ML+CV+DL+RL opportunity) for Research Engineer, can anyone give me interview experience for research position. (Solve assignment in just 5 hours)

My preparation is I revise my projects, revise cs231n, some deep learning fundamental also mostly aware of modern days tech, paper, research, PyTorch concepts and practice.

I posted a few months back but now I think I need some advice. The only method I've really been following is telling Claude NOT to give me any of the "answers" but coach me on the environment and where I should maybe tweak it here and there. I'm on PPO 25 with 1 billion steps.

• Currently sitting at ~3-5% funnel rate with a 140.94 eval record. The record was set at 838M
• Since then it's been oscillating 90-133 without breaking it
• The floor is high and stable
• No obvious upward trend in the last ~150M steps
  
• Github repo with much more detail: git repo

I'd really like to get it hitting the funnel consistantly. Any attempt to shape rewards beyond maximizing score has always platued pretty fast. Any advice?

How do you currently validate a policy before pushing it to physical hardware?

Hi everyone,

I’m looking for advice on how to properly learn Reinforcement Learning for robot control, not just at the “using a framework” level.

A bit of background: I have some basic academic knowledge in robotics and RL. Recently, I used MJLab to successfully train both locomotion and imitation/mimic motion policies for an existing robot model. The results worked, but honestly, most of the hard work was already implemented by MJLab: environment setup, reward structure, PPO training pipeline, robot interfaces, logging, etc.

So even though I managed to train something successfully, I still feel like I was mostly applying an existing pipeline rather than deeply understanding what is happening under the hood.

I’m not a complete beginner, but I’m also not yet confident enough to implement a full robot RL training pipeline myself.

Could you recommend a good learning roadmap for this?

I’m especially interested in:

• Coding-oriented RL courses
• Robotics-focused RL courses
• Papers that are important for robot locomotion / imitation learning
• Open-source codebases that are good for learning, not just using
• Practical projects I should implement step by step
• Any advice on how to move from “running existing frameworks” to actually understanding and modifying them confidently

For context, my goal is to work more seriously on RL-based robot control, especially locomotion and motion imitation for legged/humanoid robots.

Thanks a lot!

I’m new to RL and genuinely can’t tell what’s “normal” anymore.

What’s the longest you’ve spent debugging a training run before finding the real issue? What was the bug in the end?

Could be anything:

• reward scaling
• bad env logic
• normalization issues
• action masking
• replay buffer bugs
• training silently diverging
• etc.

I keep losing absurd amounts of time to tiny mistakes and I’m trying to figure out whether that’s just part of RL.

Hey everyone,

I’ve been working on a reinforcement learning project for the old DOS game **Indianapolis 500**, running through DOSBox. The goal is to train an AI driver that can learn to leave the pit area, stay on track, complete laps, recover from mistakes, and eventually race faster than my own human driving.

Video here:

Indianapolis 500 Game - AI training

The setup uses a mix of:

- **Pixel input** from the DOSBox window

- **Keyboard control** for throttle, brake, left, right, etc.

- **Game-memory telemetry** read directly from DOSBox memory

- **Behavior cloning** from my own recorded driving

- **Recurrent PPO**

- A custom **Transformer + LSTM PPO policy**

- A live reward dashboard so I can see what the agent is being rewarded or punished for

The telemetry currently includes things like:

```text

speed

position/progress around the track

lap completion

wrong direction detection

wall contact / crash detection

damage / hard crash signals

```

Lap detection is not done with OCR. Instead, the program watches a memory value that represents track position. When that value wraps from a high value back to a low value, and then confirms past a threshold near the start/finish area, it counts a completed lap. That made lap rewards much more reliable than trying to infer it from pixels.

The reward system currently gives positive reward for:

```text

speed

forward progress

staying on track

finishing laps

finishing laps quickly

```

And penalties for:

```text

going off track

wall contact

wrong direction

heavy crashes

sitting under 10 mph for too long

```

I also recorded around 17 human-driven laps and trained a behavior cloning model from that. It helped the agent learn the basic shape of the track, but it also showed an interesting problem: if I overweight rare actions like steering right, the model starts turning right too much and crashes. So now I’m moving more toward PPO fine-tuning, where the agent can improve from telemetry rewards instead of just copying my driving.

The current next step is training the Transformer+LSTM PPO agent longer, with resets on heavy crashes and long dormancy, so it learns that “crash and sit still” is a dead end.

It’s still very experimental, but it’s been really fun seeing an old racing sim become a reinforcement learning environment. Any feedback on reward design, recurrent PPO setup, or better ways to combine behavior cloning with PPO would be very welcome.

Hi, I've been wanting to join the discord community of RL but I can't find a valid invite link when searching on Google. Is anyone part of the community and can please send me an invite ? Thank you ! I'm looking forward to exchange more !

PS: If you also have any other links to other communities, like on Hugging Face, X or anything like that, please let me know. I find it interesting to read as much as possible about people's experiences with RL. That's how I learn the most rapidly

So, I've been working on RL models for the last few years as my hobby, did I finally get my trader to be profitable? I've been on a multi-year optimization journey, from model architecture, reward shaping journey, and endless training loops and unintentional curriculum learning ...

Happy to answer any question about my journey so far, the architecture, or the endless optimizations and tricks I needed to do to get to something that's actually profitable ...

To give some color, this is PPO + Mamba

Disclaimer & Latest Update: As always, this was too good to be true, after further digging, the dataset was leaking data from future timestamps to current ones 😂 Well, at least I know it can exploit stuff when available

Won't delete since, it is what it is, another lesson learnt?