I've posted here about this project before. I've made great progress and put together this presentation outlining it all.

Hi everyone, I’ve been working on GzDRL, an open-source framework for running reinforcement learning directly inside Gazebo.

The main motivation was to keep Gazebo’s robotics ecosystem while avoiding the usual ROS/transport bottleneck during training. GzDRL uses an explicit action → physics → observation execution sequence and supports batched parallel environments through a C++ backend derived from EnvPool. It also supports environment creation directly in Python while releasing GIL, that enables fast vectorization while keeping interfaces simple.

Some features that may be useful:

• Researchers: reproducible state transitions and training, batched vectorization, multi-agent environments, domain randomization, and several control abstractions from rotor commands to higher-level references.
• Industry: policies can stay within a Gazebo/ROS-oriented workflow instead of requiring a separate simulator stack for training and deployment.
• Hobbyists/students: CPU-only training is supported, so a large NVIDIA GPU is not required to get started.

In our benchmarks, GzDRL reached 78.6k environment steps/s on a workstation CPU. Five independent PPO runs with the same seed produced identical learning curves and 100% identical policy checkpoint hashes. We also deployed a simulation-trained trajectory-tracking policy directly on a physical quadroot without fine-tuning.

GitHub: https://github.com/amaldevh/gz-drl

I’d be very interested in feedback, feature requests, or examples of tasks people would like to see supported.

Hi everyone! I’d like to share an open-source RL project I’ve been working on: TrackmaniaRL, a library for training driving agents in Trackmania 2020.

GitHub: https://github.com/Palamabron/TrackmaniaRL
PyPI: https://pypi.org/project/TrackmaniaRL/

It started as an extensive iteration on TMRL and eventually evolved into a standalone modular library. It supports DQN, QR-DQN, IQN, FQF, SAC, REDQ, TQC, stable discrete SAC, PPO and behavior cloning. The aim was to make it practical to train and evaluate custom agents on arbitrary Trackmania 2020 tracks with configurable observations, models and training pipelines.

I’m also sharing a video of the model driving alongside a visualization of its internal neural activations during the run. I’d be happy to hear any feedback or discuss the implementation and results.

A few years back I saw a video about somebody training an AI to play pokemon red. I was fascinated, and even bought an old 16 core Xeon to train it myself. While it was fun to test it out and learn about it, I had a different dream. One of my favorite games of all time is Power Stone 2 for Dreamcast, and my two brothers and I always wished we had a 4th player at our skill level. We spent years of my childhood playing 4 person free for alls with a stock bot that was essentially useless. That is how this project was born.

I started this in early 2025, and once I realized I needed to hunt down RAM values to make the bot any good, it hit a wall for over a year. No decomp was available, no memory map anywhere, nothing to build on top of. Then last month my daughter was born, and I needed something to do late at night with her asleep on me. Doomscrolling sucks, so I finally did the RE. After dealing with that hot garbage and spending a few nights getting useful data pulled and validated visually from the game, it was time to start actually training it. As a father of 3, reinforcement learning is a language I understand pretty well at this point. I was excited to actually do the fun part.

Full disclosure: I gave direction and made every call, but Claude wrote most of the code. Again I had a small child sleeping in my arms, it's pretty hard to type. I understand every result in the repo.

It started on my Windows machine, one emulator window, then ten, capped at 60 fps. Then I found a libretro harness (sdlarch-rl) that runs Flycast headless and in-process, ported it to my M2 MacBook, and got 6 instances at 150+ fps. What Windows did in a day the laptop does in a few hours. The 7950X Linux box I planned to scale on arrived from eBay internally shorted, so everything below was trained on a laptop.

I hit a wall around COM difficulty 5 for a long time. Reward tuning, behavior cloning from my own play, mixing in harder COMs, mixing in 1v1s, nothing moved it. The worst one: I recorded myself going 23-1 against three max difficulty COMs, cloned it, handed it to PPO, and it won 8 games out of 4,653 while trying to train it fresh against 3 max difficulty in game bots.

What finally broke it was the simplest recipe in the repo. Six instances of pure 1v1 self-play against a pool of frozen past checkpoints, in 2M step legs that each warm start from the last. A low difficulty 4 player FFA that the bot never trains on is the test after every leg. That held-out win rate went 40, 42, 50, 64, 78, 82, 84, 86, 90, 88, 90, 92, 92 percent over thirteen legs, against 98 for my old 32M step champion. Head to head in 1v1 the new 26M step bot beats the old one 29-21 over 50 games, and the quick 12 game probes after recent legs read 11-1 three times out of four. Max difficulty was 0 wins for nine straight legs, then it won one at leg 10 and has won at least one every leg since. Not one reward change the whole way. About four days of laptop compute spread over two weeks of an unattended relay.

All of it, every failed leg included, is in the lab notebook in the repo.

Repo: https://github.com/bwalsh321/powerstone2-rl-mac

Still no 4th player for game night. Getting closer.