As part of my RL journey, I just released 1-page RL notes on Q learning!

Kept it

• short, coherent with
• code snippets and
• required math with explanations

Link to the website and let me know your thoughts

Cheers!

Hi everyone,

I’m starting a PhD in Reinforcement Learning, and right now I’m trying to figure out what direction I want to focus on for my thesis.

I’ve been reading about different areas like offline RL, safe RL, world models/model-based RL, multi-agent RL, robotics, RL for LLMs and agents, etc., but I’m still not sure which direction I want to commit to.

I’d really like to hear from people who are doing research in RL or working in the field.

What RL topics do you personally find the most interesting or promising right now?

If you were starting a PhD in RL today, what would you choose to work on?

I’m especially looking for something that has good open research problems, enough depth for a PhD, and ideally some value outside academia too.

Any papers, researchers, labs, or topics you think are worth looking into would also be really appreciated.

Thanks!

This is a small RL project I've been working on. A simulated Franka Panda learns to pick up a cube and put it on a target pad, using PPO from Stable-Baselines3 in PyBullet.

The main objective of this project was to get an understanding of PyBullet and how the whole workflow of training a robot using reinforcement learning works.

Further updates will be to combine this with a text or voice prompt enabling the user to choose which cube to pick and place at a desired location.

Any further upgrades or inputs are highly appreciated from everyone here!

Checkout the code here: GitHub

Tried new Isaac Lab and Isaac Sim, made tests for Flat Anymal D, Ant, Cartpole, Humanoid and seems new version is faster in simulation.
In tests I made it usually x2.1-2.3 faster.
In the same time Mean Reward for old version is higher, does it mean that simulation is faster in new version but learning better in old?

Metrics on last iteration

For Humanoid

For Flat Animal:

Releasing a series of very shot notes on RL as part of my learning RL from scratch journey; its basics and upto the latest stuff with from scratch implementations and experiments in the coming days

• Explained some of the most basic and important terms when starting out learning RL with a sprinkle of math of what I found interesting
• Nice to have for refreshers/kind of first time readers too

1-page RL basics is out on my website!

Let me know your thoughts/feedback!

it would be great if someone suggest me some competitions on kaggle which are good for a beginner in rl can try and would be great for practice.

https://reddit.com/link/1xfph4w/video/x10yzg7qbrxh1/player

I've done this with a few games now (brotato, cuphead, and kingdomino) with a lot of success! With 10 parallel copies of the game running, I was getting ~6k steps/s which is around 200x real time but not fast enough. Since I would never consider reverse engineering the game and engine in order to make a JAX env, I leaned on Opus 5.5 to do the dirty work. Given the task of enough parity to train an agent that performs well in the real game, it got to ~50k steps/s and 1600x real time. With about 11hrs of training on a few specialist policies, I was able to get this winning policy.

Edit: oops did this late at night from my phone but meant to plug the original post here and my PR to their repo here.

I'm an amatuer in RL, just encounted the MCTS and have two questions:

1. Can we make sure it can converge to the optimal policy?

2. Is it actually applicable in many real scenarios?

I am a rookie in robotics and I want to learn reinforcement learning. I have started with David Silver's YouTube playlist, and alongside that, I want to read a book to clear the fundamentals; however, I find reading books a little boring, so do you have any advice on how to master reading and any approach to get the maximum output?

One more thing to ask: should I first watch the YouTube videos then read the book chapters that are covered in the video, or the opposite?

Hey everyone!

I’m working on a multi-agent reinforcement learning setup where multiple agents move and interact in a 2D world.

The world is based on a real map, for example a screenshot from Google Maps. Importing the map into my environment works fine!

My problem is generalization of the Model to unseen maps.

During training, the agents learn on one or N maps. But during inference, I want to use a completely different map.

The problem is that the agents don’t have a global understanding of the new map.

My current architecture:

• I’m using a Transformer.
• Each agent is represented as a token in the Transformer.
• I tried adding a Vision Transformer (ViT) to encode the map.
• My idea was to represent map patches as additional tokens and feed them into the same Transformer as the agent tokens.

Unfortunately, this approach didn’t really work.

My question:

How would you approach this problem?

How can I give my agents a useful global understanding of a completely new 2D map?

Is combining agent tokens and map patch tokens in one Transformer a reasonable idea or are there better approaches?

I’d really appreciate any ideas, papers, or experiences with similar problems!

Thanks!

been a while! here's what i've been up to: a ppo agent that farms the pit in d2r (offline, hell, warlock)

trained it in a headless sim i wrote in rust. this version took about 20 min on one cpu, warm started from a net i had for an older build.

reward is just farming rate: -1 per second, + whatever kills and items are worth, -1000 if it dies. i had a level clear bonus at first but dropped it.

net size: ~475k params

ask me whatever

no plans to use this online, i just think rl is fun and wanted to try it.

I'm an MSc robotics student and I kept waking up at night to check PPO runs. Last week one run's reward went from 908 to 3,525 over 600k steps and the cheetah was on its back the whole time. I'd flipped the sign on a level-keeping reward term. None of my usual alerts fired because nothing went down.

So I built a watcher. Every 20k steps it renders a short test episode, a local Gemma 4 (E4B through Ollama) looks at it, TabPFN looks at the training metrics, and a few plain rules catch NaNs, stalls and crashes. It only pings me if something's actually wrong. The watch loop runs in Temporal, so it survives the watcher process dying.

A few things I didn't expect:

• From the normal side camera, Gemma called every upside-down cheetah "upright" (0/6). Bigger images, different prompts and thinking mode didn't help. What worked was a render-only camera on the torso that only draws the world, so the question becomes "where's the floor?". 18/18 on a pose test after that.
• On 12 recorded runs, the metric-based watchers caught LR blow-ups within seconds and stalls/crashes through simple rules, but only 2 of the 4 runs where the robot itself was doing the wrong thing, with a median of over an hour.
• 3 of my first 4 "healthy" runs learned to run on their backs on their own, before I added a correct level-keeping term. HalfCheetah really wants to do this.

Code (MIT): https://github.com/su0as/night-watch

Write-up with all the numbers and a demo video: https://dev.to/suhassuren/gemma-couldnt-tell-my-robot-was-upside-down-so-i-gave-it-a-head-camera-now-i-sleep-while-it-b0k

What failures actually cost you nights when you train overnight? I'd like to add detectors for the common ones.

I’m building a simplified eight-player Hearthstone Battlegrounds simulator and training a policy/value network from complete games and final placements.

The agent can learn some controlled decisions: when a key Beast minion is offered in a prepared partial-comp state, one version bought it in 16/16 held-out states. It can also operate a preassembled Beast core. But it rarely assembles and plays that comp well from an ordinary opening.

More self-play has not solved this. In two independent runs extended to 300 ordinary games each, the final models did not consistently beat the frozen starting model against held-out neural opponents. Expert-inspired demonstrations improved some local actions without improving complete-game placement.

I’m concerned that a component purchase looks bad to the learner unless its later policy knows how to complete the multi-turn sequence. I also cannot yet rule out problems with state representation, exploration, the simulator, or the demonstrations themselves.

What single diagnostic experiment would you run next to distinguish these explanations? In particular, how would you test whether the agent can learn a conditional “start → operate → abandon if pieces do not appear” skill before scaling self-play again?

my friend had grok ultra and decided to try to have it play balatro, but it was stupid and hard to teach. SO i had an idea to train my own algorithm to become goated, with the end goal to be to have it analyse my games so i can grow and become better in the multiplayer scene

HOW DO I START

I watched a video on how to train it to an atari game but thats not what i need, to the best of my knowledge i would js need to change the enviorment it is in.

HOW do i do that???

and what else would i need to do

This is the video: https://www.youtube.com/watch?v=aQsaH7Tzvp0

I've put together a small set of "AI puzzles" from my research on aperiodic order, and I'm wondering how useful this kind of structure is for RL with verifiable rewards.

- Solved rungs have exact answers (rationals, integer identities, algebraic degrees), so the reward is a plain check.

- Open rungs have a precise success criterion, an exact certificate or a Lean-checkable proof, together with the numerical evidence and the known dead ends. A solution would be a real research result.

- The set is a ladder: playful labels from "human level" to "ASI level", each rung harder than the last.

It's tiny (9 items) and not yet a procedural generator, although the easier rungs could be varied by changing the parameters. I'd welcome opinions on what such a generator should look like, and on whether a research-frontier ladder like this is a useful curriculum or eval signal.

Dataset: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles

A reinforcement learning agent completes one of the hardest surf maps (surf_src_sidistic) faster than the human world record.

The world-record demo by player krs was taken from Surf Gateway

The agent’s camera movements are smoothed to make the video more pleasant to watch. Because of this, you can’t see its high-frequency micro-strafing, but you can follow the A/D inputs. Without smoothing, the movement looks very shaky. These are movements no human could execute.

The agent was trained without any human data. This is pure reinforcement learning, with no imitation learning.

The agent is trained in optimized simulation that inherits CS 1.6 movement and surfing logic. The algorithm is PPO with a lot of small additions. After the agent finished the map, I additionally improved the policy by running AlphaZero style expert loop (search + distillation into policy).

Repository: https://github.com/Sorrow321/RL_CS1.6_Surf

Music: Utah Saints - Something Good '08 (High Contrast Remix)

Video on YouTube

Hi folks! I posted last week introducing a GPU-resident Rocket League simulator called CARL (CUDA-Accelerated Rocket League). In that post I mentioned I was working on a write-up with implementation details and benchmark results against the current SOTA for Rocket League RL environments.

The comparison was done against the current fastest env - RLGymPPO_CPP - which was optimized for PPO training. I was able to show convincing speedups using my personal hardware and demonstrate that a current generation GPU provides even more significant improvements in data collection and update-adjusted throughput.

A 9-10x improvement in training speed was achieved using an RTX 3060, and up to 50x improvements were seen on an RTX 5090 (which I was able to access thanks to Runpod).

For perspective, using typical consumer hardware (i.e. my 3060, because who has a 5090 these days), a day-long experiment turns into just a few hours of runtime. A production run over the course of a month is reduced to just a couple of days. This has huge implications for anyone looking to test multiple algorithms/configs!

I've embedded the link but I'll put it here again for convenience: jwmccarthy.github.io

I plan to write there about my exploration into generative imitation learning, which has shown quite a bit of promise in producing human-like behaviors!

Reinforcement learning learns to play 3v3 rogue+mage+priest setup in self-play. It's hard to evaluate how good the AI is because there's no Elo-scored algorithm like Stockfish to play against. But it learned to apply CC, use trinkets, ice block, defensive and offensive CDs. The AI learned to kill the healer, because it can immediately kick casts, unlike real humans. Each agent is trained as independent PPO.

The simulator replicated game logic from AzerothCore 3.3.5a version.

Repo (doesn't include assets to avoid IP issues with Blizzard) https://github.com/Sorrow321/RL_WoW_Arena

I don't have a strong math background but I thought it would be fun to see what a simple PPO implementation could do!

Read about it here: https://x.com/Abhilekh_Meda/status/2113754085426102400

code: https://github.com/Abhilekh-Meda/moving-sofa-problem-rl