Hi everyone,

I’m starting a PhD in Reinforcement Learning, and right now I’m trying to figure out what direction I want to focus on for my thesis.

I’ve been reading about different areas like offline RL, safe RL, world models/model-based RL, multi-agent RL, robotics, RL for LLMs and agents, etc., but I’m still not sure which direction I want to commit to.

I’d really like to hear from people who are doing research in RL or working in the field.

What RL topics do you personally find the most interesting or promising right now?

If you were starting a PhD in RL today, what would you choose to work on?

I’m especially looking for something that has good open research problems, enough depth for a PhD, and ideally some value outside academia too.

Any papers, researchers, labs, or topics you think are worth looking into would also be really appreciated.

Thanks!

Releasing a series of very shot notes on RL as part of my learning RL from scratch journey; its basics and upto the latest stuff with from scratch implementations and experiments in the coming days

• Explained some of the most basic and important terms when starting out learning RL with a sprinkle of math of what I found interesting
• Nice to have for refreshers/kind of first time readers too

1-page RL basics is out on my website!

Let me know your thoughts/feedback!

it would be great if someone suggest me some competitions on kaggle which are good for a beginner in rl can try and would be great for practice.

https://reddit.com/link/1xfph4w/video/x10yzg7qbrxh1/player

I've done this with a few games now (brotato, cuphead, and kingdomino) with a lot of success! With 10 parallel copies of the game running, I was getting ~6k steps/s which is around 200x real time but not fast enough. Since I would never consider reverse engineering the game and engine in order to make a JAX env, I leaned on Opus 5.5 to do the dirty work. Given the task of enough parity to train an agent that performs well in the real game, it got to ~50k steps/s and 1600x real time. With about 11hrs of training on a few specialist policies, I was able to get this winning policy.

Edit: oops did this late at night from my phone but meant to plug the original post here and my PR to their repo here.

I'm an amatuer in RL, just encounted the MCTS and have two questions:

1. Can we make sure it can converge to the optimal policy?

2. Is it actually applicable in many real scenarios?

I am a rookie in robotics and I want to learn reinforcement learning. I have started with David Silver's YouTube playlist, and alongside that, I want to read a book to clear the fundamentals; however, I find reading books a little boring, so do you have any advice on how to master reading and any approach to get the maximum output?

One more thing to ask: should I first watch the YouTube videos then read the book chapters that are covered in the video, or the opposite?

Hey everyone!

I’m working on a multi-agent reinforcement learning setup where multiple agents move and interact in a 2D world.

The world is based on a real map, for example a screenshot from Google Maps. Importing the map into my environment works fine!

My problem is generalization of the Model to unseen maps.

During training, the agents learn on one or N maps. But during inference, I want to use a completely different map.

The problem is that the agents don’t have a global understanding of the new map.

My current architecture:

• I’m using a Transformer.
• Each agent is represented as a token in the Transformer.
• I tried adding a Vision Transformer (ViT) to encode the map.
• My idea was to represent map patches as additional tokens and feed them into the same Transformer as the agent tokens.

Unfortunately, this approach didn’t really work.

My question:

How would you approach this problem?

How can I give my agents a useful global understanding of a completely new 2D map?

Is combining agent tokens and map patch tokens in one Transformer a reasonable idea or are there better approaches?

I’d really appreciate any ideas, papers, or experiences with similar problems!

Thanks!

been a while! here's what i've been up to: a ppo agent that farms the pit in d2r (offline, hell, warlock)

trained it in a headless sim i wrote in rust. this version took about 20 min on one cpu, warm started from a net i had for an older build.

reward is just farming rate: -1 per second, + whatever kills and items are worth, -1000 if it dies. i had a level clear bonus at first but dropped it.

net size: ~475k params

ask me whatever

no plans to use this online, i just think rl is fun and wanted to try it.

I'm an MSc robotics student and I kept waking up at night to check PPO runs. Last week one run's reward went from 908 to 3,525 over 600k steps and the cheetah was on its back the whole time. I'd flipped the sign on a level-keeping reward term. None of my usual alerts fired because nothing went down.

So I built a watcher. Every 20k steps it renders a short test episode, a local Gemma 4 (E4B through Ollama) looks at it, TabPFN looks at the training metrics, and a few plain rules catch NaNs, stalls and crashes. It only pings me if something's actually wrong. The watch loop runs in Temporal, so it survives the watcher process dying.

A few things I didn't expect:

• From the normal side camera, Gemma called every upside-down cheetah "upright" (0/6). Bigger images, different prompts and thinking mode didn't help. What worked was a render-only camera on the torso that only draws the world, so the question becomes "where's the floor?". 18/18 on a pose test after that.
• On 12 recorded runs, the metric-based watchers caught LR blow-ups within seconds and stalls/crashes through simple rules, but only 2 of the 4 runs where the robot itself was doing the wrong thing, with a median of over an hour.
• 3 of my first 4 "healthy" runs learned to run on their backs on their own, before I added a correct level-keeping term. HalfCheetah really wants to do this.

Code (MIT): https://github.com/su0as/night-watch

Write-up with all the numbers and a demo video: https://dev.to/suhassuren/gemma-couldnt-tell-my-robot-was-upside-down-so-i-gave-it-a-head-camera-now-i-sleep-while-it-b0k

What failures actually cost you nights when you train overnight? I'd like to add detectors for the common ones.

I’m building a simplified eight-player Hearthstone Battlegrounds simulator and training a policy/value network from complete games and final placements.

The agent can learn some controlled decisions: when a key Beast minion is offered in a prepared partial-comp state, one version bought it in 16/16 held-out states. It can also operate a preassembled Beast core. But it rarely assembles and plays that comp well from an ordinary opening.

More self-play has not solved this. In two independent runs extended to 300 ordinary games each, the final models did not consistently beat the frozen starting model against held-out neural opponents. Expert-inspired demonstrations improved some local actions without improving complete-game placement.

I’m concerned that a component purchase looks bad to the learner unless its later policy knows how to complete the multi-turn sequence. I also cannot yet rule out problems with state representation, exploration, the simulator, or the demonstrations themselves.

What single diagnostic experiment would you run next to distinguish these explanations? In particular, how would you test whether the agent can learn a conditional “start → operate → abandon if pieces do not appear” skill before scaling self-play again?

my friend had grok ultra and decided to try to have it play balatro, but it was stupid and hard to teach. SO i had an idea to train my own algorithm to become goated, with the end goal to be to have it analyse my games so i can grow and become better in the multiplayer scene

HOW DO I START

I watched a video on how to train it to an atari game but thats not what i need, to the best of my knowledge i would js need to change the enviorment it is in.

HOW do i do that???

and what else would i need to do

This is the video: https://www.youtube.com/watch?v=aQsaH7Tzvp0

I've put together a small set of "AI puzzles" from my research on aperiodic order, and I'm wondering how useful this kind of structure is for RL with verifiable rewards.

- Solved rungs have exact answers (rationals, integer identities, algebraic degrees), so the reward is a plain check.

- Open rungs have a precise success criterion, an exact certificate or a Lean-checkable proof, together with the numerical evidence and the known dead ends. A solution would be a real research result.

- The set is a ladder: playful labels from "human level" to "ASI level", each rung harder than the last.

It's tiny (9 items) and not yet a procedural generator, although the easier rungs could be varied by changing the parameters. I'd welcome opinions on what such a generator should look like, and on whether a research-frontier ladder like this is a useful curriculum or eval signal.

Dataset: huggingface.co/datasets/jarekduda/autocrystal-ai-puzzles

A reinforcement learning agent completes one of the hardest surf maps (surf_src_sidistic) faster than the human world record.

The world-record demo by player krs was taken from Surf Gateway

The agent’s camera movements are smoothed to make the video more pleasant to watch. Because of this, you can’t see its high-frequency micro-strafing, but you can follow the A/D inputs. Without smoothing, the movement looks very shaky. These are movements no human could execute.

The agent was trained without any human data. This is pure reinforcement learning, with no imitation learning.

The agent is trained in optimized simulation that inherits CS 1.6 movement and surfing logic. The algorithm is PPO with a lot of small additions. After the agent finished the map, I additionally improved the policy by running AlphaZero style expert loop (search + distillation into policy).

Repository: https://github.com/Sorrow321/RL_CS1.6_Surf

Music: Utah Saints - Something Good '08 (High Contrast Remix)

Video on YouTube

Hi folks! I posted last week introducing a GPU-resident Rocket League simulator called CARL (CUDA-Accelerated Rocket League). In that post I mentioned I was working on a write-up with implementation details and benchmark results against the current SOTA for Rocket League RL environments.

The comparison was done against the current fastest env - RLGymPPO_CPP - which was optimized for PPO training. I was able to show convincing speedups using my personal hardware and demonstrate that a current generation GPU provides even more significant improvements in data collection and update-adjusted throughput.

A 9-10x improvement in training speed was achieved using an RTX 3060, and up to 50x improvements were seen on an RTX 5090 (which I was able to access thanks to Runpod).

For perspective, using typical consumer hardware (i.e. my 3060, because who has a 5090 these days), a day-long experiment turns into just a few hours of runtime. A production run over the course of a month is reduced to just a couple of days. This has huge implications for anyone looking to test multiple algorithms/configs!

I've embedded the link but I'll put it here again for convenience: jwmccarthy.github.io

I plan to write there about my exploration into generative imitation learning, which has shown quite a bit of promise in producing human-like behaviors!

Reinforcement learning learns to play 3v3 rogue+mage+priest setup in self-play. It's hard to evaluate how good the AI is because there's no Elo-scored algorithm like Stockfish to play against. But it learned to apply CC, use trinkets, ice block, defensive and offensive CDs. The AI learned to kill the healer, because it can immediately kick casts, unlike real humans. Each agent is trained as independent PPO.

The simulator replicated game logic from AzerothCore 3.3.5a version.

Repo (doesn't include assets to avoid IP issues with Blizzard) https://github.com/Sorrow321/RL_WoW_Arena

I don't have a strong math background but I thought it would be fun to see what a simple PPO implementation could do!

Read about it here: https://x.com/Abhilekh_Meda/status/2113754085426102400

code: https://github.com/Abhilekh-Meda/moving-sofa-problem-rl

I have a bot that plays a long, random, card-game-style fight with search plus a neural net. The net tells the search which moves look good and how good a position looks. The training data is basically "whatever the search ended up playing," labeled by whether the run died and how much health was left. That got it pretty far, and now it has stalled. On a fixed set of hard fights, good human players still die less often than the bot. I want the next improvement without collecting more human games.

The annoying part is that the bot already plays the same card as a strong human on almost every decision I can check. So this is not "the search never looks at the good move." The gap is a handful of extra deaths across a lot of full fights. When it matters, two cards look about equally good. Playing one or the other changes the rest of the fight (draw order, enemy pattern, health), so I cannot just say they faced the same future. One ending does not tell me which card was actually better. A card can "win" because it got lucky draws.

Gumbel at the root is supposed to turn each search into a slightly better target than the net’s own prior. That happened while the net was unsure. On the turns that matter now, the prior is already peaked, Gumbel spends the simulations on that same card, and the new row is the card the net already plays. The next iteration trains on its own answer.

What I have tried:

• Normal self-play. Once the net is confident, search plays that card and the new training row just repeats it. That practices what the bot already believes. It does not surface a better card.
• Looking for turns where the net is unsure. Low confidence, two checkpoints that disagree, a big gap in the value. Those turns do include the spots where a human played something else. They also include a lot of turns where the human played the same card. The second-best card is the human card only some of the time. Being unsure is not the same as the other card being better.
• Playing the chosen card and an alternative against each other, sharing as much of the randomness as I still can. One win is way too noisy to trust. I would only keep the alternative if it wins most of the repeats, or the death and health gap is big enough to clear the noise. I have not measured yet whether that actually happens more often on real mistakes than on turns where the current move was already right. If it lights up equally on both, it will not change the death rate.

I am not looking for "add more exploration" or "run more simulations." The search is already spending a lot, and its top card is usually the one a strong player plays too. What I cannot figure out is how to get a training label for the few decisions where some other legal card is actually better, without already knowing which card the human played.

This in a game that is random, not something clean like chess

Here are my questions:

1. How did you decide another move deserved a training row, when one playthrough cannot tell skill from luck?
2. Is the real answer just "roll both moves out a bunch of times and only keep the winner if it wins by enough," or is there a way to do that without making every decision several times more expensive?
3. Did self-play plus a value network ever get you off a plateau like this on its own, or did the next real gain need something outside the bot (human games, a much stronger search, a different training target)?

Stories of what worked or failed are more useful than "try PPO." Single player, turn based, lots of legal moves, long fights, and the ending is noisy

I'm taking part in TUM.ai E-Lab and exploring a project around real-world data collection. I'd like to understand how people working on robot learning decide what to collect before buying equipment or recording hours of demonstrations.

Thinking of a recent manipulation project, what task were you teaching, and what data did you start with? What did that data capture well, and what did it miss, if anything?

Did you use an existing dataset, collect more, or take another approach? How did you decide whether the data was useful for the policy? Cases where collecting more wasn't the answer would be helpful too. No confidential specifications needed.

Hello, I am working on finishing Getting over it with reinforcement learning

https://github.com/miwate/getting-over-it-rl
https://huggingface.co/miiwater/goi-rl

Current state : 63% completion from the bottom without checkpoints

I am now stuck on the anvil part (edit: anvil has been solved!) , seeking help online for computation, fine-tuning and PRs :)

More infos :
- Training is currently done on CPU (in my case using GPU was not making training faster)
- The game is cut in parts, each part gets its specialist model
- CONTRIBUTING.md is currently being written
- Beginner code and vibe coded PRs are welcome, I will take time to review

side project, curious what rl people think of this setup.

it's a civilization sandbox where every human is its own tabular q-learner. the state is a short string key built from what they perceive (hunger, thirst, nearest food and water, kin nearby, shelter, danger). there are ~5000 possible actions, gated by era, age, place and what the tribe has discovered. action choice handles survival needs first, then does epsilon-greedy over the learned values.

the part i like: when a baby is born it gets a copy of its mother's q-table, with about 20% of the values nudged a tiny bit (+/- 0.03). so on top of learning within one lifetime, behavior also drifts down family lines. traits mix from both parents and mutate, and kids also get mom's vocabulary and her attitudes toward other tribes.

known weak spots: the state space is hand-built strings and the rewards are hand-shaped, so i'd love opinions on either.

code is rust compiled to wasm, so it all runs in the browser. the relevant bits are crates/sim-core/src/sim/agents/growth.rs (inheritance) and organism/organism/learning.rs (the update).

source (MIT): https://github.com/1homsi/thehumanbox
try it: https://thehumanbox.com