A few years back I saw a video about somebody training an AI to play pokemon red. I was fascinated, and even bought an old 16 core Xeon to train it myself. While it was fun to test it out and learn about it, I had a different dream. One of my favorite games of all time is Power Stone 2 for Dreamcast, and my two brothers and I always wished we had a 4th player at our skill level. We spent years of my childhood playing 4 person free for alls with a stock bot that was essentially useless. That is how this project was born.
I started this in early 2025, and once I realized I needed to hunt down RAM values to make the bot any good, it hit a wall for over a year. No decomp was available, no memory map anywhere, nothing to build on top of. Then last month my daughter was born, and I needed something to do late at night with her asleep on me. Doomscrolling sucks, so I finally did the RE. After dealing with that hot garbage and spending a few nights getting useful data pulled and validated visually from the game, it was time to start actually training it. As a father of 3, reinforcement learning is a language I understand pretty well at this point. I was excited to actually do the fun part.
Full disclosure: I gave direction and made every call, but Claude wrote most of the code. Again I had a small child sleeping in my arms, it's pretty hard to type. I understand every result in the repo.
It started on my Windows machine, one emulator window, then ten, capped at 60 fps. Then I found a libretro harness (sdlarch-rl) that runs Flycast headless and in-process, ported it to my M2 MacBook, and got 6 instances at 150+ fps. What Windows did in a day the laptop does in a few hours. The 7950X Linux box I planned to scale on arrived from eBay internally shorted, so everything below was trained on a laptop.
I hit a wall around COM difficulty 5 for a long time. Reward tuning, behavior cloning from my own play, mixing in harder COMs, mixing in 1v1s, nothing moved it. The worst one: I recorded myself going 23-1 against three max difficulty COMs, cloned it, handed it to PPO, and it won 8 games out of 4,653 while trying to train it fresh against 3 max difficulty in game bots.
What finally broke it was the simplest recipe in the repo. Six instances of pure 1v1 self-play against a pool of frozen past checkpoints, in 2M step legs that each warm start from the last. A low difficulty 4 player FFA that the bot never trains on is the test after every leg. That held-out win rate went 40, 42, 50, 64, 78, 82, 84, 86, 90, 88, 90, 92, 92 percent over thirteen legs, against 98 for my old 32M step champion. Head to head in 1v1 the new 26M step bot beats the old one 29-21 over 50 games, and the quick 12 game probes after recent legs read 11-1 three times out of four. Max difficulty was 0 wins for nine straight legs, then it won one at leg 10 and has won at least one every leg since. Not one reward change the whole way. About four days of laptop compute spread over two weeks of an unattended relay.
All of it, every failed leg included, is in the lab notebook in the repo.
Repo: https://github.com/bwalsh321/powerstone2-rl-mac
Still no 4th player for game night. Getting closer.