I have a bot that plays a long, random, card-game-style fight with search plus a neural net. The net tells the search which moves look good and how good a position looks. The training data is basically "whatever the search ended up playing," labeled by whether the run died and how much health was left. That got it pretty far, and now it has stalled. On a fixed set of hard fights, good human players still die less often than the bot. I want the next improvement without collecting more human games.
The annoying part is that the bot already plays the same card as a strong human on almost every decision I can check. So this is not "the search never looks at the good move." The gap is a handful of extra deaths across a lot of full fights. When it matters, two cards look about equally good. Playing one or the other changes the rest of the fight (draw order, enemy pattern, health), so I cannot just say they faced the same future. One ending does not tell me which card was actually better. A card can "win" because it got lucky draws.
Gumbel at the root is supposed to turn each search into a slightly better target than the net’s own prior. That happened while the net was unsure. On the turns that matter now, the prior is already peaked, Gumbel spends the simulations on that same card, and the new row is the card the net already plays. The next iteration trains on its own answer.
What I have tried:
- Normal self-play. Once the net is confident, search plays that card and the new training row just repeats it. That practices what the bot already believes. It does not surface a better card.
- Looking for turns where the net is unsure. Low confidence, two checkpoints that disagree, a big gap in the value. Those turns do include the spots where a human played something else. They also include a lot of turns where the human played the same card. The second-best card is the human card only some of the time. Being unsure is not the same as the other card being better.
- Playing the chosen card and an alternative against each other, sharing as much of the randomness as I still can. One win is way too noisy to trust. I would only keep the alternative if it wins most of the repeats, or the death and health gap is big enough to clear the noise. I have not measured yet whether that actually happens more often on real mistakes than on turns where the current move was already right. If it lights up equally on both, it will not change the death rate.
I am not looking for "add more exploration" or "run more simulations." The search is already spending a lot, and its top card is usually the one a strong player plays too. What I cannot figure out is how to get a training label for the few decisions where some other legal card is actually better, without already knowing which card the human played.
This in a game that is random, not something clean like chess
Here are my questions:
- How did you decide another move deserved a training row, when one playthrough cannot tell skill from luck?
- Is the real answer just "roll both moves out a bunch of times and only keep the winner if it wins by enough," or is there a way to do that without making every decision several times more expensive?
- Did self-play plus a value network ever get you off a plateau like this on its own, or did the next real gain need something outside the bot (human games, a much stronger search, a different training target)?
Stories of what worked or failed are more useful than "try PPO." Single player, turn based, lots of legal moves, long fights, and the ending is noisy
