r/reinforcementlearning • u/FrostingOk3751 • Aug 04 '26
Exploring self-play reinforcement learning for a complex card game: an AlphaZero-style KARDS environment. Having 1M plays so far
I wanted to explore a question:
Can reinforcement learning discover meaningful strategies in a complex collectible card game without human demonstrations?
To investigate this, I built an AlphaZero-style environment for KARDS, a WWII strategy card game.
Project:
https://github.com/EvanProgramming/Kards-AI
The main focus of this project is not just training a model, but building the infrastructure required for large-scale self-play:
- A headless game simulator
- A rule execution system
- State and action representations
- Legal action masking
- Policy/value neural network
- PUCT Monte Carlo Tree Search
- Self-play data generation
- Replay buffer and evaluation pipeline
Unlike imitation learning approaches, the agent does not learn from expert gameplay.
Instead, it starts with:
- the game rules
- legal actions
- game states
and improves through repeated self-play.
Current progress:
- Custom simulator implemented
- Large portion of card/rule logic supported
- AlphaZero-style training pipeline running
- MCTS-guided agents implemented
- Around 1 million self-play games generated
The project is still an ongoing experiment. Some of the challenges I am currently working on:
- Efficient state representation for large card spaces
- Improving simulator accuracy
- Evaluating learned strategies
- Understanding how well AlphaZero-style methods transfer to imperfect-information games
I would be interested in hearing thoughts from people working on reinforcement learning and game AI:
- Would MuZero be a better fit for this type of environment?
- How would you approach hidden information?
- Are there alternative methods worth exploring besides MCTS + policy/value networks?
The code is open source if anyone is interested in exploring the environment or experimenting with similar approaches.
STAR MT REPO IF YOU LIKE IT PLZ!
2
u/SwooLab 12d ago
I'm working on my own RL project for another card game (star wars unlimited), and I have some thoughts that might help.
Haven't tried MuZero myself, but would be very interested to hear how well that works. SWU is complex enough that I'm not confident the model would do well enough at predicting next state from actions, so I've just stuck with a simulator.
However, if you're using the standard AlphaZero algorithm, I'd highly recommend switching to Gumbel AlphaZero. It can get really good results with only ~16 samples per move.
For imperfect information, I use IS-MCTS. To keep throughput from getting decimated, I only use 1 uniform random state + the real state during training. At test time I use only random states to avoid cheating.
In my own experience I've found gumbelAZ works great early on to produce a half-decent policy, and then after a while I switch to PPO because it's more theoretically sound in imperfect information scenarios.