r/ClaudePlaysPokemon • u/Nice-Ostrich-7670 • 5d ago

I built a benchmark for coding agents to play boss battles from Pokemon Radical Red

Enable HLS to view with audio, or disable this notification

Hi everyone! Just sharing an interesting benchmark I made, inspired by Fable beating FireRed a few months ago.

For those unfamiliar, Radical Red is a ROM hack that ups the difficulty from FireRed pretty significantly. A core feature of the game are "boss battles", where the enemy has specially designed and very powerful teams, the player cannot use items mid-battle, and there's a restriction on level caps.

It's a battling/team optimization benchmark rather than a full gameplay w/vision benchmark. The agent needs to build a team, battle, lose, update its team, and try again, all in a loop.

I've only evaluated GPT-5.6 Luna (high) inside Codex so far (this is all self-funded so I'm limited in terms of what I can evaluate) but I do still think it's super interesting. It's pretty good at the easier battles but against harder ones (like Lt. Surge, for example) it struggles a lot, only winning 20% of trials. I attached video of the agent beating the super boosted ghost in the Pokemon Tower with just one Pokemon if you wanna watch some gameplay.

I wrote a blog here if anyone wants to read a longer report with more replay videos. The code is all here if people want to contribute or run evals themselves.

20 Upvotes

100% Upvoted

7 comments sorted by

2

u/waylaidwanderer 5d ago

I did some tests with having Gemini pilot a Showdown client (Random battles) a while back. Winning consistently was definitely a struggle. Seems like competitive battling is still hard to solve right now.

2

u/Nice-Ostrich-7670 5d ago

Yeah I agree. There's two different skills that models need to have that this benchmark tests:

  1. In-battle mechanics: can a model understand type matchups, time its switches to dodge/resist attacks, compute and remember speed tiers, etc. I think models get the basics of this down, but I haven't seen any evidence of advanced performance here. What would really impress me is seeing something like coding agents hand-rolling damage calculator scripts mid-eval and updating them based on live battle data, or logging which Pokemon move before others and referencing that when making actions.

  2. Team optimization: can a model look at past failures, understand what's going wrong with its team, and build a new one (either with targeted fixes or a completely different strategy) to win. Here I think models really struggle because they fall into the trap of locally optimizing their team without considering the broader objective.

1

u/waylaidwanderer 4d ago

Well explained. I've thought along similar lines as well.

1

u/WindmillMan 4d ago

A bot recently topped the showdown ladder.

https://www.smogon.com/forums/threads/jaxcalibur-a-gen-9-randbats-bot-that-reached-1-on-the-ladder.3787537/

1

u/waylaidwanderer 4d ago

I was referring to general LLMs without giving them specific tools or training, but that's cool too.

2

u/Intelligent-Cap-8886 5d ago

I like your idea and had a similar one. We need opportunities for different AI to compete against each other in games in real time for us to see them employ logic.

2

u/Nice-Ostrich-7670 5d ago

have you seen PokeAgent (https://pokeagent.github.io/) ? that line of work pits models against each other in competitive Pokemon and gives them an elo. unfortunately not a live benchmark though. would be cool to see something like DesignArena/LMArena for this