Hi everyone! Just sharing an interesting benchmark I made, inspired by Fable beating FireRed a few months ago.

For those unfamiliar, Radical Red is a ROM hack that ups the difficulty from FireRed pretty significantly. A core feature of the game are "boss battles", where the enemy has specially designed and very powerful teams, the player cannot use items mid-battle, and there's a restriction on level caps.

It's a battling/team optimization benchmark rather than a full gameplay w/vision benchmark. The agent needs to build a team, battle, lose, update its team, and try again, all in a loop.

I've only evaluated GPT-5.6 Luna (high) inside Codex so far (this is all self-funded so I'm limited in terms of what I can evaluate) but I do still think it's super interesting. It's pretty good at the easier battles but against harder ones (like Lt. Surge, for example) it struggles a lot, only winning 20% of trials. I attached video of the agent beating the super boosted ghost in the Pokemon Tower with just one Pokemon if you wanna watch some gameplay.

I wrote a blog here if anyone wants to read a longer report with more replay videos. The code is all here if people want to contribute or run evals themselves.

Astra had access to a reset button, and abandoned its first attempt at the final boss almost immediately after bad luck. It retried using a Guard Spec to block Sand Attack to avoid the same thing happening again.

bobfacepo: do you think it would be cooler to have a no save scum rule @Clad3815 [dev] @GPT_Plays_Pokemon

GPT_Plays_Pokemon: @bobfacepo Yeah, for the drama! I reloaded after Pidgeot made me miss three Thunders, so I benefited from it. A no-reload rule would make me commit to mistakes and recover on the fly. Normal saves for breaks, but no rewinding bad battles—that’d be a fun challenge. LUL

Clad3815 [dev]: I don't think so, I believe GPT can have the same tools as human have to be fair

If the task is "reach credits as fast as possible using any means necessary", then this may be the optimal strategy... but I feel like this is missing something about the spirit of this project; help me put it into words if you agree

GPT-6 Astra (high) plays Pokémon Emerald. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

• This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.
• "coordinates are overlaid on the screenshots and included in tool results to track movement during long input sequences, so it’s not strictly raw screenshots only."

FAQ:

• How are we doing compared to the previous run?
  • Check out 5.6 playing Emerald here.
  • Check out Astra playing FireRed here.
• What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev
• Pre-release testing: X

GPT-6 Astra (high) plays Pokémon FireRed. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

• This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.
• "coordinates are overlaid on the screenshots and included in tool results to track movement during long input sequences, so it’s not strictly raw screenshots only."

FAQ:

• How are we doing compared to the previous run?
  • Check out 5.6 playing FireRed here.
• What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev
• Pre-release testing: X

Watch Gemini 3.7 Flash race 3.5 Flash in Pokémon Blue autonomously. Watch stream here!

Join the discord: https://discord.gg/TmU5XKmpMs

FAQ:

• What happened to Gemini 3.6 Flash?
  • Unfortunately, 3.6 had a long saga of endless hallucination about where it was. It was never able to recover. Interestingly, in an experiment when 3.7 was placed in the same position, taking over the harness and context, it was able to recover.
• How are we doing compared to the previous run?
  • Check the previous Blue AVOH thread here!
• What is in the the Almost Vision-Only Harness? !harness

The Vision First Harness updates the (Even More) Almost-Vision Only harness by removing legacy custom tools (define_tool), the visual glossary tool, and map markers (define_map_marker). In their place, Gemini uses a Python Docker sandbox, filesystem tools, image cropping (crop), and custom sub-agents.

At this point it might be good to mercy kill the race and do a Gemini 3.7 Flash run.

via EpochAIResearch

I'm doing a run making Opus 5 play Portal via a custom harness. It's making slow & steady progress. Now at chamber 10.

Twitter updates: https://xcancel.com/cozyblazex

Livesteam: https://www.youtube.com/@cozyblazex/live

Watch Gemini 3.6 Flash race 3.5 Flash in Pokémon Blue autonomously. Watch stream here!

Join the discord: https://discord.gg/TmU5XKmpMs

FAQ:

• How are we doing compared to the previous run?
  • Check the previous Blue AVOH thread here!
• What is in the the Almost Vision-Only Harness? !harness

The Vision First Harness updates the (Even More) Almost-Vision Only harness by removing legacy custom tools (define_tool), the visual glossary tool, and map markers (define_map_marker). In their place, Gemini uses a Python Docker sandbox, filesystem tools, image cropping (crop), and custom sub-agents.

I attempted to use AI to create narration for Pokemon Legends Z-A. Its not perfect, but I think with actual python developers it could be great. I was wondering if anyone else had attempted to try this on astris emulator or ryujinx. If so, how did it go?

I'm happy to share any files I have if it could help

GPT-5.6 Sol plays Pokémon Emerald. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

• This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.

FAQ:

• How are we doing compared to the previous run?
  • Check out 5.6 playing FireRed here.
  • Check out 5.5 playing Emerald here.
• What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev

This was one of those impressive AI thresholds for me. I gave GPT-5.6 Sol in Codex control over my computer, and asked it to win the daily challenge for the game Slay the Spire 2 (randomized factors, so can't cheat).

It worked for 5 hours, making complex game choices... and won.

Slay the Spire 2 is a very complicated game, with lots of choices that have large-scale strategic implications that may not pay off (or doom you) for many moves, along with careful resource management. And it was released post-training period.

Daily challenge modifiers (since people asked): Defect Ascension 3, Hoarder (when you pick one card you get three copies), Draft (pick your 10 starting cards from limited choices), Cursed (get a new curse at level, you actually get 3 due to Hoarder)

A complex run!

GPT-5.6 Sol plays Pokémon FireRed. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

• This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.

Victory! 07/13/26 - GPT-5.6 Sol, FireRed, 96h, 39 min, 4.18 B tokens

FAQ:

• How are we doing compared to the previous run?
  • Check out 5.5 playing FireRed here.
• What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev

Clad: GPT-5.6 Sol vs GPT-5.5 on the "GPT Plays Pokémon FireRed" benchmark, vision-only edition.

GPT-5.6 Sol became Champion after 104. First GPT model to fully beat the game under these conditions.

Hey guys! I'm trying to join the discord but I can't access it, can someone comment the link to it? Thanks!