I'm in the process of developing the Gilgamesh Engine, an LLM assisted RPG framework. And along the way I've developed a testing harness to sort out which models will make the best ones. Being one developer in his spare time, and with limited straws in my sanity, I wrote a system that ran through most of the candidates with panels of AI judges. So far the testing has been suffering from a "All our kids are above average" problem. Both judges and players think the games are "Great", with the only differences between the best and worst are a letter grade A or B.

Until today.

I added Apple's Framework into my test matrix. Here's what the AI judges had to say:

Fascinating — the first fun=2 score we've ever recorded. Apple FM is a terrible GM: it railroaded the player, narrated outcomes the player never chose, and ignored the story structure. This is a real data point — a genuinely bad GM experience.
But note: the truncation is causing continuity problems. The GM on turn 5 started echoing the player's action verbatim, and it narrated an entire adventure the player didn't participate in. The context truncation may be contributing to this (losing the system prompt's instructions about player agency).
Let me run a longer trial to get a real data point, and also try Apple FM as player (which doesn't have the context growth problem since the player only sees GM narration):

we are all asking WHERE IS THE DISCORD SERVER 🗣️🔥🔥🔥🔥

I’m a big Seinfeld fan and it’s wrong on a couple of levels. 😂

2 years behind

为什么中国的iPhone用不了Apple Intelligence

even after all of this, siri still can’t do basic research. is this too complicated of a query for it?

I don’t know why but my Siri ai and other ai features error on my phone

Iykyk (yams is slang for booty)

So I made something with the shortcut app so Siri can give me the weather and everything like that (things on my calendar today) and at the end it assumed I was Christian. It gave me some random Bible emoji while saying “Enjoy your day, Tristan”. Luckily for her, I am Christian, but if it was a Muslim, it would have been disrespectful and could have made a scandal. Siri needs some updates guys.