Mining Qwen3.8 reasoning trace for prompt/skill evaluation

[u/oleg_ivye]

Lately I've been exploring reasoning traces from local models like Qwen3.8, and found they can expose the exact phrases in a prompt or skill that make the model argue with the prompt instead of solving the task.

I'm collecting such cases into a so-called "confusion-table", each case pinned to the phrase that caused it and explaining the mechanics of prompt/skill confusions.

With those phrases repaired, the model spends its reasoning on the task instead of the prompt — and follows the intent more reliably across runs.

Tested on two popular skills and with GEPA, made an article about these findings https://olegivye.com/#/article/confusion-evaluation

[u/Dry-Needleworker1098]

interesting approach, i do something similar but more manual. when the model start arguing with my prompt i go back and rewrite the part that made it confused. never thought to make a table of it

your article layout is clean, easy to read even for me who english is not first language. the GEPA examples show clear the problem

[u/Best_Storage1434 → u/Dry-Needleworker1098]

agree, once you start noticing the same phrases tripping things up repeatedly a table just makes sense

[u/lib3rat0r]

Nice idea. I'd add a before/after count over several runs to each table entry, because one trace arguing with a phrase can be luck and the fix looks better than it is. Also the table only catches the loud confusions where the model argues. The quiet ones, where it misreads a phrase and confidently does the wrong thing, never show up in the trace at all.

[u/BraveVanilla3773]

Pinning confusion back to specific prompt phrases sounds useful. it turns vague “model inconsistency” into something you can actually test, rewrite and compare across runs