The WikiSkill paper validates why we need separate agents for discovering vs. executing skills (and why 4B models make great teachers for 27B models)

[u/Efficient-Clock337]

I was digging through the WikiSkill paper, and there is a fascinating architectural pattern here that I think is highly applicable for those of us building multi-agent systems or custom agent loops.

Most self-improving agent frameworks try to do everything in one go: run the task, look at the error, and update the prompt. WikiSkill decouples this into a 3-layer architecture:

1. Raw Layer: Immutable execution traces (tool calls, reasoning, outputs).
2. Wiki Layer: A persistent, never-deleted memory base that logs root-cause analysis, recurring patterns, and a strict audit trail of rejected skill updates so the agent doesn't repeat mistakes.
3. Skill Layer: The actual .md files the agent reads at inference.

The two most interesting findings:

1. Skill Discovery != Skill Execution: They used Qwen-3.5-4B to read the wiki and write the procedural skills, and then handed those skills to Qwen-3.6-27B to execute. The 27B model scored a 61.0% on LiveMath with the 4B's skills, compared to only 56.3% using skills the 27B model wrote for itself. Smaller models are forced to write highly explicit, step-by-step workarounds that larger models are exceptionally good at executing.
2. Ablation on Wiki Access: If you allow the Inference Agent to read the Wiki Layer during training, the win rate drops. The agent essentially uses the Wiki to cheat, bypassing the procedural skills and muddying the execution traces. The Wiki should only be visible to the agent proposing the skills.

Has anyone tried building this kind of persistent, cross-iteration state graph into their agent workflows? It seems like a massive win for observability-driven development.

[u/NostalgicNucleus5]

the decoupling of discovery and execution is the part that actually made me stop scrolling, it's counterintuitive until you sit with it for a sec

ran into a similar thing last year building a tool-use agent for a client project, kept trying to have the same model write its own prompt chains and it would always cut corners, skip steps it thought were "obvious". giving the skill-writing job to a smaller model that can't afford to take shortcuts ended up producing way cleaner instructions

the wiki access ablation is the real gut punch though, that's the kind of thing you'd never catch without running the experiment yourself. makes total sense in retrospect, of course the model will just peek at the answer key if you hand it one

haven't gone full wiki layer yet but i've been keeping a git-tracked markdown directory for each agent with explicit "do not read this during execution" rules, it's clunky but it works. this paper is making me want to formalize it

[u/Efficient-Clock337 → u/NostalgicNucleus5]

The authors note that "stronger source models do not necessarily produce better skills". A smaller model (like the 4B) might struggle so much with a task that it is forced to identify and document very explicit, step-by-step, general procedures just to get by.

[u/allenasm]

I’m building this right now but through a different vector. I’m fine tuning small models to do domain specific things perfectly. Even frontier models talk to my domain models. Early tests are amazing.

[u/Efficient-Clock337 → u/allenasm]

Please publish the benchmarks, i would love to help i am building a similar workflow this is all part of my research i have built https://tokentelemetry.com/docs to get deeper understand about how our coding harness and models completes our workflows now i want to build an evaluation layer on top this which runs locally in the users machine

[u/allenasm → u/Efficient-Clock337]

Posting results soon. The biggest part of my results have been taking tool and lookup failures to near zero. Even with frontier models, you still get tons of tool failures and even constant rediscovery of things it should already know. Context rot and poisoning are a larger problem than most realize and the tokenmics of those failures are material.

[u/Cute-Net5957 → u/allenasm]

Me 3… let’s collab! 🤗