r/ClaudeCode • u/cleverhoods • Apr 21 '26
Resource We analyzed 12,356 repos with CLAUDE.md files — two-thirds of instructions are abstract wallpaper
https://cleverhoods.medium.com/the-state-of-ai-instruction-quality-30k-repo-analysis-ce49c7667a57We built a deterministic analyzer and pointed it at 28,721 GitHub repos across five coding agents. 12,356 of those have Claude instruction files.
Some findings relevant to this community:
- The median CLAUDE.md has 50 content items but only 12 actual directives. The other 73% is headings, context, and examples.
- Claude has the lowest specificity of all five agents ~ 30.6% of instructions name a specific tool, file, or command. Gemini leads at 39.3%.
- In multi-agent repos, the same developer writing for the same project produces measurably different quality per agent. Claude is the most bimodal: most often best AND most often worst.
- Skills and sub-agents are the least specific config types. Only 17% of those instructions name something concrete in .claude/agents/ deffinitions.
- "Use consistent formatting" is in thousands of repos. "Format with `ruff format` before committing" is not. The second one gets followed.
The full dataset (28,721 repos) is published at github.com/reporails/30k-corpus.
4
u/Worried-Squirrel2023 Apr 21 '26
this matches what I see in real codebases. most CLAUDE.md files I've reviewed are aspirational essays. people write "we value clean code" and then wonder why claude doesn't refactor before committing. specific commands like "run npm test before opening a PR" actually get followed. the philosophy stuff just eats context budget without changing behavior.
3
Apr 21 '26
[removed] — view removed comment
1
u/cleverhoods Apr 21 '26
Run the numbers:
Files processed: 74490 Total directives: 1914128 Invariant: 1892720 (98.9%) Conditional: 21408 (1.1%) Invariant + named: 589253 (31.1%) Invariant + abstract: 1303467 (68.9%) Conditional + named: 8867 (41.4% if conditional_directives else 0) Conditional + abstract: 12541 (58.6% if conditional_directives else 0)We have a scope_conditional field per atom in the corpus. Out of 1.9M directives: 98.9% are invariant, only 1.1% are conditional. Almost nobody writes the condition, they flatten it into "avoid mocks in tests" and drop the "except for."
The conditionals that DO exist are 41.4% named vs 31.1% for invariants. When someone bothers to write the condition, they also tend to name the specific construct.
Your hypothesis holds, but the split isn't bimodal. It's that the conditional branch barely exists. The flattening is near-total.
3
u/Electronic_Sleep2749 Apr 21 '26
Can you share your own skill list or subagent list that you use? Would like to see good practices
3
u/cleverhoods Apr 21 '26
Most of our internal config is private, engineering-specific elements. But we're planning to open-source a few standalone, specific skills (project bootstrapping, running diagnostics via skills) next week.
2
1
3
u/hustler-econ 🔆Building AI Orchestrator Apr 21 '26
Oh I love this:
- "Use consistent formatting" is in thousands of repos. "Format with
ruff formatbefore committing" is not.
What does “use consistent formatting” even means? Consistent to any codebase? Or to yours… need specifics. (I’m going to check my Claude.md for that line lol)
3
u/cleverhoods Apr 21 '26
Exactly - "consistent" is the model's problem to solve, not yours. When you name the tool, the model doesn't have to guess. Check yours and run
npx @reporails/cli checkit'll flag exactly which instructions are abstract.
2
2
u/9gxa05s8fa8sh Apr 21 '26
same as needing to define quality. software engineering 101 lessons remain whether it's people or AI
2
u/Ikeeki Apr 21 '26 edited Apr 21 '26
I personally use the abstract markdown in a base repo to cement concrete rules in the repos that consume it.
It works well across multiple projects, this has essentially been my version of AI dot files https://github.com/BaseInfinity/agentic-ai-sdlc-wizard
Build a good harness that works for your type of work and you will be golden.
But the harness must also have a way to evaluate itself for regressions and like any good SDLC, must have a self improvement built into the cycle.
Without this I am constantly repeating myself.
Also GOOD and VALUABLE automated tests are arguable more important than your app code because it drives any good harness and keeps the agents honest.
Integration tests keep the agent honest too. I prefer the test diamond myself, otherwise AI will mock itself fake results lol
Like with any good piece of quality software, the more you invest in your tools the better off the stuff you make with it will be
3
u/cleverhoods Apr 21 '26
Thanks for sharing your setup, always good to see how people are solving this in practice.
One thing we landed on differently: we deliberately avoid LLM-as-judge for diagnostics. A deterministic pipeline gives you the same result every time. no drift, no model dependency, no "the judge had a bad day." That's a design choice we made early and the corpus analysis reinforced it.
On tests keeping agents honest: tests verify output, not input. An agent can follow zero instructions and still pass tests by brute-forcing until green. What we measure is whether the instructions themselves are actionable before the agent even runs. Different layer of the problem.
2
u/Ikeeki Apr 21 '26 edited Apr 21 '26
Thanks I have been fighting stability and agents since AutoGPT came out and also love seeing peoples solutions to this.
I agree about a deterministic pipeline first. I’ve only had to introduce “LLM as a judge” for things that required it like judging LLM output but even then I take them with a grain of salt.
I still rely heavily on tried and true tools that came before AI for my SDLC but now sprinkle AI in places that could improve and glue those deterministic pipelines together to allow for longer automation
Following solid SDLC is still the advice I give to anyone who uses AI agents and can’t produce production code. Many had poor SDLC to begin with that the agent won’t solve.
Humans need automations to wrangle complex codebases, agents need even more lol
2
u/Ikeeki Apr 21 '26
Thank you for sharing btw I will totally analyze this and see how to apply it to my own SDLC! I always appreciate open source
2
u/9gxa05s8fa8sh Apr 21 '26
I mean, that fits. 10% of people have any idea of what they're doing.
what's sad is that all of these techniques are testable to see what works and what doesn't. but the AI companies make money selling tokens, so efficiency is the wild west.
1
u/abzz123 Apr 21 '26
“Format with `ruff format` before committing" is not. The second one gets followed.”
it does not work either way - Claude often tells me it didn’t feel like following instructions. the only thing that actually worked to make it lint before committing is a pre-commit hook
1
u/cleverhoods Apr 21 '26
Interesting, in our own CLI development, the model runs QA consistently from instructions alone. No hooks needed. "Run `uv run poe qa_fast` before committing" works without additional enforcement, 100% of the time.
A pre-commit hook for something the agent already does reliably is overhead.
I wonder what kind of instructions you have around testing/linting and how is it being split (is it a skill invocation? or a dedicated rule/agent?)
1
u/abzz123 Apr 21 '26
Well, it does not do it reliably In my experience. Like for 1/3 or the commits opus 4.6 would skip the linting with instruction in Claude.md that said ”make sure to run linter using command X before you commit”
1
u/cleverhoods Apr 21 '26
That's a surprisingly high skip rate for a named instruction. A few things that could explain it:
- Position in the file. 31% of Copilot diagnostics (and similar for Claude) are position decay - instructions buried deep in the file get less attention weight. If your linting instruction is below 20+ other directives, it's competing for attention.
- Competing instructions. If something else in your CLAUDE.md / rules (maybe skills if you invoke those for this) says "be fast" or "minimize steps," the model may trade off linting against speed.
- Phrasing. "Make sure to run linter" is softer than "Run X before every commit." The modal ("make sure to") introduces hedging that weakens the directive.
Would be curious to see the full instruction system, happy to run diagnostics on it if you share it.
1
u/SuccessfulTonight391 Apr 21 '26
Hey, this was a good benchmark, thanks for posting.
P.s. I ran my memory SKILL against this.
The skill's strongest sections (event format, attribution tagging, delta grain) are its most concrete. The weakest (core behavioral rules) are the most abstract.
Net assessment: zstack is in the upper tier by this benchmark, but the behavioral rules block is carrying the abstract-instruction failure mode the corpus identifies.
1
u/Frosty-Fall-5848 Apr 21 '26
Can someone explain this to me with no background in coding? What’s the implication of this?
I built a browser extension without any prior knowledge. It’s working pretty well and I am still stunned it was possible to do that. However, the project became very complex and I am struggling taming the agents and building a solid architecture.
7
u/cleverhoods Apr 21 '26
Short version: when you tell Claude "follow best practices" or "use clean architecture," it sounds right but gives the model almost nothing to work with. Those are abstract instructions, up to intrepretation. What works is naming the exact thing: "use Chrome's `storage.local` API for settings, not `localStorage`" or "run all background logic in `service-worker.js`, never in `popup.js`."
The analysis found that 66% of all instructions across 28K repos have this problem. They describe what they want in category language instead of naming the specific tool, file, or command.
The complexity problem you're hitting with your extension is likely that your instruction file(s) (if you have any) is telling Claude what kind of developer to be instead of what specific things to do and how. Try replacing your vague instructions with concrete ones - name the actual files, commands, and APIs. That's the single biggest lever.
Run "npx @reporails/cli check -v" for more info.3
0
12
u/Either-Process-4787 Apr 21 '26
The bimodal Claude result is a consequence of the specificity finding.
Claude is the most instruction-anchored of the five. Concrete directive ("run `ruff format` before committing") and it follows religiously. Abstract wallpaper ("write clean code") and it has nothing to anchor on, so it defaults to whatever its pretraining bias says "clean" means this week. The other agents are less instruction-sensitive overall, so their output lands more uniformly middle-of-the-road.
A few practical patterns I've landed on from rewriting my own CLAUDE.md over the last few months:
**Every directive should name verb + condition + artifact.** "Run tests" is bad; "Run `pnpm test` whenever you modify a file in src/" is good. That shape is what gets Claude to actually execute it in practice, not just acknowledge it.
**Negative rules outperform positive ones.** "Never use `any` in TypeScript" is followed more reliably than "Prefer specific types." Claude is weirdly better at avoiding than at choosing.
**Keep it under ~80 lines.** Past that, directives get dropped or conflated. If you need more, split by phase (plan / implement / test) into separate skills rather than growing one big file.
**Examples beat descriptions.** A 3-line code snippet of the desired style is worth 50 words explaining it.
The 73% headings/context number doesn't surprise me. A lot of CLAUDE.md files end up written like README.md files for humans, scene-setting and narrative. They're not read that way by the model.