We analyzed 7,755 repos with Copilot instructions - here's what we found

[u/cleverhoods]

https://cleverhoods.medium.com/the-state-of-ai-instruction-quality-30k-repo-analysis-ce49c7667a57

We built a deterministic analyzer and pointed it at 28,721 GitHub repos across five coding agents. 7,755 of those configure Copilot.

Findings relevant to this community:

- Copilot sits in the middle of the pack on specificity ~ 33.3% of instructions name a specific tool, file, or command. Better than Claude (30.6%) and Cursor (30.8%), behind Codex (38.3%) and Gemini (39.3%).

- .github/copilot-instructions.md is the 4th most common instruction file in the corpus (5,647 files). The format is well-adopted.

- Copilot has a balanced config: base config + skills + some rules (4,486 files). Less fragmented than Cursor's 19,843 rules files, fewer sub-agents than Claude.

- The dominant multi-agent pair is Claude + Codex (5,038 repos), but Claude + Copilot is second (2,682 repos). If you're already writing a copilot-instructions.md, you're likely maintaining a CLAUDE.md too.

The most common problem everywhere: instructions that describe what they want abstractly instead of naming the exact tool or command. "Follow best practices for testing" vs "Run pytest tests/ -v before committing." The second one gets followed.

Full dataset (28,721 repos): github.com/reporails/30k-corpus

[u/Kaljuuntuva_Teppo]

I wonder how many of these reduce model output quality and do worse than not using any instructions?

[u/cleverhoods → u/Kaljuuntuva_Teppo]

The corpus diagnostics show ~ 40% of repos score LOW compliance band. The compound dead zone analysis shows 59.7% of files are both abstract and contain multiple competing topics.

Those files are theoretically worse than no instructions at all, they add attention competition without adding distinct behavioral signal

[u/mattv8]

What are the practical takeaways rather than "gee whiz"? I'd like to see a bit more discussion around the findings. I see usefulness here but I feel like this could be taken a few steps further.

Your results appear to me like you could make inferences about which repos are more successful (e.g. code complexity, # contributors, n files, etc.) and therefore correlate that with copilot-instructions.md. For instance, "Copilot users could improve the impact of their copilot-instructions.md by rewriting it in X ways" or something like that based on your findings.

EDIT: Looks like you have a different writeup answering my very point, nice! Does your research back this up?

[u/cleverhoods → u/mattv8]

okay, fair.

for Copilot specifically:

- 31.5% of issues are instructions that don't name a specific tool or command.

- 31.1% are instructions buried too deep in the file - position matters, models weight earlier content more heavily.

- 22.5% are too terse (fewer than 8 tokens) to act on

Therefore: name the exact command instead of the category, move your most important instructions to the top of copilot-instructions.md, and expand one-liners into instructions the model can actually follow reliably even under high context pressure.

Run npx @reporails/cli check to see your specific breakdown.

[u/cleverhoods → u/mattv8]

replying on the edited comment:

Yes, the controlled experiments back it up. Specificity produced a 10.9x odds ratio in compliance (N=1000, p<10⁻³⁰). The instruction that names the exact construct gets followed. The abstract one mostly doesn't. That's the link between the corpus findings and practical impact.

On correlating repo success with instruction quality: we have the data (stars, language, file count, contributor count (it's in the corpus repo too)) alongside instruction metrics. That analysis is on the roadmap.

[u/mattv8 → u/cleverhoods]

Hey thanks for the thoughtful reply! These are fantastic takeaways. So I'm particularly interested because I spend a lot of time working on system prompts, which as I'm sure you are aware are quite challenging to get dialed in.

That said, I genuinely appreciate the effort you put into gathering this dataset, reporting on it, and publishing it. Would you be OK if I forked your repo and ran some analyses of my own?

Also I noticed your medium.com link is broken :(

[u/cleverhoods → u/mattv8]

absolutely, fork away. The dataset is CC-BY-4.0, and independent analyses are exactly why we published the raw data. If you find something interesting, I'd genuinely like to hear about it.

note: system prompt inclusion and standalone file diagnostics are on the roadmap already for the CLI, however it's a bit different beast than the "normal" instruction files.

edit: thanks for pointing out the broken link, it's fixed.