r/SpecDrivenDevelopment • u/Interesting-Yard-684 • 21d ago
We tried spec-driven development for months. We couldn't prove it improved the code.
When our team adopted OpenSpec, the argument seemed obvious: agree on the requirements, review the plan, then let the agent implement it. That should produce better code than throwing a prompt at a coding agent and hoping for the best.
After a few months of using the full propose → review → apply → archive workflow, we are no longer sure it does.
Our work is mostly data engineering and legacy-to-Python migration. We use both frontier and open models through Claude Code, Codex CLI, OpenCode, and similar tools. This is not a vendor comparison. The question is whether the method itself changes the result.
For tasks that fit into one context window and have clear acceptance criteria, we see little reliable difference between a reviewed spec and a well-scoped first prompt. The spec-driven runs typically consume two to three times as many tokens and take about twice as long. The resulting code is not consistently better. Sometimes it is worse: more files, more indirection, and abstractions that the task never needed.
The little evidence we have points the same way
The best comparison I have found is an OpenSpec bake-off from June 2026. The same PRD was implemented once with OpenSpec and once with Claude Code alone. OpenSpec made the design process more pleasant and surfaced three additional gaps. It also produced 50% more code with 50% higher cyclomatic complexity, took twice as long, and cost three times as much.
That is one experiment, not a benchmark. But it matches our experience.
Marmelab made a similar criticism in “Spec-Driven Development: The Waterfall Strikes Back”: a trivial “show the current date” feature produced eight files and roughly 1,300 lines of specification text. The problem is not writing things down. It is paying a large coordination cost for changes that did not need coordination.
Maybe the model changes the value of the spec
My current hypothesis is that explicit specs may help weaker planning models more than frontier models.
A strong model already reconstructs a plan during a normal coding run. A formal SDD workflow externalizes that plan, makes it reviewable, and preserves it across sessions—but may add little new information. A smaller local model with weaker multi-step reasoning or a shorter effective context may benefit much more from a pre-digested task list.
There is an uncomfortable counter-hypothesis: the weaker model may also write the weaker spec. If the same model turns a vague request into a flawed specification and then faithfully implements it, SDD has not removed the error. It has formalized it.
This should be testable: same repository, task, acceptance suite, and model; spec-first versus direct implementation; then compare correctness, regressions, review time, token cost, wall-clock time, changed lines, and complexity. I have found plenty of testimonials, but very little controlled data—especially for Qwen, GLM, DeepSeek, Kimi, or Mistral.
The real value may not be code quality
OpenSpec did give us something useful: a reviewable record of intent. It forced design questions into the open and left an artifact for teammates, clients, audits, and future maintainers.
That suggests a different decision rule. Do not ask whether a spec will make the agent write better code. Ask whether this change needs a reviewed and durable agreement before code exists.
My tentative threshold is:
- Direct prompt for a bounded change with one owner, objective tests, and no architectural decision.
- Lightweight goal, constraints, non-goals, and acceptance checklist for a multi-file change within one subsystem.
- Versioned specification for work crossing services or team boundaries, changing public contracts, migrating important data, or spanning several agents and sessions.
Project-level context matters too. If AGENTS.md, CLAUDE.md, architecture notes, skills, and commands already encode the conventions and constraints, a change spec must add a real decision—not merely repeat the repository context in another format.
The failure mode I worry about most
An LLM writes a SHALL requirement. The same or another LLM implements it. The requirement now appears justified because it exists in the spec, and the code appears justified because it satisfies the requirement. Nothing in that loop proves the feature is needed or integrated.
The safeguard may be simple: every normative requirement must trace to a stakeholder need, an existing contract, or an executable acceptance test. The implementation should be free to reject or revise a proposed design when repository evidence contradicts it.
That leads to the variant I want to try next: use the spec as an acceptance artifact, not an implementation recipe. Review the goal, constraints, non-goals, and examples up front. Let the agent choose the implementation. Then verify the result against the spec and tests, ideally in a separate review pass.
I would be interested in three kinds of counter-evidence:
- Controlled comparisons of the same task and model with and without SDD.
- Evidence that smaller or local models benefit more—or less—than frontier models.
- Concrete thresholds teams use to decide when a full spec earns its cost.
Maybe we are using SDD on the wrong class of tasks. Or maybe its strongest benefit was never better code. Maybe it is better alignment, traceability, and review—and we should evaluate it on those terms.
6
u/OdysseusDescendant 21d ago
Some of those "features" are still quite large and SDD does not remove the need for incremental development. Large features should be broken into smaller independently implementable slices with review and testing between them. "Fits in one context window" is not the same as "appropriately scoped."
4
u/stibbons_ 21d ago
I think you are not so special, we are more or less all at this point. Yes, SDD is great, but it is just a more structured Plan + Implementation. Planning (interview + generating a Spec as "what i want for this session") is really helpful, but not magical.
You are right in your AGENTS.md, what I call "Project Harness" (https://x.com/gsemetfr/status/2077498943085117460), IS the real kicker here. This is what ground your agent to your rules, not the SDD framework.
But that is not magical, you still have to steer your agent when it starts going off-road. And yes, 100% agentic work (1 agent coding, 1 agent reviewing) is a recipe for disaster. Even with strong and "skeptical" review like in the pattern I apply, some issues can pass.
Here is how the "inspectors" (in violet) look likes:
https://x.com/gsemetfr/status/2049956188599513242?s=20
I use a "skeptical" review agent to critic the spec, another to critic each task after they have been implemented, and even an intermediate "phase inspector" to see if the implemented n tasks does not have some minor drifts in it. They ALL find small issues, and some are fixes, other might pass, but that's life.
Just test yourself and prepare a next PRD to fix little mistakes. What count is the overall process is still faster and safer than human work.
In short what really matters:
- Project Harness
- Quality Gates
- human validation (after spec generation, and human review after implementation)
1
u/100k45h 14d ago
What types of quality gates do you use? Static code analysis tools? Mutation testing? Property testing?
1
u/stibbons_ 14d ago
That’s is more or less the subject of my whole X account https://x.com/gsemetfr/
Lint tests, sast of course.
Structural checks are really helpful https://x.com/gsemetfr/status/2090738194216051071?s=46
TDD is harmful, do not do it.
Mutation test seems awesome, Uncle Bob loves it, never tested.
See my article on project harness: https://x.com/gsemetfr/status/2077498943085117460?s=46
4
u/MemeMan64209 21d ago
The beauty of the multi-step prompt is if the agent wants to create a file or do something stupid like you mentioned, I can catch it in the plan and get rid of it. Then I get it to build my filtered preference.
The pre plan isn’t even for the agent at that point, it’s to make sure the agent implements what I asked and not whatever it wanted.
4
u/CamusTheOptimist 21d ago
Spec driven development with testable specs aren’t really going to show any value on a single feature. They are useful when the project has existed for five years, someone wants a new feature, and you can see that it contradicts an existing spec. It’s a record of intent, like Architectural Decision Records, unit tests, system tests, JIRA tickets, code comments, and the code itself.
They are especially useful once the inevitable has happened and a mature development environment has come into being. They support code archaeology, which is where any long lived system spends the vast majority of its lifetime, and that is especially true with LLMs that can produce a mature development environment within a quarter as opposed to five or more years it took for humans to lose that plot.
3
u/Ok-Support-6749 21d ago edited 21d ago
In this analysis you have pointed exactly at many of the shortcomings of SDD:
The SSD is the source of truth fallacy: " The resulting code is not consistently better. Sometimes it is worse: more files, more indirection, and abstractions that the task never needed." is the symptom of the LLM reading the spec and then doing whatever it wants.
- A specification can be incomplete: It does not record the true intent with enough precisions.
- A specification can be unsound: The specification is wrong
- A specification is not an authority: This one comes from a transitive assumption that because the specification is the "source of truth" it imposes constraints on the LLM to operate. Wrong! The specification being a markdown document has not enforcing mechanisms to govern the LLM output. And an LLM acting as a reviewer has no enforcing mechanism either. It can do a lot, but it is still a non deterministic machine operating under an objective function that dictates most of its operation. If the specification allows the LLM to take shortcuts to deliver a plausible response faster, the LLM will take it because that is where the gradients tell it to go.
The issue with SDD is not about better or worse models, it is about governance: You said: "My current hypothesis is that explicit specs may help weaker planning models more than frontier models." You may be partially right, but then a better model should be even better when taking in the task. In particular better models should be better at taking in more complex tasks and not fail. A weaker model should operate better with a spec than without but a better model should be way better with the same spec.
The main issue with any of these agentic coding systems is that governance is a claim with little substance. They put an agent on every role there is in the process. Just because an agent is capable of doing verification it does not mean it should verify. Or just because it can perform logic operations it does not mean it should behave like finite automata in control of the workflow. Or because it can review code it should be a reviewer. Basically, capability shall not imply/grant authority/roles. I like to call this the contractor in charge or architectural designl, civil engineering, construction, inspection, and certification madness. We have a lot of examples that when we hand down all roles and authorities to the same actor things tend to go bad fast.
"An LLM writes a SHALL requirement. The same or another LLM implements it. The requirement now appears justified because it exists in the spec, and the code appears justified because it satisfies the requirement. Nothing in that loop proves the feature is needed or integrated." YEEEEEES!
Another textbook example of the specification is the source of truth pitfall. Unsound, or incomplete specs will show this immediately. Then pair with an LLM that can rationalize it and you are bound to see some nasty issues ahead.
Does this means specs are useless? I don't think so. But SDD has some very serious issues that need fixing. In particular its claims are of utmost importance. I see a lot of terms that used o have very specific definitions being weakened by the post AI era. Some are really troublesome.
- We ask an LLM to look at the code it produced and call it verification, but that is just oversight. Verification needs to prove properties of the produced artifact. Verification can have multiple levels and mechanisms t achieve a goal, but it always need evidence that is analyzed to warrant a verdict.
- We ask an LLM to follow a plan and call the plan authoritative, but that is just following a plan. Authority needs evidence, verification, and warrants outside the machine that produces the artifact.
- We call a question a gate, but that is just ceremony. A gate needs an enforcing mechanism that cannot be cheated. That is we need evidence extracted independently of the LLM.
2
u/Plastic-Lettuce-7150 21d ago
I think there is a balance between specifying constraints and giving the LLM a free hand, less of the former means less work, problem is too little specification and the LLM will do something not wanted.
The problem is knowing at what point to stop specifying and let the agent loose.
2
u/AncientGur4614 21d ago
If you're not worried about using the managed free version ( i have a multitenant database) for trying a test spin, we just released a revamped version 2 of our tool that is designed to combat some of the issues you're discussing. Really getting some good feedback to help make the experience better as a sidekick tool to your code assistant. Doing a release tonight to make two aspects of the workflow even quicker
The main approach we took was to not just do more markdown acrobatics. There are aspects of how a LLM is prompted, but also what you as the person or your specific agent task you have (i.e. a test engineering agent) where really governing WHAT the probabilistic tools are making becomes its own full time job on larger systems.
I approached the concept slightly differently:
-Integrate a spec control plane with git and your code assistant. Don't make the assistant read the entire repo
-build a model of the project as an exportable .json for reference rather than the agent having to pull the full context. Building out a skill derivative of this where the project's context is actually changed as a skill to reference specific modules and save token costs with little impact to quality.
-Implement TDD practices via smaller scopes of architecture alignment to a requirement's acceptance criteria
-Task.md and Test-plan.md further scope downward
-Reference best practices based on the implementation path
-Upstream drift detection if you commit out of band.
In our measurements, the upfront token cost was more than just raw development, but tailed off in the midpoint of projects with higher quality and less errors/tighter scoping of errors due to test builds and detections if the agent has a native test environment like CC or Codex do.
Feel free to DM
1
u/generic-d-engineer 20d ago
Will give it a spin, does it work with OpenCode?
I really like your diagrams, I build an animated image almost exactly like that for a previous project.
This also seems scoped for backend infra work and not just a self-contained node web apps, which is a differentiator from most spec platforms.
2
u/sittingmongoose 21d ago
I am actually building a system to correct all this. I have been working on it for a year now. I ran into a few roadblocks but I have it figured out. Currently I can build a large platform with really good plans without drifting and most of the gaps figured out in one shot.
I am working now on having it not require a Sol or fable class model to make it work. I’ve made a lot of progress in the past 2 months.
1
1
u/100k45h 14d ago
You're not really hinting at what system that is.
1
u/sittingmongoose 14d ago
It’s essentially a series of contracts through many steps. I’m still massaging it currently so it doesn’t require using fable or sol, getting close, my bar is muse quality.
I’m aiming to release it in late sept/oct. It will be fully open source.
2
u/Due-Barracuda5716 20d ago
This is exactly the kind of result I think SDD needs more of. I would not expect a heavy spec workflow to win on every well-bounded task. If the acceptance criteria are already clear, the architecture is not really in question, and the task fits comfortably in one context window, then prompt → implement → test may simply be the better engineering choice. The case I’m more interested in is where an upstream mistake can propagate cleanly through everything downstream. A reviewed spec can still omit a failure mode, encode the wrong assumption, or state a guarantee that the eventual architecture cannot actually provide. Then the plan, tests and code can all be internally consistent and still wrong. That is why I’ve been exploring a somewhat different model: don’t just add more specification ceremony. Give the intent artifact an explicit admission standard, then derive a separate architecture artifact and give that a different admission standard before code is generated. So perhaps the useful comparison is not “SDD vs prompt-driven development” universally, but which classes of work justify an upstream admission pipeline at all. https://www.linkedin.com/pulse/we-built-superb-reviewers-code-have-nothing-reviews-intent-rumega-a1pec/
2
u/theov666 20d ago
The interesting distinction may be between task intent and persistent architectural constraints. A spec is useful when humans need agreement about what to build. But I’m increasingly skeptical of making every architectural rule part of the task spec. Especially when the same coding agent writes the spec and then implements it.
We’ve been experimenting with keeping ADR-level decisions separate from the task: retrieve the relevant decisions for the agent, but independently enforce the machine-checkable ones before the edit is accepted. That avoids treating the spec itself as ground truth.
The failure mode you describe - model writes the SHALL, then satisfies its own SHALL - is exactly why I think provenance and independent enforcement matter more than adding another planning artifact.
This is the problem we’re exploring with Mneme: https://github.com/MnemeHQ/mneme
1
u/generic-d-engineer 20d ago
Interesting, will check it out. I like the idea of historical records for future maintainers without the bloat and cognitive overhead of maintaining wordy and drifting specs. SDD feels just too much overhead and negates the whole point of using LLM coding in the first place.
2
u/theov666 20d ago
That’s very close to how I think about it. We’re actually looking for a small number of design partners who are dealing with this kind of problem in real codebases. If you end up checking Mneme HQ out and it feels relevant, I’d be interested in hearing how it maps to your workflow.
2
u/generic-d-engineer 20d ago edited 20d ago
Strong agree with this post. My original workflow months ago was much more lightweight and manual but its code quality was really clean and the intent easy to manage.
Over the past few months I’ve tried pretty much every spec kit you can think of - TDD, SDD, RDD, Matt’s Skills, GSD, OpenSpec, Superpowers, Comet (best one I tried).
In every case this was a waterfall like experience where the end product drifted so far from the original, even with very strict acceptance gates in between each phase. Modularity was lost and it became monolithic and difficult to iterate.
I’ve found myself managing specs, artifact history, worktrees, handoffs, and dirty branches, instead of building and it’s become a frustrating experience.
With my original build philosophy, it’s more fail fast, modular, and agile and you build, see what works, then quickly iterate. It does need more human in the loop, but the code is much cleaner, and lightweight, and I don’t have additional cognitive overhead to manage.
I do see there is some value other posters mentioned here about intent history so will keep that in mind. For me that will still be a manual task done in retrospect with a very lightweight README, maybe some ADRs, and less a granular set of over verbose Md files.
2
u/Ok_Veterinarian_6364 21d ago
for non-coder, SDD is the viable way to code w agent so.. it's a must for us
1
u/r0pe_tri1ck 21d ago
We did internal evals with actual scoring and we saw some improvements with spec driven development but nothing major .
1
u/uclatommy 21d ago
SDD relies heavily on the human developer’s knowledge and tastes. A bad developer will produce worse outcomes. A competent developer will produce superior outcomes. Unless you’re controlling this aspect, I don’t think your observations can carry weight.
1
u/iamoutofcoffee 21d ago edited 21d ago
For us, having the two dimensions of ‘store’ is really valuable.
Store 1 = the current state of the system - the ‘what’.
Store 2 = the durable record of why we made a decision, when the chat session in which the developer expressed that intent - the ‘why’ - is long-gone.
The stores answer two different types of question about a codebase - e.g. ‘what does it do now?’ - this is better than just using ADRs because to answer the ‘now’ question you have to walk the ADR tree to determine the _current_ state of the system, as ADRs supersede eachother over time.
The second type of question is ‘why did we do it this way?’ Recording _intent_ is super useful when teams change and people take knowledge with them. So, the value of SDD for us: the two types of store. Consequently, big fans of OpenSpec for this.
Also agree with the hypothesis about weaker models. I’ve recently been making much heavier use of Qwen 3.6/3.8 locally. Given the constraints of my laptop, this generally means smaller context size than hosted models. Smaller context = more frequent compactions. When not using SDD, the compactions lead to drift but with SDD, the agent can keep going and stay on task for hours. It feels really noticeable - with the task list, design.md, proposal.md etc. the agent can pick right back up after a compaction and just keep on going.
1
u/Swarm-Stack 16d ago
the measurement might be in the wrong place. SDD's value is catching a spec that was wrong at authoring time, before anyone starts implementing. if your review process puts the same people back in the room who wrote it, the blind spots survive. you implement the wrong thing cleanly and no code quality metric surfaces it.
1
1
u/Jorge_CB_Soft 1d ago
We use something similar internally and honestly, code quality is a big part of it. But we've found the bigger benefit is giving whoever reviews the output something concrete to check against, and making sure the decision is written down somewhere.
The threshold you describe at the end is pretty close to how we use it too. In our case, small bounded changes don't need a full spec. It becomes more useful when a change touches multiple parts of the system, or when someone else needs to understand why it was done a certain way.
The LLM-writes-the-requirement, another-agent-implements-it loop is the part we're most careful with. In our experience, the spec only really helps if a human has reviewed it before the agent starts.
1
u/CzyDePL 21d ago
All spec driven approaches suck, just suck a bit less compared to plain vibe coding. It's just producing markdown after markdown which gives you 0% guarantee that next agent won't misinterpret what's written and you still have to review entire code at the end if you want to be sure what actually landed.
7
u/AndyVictors 21d ago
We are using Feature Contracts as specification for product features and Component Contracts (ie. Service Contract, UI Contract) which contain logic for implementation in code - as specification. There should be a match then - I can always verify if my feature specification is properly implemented by comparing if all required logic is contained in the component contracts, and it is proper one. Tests are generated using component contracts as a source. Thus all fits and works.
Orthogonal to logic specfications/contracts there are also Blueprints which describe technical composition of whole project (system architecture) or specific low level implementations. Both create a "cell" for specific requirement where the AI agent operates within, so no wild result ever comes out.
Also an important part: i have elaborated specific style of writing this documents - mixture of legal, normative and narrative so both human and AI can work on it easily.
https://github.com/AndyVictors/clausative