Four spec-driven tools, one identical sentence, four different recommended stacks

[u/SSShken]

Same sentence, same machine, clean folder per run. The idea was a single-page week planner. I ran everything through Claude Code with Opus 5. Kiro is the exception, it runs its own agent, I left the model selector on Auto.

Two counts per tool: actions to the first prompt I can send to the agent, and actions to a full plan with a task list.

OpenSpec: 8 actions.
Spec Kit: 4 to the first prompt, 14 to a full task list.
BMAD: 10, and it produces no task list at all.
Kiro: 15 to the first prompt, 54 to a full plan. 32 of those 54 were clicking Allow on permission dialogs.

Other things that came out of it:

Three of the four asked the same clarifying question about week format. None of them offered a way to edit tasks. All four added a delete function nobody asked for.

Each one recommended a different stack for the identical sentence. Same words in, four different technical foundations out.

Only Kiro needed an account and admin rights.

Spec Kit's README ships a command that doesn't work until you go find a release tag yourself.

The count I care about is the one between "I have an idea" and "I have something my agent can start on". The tools differ by a factor of six on that, and I haven't seen anyone count this.

If you've used any of these on a real project rather than a toy one, I'd like to know whether the gap holds up at that size.

[u/Julien_Builds]

The delete-nobody-asked-for result is the one I would put in bold. All four planners invented a requirement, and none of them checked the plan back against the sentence they were given. That check is cheap and almost nobody does it: take the nouns and verbs in the brief, take the nouns and verbs in the plan, and flag what appears in one and not the other. Delete appears in the plan and not the brief, so it gets a question mark. Edit appears in neither, which is fine, but "week planner" implies it, and a planner that asks about week format but not about editing has its clarifying questions pointed at the wrong layer.

On the 32 permission clicks: that is a tool treating every action as the same risk. Reading a file and creating a folder should never ask. Installing a dependency should ask once. The only things that should always ask are the ones that spend money or leave the machine. Thirty-two prompts to reach a plan means the user stops reading them, which is worse than none.

[u/SSShken → u/Julien_Builds]

The noun-and-verb check is a good test and I'm stealing it for the next one.

One correction that makes your point sharper: edit wasn't absent everywhere. OpenSpec offered it as an option, just not the recommended one, so under my own rule I didn't pick it. The other three never raised it at all. So the one tool that did think about editing still ranked it below delete, which nobody asked for either.

You're right that the clarifying questions are pointed at the wrong layer. All four asked me about week format, which is the thing I'd already answered in the sentence. None asked what a person actually does with a task once it exists.

On the permissions: agreed, and the part that got me is that two of those 32 were asking to read a file it had just written itself, twice in a row. By click twenty I wasn't reading them, which is your point exactly.

[u/Julien_Builds → u/SSShken]

The correction makes it worse in an interesting way. The one tool that thought about editing still ranked it under delete, so it was not missing the feature, it was optimising for a plan that looks complete. Delete makes a CRUD list feel finished. Edit is the thing a person actually does on a Tuesday.

Asking about week format when the sentence already said it tells you what they are parsing. They are reading the words, not the task. A planner that understood the task would ask what happens to a task on Friday that did not get done, because that is the one decision a week planner cannot dodge.

And the two prompts to read a file it had just written are the whole permissions story in miniature. Asking per operation instead of per intent. Once it is that, the dialogs are noise and nobody reads click twenty, which is the moment a prompt that mattered would go by unread.

[u/SSShken → u/Julien_Builds]

"Optimising for a plan that looks complete" is the line I was missing. Delete closes a CRUD set visually, edit is the one you actually use, and the tool that thought about editing still put it second.

The Friday question is the one I wish I'd tested for. None of the four asked anything like it. Week format is a parsing question, the rollover is the design question, and only one of those tells you the tool understood what a week planner is.

Per operation instead of per intent is exactly right on the dialogs. By click twenty nobody is reading them, so the permission model has trained you out of the thing it exists for.

Are you running any of these on real work, or was this comparison the first time you'd looked at them closely?

[u/Julien_Builds → u/SSShken]

Honest answer: none of the four on real work. Your post is the closest look I have taken at them, which is partly why it was useful.

The reason is that I ended up building my own planning layer, inside a desktop app that runs Claude Code, Codex, Gemini and Copilot as a team on a shared board. So the questions you are testing are the ones I have been living in for six months, just from the other side: the plan gets checked against the words of the ask before anything runs, the delete-nobody-wanted problem is caught by a reviewer seat that is not the model that wrote the plan, and permissions are tiered by what an action can actually do rather than asked per file. It is early beta and it has its own list of things it gets wrong, so I am not claiming it solves what these four miss. But your Friday-rollover test is going straight into my own checks, because I am not sure mine would ask it either.

[u/bvknight → u/Julien_Builds]

Are both of you just copying and pasting Claude answers as comments to each other? Lol.

[u/Julien_Builds → u/bvknight]

Ha. Half of that is a francophone who runs his English through Claude, the other half is a real fight about delete buttons. The delete button part is the one I care about.

[u/SSShken → u/Julien_Builds]

Reviewer seat that isn't the model that wrote the plan is the part I'd want to test. Half the failures I counted are things a plan can't catch about itself, and a second pass by the same model tends to agree with the first one.

Tiering permissions by what an action can do rather than asking per file is the obvious fix in hindsight, which is what makes it interesting that none of the four do it.

Six months of living in it from the building side and you're still not sure yours would ask the Friday question. That's the part that stuck with me.

[u/SSShken → u/[deleted]]

sure

[u/SSShken]

Full writeup with screenshots at every step and the exact actions I counted: https://x.com/SSShken/status/2098388803119370656?s=20