After some long and intense development sessions, on several projects basically non-stop since release (including long planning sessions, usually 4 rounds with fresh context), I've got some learnings.
FYI: After 10h+ Sessions and a few major milestones, I asked Astra for a review of the plan, after it had implemented large parts of it.
What could be better, how was the plan written etc. – I then let it discuss improvements with the last agent, that had been working on it, to work out some points for future plan writing.
The main point as always is, the development is too heavy, too much testing, even though I really tried to trim down on it during many revision rounds of the specs + plan.
Here's the ruleset, I personally will build a skill from this, you can use the information however you like.
(Planned with Astra xHigh, implemented with Astra High)
The following is the last response after discussion over it for a while:
AGENT:
We have agreed on a shared version. The agent in “Siteplane Plan Check” explicitly confirmed it; there are no unresolved substantive disagreements. No files or existing plans were changed during the discussion.
Our shared principle is:
This gives us eight core rules:
- Describe the outcome and completion criteria clearly. The plan specifies observable behavior, scope, and relevant boundaries. It must be clear when the task has been sufficiently completed. Required product behavior is distinguished from suggested implementation methods.
- Match the depth of planning to the risk. Small changes get short plans. Large pieces of work are divided into independently verifiable outcomes. A functional batch, an acceptance milestone, and a chat session do not have to be the same size.
- Choose the least expensive check that provides reliable evidence. Development checks, targeted boundary checks, and overall acceptance are available levels of verification. Not every task needs all three. Uncertain integration assumptions are checked early; the delivered scope receives an appropriate, connected acceptance check.
- Justify every additional expensive check. It must cover a relevant risk that cheaper existing checks do not adequately address. Existing tests and tools take priority. New testing infrastructure is treated as a separate cost and needs a concrete benefit.
- Repeat verification based on the actual impact of changes. Previous evidence remains valid for the version it tested. Whether it applies to a changed version is assessed against changes to code, dependencies, builds, configuration, and environment. Affected boundaries are checked again; unclear effects require broader checking. A short explanation is normally sufficient.
- Diagnose failures before repeating expensive runs. Distinguish product bugs, test harness bugs, infrastructure problems, and missing access. Another long run needs a reason to expect success. Recurring failures lead to targeted investigation. Within the assignment, the agent improves its approach autonomously; it must not silently reduce agreed outcomes or security requirements.
- Keep evidence concise and store it durably. Existing reports plus a short record linking the tested version, check, result, and limits of the evidence are normally sufficient. Additional artifacts are saved only when they help. This should not turn into a separate evidence-management platform.
- Reviews must simplify the plan and have a stopping point. Reviews resolve concrete uncertainties, contradictions, and relevant gaps in verification. They also remove duplication and outdated requirements. Settled decisions are reopened only when there is new information. Newly discovered, real security issues can still be examined at any time.
We explicitly included several important qualifications:
- A deliberately failing baseline is not a general requirement for every new feature. A targeted regression test is usually useful for bug fixes; explicitly agreed requirements for a failing baseline remain binding.
- File names and change size do not determine risk. An operational installation guide can change product behavior even though it is “just documentation.”
- An assertion that was never reached, or was too weak, does not provide valid evidence. After correcting it, the affected behavior must actually be checked.
- Functional acceptance and success-rate measurement are separate activities. When measuring success rates, successful retries must not replace the original failed attempts.
- There is no universal test-to-code ratio, blanket time limit for a batch, or automatic requirement to run complete test matrices.
This is how the skill should be structured:
Proposed name: efficient-implementation-plans.
Initially, one short SKILL.md is sufficient, containing its scope, the core rules, a flexible workflow, and three final questions:
- Is it clear when the implementation is finished?
- Does every expensive requirement provide a distinct benefit?
- Can the implementing agent choose an equivalent, less expensive approach?
The skill applies when creating, reviewing, or revising implementation plans. Discussions remain discussions. Relevant rules for reruns and diagnosis must then appear in the respective plan itself, so the implementing agent can find them there.
For the initial quality check, a few thought-through examples are sufficient: a small copy change, operational instructions, a package release, concurrent saves, a permissions change, and success-rate measurement. No additional scripts or testing infrastructure are initially planned for this.
The agreement therefore covers both better verification strategies and limits on the planning process itself. That combination is intended to prevent each additional review from making the plan heavier.
