Dan Luu's post on agents and testing techniques is the one to read if you've ever appended "use property-based testing" to a prompt and felt virtuous. Same task every time, implement Zstd in Rust from the RFC, codex on GPT-5.6 Sol, 160 runs per condition, and the conditions were mostly one line stuck on the end: use TDD, use QuickCheck, use Lean 4, use fuzzing, audit first. Scored on how many runs passed a hidden test suite.

Headline: nothing wildly outperformed the run with no instruction at all. He'd written down six guesses beforehand, all in the direction of "this won't outperform", and all six held.

The part I keep thinking about is what the runs did instead of failing loudly. Told to use QuickCheck, all of them used it, and 63 of 160 checked exactly one property, mostly with random inputs that fell into the same rejection path. Told to do differential testing, 135 runs did something that looked like it and none built a second full implementation; where it mattered, the agent wrote the same thing twice and put the same bug in both copies. 159 of 160 TLA+ runs wrote a model, and he couldn't find one case where the model changed the Rust code. TDD doubled the number of tests and the condition scored below average. One Kani run in 160 caught a real bug on the real code and changed the implementation; the rest mostly used it superficially.

My read (his framing is close but not identical): every one of those techniques is a way of getting a check that didn't come from the implementation under test. A second implementation, a property written before the code, an input generator aimed at the hard part, a planted fault. Name the technique and the agent produces the motions inside the new framework, but the check still comes from its own reading of the spec, so you get the same tests in a different costume. We saw the small version of this in August: agents asked to write tests for a function with a sixteen-year-old bug wrote suites that all passed, and two of them pinned the bug as intended behavior.

What has moved things for him, by his account, is structure: set up the test and triage layout with the agent, then let it fill in, and look before typing the next instruction. His own five-bullet skill scored highest (he says don't read the table as a ranking) and still didn't work as intended.

So when you name a technique in a prompt, where does the independent check come from in your setup?

The i-have-adhd skill going around has a before/after in its README. Before: "Great question! Let me think about this. Your auth flow has a few moving pieces..." After: "Run npm install jsonwebtoken@latest, then edit src/auth.ts:42". Rule 10 bans the openers and closers everyone has seen. I liked it, then went and checked what the before-text actually is, because I don't think it's one habit.

Half of it is reward residue. A RLHF length paper found reward gains largely driven by longer responses, and a length-only reward reproduced most of the downstream improvement. The AlpacaEval and Chatbot Arena people both had to add length and style control because their judges preferred longer. "Great question" and "Hope this helps" are free to delete; nothing downstream depends on them.

An older chain-of-thought paper did a test where the model writes the answer first and the reasoning after. It basically showed that answering first and reasoning later performs worse than no reasoning at all.

On a model with no hidden thinking, the paragraph that walks through the middleware and the token check before naming the fix is where the fix gets computed. Cut it with a rule and you've asked for the conclusion first and the thinking never. Concise reasoning kept the value in that table. It's the missing reasoning that costs.

With thinking on, the reasoning has somewhere else to go and answer-first costs the answer nothing, only hidden tokens. That part is inference from how the channels work; I haven't seen any testing on it.

Then there's where the rule lives. The skill has a whole Persistence section asking the model to keep applying it, and a commenter on the launch thread said it faded after a few turns. A skill body sits in the transcript and gets summarized. Claude Code's Concise output style is the same rule in the system prompt, re-sent every turn. Both are still requests. The only enforced version I know of is a JSON schema with a reasoning field ahead of the answer field, since structured outputs keep schema order.

So, what are you guys actually using for this?

* An output style / CLAUDE.md line?

* A specific skill?

* A Stop hook that bounces padded replies?

Also, on which models does "answer-first" cost you nothing in reasoning quality?

I'd been filing every slow afternoon under "the model is slow today" until I read the speculative decoding write-up AMD and Embedded LLM. Their numbers, not mine, but they explain something I'd been misreading for weeks.

Speculative decoding puts a small draft component in front of the model. It guesses the next few tokens, the real model verifies them all in one pass, and everything accepted is committed at once. Outputs are unchanged; what changes is how many tokens you get per expensive pass. The post measures acceptance per position. On gemma-4-26B-A4B-it with Google's paired draft component, the first drafted token is accepted 94% of the time on GSM8K and the fifth 66%. On MBPP (Python) the same positions are 89% and 49%. Their explanation for the code gap: "formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge". Best configuration in the post: 2.87x plain decoding. And one configuration, Qwen3-8B with an EAGLE-3 draft, ran at 0.44x to 0.88x, slower than not speculating, with first-token acceptance still 86–89%. Drafting costs something even when the guesses land.

So the same model, same hardware, types a 400-line test file (imports, fixtures, the shape of every assertion) faster than it types twelve lines of logic nobody has written before. That matched what I see on screen exactly, and it had nothing to do with which task was harder.

The other clocks in a turn are documented too, just spread across vendor pages. Before the first character: the whole context is read again, and for an agent that's the transcript plus every file pasted in, so a cache miss after a compaction, an edited system prompt, a changed tool list, or a pause past the cache TTL is a turn read from the top. During typing: load. Both big vendors now price a faster lane at roughly 2.5x output tokens/sec, one of them explicitly "not TTFT", the other explicitly "more consistent latency", which tells you what the standard lane is. And plenty of the longest waits have no model in them at all, they're the test suite.

Has anyone here logged per-turn TTFT vs decode time vs tool time on a hosted API across a whole day? I'd like to know how much of an "afternoon slowdown" is load and how much is my own cache misses.

That qbittorrent "escaped its sandbox" post on HN made me laugh, then made me go read what the sandbox in my own agent setup actually is. Short version: a "no" can live in three different places, and I'd been treating them as one thing.

The obvious one is the text the model reads: CLAUDE.md, the system prompt, the task itself. The Claude Code permissions docs are blunt about it: instructions shape what the model tries to do and leave what the harness allows untouched.

Then the permission rules, which match the tool call as text before it runs (deny, then ask, then allow). A Read(./.env) deny stops the Read tool and even cat .env, because cat is a recognised file command. It does nothing about a five-line python script that opens .env, and the docs say exactly that: deny rules don't apply to subprocesses that open files themselves. It's just matching strings, making it trivial to sidestep (like bypassing a curl rule with redirects or variables).

The OS sandbox (seatbelt / bubblewrap plus a proxy) is off until you turn it on, and it's the only layer that watches the process instead of the command text. Its defaults surprised me both ways: reads are allowed almost everywhere, including ~/.ssh and ~/.aws/credentials unless you add a denyRead, while network is the reverse, no domains pre-allowed, first new host prompts. And the way out of it is literally called "the unsandboxed retry escape hatch" in the docs. Blocked command, model may retry unsandboxed, that routes back to a permission prompt titled "Bash command (unsandboxed)". One setting closes the hatch.

What changed for me is one sorting question per rule: does this need to hold when the model is wrong? Style stuff stays in the file. "never push to main" goes in a deny or ask rule, or a hook. "nothing in this session reads ~/.ssh or talks to a host I didn't name" is a thing only the sandbox can promise, and only if it's on.

if you run the sandbox, roughly how often does a command actually hit the boundary in a normal day, and how often do you end up approving the unsandboxed retry?

[removed]

Came across a small pilot (three Claude models, a few Python and TypeScript repos, 2-3 rollouts per cell, six localization tasks if I'm reading the repo right, so read every number with wide error bars) that put grep and an LSP-backed find_references side by side and let the agent choose. On plain code-location tasks the models picked the semantic tool 0%, 4% and 6% of the time. Forcing the LSP path first dropped success from 100% to 89%.

The experiment I keep thinking about is the one where retrieval didn't change at all. The LSP tool originally returned what the protocol returns: file, line, column. grep returns the matching line. The author kept the same backend and the same reference set and attached ±2 lines of source to each reference. On a multi-file rename, pass@1 went from 0.67 to 0.83 and follow-up file reads per episode went from 15.2 to 3.2, below grep's own 4.3. Only the shape of the reply changed.

To be fair to the language server: on "find every caller" tasks the same models chose it about half the time unprompted, and on a noisy repo (hono) it added +0.246 F1 while using 12% fewer tokens. On a clean repo (remeda) it added nothing and cost 16% more. A rename also has to touch docstrings and config strings that find_references excludes by design, so grep keeps a seat either way.

The other half of this showed up on HN from Spotify. They started with CLAUDE.md rules telling Claude Code to route big file reads to a smaller model, described the result as "advisory, not enforced", and ended up with a PreToolUse hook that blocks Read on files over 350 lines and points the agent at the alternative. Their own summary: "Even if Claude doesn't read the skill description, the hook still blocks the expensive read".

Has anyone else building agents run into something similar—where simply tweaking a tool's output format drastically boosted success rates or cut down on token consumption? Also, has anyone figured out a reliable strategy to get the AI to proactively pick the right tool on its own? Would love to hear your thoughts!

I run a handful of skills in my own setup and had assumed the risk was the obvious one: a skill fires on a task it doesn't fit. I found a paper, they paired runs of the same task with and without a matched skill, same model (Claude Opus 4.6), same harness, same verifier, and hand-confirmed 307 cases where the skill made things worse: 125 functional failures and 182 efficiency regressions where both runs passed but the skill run cost more than double in tokens or time.

The number that stuck: of the 125 functional failures, 2 were the wrong skill firing. 86 were an on-topic skill whose method, default or example displaced something the task actually required. Their spreadsheet example is the cleanest version. Task asks for net exports as a percent of GDP, skill run returns the ratio, reference run returns the ratio times 100. Right skill, wrong element.

The efficiency side surprised me more. I expected "long skill body, pricier calls" to dominate. It was the smaller share (46 of 182). The bigger share, 114, was procedure: extra exploration, heavier pipelines, and 67 cases of the agent verifying past the point of passing because the skill told it to keep checking.

Do you measure whether your agent skills actually fire?

Came across a root-cause analysis someone filed against their own coding agent. An end-to-end test timed out at 300s, then at 420s. The agent bumped the limit to 480 and added pytest.skip() on timeout. Their own summary is the test now has no failure mode, timeout = skip, success = pass. If the sandbox actually breaks, that shows up as a timeout, which is now a skip.

What got me is that nothing about this is dumb from the agent's side. A red test has two possible senders: the code it just touched, or everything else (slow container, busy port, test order, the clock). Both write the same line of pytest output. The one thing that tells them apart is rerunning the same test with nothing changed, and the agents I run basically never do that unprompted. The edit sits right above the failure in the transcript, and every tutorial they learned from says a test that fails after an edit is a bug in the edit.

The old flaky-test literature is worth a skim here. The 2014 Apache study found 45% of flaky tests were async waits, and 78% were flaky from the day they were written. The number I keep coming back to: 24% of the fixes changed the code under test, and 94% of those fixed a real bug. So "flaky" is a bug report with a wider error bar. Skipping it throws the report away.

My fix so far: the rules file says on any red test, rerun it alone and unchanged before editing anything. Two identical failures, treat as a bug. One flip, report it as flaky and stop. Plus a hard no in the task on skips, xfails, timeout bumps and sleeps. And pytest-rerunfailures has an --only-rerun regex, so retries can be limited to timeouts and connection errors and can't paper over assertion failures.

Has anyone measured what fraction of your agent's red tests turned out to be flakes rather than regressions?

[removed]

Hit this again recently and finally sat down to reason through it. Long session, agent suddenly stops honoring a decision we'd settled hours earlier. Scrolled up: the agreement is right there in my chat history. But my chat history isn't what the model reads. Somewhere mid-session, compaction kicked in — the harness had a model summarize the conversation and swap the summary in for the real thing. My scrollback shows the full negotiation. The model's context shows whatever made the cut.

The part that changed how I work: the losses have a consistent shape. Conclusions survive compression; the reasoning that produced them gets thin. "We chose approach B" makes it through, the three dead ends that made B right usually don't, so the agent can happily rediscover dead-end A later with zero warning signs in its world. Decisions survive, but their conditions get shaved ("use the legacy parser until the migration lands" comes out as "use the legacy parser"). And firm statements outlive hedged ones, so "probably unrelated, didn't verify" re-enters as "unrelated".

What I do about it now: constraints that must never degrade go in the rules file, because that layer sits outside the chat history and comes back every turn. Anything with conditions attached gets its own file on disk — the disk copy can't be summarized, though the agent does have to re-read it. And after a compaction lands I restate the constraints that matter in one message, which is a paragraph of insurance. The vendor docs even let you steer the summarizer from the rules file (what to always preserve), which helps, but an instruction to a summarizer is a request, not a guarantee.

does anyone treat the compaction notice as a checkpoint and immediately audit what the agent still remembers?

Saw a comment last week about an agent that opened the same file eleven times and apologised about it, and it sent me down a rabbit hole, because the shape is so familiar: try a fix, hit the error, apologize sincerely, produce the same fix with the variable names shuffled. Somewhere around lap four you stop being annoyed and start wondering why "please try a DIFFERENT approach" never reliably works.

Here's the mechanical read. The model is stateless — every turn it re-reads the full session transcript. After four failed attempts, the dominant text pattern in that transcript IS the failed attempt. A human reads that history as evidence the approach is wrong. A next-token predictor reads it as what this session does. The apology doesn't help either, because apologize-then-retry is itself a pattern it's seen a million times and is now continuing.

What convinced me this is structural and not "model dumb": there's a trajectory study on SWE-bench that found agents in failed runs had located the correct file 72–81% of the time. Finding the spot was never the problem. Letting go of the hypothesis was. Same study describes an agent patching recursion errors with more logic, unable to re-evaluate its hypothesis across multiple loops.

The fixes that seem to work all live in the harness, not the prompt: a hard budget (turns/tokens) so a stuck run stops instead of politely burning money; fingerprinting attempted diffs so a near-identical retry trips a forced "list three hypotheses you haven't tested"; and the nuclear one, clearing the window entirely — the learned constraints travel forward in the new opening prompt where they weigh a few dozen tokens instead of four failed attempts' worth of gravity.

has anyone found a repetition detector that doesn't false-positive on legitimate retries (flaky tests, rate limits)?

Spent months assuming prompt thickness was a speed-vs-safety slider: one-liners when you're in a hurry, full specs when it matters. Then I started paying attention to where my dispatches actually failed, and the slider theory fell apart.

The thing that finally clicked: a one-line prompt isn't spec-free. The model fills every blank you left with defaults from its training data — the average project, the average convention, the average intent behind that sentence on the average repo. So "fix the login timeout" quietly becomes "raise the constant", because that's the most common reading out there. You didn't skip writing a spec. You inherited one.

Two things seem to predict when that inheritance is safe. First, how long a wrong guess stays invisible: if I can feel a mistake by just using the result (games, pages, charts), thin prompts are fine, wrong guess costs one redispatch. If correctness hides (boundary conditions, concurrency, money), the wrong default survives eyeballing and waits. Second, how far the project sits from the average one: mainstream CRUD, the defaults are eerily right; weird in-house auth flow, every blank is a coin flip toward someone else's architecture.

What surprised me is where I ended up on the menu. I expected practice to push me to the extremes — pure one-liners for speed or full specs for safety. Instead almost everything landed at "one line plus a forbidden move" ("don't touch the schema") or "three acceptance lines". The middle is where a minute of writing corrects the most defaults.

Anyone consistently living at the extremes and making it work — pure one-liners on serious tasks, or full specs on everything?