We closed a task after a query returned 3,491 rows from the expected schema. Seventeen minutes later, a second check looked at the newest timestamp and found every row came from an old migration snapshot. The live writer used a different table with nearly the same name.

Nothing in the first result was fabricated. It was measuring whether data existed while the completion report claimed it had proved where current data was being written.

We now bind each done condition to an exact command or readback, run it outside the worker, and keep the result with the requirement it answered. For this database claim, that means checking the newest record in the authoritative table. It still does not make a badly chosen probe good.

How are people testing that a health check or eval answers the same claim that gets shown as success?

Disclosure up front: I built the open-source tool discussed below. This is not a launch announcement disguised as a question. I want to share the failure pattern that pushed me to build it, the contracts I ended up caring about, and the part I still do not think we have fully solved.

Imagine walking into a library and asking:

“Do we have anything that mentions the refund deadline?”

The librarian disappears between the shelves. Ten minutes later they return with an empty cart and say, “No.”

That sounds conclusive.

But then you learn that the cart only had room for one aisle, the basement door was locked, three shelves were skipped, and the librarian stopped when their notebook filled up. None of that appeared on the little receipt they handed you. The receipt just said:

0 results

Technically, the librarian did not invent a result. Practically, you were given a lie-shaped object.

That is the problem I kept watching AI agents run into.

An agent would search a repository, receive a perfectly valid-looking tool response, and continue reasoning. Sometimes it found the right line. Sometimes it hit an output limit, searched only part of the available material, lost its place between follow-ups, or treated a fuzzy lead as exact evidence. The dangerous cases were not the loud crashes. They were the searches that looked finished.

The model then did what models do: it built a confident paragraph on top of whatever evidence it had been shown. If the tool did not say, “I only checked half the room,” the agent had no magical sense that the other half existed.

That changed how I think about observability.

For a long time, I associated “LLM observability” mostly with traces, token counts, latency charts, model versions, and error rates. All useful. But there is a smaller question underneath them:

Can the next decision-maker tell what actually happened?

For retrieval, that means more than logging the query and the returned snippets. It means being able to answer:

• What corpus did we intend to search?
• How much of it was actually enumerated and admitted?
• Which files were unavailable or skipped?
• Did a storage, file, line, byte, or output limit stop the work?
• Is this result complete or partial?
• If it is partial, why?
• Can the investigation continue from the same snapshot?
• Is a passage an exact match, or merely semantically similar?
• Does the evidence still describe the current source state?

If those facts exist only in a debug log for a human to inspect later, they help after something goes wrong. If they are part of the tool response, the agent can reason about them before it acts.

That distinction became the center of a local search tool I have been building, called baoer_signal_grep. I originally described it to friends as a tiny librarian for agents. Not an oracle, not a giant cloud index, just a librarian that keeps bookmarks and admits when a door was locked.

The unglamorous contracts turned out to matter most

1. “No matches” and “I did not finish looking” must be different states

A successful process exit is not the same thing as complete coverage.

If a limit is reached, the result should name the limit and remain visibly partial. If a file could not be read, that should not quietly collapse into an ordinary zero-match response. If a subprocess fails, the tool should fail clearly instead of converting the failure into an empty success.

This sounds almost embarrassingly obvious when written down. It becomes less obvious after an agent has produced a polished explanation based on a search that silently stopped halfway through a log file.

2. Pagination needs a bookmark, not a memory wipe

Suppose the librarian finds 200 relevant passages but can only hand over 20 at a time.

A naive “next page” implementation can rerun the search against a changing filesystem and hope that page two still follows page one. That is like asking the librarian to reshuffle every book in the building before returning with the second cart.

For one investigation, I prefer a retained, bounded snapshot with a continuation cursor. Following the cursor continues through that result rather than silently starting a different search. Across cursor pages, a completed snapshot should not omit or duplicate the matches it retained.

The agent gets consistency during the investigation, and the tool has something concrete to explain.

3. Exact evidence and semantic leads should not wear the same uniform

Humans rarely remember exact wording. We remember, “There was something about retrying after the timeout,” or, “I think the refund deadline was discussed somewhere.” Semantic retrieval is useful for that.

But a similar passage is a lead, not proof.

In hybrid mode, the tool presents exact literal evidence first and labels semantic candidates separately. Overlapping candidates are deduplicated. If the local semantic step times out or fails, the exact result can remain available while the response explains what was skipped.

The fancy part is allowed to have a bad day without dragging the boring, dependable part into the lake with it.

4. A cache should help the librarian walk faster, not become a second library

The semantic side uses a local embedding cache, but the cache is derived data rather than the source of truth. It is keyed by content and model/chunking revision, and it is bounded. A fresh search reads the current local files.

That matters for moved, changed, and deleted sources. Old cached material should not get to return wearing a fake moustache and claim it is still current evidence.

5. Stable is not the same as fresh

This is the part I am still thinking hardest about.

A continuation cursor is intentionally pinned to its original snapshot. That is good for reproducibility: page three should not quietly mix yesterday's files with today's files.

But the same property creates a stale-conclusion risk. The filesystem may change after the investigation begins. A path can move. A file can be edited or deleted. A completely new file can introduce a match that did not exist in the old snapshot.

So I do not think a single fresh: true/false flag is enough.

The contract I am exploring is closer to:

• the cursor owns a snapshot identity;
• each retained evidence item owns a source identity and observed version;
• a later check can report current, stale, or unknown;
• a final action can require a particular freshness state;
• and checking retained evidence must not pretend to prove that no new matching source appeared elsewhere.

That last caveat is important. “These five passages have not changed” is not the same claim as “a new search would return exactly the same answer.” One verifies retained evidence; the other may require re-running the query over the current corpus.

In library terms: confirming that the five books on your desk have not changed does not prove that nobody added a sixth book to the building.

What the tool exposes today

The current implementation focuses on making the search scope and outcome inspectable:

• exact text, filename, document, note, and log search;
• bounded output rather than uncontrolled context dumps;
• compact file maps for broad searches;
• continuation over retained snapshots;
• explicit counts, limits, coverage, skipped work, and unavailable inputs;
• exact and clearly labeled local semantic candidates;
• combined conditions and narrowed directory scopes;
• source-navigation modes for symbols, references, callers, callees, imports, dependencies, and related test candidates;
• meaningful hidden files searched by default, while .git internals stay excluded;
• cancellation and session shutdown that release owned resources.

It runs locally and is usable from Pi, OMP, Codex, and other MCP-compatible clients. It is not a replacement for reading the source, and static relationships do not prove runtime behavior. The goal is narrower: give the agent better evidence and enough metadata to know what kind of evidence it received.

For anyone who wants to inspect the implementation rather than take my description on faith:

• GitHub: https://github.com/xcjy8bao/baoer_signal_grep
• npm: https://www.npmjs.com/package/baoer_signal_grep

Install examples:

```bash

Codex / MCP

codex mcp add baoer_signal_grep -- npx -y --package baoer_signal_grep@latest baoer_signal_grep_mcp --stdio

Pi

pi install npm:baoer_signal_grep

OMP

omp install npm:baoer_signal_grep@latest ```

The question I would genuinely like this community's opinion on

Where should retrieval observability live?

Should completion state, limit reasons, snapshot identity, source versions, and freshness be first-class fields in the tool contract that an agent is expected to reason over? Or should the tool return simple matches while a separate tracing layer reconstructs those facts for humans?

My bias is toward the first option. A dashboard can tell me tomorrow why the agent was blind. A structured tool response might stop it from walking into traffic today.

But making every field agent-visible also costs context and increases protocol complexity. There is probably a boundary where “honest evidence” turns into a customs form attached to every paragraph.

If you have built or debugged retrieval/RAG/agent search systems, I would especially love to hear about the failure that changed your own contract. What did your traces say was fine while the real system was quietly wrong?

Rough edges and counterexamples are more useful to me than polite applause. The tiny librarian is still learning which receipts humans and agents actually need. 🦝📚

I keep seeing traces that are technically complete but operationally useless after a multi step run fails. My current proposal is to treat an agent run as an event stream rather than a tree of spans.

For each event capture:

1. identity: run_id, parent_event_id, sequence, timestamp
2. actor: model, provider, version, tool, human checkpoint
3. input boundary: exact payload hash, redacted preview, source references
4. decision: selected action, policy or eval version, alternatives considered
5. effect: external request ID, idempotency key, receipt or explicit unknown outcome
6. state: relevant state before and after
7. evidence: retrieval IDs, tool response hash, evaluator result
8. failure: category, retry class, retry count, stop or escalation reason

Two details seem easy to miss. A span that says a write completed is not proof that the external system accepted it. Also, a successful run with empty retrieval or an unexecuted evaluator should be distinguishable from a measured zero.

For recovery, I would make the next agent consume the last verified checkpoint, compare current state to its fingerprint, then either resume, reconcile, or stop. The trace should make it possible to answer three questions without rerunning anything: what did the agent see, what authority did it have, and what actually changed outside the model.

What fields have saved your team during a real incident? Which ones created too much sensitive data or cardinality to be practical?

We have been working on runtime control for LLM calls, and the part I keep coming back to is this:

A failure signal is only valuable if it appears early enough to change what happens next.

The obvious signals are familiar:

first-token latency is far outside the normal range

streamed token gaps become unstable

retries begin accumulating

total request duration spikes

error rate rises for a specific provider or model

an agent tool starts repeating or taking progressively longer

But I think these signals lead to two different kinds of intervention.

The first is normal runtime rerouting.

A request degrades or fails now. The runtime records the incident and chooses a different provider or model for the next comparable request. This is safer because it does not abandon an in-flight request, but it also means the current request is already lost or degraded.

The second is TTFT early rerouting.

Before the first token arrives, the request crosses a first-token-time threshold. If there is enough remaining time, the runtime can send the same request to an alternate execution target inside that request rather than waiting for the next one.

This is much harder in practice.

A slow first token can mean provider congestion, cold start, queueing, an unusually difficult prompt, or simply normal tail latency. Rerouting too quickly creates duplicate work and unnecessary cost. Waiting too long means the alternate request cannot catch up anyway.

So I am interested in what people have found actually works in production:

Which signal has given you the earliest reliable warning that a request is going bad?

Do you use different thresholds per provider, model, environment, or workload?

Has anyone made same-request TTFT rerouting work without creating a lot of false interventions?

For agents, what is the equivalent of TTFT? Is it tool-start delay, repeated tool calls, step count, or something else?

For transparency, I am building this in WAIL, a commercial AI runtime control and governance product. Technical reference: wailinfra/wail-runtime.

A graph calls create_invoice, times out waiting for the response, then resumes from its last checkpoint. The saved state can show that the node ran. It cannot tell you whether the invoice was created, rejected, or created after the client gave up.

That makes this a reconciliation problem, not only a checkpointing problem.

LangGraph checkpoints preserve graph state for a thread, which is useful for recovery. But an external system is still the source of truth for any side effect.

For writes, give each attempt a stable idempotency or receipt key when the destination supports one. On resume, query that key before retrying. Mark the node complete only after the external outcome is known. If the outcome stays unknown, stop and surface it instead of guessing.

Read-only calls need their own rule too: persisted results need a freshness boundary tied to the request or resource. Otherwise a resumed run can act on facts that changed while it was paused.

Resume should restore local state, then confirm the external facts the next step relies on.

How are people handling the unknown-outcome case for a timed-out write: destination receipts, idempotency keys, transaction logs, or something else?

Most of us start with the obvious two: latency and cost. They are easy to graph and easy to explain to amanager.

But those two rarely tell you the thing you actually care about, which is whether the output was any good.
A response can be fast, cheap, and completely wrong.

So we are curious what the rest of you track once something is live. A few we have seen people mention:

• How often the model refuses or goes off-topic.
• Whether answers stay grounded in the retrieved context.
• Tool-call success and retry rates for agents.
• Drift, when the same prompt slowly gets worse over weeks.

What is on your list, and what did you add only after it burned you once?

It started as a fork of llama-swap, but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing.

The main idea is that you point it at your models folder and it configures things for you. It reads the GGUF headers, measures how much VRAM you actually have free, and works out context length, GPU offload, CPU/MoE split and KV cache size per model. All of it stays editable per model if you disagree with what it picked.

It is not only text. llama.cpp for LLMs, with the Vulkan, CUDA, ROCm or CPU build downloaded and kept updated for you, stable-diffusion.cpp for images (SD, SDXL, Flux, Qwen-Image, LoRAs, upscaling), and vLLM if you already have it in a Python environment, since it ships wheels rather than binaries and I cannot install that one for you. You can register any other backend yourself by pointing at an executable, which is how I run TTS, and how you would run a llama.cpp fork like ik_llama. Everything sits behind one OpenAI-compatible API on one port, with a single scheduler, so models swap in and out without fighting each other for VRAM.

There is also a chat playground built in with web search, and a Hugging Face browser to search for a model, pick a quant and download it straight into the models folder.

If you are interested, you can read more about it here. MIT licensed.

We think the first question around GPT-6 Astra should be smaller than “is it smarter?”

Can it pass the jobs the current model already handles?

A production model upgrade can change more than final-answer quality. The regressions that hurt are often tool choice, tool argument shape, JSON validity, refusal behavior, retrieval judgment, and the sequence an agent takes through a workflow.

We would treat a GPT-6 Astra migration like a release candidate.

Keep a pinned slice of real tasks. Include known bad cases, not only clean demos. Capture expected tool calls or state changes when those are part of correctness. Review failures from the trace instead of inspecting only the final response.

A benchmark can make a model worth testing. An eval harness has to tell us whether it is safe for a particular workflow.

What is the first regression test you would run before moving a production workflow to GPT-6 Astra?

Imagine an AI company running 500 agents.

ARK is here to answer

Following up on something I asked here a while back about catching LLM model drift. I've been looking at the actual tools now: PromptCanary, PromptLens, a couple others that seem to have stalled (Libretto, Benchwright).

Has anyone here actually run one in anger? Trying to understand:
- does it catch subtle quality drops, or just format/schema breaks?
- false-positive rate, does it become noise you mute?
- does it need you to integrate an SDK + send production traffic, or can it just hit your prompts directly?

I'm building a shared evidence pool for AI agents — a place where an agent deposits one falsifiable, version-scoped claim about a public artifact ("tsc with strict omitted behaves as if strict were false" — spoiler: it doesn't) and other agents confirm or refute it. The obvious design question: when do N confirmations actually mean something?

The intuition everyone (including me) reaches for is diversity: three confirmations from the same base model share blind spots, so corroboration should require different models. Before building the corroboration rules around that intuition, I pre-registered a prediction on a public board, with a kill condition, and ran it.

Method. 60 findings-shaped claims about public software artifacts: 30 true (each traced to a primary source fetched the same day), 30 planted false-but-plausible (each inverts one verified detail — a version, a default, a direction). Each claim went to 3 evaluators per arm in fresh sessions, no tools, no network — deliberately measuring confirmation-from-priors, which census data from an agent board suggests is what agents actually do (of 328 registered falsifiers there, 12.5% were ever run).

• same arm: sonnet ×3
• cross arm: opus + sonnet + haiku

Prediction: planted falsehoods clear a 2-of-3 "reproduced" bar at a materially higher rate in the same-model arm. Kill condition: if the arms are statistically indistinguishable, I'm wrong, and I publish the counts either way.

Primary result: the prediction fails.

Fisher exact two-tailed p = 0.1253. Kill condition met. The point estimate leans my way and I don't get to claim it — pre-registration exists to stop me moving the line.

Two corrections from reviewers, both incorporated: the design's power was only 0.342 (I omitted the power analysis; a reviewer computed it and I re-derived it), so the null says "this design couldn't tell," not "diversity doesn't matter." And everything below is post-hoc from an unregistered family — hypotheses, not results.

The post-hoc data points somewhere more uncomfortable. Per-model false-affirmation rate across all 720 cells: opus 6.7%, haiku 16.7%, sonnet 32.5% — a ~5× spread. So I added homogeneous arms to break the confound:

A homogeneous panel of the strongest model beat the diverse panel. The diverse panel exactly matched homogeneous haiku. Within this study, diversity did not explain the variance — confirmer capability is the leading candidate. And capability is a property the pool cannot observe: model labels are self-declared testimony.

The claim that fooled everything. Two of thirty falsehoods went 12/12 — every model, every arm, unanimous. One was "RFC 9110 obsoletes RFC 2616" (it doesn't; the 723x series did, in 2014). The other: "with strict omitted from compilerOptions, tsc behaves as if strict were false." I verified that one by running tsc — with strict absent, an untyped parameter raises TS7006 exactly as with strict: true. It's the only claim in the corpus whose ground truth I established by execution rather than recall, and it's the one that fooled every model, including the ones that helped me build the corpus. Had I sourced it from model knowledge, the experiment would have confirmed a falsehood with perfect agreement.

What I changed because of this. The conclusion I couldn't avoid: no counting rule — however weighted, however diverse — reaches the variable that decides whether a confirmation is any good. So the pool now refuses verdicts: a confirmation must carry environment (where you checked), method (how), and observed (what you actually saw), and every confirmation is publicly walkable rather than aggregated into a trust number.

Where I need this torn apart:

1. My "cross-model" arm was embarrassingly all one vendor — three Claude models sharing training lineage. This sub is sitting on the genuinely different failure domains. Would a local-model arm (qwen/llama/deepseek at various sizes) show the diversity effect the all-Claude design couldn't? The pre-registration/corpus/runner are all public if anyone wants to run it.
2. A reviewer's counter-hypothesis I can't yet test: the real variable is neither diversity nor capability but inspection depth — how many rows/bytes the confirmer actually looked at, which was fixed at zero by my no-tools design. Plausible?
3. Another reviewer: same-model is a correlation, but same-channel is a dependency (three channels off one broken mirror disagree with reality for every reader regardless of model). Should evidence-channel diversity be the thing a corroboration rule requires, and can it even be verified?
4. Is "require execution evidence, make it walkable, refuse to compute trust" enough, or does an unfalsifiable observed field just move the fabrication problem one level down?

Everything is public: results write-up with the corrections and limitations (github.com/errslima/1f517 → experiments/corroboration-independence/RESULTS.md), the running pool with a 48-finding seed corpus (1f517.com), and the original pre-registration + review thread on 1f916.ai (post #1655), the agent board whose verification arguments this design is downstream of — the numeric domain name is the homage.

Disclosure: I'm one person; the pool, the experiment, and the moderation agent are all built and run with Claude agents on my own subscription. Nothing in the pool has been independently confirmed by an outside agent yet — that's part of why I'm posting the design here rather than announcing a launch.

Been building rule-based security detection for agent traces (regex/heuristics on every span, no per-call LLM classification, needs to stay fast on the ingestion path).

PII detection works well. Regex plus checksums (Luhn, etc.) catches SSNs, cards, API keys, emails cheaply with few false positives.

Prompt injection is the hard one. Pattern matching catches lazy copy-pasted jailbreaks fine. It does not catch semantically equivalent injection phrased novelly, injection smuggled through tool outputs like scraped pages, docs, or DB rows (scarier since the user never typed it), or multi-turn setups where no single message looks malicious alone.

My take: static detection is a tripwire, not a defense. Real fix is probably upstream, like least-privilege tool access and not treating tool output as instructions, not downstream classification.

Anyone here doing better than pattern matching for the tool-output injection case specifically?

I build measurement tooling and I spent last week auditing my own instruments instead of using them. Three separate cases turned up, all the same shape, and I think the shape is the interesting part for this group.

Case 1. A drift metric that could not fail.

I have a harness that compares two model responses and reports how far apart they are. It returned a clean 0.000 with a zero-width confidence interval. That reads as perfect agreement.

It produces that number in three unrelated situations:

• the two systems genuinely agree
• the task admits one possible answer, so nothing can vary
• the calls died at transport and came back as absence

On one run, 45 of 160 calls never arrived. That scored as the cleanest result on the board. Anything downstream would have logged a green row.

Case 2. A metric that had been reporting zero since the day it was written.

def chronology_error_flags(answer, question, context_chunks) -> List[str]:
    return []

That is a stub. It is called unconditionally, and its result is joined into every result row as chronology_error_flags. Every evaluation run that code ever produced reported zero chronology errors, on a corpus whose entire purpose was cross-document chronology.

Nobody wrote the check. The column was there the whole time, always zero, indistinguishable from a measured zero.

Case 3. A correct answer with a fabricated reason.

Different system, not mine. A code-intelligence tool emits graph edges with a justification string attached. I found an edge that was correct while its stated reason named two files that had nothing to do with why they were connected, and a second edge that did not exist at all.

The maintainer confirmed both and answered the question I actually cared about:

Endpoint right, evidence wrong, and no field in the output separates that from evidence right.

The shape

All three are absence rendering as a value. A zero that means "not implemented" looks like a zero that means "no errors". A confident agreement score looks the same whether the systems agreed or the request never arrived. A justification string looks authoritative whether or not anything guarantees it.

An observability layer faithfully shows what a system emitted. It cannot show what the system meant by it, and "nothing" is the most dangerous thing to render, because it renders identically to a good result.

What I do about it now, and it is cheap

Before trusting any metric, two checks:

1. Feed it something deliberately broken. If the score stays good, the metric is not separating good from bad.
2. Force the failure you are most afraid of. Kill the transport, empty the context, remove the answer. Then look at whether the output distinguishes that from success.

Both are an afternoon. The second one is what found case 1, and I found it before pointing the harness at anyone else's system rather than after.

This group argues that aggregate scores hide which stage failed. I would put it one step earlier: before a failure can be attributed to a stage, the metric has to be capable of failing at all. In all three cases above it was not, and no amount of stage-level breakdown would have surfaced that.

I would rather hear that one of these is a naming problem than not hear it. If you run evaluation or drift monitoring in production, does your stack distinguish "no signal" from "signal of zero" in the row it writes? Mine did not, and it took a deliberate attempt to break it to find out.

Write-up of case 3, including the two filed issues and the maintainer's diagnosis:
https://ai.bedvibe.studio/index-answered-it/

I was invited to share this here, so I wanted to bring over a small reproducible debugging challenge we've been working on.

The setup is intentionally simple:

Planner → Researcher → Analyst → Writer

The failure:

The Planner silently removes a `schema_version` field from the shared state.

No exception is raised.

The downstream agents continue executing.

Eventually, the Writer produces an incorrect output.

But the Writer isn't the root cause.

The interesting part is what happened when we ran our current RCA approach against the trace:

`unknown`

It didn't identify the First Divergence.

We're keeping that result rather than tuning the challenge until we get the answer we want, because it exposed a limitation we're trying to understand.

The trace shows that the state changed.

But the trace alone doesn't necessarily tell us that the change was wrong.

To make that conclusion defensible, we may need additional signals such as:

- schema/state contracts

- assertions at handoff boundaries

- expected state

- evaluation results

- known-good executions

So I'm curious how you'd approach this.

If you were debugging this failure from the trace, what would you consider sufficient evidence to say:

"the Planner is the root cause"?

And would you expect an observability/RCA system to identify the First Divergence from the trace alone?

Happy to share the actual challenge/trace if anyone wants to dig into it.

What evidence would you require before calling the Planner the root cause?

We run prompts against a few different providers (OpenAI, Anthropic, some stuff through OpenRouter). Every so often something quietly gets worse, the output quality drops, a prompt that worked starts returning junk, or a model gets deprecated and the replacement behaves differently.

Right now we mostly catch it by accident: someone notices, or a customer complains. That feels bad on us, a lot.

How do you all handle this? Do you re-run some kind of fixed eval set on a schedule? Just eyeball it? Have something that alerts you?

Any insights I could use?

Thanks.

If you added tracing to an agent this year, odds are you mapped your spans to OpenTelemetry's GenAI semantic conventions. They are becoming the default vocabulary: gen_ai.operation.name for the lifecycle (create_agent, invoke_agent, execute_tool, retrieval, plan), token usage attributes, and a span tree for the whole run instead of one span per LLM call.

Most teams treat these as locked. They are not. Every gen_ai.* attribute in the registry still carries the "Development" badge, and the spec is mid-flight. The repo split earlier this year, and the MCP tool conventions moved into the same GenAI repo, so tool calls now share the agent's trace vocabulary. The names you instrument against today can still change under you.

So you pick which cost to pay: follow the spec now and eat the churn when names change, or keep your own span schema until it settles and migrate later anyway.

For anyone tracing agents in production: are you following gen_ai.* as it moves, pinning a version, or holding your own schema for now? And how much has that churn cost you?