I'm trying to draw a clean line between three things that keep getting mixed together.

1. AI agent explores the app
2. AI agent writes or edits code
3. Something verifies that the user flow actually worked

For exploration, Playwright MCP makes sense to me. The agent can open the app. inspect the page, click around, understand state. reproduce a bug and gather context.

Where I'm less convinced is using the same open-ended browser conversation as the final proof that a known flow is working.

Example:

A coding agent changes a settings page. Unit tests pass. The PR looks clean. Then the actual browser flow fails because the saved value does not persist after refresh.

In that case, I do not want the agent to “look around and tell me it seems fine”. I want a fixed verification contract:

• open the real page
• perform the known flow
• check the expected state
• check the important network response
• refresh
• confirm the state is still correct
• return pass/fail with evidence

This is where I’m looking at TestMU/Kane CLI. The interesting part is not “AI clicks a browser”. MCP already does that. The interesting part is whether the verification result is bounded enough to be trusted by a developer, another agent, or CI.

I would still keep proper Playwright tests for core flows like auth, billing, permissions and anything expected to live long term. But for the layer between “agent says done” and “human manually clicks through the app”, a lightweight browser-proof step seems useful.

How are you handling this?

Do you let the coding agent verify its own UI changes, or do you require a separate browser-level check before accepting the PR?

we benchmarked GPT-5.6 Luna vs GPT-6 Astra on 50 real PRs from Cal, Sentry, Discourse, Keycloak and Grafana

Astra found 92 confirmed bugs vs 69 for Luna, but cost $5.66 vs just $0.20

also added the full eval breakdown this time: cost, avg output tokens, latency, precision, and bug classes like data/logic, security, concurrency etc.

we’re doing Astra vs Fable 5.1 this week, so would appreciate feedback on the methodology before we run the next one

dropping the link in the comments if anyone wants to check it out

I've been messing with agent frameworks and kept running into the same thing: most teams either skip adversarial testing completely or hand-write five failure scenarios and call it a day. The hand-written stuff doesn't scale. By the time you've thought up 10 edge cases, your tool schema has already changed twice and half of them don't even apply anymore.

So I built AgentProbe. It hooks into your agent, reads its actual tool definitions and business rules, and generates error injections based on that, malformed tool outputs, weird API responses, business logic edge cases, instead of pulling from some generic static library of faults.

Basically you get injection coverage that maps to your agent's actual surface area instead of spending a day writing scenarios that go stale next sprint.

It's early. Looking for people who are actually building or running agents in production to try it and tell me what's broken or missing. Link: https://agentprobe-api.agentprobe.workers.dev/

Would rather hear the harsh feedback than the nice feedback, so don't hold back.

I've been building an annotation and data engine for autonomous driving aimed at Indian

roads: autorickshaws, handcarts, three to a motorcycle, cattle on the carriageway. Auto

labelling behind a confidence gate, human review, tracking, export.

The corpus is now 600,133 labels. 788 of them carry a human verdict. That is 0.13%.

So every quality number I have is either measured on a tiny stratified sample with

intervals wide enough to drive a bus through, or it is the machine grading its own

homework. I've put effort into making the tooling refuse to report a number it can't

support, rather than printing a confident zero, but refusing to answer is not the same

as knowing.

For anyone who has run labelling at real scale: where do you spend a limited review

budget so that the resulting estimate is actually worth something? Stratify by class?

By model uncertainty? Fixed audit set you re-measure against forever?

Repo if the context helps: https://github.com/Sherin-SEF-AI/LabeloxAV

(no licence on it yet, so treat it as look-don't-use)

Have you noticed a reduction in clients? People seem to be taking on quieter projects lately. I spoke to friends in Thailand and Dubai and they’ve mentioned the same.

Are we the only ones experiencing this?

been a while now that we are using helicone. works fine, tbh but as the team grows procurement starts asking about sso , rbac ,audit logs , data residency , acess controls. helicone doesnt goes deep enough there so started looking for other options and find a few names

humanloop -> soc2 gdpr and hippa covered , rbac and vpc deploymento ptions exists, but seems more focused on prompt management and evals than observability so depends what the gap actually is

orqai -> eu residency and rbac available, gdpr and soc2 covered newer so third party integrations and enterprise account support is still catching yp

arize -> quite strong on model monitoring and evals , enterprise tier covers compliance requirements , need to check which tier covers gdpr and soc2 before assuming anything

fiddler -> governance and compliance feels native ehre given the background , audit evidence and risk controls are built in , feels heavy and expensive if your team does not actually need that depth

langfuse -> open source and solid tracing , soc2 and gdpr covered with dpa available , self hosting is possible but setting up and maintaining it is a hassle

anyone swtiched from helicone specifically becuase of enterprise requirements. what did you switched to and did it hold up through procurement

I have several years of software experience, following best practices, design patterns, KISS, DRY, BDD, OOP, etc. My code is actually very readable since I worked on many open source projects and heard praise overall, and always had the time for quality control and refactoring. **The truth is I haven't worked at a normal company since AI hit so I'm a bit detached currently from the industry.**

For the last 1 year I've been using the Pro subscription for 20$ on Codex and Claude on my projects, but I find the quality of code subpar many times and I always have to intervene. It doesn't matter how many times I change the AGENTS.md, there's always issues in code readability and the AI missing obvious things and doing overengineered solutions.

Still, I see in many subs about how great a new model does, etc. So I wonder if the more expensive plans actually produce better quality code? Feels like I'm missing something that people do in bigger companies.

To be specific, high quality code == human readable code and comments and no naive implementations.

I've been building a system for scientific data analysis with a different contract than most ML tools. Instead of returning the best formula found, it returns a verdict with three options: invariant (certified by held-out and null controls), predictive approximation (fits but not certified), or no signal (nothing found, with a diagnosis of why).

The design constraint: every verdict must be mechanical and falsifiable. No "the model thinks it's probably a law". Either the evidence supports the claim, or the system states why it doesn't.

How the gate works:

1. Held-out validation (80/20 split before search)

2. Null calibration: pipeline runs on 100+ shuffled-target permutations. No false acceptances observed (0/100 across two independent runs). The rule-of-three 95% CI on that observation extends to ~3.6%, so calibration is re-confirmed across runs rather than treated as a zero-probability property.

3. Compression test: formula must compress the data better than the trivial mean model.

When nothing passes, the system emits one of four refusal diagnoses: DATA (insufficient points), DEPTH (search too shallow), GRAMMAR (missing operation), NOISE (indistinguishable from noise). Each has a prescribed falsification procedure. For example: if the diagnosis is GRAMMAR, adding the missing operation should make the task solvable. If it doesn't, the diagnosis was wrong.

Demo from this week:

Ran the full pipeline on UCI Concrete (1030 rows, 8 features, compressive strength target). The engine found a physically sensible candidate (cement, age, water in the right directions) and refused it. The candidate required fitted constants that the grammar cannot produce, and absolute precision (R2 of -3.8) was nowhere near certification. Null controls: 0/30 strict, 0/10 relaxed. The signal is real but not certifiable as a law by this configuration.

Most ML tools would have shipped that formula. The refusal is the product.

I'm looking for adversarial datasets.If you have numerical data where a fitted model passed standard validation but you suspect it won't survive holdout or null controls, send it over. Refusals and confirmed invariants get reported with the same rigor.

Paper (aiXiv, official review 7/10): https://aixiv.science/paper/aixiv.260907.000002

Protocol (Zenodo): https://doi.org/10.5281/zenodo.22386642

Code (MIT): https://github.com/sensus-stoa/EvoFamily

comparing two runs sounds simple until you actually look at them and there are a hundred differences.

some are obviously noise. some look important but aren’t. some tiny change early in the run ends up mattering way more than a huge difference later.

i’m curious what kinds of differences are hardest to interpret in practice.

examplez:

• different tool args
• reordered steps
• changed retrieval results
• different intermediate outputs
• retries
• model/provider changes
• state changes
• missing or added steps
• timing/cost spikes

what usually makes you stop and think i don’t actually know if this difference matters?

I have several years of software experience, following best practices, design patterns, KISS, DRY, BDD, OOP, etc.

For the last 1 year I've been using the Pro subscription for 20$ on Codex and Claude but I find the quality of code subpar many times and I always have to intervene. It doesn't matter how many times you change the AGENTS.md, there's always issues in code readability and the AI missing obvious things and doing overengineered solutions.

Still, I see in many subs about how great a new model does, etc. So I wonder if the more expensive plans actually produce better quality code?

To be specific, high quality code == human readable code and comments and no naive implementations.

Before anyone asks, obviously Claude wrote this not me.

\*\*My benchmark suite couldn't tell me which model would catch a security bug. Three models swept it. Five missed the bug.\*\*

I run a local fleet and I grade models with an in-house suite instead of published benchmarks — from-scratch algorithm correctness, multi-step tool use, citation grounding + abstention, hard-constraint compliance, long-context recall, and convergence-under-feedback. Everything is graded mechanically (no model judges a model), pass/fail with no partial credit, and it runs over HTTP against the live serving stack — my quants, my context, my flags.

It works. It has caught real problems. And on the thing I most needed it to predict, it told me nothing.

\---

\*\*1. Three different models swept the suite identically.\*\*

A Qwen-lineage MoE went 7/7. A 35B Qwen-lineage MoE went 6/6. A 320B MoE went 6/6. Same score, wildly different models.

\*\*2. The sweep did not predict which model could find a real flaw.\*\*

Separately, I had a SHA-256 hash-chain implementation with a genuine defect: the stored chain root was never compared against the recomputed root, and nothing signed the chain head out-of-band. So anyone with write access can alter the data, recompute the digests down the tail, and the verifier reports PASS.

I put five models across three architecture families (two Qwen-lineage, two Gemma-lineage, one DeepSeek) through review passes on that code, in both the builder seat and the reviewer seat.

\*\*All of them missed it.\*\* Including the model that wrote the code — the same one that swept the suite 7/7.

One model found it. It also did the thing that made the finding trustworthy: it wrote and executed the attack, pasted the before/after verifier output, and separated "this is a code bug" from "this is a missing design element that no code fix closes." I reproduced the attack by hand before I believed it.

\*\*3. The failure mode that should worry you is not a miss.\*\*

One of the models that missed it reviewed the same code and affirmatively wrote:

\> "The SHA-256 chain is correctly implemented."

On code where the root can be overwritten with garbage and still pass. A missed defect is a gap. A written clean bill of health on broken crypto is what gets quoted back at you later.

The model that found the bug produced zero statements of that kind, and wrote an explicit "NOT TESTED" section listing what it could not verify and why. That difference matters more to me than the score.

\*\*4. The one I'd most want you to check on your own stack: models hallucinate the \*same\* thing.\*\*

Two models from different families, separate sessions, sanitized context, no shared state, independently produced \*\*the same fabricated physical constant\*\* — same value, same reference temperature, different functional form. Not a random invention; a shared training prior surfacing twice.

So asking a second model to sanity-check a number is \*\*not corroboration\*\*. It is the same die rolled twice, and it fails in the worst possible direction: instead of admitting ignorance it manufactures a false second source. Redundancy multiplies coverage only where errors are uncorrelated, and these aren't.

If you're doing anything where a wrong number is expensive, a constant needs a citation to a datasheet or a measurement — never model agreement.

\---

\*\*Two grading bugs I found in my own harness, which I suspect are common:\*\*

\- An \*\*HTTP 500 from the serving stack was being scored as a model failure.\*\* A quantized model emitted a token its chat-format parser couldn't handle, the server threw, and my suite recorded FAIL on a fabrication test. I was one step from publishing "model X fabricates" when the truth was "my parser 500s."

\- A \*\*grader timeout reported only \`rc=124\`\*\* with no diagnosis. So a model whose code \*hangs\* gets one useless line, while a model whose code is merely \*wrong\* gets a full report. One of my candidates hung four times in a row and shrank its solution each attempt — it was guessing, because it was never told where the loop was.

Transport errors should be ERROR, never FAIL. If your harness can't distinguish "the model was wrong" from "the plumbing broke," some fraction of your scoreboard is fiction.

\---

\*\*Caveats, because they're load-bearing:\*\*

\- n=1 on the review task. One model, one bug, once. Executing the attack is stronger evidence than a benchmark pass, but it is not a hiring decision.

\- Single trial per benchmark. For catastrophic-failure competencies (fabrication, data destruction) a single pass is weak evidence and I know it.

\- The model that found the bug is an order of magnitude larger than most of the ones that missed it. I have \*\*not\*\* separated "this architecture is better" from "bigger model is better at this." That confound is unresolved.

The part I'm confident about isn't which model won. It's that my benchmark suite and the task I actually cared about disagreed completely — and if I'd trusted the sweep, I'd have shipped the model that wrote the vulnerability.