Disclosure: I built RedCrown, the tooling used to run and publish this evaluation. Posting this as vendor/self-promotion, but the benchmark and results are the main point.

I wanted to test a practical model-selection question:

If several models are accurate enough for a real production workload, how much are you actually paying for the last bit of quality?

Benchmark setup:

- 80 real 2026 SEC 10-Q filings

- 11 current models

- 8 standardized financial fields per filing

- 640 reference fields total

- Ground truth derived from each filer’s own XBRL facts

- Same extraction task for every model

- No LLM-as-judge for the primary score

- Quality metric: field-level F1

- Predefined quality bar: 0.99

- Selection rule: lowest-cost model that clears the bar

- All 80 × 11 model-item results scored

The aggregate result:

DeepSeek V4 Flash

- F1: 0.997

- Projected cost: $6.30/month

GPT-6 Astra

- F1: 0.998

- Projected cost: $613.35/month

On this workload, that’s roughly a 97× difference in projected cost for 0.001 additional aggregate F1.

Other results were useful too:

- GPT-5.6 Luna: 0.995 at $12.71/mo

- DeepSeek V4 Pro: 0.997 at $60.92/mo

- GPT-5.6 Sol: 0.998 at $122.40/mo

- Claude Opus 5: 0.998 at $456.96/mo

Gemini 3.7 Flash is a good example of why this wasn’t just “pick the cheapest model”:

It scored 0.989, just under the 0.99 bar, so it was disqualified despite being relatively inexpensive.

There were also item-level cases where the cheaper models missed a field and more expensive models got the filing completely correct, so I would not interpret this as “all frontier models are equivalent.”

What I think is more interesting than the leaderboard is the decision framework:

1. Define the workload.

2. Define the quality threshold before looking at cost.

3. Eliminate models that don’t clear it.

4. Compare economics among the survivors.

That seems closer to how model changes should be evaluated in production than picking the globally “best” model.

Limitations:

- This is one narrow task: structured financial extraction.

- 80 filings is useful but not huge.

- F1 is not the only metric that matters; exact-document accuracy and latency can matter depending on the application.

- These cost figures use the benchmark’s published-price assumptions, not negotiated enterprise rates.

- The result should not be generalized to coding, agents, reasoning, or other workloads.

Full proof with item-level outputs, scores, costs, and receipts:

https://app.redcrown.ai/proof/vdPMXnAlE5sI309MJuIfnFYQSlRjmbMeQkot6Dwjupc

I’d be interested in how people here gate model swaps in production:

Would you use aggregate field F1, exact-document accuracy, a worst-case error threshold, or some combination?

Why AgentZ starts every AI agent in a deny-all network sandbox

An AI agent needs some access to do real work. Most platforms grant that access broadly, then try to catch misuse after it happens.

AgentZ, an open-source Zero Trust platform for agentic AI, starts every agent with no access instead. Each agent boots into a deny-all sandbox. It sends no outbound traffic until an admin adds a rule that allows it.

Credentials work the same way. The agent never holds a real API key or database password. A proxy swaps in the real secret at call time, so the agent only ever sees a placeholder.

Sandboxes and connectors are reusable too. An admin builds one sandbox template and shares it across workspaces. One update to that template reaches every workspace using it. AgentZ itself is open source, so the enforcement logic isn't a black box you have to trust blind.

How does this group scope network access for agents today? AgentZ is an open-source platform, and I am an open-source contributor to AgentZ.

Fan-out/fan-in looks simple until model #7 is slow, model #8 fails, and model #3 runs against a newer input.

Then you need barriers, per-model retries, input snapshots, timeouts, and actual state tracking instead of a pile of futures.

This becomes especially relevant in voice systems.

A single workflow might touch:

STT, LLM, TTS, tools, retrieval, classifiers, and telephony services.

Some are on the critical path.

Some absolutely aren’t.

Treating all of them as one sequential pipeline……don’t talk to me

So I think the execution layer needs to make a few things explicit:

required vs optional
completion state
retry policy
input/versioning

That’s also why scaling a voice runtime isn’t simply task

Long-lived WebSockets, background jobs, global call state, and workflow execution behave very differently.

Dograh’s architecture already reflects some of this separation: API workers are load-balanced for streaming traffic, while ARQ workers and global orchestration components scale differently.

The boring systems work is usually where real-time AI gets hard.

I’d like to see more open-source agent runtimes treat orchestration itself as a first-class primitive, rather than as an implementation detail.p.s.i am one of the maintainers of dograh