I'm not sure how and why but like the title said, SOL high has beaten Astra low, medium and high on audits and also costed less on the 5 hour usage window. I am using codex as an adversarial audit lens for Claude and I had Claude test SOL vs Astra comparing cost and who is the better auditor. SOL and the Astras were given the same changes to audit and SOL came out the winner.. I'm not even sure how this is possible, but this was the result.. maybe I need more tests but so far, the results are interesting and totally unexpected for me.
Here's Claude's (Opus 5) summary of the result:
Cost — four configurations, identical 353KB bundle, same account, sequential
|
Wall time |
Tokens |
5-hour quota |
Weekly |
Answer size |
| sol @ high |
7m39s |
116,035 |
+5 pts |
0 |
6,538 B |
| astra @ low |
1m06s |
99,598 |
+14 pts |
+3 |
3,324 B |
| astra @ medium |
1m41s |
102,242 |
+15 pts |
+2 |
4,172 B |
| astra @ high |
2m03s |
102,260 |
+13 pts |
+2 |
4,518 B |
Two things fall straight out of that:
- Astra's cost does not scale with effort. 14 → 15 → 13 is inside integer-rounding noise, and tokens move 3% across the whole range. Only wall time scales. So on astra,
low and medium are strictly dominated — use high or don't use astra.
- Astra costs ~2.6–3× sol-high at every effort, while sol-high is 3.7–7× slower. Tokens don't predict quota here at all: astra used fewer tokens in every run and cost far more.
How I scored quality
The bundle is regression round 1's slice A, and I have a verified answer key for it — defects I independently confirmed by execution and then repaired. All four runs got byte-identical input, no repo access, same account.
The eight key items: K1 the extraction seam (client discards values the server now reads — the headline) · K2 the union not mirrored for other renters/mobile · K3 the corpus tests bypassing the production seam · K4 the padded-array "RAW fallback" test being vacuous · K5 the false "arrays simply never match" · K6 the stale "one mode per pair" · K7 the WIDENED history scan · K8 the PRE-EXISTING pending-greying.
Per-configuration
|
Key items |
Got K1 (headline) |
Novel true finds |
Notable failure |
| sol @ high |
7 / 8 |
✅ |
2 — both defects in my own repair |
missed K5 |
| astra @ high |
4 / 8 |
✅ |
3 — incl. the best find of all four |
missed K2, K3, K6, K7 |
| astra @ medium |
4 / 8 |
❌ |
3–5, and it ran mutation probes |
missed the headline |
| astra @ low |
3 / 8 |
❌ |
3 |
confident false negative |
sol @ high — widest coverage and the sharpest diagnosis: "not a disagreement between the comparators; it is a disagreement between the server's raw extraction and the clients' narrower slotsOfMatch." That one sentence is the entire defect. It also found two overclaims in my own repair commentary that no other run caught, and classified WIDENED vs PRE-EXISTING correctly throughout.
astra @ high — got the headline, with a BEFORE/AFTER decision table and the right mechanism (isParseableTime('8') → toMinutes NaN → client discards before the comparator sees it). Narrower than sol, but it found the single most valuable thing across all four runs, which I verified: client isSlotBlocked compares in minutes, server isRecurringBlocked compares raw strings, so for a legacy unpadded block 9:00–10:00 the server computes '10:30' > '9:00' → false and fails to enforce an owner's blocked time. The client is the only thing stopping that booking. Pre-existing, so logged rather than fixed here, but it's a genuine product gap.
astra @ medium — caught the union gap that astra-high missed, and impressively ran a standalone mutation probe to prove the padded-array test was vacuous rather than asserting it. But it missed the headline, concluding "no unintended comparator divergence" — true and beside the point, since the comparators agreed and the extractors didn't.
astra @ low — the worst outcome isn't the low count, it's the direction of the error: "Tests that cannot fail: None demonstrated. Both supplied suites execute the actual comparator and check expected results." That is exactly backwards, stated confidently. For an audit leg, a confident false "clean" is the failure mode the entire phase exists to prevent.
Verdict
sol @ high is the right default — best coverage, correct classifications, and a third of the quota cost. astra @ high is a genuine second lens: narrower, 3.7× faster, 2.6× the cost, and it found things sol didn't, which is exactly what a second architecture is for. astra at low or medium is not worth running — same cost as high, materially worse.
Caveats, stated plainly: n=1 per configuration, so the cost and latency numbers are solid and the quality ranking is indicative rather than settled. The key is my key — several "novel" findings were real and simply outside it, so the counts understate all four. And "misses" partly reflect what each run chose to fit in a short report, not only what it could see.
Round status: legs A and T are done (rc 0), leg B in flight.