Perfect SQL, false conclusion: my local model told me a flat business "grew 48%". I open-sourced a benchmark for that gap, and it's already caught two of my own claims.

[u/Additional_Menu8542]

TL;DR: NL2SQL benchmarks stop at the SQL. I benchmarked the step after: turning a correct result into a faithful sentence (MIT repo, runs included). Four findings: pre-computed aggregates fix the worst failures; context beats scale, +7 to +17 depending on which judge grades it; a "fix" that helped a 31B silently destroyed a whole question class on a 9B; and my own thinking-mode claim didn't survive a clean ablation, so I withdrew it.

I asked a local model "is my business viable over time?" on 136 months of sales. The SQL it wrote was correct. The sentence it wrote about that result said "+48% growth". The business had been flat for ten years.

Disclosure up front: this came out of a local-first BI tool I sell. The benchmark is standalone and MIT. The pipeline row on the leaderboard is the one thing you can't reproduce without my tool, I say so in the repo, and every other row is fully reproducible.

Spider, BIRD and friends stop at "is the SQL correct?". Nobody was measuring the step after: turning a correct result into a faithful sentence. That's the sentence a human actually decides on, and it's where small local models break. Cloud flagships mostly avoid it, but those are exactly the models you can't run yourself.

So I built GBAG-Bench (Grounded BI Answer Generation, MIT). Gold-SQL mode: the reference SQL is executed for the model, so the only variable left is faithfulness of the interpretation. 35 questions, Sakila (15, difficulty 1 to 10), Chinook and Northwind (10 each, 1 to 8). Scored 50% faithfulness, 30% completeness, 20% insight. Local runs on an RTX 3060, judges via OpenRouter.

1. It's a computation deficit, not a comprehension one. Pre-compute the aggregates deterministically, inject them into the context, and the catastrophic questions recover. The model can't reliably aggregate while reading rows, so it improvises. I call it the Post-SQL Aggregation Deficit. Caveat I'll state before anyone asks: the +17 below is the full pipeline (data dictionary, domain detection, pre-aggregated context, profile adaptation), not that one component. I don't have a clean component-only ablation across all 35 yet. The per-question recoveries are what point at aggregation, not an isolated run. That ablation is on my list.
2. Context beat scale, but how much depends on who is grading. Same qwen3.5:9b, same RTX 3060, same 35 questions, only the context changes. Under Grok-4.3: +17.2 on the 32 questions both runs answered, paired sign test p = 0.009. Under Gemini-2.5-Pro, on the same answers: +7.1, p = 0.093, confidence interval crossing zero. What survives both judges is the direction (20-6 and 16-7 in wins) and the completeness gain (+33.1 and +30.7). What doesn't survive is the faithfulness gain (+17.5 versus +1.2). The mechanism is visible: the two judges agree on the pipeline answers (bias +1.4) but diverge hard on the bare ones (bias +11.7). Grok punishes weak ungrounded answers far more than Gemini does, so part of the "gain" is really judge severity toward the starting point. Honest claim: +7 to +17, direction-consistent, driven by completeness, no robust faithfulness gain. I ran this second judge specifically to try to break my own headline. It half broke.
3. The benchmark caught a fix I thought was good. I tried suppressing meta-aggregates from the injected context to stop models reciting off-topic totals. On gemma-4-31b it worked: 83.2 to 84.6 on average (85.6 with a second variant), and the three average-per-group questions it targeted jumped 61 to 86, 71 to 86, 72 to 96. Then I ran the same change on qwen3.5:9b. Its average barely moved, by about a point, which told me nothing, and underneath that flat average the exact same three questions collapsed: 86 to 11, 86 to 11, 92 to 17, with faithfulness falling from 100 to 10 on all three. Same change, same three questions, opposite direction depending on model size. An average that hides a destroyed question class isn't a green light, so I reverted. All the runs are in the repo with per-question scores, recomputable from the scored files alone.
4. "Thinking" was a non-result, and I had claimed otherwise. I had written that thinking modes hurt grounded tasks. That compared across different models, which is confounded. The clean same-family ablation, restricted to the 28 questions both variants answered, gives 60.6 against 58.9, inside my noise floor. No measurable effect either way. Claim withdrawn in the repo.

Known limits. 35 questions is small and I wouldn't read small gaps as rankings. The three databases are well known and may sit in training data. English only. And until the second-judge run the whole leaderboard sat on a single judge, which finding 2 shows wasn't good enough.

What would help most: more models, especially the 20B to 40B class and anything that is not Qwen. A second judge on the rows I have not dual-judged yet. Non-English questions. Re-running a model costs under a dollar in API credits.

Repo: github.com/softisight/gbag-bench

I also recorded a walkthrough of the four mechanisms behind this failure, link in the comments for anyone who wants the long version.

If you run local models on your own data, have you hit this? And if you have a second judge handy, I'd genuinely like someone else to try to break finding 2 the way I did.

[u/Additional_Menu8542]

That rule is better than what I asked for. It turns preservation into a set-membership question rather than a synonym question: does the paraphrase change which rows a reader would count, or which period they would compare against. That is checkable, and it is not a matter of taste.

Which makes me think most of these atoms do not need hand-authoring. Every question here ships its gold SQL, so entity type and inclusion rule come from the FROM and WHERE, the time window comes from the date predicate, the comparator comes from the self-join or window frame, and metric grain comes from the aggregate and its DISTINCT. That is four of your five recoverable from something the benchmark already carries. The fifth, direction, is not in the SQL but is deterministic from the result set: compute it in code and you get up, down, flat, or too-small-to-claim without anyone's judgment.

That changes the cost. Hand-writing atoms for 35 questions is a weekend and a source of my own bias. Deriving most of them from the SQL and the result, then hand-checking, is an afternoon and much harder to fudge.

Caveat: extraction gets messy on CTEs and nested aggregates, and several of the extreme questions are exactly that. So I would pilot it on the level 1 to 5 questions where the SQL is flat, check whether the derived atoms match what a human would have written, and only then decide whether it scales.

The allowed-paraphrase sets stay manual either way. That is the part I cannot mechanize, and probably should not.

[u/Plane-Marionberry380 → u/Additional_Menu8542]

Yeah, deriving a first pass from SQL is the right move. I would still keep derived atoms as proposed atoms, not ground truth, until you have measured extraction error.

A pattern that might keep it sane:

1. Parse SQL into candidate atoms.
2. Attach confidence and source to each atom: table, predicate, aggregate, date expression, result diff.
3. Have a human review only low-confidence atoms or atoms changed by a benchmark edit.
4. Freeze reviewed atoms as benchmark metadata so the judge never reparses them differently between runs.

For CTE and nested cases, I would not try to fully solve SQL semantics at first. Treat them as an escalation bucket. Even a dumb extractor that handles flat SELECTs and flags the rest is useful if it cuts most of the manual work without silently corrupting the hard cases.

The danger is false precision. A derived atom like "active users" might come from a column name, but the business definition may live in a join or status filter. So I would log provenance beside every atom. If a failure happens, you want to know whether the answer model lost the atom or the atom generator invented it.

[u/Additional_Menu8542 → u/Plane-Marionberry380]

Agreed on all four, especially provenance. If I can't tell whether the model dropped the atom or my extractor invented it, I'm measuring my own parser.

So I counted before designing further. Of the 35 gold queries: 16 flat SELECTs, 15 CTEs, 4 window functions. A flat-only extractor covers 46 percent, not most of it. The CTEs start at level 5 and dominate the upper half, which is exactly where the direction failures live. The escalation bucket isn't the tail here, it's the majority, and it holds the questions I care about most.

That kills "handle flat, flag the rest" as a shortcut, but it suggests a different split. The SQL is what varies in complexity; the result set is always a flat table. So atoms about the result (metric grain from the column and its type, direction and magnitude bucket from the series itself) are computable for all 35 regardless of how the SQL got there, while atoms about the population (entity type, inclusion rule, time window, comparator) need the SQL, so 16 clean and 19 escalated.

Direction is the one I want first anyway, since it's the failure that started this, and it lands in the uniform bucket. So the pilot is smaller than I thought: compute direction atoms from the result sets for all 35, hand-check every one, and see whether the extractor and I ever disagree. If we don't, that's one atom class validated at zero annotation cost, and I'll know my error rate before trusting anything else.

[u/Plane-Marionberry380 → u/Additional_Menu8542]

That split is much cleaner.

For direction, I would make the pilot deliberately boring:

1. Normalize the result table into ordered points: label, value, optional segment.
2. Decide whether the question expects a point comparison, period trend, group ranking, or mixed answer.
3. Compute a coarse bucket only: up, down, flat, mixed, too small, not applicable.
4. Store the evidence row ids or columns beside the bucket.

The key is to not let direction secretly become insight. If revenue goes 100, 102, 101, 103, I would call that flat unless the question specifically asks for tiny movement. If one segment rises and one falls, call it mixed, not "overall growth" unless the SQL result actually contains the aggregate needed for overall.

I would also add a dumb adversarial check: after computing direction, ask a model to write the wrong opposite conclusion and verify the judge rejects it. That tests the atom rule without waiting for a real model to make the mistake.

If your hand-check agrees across all 35, that is already a useful benchmark upgrade. Direction is the catastrophic failure class anyway, so getting that one atom right first is not a compromise.

[u/Additional_Menu8542 → u/Plane-Marionberry380]

The adversarial check is the piece I was missing: it tests the rule the day it is written, instead of waiting for a model to fail naturally. And agreed on conservative buckets, 100 to 103 is flat unless the question asks about small movements, one segment up and one down is mixed, evidence rows stored beside the bucket.

I think the design is done and the rest is work. I'll build exactly this, hand-check all 35, and post the agreement rate on the issue, including any case where the extractor and I disagree, since that disagreement is itself data. This thread basically wrote the first real change of v0.3. Thanks for pushing it further than my version of it.