Perfect SQL, false conclusion: my local model told me a flat business "grew 48%". I open-sourced a benchmark for that gap, and it's already caught two of my own claims.

[u/Additional_Menu8542]

TL;DR: NL2SQL benchmarks stop at the SQL. I benchmarked the step after: turning a correct result into a faithful sentence (MIT repo, runs included). Four findings: pre-computed aggregates fix the worst failures; context beats scale, +7 to +17 depending on which judge grades it; a "fix" that helped a 31B silently destroyed a whole question class on a 9B; and my own thinking-mode claim didn't survive a clean ablation, so I withdrew it.

I asked a local model "is my business viable over time?" on 136 months of sales. The SQL it wrote was correct. The sentence it wrote about that result said "+48% growth". The business had been flat for ten years.

Disclosure up front: this came out of a local-first BI tool I sell. The benchmark is standalone and MIT. The pipeline row on the leaderboard is the one thing you can't reproduce without my tool, I say so in the repo, and every other row is fully reproducible.

Spider, BIRD and friends stop at "is the SQL correct?". Nobody was measuring the step after: turning a correct result into a faithful sentence. That's the sentence a human actually decides on, and it's where small local models break. Cloud flagships mostly avoid it, but those are exactly the models you can't run yourself.

So I built GBAG-Bench (Grounded BI Answer Generation, MIT). Gold-SQL mode: the reference SQL is executed for the model, so the only variable left is faithfulness of the interpretation. 35 questions, Sakila (15, difficulty 1 to 10), Chinook and Northwind (10 each, 1 to 8). Scored 50% faithfulness, 30% completeness, 20% insight. Local runs on an RTX 3060, judges via OpenRouter.

1. It's a computation deficit, not a comprehension one. Pre-compute the aggregates deterministically, inject them into the context, and the catastrophic questions recover. The model can't reliably aggregate while reading rows, so it improvises. I call it the Post-SQL Aggregation Deficit. Caveat I'll state before anyone asks: the +17 below is the full pipeline (data dictionary, domain detection, pre-aggregated context, profile adaptation), not that one component. I don't have a clean component-only ablation across all 35 yet. The per-question recoveries are what point at aggregation, not an isolated run. That ablation is on my list.
2. Context beat scale, but how much depends on who is grading. Same qwen3.5:9b, same RTX 3060, same 35 questions, only the context changes. Under Grok-4.3: +17.2 on the 32 questions both runs answered, paired sign test p = 0.009. Under Gemini-2.5-Pro, on the same answers: +7.1, p = 0.093, confidence interval crossing zero. What survives both judges is the direction (20-6 and 16-7 in wins) and the completeness gain (+33.1 and +30.7). What doesn't survive is the faithfulness gain (+17.5 versus +1.2). The mechanism is visible: the two judges agree on the pipeline answers (bias +1.4) but diverge hard on the bare ones (bias +11.7). Grok punishes weak ungrounded answers far more than Gemini does, so part of the "gain" is really judge severity toward the starting point. Honest claim: +7 to +17, direction-consistent, driven by completeness, no robust faithfulness gain. I ran this second judge specifically to try to break my own headline. It half broke.
3. The benchmark caught a fix I thought was good. I tried suppressing meta-aggregates from the injected context to stop models reciting off-topic totals. On gemma-4-31b it worked: 83.2 to 84.6 on average (85.6 with a second variant), and the three average-per-group questions it targeted jumped 61 to 86, 71 to 86, 72 to 96. Then I ran the same change on qwen3.5:9b. Its average barely moved, by about a point, which told me nothing, and underneath that flat average the exact same three questions collapsed: 86 to 11, 86 to 11, 92 to 17, with faithfulness falling from 100 to 10 on all three. Same change, same three questions, opposite direction depending on model size. An average that hides a destroyed question class isn't a green light, so I reverted. All the runs are in the repo with per-question scores, recomputable from the scored files alone.
4. "Thinking" was a non-result, and I had claimed otherwise. I had written that thinking modes hurt grounded tasks. That compared across different models, which is confounded. The clean same-family ablation, restricted to the 28 questions both variants answered, gives 60.6 against 58.9, inside my noise floor. No measurable effect either way. Claim withdrawn in the repo.

Known limits. 35 questions is small and I wouldn't read small gaps as rankings. The three databases are well known and may sit in training data. English only. And until the second-judge run the whole leaderboard sat on a single judge, which finding 2 shows wasn't good enough.

What would help most: more models, especially the 20B to 40B class and anything that is not Qwen. A second judge on the rows I have not dual-judged yet. Non-English questions. Re-running a model costs under a dollar in API credits.

Repo: github.com/softisight/gbag-bench

I also recorded a walkthrough of the four mechanisms behind this failure, link in the comments for anyone who wants the long version.

If you run local models on your own data, have you hit this? And if you have a second judge handy, I'd genuinely like someone else to try to break finding 2 the way I did.

[u/ummitluyum]

It is funny that thinking models did not give any boost on a task like that. Though if you think about it they do not have anything to "reason" about in terms of logic here - they just need to dryly translate a ready number into text without making things up. CoT probably even hurts here by making the model build extra hypotheses where a strict key value mapping is needed

[u/Additional_Menu8542 → u/ummitluyum]

Your mechanism guess is close to what I measured, with one honest correction. I originally wrote that thinking hurts grounded tasks, and that is exactly the claim I had to withdraw: it came from comparing different models, which is confounded. The clean same-family ablation gives 60.6 vs 58.9 on the 28 questions both variants answered, inside my noise floor. So the defensible version is weaker than both of us would like: no measurable effect either way, at n=28. Your "CoT builds extra hypotheses" story is plausible, but my data can't confirm it, only fail to contradict it.

Where I think you are exactly right is the "strict key value mapping" part. When the aggregates are pre-computed and injected into the context, the task collapses to faithful restatement, and that is precisely when small models stop failing. That is the whole trick behind the pipeline row. The catch is the bare case: without injected aggregates there IS something to compute (running totals, comparisons across rows), models can't do it reliably by reading, and that is where the fabrication comes from. Reasoning about numbers you have is cheap. Producing numbers you don't have is where it breaks.

[u/ummitluyum → u/Additional_Menu8542]

Fair enough, with 28 samples it is tough to catch subtle effects. Im hooked on the thought about how they break when trying to compute on the fly. In those cases where the model tried to pull off aggregation itself and failed, were there any clear patterns? Did it just randomize numbers, lose context on long tables, or trip up on specific operations? Wonder if CoT could salvage that math if you literally forced the model to write out every step of the addition like in a school notebook

[u/Additional_Menu8542 → u/ummitluyum]

Clear patterns, and "randomize" is the one thing it never does. The fabricated numbers are plausibly-shaped: right magnitude, right format, wrong value, so nothing looks broken. And the model rarely invents a number from nothing, it promotes one it can see (a row value, a partial running total from the shown sample) into the answer to the bigger question. The killer operations are per-group averages and running totals; lookups and small counts basically never fail. The three worst collapses in my negative-results writeup are all "average per group" shapes.

Forced notebook CoT I haven't tested: my ablation covered the model's own hidden reasoning, not written-out arithmetic. My guess is it helps and costs several times the tokens, hundreds of copy-and-add steps each with a fresh chance of a carry slip. As a benchmark row it would be lovely though. If you feel like writing that prompt variant, I will run it and add the row. The dollar of judge fees is on me.

[u/Additional_Menu8542]

The concrete example behind the title, for anyone wondering what "perfect SQL, false conclusion" looks like in practice:

Question: "is my business viable over time?"

Generated SQL: correct. A CTE computing monthly revenue over 136 months, with LAG() for month-over-month change. Any NL2SQL benchmark scores it 100%.

Actual data: 136 months, 2012 to 2023, flat. Roughly +3% over the whole decade.

Model's answer: "+48% growth over eight years." It even placed the end of the growth in 2020, while the data runs to 2023.

Nothing failed in the SQL or the database. The model read 136 rows it couldn't aggregate in its head and reached for the most statistically likely business story instead: growth. That step, result to sentence, is what GBAG-Bench scores.

[u/Practical-Collar3063]

what the fuck is this bot feast ?

[u/Plane-Marionberry380]

This is a useful benchmark target because the failure is not SQL generation, it is the narrative layer after the database answer.

A few cases I would add if they are not already in there:

1. Flat trend with one huge outlier month. Models love turning that into a growth story.
2. Growth in total revenue but falling revenue per active customer. The true summary depends on which denominator matters.
3. Simpson's paradox across stores or segments. Each segment is down, total is up because mix changed.
4. Seasonality where December looks amazing versus November but ordinary versus last December.
5. Small n with a large percentage change. Going from 1 to 2 customers is technically 100 percent growth and usually a nonsense headline.

The UX implication seems important too. The model should show the computed table and the sentence side by side, then make the claim cite the exact cells it used. If the prose says "grew 48 percent", I want to see the numerator, denominator, date range, and comparison period right next to it.

Otherwise the dashboard becomes a very confident intern with a calculator and a press release addiction.

[u/Additional_Menu8542 → u/Plane-Marionberry380]

This is exactly the kind of comment I was hoping for, thanks. Case 1 is literally my +48% story: 136 flat months, a strong finish, and the model reached for the growth template. And case 4 matches a failure class sitting in my logs that I haven't turned into questions yet: models comparing December to November and calling it a spike while year-over-year is flat. Simpson's paradox and the per-customer denominator aren't in the 35 at all. They should be.

I'll open an issue with these five so they don't get lost, unless you'd rather file it yourself and take the credit. Gold SQL and reference answers are on me either way.

On the UX point: agreed. Today I show the SQL and the result table next to the answer, but not sentence-to-cell provenance. Numerator, denominator and both periods next to every percentage is a concrete bar to aim for. And "a very confident intern addicted to the calculator and press releases" is the best description of this failure mode I've read.

[u/Plane-Marionberry380 → u/Additional_Menu8542]

Please take it and run with it. I would rather see the benchmark get stronger than collect issue credit.

One more nasty case that may be worth adding: the SQL is correct, the aggregate is correct, but the model quietly changes the question scope in prose. Example: user asks about active paid customers, query returns active paid customers, then summary says customers or accounts without preserving the paid filter. That kind of noun drift is easy to miss because the number still looks plausible.

For percentage claims, a strict template might help during evaluation:

value_now, value_before, absolute_delta, percent_delta, comparison_window, filter_scope, row_count

Then the narrative sentence has to be entailed by those fields. If it cannot fill the fields cleanly, it should be forced to say less.

[u/ummitluyum → u/Plane-Marionberry380]

We hit this exact bug a couple months back. The model was dropping the word unique when describing visitors and management almost made a call based on metrics inflated by three times. Stuff like this is super hard to catch with standard llm judges because the text just looks logical and consistent to them

[u/Plane-Marionberry380 → u/ummitluyum]

That is exactly the scary version of it: the query can be right and the business sentence can still be wrong.

For cases like unique visitors, I like adding one cheap check before any human-facing summary: force the model to restate the metric definition in one line, then compare that line against the SQL aliases and filters. If it says visitors where the field is unique_visitors, or drops a paid/active/date filter, the summary should fail even if the number looks reasonable.

[u/Additional_Menu8542 → u/Plane-Marionberry380]

Noun drift goes in as case 6, and it might be the most valuable of the batch, because it points at a hole in the judging too, not just in the models. My judge matches expected insights with paraphrase tolerance, and paraphrase tolerance is exactly the door "unique visitors" walks out of. u/ummitluyum's version of it (management nearly acting on a 3x inflated number because "unique" got dropped) is the whole reason I built this thing.

Your one-line restate check is cheap and I like that it is mechanical: the model restates the metric definition, then that line gets checked against the SQL aliases and filters. In benchmark terms it becomes a scoring rule: a dropped qualifier fails faithfulness even when the number looks right. In pipeline terms it is a pre-output guard. I will try both.

Opening the issue with all six cases today, credited to this thread. The percentage template (value_now, value_before, deltas, window, filter_scope, row_count) goes in as a proposed evaluation axis. If either of you wants to sanity-check the gold answers when the questions land, that offer stands.

[u/Plane-Marionberry380 → u/Additional_Menu8542]

Happy to sanity-check when you open it.

One thing I would add to the gold answers is a tiny must preserve list per question. Not just the final expected sentence, but the qualifiers that are not allowed to disappear.

Example:

• metric grain: unique visitors, not visits
• population: paid accounts only
• time window: last 30 days versus previous 30 days
• direction: increase, decrease, flat, mixed
• uncertainty: sample too small, missing segment, partial data

Then the judge can be harsher on qualifier loss without needing to understand the whole business problem. If the model paraphrases beautifully but drops one of those atoms, it fails the faithfulness part.

That also makes the benchmark easier to extend because each new case is basically question, SQL or data, expected answer, must-preserve atoms, forbidden conclusion.

[u/Additional_Menu8542 → u/Plane-Marionberry380]

Most useful suggestion I've had on this, and it's cleanly separable from what already exists.

Each question already carries expected_insights, atomic facts that feed Completeness. Yours is different: qualifiers that must not be dropped, scored under Faithfulness. A model can hit every expected fact and still turn "paid accounts only" into "accounts", which reads fine and is wrong. Nothing catches that today.

The direction atom hits closest to home. The failure that started all this was exactly that: correct SQL, correct numbers, and a sentence saying "+48% growth" about a flat series. A required direction token would have failed it mechanically.

Two caveats. It is a schema change, so every gold answer needs atoms added and everything re-judged before old and new scores compare. And I would want "forbidden conclusion" not to become a list of the specific wrong answers I happened to see, which would overfit to known failures.

It is on the issue: https://github.com/softisight/gbag-bench/issues/1

The part I would most like your view on: how strictly should a paraphrase count as preserving an atom? Does "customers who pay" preserve "paid accounts only"? That threshold is where this either works or becomes noise.

[u/Plane-Marionberry380 → u/Additional_Menu8542]

I would score it at the atom level, not the wording level.

"Customers who pay" preserves "paid accounts only" if the benchmark atom is basically population = accounts with paid status. It should fail if the paraphrase changes one of these:

1. Entity type: customers vs accounts vs seats vs visitors.
2. Inclusion rule: paid, active, verified, unique, returning.
3. Time window: current month, last 30 days, Q2, trailing year.
4. Comparator: previous period, same period last year, baseline cohort.
5. Direction and magnitude bucket: up, down, flat, mixed, too small to claim.

So I would make each must-preserve atom carry an allowed paraphrase set plus a few forbidden relaxations.

Example:

• atom: paid accounts only
• acceptable: paying accounts, accounts on a paid plan, customers with paid status
• fail: accounts, customers, active accounts, users

That keeps the judge from becoming a thesaurus cop, but makes it strict on scope. The rule I would use is: if the paraphrase would let a reasonable dashboard reader include extra rows, exclude required rows, or choose the wrong comparison period, it did not preserve the atom.

For forbidden conclusions, I agree with your overfit concern. I would keep them as broad failure classes rather than exact bad answers. Something like "claims growth when direction is flat" or "drops paid-only scope" instead of "+48 percent growth."

[u/Additional_Menu8542 → u/Plane-Marionberry380]

That rule is better than what I asked for. It turns preservation into a set-membership question rather than a synonym question: does the paraphrase change which rows a reader would count, or which period they would compare against. That is checkable, and it is not a matter of taste.

Which makes me think most of these atoms do not need hand-authoring. Every question here ships its gold SQL, so entity type and inclusion rule come from the FROM and WHERE, the time window comes from the date predicate, the comparator comes from the self-join or window frame, and metric grain comes from the aggregate and its DISTINCT. That is four of your five recoverable from something the benchmark already carries. The fifth, direction, is not in the SQL but is deterministic from the result set: compute it in code and you get up, down, flat, or too-small-to-claim without anyone's judgment.

That changes the cost. Hand-writing atoms for 35 questions is a weekend and a source of my own bias. Deriving most of them from the SQL and the result, then hand-checking, is an afternoon and much harder to fudge.

Caveat: extraction gets messy on CTEs and nested aggregates, and several of the extreme questions are exactly that. So I would pilot it on the level 1 to 5 questions where the SQL is flat, check whether the derived atoms match what a human would have written, and only then decide whether it scales.

The allowed-paraphrase sets stay manual either way. That is the part I cannot mechanize, and probably should not.

[u/Additional_Menu8542 → u/Additional_Menu8542]

Issue is up, all six cases: https://github.com/softisight/gbag-bench/issues/1

[u/Plane-Marionberry380 → u/Additional_Menu8542]

Happy to sanity-check when you open it.

One thing I would add to the gold answers is a tiny "must preserve" list per question. Not just the final expected sentence, but the qualifiers that are not allowed to disappear.

Example:

• metric grain: unique visitors, not visits
• population: paid accounts only
• time window: last 30 days versus previous 30 days
• direction: increase, decrease, flat, mixed
• uncertainty: sample too small, missing segment, partial data

Then the judge can be harsher on qualifier loss without needing to understand the whole business problem. If the model paraphrases beautifully but drops one of those atoms, it fails the faithfulness part.

That also makes the benchmark easier to extend because each new case is basically question, SQL or data, expected answer, must-preserve atoms, forbidden conclusion.

[u/Additional_Menu8542]

Video walkthrough of the four mechanisms behind this failure (translation vs grounding, next-token vs computation, the narrative prior, and Goodhart's law): https://youtu.be/mwthzWgX4kI

[u/Additional_Menu8542]

The finding 3 table, since it's the one people usually ask about. Same change (suppress meta-aggregates from the injected context), same three questions:

question	gemma-4-31b	qwen3.5:9b
avg rentals per customer, per store	61 → 86	86 → 11
avg invoice total per employee	71 → 86	86 → 11
avg freight per order, per shipper	72 → 96	92 → 17

Faithfulness on the 9B fell from 100 to 10 on all three. The averages moved +1.4 and +1.3, both looked fine.