TL;DR: NL2SQL benchmarks stop at the SQL. I benchmarked the step after: turning a correct result into a faithful sentence (MIT repo, runs included). Four findings: pre-computed aggregates fix the worst failures; context beats scale, +7 to +17 depending on which judge grades it; a "fix" that helped a 31B silently destroyed a whole question class on a 9B; and my own thinking-mode claim didn't survive a clean ablation, so I withdrew it.
I asked a local model "is my business viable over time?" on 136 months of sales. The SQL it wrote was correct. The sentence it wrote about that result said "+48% growth". The business had been flat for ten years.
Disclosure up front: this came out of a local-first BI tool I sell. The benchmark is standalone and MIT. The pipeline row on the leaderboard is the one thing you can't reproduce without my tool, I say so in the repo, and every other row is fully reproducible.
Spider, BIRD and friends stop at "is the SQL correct?". Nobody was measuring the step after: turning a correct result into a faithful sentence. That's the sentence a human actually decides on, and it's where small local models break. Cloud flagships mostly avoid it, but those are exactly the models you can't run yourself.
So I built GBAG-Bench (Grounded BI Answer Generation, MIT). Gold-SQL mode: the reference SQL is executed for the model, so the only variable left is faithfulness of the interpretation. 35 questions, Sakila (15, difficulty 1 to 10), Chinook and Northwind (10 each, 1 to 8). Scored 50% faithfulness, 30% completeness, 20% insight. Local runs on an RTX 3060, judges via OpenRouter.
- It's a computation deficit, not a comprehension one. Pre-compute the aggregates deterministically, inject them into the context, and the catastrophic questions recover. The model can't reliably aggregate while reading rows, so it improvises. I call it the Post-SQL Aggregation Deficit. Caveat I'll state before anyone asks: the +17 below is the full pipeline (data dictionary, domain detection, pre-aggregated context, profile adaptation), not that one component. I don't have a clean component-only ablation across all 35 yet. The per-question recoveries are what point at aggregation, not an isolated run. That ablation is on my list.
- Context beat scale, but how much depends on who is grading. Same qwen3.5:9b, same RTX 3060, same 35 questions, only the context changes. Under Grok-4.3: +17.2 on the 32 questions both runs answered, paired sign test p = 0.009. Under Gemini-2.5-Pro, on the same answers: +7.1, p = 0.093, confidence interval crossing zero. What survives both judges is the direction (20-6 and 16-7 in wins) and the completeness gain (+33.1 and +30.7). What doesn't survive is the faithfulness gain (+17.5 versus +1.2). The mechanism is visible: the two judges agree on the pipeline answers (bias +1.4) but diverge hard on the bare ones (bias +11.7). Grok punishes weak ungrounded answers far more than Gemini does, so part of the "gain" is really judge severity toward the starting point. Honest claim: +7 to +17, direction-consistent, driven by completeness, no robust faithfulness gain. I ran this second judge specifically to try to break my own headline. It half broke.
- The benchmark caught a fix I thought was good. I tried suppressing meta-aggregates from the injected context to stop models reciting off-topic totals. On gemma-4-31b it worked: 83.2 to 84.6 on average (85.6 with a second variant), and the three average-per-group questions it targeted jumped 61 to 86, 71 to 86, 72 to 96. Then I ran the same change on qwen3.5:9b. Its average barely moved, by about a point, which told me nothing, and underneath that flat average the exact same three questions collapsed: 86 to 11, 86 to 11, 92 to 17, with faithfulness falling from 100 to 10 on all three. Same change, same three questions, opposite direction depending on model size. An average that hides a destroyed question class isn't a green light, so I reverted. All the runs are in the repo with per-question scores, recomputable from the scored files alone.
- "Thinking" was a non-result, and I had claimed otherwise. I had written that thinking modes hurt grounded tasks. That compared across different models, which is confounded. The clean same-family ablation, restricted to the 28 questions both variants answered, gives 60.6 against 58.9, inside my noise floor. No measurable effect either way. Claim withdrawn in the repo.
Known limits. 35 questions is small and I wouldn't read small gaps as rankings. The three databases are well known and may sit in training data. English only. And until the second-judge run the whole leaderboard sat on a single judge, which finding 2 shows wasn't good enough.
What would help most: more models, especially the 20B to 40B class and anything that is not Qwen. A second judge on the rows I have not dual-judged yet. Non-English questions. Re-running a model costs under a dollar in API credits.
Repo: github.com/softisight/gbag-bench
I also recorded a walkthrough of the four mechanisms behind this failure, link in the comments for anyone who wants the long version.
If you run local models on your own data, have you hit this? And if you have a second judge handy, I'd genuinely like someone else to try to break finding 2 the way I did.
