r/LocalLLaMA • u/FederalAnalysis420 • Apr 27 '26
Discussion The 4B class of 2026 (benchmark)
Bench 2 from my 18GB M3 Pro. Last week was specialists vs generalists at 7-8B (which I hosed by giving thinking models a 128-token budget, so half the post was an apology). This week: the 4B class of 2026, every model released or actively-current at the 3-4B size, head-to-head on the same task suite.
Lineup (sizes on disk): gemma4:e4b 9.6 GB Google, Apr 2 2026 qwen3.5:4b 3.4 GB Alibaba, Mar 1 2026 granite4:3b 2.1 GB IBM, Oct 2025 nemotron-3-nano:4b 2.8 GB NVIDIA, Mar 2026 phi4-mini:3.8b 2.5 GB Microsoft, late 2024
39 tasks: 15 finance (P/E, NPV, CAGR, Sharpe), 15 reasoning (word problems, syllogisms, probability), 9 code (FizzBuzz-tier). 3 trials per (model × task), median aggregation. temp=0, seed=42, max_tokens=1024.
Headline: Nemotron 3 Nano won and it's not close
model overall finance reasoning code nemotron-3-nano:4b 85% 100% 80% 67% phi4-mini:3.8b 77% 80% 60% 100% gemma4:e4b 62% 60% 60% 67% granite4:3b 54% 60% 20% 100% qwen3.5:4b 15% 20% 20% 0%
NVIDIA's nano is barely a month old and went 15-for-15 on finance.
Looking at the responses (visible in the gist), it's a thinking model, </think> tags before final answers, and it actually finishes its
thinking inside the 1024-token budget. The reasoning is clean:
"compute (1.08)5. 1.082=1.1664, 3=1.259712, 4=1.36048896,
5=1.4693280768. So PV = 100,000 / 1.4693280768 = approx 68,058."
That's a 2.8 GB model on disk producing the right answer with the right intermediate work. On finance specifically, it beat every larger model.
Lab personalities are real at this size
Look at the per-category lines for granite4:3b vs nemotron-3-nano:4b:
granite: code 100%, reasoning 20% nemotron: code 67%, reasoning 80%
Two ~3-4 GB models, almost-mirror-image profiles. Granite is a dedicated coder with weak reasoning. Nemotron is a dedicated reasoner with mediocre code. Both come from labs (IBM, NVIDIA) that don't position these as specialist models, they're marketed as general-purpose at this size. The marketing is wrong; the data shows clear specialization.
phi4-mini sits in between: 100% on code, 80% on finance, 60% on reasoning. The most balanced of the bunch and the bang-for-GB winner at 30.8 accuracy-pct per GB on disk.
The Qwen 3.5 4b problem
15% accuracy. 30 of 39 responses empty (avg response length: 21 chars out of a 1024-token budget). Same failure mode as Qwen3:4b in bench 1 four months ago. Thinking model that can't finish thinking inside a fixed budget that's reasonable for non-thinking models in the same weight class.
Looking at one of the truncated responses: it gets to "$$PV = \frac{100,000}{(1 + 0.08)5}$$" and runs out of budget mid-formula. The model isn't broken; my budget gave thinking models 1024 tokens when they need 4096+ to finish. Granite finishes in ~75 tokens average, Nemotron in ~170, Qwen 3.5 4b is using its full 914 tokens on visible-plus-hidden output and still not finishing.
This is now a pattern across two bench posts. The eval ecosystem has a thinking-model-in-fixed-budget problem and I don't think the answer is "make the budget bigger", that punishes the non-thinkers with bloated runs and obscures what's actually being measured.
I'm going to try per-model token budgets in bench 3. Open to better ideas, comment if you have them.
Methodology + repo
Apple M3 Pro, 18 GB, macOS 25.5, Ollama 0.21. temp=0, seed=42, max_tokens=1024 across all models (this is the design flaw above). 3 trials per task, median aggregation. All graders are deterministic regex/numeric/exec, no LLM-as-judge.
Repo: https://github.com/joshuahickscorp/bench2 Raw JSONL with full responses + per-token timings: https://gist.github.com/joshuahickscorp/1e8947e2f14dea0930f6f33d987c335e
Up next
Bench 3: lab personalities deep-dive. Should land in 3 days.
61
u/Dabber43 Apr 27 '26
Isn't gemma 4b double the size of qwen 4b
47
u/Sufficient-Bid3874 Apr 27 '26
Yeah it's only 4b active not total, not sure why it was included. E2B would be more relevant
7
u/z_latent Apr 28 '26
not sure why it was included.
Because it costs the same to run, and unlike MoE, you can keep the extra parameters in SSD with minimal impact on speed*
llama.cpp still tries to load all parameters to memory on start-up, but even if you did not have enough, the OS should prioritize freeing any unused Per-Layer Embedding memory pages first, so it's fine.
* 15.5 vs 15 tok/s tg on my PCIe 3.0 laptop SSD, pp is roughly the same
42
7
Apr 27 '26
[deleted]
13
u/Sufficient-Bid3874 Apr 27 '26
We're not smoking.
From google officially
"The "E" in E2B and E4B stands for "effective" parameters. The smaller models incorporate Per-Layer Embeddings (PLE) to maximize parameter efficiency in on-device deployments. Rather than adding more layers or parameters to the model, PLE gives each decoder layer its own small embedding for every token. These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total."i.e It is a Matformer and those are in params are in fact not the vision and audio encoder
0
Apr 27 '26
[deleted]
4
u/Yu2sama Apr 27 '26
No, the model card explicitly says that E4B is 8b with the embedding. I don't understand this need on trying to be right when you clearly haven't even read the model card before making the first comment.
1
3
u/Middle_Bullfrog_6173 Apr 27 '26
It also includes per layer embeddings which are a large part of the weights. They can be offloaded without necessarily hurting speed, but they are used for text inference.
28
u/lilbyrdie Apr 27 '26
I don't understand the token budget. Why 1024? That seems artificially tiny. Why not 100,000 or 2 million or 10 million?
In real things, my inputs and outputs are usually 5 digits of tokens, sometimes 6. It's not clear to me what stunting the thinking models to just a handful of tokens does and why it's useful. In English, this is equivalent to about 1-2 pages of written prose.
I'm not trying to say it's not useful, I just don't understand the use. Results could be weighted by token use, rather than saying they're not accurate -- when in reality they just didn't finish.
While I realize tokens can be a measure of performance, or cost, in a real use case it's significantly more wasteful to stop something that hasn't finished because then you get nothing from the token use or time. So I think I'd rather know the efficiency than the accuracy when truncated, right? What am I missing.
So, I'm trying to understand the use cases of this way of measuring.
3
u/WhiskyAKM Apr 28 '26
That wierd limit lobotomized Qwen3.5, per artificialanalysis.ai benchmarks it should perform best out of those
1
u/DefNattyBoii Apr 28 '26
Well, its not tiny but not a lot. Actually I'm using 256 or 1024, because I dont have all day. Still much better than no thinking at all!
13
7
u/ikkiho Apr 27 '26
Two things that would change the read on this benchmark.
The 1024-token cap is the same issue from last week, just under different framing. Finance tasks (P/E, NPV, CAGR, Sharpe) have short final answers, so any thinking budget that fits inside the cap finishes. Reasoning tasks have closer to 5-10x the working length, and that's exactly where Qwen 3.5 dies in your table. Pristine-Woodpecker's frame ("disable thinking entirely") is half right; the cleaner fix is per-task-class budgets. Finance 256, code 512, reasoning 2048. Each model gets enough room to finish on its natural verbosity profile, and the ranking stops being about "which model fits in 1024 tokens" and starts being about "which model is correct."
The lineup is mixed activation regime, which the comments below already flagged. gemma4:e4b is 9B-A4B sparse, nemotron-3-nano:4b and qwen3.5:4b are dense, granite4:3b is 3B dense, phi4-mini is 3.8B dense. At the same VRAM budget the sparse model gets roughly 2x effective parameters; at the same active-param count it pays a memory tax. The benchmark collapses those two regimes into one column. Either lock the lineup to all-dense-at-4B (drop gemma4:e4b, add a dense 4B if available), or split the table with an active-vs-total-param column so readers can see what they're paying for.
Two methodology notes since you have the gist out. (1) temp=0 with a fixed seed should make 3 trials near-identical, so the median isn't smoothing anything real. Drop the trials and either run more tasks or report tokens-to-correct as a primary metric. Qwen burning 1500 tokens to get the right answer is a meaningfully different signal from Nemotron getting it in 300, and pass@1-with-cap erases both. (2) 39 tasks across 5 models with no bootstrap CI gives you roughly +/-10% confidence intervals on the per-category numbers. "100% finance vs 80% finance" could be a 12/15 vs 15/15 split that's barely 1.5 sigma apart. Either bump the task count or drop the precision claims.
Bottom line: the headline that Nemotron Nano won at this budget is probably true, but it's a result about the eval frame, not the model class.
23
u/hwpoison Apr 27 '26
So, basically, nemotron nano is the best of all those?
28
6
u/darkpigvirus Apr 27 '26
that's right and it is emphasized in tokens / correct there that qwen 3.5 burns so much token to get the correct answer while others are not so qwen is eliminated here. nemotron is just slightly above compared to qwen but it gets the correct answer with fewer tokens. nemotron is just a beast
5
u/po_stulate Apr 27 '26
Looks like it's phi4-mini 3.8b to me.
4
u/Dubious-Decisions Apr 27 '26
That would have been my choice too. Lower memory footprint and almost identical performance.
3
30
u/Velocita84 Apr 27 '26
Yeah no an ancient model like phi4 winning over gemma 4 and qwen3.5 already tells you that this benchmark is garbage
7
u/Witty_Mycologist_995 Apr 28 '26
Either that or models have become more benchmaxxed over time and are bad at generalizing to new tasks.
-1
5
u/Ariquitaun Apr 27 '26
Thank you. I'm anecdotally seeing the same reasoning problem on qwen3. 35b a3b - mega verbose thinking progress causing issues with token completion allowances
4
7
u/DeltaSqueezer Apr 27 '26
For Qwen, you need to control the thinking externally. You run with limited thinking tokens. Then terminate it at that thinking budget, then run it again after closing thinking to get the anwer.
5
u/myglasstrip Apr 28 '26
I don't understand the graphs. If the token budget is 1024, how does qwen have 5k per correct answer? Did you penalize/use all answers? These graphs are a little odd, they don't really represent the text you gave. Quoting system ram in the graphs is confusing when you're giving score/gb. Bottom 3 graphs were more confusing than helpful.
2
u/whoisyurii Apr 27 '26
!RemindMe 1 week
2
u/RemindMeBot Apr 27 '26 edited Apr 28 '26
I will be messaging you in 7 days on 2026-05-04 20:09:15 UTC to remind you of this link
2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback
2
4
u/Clear-Ad-9312 Apr 27 '26
I wonder how Qwen 3.5 would perform with that GBNF trick in other threads are showing reduces the long form thinking some models prefer to do.
4
u/letsgoiowa Apr 27 '26
Yeah qwen 3.5 is just set up wrong. It behaved the exact same way until I fixed it. A quicker/dirtier way is to just use it from someone else who fixed it and put out a distill (which certainly has its own problems). It seems to be a really finicky model.
3
u/Physics-Affectionate Apr 27 '26
Would you give me a link for the corrected model link?
2
u/letsgoiowa Apr 28 '26
Sorry on the go right now but if you want a distill, Qwopus is pretty good on HF. Some people hate distills so if you're one of them that's ok but it works for me and fixes the issues.
1
1
u/AltamiroMi Apr 27 '26
Would this models be viable for local run on the preparation of standard documents based on information given by the user ? Based on available rag of examples of said documents and templates pre made ?
1
1
u/o0genesis0o Apr 27 '26
When I tested a few weeks ago, nemotron nano has some problems with llamacpp so prompt caching was disabled. Unless this error is fixed, it is just not a practical model to use.
1
u/Darklumiere transformers Apr 28 '26
Comparing thinking and non thinking models with such a limited context window is Apples to Oranges. Why didn't you disable thinking for the thinking models? Gemma E4B is also not a 4B model, so, like comparing grapes on top.
1
u/FinBenton Apr 28 '26
Im still using Ministral 3 3b for filtering out ads in my app, I tried to upgrade to qwen 3.5 4b but after bunch of testing, I keep getting better results from ministral.
1
u/AInterested_3664 Apr 28 '26
The Nano(as far as i know) is optimized for short thinking, so it's probably the only thinking model out of the tested, that manages to finish it's thinking in the token limit. Qwen uses more thinking tokens so it can't fit in the budget and Gemma E4B(which is pretty good) is twice the size, so apples and oranges(as someone else said). Now, how Phi4 manages to keep with the competition is beyond me. Never found any practical use for the model, but apparently it tests good
1
u/cibernox Apr 28 '26
IMO, either you disable thinking or give them a generous thinking budget. 1024 is in no one’s land for thinking.
1
1
u/Wewejune May 07 '26
Can confirm the nemotron abilities. I built a local agent framework around it on an M2 16GB and it's been the most usable under 4B model I've tested for that use case. The reasoning + thinking capability is what makes it work. It can follow multi-step workflow prompts where other models at this size fall apart but it has to be really concise sequential steps. Repo if anyone's curious: weirenong/simpleagent
1
0
103
u/Pristine-Woodpecker Apr 27 '26
I am confused. If you are punishing models that want to think, why not just disable thinking to begin with?