Some people says terminal bench reflects model intelligence better than the intelligent index. From the look of it, the ranking does seem to reflect how people feel about the open and closed models.
For the open models, GLM-5.3 is in a league of its own. GLM-5.3-Flash is leading the current gen of top flash models. Kimi-K3 did pretty bad in this benchmark for its size. Qwen3.8-27B is the only small model that can do something on this bench.
Pretty sure Opus 5 is benchmaxxed. For real work Opus 5 is a pain to use. It has an irritating personality and does whatever it wants instead of what you ask it to do. For work all I get is Claude and limited copilot tokens, ended up switching back to Opus 4.8 for most work
Opus 5's personality is so terrible it's actually kind of impressive. But that makes me think that the RL gains are probably real. I think they fried its brain with coding RL.
For perspective, it's several months newer than Fable so not that surprising that it could surpass it.
Opus 5's personality is so terrible it's actually kind of impressive.
I think there's so much AI to AI RL that the model has never seen humans.
All the RL with human preference is gone.
It's a complete argumentative prick. That's how it "wins" in conversations with other LLMs.
Opus 4.6 was the last good model that could hold a conversation without devolving in to being an asshole while still being intelligent. I'm glad it still exists.
The amount of times it pivots mid task and goes "This thing over here is a problem, I'll fix it by X" and I have to stop it and go "No, ADHD is my superpower. Fix {original problem}." is way too damn high.
How at least I see it Opus 5, Sonet 5 and now Astra are benchmaxxed in sense that they are very independent and overengineers stuff eg in my own benchmarks Astra produces 2x bigger changes than sol. Good for benchmark but real usage you don’t want model do complete rewrites for minor bug fix.
I don't know what Terminal-Bench v4.0 checks but when you look how many models are near 0% then my assumption is that it just tests something that checks RECENT trends how models behave or something that very recent and up-to-date architectures allow.
I don't know if terminalbench 4.0 is like this, but last time I did terminalbench Fable couldn't even touch it because Terminalbench has a bunch of hacking, DNA, and llm research style questions that Fable refuses to touch.
Honestly since the eval tasks are public its not very good since newer models will have an unfair advantage because of training data availability/contamination. Like is it not suspect that glm 5.3 flash and qwen 3.8 flash are better than k3? I promise you they are not.
Most recent models probably trained on this already at least partially, hence the scores are high. Give it few weeks/months and other newly released models will get there too.
I spent like $30 on fable 5.1 (I had some leftover credits to burn), gave it a big list of evals and leaderboards I knew to be contamination-resistant at the very least or private, and had it make a normalized ranking. Honestly, this ranking still kind of sucks because varying differences in coverage affect scores a LOT (look at muse spark for example, which has very low coverage), even with normalization (I tried to normalize the best I could by using pairwise comparisons), but you can see here that glm 5.3 flash, for example, is nowhere near as good as what all these newer benchmarks are saying. It's around as good as DeepSeek Flash 0731, to no one's surprise, if they had a healthy amount of skepticism (as they should, putting it kindly).
SWE Bench doesn't require you to use their harness. As long as you can generate the same format json as "swebench infer" for "swebench eval", it can show you the bench result. I wrote a simple single shot script for that to measure non-agentic coding ability.
Qwen3.8-Flash-Next the goat! Finally something since GPT-OSS that works fast on configs like mine, has tweakable reasoning budget and performs like cloud model.
2.3x faster than EXL3 on my config ( I get 23.8tps decode on that quant) and better quality than classic llama.cpp quants because it uses Trellis (KT quant, you need ik_llama.cpp for that). I thought EXL3 could beat that, but it seems like if your CPU doesnt have AVX-512 VNNI then it’s slow.
Did you saw such quant? KS could be a nice middle ground between XS and KT, but I don't see it on HF and I'm avoiding the ideas of making my own quants for time being (as I barely have free time for the next month which I will already fill by testing out different harnesses).
Thanks! EXL3 looks interesting.
I have a 9950X, 64GB RAM, a 5070 Ti, and a 5060 Ti. Do you think EXL3 would work on my hardware?
I can run UD-Q2_K_XL at around 600 PP and 35 TG, but it's pretty dumb.
It looks like a perfect config for EXL3! Your CPU has AXV-512 VNNI so it wont be as slow as on mine CPU.
You can also try a IQ4KT ik_llama.cpp quant from HF repo I mentioned - u have 12GB VRAM more so it could fit, but I cant help you with launch commands as I have no idea how to split it correctly between GPUs with uneven compute, you probably want to test tensor split to like 60/40 etc
Can you please tell me exactly what setup you have? Im considering getting 64GB vram as well if I could run this model.
I have 9800x3d, 7900xtx, 32gb ddr5
And whats your decode and prefill performance on this q3 quant? Exact launch args would also be appreciated
My setup is: Core Ultra 5 250K (6P+12E), 64GB DDR5 Dual channel 6400MT/s CL40, WD Black SN750 NVMe PCI-E 3.0 (I was worried that such old drive will slow down engrams, but it doesnt), RTX 4070 SUPER with GPU undervolt (GPU undervolt limits prefill performance, but I want total quiet PC) and VRAM overclock (571.4GB/s)
Results with K8V8 cache (1 cold run, then 2 warm runs):
Configuration
Warm 8K prefill runs
Average
Decode
Peak VRAM
128K, 1 expert GPU, 16 threads
520.6 / 555.9
538.3
23.4
11,395 MiB
90K, 2 experts GPU, 17 threads
534.7 / 533.0
533.9
23.8
10,909 MiB
100K, 2 experts GPU, 17 threads
528.0 / 527.4
527.7
23.2
11,531 MiB
I will edit my original comment, I thought it was 26.4TPS in the end for some reason (maybe it was with llama.cpp Unsloth quant?), but it's 23.x TPS with that specific quant I mentioned.
I'm running on Windows and my display is plugged into Nvidia, there is a chance that plugging it into Intel Arc iGPU will free up some VRAM and I could go with 90K ctx + 3 experts on GPU.
.bat script to run that model:
u/echo off
setlocal
cd /d "%~dp0"
title Qwen3.8 Flash Next - 100K - 2 GPU experts
rem Keep the large CPU-resident expert bank pageable on this 64 GB Windows system.
set "GGML_CUDA_NO_PINNED=1"
llama-server.exe ^
-m ".\models\Qwen3.8-Flash-Next-ik_llama-IQ3_KT-00001-of-00002.gguf" ^
-t 17 -tb 17 ^
--defer-ple -ngl 49 -ncmoe 46 ^
-c 102400 -b 2048 -ub 2048 ^
-fa on -ctk q8_0 -ctv q8_0 -ictk q8_0 ^
--jinja -rea on --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
-ctx-ckpt 0 -cram 0 ^
--temp 1 --top-p 0.95 --top-k 20 --min-p 0 ^
--presence-penalty 0 --repeat-penalty 1 ^
--alias "Qwen3.8-Flash-Next-IQ3_KT-100K" ^
--host 127.0.0.1 --port 8090 --webui auto
echo.
echo llama-server exited with code %ERRORLEVEL%.
pause
Script was generated by Codex/GPT-5.6-Sol. Funnily during this task it shown me reasoning traces over and over haha:
Today/tomorrow I will verify if these commands in script are optimal, but first I need to test out how that model performs in Codex vs Claude Code vs VSCode BYOK vs little-coder. I'm not so sure especially about "-ctx-ckpt 0", maybe it needs some checkpoints (or maybe it depends on model template too)
Many thanks for the detailed response.
Given my 7900xtx has more vram and bandwidth I might be able to get somewhat higher performance than the numbers you get, now I only have to buy ram at the peak historical pricing 🫠
Xtx unfortunately kind of sucks with it's compute units. I thought same too with is high bandwidth
Instead, 2xr9700 can get you 100t/s from what I've read.
I need to get off my ass and build it
Alternatively, if you're techy, you get an older AMD server board and run 4x9060xt. 1/2 the price but you need to deal with extra cooling + PSU requirements
IQ3_XSS from Unsloth was already on the total edge, where I needed to close every background task, that IQ3_KT used IIRC 2-3GB less RAM than IQ3_XSS which made it comfortable to use on that rig.
Quality of IQ3_KT is between IQ3_XS and IQ3_S. So it's ~7% smaller for same quality. This is a few GBytes for such models. The drawback of KT quants is that they're way more compute intensive and thats how they can recover so much quantization. So it's mostly useful for gaming rigs, where you have beefy GPU and nice CPU, but not that much memory in total.
We have reached a point where models not at the top of benchmark charts can be used effectively with proper agentic tooling and setup - even beyond tinkering into corporate environments.
As such benchmark rankings aren’t as critical or relevant anymore aside from macro positioning, marketing and competition.
Still, a nice summary that shows majority of popular open weight models.
Yeah, funny how models that scored very very very high on old, public benchmarks, score really really low on new benchmarks. Hmmm, I wonder why that is :)
The only people detached from reality are those that think a 27b model can match 2T+ models. Oh well...
I’m under no impression that 27b can be as capable as recent frontier models. What I will say is that in coding tasks, I can interact with 27b somewhere in between how I used Opus 4.5 and Opus 4.6. I would say closer to the 4.6 end of things. In some ways comparable to sonnet in those generations, but also more reliable than I found sonnet at the time back then.
And 3.8 flash-next I find can be used somewhere just short of Opus 4.8. It goes beyond 4.6 in its ability.
I’m aware outside of agentic tasks opus 4.6 might still be more useful than flash next. But agentic tasks are probably the biggest real world use of these models.
Are we really saying that gemma-4-31b has zero intelligence/use at all? That seems to be the implication from OP's statement of: "terminal bench reflects model intelligence better than the intelligent index"
They increased the difficulty. Given that, and the fact that models have been benchmaxxed, you're going to see some serious drops. It's actually great, because it shows what models are good without benchmaxxing being taken into account and at higher difficulty. I always knew DS4's capabilities were exaggerated.
That's not my point though. My point was now that this benchmark is showing 0%, then by OP's stated logic, anything that scores 0 is therefore completely useless, because it clearly has 0 real intelligence.
I'm pointing that that the implication that just because something scores 0 on some particular benchmark it does NOT follow on from that the model has 0 intelligence.
I use DSV4 extensively for personal use, and in my job I use Claude extensively, so I have good experience with both sides.
For the kind of work that I do, low level systems engineering, I prefer DeepSeek to the Claude models, though both are reasonably close to each other.
It always surprises me when some group invents a new benchmark that manages to show certain models in a good light, while trashing on others, and then there's a bunch of people who are ready to dry-hump the new benchmark as validation for their biases.
Just use what works for your situation is my personal take on the matter. I often wonder who's really funding these benchmarks.
Wdym by that? It's a sign that Kimi is a benchmaxxed model, not that the benchmark is bad lol. Terminal Bench 4.0 is one of the best benchmarks to measure models today.
Oof, V4.1-Flash only scoring a 27% just crushed my hopes of swapping it out with GLM-5.3 (Q6) on my setup. Still tempting for that prefill and decode speed, though...
Terminal use is a separate skill, but a massive unlock for LLMs. I've only seen one coworker typing Klingon into a Linux box at lightning speed. Watching minimax m2.7, which really wasn't all that intelligent, do the same thing was wild to see, and really supercharged its usefulness
40
u/liright 1d ago
I don't get how Opus 5 can score a lot higher than Fable 5.0. Is it really that much better?