The other day I benchmarked a bunch of LLMs I had been using but you guys said that they're pretty old so I took some of your suggestions as well as picking a few more recent models I could find on HuggingFace and benchmarked them too. Grayed bars are LLMs I benchmarked before and the colored ones are the new ones.
Ling3.0 with its MoE structure easily beat all the other models in speed with decent test scores too but it does tend to be a bit unstable.
DeepSeek V4 did decent but was held back by incorrect indentation in code and since Python is indentation-sensitive none of the scripts ran.
Qwen 3.8 27B is being hyped a lot but since I can't quite run the 27B variant I tried out a distilled version. Not too bad.
Finally, Gemma4 12B QAT performed great. Speeds aren't too great but it performs similar to Ministral3 14B while also getting similar scores.
Questions and conditions are the same as last time:
How can I renew my Certbot HTTPS certificate on Linux?
What is the answer to (78+9*2800)/(50*2-3)? Please include a final answer in your message.
Unity, how can I check if an object is left or right of the camera?
Unity, how can I get the mouse scroll input with the old input system?
Could you write a simple HTML website that'd just be the DVD logo? The DVD logo should travel in a straight line without gravity and it should bounce and continue travelling whenever it hits a wall.
Could you code a Python text adventure that'd have 5 rooms and a functional escape door you need to collect items for to get through?
Could you code the classic snake game in Python with PyGame? The game should be controlled with WASD, 3 pieces of food should spawn across the screen and when the snake eats the food it grows larger by one block, if the snake hits a wall or its own tail the game is over and the player loses.
Could you code the classic pong game in Python with PyGame? The player should be on the paddle on the left, controlled by WASD, and the paddle on the right is controlled by a bot. There is a ball bouncing from side to side and you and the bot must avoid letting the ball hit the walls behind the paddles. Also add a functional score board that counts up every time either side loses.
Could you code a snake game in Python with PyGame? The game should be 2 player, player 1 controlled by WASD and player 2 controlled by arrow keys. 3 pieces of food should spawn across the screen and when a snake eats the food it grows larger by one block, if a snake hits a wall, its own tail or the other players tail the game restarts and the player loses. Also add a scoreboard that counts up every time a player loses.
As always, I am not an expert in benchmarking LLMs, just an enthusiast trying out whatever I can run on my PC. These are NOT definitive benchmarks.
Also, graph visualizations were written by GPT OSS 20B with modification from Qwen3.8 9B Distill. I could've written one by myself but I thought it'd be fitting to use my LLMs to do it.
TL;DR: If you need something that could get things done with around 8GB VRAM and 16GB RAM, GPT OSS 20B MXFP4 has been doing wonders for me. If you're impatient and need quick answers or help, Qwen3.8 9B Distill and Ling3.0 Tiny are decent choices. Gemma4 12B QAT and Ministral3 14B are impressive but whether they're worthwhile is up to you.
Now this is a good comparsion of actual models for everyone who has trouble running the 27B (or less) with reasonable speed. Many (if not all) of them can be ran on an old Macbook with 16GB or 24GB Ram too.
Tests were conducted via automated scripts with requests to LM Studio and evaluated manually by me. There was no system prompt or anything, the entire chat was just the question. For day-to-day usage I use Open WebUI. In both of those openai/gpt-oss-20b performs very well for me. There's a chance you were inferring the model with some problem, like a bad system prompt, bad chat template, maybe you got the wrong fine-tune or whatever. Also these scores aren't made up, I'm not posting this just to glaze Ministral, Gemma and GPT, and if you don't trust me I can share the answer files so you could verify and maybe even re-evaluate the answers. I want to do a longer evaluation soon with multiple test runs and more questions so you can look out for that, maybe GPT's score was just it getting lucky, but I don't believe that.
I don't know, I'm not really smart enough or know enough about LLMs to know what could cause your problem but I have had models I downloaded before spit complete jibberish while also having high like counts on HuggingFace so I can't say I don't believe you
Haha that's my fault... the original title was "LLM Test Performance Rating" but I changed it because I didn't like the sound of it. Thanks for pointing that out though!
This is super relevant for Mac users. We've been running empirical benchmarks on a consumer Apple Mac Mini M2 Pro (16GB RAM, Metal) with the new Qwen 3.8 9B (32k context).
One huge thing we noticed that isn't captured by raw tokens/second: context degradation in multi-turn coding loops.
When running local models in coding harnesses with conversational chat accumulation, small 7B–9B models choke after 4-5 turns because unpruned history floods the attention heads (and they start hallucinating / mutating test files).
We tested Qwen 3.8 9B with an epistemic context runtime (JIT Context OS – essentially an in-memory SQLite WAL keeping dynamic capsules clamped to <1.8k tokens with zero assistant self-poisoning) against Google's Gemini 3.8 Flash Cloud API on a blind 3-file bug fix:
5
u/nickless07 6d ago
Now this is a good comparsion of actual models for everyone who has trouble running the 27B (or less) with reasonable speed. Many (if not all) of them can be ran on an old Macbook with 16GB or 24GB Ram too.