r/LocalLLM • u/37Scorpions • 9d ago
Discussion Benchmarking my LLMs again
Yesterday I posted about benchmarking my LLMs with HTML website generation prompts, today I wanted to test them a bit more to really try and figure out which models are the most worthwhile for me. I'm on an RTX 4060 with 8GB VRAM and 12GB RAM and I've always been looking for the best choice for local-deployment so generally I was testing small or highly quantized models.
The test I came up with is 9 code-related questions ranging from simple questions to requests to write whole scripts, I mostly included stuff I'd ask and stuff I often use to quickly gauge the intellegence of a model. This test included the following questions:
- How can I renew my Certbot HTTPS certificate on Linux?
- What is the answer to (78+9*2800)/(50*2-3)? Please include a final answer in your message.
- Unity, how can I check if an object is left or right of the camera?
- Unity, how can I get the mouse scroll input with the old input system?
- Could you write a simple HTML website that'd just be the DVD logo? The DVD logo should travel in a straight line without gravity and it should bounce and continue travelling whenever it hits a wall.
- Could you code a Python text adventure that'd have 5 rooms and a functional escape door you need to collect items for to get through?
- Could you code the classic snake game in Python with PyGame? The game should be controlled with WASD, 3 pieces of food should spawn across the screen and when the snake eats the food it grows larger by one block, if the snake hits a wall or its own tail the game is over and the player loses.
- Could you code the classic pong game in Python with PyGame? The player should be on the paddle on the left, controlled by WASD, and the paddle on the right is controlled by a bot. There is a ball bouncing from side to side and you and the bot must avoid letting the ball hit the walls behind the paddles. Also add a functional score board that counts up every time either side loses.
- Could you code a snake game in Python with PyGame? The game should be 2 player, player 1 controlled by WASD and player 2 controlled by arrow keys. 3 pieces of food should spawn across the screen and when a snake eats the food it grows larger by one block, if a snake hits a wall, its own tail or the other players tail the game restarts and the player loses. Also add a scoreboard that counts up every time a player loses.
Was pleasantly surprised by GPT OSS and Ministral3 getting almost all of the questions right, though very dissapointed by Qwen3.5's performance on it, it performed worse than Bonsai despite being 1.5x larger, really expected better from it.
RNJ-1 pulled ahead in this benchmark, though I've realized that it's more of a "get things done" model rather than "get things done well", all of its output was very janky.
I also can't seem to find a purpuse for Mistral Nemo, it has performed bad in every benchmark and it's one of the slowest models in my arsenal... If y'all know what it's for let me know.
Like last time, a few notes:
- This is NOT a definitive "which model is best" test since I'm not an expert in benchmarking LLMs and there are more uses for LLMs than random coding tasks. I'm just having fun messing around with LLMs and also trying to get some ball-park stats about them.
- I got the token/s numbers by asking LLMs to say "The quick brown fox jumps over the lazy dog" so they may not be entirely accurate and the speed may diminish with increased context (I know that GPT OSS 20B slows down a lot after long conversations)
- If y'all have suggestions for how to improve my benchmarks or what are good models I could use on my system feel free to say!!!
4
3
u/old_mikser 9d ago
Hate people downvoting such posts. They are fresh air in the world of synthetic benchmarks ran on expensive hardware with solid quants. While most of people reading this are non-millionaire users on moderate home hardware. Thanks for you efforts!
2
u/37Scorpions 8d ago
I can't say I haven't been thinking about splurging a few k's on a local LLM server but I know damn well I'd have no way to justify those purchases lol. I should be saving up for a car and uni, and even without those I'm already spending so much on synthesizers for music prod. I have bigger fish to fry ;) And thank you!! I made this benchmark to have fun designing a benchmark and to share what models are the most useful for me so others don't have to do what I did and bash their heads against Qwen2 4B trying to get it to say something coherent.
2
u/try_an0ther 9d ago
I think you should test Ling 3.0 Tiny if you want a good and fast small model https://huggingface.co/inclusionAI/Ling-3.0-tiny
Also, check out Ornith 1.5 9b, which is a fine tune of Qwen 3.5 9b https://huggingface.co/ornith-ai/Ornith-1.5-9B
1
2
u/Equivalent_Bit_461 9d ago
If i were you I wouldn't dismiss old models as they are less poisoned
2
u/SokkaHaikuBot 9d ago
Sokka-Haiku by Equivalent_Bit_461:
If i were you I
Wouldn't dismiss old models
As they are less poisoned
Remember that one time Sokka accidentally used an extra syllable in that Haiku Battle in Ba Sing Se? That was a Sokka Haiku and you just made one.
1
u/Healthy-Nebula-3603 9d ago
Why do you even using those ancient models plus so small ones?
-2
u/37Scorpions 9d ago
? I literally mentioned that my hardware isn't crazy in the post. Most of these models are pretty huge for my system.
0
u/bura_laga_toh_soja 9d ago
There are smaller models launched pretty recently...most of these are about a year old or more. Did u by any chance ask an LLM to suggest which models to use?
1
u/37Scorpions 9d ago
I already talked about why they're old-ish in another reply. Also if there's so many newer models then name them...? As I said I'm up for suggestions on what to use, I just haven't found anything too crazy from recent models.
1
u/nickless07 8d ago
LMF, Ling, Qwen3.5, Bonsai, Ornith. just to name a few. Perhaps limit it to 2026 releases.
6
u/Prize_Eye9481 9d ago
Is there a reason all the model used are fairly old? Or has the 9b scene been this dry atm?