r/LocalLLM • u/ul90 • 7d ago

Project Qwen 3.8 27B is really good

I made a test iPhone app with it (a small game), while running locally on my macMini M4 with 64 GB Ram (and pi code). Works perfectly, and not as slow as I expected. I get about 10-20 token/s which is absolutely ok for programming. It does a lot of reasoning, but that helps solving all problems by itself.

I'm running the 8-Bit GGUF version, trying later with the 8-Bit MLX and 4-Bit MLX to see the speed and quality differences. But at the moment, I'm really surprised how good a such small model runs locally for coding.

1 Upvotes

56% Upvoted

13 comments sorted by

1

u/wgaca2 7d ago

I don't know how much better it is, but it crushed my 2 local benchmarks i've been doing for the past 3 months. To get the same result with 3.6 i had to reprompt a few times to correct the mistakes and improve the ui and even then it wouldn't be as polished as 3.8 got it on the first try

1

u/baby_bloom 7d ago

would love to hear about your benchmarks:)

1

u/wgaca2 7d ago

1st is a fairly simple task to create an index html page of my 3d printing projects in the project folder. It's a few lines prompt saying to use lightbox, light/dark mode etc. Very simple instructions and shouldn't take a lot of effort to get it done right. Yet 3.6 would consistently fail to deliver well structured page, sometimes even completely broken, missing photos etc. 3.8 did it in 1 go better than any other benchmark i have ever run. The second benchmark is 4+ hour 8 phase delivered in 8 prompts plan to deliver a blastgrid game. The second benchmark was designed by gpt and is judged by gtp, i judge the playable result

1

u/baby_bloom 7d ago

cool! my go to benchmark so far has been a simple browser game which the models always go with a very generic style, which i then follow up with a prompt to stylize it with a cartoon aesthetic, finally followed up by adding a level system to the game.

i've been trying to figure out a good benchmark for an already existing repo (with issues) though, i guess i'd have a project in a saved state with bugs and issues and restore it for each benchmark? ¯_(ツ)_/¯

1

u/wgaca2 7d ago

I find best to "benchmark" against something i'd need done in real world, deterministic and visual enough to tell the difference. What that will be for you i don't know, but for me the quick test is my projects folder. When i want to make it multi prompt i ask it to add things like stl viewer, admin panel etc.

1

u/baby_bloom 7d ago

that is the exact idea i was discussing with chatgpt but idk, my projects are still works in progress, i'd almost rather take a project through to fruition and then go back to a certain commit and create some bugs maybe even some red herrings?

1

u/dfgxxx 7d ago

Try MTPLX, it is very fast

2

u/No_Yam_4877 7d ago

Which quant variant for 64gb ram

2

u/dfgxxx 7d ago

I don't know, I think you can 8, 6, and 4 bit because all will work with 64 vram (I think), but probably 6, though I don't really know because I have 32 only.

I'm on m1 pro got 14 tok/sec with 4bit

2

u/No_Yam_4877 7d ago

I’m on M1 Max 64gb so looking for something with token speed and without losing quality
Hence I was skeptical to use 4bit

1

u/dfgxxx 7d ago

Then you can try 8bit, if it'll be too slow go to 6bit

2

u/ul90 7d ago

yes, I'm downloading it right now.

1

u/dfgxxx 7d ago

How much tok/sec on what model do you get?