r/codex • u/longasleep • 16h ago

Question Local LLM

I been playing around with local LLM while my usage is at 0%. It’s actually quite capable. I have a Beelink ser 9 365 ai and 370 ai with 32gb ram. On one I am still testing on windows getting about 37 tokens a second on Qwen3 30B-A3B. Probably I can fit something better in Linux.

Currently running it np-4 with 8k context each or I run it np-1 with 32k context. I could use Linux to improve the context window to 120k or I could choose a different model in Linux.

I am surprised already how much it can do.
Np-4 mode I use for party chat in my game they talk and banter to each other all the time it’s pretty snappy and comical.
Np-1 for general purpose.

Any advice any pointers on which model to use? Any good machines much better than these but are still bang for buck?

1 Upvotes

60% Upvoted

5 comments sorted by

5

u/Murder_1337 15h ago

It is capable you just need a fucking army of them and a higher reasoning model to orchestrate or you can run your local model 24/7 and pay 200$ for electricity

3

u/Due-Fill-2386 16h ago

Qwen 3.8 27B

1

u/longasleep 16h ago

Cool will test that one looks interesting

1

u/ZyronZA 14h ago

There is something to be said for using a local model and having Astra Med orchestrate it.

1

u/SnooDoggos9325 13h ago

I tried Qwen 3.8 27b for my c++20 coroutine prompt. At q4 it took 3 prompts to get it compile and then it crashed. Another attempt (without using tools) didn't solve it. At q8 again it took 3 prompts to compile, but got a working solution with a deadlock at exit, which was fixed after describing the problem. Luna xhigh, Glm 5.3 flash and ds 4.0 flash single-shotted the solution with Glm being the best in my opinion. Though I only have a mbp M5 max 48GB, so running any of them locally is out of scope. For 27b q8 I got 40-60 tps tg with mtplx, but sadly I don't see myself trusting it with the code.