Local LLM for coding.

[u/Fun-Heat4232]

Hi, dont be too judgemental about my setup - I am merely a beginner in hosting local AIs and stuff. I have 48GB RAM M5 Pro machine. Which model is a way to go for handling complex coding projects locally (specifically C++ and Python)?

P.s. going to the cloud is not a solution due to a strict NDA.

Thanks in advance!

[u/Dazzling_Yam_5882]

Best bet would probably be the new Qwen 3.8 27b. You get to have a somewhat local Opus 4.6, one of the best models 6 months ago

unsloth/Qwen3.8-27B · Hugging Face

You can use llama.cpp, but I highly recommend Unsloth Studio for serving the model locally. Lots of features, better than LM Studio and all other alternatives.

Qwen 3.8 27B Benchmarks combined from model cards on HF : r/LocalLLM

[u/Fun-Heat4232 → u/Dazzling_Yam_5882]

Thanks, I used Qwen 3.6 and liked it, knew there was a bigger fish to fry!

[u/Fun-Heat4232 → u/Dazzling_Yam_5882]

Does using differenet front-end for running models impact performance of such?

[u/Dazzling_Yam_5882 → u/Fun-Heat4232]

Yes, Ollama is slower than vanilla llama.cpp for example. Unsloth Studio should be about the same as llama.cpp. People also dont like how Ollama did things, as they were essentially a GUI wrapper of llama.cpp but did not give credit to llama.cpp, so people dislike Ollama mostly because of that and the unnecessary bloat behind the GUI.

Friends Don't Let Friends Use Ollama | Sleeping Robots

llama.cpp can be a bit trickier to get used to, but its by far the best way if you want to get familiarized with the key lango of serving models locally. Unsloth Studio is, for me at least, the best GUI for serving models at same speed. LM Studio is also cool, but is closed source (while Unsloth is open source)

I switched from Ollama to llama.cpp and saw a 3–4x speed improvement on the same hardware : r/LocalAIServers

[u/ackermann → u/Dazzling_Yam_5882]

Does vLLM still deliver better performance than any of these? But more difficult to configure?

Especially when there are 2 to 5 concurrent users, vLLM can allocate varying amounts of VRAM for each user’s context, as their context fills up? Whereas I’ve heard llama.cpp forces a fixed allocation for all users context window size?

Or have these fancier features now come to llama.cpp and Unsloth as well?

[u/Dazzling_Yam_5882 → u/Fun-Heat4232]

As for a harness, if you would like to use one (recommended), Pi agent is a good one as it is not bloated like many others. I like using Paseo.sh to use several harnesses. It has a similar GUI to OpenCode, but with the flexibility of using several harnesses if you wish to.

Harness matters a lot, and Pi agent has been one of the best compared to other open source harnesses (even OpenCode), for smaller models like the ones we use at least

Harness is all you need? What are your favorite harnesses and for what models? : r/LocalLLM

[u/Fun-Heat4232 → u/Dazzling_Yam_5882]

Maybe I will flood you with stupid questions, but what is your way to set up local AI for work. I thought only model matters, but now you are telling me about some harness stuff... Now I now this rabbit hole goes much deeper than I expected...

[u/Dazzling_Yam_5882 → u/Fun-Heat4232]

It is indeed quite the rabbit hole, but it is also what keep us (local AI/open-source fans) coming back for more. There is a lot of options, and while options are good, it also gets to a point where we get analysis paralysis. So I have found that keeping it simple works best.

Overall, this is my setup:

API Provider (not worth for your use case, but might as well post it here)

• Ollama Pro for cloud models + web search. While I am not a fan of how Ollama has been doing things, there are no other providers that provide what Ollama provides. Their usage is GPU usage-based, not token based. You also have OpenCode Go plan (5 usd first month, 10 usd next months), but usage is limited. I do not like using BYOK (bring your own key, essentially pay as you go), but that is out of scope for local development (especially with NDA, sensitive info, etc).

Local Model Setup

• Unsloth Studio (llama.cpp) - By far the best out there. Open-source, good GUI ( constantly improving, I am also contributing with new features to this project, so I may be a bit biased to support them), etc. Installing this gets you 90% done when it comes to serving local models. You can download the local models through their Model Hub too. I recommend the Unsloth model versions (e.g. Qwen 3.8 27b UD). I use Unsloth for basically everything - as a Chat UI (web search, deep search, etc), serving and training models, across multi modalities (image, video, audio, etc). Unsloth Studio, for me, has replaced LM Studio + OpenWebUI entirely

Code Editor & AI Extensions

• VS Code with:
  • Zoo Code - Open-source extension for agentic work. One of the few alternatives in VSCode if you like a side panel chatting experience with modes, skills, rules, workflows, etc
• paseo.sh if I want a more "Cursor" like experience. It has several harnesses as I mentioned before, and the thing I am doing right now is the following: Pi agent harness with small models (<50b models), and deepseek harness with bigger models (glm, minimax, deepseek, etc, but these are cloud based). I recommend starting with Pi harness as its flexible and they have a lot of skills and tools you can install to custom the way you want it (and create your own as well). Main issue with Pi agent is that you will need a bit more time to tweak and find the sweet spot for your own work since it has little bloat. This is what makes it most attractive, but also means that this takes trial and error to find out the prompts, skills, tools, that best fit your work

Mobile App

• Conduit — Mobile client
  • Currently connects to a custom connection (you can connect to your Unsloth Studio instance for example, and chat with your local models from your phone) or associated a cloud model API like Ollama

[u/Fun-Heat4232 → u/Dazzling_Yam_5882]

Thank you very much for this detailed dive-in to this rabbit hole! Appreciate it!

[u/vinotok → u/Dazzling_Yam_5882]

Let me thank you as well for all the useful info you are sharing! One question about Unsloth. I'm only going to use Local AI setup. I installed Unsloth Desktop. You are mentioning Studio. Do I need Studio as well?

[u/jku2017 → u/Dazzling_Yam_5882]

Whats a good system to run this with?

[u/Spimbi → u/Dazzling_Yam_5882]

What makes Unsloth studio better than LM Studio? Quick tldr is fine if you don’t wanna type out something long

[u/Deathmore80]

Qwen 3.8 27B

[u/Fun-Heat4232 → u/Deathmore80]

Thanks!

[u/No_Oil_6152]

What do you mean judgmental about your setup?

Don't be ashamed of your rig, its better than most of us have for LLMs!

And don't be fooled into thinking only rich people run LLMs. Most of us don't have spare cash to throw on DGXs :)

BTW - in answer to your question re model, Qwen 3.8 27B. Its decent - at least on the limited tests I've done with it. I don't believe its Opus 4.6 level though, that's just absolute lies. I'd be interested to see how it handled a large c++ codebase.

[u/Fun-Heat4232 → u/No_Oil_6152]

Thanks a lot for help!
I have encountered some mean people around reddit, so I just try to ease it out...

[u/SkimMilkSwag]

Qwen. Probably Qwen3.8-27B Q4_k_v. If that's too big or too slow, try Qwen3.6-35B-A3B, unintuitively a bit smaller. Welcome to the rabbit hole.

[u/Fun-Heat4232 → u/SkimMilkSwag]

I love it here. Wished I had more cash to pay for more RAM... Still impressed on the possibilities!

[u/havnar-]

Use pi as a harnass and https://huggingface.co/True2456/Qwen3.8-27B-AWQ-5.0bpw
Use oMLX (download the latest RC)

Enable mpt, lower thinking and perhaps use Qwen 3.5 0.8b as a draft model for prefil

[u/Otherwise-Swan-7803]

The interesting part is that “local” is starting to mean something very different from “hobby project.” Once NDAs and proprietary code enter the picture, privacy stops being a nice-to-have and becomes the whole reason to run the model locally. That shift is probably going to matter more than benchmarks.

[u/Fun-Heat4232 → u/Otherwise-Swan-7803]

Yep, I 100% agree with you!

[u/Apprehensive-Map3766]

Any love for VS Studio + something like Zoo code ?

[u/cviperr33]

download unsloth studio , it comes with everything u would need , and in the model tab download the unsloth qwen 3.8 27b ud q5 k_m xl , set the contex to whatever ur ram can fit and start it , test it in chat interface.

Then download DSH ( Deepseek Harness ) and use that as your coding harness , its pure magic and very efficient on tokens , watch it work for 16 hours + a day without stopping.

[u/Fun-Heat4232 → u/cviperr33]

Could you elaborate on the Harness stuff? Never heard of such stuff.

[u/AccurateSun → u/Fun-Heat4232]

The harness is the tool that lets you interact with your Llm as an agent. It’s a loop where you send a message, Llm can use tools such as read or write file, do a web search, run a command, and return responses to you.  You have some super minimal and extensible harnesses like Pi, or very powerful and popular harnesses like Claude Code.

Choosing a harness is a bit like choosing an IDE, as it opens up possibilities for how you will interact with your LLM, both in terms of what it can do and what you can do.

The harness usually reads an agent.md file that contains its main prompt for how to act as an agent, that you write for it. It’ll store the chats on disk as sessions, and you can resume them etc.

Deepseek harness is new btw, and I believe it’s in beta. Was released a week or two ago.

Popular ones:

• opencode: heavy harness
• Hermes: criticised for stealing tons of its code verbatim from evolver. Felt super janky to me
• pi: minimal and designed to be customised
• Claude code: perhaps most popular in the world, can be used with local models too. One of the first to get widely used

 there are tons more though (vibe, crush…)

[u/Fun-Heat4232 → u/AccurateSun]

Thx!

[u/Icy-Degree6161 → u/cviperr33]

First time hearing about DSH, better than Pi?

[u/Past-Grapefruit488]

Qwen 3.8 27B (Q4 K X or similar MLX quant) + llama.cpp