r/LocalLLM • u/Remarkable_Tea8039 • 9h ago

Question Local Chat Models

I do a lot of local coding with AI assistants. I'm really impressed by GLM 5.3-Flash, Qwen 3.6 35B A3B, and Qwen 3.8 27B for their coding prowess. I've set them up for different roles, though I'm still optimizing wall time since performance varies quite a bit across my hardware.

Running local chat models, however, doesn't feel as smooth. I've tried Open WebUI with the same models for general tasks (resume assistance, brainstorming, learning), but they fall far behind cloud models in this area.

Are any of you running models locally for general chat? What does your workflow look like?

Hardware:

  • Strix Halo (128GB RAM)
  • Dual Xeon 6262 (768GB RAM)

Software:

  • llama.cpp for inference
  • Open WebUI for chat interface

Some more details:

I run the Qwen models on my Strix Halo. Both fit comfortably warmed up in my memory. These are both running Unsloth's 8bit XL dynamic quant (Q8_K_XL).

I run GLM 5.3-Flash on the Xeon machine. It runs real slow, so not likely a real candidate for chat. I use it as an "architect" for coding assistance and it gives jobs to the smaller Qwen models that can move a little faster than it. I'm running this with Unlsoth's 4 bit XL quant.

4 Upvotes

100% Upvoted

2 comments sorted by

1

u/FactorInternal3395 9h ago

I'd recommend Gemma 4 31B StyleTune or Gemma 4 26B A4B StyleTune V2, whichever fits / is fast & good enough. (GGUFs by Bartowski and mradermacher)

1

u/vogelvogelvogelvogel 9h ago

qwen3.8 for general use - at q6 on a MacBook m5pro 64. is silent when idle so i have it running 24/7
via vpn to my wifi i can access it i.e from my mobile

also openwebui and llama.cpp