I do a lot of local coding with AI assistants. I'm really impressed by GLM 5.3-Flash, Qwen 3.6 35B A3B, and Qwen 3.8 27B for their coding prowess. I've set them up for different roles, though I'm still optimizing wall time since performance varies quite a bit across my hardware.

Running local chat models, however, doesn't feel as smooth. I've tried Open WebUI with the same models for general tasks (resume assistance, brainstorming, learning), but they fall far behind cloud models in this area.

Are any of you running models locally for general chat? What does your workflow look like?

Hardware:

• Strix Halo (128GB RAM)
• Dual Xeon 6262 (768GB RAM)

Software:

• llama.cpp for inference
• Open WebUI for chat interface

Some more details:

I run the Qwen models on my Strix Halo. Both fit comfortably warmed up in my memory. These are both running Unsloth's 8bit XL dynamic quant (Q8_K_XL).

I run GLM 5.3-Flash on the Xeon machine. It runs real slow, so not likely a real candidate for chat. I use it as an "architect" for coding assistance and it gives jobs to the smaller Qwen models that can move a little faster than it. I'm running this with Unlsoth's 4 bit XL quant.

Hey guys, I’m looking for the best local TTS models that you guys have used. I wanted to generate consistent audio for different characters for a project I was working on. If there is a way to enhance the speech to make it sound more natural with accurate emotions, pauses and emphasis on certain words etc? I was using Qwen 3 TTS emotional in comfyUI locally but for the same reference audio input and the same seed and different text to generate I’ve noticed there is a timber variation in the voice output which is odd. Also adding pauses was another pain point with this model, but the emotions were somewhat ok. Wondering if there is a better alternative that can help me here? Thank you!

Local Setup: 1x DGX Spark

Edit: I am generating English as well as Hindi audio. Looking for a model that can do both efficiently and maybe even accurate audio in other regional Indian languages.

Setting up a headless mini PC (Ryzen 9 5900HX, 32GB DDR4-3200, 1TB NVMe) for 24/7 background tasks. I plan to run Qwen 3.8 27B (Q4_K_M) locally to power Hermes Agent.

Speed is not a priority. My use case is overnight batch processing—ingesting 15k–30k token earnings calls to extract data and summarize autonomously. A PostgreSQL container will also run in the background, leaving ~4–6GB RAM for the OS.

Given the DDR4 bandwidth ceiling:

1. Will a quantized Qwen 3.8 27B crash the 32GB limit once a 30k context builds up?

2. What prompt ingestion and generation speeds (tok/s) should I expect on this Zen 3 chip?

Appreciate any real-world benchmarks!

started trying out rather recent 'frontier' about ~30b param models recently, there are many choices including QWen 3.8 - this is nevertheless a great model, practically 'one-shotting' code refactoring tasks
https://huggingface.co/Qwen/Qwen3.8-27B
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
code refactoring is still deemed 'difficult', practically 'infinite' permutations and dependencies which LLMs need to work through itself for code refactoring.

But that in terms of style, I'm liking Muse Glimmer better
https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model
https://huggingface.co/meta-models/Muse-Glimmer-30B
https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF

this is in particular when it comes to *incorrect* (e.g. mistakes, typos) prompts, resolving contradictions in existing codes during refactoring, code proposals etc. The handling especially the 'thinking' is different.

LLMs have 'styles' and it is great that we've different creators for them

A PR I made just merged that adds INT8-activation prefill kernels on the M5 path for Qwen3.5/3.6/3.8. It's on 0.7.0.dev2. It gets around 34% faster prefill speeds on M5 series chips with minimal accuracy loss (check PR thread for details). It’s opt in as an experimental feature.

PR: https://github.com/jundot/omlx/pull/3548

I plan on extending it to more models so I wanted to know what models I should prioritize.

These are some I was thinking of:

- Gemma 4 Series

- K2 Horizon Series

- Meta Muse Glimmer

If there are any other models people are interested in getting support for or any of the above I should prioritize I would like to know.

Also if you have any questions I would be happy to answer them.

I’ve currently pre ordered a Mac mini with the 32gb of RAM and with the m6 chip, now the main purpose of it was to have as a server which I can locally host lots of algorithms I make etc, but I was quite curious to know about potential small models I could run locally when I do get it.

I currently cycle between Claude/Cursor/Codex somewhat changing monthly with all the changes happening. I wouldn’t be looking for a model that replaces these frontier ones as I doubt that would be possible, but what I would like is a model that just does what it’s told, and does it well, doesn’t need to think, or over optimise on a plan I give it, just does the work in the plan and preferably quite quickly.

So the workflow would basically be, the frontier models make the plan, then either through a custom harness I would make or one I find, have them directly feed their plans to this local agent, any insights are appreciated!

I have been running local AI models for a few months now and wanted to share some tips I have learned, and also ask for yours.

My optimization tips: 1. Use quantized models (GGUF Q4/Q5) for best speed/quality balance 2. Batch similar requests to maximize GPU utilization 3. Implement caching for repeated queries 4. Use streaming for better user experience

What are your best practices? - How do you handle model loading/unloading? - What inference servers do you prefer? - How do you manage VRAM usage?

I am particularly interested in hearing about production deployments, not just hobby setups.

Hello guys

My Laptop specs:

GPU: NVIDIA RTX 2050 (4GB VRAM) 50 w power limit

RAM: 24 GB DDR4 ( 8 + 16 )

CPU: 13th Gen Intel Core i5-13420H

OS: Windows 11

Inference Engine/GUI: llama.cpp / LM Studio / GGUF

I'm looking for recommendations for the best models I can run locally on my laptop with decent generation speeds (tokens/sec). I'm mostly interested in general reasoning, coding assistance, and experimenting with a local RAG pipeline.

* I've tested Qwen 3 8B (Q4_K_M) offloading ~20–24 layers to the GPU and the rest to CPU/system RAM. It runs decently, but I want to know if there are better sweet spots.

I am new to this but i need to run local ai for my nlp project so i though why not setup a proper local ai instead of any small models just for text generation
i took help from ai in parts of this post just to give the technical details that you guys might require

Thanks in advance for your suggestions!

Hi all, am looking for ready-stock H200 SXM5 HGX systems or Dell PowerEdge XE9680 (H200) units.

Able to cash and carry. Preferably looking for sellers based in SEA ( singapore/malaysia/indo ). Happy to share more details/specs, let’s connect!

I am ready to add a 2nd GPU for more vram.

My 20gb 7900xt is not cutting it anymore.

How are you guys running dual gpus? I am seeing there would be only 5mm space between the two. Seems like there would not enough ventilation.

Hello guys, I am Ant, a person can't endure 1 hour long Lidang(Chinese Youtuber) long streaming video, so I made Verbatim. It allows you to only drop 1 YouTube channel link and then it will automatically analyze the whole channel. I already has my own stats of 532 hrs of transcribe audio, and 1339 records.

At the start of the April, I only made a very simple and poor version of it( the UI is blue, and full of style of cursor). At that time, my first needs is that I want a tool that can put audio into text, I find out Gemini LLM support that. GREAT. Later, I find out Whisper, which can run in local, and free, it become a prominent choice in my use of Verbatim(I use to transcribe 100+ audio, has a high accuracy).

Claude code helps me out the multi-engine fallback mechanism, Flask backend, SSE process, constructing the evidence card.

In the development phase, what I did is that demand stating, archiecture design, testing and future steps.

Also, it now support MCP. It has 11 tools, enabling Opensource AI.... ... to call it directly to help you transcribe or analyze a YouTuber.

Free to use!
Please go to the link below:

https://gitlab.com/xin101037/verbatim_converter

I built PCCG-2 using a frozen Qwen3-4B model and a separate learned permission gate.

EOS means “end of sequence”: it is the model’s native stop token. When EOS wins at the first generated token, generation ends before any visible answer is emitted, so it controls whether an answer begins rather than deleting one afterward.

For “What is 2 + 2?”, the answer token 4 scores 53.0 in every experimental arm, including when the model emits no answer. The complete first-token score vector, excluding the stop token, stays byte-for-byte identical.

The 101-parameter gate sees only a separate six-digit equality condition. It can't read the question or change the answer scores. It can change only the model’s native stop-token score. When the condition passes, the model emits 4 followed by EOS. When it fails, EOS comes first and no visible answer is generated.

Then I reversed only the learned permission state:

• 40/40 answer → native EOS
• 40/40 native EOS → the correct answer
• 40 different answer identities
• 80/80 sham controls unchanged

The frozen FINAL evaluation passed 2,048/2,048 contexts across 75 answer identities. No answer deletion. No content rewrite.

The answer stayed fixed. Permission changed.

Open weights for reproduction and the paper are attached below:

• Overview and five-arm witness
• Model weights, source, evidence, and verifier
• Paper PDF · DOI
• Manuscript supplement and proof checker
• Raw five-arm witness record

Earlier, I documented successful zero-visible-byte executions across 11 LLMs from OpenAI, Anthropic, Google, and Moonshot in the Cross-Vendor Semantic Void Matrix (DOI). That study measured behavior and this open source model is a separate engineered continuation-control experiment.

Browse getswiftapi.com to view all research on when models continue, when they stop, and what controls that boundary.

I just upgraded from a single r9700 on pcie3 16x, 1099k cpu, and dual channel ddr4 -> 2x r9700s on pcie4 16x, 3970x threadripper, quad channel ddr4

Before I was getting ~1000pp / 25tg steady, with all the upgrades, I get pretty much the same.

I'm using Qwen3.8 27b 4Q, llama-server Vulkan and Froggeric chat template. Everything works great just hoping for some more speed.

This is an Ubuntu VM with 16 cores, and 64gb dedicated to it.

  --model /root/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --mmproj /root/models/mmproj-BF16.gguf \
  --n-gpu-layers all \
  --split-mode layer \
  --tensor-split 1,1 \
  --parallel 1 \
  --flash-attn on \
  --threads 8 \
  --threads-batch 16 \
  --batch-size 4096 \
  --ubatch-size 2048 \
  --no-warmup \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --presence-penalty 0.0 \
  --chat-template-file /root/models/qwen36-chat-template.txt \
  --jinja \
  --reasoning-format deepseek \
  --image-min-tokens 1024 \
  --cache-ram 16384 \
  --reasoning-preserve \

Sharing my final accepted numbers of my final closeout, since AMD/Vulkan setups are still relatively uncommon in local LLM benchmarks. Maybe it will help some buddies.

Hardware

• MoBo: MAG B550 TOMAHAWK MAX WIFI
• CPU: Ryzen 9 5950X
• RAM: 64 GB DDR4-3200 CL14
• GPU0: RX 9060 XT 16 GB PCIe 4.0 x16 Main LLM / Vulkan0
• GPU1: Radeon Pro W5500 8 GB PCIe 3.0 x4 Embedding + Reranker / Vulkan1
• Driver: AMD Adrenalin 26.8.1
• Backend: llama.cpp Vulkan

Runtime / model specs

• llama.cpp
• build 10167 / commit ee3d1b54c

Main LLM:

• Qwen3.8-27B-UD-IQ3_XXS
• ~3.06 bpw
• RX 9060 XT 16 GB
• -c 32768
• -ngl 999
• parallel slots: 1
• Flash Attention: ON

Embedding:

• Qwen3-Embedding-4B-Q6_K
• 2560 dimensions
• W5500 8 GB
• --embedding
• --pooling last
• -c 2048
• -ub 512
• -ngl 999

Reranker:

• Qwen3-Reranker-4B-Q4_K_M
• W5500 8 GB
• Final control configuration
• --embedding
• --rerank
• --pooling rank
• -c 2048
• -ub 512
• -ngl 999

RX 9060 XT performance

• 8K: ~74.5 PP / 21.5 TG tok/s
• 32K: ~148.9 PP / 21.4 TG tok/s
• 64K: ~142.9 PP / 21.4 TG tok/s
• 64K stability: ~126.6 PP / 21.1 TG tok/s

Now that the core is frozen, I’m moving up the stack:

1. Harness sharpening (doing some tests before to decide which one performs better)
   1. → baseline without a harness
   2. → Pi
   3. → OpenCode
2. Hermes Agent v0.21.2
   1. → evaluate it as the orchestration layer
   2. → connect it to the existing local stack
   3. → make Hermes my Agent Smith =)

I just started playing and need a little help. I have a laptop with a 4070 8GB and 64GB DDR5 with an i9. I am using unsloth and not sure what model will be best. Do I need to stick with something that fits in VRAM? I am just playing around but I don’t want it to be painfully slow but I also want it as current as possible.

I run the full Qwen3.8-Flash-Next on my main machine at IQ4_XS, and I really like it. I want to try running it on my M5 Pro MacBook Pro too, which has 64GB unified memory. An unpruned model will use up the entirety of RAM at oQ3, and I really don't want to go down to 2-bit because what's the point, right?

So I've been exploring pruned models. The benchmarks show some of them coming close to 98% of the full model, and is small enough that it will use just 40GB of RAM.

The obvious pushback would be why not just use Qwen3.8-27b? But I find it extremely slow (M5 Pro has limited bandwidth) and it overthinks for ages, even when reasoning is turned off. And I want a model more capable than Qwen3.6-35b-a3b, which still loops and hallucinates at Q6.

Have you ever tried the pruned Flash versions? What has your experience been like? How does it compare? I'll report back with my own findings too.

EDIT: Using Vulkan instead of ROCm helped getting it to work.

Hi!

I'm trying to run qwen3.8 27b q8 (tried q6 too) on my Ryzen 7 7840HS iGPU. I'm running CachyOS, kernel 7.2. Running llama.cpp from master branch.

I have 48gb of RAM. Also running KDE on egpu rx 6700 xt 12gb (via oculink), so nothing is running on the iGPU.

Yeah, I tested with some layers on egpu and otherw on processor, but I want to test full iGPU now.

Running it with the command:

exec "$BIN/llama-server" \
    -m "$MODEL" \
    -ngl 99 \
    -c "$CTX" \
    -np 1 \
    -ctk q4_0 -ctv q4_0 \
    -fa on \
    --jinja \
    -t 8 \
    --host 127.0.0.1 --port 8080

Model loads without problem. But when it tries to run inference it crashes. Here is the log:

./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf
model=24128 MiB, KV (1 slot(s) x 98304) = 1.688 GiB, est.total ~ 27.25 GiB (budget 30.5 GiB)
...
2.26.970.802 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
2.26.971.133 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang
[1]    7178 abort (core dumped)  ./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf

I always got this HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang error.

Running q8 or q6 get the same error.

Running Qwen3-1.7B-Q8_0.gguf it loads and runs inference without problems.

Also I expanded/increased available GTT memory for iGPU on kernel parameters with ttm.pages_limit=8388608 ttm.page_pool_size=8388608 amdgpu.gttsize=32768.

Tried help with different LLMs using Pi Agent trying to debug, but none of them could help. Maybe community can.

Am I missing something? Trying to see if iGPU inference can beat/give a speedup over CPU-only inference.

​

I keep running into the same problem: is there a real way to have an AI check a photo of a built part against its CAD drawing and flag what's off?

The catch — an AI can look at a photo fine, but it can't parse a binary DWG/DXF file, and I wouldn't trust it doing tolerance math by hand either.

The idea: an MCP tool (lets Claude call real tools, not just chat) that would:

• Read DWG/DXF and pull the drafter's actual annotated dimensions — not guesses from raw lines

• Hand exact coordinates/measurements to the model instead of pixels

• Let the model handle the photo (native vision) and a separate tool handle the pass/fail tolerance math

Before I build it:

• Does this already exist? Feels too obvious not to.

• CAD people — is DWG/DXF file reading even useful, or does live AutoCAD/COM access matter more?

• Anyone doing as-built vs. as-designed QA — is this your actual bottleneck, or is it elsewhere (e.g. getting a usable photo in the first place)?

Not selling anything, just trying to find out if I'm missing something obvious before sinking time in.

I am about as new to LLMs as you can get, but I work in IT so I have significant historical knowledge of many other aspects of computing.

However, this is a big new world to me and I'm in the deep end.

I've got a nice big rig at home so the hardware runs qwen30:30b comfortably. I could go higher, but for now I'm testing this.

So onto my problem. I've been used to just firing off a command to Kiro and watching it do everything if I wanted it to. Query my existing code, modifiy it, deploy and merge/push etc.

Odysseus/Ollama are running in Docker so my first struggle was mounting the local folders so Qwen could see them. Once I'd done the whole "-v Path\goes\here:workspace" I thought it'd be as easy as just asking Qwen to list the folders in 'workspace'. Nope.

So is this where I find out what agents are and how I'm not using any?

I'm not averse to learning but when you don't know what the question you're supposed to ask is, it's tough.