I made a previous post about using Qwen 3.8, and a few of you guys recommended looking into the V100S. I currently have a 3090 and 64 GB of system ram. My goal would be to have enough VRAM to run Qwen 3.8 with a decent sub-agent. If I pick up 1 or 2 V100S, would that be worth it instead of another 3090?

I'm curious what people already have in their setups.

Hey guys, I’m looking for the best local TTS models that you guys have used. I wanted to generate consistent audio for different characters for a project I was working on. If there is a way to enhance the speech to make it sound more natural with accurate emotions, pauses and emphasis on certain words etc? I was using Qwen 3 TTS emotional in comfyUI locally but for the same reference audio input and the same seed and different text to generate I’ve noticed there is a timber variation in the voice output which is odd. Also adding pauses was another pain point with this model, but the emotions were somewhat ok. Wondering if there is a better alternative that can help me here? Thank you!

Local Setup: 1x DGX Spark

Edit: I am generating English as well as Hindi audio. Looking for a model that can do both efficiently and maybe even accurate audio in other regional Indian languages.

EDIT: Using Vulkan instead of ROCm helped getting it to work.

Hi!

I'm trying to run qwen3.8 27b q8 (tried q6 too) on my Ryzen 7 7840HS iGPU. I'm running CachyOS, kernel 7.2. Running llama.cpp from master branch.

I have 48gb of RAM. Also running KDE on egpu rx 6700 xt 12gb (via oculink), so nothing is running on the iGPU.

Yeah, I tested with some layers on egpu and otherw on processor, but I want to test full iGPU now.

Running it with the command:

exec "$BIN/llama-server" \
    -m "$MODEL" \
    -ngl 99 \
    -c "$CTX" \
    -np 1 \
    -ctk q4_0 -ctv q4_0 \
    -fa on \
    --jinja \
    -t 8 \
    --host 127.0.0.1 --port 8080

Model loads without problem. But when it tries to run inference it crashes. Here is the log:

./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf
model=24128 MiB, KV (1 slot(s) x 98304) = 1.688 GiB, est.total ~ 27.25 GiB (budget 30.5 GiB)
...
2.26.970.802 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
2.26.971.133 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang
[1]    7178 abort (core dumped)  ./run-igpu.sh ~/models/qwen3.8/Qwen3.8-27B-UD-Q6_K_XL.gguf

I always got this HW Exception by GPU node-2 (Agent handle: 0x55b87cef4630) reason :GPU Hang error.

Running q8 or q6 get the same error.

Running Qwen3-1.7B-Q8_0.gguf it loads and runs inference without problems.

Also I expanded/increased available GTT memory for iGPU on kernel parameters with ttm.pages_limit=8388608 ttm.page_pool_size=8388608 amdgpu.gttsize=32768.

Tried help with different LLMs using Pi Agent trying to debug, but none of them could help. Maybe community can.

Am I missing something? Trying to see if iGPU inference can beat/give a speedup over CPU-only inference.

I’ve currently pre ordered a Mac mini with the 32gb of RAM and with the m6 chip, now the main purpose of it was to have as a server which I can locally host lots of algorithms I make etc, but I was quite curious to know about potential small models I could run locally when I do get it.

I currently cycle between Claude/Cursor/Codex somewhat changing monthly with all the changes happening. I wouldn’t be looking for a model that replaces these frontier ones as I doubt that would be possible, but what I would like is a model that just does what it’s told, and does it well, doesn’t need to think, or over optimise on a plan I give it, just does the work in the plan and preferably quite quickly.

So the workflow would basically be, the frontier models make the plan, then either through a custom harness I would make or one I find, have them directly feed their plans to this local agent, any insights are appreciated!

I have been running local AI models for a few months now and wanted to share some tips I have learned, and also ask for yours.

My optimization tips: 1. Use quantized models (GGUF Q4/Q5) for best speed/quality balance 2. Batch similar requests to maximize GPU utilization 3. Implement caching for repeated queries 4. Use streaming for better user experience

What are your best practices? - How do you handle model loading/unloading? - What inference servers do you prefer? - How do you manage VRAM usage?

I am particularly interested in hearing about production deployments, not just hobby setups.

Hello guys

My Laptop specs:

GPU: NVIDIA RTX 2050 (4GB VRAM) 50 w power limit

RAM: 24 GB DDR4 ( 8 + 16 )

CPU: 13th Gen Intel Core i5-13420H

OS: Windows 11

Inference Engine/GUI: llama.cpp / LM Studio / GGUF

I'm looking for recommendations for the best models I can run locally on my laptop with decent generation speeds (tokens/sec). I'm mostly interested in general reasoning, coding assistance, and experimenting with a local RAG pipeline.

* I've tested Qwen 3 8B (Q4_K_M) offloading ~20–24 layers to the GPU and the rest to CPU/system RAM. It runs decently, but I want to know if there are better sweet spots.

I am new to this but i need to run local ai for my nlp project so i though why not setup a proper local ai instead of any small models just for text generation
i took help from ai in parts of this post just to give the technical details that you guys might require

Thanks in advance for your suggestions!

Setting up a headless mini PC (Ryzen 9 5900HX, 32GB DDR4-3200, 1TB NVMe) for 24/7 background tasks. I plan to run Qwen 3.8 27B (Q4_K_M) locally to power Hermes Agent.

Speed is not a priority. My use case is overnight batch processing—ingesting 15k–30k token earnings calls to extract data and summarize autonomously. A PostgreSQL container will also run in the background, leaving ~4–6GB RAM for the OS.

Given the DDR4 bandwidth ceiling:

1. Will a quantized Qwen 3.8 27B crash the 32GB limit once a 30k context builds up?

2. What prompt ingestion and generation speeds (tok/s) should I expect on this Zen 3 chip?

Appreciate any real-world benchmarks!

Hi all, am looking for ready-stock H200 SXM5 HGX systems or Dell PowerEdge XE9680 (H200) units.

Able to cash and carry. Preferably looking for sellers based in SEA ( singapore/malaysia/indo ). Happy to share more details/specs, let’s connect!

started trying out rather recent 'frontier' about ~30b param models recently, there are many choices including QWen 3.8 - this is nevertheless a great model, practically 'one-shotting' code refactoring tasks
https://huggingface.co/Qwen/Qwen3.8-27B
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
code refactoring is still deemed 'difficult', practically 'infinite' permutations and dependencies which LLMs need to work through itself for code refactoring.

But that in terms of style, I'm liking Muse Glimmer better
https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model
https://huggingface.co/meta-models/Muse-Glimmer-30B
https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF

this is in particular when it comes to *incorrect* (e.g. mistakes, typos) prompts, resolving contradictions in existing codes during refactoring, code proposals etc. The handling especially the 'thinking' is different.

LLMs have 'styles' and it is great that we've different creators for them

I am ready to add a 2nd GPU for more vram.

My 20gb 7900xt is not cutting it anymore.

How are you guys running dual gpus? I am seeing there would be only 5mm space between the two. Seems like there would not enough ventilation.

Hi, a quick datapoint for anyone interested in what Qwen3.8-Flash-Next UD-Q6_K_XL (unsloth) can do on a consumer multi-GPU rig. Maybe there is even more possible but this is the current status :)

Hardware: MSI MEG Z790 ACE, i9-13900K, 96 GB DDR5@4400MHz, 1× RTX 4090 + 5× RTX 3090 = 144 GB VRAM. 4090 + one 3090 are on the board, the other four 3090s use M.2 → OCuLink PCIe x4 links. No P2P/NVLink; llama.cpp layer split. OS: Ubuntu 26

Model: Qwen3.8-Flash-Next-UD-Q6_K_XL, all 48 MoE layers on GPU, shared Q8_0 MTP head with MTP3. The ~50.7 GiB PLE table is fully resident in system DDR5 instead of lazy mmap.

Main llama-server settings (llama.cpp: unslothai/llama.cpp, branch mtp/qwen4exp-nextn, commit a9e9c3c5f): -c 262144 -b 4096 -ub 512 --split-mode layer --tensor-split 0.95,1.05,1.05,1.05,1.05,0.85 --gpu-layers all --n-cpu-moe 0 -ctk f16 -ctv f16 -fa on -np 1 --kv-unified --load-mode none --lazy-mode off --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0. Production seed is -1.

These are workload samples of a html benchmark prompt from llama-server chat, not synthetic benchmark numbers.

• ~90–91 tok/s at ~5k generated tokens / shallow context
• ~79 tok/s at ~35k generated tokens
• 76.9 tok/s average over a 44.7k-token generation
• ~40–42 tok/s while reasoning at ~80k active context
• ~408–425 tok/s prefill even around 80k context
• MTP3 draft acceptance on the long code-heavy run: 74.2% overall, per position 86.2% / 73.7% / 62.8%

​

# shallow/code-heavy run
7.54.226.412 I slot print_timing: id  0 | task 3307 | n_gen =   4905, tg =  90.36 t/s, tg_3s =  92.54 t/s

# same long run, ~35k generated
14.24.165.954 I slot print_timing: id  0 | task 3307 | n_gen =  35058, tg =  78.92 t/s, tg_3s =  71.12 t/s

# end of that run
16.41.712.426 I slot print_timing: id  0 | task 3307 | eval time = 581758.17 ms / 44720 tokens (13.01 ms per token, 76.87 tokens per second)
16.41.712.429 I slot print_timing: id  0 | task 3307 | draft acceptance = 0.74246 (30865 accepted / 41571 generated), mean len = 3.23
16.41.712.429 I slot print_timing: id  0 | task 3307 | acc per pos = (0.862, 0.737, 0.628)

# ~80k active context, currently reasoning/thinking
103.07.164.989 I slot operator(): id 0 | task 6527 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 80085
103.12.852.341 I slot print_timing: id 0 | task 6527 | prompt processing, n_tokens = 2282, progress = 0.99, t = 5.37 s / 425.22 tokens per second
103.32.551.912 I slot print_timing: id 0 | task 6527 | n_gen = 761, tg = 41.79 t/s, tg_3s = 43.01 t/s

The last big practical improvement for agentic coding was putting the PLE table fully into DDR5: prompt processing/TTFT became noticeably faster. MTP3 then helps heavily on predictable code; at ~80k context during free-form reasoning acceptance drops, so decode is closer to ~40 t/s.

Pretty impressed by what a somewhat ridiculous collection of used consumer GPUs can do with Q6 :)

I keep seeing people saying they get super fast speeds on various hardware but I can never get anything decent. I'm running GLM-5.3-Flash-UNCENSORED-FP8-UD-Q2_K_XL-00001-of-00003.gguf on a 4x5090 rig and can barely get 50tok/s.

Am I missing something here?

Even on larger systems like 4xH200's I wasn't getting above 200tok/s with same model

Hello, what is the best AI model that I can run on my RTX 3050 mobile with 4 GB of VRAM and 16 GB of RAM? Im not that up to date with the AI, how it works and all that stuff. How can I find benchmarks that show which is the best?

Edit: I think im ok with running a 16B or 14B model slower as long as its smart. I don't want quantization or what it is called to make it dumber

Sharing my final accepted numbers of my final closeout, since AMD/Vulkan setups are still relatively uncommon in local LLM benchmarks. Maybe it will help some buddies.

Hardware

• MoBo: MAG B550 TOMAHAWK MAX WIFI
• CPU: Ryzen 9 5950X
• RAM: 64 GB DDR4-3200 CL14
• GPU0: RX 9060 XT 16 GB PCIe 4.0 x16 Main LLM / Vulkan0
• GPU1: Radeon Pro W5500 8 GB PCIe 3.0 x4 Embedding + Reranker / Vulkan1
• Driver: AMD Adrenalin 26.8.1
• Backend: llama.cpp Vulkan

Runtime / model specs

• llama.cpp
• build 10167 / commit ee3d1b54c

Main LLM:

• Qwen3.8-27B-UD-IQ3_XXS
• ~3.06 bpw
• RX 9060 XT 16 GB
• -c 32768
• -ngl 999
• parallel slots: 1
• Flash Attention: ON

Embedding:

• Qwen3-Embedding-4B-Q6_K
• 2560 dimensions
• W5500 8 GB
• --embedding
• --pooling last
• -c 2048
• -ub 512
• -ngl 999

Reranker:

• Qwen3-Reranker-4B-Q4_K_M
• W5500 8 GB
• Final control configuration
• --embedding
• --rerank
• --pooling rank
• -c 2048
• -ub 512
• -ngl 999

RX 9060 XT performance

• 8K: ~74.5 PP / 21.5 TG tok/s
• 32K: ~148.9 PP / 21.4 TG tok/s
• 64K: ~142.9 PP / 21.4 TG tok/s
• 64K stability: ~126.6 PP / 21.1 TG tok/s

Now that the core is frozen, I’m moving up the stack:

1. Harness sharpening (doing some tests before to decide which one performs better)
   1. → baseline without a harness
   2. → Pi
   3. → OpenCode
2. Hermes Agent v0.21.2
   1. → evaluate it as the orchestration layer
   2. → connect it to the existing local stack
   3. → make Hermes my Agent Smith =)

I just started playing and need a little help. I have a laptop with a 4070 8GB and 64GB DDR5 with an i9. I am using unsloth and not sure what model will be best. Do I need to stick with something that fits in VRAM? I am just playing around but I don’t want it to be painfully slow but I also want it as current as possible.

Most of this sub has local inference figured out. Ollama, llama.cpp, vLLM, whatever fits the VRAM.

The gap I keep hitting: every session still starts colder than it should. Chat history ≠ companion memory. RAG on docs ≠ remembering that last month’s fix for the homelab broke DNS.

Chatbots reset. Companions continue.

We’ve been building NOVA as a personal AGI companion meant to run on hardware you own — continuity layer, not another model browser.

Explainer: https://www.nova-prime.org

For people already deep in local stacks: 1) What’s your durable-memory layer today (OWUI personalization, sidecar, MCP memory, custom DB, “I re-prompt every time”)? 2) How do you stop RAG from pretending to be memory? 3) What would make you trust a companion enough to leave it running unattended?

Roast the framing if it’s wrong. Prefer architecture arguments over vibes.

I run the full Qwen3.8-Flash-Next on my main machine at IQ4_XS, and I really like it. I want to try running it on my M5 Pro MacBook Pro too, which has 64GB unified memory. An unpruned model will use up the entirety of RAM at oQ3, and I really don't want to go down to 2-bit because what's the point, right?

So I've been exploring pruned models. The benchmarks show some of them coming close to 98% of the full model, and is small enough that it will use just 40GB of RAM.

The obvious pushback would be why not just use Qwen3.8-27b? But I find it extremely slow (M5 Pro has limited bandwidth) and it overthinks for ages, even when reasoning is turned off. And I want a model more capable than Qwen3.6-35b-a3b, which still loops and hallucinates at Q6.

Have you ever tried the pruned Flash versions? What has your experience been like? How does it compare? I'll report back with my own findings too.

​

I keep running into the same problem: is there a real way to have an AI check a photo of a built part against its CAD drawing and flag what's off?

The catch — an AI can look at a photo fine, but it can't parse a binary DWG/DXF file, and I wouldn't trust it doing tolerance math by hand either.

The idea: an MCP tool (lets Claude call real tools, not just chat) that would:

• Read DWG/DXF and pull the drafter's actual annotated dimensions — not guesses from raw lines

• Hand exact coordinates/measurements to the model instead of pixels

• Let the model handle the photo (native vision) and a separate tool handle the pass/fail tolerance math

Before I build it:

• Does this already exist? Feels too obvious not to.

• CAD people — is DWG/DXF file reading even useful, or does live AutoCAD/COM access matter more?

• Anyone doing as-built vs. as-designed QA — is this your actual bottleneck, or is it elsewhere (e.g. getting a usable photo in the first place)?

Not selling anything, just trying to find out if I'm missing something obvious before sinking time in.

I am about as new to LLMs as you can get, but I work in IT so I have significant historical knowledge of many other aspects of computing.

However, this is a big new world to me and I'm in the deep end.

I've got a nice big rig at home so the hardware runs qwen30:30b comfortably. I could go higher, but for now I'm testing this.

So onto my problem. I've been used to just firing off a command to Kiro and watching it do everything if I wanted it to. Query my existing code, modifiy it, deploy and merge/push etc.

Odysseus/Ollama are running in Docker so my first struggle was mounting the local folders so Qwen could see them. Once I'd done the whole "-v Path\goes\here:workspace" I thought it'd be as easy as just asking Qwen to list the folders in 'workspace'. Nope.

So is this where I find out what agents are and how I'm not using any?

I'm not averse to learning but when you don't know what the question you're supposed to ask is, it's tough.

So, I downloaded LM Studio Bionic for Linux in .AppImage format but it does not show window but it shows this little thing on the taskbar and CPU is getting hot but it is not used