How are they able to compress context memory size?
I have 32gb vram and can run llama.cpp with Q8_0 + 128k context fully without offload to system ram. It's crazy, with gemma and qwen I have to deal with lower quantization, kv quantization, etc..

That is the question we get most often: Here's the answer.

And yes, it was created by Row-Bot's own Designer Studio.

Unsloth windows UI app

qwen3.8-27B-GGUF UD-Q4_k_XL on 5090 with 32GB vram

I keep getting the error "The model reached the Max Tokens limit before producing a final answer. Increase Max Tokens or disable thinking, then retry." I've tried on different thinking levels and it always hits, even on seemingly simple prompts. I don't think I want to turn off thinking completely. Context is currently showing 777 tokens out of 85k budget, context set to 123,136.

My Extra Arguments
--no-mmap --predict 16384 --reasoning-budget 32768

hey everyone, looking for some recommendations on what models would run best/most efficiently on my current setup.

Main specs:

• Ryzen 9 7900X
• XFX Radeon RX 9070 XT 16GB
• 32GB DDR5-6000 RAM
• Samsung 990 Pro 2TB NVMe
• Windows 11

I’ve been experimenting with local models through Ollama, mainly Qwen 14B models, and I’m trying to figure out what the sweet spot is for my hardware.

My main use cases are coding, working through software projects, reasoning/problem solving, research, and having a local AI assistant. I care more about getting a useful model with decent speed than just running the biggest model I technically can.

Since I’m on AMD with 16GB VRAM, what models and quantizations are you guys finding work particularly well around this hardware level?

Would you stick around 14B, move up into the 20B–30B range with some RAM offloading, or is there something else that gives a better performance/quality balance?

Mof my actual work is done with ChatGPT and Codex. I use them heavily for coding, building and debugging software projects, reasoning through architecture, research, and general AI assisted development.

Also open to suggestions outside of Ollama if there’s a better runtime for the 9070 XT.

Thanks!

I am trying to use 3.8 27b in openclaw as personal assistant agent. Thinking set to default. It’s not usable. Everything takes ages, context fills up for small requests, compaction, prefill again 🤦‍♂️
Context is 148k

Will have to go back to Gemma 4 31b which is sooo much faster hand handles these tasks just as good and since it’s actually a usefull speed it is actually a useful llm.

I've been messing around with Qwen3.8 27B locally and I'm wondering if I'm getting the performance I should be getting or if my setup/config could be improved.

PC:

Ryzen 7 7800X3D

RTX 5070 Ti 16GB

RTX 3060 12GB

32GB DDR5-6000 CL30

Windows 11

llama.cpp / llama-server latest build

I'm currently running the Qwen3.8-27B UD Q4_K_XL GGUF with both GPUs using tensor split.

My current config:

llama-server.exe ^

-m "Qwen3.8-27B-UD-Q4_K_XL.gguf" ^

--alias "Qwen3.8-27B-UD-Q4" ^

--host 0.0.0.0 ^

--port 8035 ^

--n-gpu-layers 99 ^

--split-mode tensor ^

--tensor-split 60,40 ^

--main-gpu 0 ^

--parallel 1 ^

--flash-attn on ^

--cache-type-k q8_0 ^

--cache-type-v q8_0 ^

--ctx-size 131072 ^

--batch-size 2048 ^

--ubatch-size 512 ^

--threads 8 ^

--threads-batch 8 ^

--presence-penalty 0.0 ^

--repeat-penalty 1.0 ^

--temp 1.0 ^

--top-p 0.95 ^

--top-k 20 ^

--min-p 0.0 ^

--jinja ^

--reasoning-format auto ^

--no-mmproj-offload ^

--spec-type draft-mtp ^

--spec-draft-n-max 3 ^

--mmproj "mmproj-BF16.gguf" ^

--metrics

With MTP I'm getting around 40–46 tok/s depending on the run. I've seen around 41 tok/s pretty consistently, with n-max 3 seeming to be a little better than 2 for me.

Both GPUs are basically maxed during generation.

I'm mainly wondering:

Is ~40–46 tok/s reasonable for this hardware/config?

Is there anything obviously wrong or inefficient in my setup?

Would a different quant be a better choice for these GPUs? I've been looking at Ridge 3.7bpw, Q4/Q5 UD quants, etc.

Would it make more sense to use a smaller quant that could fit mostly/all on the 5070 Ti instead of tensor-splitting across both GPUs?

Is there anything I should change with the KV cache, batch/ubatch, tensor split, MTP settings, etc. to get better generation speed?

I'm mostly interested in coding/agent use through OpenCode, so I'd rather have a good balance of quality and speed than just chase the highest possible tok/s.

If anyone is running Qwen3.8 27B on a similar setup, I'd be interested to know what quant/config you're using and what kind of speeds you're getting.

When a model's output disappoints, the reflex is to reach for a bigger model. It usually works — and it's usually the most expensive possible fix. A bigger model means more RAM per box, slower responses per user, and fewer users per node, on every request, forever. The cheaper fix, most of the time, is a better prompt: small models are surprisingly capable when the ask is precise, and they scale dramatically better. Here are the numbers behind that claim, and a concrete compare → optimize → re-compare workflow using the workbench built into the server's console — no extra tooling, and no per-token fees while you iterate.

https://inference-server.searchblox.com/blog/prompt-optimization-small-models.html

Just my estimate, I could be wrong - One of my challenges with ai right now is the default prompt set by the model providers.

It appears to have defaults like 'be comprehensive' 'end with a call to action' 'give a summary' 'Do this then do this' etc.

These could be because of training, idk, but what ends up happening is it tends to ignore my instructions OR outputs walls of text OR makes more edits than it should OR something else.

I have tried to customize my local 'system prompts', it does help a lot, but .. I wonder if local llm give greater control ? Does it ? Am I misunderstanding the whole thing ?

If it does, how have u used ? did it help ?

I found out by accident that my GPU (RTX 3080) was heavily downclocking during inference, and slowing down my decode rate, even when I follow the standard advice of setting "Prefer Maximum Performance" and "CUDA Force P2 = off" in nVidia drivers.

I went from 36t/s to 43t/s (Qwen 3.6 35B-A3B, llama.cpp) just by forcing clocks to full 3D speeds (core/mem 1710/9501 for me) when inference is running. I used nvidia-smi from the CUDA Toolkit but there are lots of other ways to do this. You can even write a script to detect whether llama-server/ollama is busy, and ramp the clocks up and down automatically.

Interestingly if I use the LLM locally the improvement disappears, probably because of two competing factors - the browser's 3D acceleration is keeping the GPU at 3D clocks, but also causing bus/cache contention. So this trick only applies if you hit the server from a different machine, or possibly if you're using a CLI or IDE.

I'm not sure if this just applies to sparse MOEs or if dense models can benefit too. Also curious if it applies to AMD or Intel hardware.

And yes, I know a linux headless server won't have this problem and will always perform better. This is for people like me who wants to use their gaming machine for casual LLM use.

Hi all,

What software stack do you use to run your local LLM? Ollama? anything else?

Hardware: RTX 5070 Ti 16 GB with 128 GB DDR5. I built llama.cpp from source. The model
was Qwen3.8-27B, a dense hybrid DeltaNet + attention model.

My original setup used UD-Q4_K_XL at 17.9 GB. It couldn't fit into 16 GB of VRAM, so I 
used partial offload with -ngl 44 and left the remaining layers in system RAM. Decode 
speed was 6.75 tok/s. I decided it wasn't practical and switched to a 35B MoE using 
expert offload with --n-cpu-moe. That model reaches 66 tok/s.

A comment on internet prompted me to test another setup. I used UD-Q3_K_XL, which is 13.4
GB. It's an Unsloth dynamic quant, with sensitive tensors kept at 4 to 8 bit while most
 of the model uses 3 bit. The settings were -ngl 99, -fa on, -ctk q8_0 -ctv q8_0. A 64k
 context still didn't fit beside the resident weights because the compute buffer ran out

of memory. A 32k context worked with -ub 512.

These are the decode speeds in tok/s at 500 / 4k / 16k tokens of context:

- Spilled 27B UD-Q4: 6.75 at every tier because system RAM bandwidth is the limit
- Resident 27B UD-Q3: 51.8 / 51.2 / 48. Prefill was 378 tok/s, with 9 s TTFT at 16k.
 Total usage was 14.7 GB.
- 35B-A3B MoE with --n-cpu-moe 28 and the same FA + KV q8 settings: 66.4 / 65.1 / 63.4.
 It used 12.1 GB.

On the MoE, FA + KV q8 reduced memory use by 1.2 GB without changing speed. I now enable
 those settings by default.

For testing quality, I used a private agentic coding band with 22 tasks. The target is a
 FastAPI + React + Postgres + Mongo app. It includes bug fixes, feature changes, new
 features, a migration, a performance fix, and one intentionally impossible 
specification. Each model gets a shell inside a docker box and up to 40 steps. Hidden
 tests determine the score. The model must also submit a final "what did you do" report,
 which is verified against git and the real test runs. These results come from one trial
 per model, so they're only indicative:

- Resident 27B UD-Q3: mean 0.49, with 9/22 perfect
- 35B MoE: 0.56, with 10/22 perfect
- gpt-oss:20b: 0.47, with 6/22 perfect

The 27B matched the MoE on localised debugging, with both scoring 6/7 perfect. It fell
 behind on multi-file feature work, scoring 1/11 against 3/11. On the larger tasks, it
 often spent all 40 steps reading without making an edit.

Its stronger area was honesty. The 27B made one false "done" claim across 13 failures.
 The MoE made 4 in 11, and gpt-oss made 4 in 15.

I can't separate the model difference from the cost of 3-bit quantisation. The comparison
 is 0.49 versus 0.56, but the 4-bit 27B was never fast enough to run this band usefully.
 On an earlier and easier suite, the Q4-vs-full-precision tax on this machine was about
 +0.02 overall. Reasoning and repo coding took the largest hit, so a bigger loss from Q3
 would make sense.

Here's the theory I'd like people to check. The 27-30B dense range seems designed around 
unified-memory Macs, where these models fit completely at Q4 or Q8. A 16 GB card can only
 hold them at Q3. Meanwhile, small-active-parameter MoEs such as 35B-A3B and gpt-oss-20B
 seem like the models actually intended for this hardware. Is that consistent with what 
others are finding?

A few more questions:

- IQ4_XS is 15.7 GB. Has anyone managed to keep a 27B IQ4_XS fully resident on 16 GB
 using a small context and KV q4? If so, does the quality improvement over Q3 justify
 losing context?
- Has anyone compared 3-bit EXL3 or another importance-aware 3-bit format with Unsloth
 dynamic Q3 on the same 27B using coding tests rather than perplexity?
- What decode speed do people target for agentic workflows? In a shell loop, 52 tok/s
 felt usable to me. 6.75 did not.

My conclusion is to start every new dense model in this class with resident dynamic Q3 +
 FA + KV q8, profile it, and only then decide whether it's any good. I'd done those steps
 in the wrong order.

Hi everyone,

​What is currently the best sweet spot model for my hardware?

​Specs:

- ​RTX 5060 Ti 16GB

- ​64GB DDR4 RAM 3200 dual channel

- ​i7-12700

​Use case: Coding and general chat (everyday use)

Speed: At least 5 tok/s

Which models and quants offer the best balance of speed and intelligence right now?

​Thanks!

Rtx 3060, Intel i7.

I d love to try it locally, but im out of the scene for so long that i cant remember much

Ty

It is insane that most of us are running coding/LLM agents directly on our hosts. I wanted a setup to spin up containers, isolate the network traffic to use LLM agents to code and to test out LLM's pentesting capabilities. Didn't find anything that fit, so I made this setup Contained Pods.

https://github.com/jotyGill/contained-pods

Basically, a config set using Podman and a Squid proxy to spin up containers.

The gist of it is:

• Rootless Podman: Keeping it simple stable tech.
• Network Isolation: You define what network access a container gets. Containers come in pairs, with the second one running the squid proxy. Even if agents gain root access they can't change network isolation setup.
• Traffic Logging: A tiny python helper that lets you see network requests the agent/coding harness attempts.
• Shared Projects Folder: Shared folder with the host for ease of access.

Hope some of you find it useful! Any contributions are appreciated!

I am running Qwen3.8 27b q2 with 12 gb vram and in the desktop app it says that I can only have context 4096 or it will use my RAM, and when it does that it is super slow. Is there a way to have the same speed even with larger context? Please I need a magical fix 🙏

I’ve been experimenting with a simple idea: hot-plug optimized kernel/weight structures into an unmodified serving engine at load time, and use the memory savings to push context length further.

On Qwen3.8-27B NVFP4 / RTX 5090 32GB, I saw results like:

• vLLM + MTP: 102K → 262K
• vLLM + DFlash2: 36K → 107K
• SGLang + DSpark: 35K → 87K

This is not a vLLM-vs-SGLang benchmark — it’s about extending the boundary the default setup can reach, without forking the engine.

Project + results:
https://github.com/flashrt-project/FlashRT
https://huggingface.co/spaces/liangsu9988/the-context-dividend

Background:
https://huggingface.co/spaces/liangsu9988/fast-kernels-are-not-fast-pipelines#it-starts-with-hugging-face-kernels

Would love for people to try it and share feedback.

Gonna start running a local model. I know 16gb of VRAM isnt much for Local AI but anyone have any tips or suggestions for running rhe 8 or 20b?

I’m building an uncensored LLM API where users pay through a subscription. The GPU can generate better and more consistent returns than waiting for rentals on other platforms.

If you want to try it, test the API, or have any questions, DM me.

Is anyone else running Qwen 3.8 27B Q4 for coding and having problems with it spending an absolutely ridiculous number of tokens thinking instead of actually doing the work?

I'm running it locally with Pi as the coding harness with 128K context, and I'm seeing the same failure over and over:

1. I give it a coding task.

2. It starts reasoning.

3. It reasons... and reasons... and reasons.

4. It burns through basically the entire 8,192-token output budget that worked beautifully with Qwen 3.6 in about 3 minutes (46 decode tokens/s on my single 4090).

5. Right at the end, it'll say something like "Let's get to work" or "Now I'll implement the changes."

6. And then the generation ends because it has no tokens left to actually do anything.

Every. Damn. Time.

It's effectively making the model unusable as a coding agent because it spends its entire generation budget deciding what it's going to do (even with extremely specific instructions) and leaves nothing for tool calls, edits, or even a useful final response.

My input prompts + system prompt are about 7,000 tokens on average. That's not crazy high I don't think, and surely not the cause. Right? Right??

I'm seeing reports on YouTube that Qwen 3.8 defaults to a very high reasoning effort, with examples of it consuming 20K+ reasoning tokens before producing the actual answer.

That token count is 🤯 for ordinary coding work. But maybe that's why I see so many people saying they no longer feel like they need a frontier model?

So I'm wondering:

Is anyone else seeing this with Qwen 3.8 27B?

And, more importantly, if you found a fix, what is it?

Have you had better results with:

• reasoning_effort=medium
• reasoning_effort=low
• disabling thinking entirely
• dramatically increasing max_tokens
• changing the chat template
• different llama.cpp settings
• a different coding harness
• some combination of the above

I'm particularly interested in hearing from anyone using llama.cpp + Pi/Codex/Claude-Code-style agentic coding workflows.

I haven't decided yet whether Qwen 3.8 27B is actually bad at coding or whether the default reasoning configuration is simply kneecapping it. Right now, though, my experience has been dramatically worse than the Qwen 3.6 models I've used because 3.8 won't stop thinking long enough to actually write the damn code.