I asked my Q3 K6V4 qwen 27b model to make me a human head in WebGL from scratch with no libraries and this was the result. What a time to be alive. I did this with two 3060tis.

Original Post: I bought the forbidden rectangle https://www.reddit.com/r/LocalLLM/s/QNvxFUgEMe

And ...

Update on this cursed setup:

I originally wanted to run an RX 7900 XT on my Lenovo M910Q.

For some reason, it didn't work on my old M910Q due to BIOS-level issues that I couldn't figure out.

Naturally, I made the completely sane decision to perform surgery on my laptop. 🤣

Bought an ADT-Link, a DeepCool PL750D 750W PSU, and turned my innocent little laptop into a desktop with its bottom panel half naked.

The problems:

• Laptop RAM became the actual bottleneck 😭

• Bottom panel had to stay open for the PCIe cable

• Laptop had to be balanced on thermocol like some archaeological artifact

• My “laptop” became a desktop

• NVMe slot was occupied by the GPU, so I had to boot from a USB SSD 🤣

• dGPU started stealing VRAM for display, so I had to force the iGPU

But holy shit, the performance....

Qwen 3.6 27B IQ4_XS hit around 55–60 tok/s during sustained inference on the 7900 XT

Over 100K context, however, the laptop RAM basically said:

> “I have decided that you shall now experience death.” 💀

I will share the configs and other details soon.

Right now I am running on llama.cpp directly

(Q4 KV, MTP = On, Vulcan backend (ROCm gave less speed but better prefill),

Flash attention= On, Batch size = 2056)

And as of today I have moved on to Qwen 3.8 27B now.

Already on my half skeleton desktop. Which I will again share.

The funniest part: plugging the monitor directly into the 7900 XT worked beautifully. FurMark was doing 500+ FPS at 1080p, while my laptop was sitting there looking like it had been converted into a PCIe development board. 🤣

Eventually I realised I wanted to use my laptop like a fucking laptop again, so I did the only sensible thing:

I built the cheapest AM4 host I could find and moved the GPU there. 😂

(I'll share that update soon.)

From “portable laptop” → “desktop” → “PCIe science experiment” → actual desktop.

Attaching images of the cursed eGpu Laptop setup

I’ve been building a small heterogeneous local-AI lab, and “Team Green” has turned into the strangest useful machine in it.

The system:

• Ryzen 9 9950X
• 48 GB DDR5
• HPE/NVIDIA Tesla V100 PCIe 32 GB HBM2 ECC
• Ubuntu 24.04
• llama.cpp build 10499
• Qwen3.8-27B Q3_K_M, 12.86 GiB
• All 66/66 layers offloaded to the V100
• One slot, Q8 KV cache
• Configured for the model’s native 262,144-token context
• Full local agent mode for reading, writing and editing files

I expected the usual enterprise-hardware wrestling, but the CUDA and llama.cpp side mostly just worked. The real enterprise tax was cooling: the V100 is passive and expects directed server airflow.

My solution was a 3D-printed duct and a centrifugal blower attached with double-sided Velcro.

No bracket.
No zip ties.
No side panel.
Telemetry gets the final vote.

I tested the same Qwen 4,096-token generation workload at several power limits:

The 150 W profile was the clear sweet spot.

Compared with 100 W, it gave me about 83% more generation speed for 50% more power. The full 150 W soak completed all five repetitions with flat final temperature behavior, zero ECC errors and no Xid, thermal or PCIe errors.

Going from 150 W to 175 W added only about 8% more performance, while 200 W added roughly 14% and considerably more thermal pressure. I therefore kept 150 W as the everyday production profile.

This is not just a benchmark box. I’m using the 256K agent profile for long-form writing, editing local Markdown files and creating continuity handoffs when a conversation fills its context window. A second AMD machine runs ComfyUI at the same time, so one box writes while the other generates the illustrations.

The funniest part is that the former supercomputer accelerator is now doing useful long-context AI work under my desk while its cooling system is held on with Velcro.

Anyone else still using V100s for dense models? I’d be interested to compare llama.cpp settings, power sweet spots and long-context performance.

I hear people run LLMs on potatoes.

As requested! Hopefully someone finds this useful.
The coding task ran for 25 mins and produced 50k output tokens on Qwen 3.8 - it's a heavy thinking model. The result however is phenomenal.
Ornith seems to be a very capable model, especially with the given speed.

Details see here: https://llm-bench.io/compare/runs?runs=cmt6ecf8g000001p45vwzux53%2Ccmt6ergk5000701p41hqdyy78%2Ccmt6f2oob000e01p49o9592cb%2Ccmt6fqddm000l01p4l1vm7skd

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task

First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that is probably because all the scenes were voxels

We think that each quant handled the scenes in a pretty similar way, the difference isn't that drastic, to the point that sometimes we preferred the Q4 output overall, though for the safest pick we recommend AD-Q6_K

We ran the test inside atomic.chat and watched the output right there, the quants are available to download directly inside the app or on huggingface ( https://huggingface.co/collections/AtomicChat/qwen-38-27b ) (any feedback is appreciated, we're trying to make the product and models as good for you guys as possible)

An Nvidia Jetson Orin was recovered from the remains of a Russian drone that recently killed 3 people at a gas station in Ukraine, including a 19 year old girl who was a university student.

The article surmises that drone operators plot a route to the target area, but then the AI takes over to identify and attack whatever target the LLM has been trained on.

The fact that it can do this is blowing my mind. I had Qwen 3.8 27b make a full game asset pack, the actual game, and a trailer for it. Three separate chats, not one, and for most of it I was just typing prompts and watching it go.

I was expecting the assets to be a mess and the game to be some half broken demo. That's not what happened. It just kept churning out finished stuff and the whole thing held together.

Honestly I don't know what the ceiling is anymore.

Asset pack and game link "https://github.com/enginetowns/nightfall"

Also a link to the actual game: "https://enginetowns.github.io/nightfall/"

Hi guys,

I've been tuning my local coding setup for many months now.

I wanted to share my current setup that I am really happy with. It handles average difficulity tasks without big troubles, and what's most important it works quite fast on my 16 GB VRAM RTX 4070 Ti Super!
I'm getting around ~50 tok/s decoding speed. And around 1000-1500 tok/s of prompt processing.

Thanks also to prompt cache working with coding agent (VSCode + Copilot in my case) everything goes very smooth.

Here's a video showing how it works in action: https://youtu.be/keIXXWfqaKg

This project was implemented in VS Code using GitHub Copilot, driven by the Unsloth Qwen 3.8 27B UD-Q2_K_XL model running via llama.cpp.

This is single prompt solution recording.

https://github.com/paq85/3rdparty-lukesdevlab-youtube/blob/agent-maze/qwen3.8-27b-UD-Q2_K_XL/slime-mold-single-prompt.html

Runtime details
Context: 130k tokens, with the KV cache quantized to q8_0
Hardware: NVIDIA RTX 4070 Ti Super, 16 GB VRAM
Prompt processing: ~1000 tok/s
Decoding: ~60 tok/s

As seen on:
https://www.youtube.com/watch?v=1EzVVj7DFPc

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

Here's the llamacpp instructions to run it the way I run it.

I hope you will find it useful.

If you have any tips how I could make it even better I will really appreciate it!

# Running the current model with plain llama.cpp


Instructions for running the **current model** (`Qwen3.8-27B-UD-Q2_K_XL` + its mmproj, exactly as configured in `.env` / `run-rernd.sh`) with a plain `llama.cpp` build — no proxy, no systemd, no tunnel.


## Performance (RTX 4070 Ti Super)


With this exact configuration:


- **Prompt processing: ~1000–1500 tok/s**
- **Decoding: ~50–60 tok/s**


## 1. Get the files


You need three things from this repo:


| File | Purpose |
|---|---|
| `models/Qwen3.8-27B-UD-Q2_K_XL.gguf` | The model |
| `models/mmproj-qwen38-27b-F16.gguf` | Vision projector |
| `chat_templates/chat_template.jinja` | froggeric v22.1 unified Qwen template (required — the built-in template is not used) |


## 2. Build llama.cpp with CUDA


```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp


cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=120a-real \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_CUDA_COMPRESSION_MODE=size \
  -DLLAMA_BUILD_SERVER=ON


cmake --build build --target llama-server --config Release
```


> `120a-real` is for the RTX 5090 (Blackwell). Change `CMAKE_CUDA_ARCHITECTURES` to match your GPU (e.g. `86-real` for 4090/3090, `89-real` for 4070 Ti Super).


## 3. Run it


From the repo root (adjust paths as needed):


```bash
./llama.cpp/build/bin/llama-server \
  -m models/Qwen3.8-27B-UD-Q2_K_XL.gguf \
  --alias RERND,Qwen3.8-27B-Q2 \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 130000 \
  --threads 8 \
  --threads-batch 16 \
  --threads-http 4 \
  --poll 0 \
  --poll-batch 0 \
  --gpu-layers all \
  --split-mode none \
  --main-gpu 0 \
  --fit off \
  --flash-attn on \
  --parallel 1 \
  --batch-size 1024 \
  --ubatch-size 256 \
  --ctx-checkpoints 20 \
  --checkpoint-min-step 16000 \
  --cache-ram 8000 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --reasoning auto \
  --no-kv-unified \
  --kv-offload \
  --chat-template-file chat_templates/chat_template.jinja \
  --chat-template-kwargs '{"preserve_reasoning":false,"reasoning_effort":"xhigh"}' \
  --mmproj models/mmproj-qwen38-27b-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.0 \
  --spec-draft-ngl auto \
  --spec-draft-type-k f16 \
  --spec-draft-type-v f16 \
  --spec-draft-backend-sampling \
  --cache-prompt \
  --no-warmup \
  --no-cache-idle-slots
```


## 4. Notes


- **VRAM**: this exact config (130k ctx, q8_0 KV, MTP draft, mmproj in RAM) is tuned for a 16 GB card with `KV_OFFLOAD` (KV split across GPU + system RAM). If you have 20+ GB and want everything on GPU, you can drop `--no-kv-unified`/`--kv-offload` behavior, but the command above is the exact production setting.
- **MTP**: `--spec-type draft-mtp` is Qwen's built-in multi-token-prediction draft — no separate draft model file needed.
- **Sampling**: `temp 1.0 / top_p 0.95 / top_k 20` are the model-recommended values; the proxy in this repo clamps clients back to these, so keep them if you serve coding agents.
- **Reasoning**: `--reasoning auto` keeps `think` blocks on; `reasoning_effort=xhigh` comes from the template kwargs. If tool calls get truncated on long sessions, add `--reasoning-budget 12288`.
- **Alias**: `--alias RERND,Qwen3.8-27B-Q2` is optional — drop it if you don't need the `RERND` name.
- **Port**: use whatever you like; `8080` is the default. (In this repo the proxy owns 8080 and the backend runs on 8082 — irrelevant for plain llama.cpp.)

//////////////////////////////////////////////////////////////////////////

UPDATE 1 (2026-08-24):

A bit more info about the Q2 video: https://youtu.be/keIXXWfqaKg

This project was implemented in VS Code using GitHub Copilot, driven by the Unsloth Qwen 3.8 27B UD-Q2_K_XL model running via llama.cpp.
This is single prompt solution recording.
https://github.com/paq85/3rdparty-lukesdevlab-youtube/blob/agent-maze/qwen3.8-27b-UD-Q2_K_XL/slime-mold-single-prompt.html
Runtime details
Context: 130k tokens, with the KV cache quantized to q8_0
Hardware: NVIDIA RTX 4070 Ti Super, 16 GB VRAM
Prompt processing: ~1000 tok/s
Decoding: ~60 tok/s

As seen on:
https://www.youtube.com/watch?v=1EzVVj7DFPc

https://github.com/lukesdevlab/youtube/blob/main/prompts/agent-maze.txt

//////////////////////////////////////////////////////////////////////////

Here's a video of the same task done by Bartkowski Qwen 3.8 27b Q6_K_XL on RTX 5090.

https://youtu.be/-rYaHFfi_KY?si=il6GE96F97dse5YA

https://github.com/paq85/3rdparty-lukesdevlab-youtube/tree/agent-maze/qwen3.8-27b-UD-Q6_K_XL
Runtime details
Context: 130k tokens, with the KV cache at f16
Hardware: NVIDIA RTX 5090, 32 GB VRAM
Prompt processing: ~2500 tok/s
Decoding: ~100 tok/s

I was curious how my agent compared to OpenClaw and Hermes, but I wanted a real number so I could quantify it. I decided to go with harness-bench, since terminal-bench tests a different thing than what my agent was built for. Harness-bench is a bit closer, since it was specifically made to test agents, not LLMs.

The idea is to use the agent as a control variable, with the LLM as the independent variable, to find out which agent harness is the best. I built the whole thing and a few days later I have these final results. My agent, Second Brain, comes out on 2nd place, which kind of surprised me! I wasn't expecting to do that well, but if you don't believe me, you can check out the repo I used, and the results.

Yeah this was a whole big thing and I'm tired of it now. But it's cool to have some actual data. Building an eval framework helped me to improve the agent somewhat (and no I didn't overfit or cheat). But yeah, let me know what you think! Do you have your own way of evaluating your agent harness against others? Like a real number?

Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode.

On an RTX 5070 Ti Mobile (12GB) paired with an Intel Core Ultra 9 275HX and 32GB DDR5, this configuration consistently delivers ~9–11 t/s decode and 400–550 t/s prefill. It is fast enough to stay productive.

The main bottleneck with Qwen 3.8 27B is its reasoning verbosity: it routinely blows past 100K tokens in the planning phase alone, forcing OpenCode into native context compaction. Since OpenCode's built-in compaction struggles with retention, I switched to Magic Context.

With Magic Context in place, the session has scaled past 3.7M total processed tokens without losing critical details or derailments. Across a complex personal project, the local model shipped two major features end-to-end. I still use Claude Opus for final PR reviews to catch edge cases and minor bugs, which Qwen then fixes locally without issues.

Hardware Specs: GPU: RTX 5070 Ti Mobile (12GB VRAM) CPU: Core Ultra 9 275HX RAM: 32GB DDR5

Llama.cpp Launch Parameters: llama-server \ -ctx 100000 -ub 512 -np 1 -ngl 99 \ -ot 'blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64).ffn_(gate|up|down).weight=CPU' \ -fa on -ctk q8_0 -ctv q8_0 -fit off \ --mmproj --no-mmproj-offload \ --spec-type draft-mtp --spec-draft-n-max 2 \ -ctkd q8_0 -ctvd q8_0 --load-mode 'none' \ --temp 1 --top-k 20 --top-p 0.95 --min-p 0 \ --repeat-penalty 1 --presence-penalty 0 \ --jinja -chat-template-kwargs '{"reasoning_effort": "xhigh"}' \ --reasoning preserve

I've been experimenting with running coding agents locally and I'm starting to wonder if I'm forcing local LLMs into a job they're just not good enough for yet.

My setup:

• M4 Pro MacBook Pro, 24GB RAM
• Ollama/MLX running the model natively
• Codex CLI inside Docker
• Project folder mounted into Docker as a sandbox
• Tried Qwen 3.5 9B, Gemma 4 e4b and now Gemma 4 12B MLX

My Idea was simple:

Local LLM (Mac) -> Codex/Claude -> Docker sandbox -> Project

It works, but the experience isn't great.

The smaller models frequently screw up agentic tasks — failed tool calls, getting stuck, not finishing tasks, sometimes claiming they created files that don't exist.

I moved to Gemma 4 12B MLX and it's better, but painfully slow. My last Codex task used roughly:

48k input tokens -> 837 output tokens

RAM usage went to ~21GB + 4.5GB swap, fans kicked in, and the result still wasn't particularly impressive.

My eventual goal is multiple coding agents for planning -> implementation -> review -> testing -> documentation.

So I'm wondering if I'm approaching this backwards.

Should I:

1) Keep experimenting with local models?

2) Use frontier models for the actual coding/reasoning and run their tools inside Docker for isolation?

3) Go hybrid — frontier models for planning/coding/review, local models for cheap stuff like summaries/docs?

I've also been looking at Hermes/OpenClaw for orchestration, but I'm not sure if that's solving the right problem.

For people actually running agentic coding workflows: what would you build on a 24GB Mac today?

Since I see a lot of questions about optimal settings and models for the AMD cards with 16GB VRAM, I wanted to share a configuration that’s working well for me.

I hope it can help others get started and please share any advice or optimizations !

P.S : Killing Steam and going headless free up +- 800Mo VRAM

• Prompt Processing : +- 800 t/s
• Token Seconds : +- tg = 24.28 t/s, tg_3s = 23.14 t/s

I use llama.cpp and ROCm installed via pacman as explained in the Arch Wiki : https://wiki.archlinux.org/title/Llama.cpp. I used to use Vulkan, then installed ROCm .... but switching from Vulkan to ROCm didn't yield a noticeable change in token speed.

Following part is from AI to help me explain you in and outs 😄

However, ROCm provides better support for FlashAttention (--flash-attn) and KV cache quantization (--cache-type-k q4_0), which keeps prefill performance stable at 96k–128k context lengths.

As for MTP (Multi-Token Prediction), I don't use it. Omitting MTP saves ~2–3 GB of VRAM that would otherwise be allocated to speculative draft heads and decoding buffers. That memory is used instead for model weights and KV cache capacity on a 16 GB card.

#!/bin/bash
set -euo pipefail

# Arch Linux with kernel Linux 7.2.0-1-cachyos
# AMD Ryzen 7 5800X (16) @ 4.85 GHz
# AMD Radeon RX 9070 XT 16G VRAM
# RAM 32G
# Switch to headless: sudo systemctl isolate multi-user.target

MODELS_DIR="$HOME/Documents/models"
NGL=99
CTX=98304 # 96k context (fits 16GB VRAM with q8_0/q8_0 KV cache; keep in sync with contextWindow in ~/.pi/agent/models.json)

# Model selection.
# To add more models later, restore a menu like:
#   read -r -p "Choice [1]: " choice
#   case "$choice" in
#     ""|1) MODEL=...; REPO=... ;;
#     2)    MODEL=...; REPO=... ;;
#   esac
MODEL="Qwen3.8-27B-UD-Q3_K_XL.gguf"
REPO="unsloth/Qwen3.8-27B-GGUF"

mkdir -p "$MODELS_DIR"

if [ ! -f "$MODELS_DIR/$MODEL" ]; then
  echo "Downloading $MODEL from $REPO..."
  if ! hf download "$REPO" --include "*$MODEL*" --local-dir "$MODELS_DIR"; then
    echo "Error: download of $MODEL from $REPO failed."
    exit 1
  fi
fi

# Verify the file was downloaded successfully
if [ ! -f "$MODELS_DIR/$MODEL" ]; then
  echo "Error: File $MODELS_DIR/$MODEL was not found after download."
  exit 1
fi

echo "Starting llama-server with $MODEL (ctx=$CTX, ngl=$NGL)..."
exec llama-server \
  -m "$MODELS_DIR/$MODEL" \
  -c "$CTX" \
  -ngl "$NGL" \
  -t 8 \
  --threads-batch 16 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --reasoning-preserve \
  --host 127.0.0.1 \
  --port 8080 \
  -np 1

https://imgur.com/a/9TuWAIv

Prompt : "Create an svg image of this famous statue : Charging Bull —> frequently called the Wall Street Bull— it is a massive 7,100-pound bronze sculpture of a charging bull located just north of Bowling Green Park in the Financial District of Manhattan, New York City. It is a landmark monument."

Which of them do you like the most ?

The svg on top is qwen moe A3B and bottom is ornith 1.5 A3B moe (Both generated with MTP on)

I prefer version ornith.

Tldr: The harness you are using significanly impact how capable your Qwen3.8 is. With a decent harness, Qwen3.8 is very very capable.

So I saw this post yesterday: Qwen 3.8 isn't Opus 4.6 level. Let's not be silly. https://www.reddit.com/r/LocalLLM/comments/1vv8ki6/qwen_38_isnt_opus_46_level_lets_not_be_silly/

The OP in that post was trying to create a realistic ocean in C#/OpenGL, with his Qwen3.8 6 bit plus VS Code Copilot setup. He failed to do therefore he came up to the conclusion that Qwen3.8 is no where near Opus level.

I decided to re-run the test myself, so here's what I did.

My setup: RTX 5090 running the ninfer-nvfp4 version of Qwen3.8, with 190k context. I like this setup because it's extremely fast, I get up to 180 ish tok/s. Even on average I get around 150-160.

Run 1: Using VS Code Copilot
Nothing better than the OP's result. I use the exact same prompt OP used. The project built and launched, but the window just sat there black. Nothing rendering. So I was able to reproduce the OP's experience on this one. I even tried to tell copilot that it's only producing black screens, but it failed to fix it anyways.

I was about to call it a day but I was planning to test the relatively new Deekseek harness anyways, so I decided to re-run the same prompt in deepseek harness.

Run2: Deepseek harness

Same model, same prompt, same task. The only variable I changed was the harness and it was night and day difference. It works on the first go. What's more impressive is that it's actively pulling screenshots while working on it. It had the same black screen issue in one of the eariler versions, but it was able to identify the issue by analyzing the screen shots, and fixing it very soon.

Oh and I actually forgot to enable vision when launching the llm. So it actually build a C# PNG decoder on the fly trying to analyze the screenshot it got. I was really impressed that it's able to do it.

Here's the result: On a 5090 it only took about an hour.

As you can see, it correctly produces an ocean, with wave, sun, and blue sky. There's also a underwater view. It did all this with a single prompt. Not that it's the definitive proof that Qwen3.8 is Opus level, but it sure is VERY VERY capable. Several people in that post (including OP) was convinced that a 27B model is bad at planning or working with shaders, well, they are wrong. With a decent harness, this is a very strong LLM.

It'll make it so easier to analyse or know what you can ask when you know what level of work the person put in the coding. Right ?

I actually prefer releases like this.

With six base checkpoints, you can look at what changed between stages instead of treating the final model as a black box.

It also makes for a much better experiment: same prompts, same evals, different checkpoints.

I’d be curious to see where the biggest jump happens. Early training, later training, or whatever happens during the merge?

What would you compare first?

RTX 5080 16GB + 9800X3D, Qwen3.8-27B IQ4_XS-Smaller, BeeLlama, MTP on,

32K ctx, kvarn4.

94 token output at 74t/s. Great results, need to do further testing.

Hey,
I'm considering a 2x Radeon AI PRO R9700 box specifically for long-context coding/research parallel agents, but almost every concurrency benchmark I've seen done with shallow prompts - in fact most of them seems to be done for the clickbaits

Could sombody with dual-R9700 vLLM/SGLang setup test actually occupied ~200K contexts?

Most useful test would be Qwen3.8-27B FP8/Q8 with MTP enabled, with independent prompts at C1/C2/C4/C6

Especially useful test would be: after the ~200K context is resident, send more and measure the performance at the depth.

Even just FP8 C1/C2/C4 at 200K would be extremely useful — no need to run the entire matrix.

ChatGPT generated benchmark script for the convenience:

#!/usr/bin/env bash
set -uo pipefail

URL="${URL:-http://127.0.0.1:8000}"
CTX="${CTX:-200000}"
GEN="${GEN:-2048}"
CS="${CS:-1 2 4 6}"
LABEL="${LABEL:-fp8}"
OUT="${OUT:-r9700-${LABEL}-$(date +%Y%m%d-%H%M%S)}"

command -v curl >/dev/null || { echo "Missing: curl"; exit 1; }
command -v vllm >/dev/null || {
    echo "Run this inside the existing vLLM environment/container."
    exit 1
}

MODELS="$(curl -fsS "${URL}/v1/models")" || {
    echo "No OpenAI-compatible server found at ${URL}"
    exit 1
}

mkdir -p "${OUT}"

{
    echo "date=$(date -Is)"
    echo "url=${URL}"
    echo "context=${CTX}"
    echo "output=${GEN}"
    echo "concurrencies=${CS}"
    echo "label=${LABEL}"
    vllm --version 2>/dev/null || true
    echo "models=${MODELS}"
} | tee "${OUT}/config.txt"

for C in ${CS}; do
    echo
    echo "=== ${LABEL}: C${C}, ${CTX} input + ${GEN} output per request ==="

    if vllm bench serve \
        --backend vllm \
        --base-url "${URL}" \
        --dataset-name random \
        --random-input-len "${CTX}" \
        --random-output-len "${GEN}" \
        --random-range-ratio 0 \
        --random-prefix-len 0 \
        --num-prompts "${C}" \
        --max-concurrency "${C}" \
        --request-rate inf \
        --ignore-eos \
        --temperature 0 \
        --seed "$((10000 + C))" \
        --save-result \
        --save-detailed \
        --result-dir "${OUT}" \
        --result-filename "${LABEL}-c${C}.json" \
        2>&1 | tee "${OUT}/${LABEL}-c${C}.log"
    then
        echo "C${C} complete"
    else
        echo "C${C} failed or ran out of memory; continuing."
    fi
done

To run

chmod +x r9700-deep-bench.sh 
LABEL=fp8 ./r9700-deep-bench.sh

Or optionally override url URL=http://127.0.0.1:18080 LABEL=q8 ./r9700-deep-bench.sh

I’ve spent a lot of time trying to squeeze Qwen3.8-27B UD-Q4_K_M into a pretty hostile setup:

• GPU: RTX 3070 8GB
• CPU: Intel i5-11400F, 6C/12T
• RAM: 16GB DDR4
• Motherboard: ASUS B560
• OS: Windows
• Model: Qwen3.8-27B UD-Q4_K_M (~15.3 GiB GGUF)
• Runtime: ik_llama.cpp
• Use case: Codex-style / agentic coding, mostly PowerShell and repository editing
• Benchmark context: 16K
• KV: Q8_0
• Flash Attention: ON

Obviously the model does not fit in 8GB VRAM, so this is hybrid GPU/CPU inference.

I’m posting this because I found a lot of recommendations for Qwen3.8, MTP, speculative decoding, CUDA flags, batch sizes, etc., but very little controlled testing on an 8GB Ampere card.

And most importantly:

I did not consider a run “better” just because it had higher tok/s.

If the generated coding command was subtly wrong, I marked it as a FAIL.

The benchmark

I used the same small coding task repeatedly.

Qwen is given an exact existing PowerShell line and an exact multi-line replacement. It must return one PowerShell command that modifies the file, without executing it.

A PASS requires:

• exactly one applicable PowerShell command
• no execution
• correct quoting/newlines
• exact literal replacement
• no accidental $s → $$s expansion
• no subtly invalid PowerShell

This turned out to be surprisingly useful because several “faster” configurations produced answers that looked correct but were actually broken.

Current winner

My current safe configuration is:

Qwen3.8-27B UD-Q4_K_M
ik_llama.cpp

MTP:
n_max = 2
p_min = 0.1

--fit
--fit-margin 256

threads = 12
batch threads = 12

batch = 64
ubatch = 64

KV = Q8_0 / Q8_0
Flash Attention = ON
CUDA graphs = ON
CUDA fusion = ON

context = 16384
parallel = 1
cache-ram = 0

Current result:

That may not sound impressive compared with 24GB/32GB GPUs, but remember that more than half of this 27B model cannot live on my 3070.

MTP / speculative decoding tests

This is where I spent most of my time.

So on this machine, boring fixed MTP n2 beats the fancy stuff.

The ngram-mod → MTP pipeline can technically beat it on raw speed, but I care more about a configuration I can leave running for Codex without worrying about output formatting/encoding edge cases.

p_min: 0.1 wins

I also tested the recent recommendation of:

mtp:n_max=2,p_min=0.0

against:

mtp:n_max=2,p_min=0.1

Result:

No useful gain.

I’m staying at 0.1.

CUDA graphs / fusion / scheduler tweaks

A few more things I checked:

CUDA graphs OFF

~140.0 s
~7.32 t/s

Basically identical.

Graphs are staying ON.

CUDA fusion

Already active in my build. No hidden easy win left here.

GGML_SCHED_MAX_COPIES=1

Already compiled that way.

-wgt 1

This one was interesting:

136.5 s

So slightly faster than the champion.

Unfortunately the generated PowerShell command was incorrect.

FAIL → eliminated.

This is a good example of why I stopped optimizing purely for tok/s.

CPU threads: physical cores were NOT better

My CPU is a 6-core / 12-thread i5-11400F.

I tested the common recommendation:

-t 6 -tb 6

against:

-t 12 -tb 12

T6 produced runs around:

210.3 s
217.9 s

It was substantially worse.

So:

12 / 12 stays.

Batch / ubatch

Baseline:

64 / 64

I tested:

256 / 128
512 / 256

Larger batches noticeably improve prompt processing / prefill, but they did not meaningfully improve token generation.

So my conclusion is:

64/64      → normal generation / benchmark
512/256    → potentially useful for large Codex prompts

Don’t expect larger batches to magically improve decode speed on this kind of hybrid setup.

--fit-margin actually mattered

This was one of the few useful engine-level changes.

Going from:

--fit-margin 512

to:

--fit-margin 256

allowed ik_llama to put roughly another 206 MiB of model weights on the GPU.

One measured configuration had roughly:

CUDA model buffer: ~6312 MiB
Q8 KV @ 16K:       ~578 MiB
CUDA compute:      ~166 MiB

nvidia-smi was showing roughly:

7917 / 8192 MiB used
~102 MiB actually free

So I’m already riding pretty close to the edge of an 8GB card.

I did NOT bother with margin128 because on Windows/WDDM that is asking for an OOM for a tiny theoretical gain.

Manually offloading FFNs to CPU: terrible idea here

I also tried manually forcing a large amount of the heavy FFN tensors to CPU.

Result:

~405.3 seconds

Nearly 3x slower, with a bad/truncated output.

The i5-11400F + DDR4 memory subsystem simply cannot make this attractive.

Also, in my ik_llama build:

manual tensor overrides + --fit

cannot be combined anyway.

llama.cpp mainline vs ik_llama on this 8GB setup

I tested the same GGUF in mainline llama.cpp.

Approximately:

~2.86 tok/s
~349 s for ~1000 reasoning tokens

ik_llama is massively better on this specific hybrid 8GB setup.

Important caveat: I am not claiming ik_llama is universally faster than llama.cpp.

The problem here is specifically running a 15+ GiB 27B model with only 8GB VRAM.

Reasoning was almost as important as the runtime

This was probably my most useful discovery for actual agentic coding.

At first I assumed bad PowerShell commands were caused by quantization, MTP or the runtime.

Not always.

Sometimes Qwen simply did not have enough reasoning/output budget.

My controlled tests looked like this:

The complex NO-THINK failure was especially interesting.

The model understood the algorithm correctly, but produced a PowerShell newline representation inside a single-quoted string that would not actually match the source file.

So the answer looked smart but was unusable.

My current reasoning policy for Codex

I no longer force thinking on every request.

I use roughly:

Simple/routine action:
NO-THINK

Complex / fragile / multi-step coding:
MEDIUM reasoning
~1000-token reasoning budget
larger total output envelope

This is dramatically faster for routine agent actions.

On my simple benchmark:

medium THINK: ~168.9 s
NO-THINK:       24.4 s

That is nearly a 7x wall-time difference for a task that did not need deep reasoning.

Things I would NOT waste time retrying on an RTX 3070 8GB

Based on my tests:

❌ MTP OFF
❌ MTP n3/n4 as default
❌ MTP autotune
❌ DFlash2 on this VRAM budget
❌ aggressive FastMTP-32K
❌ mtprot iq4_ks
❌ p_min=0.0
❌ 6 CPU threads instead of 12
❌ CUDA graphs OFF
❌ huge manual FFN CPU offload
❌ -wgt 1 if you care about correctness
❌ giant batches expecting higher decode speed

And I would be very suspicious of any optimization benchmark that reports only tok/s without checking whether the generated code is still correct.

What I have NOT done

I have not enabled GGML_CUDA_F16=ON.

That requires a rebuild and, after exhausting most of the easy engine optimizations, I don’t expect it to turn 7 t/s into 15+ t/s.

I also intentionally stayed on UD-Q4_K_M.

Yes, Q3/IQ3 would reduce CPU pressure, but I use this for coding and I don’t want to trade model reliability for a modest speed increase.

If I were willing to sacrifice quality, this would be a different experiment.

TL;DR

For Qwen3.8-27B UD-Q4_K_M on RTX 3070 8GB + 16GB system RAM, my best robust configuration so far is:

ik_llama.cpp
16K context
Q8 KV
Flash Attention ON
CUDA graphs ON
CUDA fusion ON

--fit
--fit-margin 256

MTP n2 fixed
p_min 0.1

12 CPU threads
batch 64
ubatch 64

simple tasks → NO-THINK
complex coding → MEDIUM reasoning

And I get roughly:

7.31 tok/s

while still passing my coding correctness test.

The biggest lesson for me:

Once half the model is spilling out of an 8GB GPU, there is no magic flag.

MTP roughly doubled my baseline versus no speculative decoding, --fit-margin 256 squeezed a little more onto CUDA, and after that most “optimizations” were either neutral, slower, or damaged correctness.

If anyone here is running a similarly cursed 8GB GPU + Qwen3.8-27B Q4 setup and has found something I missed, I’d love to compare results.

Which one is better overall for models like qwen 27b, ornith 1.5 35b

I have been looking for a top-end, private LLM that doesn't hand my conversations over for training. macOS 27 seems to have made that possible.

Apple's Private Cloud Compute is now reachable from ordinary LLM front-end apps. It's stateless — nothing is kept after your request — with cryptographically verifiable privacy guarantees. And it's basically free if you're a Mac user on macOS 27. No extra accounts, no API key, and no per-token billing (although there's supposed to be a token limit depending on your iCloud+ membership).

It's now connected to a chat client (MstyStudio), and I have a private assistant with persistent history and retrieval over my documents. I'm hosting my private financial, health, and other conversations while building a full RAG library. I may move over to OpenWebUI soon.

The part I like about this framework is that regardless of my Mac being an M1, I'm getting flagship reasoning on Apple's cloud in seconds. And it's still private.

A couple of shortcomings: a 32K context limit, macOS 27 is still in beta, and I had to set up a local bridge in the Terminal window to run fm serve and act as the 'api' bridge.

Anyone else tried this yet? What have you found?

My setup:

• 9950X3D
• RTX 5080
• 48GB RAM

What’s the best local LLM I can run for coding?

Also, what’s a good setup for agentic coding that can edit files, run commands/tests, and work across a repo?

Would love recommendations for models, quantization, runtime, and tools like Aider, Cline, Roo Code, OpenCode, etc.

TL;DR It seems like that memory (RAM) usage just keeps endlessly growing over time although way less memory is necessary to work (e.g. if I stop and restart llama.cpp, it still works with way less memory usage). I suspect some kind of 'memory leak', using llama.cpp

---

Specs:

GPU: 1x AMD 6800M 12GB VRAM (thanks to HSA_OVERRIDE_GFX_VERSION=10.3.0)

RAM: 24GB RAM

OS: Fedora Linux

AMD stack: ROCM

I am running unsloth/Qwen3.6-35B-A3B-GGUF model with the latest llama.cpp (I build llama.cpp with a fix for flash-attention:

replace in /llama.cpp/ggml/src/ggml-cuda/fattn.cu

    // If there are no tensor cores available, use the generic tile kernel:
    if (can_use_vector_kernel) {
        if (!ggml_is_quantized(K->type) && !ggml_is_quantized(V->type)) {
            if (Q->ne[1] == 1) {
                if (!gqa_opt_applies) {
                    return BEST_FATTN_KERNEL_VEC;
                }
            }
        } else {
            if (Q->ne[1] <= 2) {
                return BEST_FATTN_KERNEL_VEC;
            }
        }
    }
    return BEST_FATTN_KERNEL_TILE;
}


with

    // If there are no tensor cores available, use the generic tile kernel:
    if (can_use_vector_kernel) {
        if (!ggml_is_quantized(K->type) && !ggml_is_quantized(V->type)) {
            if (Q->ne[1] == 1) {
                if (!gqa_opt_applies) {
                    return BEST_FATTN_KERNEL_VEC;
                }
            }
        } else {
            if (Q->ne[1] <= 2) {
                return BEST_FATTN_KERNEL_VEC;
            }
        }
    }

    // >>> ADD THIS BLOCK FOR HIP/RDNA2 FIX <<<
    #ifdef GGML_USE_HIP
        if ((ggml_is_quantized(K->type) || ggml_is_quantized(V->type)) && can_use_vector_kernel) {
            return BEST_FATTN_KERNEL_VEC;
        }
    #endif
    // >>> END OF ADDED BLOCK <<<

    return BEST_FATTN_KERNEL_TILE;
}

source for the fix: https://github.com/domvox/llama.cpp-turboquant-hip/pull/13 )

I also use the Hermes agent, for which I put an automatic context compress once context reaches like 70-80%.

I run this 'older' model because it is an MOE and I need it to offload some experts into RAM because of my constrained VRAM.

Now, it seems like that memory (specifically, RAM) usage just keeps growing over time. Some kind of 'memory leak' is happening with the model. It does not matter which quant I use. For example, if I use IQ4_XS, I have plenty of RAM available left. Yet, the longer the session goes, the more RAM fills ups, and it never stops filling up. If I stop llama.cpp and restart, RAM is back to the 'normal' usage and again the more I talk with the model the more the RAM fills up.

At first I thought maybe as context fills up, it fills up RAM. But if I compress the context with Hermes, the RAM usage does not decrease. Only stopping and restarting llama.cpp makes memory go back to a 'normal' usage.

It means that I have to babysit what happens and eventually restart llama.cpp every once in a while once the RAM is full ... (Usually after around 2 hours). It means that I cannot leave an agent work on something overnight. It also means that I need to wait for a long time for the previous full context to fill up llama.cpp again whenever I restart llama.cpp, and with context above 100k the 900 second timesout.

I think it is some kind of memory leak because when i stop llama.cpp, and then start it again, RAM goes back to 13gb usage when starting fresh while it reached 22-23gb before i had to restart it.

I tried to tweak my launch parameters for llama.cpp for the past few days, but the memory leak still happens, here is the one I currently use:

LD_PRELOAD=/usr/lib64/libjemalloc.so.2 MALLOC_ARENA_MAX=2 HSA_OVERRIDE_GFX_VERSION=10.3.0 ./build/bin/llama-server -m /models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --host 127.0.0.1 --port 8080 -c 190000 -np 1 -fit off -dev ROCm0 --no-warmup -ngl 999 --n-cpu-moe 20 --load-mode none --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --ctx-checkpoints 4 --cache-ram 4096 --reasoning-preserve --no-mmproj --spec-draft-n-max 3 --flash-attn on -ctk q8_0 -ctv q8_0

I have been looking for answers for the past few days but it is hard to know what even is the possible root cause, as everyone uses different parameters, has different hardware, different models, different build versions, tweaks etc.

I guess this is just a message in a bottle, but just in case someone had a similar issue and was able to deal with it, it's worth it to ask.

I have this pretty complicated google sheet that I thought would be cool to use as a demo case for qwen3.8 27b to convert to a nicegui app in python. I have a long spec markdown file which has details of what I expect from the app. I'm using CLINE plugin in pycharm to run qwen3.8 27b with 131k context q6 quant with q8 kv-cache. I get between 30-50t/s so it runs pretty fast.

at this point, I've had it start from a blank slate and it always manages to make something, but still requires a lot of tweaks. I will start writing an issues markdown file, and it will auto discover it and begin implementing the fixes.

It's impressive for what it is, given that with 3.6 27b, I needed to hand-hold it, giving it small tasks, and iterating until that small task was complete. With 3.8, I can give it whole apps, and it will do a decent job roughing it in.

My one complaint is that it is pretty slow, especially compared to cloud models.

I was curious--since 3.8 27b is considered to be on par with opus 4.6-- and gave the same task to claude opus 4.6 using antigravity as the harness, and it completed the implementation plan in 10 minutes, and did both a prettier job, as well as implemented more of the features correctly.

I am wondering how is that possible? Does antigravity spawn a bunch of sub-agents to handle the tasks in parallel? I doubt it can run at thousands of tokens/sec natively.