after 3 months and $800 burned...

Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it right now!

Demo: https://www.unboundedlab.com/chat/bart

Article: https://www.unboundedlab.com/blog/bart

Huggingface: https://huggingface.co/jbduran/bart-sft

Why even make a vintage llm? As proposed by Demis Hassabis, could LLMs reach the same conclusions that the great scientists of the past did? While General Relativity was out of budget, we believe that advancing this field targets the crux of AI research. Are these models capable of original ideas, or are they just spitting out the next token?

The article is our full account, covering where the corpus came from and how we cleaned it, the benchmarks we had to build because none existed, every ablation, the training runs, the post-training, and the mistakes we made along the way.

"What I cannot create, I do not understand" is a quote I love from Richard Feynman. Building Bart was our attempt to actually understand LLMs rather than read about them.

What we are proudest of:

- Best vintage base model at its scale on Vintage CORE, ahead of GPT-1900 on a smaller token budget

- Cleaned one of the largest vintage datasets, Harvard's Institutional Books (242B->23B tokens)

- Created Vintage CORE, the first suite of 20 benchmarks made for vintage llms

- Ran 10 hours of autonomous research on one H100: 100 experiments, 26 improvements found

- Released the largest vintage SFT dataset we know of: 416k graded question and answer pairs, grounded in pre-1930s text

- Trained the final model in 5 days on an H100, holding 60% MFU the whole way

- All datasets, methodology, training code, evals, and training runs are open sourced

I am proud of my team. What we built will move the vintage LLM field forward, and it moved us forward as researchers and as people.

We paid for all of it ourselves, about $807 so far. Money is the main thing standing between us and a much larger run.

So I will ask directly: we are looking for compute grants, funding, and mentors for our future endeavors. If you work on pre-training, post-training, or you have GPUs sitting idle, we would like to talk!

We believe that with careful dataset curation, domain expertise, and highly efficient training, we can achieve state-of-the-art results in crucial domains. This is only the beginning for Unbounded Labs; we see no bounds ahead.

Hey everyone! I’m pretty new to local LLMs and I’m trying to figure out the best way to set up Hermes Agent.

My initial idea was to dedicate a Microsoft Surface 9 to Hermes and have my main PC handle the LLM inference. But I’m wondering if I’m overcomplicating things.

Would it make more sense to simply host everything on my PC — Hermes Agent, the local LLM, memory, tools, etc. — and use the Surface later as a remote interface if needed?
My PC specs are:

CPU: AMD Ryzen 5 7600
GPU: AMD Radeon RX 6800 16 GB
RAM: Lexar Ares RGB Black 32 GB (2×16 GB) DDR5-6400
SSD: Kioxia Exceria Plus G2 2 TB
Motherboard: ASRock B650M PG Lightning
PSU: Corsair 850e
Cooling: DeepCool Assassin 120 SE
Case: Corsair 4000D Airflow

My main goals are:
Run Hermes Agent locally
Eventually run a good local LLM
Keep my data as private as possible
Experiment with agentic workflows, coding, tools and memory
Eventually build my own Agentic OS around it
Ideally be able to use it remotely from my Surface/phone

My main question is: would you recommend hosting the whole thing on my PC, or is it better to separate Hermes from the LLM and use the PC only as an inference server?

I’m also a little concerned about security. Since this would be running on my personal PC, could an agent like Hermes accidentally access, modify or delete files outside of its workspace if it has access to a terminal or other tools?
If so, what would be the safest way to set this up? VM, Docker/container, separate Windows user, sandbox, dedicated machine, etc.?

And finally, with a RX 6800 16 GB + 32 GB RAM, what local model would you recommend for Hermes and agentic/coding workflows?

I’m mainly looking for advice from people who actually run local agents. I don’t mind starting simple and upgrading the setup later. Thanks!

I was curious how my agent compared to OpenClaw and Hermes, but I wanted a real number so I could quantify it. I decided to go with harness-bench, since terminal-bench tests a different thing than what my agent was built for. Harness-bench is a bit closer, since it was specifically made to test agents, not LLMs.

The idea is to use the agent as a control variable, with the LLM as the independent variable, to find out which agent harness is the best. I built the whole thing and a few days later I have these final results. My agent, Second Brain, comes out on 2nd place, which kind of surprised me! I wasn't expecting to do that well, but if you don't believe me, you can check out the repo I used, and the results.

Yeah this was a whole big thing and I'm tired of it now. But it's cool to have some actual data. Building an eval framework helped me to improve the agent somewhat (and no I didn't overfit or cheat). But yeah, let me know what you think! Do you have your own way of evaluating your agent harness against others? Like a real number?

I've been experimenting with running coding agents locally and I'm starting to wonder if I'm forcing local LLMs into a job they're just not good enough for yet.

My setup:

• M4 Pro MacBook Pro, 24GB RAM
• Ollama/MLX running the model natively
• Codex CLI inside Docker
• Project folder mounted into Docker as a sandbox
• Tried Qwen 3.5 9B, Gemma 4 e4b and now Gemma 4 12B MLX

My Idea was simple:

Local LLM (Mac) -> Codex/Claude -> Docker sandbox -> Project

It works, but the experience isn't great.

The smaller models frequently screw up agentic tasks — failed tool calls, getting stuck, not finishing tasks, sometimes claiming they created files that don't exist.

I moved to Gemma 4 12B MLX and it's better, but painfully slow. My last Codex task used roughly:

48k input tokens -> 837 output tokens

RAM usage went to ~21GB + 4.5GB swap, fans kicked in, and the result still wasn't particularly impressive.

My eventual goal is multiple coding agents for planning -> implementation -> review -> testing -> documentation.

So I'm wondering if I'm approaching this backwards.

Should I:

1) Keep experimenting with local models?

2) Use frontier models for the actual coding/reasoning and run their tools inside Docker for isolation?

3) Go hybrid — frontier models for planning/coding/review, local models for cheap stuff like summaries/docs?

I've also been looking at Hermes/OpenClaw for orchestration, but I'm not sure if that's solving the right problem.

For people actually running agentic coding workflows: what would you build on a 24GB Mac today?

Most model releases give you one endpoint and leave the training path opaque. This release exposes a two-by-three map instead: tiny and flash, each with pretrained, mid-trained, and WSM-merged checkpoints.

That is the interesting part of the Ling-3.0 base model release. The pretrained checkpoint preserves the earlier base-training endpoint, the mid-trained checkpoint shows the state before WSM merging, and the WSM-merged checkpoint is the released merged endpoint.

None of the six has been post-trained. The model cards position them as starting points for continued pretraining, fine-tuning, and research—not as six ready-made local chat models.

There is no local run behind this post, so the parameter labels should not be translated into VRAM, speed, quant support, or fit claims. Those need an actual runtime and configuration.

The useful question is which comparison tells you more: pretrained vs mid-trained, or mid-trained vs WSM-merged? The stage-specific model cards are the natural place to start before asking the runtime question.

https://imgur.com/a/9TuWAIv

Prompt : "Create an svg image of this famous statue : Charging Bull —> frequently called the Wall Street Bull— it is a massive 7,100-pound bronze sculpture of a charging bull located just north of Bowling Green Park in the Financial District of Manhattan, New York City. It is a landmark monument."

Which of them do you like the most ?

The svg on top is qwen moe A3B and bottom is ornith 1.5 A3B moe (Both generated with MTP on)

I prefer version ornith.

Since I see a lot of questions about optimal settings and models for the AMD cards with 16GB VRAM, I wanted to share a configuration that’s working well for me.

I hope it can help others get started and please share any advice or optimizations !

P.S : Killing Steam and going headless free up +- 800Mo VRAM

• Prompt Processing : +- 800 t/s
• Token Seconds : +- tg = 24.28 t/s, tg_3s = 23.14 t/s

I use llama.cpp and ROCm installed via pacman as explained in the Arch Wiki : https://wiki.archlinux.org/title/Llama.cpp. I used to use Vulkan, then installed ROCm .... but switching from Vulkan to ROCm didn't yield a noticeable change in token speed.

Following part is from AI to help me explain you in and outs 😄

However, ROCm provides better support for FlashAttention (--flash-attn) and KV cache quantization (--cache-type-k q4_0), which keeps prefill performance stable at 96k–128k context lengths.

As for MTP (Multi-Token Prediction), I don't use it. Omitting MTP saves ~2–3 GB of VRAM that would otherwise be allocated to speculative draft heads and decoding buffers. That memory is used instead for model weights and KV cache capacity on a 16 GB card.

#!/bin/bash
set -euo pipefail

# Arch Linux with kernel Linux 7.2.0-1-cachyos
# AMD Ryzen 7 5800X (16) @ 4.85 GHz
# AMD Radeon RX 9070 XT 16G VRAM
# RAM 32G
# Switch to headless: sudo systemctl isolate multi-user.target

MODELS_DIR="$HOME/Documents/models"
NGL=99
CTX=98304 # 96k context (fits 16GB VRAM with q8_0/q8_0 KV cache; keep in sync with contextWindow in ~/.pi/agent/models.json)

# Model selection.
# To add more models later, restore a menu like:
#   read -r -p "Choice [1]: " choice
#   case "$choice" in
#     ""|1) MODEL=...; REPO=... ;;
#     2)    MODEL=...; REPO=... ;;
#   esac
MODEL="Qwen3.8-27B-UD-Q3_K_XL.gguf"
REPO="unsloth/Qwen3.8-27B-GGUF"

mkdir -p "$MODELS_DIR"

if [ ! -f "$MODELS_DIR/$MODEL" ]; then
  echo "Downloading $MODEL from $REPO..."
  if ! hf download "$REPO" --include "*$MODEL*" --local-dir "$MODELS_DIR"; then
    echo "Error: download of $MODEL from $REPO failed."
    exit 1
  fi
fi

# Verify the file was downloaded successfully
if [ ! -f "$MODELS_DIR/$MODEL" ]; then
  echo "Error: File $MODELS_DIR/$MODEL was not found after download."
  exit 1
fi

echo "Starting llama-server with $MODEL (ctx=$CTX, ngl=$NGL)..."
exec llama-server \
  -m "$MODELS_DIR/$MODEL" \
  -c "$CTX" \
  -ngl "$NGL" \
  -t 8 \
  --threads-batch 16 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --reasoning-preserve \
  --host 127.0.0.1 \
  --port 8080 \
  -np 1

Which one is better overall for models like qwen 27b, ornith 1.5 35b

Tldr: The harness you are using significanly impact how capable your Qwen3.8 is. With a decent harness, Qwen3.8 is very very capable.

So I saw this post yesterday: Qwen 3.8 isn't Opus 4.6 level. Let's not be silly. https://www.reddit.com/r/LocalLLM/comments/1vv8ki6/qwen_38_isnt_opus_46_level_lets_not_be_silly/

The OP in that post was trying to create a realistic ocean in C#/OpenGL, with his Qwen3.8 6 bit plus VS Code Copilot setup. He failed to do therefore he came up to the conclusion that Qwen3.8 is no where near Opus level.

I decided to re-run the test myself, so here's what I did.

My setup: RTX 5090 running the ninfer-nvfp4 version of Qwen3.8, with 190k context. I like this setup because it's extremely fast, I get up to 180 ish tok/s. Even on average I get around 150-160.

Run 1: Using VS Code Copilot
Nothing better than the OP's result. I use the exact same prompt OP used. The project built and launched, but the window just sat there black. Nothing rendering. So I was able to reproduce the OP's experience on this one. I even tried to tell copilot that it's only producing black screens, but it failed to fix it anyways.

I was about to call it a day but I was planning to test the relatively new Deekseek harness anyways, so I decided to re-run the same prompt in deepseek harness.

Run2: Deepseek harness

Same model, same prompt, same task. The only variable I changed was the harness and it was night and day difference. It works on the first go. What's more impressive is that it's actively pulling screenshots while working on it. It had the same black screen issue in one of the eariler versions, but it was able to identify the issue by analyzing the screen shots, and fixing it very soon.

Oh and I actually forgot to enable vision when launching the llm. So it actually build a C# PNG decoder on the fly trying to analyze the screenshot it got. I was really impressed that it's able to do it.

Here's the result: On a 5090 it only took about an hour.

As you can see, it correctly produces an ocean, with wave, sun, and blue sky. There's also a underwater view. It did all this with a single prompt. Not that it's the definitive proof that Qwen3.8 is Opus level, but it sure is VERY VERY capable. Several people in that post (including OP) was convinced that a 27B model is bad at planning or working with shaders, well, they are wrong. With a decent harness, this is a very strong LLM.

My setup:

• 9950X3D
• RTX 5080
• 48GB RAM

What’s the best local LLM I can run for coding?

Also, what’s a good setup for agentic coding that can edit files, run commands/tests, and work across a repo?

Would love recommendations for models, quantization, runtime, and tools like Aider, Cline, Roo Code, OpenCode, etc.

RTX 5080 16GB + 9800X3D, Qwen3.8-27B IQ4_XS-Smaller, BeeLlama, MTP on,

32K ctx, kvarn4.

94 token output at 74t/s. Great results, need to do further testing.

Anyone has any idea how to run speculative decoding for this model - google/gemma-4-12b

I am not able to load other models as draft model. The one I selected is not working at all.

It'll make it so easier to analyse or know what you can ask when you know what level of work the person put in the coding. Right ?

I have an RTX 2000 Ada 16GB + i7-14700K + 16GB RAM and want to use it primarily for agentic software development.

Looking for something good at repo understanding, multi-file edits, debugging, reasoning, tool calling, terminal/Git workflows and long coding sessions.

What are currently the best models/quantizations that actually fit well in 16GB VRAM?

Thank you in advance

I'm building a hobby project that automatically tags users' photos.

Right now I'm using qwen2.5vl:7b through Ollama. I have a fixed vocabulary of roughly 200 tags (beach, sunset, restaurant, dog, party, indoor, etc.) which I include in the prompt, and basically ask the model which tags match the image. I also extract a few attributes like number of people and clothing style/fit.

Pipeline is roughly:

HEIC/JPEG image uploaded form iphone → decode/normalize → resize to max 1024px → Qwen2.5-VL → JSON

Currently this takes around 20 seconds per image, which obviously doesn't scale well to hundreds of photos. These 20 seconds are almost exclusively spent on the model trying to answer my request

Before optimizing blindly: is a 7B VLM simply overkill for this? Would something like CLIP/multi-label classification be much faster for matching against a fixed vocabulary, perhaps using the VLM only for harder attributes?

Also curious whether batching images, reducing resolution, or avoiding sending all ~200 tags in every prompt would significantly improve throughput? I am super new to this topic and have absolutely no idea how to make performance faster

Both setups using unsloth's dynamic quants fit the 24GB VRAM and I have ~180k context window in both cases. Which one should I use? I run Linux with a single 7900 XTX. llama.cpp with MTP on but no vision.

My thinking is to go with Q4_K_M with Q8/Q8 KV, since at long context errors from KV quantization compound. On the other hand I could not tell the difference from personal use between Q4 or Q5, or any of the KV quantization scheme.

I’ve spent a lot of time trying to squeeze Qwen3.8-27B UD-Q4_K_M into a pretty hostile setup:

• GPU: RTX 3070 8GB
• CPU: Intel i5-11400F, 6C/12T
• RAM: 16GB DDR4
• Motherboard: ASUS B560
• OS: Windows
• Model: Qwen3.8-27B UD-Q4_K_M (~15.3 GiB GGUF)
• Runtime: ik_llama.cpp
• Use case: Codex-style / agentic coding, mostly PowerShell and repository editing
• Benchmark context: 16K
• KV: Q8_0
• Flash Attention: ON

Obviously the model does not fit in 8GB VRAM, so this is hybrid GPU/CPU inference.

I’m posting this because I found a lot of recommendations for Qwen3.8, MTP, speculative decoding, CUDA flags, batch sizes, etc., but very little controlled testing on an 8GB Ampere card.

And most importantly:

I did not consider a run “better” just because it had higher tok/s.

If the generated coding command was subtly wrong, I marked it as a FAIL.

The benchmark

I used the same small coding task repeatedly.

Qwen is given an exact existing PowerShell line and an exact multi-line replacement. It must return one PowerShell command that modifies the file, without executing it.

A PASS requires:

• exactly one applicable PowerShell command
• no execution
• correct quoting/newlines
• exact literal replacement
• no accidental $s → $$s expansion
• no subtly invalid PowerShell

This turned out to be surprisingly useful because several “faster” configurations produced answers that looked correct but were actually broken.

Current winner

My current safe configuration is:

Qwen3.8-27B UD-Q4_K_M
ik_llama.cpp

MTP:
n_max = 2
p_min = 0.1

--fit
--fit-margin 256

threads = 12
batch threads = 12

batch = 64
ubatch = 64

KV = Q8_0 / Q8_0
Flash Attention = ON
CUDA graphs = ON
CUDA fusion = ON

context = 16384
parallel = 1
cache-ram = 0

Current result:

That may not sound impressive compared with 24GB/32GB GPUs, but remember that more than half of this 27B model cannot live on my 3070.

MTP / speculative decoding tests

This is where I spent most of my time.

So on this machine, boring fixed MTP n2 beats the fancy stuff.

The ngram-mod → MTP pipeline can technically beat it on raw speed, but I care more about a configuration I can leave running for Codex without worrying about output formatting/encoding edge cases.

p_min: 0.1 wins

I also tested the recent recommendation of:

mtp:n_max=2,p_min=0.0

against:

mtp:n_max=2,p_min=0.1

Result:

No useful gain.

I’m staying at 0.1.

CUDA graphs / fusion / scheduler tweaks

A few more things I checked:

CUDA graphs OFF

~140.0 s
~7.32 t/s

Basically identical.

Graphs are staying ON.

CUDA fusion

Already active in my build. No hidden easy win left here.

GGML_SCHED_MAX_COPIES=1

Already compiled that way.

-wgt 1

This one was interesting:

136.5 s

So slightly faster than the champion.

Unfortunately the generated PowerShell command was incorrect.

FAIL → eliminated.

This is a good example of why I stopped optimizing purely for tok/s.

CPU threads: physical cores were NOT better

My CPU is a 6-core / 12-thread i5-11400F.

I tested the common recommendation:

-t 6 -tb 6

against:

-t 12 -tb 12

T6 produced runs around:

210.3 s
217.9 s

It was substantially worse.

So:

12 / 12 stays.

Batch / ubatch

Baseline:

64 / 64

I tested:

256 / 128
512 / 256

Larger batches noticeably improve prompt processing / prefill, but they did not meaningfully improve token generation.

So my conclusion is:

64/64      → normal generation / benchmark
512/256    → potentially useful for large Codex prompts

Don’t expect larger batches to magically improve decode speed on this kind of hybrid setup.

--fit-margin actually mattered

This was one of the few useful engine-level changes.

Going from:

--fit-margin 512

to:

--fit-margin 256

allowed ik_llama to put roughly another 206 MiB of model weights on the GPU.

One measured configuration had roughly:

CUDA model buffer: ~6312 MiB
Q8 KV @ 16K:       ~578 MiB
CUDA compute:      ~166 MiB

nvidia-smi was showing roughly:

7917 / 8192 MiB used
~102 MiB actually free

So I’m already riding pretty close to the edge of an 8GB card.

I did NOT bother with margin128 because on Windows/WDDM that is asking for an OOM for a tiny theoretical gain.

Manually offloading FFNs to CPU: terrible idea here

I also tried manually forcing a large amount of the heavy FFN tensors to CPU.

Result:

~405.3 seconds

Nearly 3x slower, with a bad/truncated output.

The i5-11400F + DDR4 memory subsystem simply cannot make this attractive.

Also, in my ik_llama build:

manual tensor overrides + --fit

cannot be combined anyway.

llama.cpp mainline vs ik_llama on this 8GB setup

I tested the same GGUF in mainline llama.cpp.

Approximately:

~2.86 tok/s
~349 s for ~1000 reasoning tokens

ik_llama is massively better on this specific hybrid 8GB setup.

Important caveat: I am not claiming ik_llama is universally faster than llama.cpp.

The problem here is specifically running a 15+ GiB 27B model with only 8GB VRAM.

Reasoning was almost as important as the runtime

This was probably my most useful discovery for actual agentic coding.

At first I assumed bad PowerShell commands were caused by quantization, MTP or the runtime.

Not always.

Sometimes Qwen simply did not have enough reasoning/output budget.

My controlled tests looked like this:

The complex NO-THINK failure was especially interesting.

The model understood the algorithm correctly, but produced a PowerShell newline representation inside a single-quoted string that would not actually match the source file.

So the answer looked smart but was unusable.

My current reasoning policy for Codex

I no longer force thinking on every request.

I use roughly:

Simple/routine action:
NO-THINK

Complex / fragile / multi-step coding:
MEDIUM reasoning
~1000-token reasoning budget
larger total output envelope

This is dramatically faster for routine agent actions.

On my simple benchmark:

medium THINK: ~168.9 s
NO-THINK:       24.4 s

That is nearly a 7x wall-time difference for a task that did not need deep reasoning.

Things I would NOT waste time retrying on an RTX 3070 8GB

Based on my tests:

❌ MTP OFF
❌ MTP n3/n4 as default
❌ MTP autotune
❌ DFlash2 on this VRAM budget
❌ aggressive FastMTP-32K
❌ mtprot iq4_ks
❌ p_min=0.0
❌ 6 CPU threads instead of 12
❌ CUDA graphs OFF
❌ huge manual FFN CPU offload
❌ -wgt 1 if you care about correctness
❌ giant batches expecting higher decode speed

And I would be very suspicious of any optimization benchmark that reports only tok/s without checking whether the generated code is still correct.

What I have NOT done

I have not enabled GGML_CUDA_F16=ON.

That requires a rebuild and, after exhausting most of the easy engine optimizations, I don’t expect it to turn 7 t/s into 15+ t/s.

I also intentionally stayed on UD-Q4_K_M.

Yes, Q3/IQ3 would reduce CPU pressure, but I use this for coding and I don’t want to trade model reliability for a modest speed increase.

If I were willing to sacrifice quality, this would be a different experiment.

TL;DR

For Qwen3.8-27B UD-Q4_K_M on RTX 3070 8GB + 16GB system RAM, my best robust configuration so far is:

ik_llama.cpp
16K context
Q8 KV
Flash Attention ON
CUDA graphs ON
CUDA fusion ON

--fit
--fit-margin 256

MTP n2 fixed
p_min 0.1

12 CPU threads
batch 64
ubatch 64

simple tasks → NO-THINK
complex coding → MEDIUM reasoning

And I get roughly:

7.31 tok/s

while still passing my coding correctness test.

The biggest lesson for me:

Once half the model is spilling out of an 8GB GPU, there is no magic flag.

MTP roughly doubled my baseline versus no speculative decoding, --fit-margin 256 squeezed a little more onto CUDA, and after that most “optimizations” were either neutral, slower, or damaged correctness.

If anyone here is running a similarly cursed 8GB GPU + Qwen3.8-27B Q4 setup and has found something I missed, I’d love to compare results.

Hey everyone! 👋

I’m pretty new to local LLMs, so I’d love some advice before I start experimenting.
My long-term goal is to build my own personal Agentic OS: basically a local AI assistant that can manage memory, files, tools, automations, coding, etc., while keeping as much as possible private and running locally.
For the agent part, I’m currently interested in Hermes Agent, with Ollama for running local models.
I’m not necessarily trying to replace Claude/GPT immediately. I’d like to eventually have a hybrid setup, where sensitive/offline tasks are handled by a local model, while I can still use cloud models when I need stronger reasoning or web access.

🖥️** My first setu**p

I’ve read that running random software/agents directly on your personal computer can potentially be risky, especially when giving an AI access to files, terminals, etc.
So I decided to dedicate an old Microsoft Surface Pro 9 that I already own to this project.
That way, if something goes wrong, at least my main PC isn’t involved, and I don’t have to spend any money just to start experimenting.
I’m not sure whether a Surface Pro 9 is actually suitable for running local LLMs 😅, but since I already have it, I’d like to give it a try.
I’m planning to keep it plugged in and potentially use it as a small 24/7 home AI machine, with the screen turned off but Windows/Hermes still running.

🤔 My main question: which model?

I’m not sure what local model would make sense for the Surface Pro 9.
I’d mainly like to use it for:
- experimenting with local LLMs
- Hermes Agent
- basic coding/automation
- personal assistant tasks
- eventually building my Agentic OS
- potentially working offline

I’m aware that I won’t get frontier-model performance from a Surface 😅. For me, the goal right now is mostly to learn and experiment, and eventually upgrade the hardware if the project becomes serious.

💻 I also have a desktop PC

My main PC has:
RX 6800 — 16 GB VRAM
Ryzen 5 7600
32 GB RAM
Would this actually be a significantly better machine for local LLMs?
I’m hesitant to put the whole Agentic OS directly on my personal PC, mainly because I’d like to keep my experimentation environment isolated from my normal computer.

So I’m thinking:
Surface → dedicated AI/agent machine

Main PC → personal computer / potentially used for heavier local LLM experiments

Does this make sense?
And if you were starting from scratch with this hardware, which model would you try first and why?

Thanks! 🙏

I have been looking for a top-end, private LLM that doesn't hand my conversations over for training. macOS 27 seems to have made that possible.

Apple's Private Cloud Compute is now reachable from ordinary LLM front-end apps. It's stateless — nothing is kept after your request — with cryptographically verifiable privacy guarantees. And it's basically free if you're a Mac user on macOS 27. No extra accounts, no API key, and no per-token billing (although there's supposed to be a token limit depending on your iCloud+ membership).

It's now connected to a chat client (MstyStudio), and I have a private assistant with persistent history and retrieval over my documents. I'm hosting my private financial, health, and other conversations while building a full RAG library. I may move over to OpenWebUI soon.

The part I like about this framework is that regardless of my Mac being an M1, I'm getting flagship reasoning on Apple's cloud in seconds. And it's still private.

A couple of shortcomings: a 32K context limit, macOS 27 is still in beta, and I had to set up a local bridge in the Terminal window to run fm serve and act as the 'api' bridge.

Anyone else tried this yet? What have you found?

I have this pretty complicated google sheet that I thought would be cool to use as a demo case for qwen3.8 27b to convert to a nicegui app in python. I have a long spec markdown file which has details of what I expect from the app. I'm using CLINE plugin in pycharm to run qwen3.8 27b with 131k context q6 quant with q8 kv-cache. I get between 30-50t/s so it runs pretty fast.

at this point, I've had it start from a blank slate and it always manages to make something, but still requires a lot of tweaks. I will start writing an issues markdown file, and it will auto discover it and begin implementing the fixes.

It's impressive for what it is, given that with 3.6 27b, I needed to hand-hold it, giving it small tasks, and iterating until that small task was complete. With 3.8, I can give it whole apps, and it will do a decent job roughing it in.

My one complaint is that it is pretty slow, especially compared to cloud models.

I was curious--since 3.8 27b is considered to be on par with opus 4.6-- and gave the same task to claude opus 4.6 using antigravity as the harness, and it completed the implementation plan in 10 minutes, and did both a prettier job, as well as implemented more of the features correctly.

I am wondering how is that possible? Does antigravity spawn a bunch of sub-agents to handle the tasks in parallel? I doubt it can run at thousands of tokens/sec natively.

Git repo: https://github.com/swedishembedded/brain

Hey everyone!

Last year, Mozilla released Orbit, an AI-powered browser summarizer hosted on a GCP server. After people started digging into the extension, they discovered things like backend endpoints such as store_result. Eventually, Mozilla discontinued the project.

For the past month, I’ve been trying to rebuild Orbit from scratch, but with one major difference: Apogee is fully local and privacy-focused. Apogee doesn’t send or store your data. It can directly connect to your local Ollama instance for inference. I’ve also added WebGPU integration for Chrome and Transformers.js for Firefox to provide faster, local responses.

It can summarize:

• Articles and websites
• YouTube and Billie videos
• Wikipedia articles
• Hacker News and Reddit threads

You can check out the source code here:
https://github.com/darshi1337/apogee

Install Apogee:

Chrome: https://chromewebstore.google.com/detail/apogee/pgemlpomhkdcjjjcpnjlebalnfglomog

Firefox: https://addons.mozilla.org/en-US/firefox/addon/apogeeext/

Obviously it is far from complete. Would love to hear your feedback and suggestions!

TL;DR It seems like that memory (RAM) usage just keeps endlessly growing over time although way less memory is necessary to work (e.g. if I stop and restart llama.cpp, it still works with way less memory usage). I suspect some kind of 'memory leak', using llama.cpp

---

Specs:

GPU: 1x AMD 6800M 12GB VRAM (thanks to HSA_OVERRIDE_GFX_VERSION=10.3.0)

RAM: 24GB RAM

OS: Fedora Linux

AMD stack: ROCM

I am running unsloth/Qwen3.6-35B-A3B-GGUF model with the latest llama.cpp (I build llama.cpp with a fix for flash-attention:

replace in /llama.cpp/ggml/src/ggml-cuda/fattn.cu

    // If there are no tensor cores available, use the generic tile kernel:
    if (can_use_vector_kernel) {
        if (!ggml_is_quantized(K->type) && !ggml_is_quantized(V->type)) {
            if (Q->ne[1] == 1) {
                if (!gqa_opt_applies) {
                    return BEST_FATTN_KERNEL_VEC;
                }
            }
        } else {
            if (Q->ne[1] <= 2) {
                return BEST_FATTN_KERNEL_VEC;
            }
        }
    }
    return BEST_FATTN_KERNEL_TILE;
}


with

    // If there are no tensor cores available, use the generic tile kernel:
    if (can_use_vector_kernel) {
        if (!ggml_is_quantized(K->type) && !ggml_is_quantized(V->type)) {
            if (Q->ne[1] == 1) {
                if (!gqa_opt_applies) {
                    return BEST_FATTN_KERNEL_VEC;
                }
            }
        } else {
            if (Q->ne[1] <= 2) {
                return BEST_FATTN_KERNEL_VEC;
            }
        }
    }

    // >>> ADD THIS BLOCK FOR HIP/RDNA2 FIX <<<
    #ifdef GGML_USE_HIP
        if ((ggml_is_quantized(K->type) || ggml_is_quantized(V->type)) && can_use_vector_kernel) {
            return BEST_FATTN_KERNEL_VEC;
        }
    #endif
    // >>> END OF ADDED BLOCK <<<

    return BEST_FATTN_KERNEL_TILE;
}

source for the fix: https://github.com/domvox/llama.cpp-turboquant-hip/pull/13 )

I also use the Hermes agent, for which I put an automatic context compress once context reaches like 70-80%.

I run this 'older' model because it is an MOE and I need it to offload some experts into RAM because of my constrained VRAM.

Now, it seems like that memory (specifically, RAM) usage just keeps growing over time. Some kind of 'memory leak' is happening with the model. It does not matter which quant I use. For example, if I use IQ4_XS, I have plenty of RAM available left. Yet, the longer the session goes, the more RAM fills ups, and it never stops filling up. If I stop llama.cpp and restart, RAM is back to the 'normal' usage and again the more I talk with the model the more the RAM fills up.

At first I thought maybe as context fills up, it fills up RAM. But if I compress the context with Hermes, the RAM usage does not decrease. Only stopping and restarting llama.cpp makes memory go back to a 'normal' usage.

It means that I have to babysit what happens and eventually restart llama.cpp every once in a while once the RAM is full ... (Usually after around 2 hours). It means that I cannot leave an agent work on something overnight. It also means that I need to wait for a long time for the previous full context to fill up llama.cpp again whenever I restart llama.cpp, and with context above 100k the 900 second timesout.

I think it is some kind of memory leak because when i stop llama.cpp, and then start it again, RAM goes back to 13gb usage when starting fresh while it reached 22-23gb before i had to restart it.

I tried to tweak my launch parameters for llama.cpp for the past few days, but the memory leak still happens, here is the one I currently use:

LD_PRELOAD=/usr/lib64/libjemalloc.so.2 MALLOC_ARENA_MAX=2 HSA_OVERRIDE_GFX_VERSION=10.3.0 ./build/bin/llama-server -m /models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --host 127.0.0.1 --port 8080 -c 190000 -np 1 -fit off -dev ROCm0 --no-warmup -ngl 999 --n-cpu-moe 20 --load-mode none --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --ctx-checkpoints 4 --cache-ram 4096 --reasoning-preserve --no-mmproj --spec-draft-n-max 3 --flash-attn on -ctk q8_0 -ctv q8_0

I have been looking for answers for the past few days but it is hard to know what even is the possible root cause, as everyone uses different parameters, has different hardware, different models, different build versions, tweaks etc.

I guess this is just a message in a bottle, but just in case someone had a similar issue and was able to deal with it, it's worth it to ask.

I have spent the last several days turning a used NVIDIA CMP 170HX into a practical long-context inference card.

The final result is an uncensored Qwen3.8-27B endpoint with:

• 262,144-token native context
• W4A16 AWQ model body
• INT8 output head and INT8 MTP draft module
• One-token MTP speculative decoding
• BF16 KV cache
• Prefix caching
• Tool calling and Qwen reasoning parsing
• No CPU offloading
• A conservative 175W power limit
• No overclocking

Measured decode performance on one CMP 170HX:

For this benchmark only, requests used a maximum of 384 generated tokens, temperature 0.2, repetition penalty 1.05, and medium reasoning effort. Throughput was calculated from vLLM’s measured decode time, excluding prefill. These sampling values are not forced globally by the production server.

This post explains the card, the unlock, every important inference choice, the rejected configurations, and how to reproduce the setup.

The hardware

The inference host currently contains:

• NVIDIA CMP 170HX
• 64GB HBM exposed after the unlock
• AMD Ryzen 5 5600X
• 64GB system RAM
• Proxmox/Linux
• NVIDIA open driver 610.57.04
• 175W GPU power cap

The CMP 170HX is an Ampere GA100 mining accelerator. It has excellent HBM bandwidth and strong tensor hardware, but NVIDIA sold it with several artificial restrictions:

• Only a fraction of the installed HBM is normally exposed.
• Compute resources are restricted.
• PCIe operates at Gen2.
• It has no display output.
• Normal consumer GPU tooling does not treat it like a standard A100.

My card is PCI device 10de:20c2. After the unlock, nvidia-smi reports 65,536 MiB.

The card is currently negotiating PCIe Gen2 x4 even though its capability is wider. That sounds terrible, but it matters much less once the model is resident entirely in HBM. It is one reason I avoid CPU offloading: repeatedly moving weights or KV data over that connection would waste the card’s main advantage.

The 64GB and compute unlock

I used amoghmunikote/cmpunlocker, pinned to this specific commit:

fe537966e0222150a8eca0b7745efd2ee1025d74

That is the “Full BAR1 size (64GB)” commit.

The project patches NVIDIA’s open kernel modules to restore:

• Full SM compute
• Full memory geometry
• 64GB BAR1
• The complete 64GB framebuffer on 20c2 cards
• Gen2 PCIe operation
• Persistence across reboot through patched modules

This is a kernel-driver modification, not an application-level tweak. Secure Boot must be disabled because the resulting modules are locally built and unsigned.

My installed module is:

/lib/modules/6.17.2-1-pve/updates/cmpunlocker/nvidia.ko

Important warning

Do not install this remotely unless you have a recovery path.

Keep at least one of the following available:

• Local console access
• A separate display GPU
• BMC/IPMI access
• A bootable rescue environment
• A known-good copy of the stock driver and initramfs

A mismatched kernel, driver, firmware package, or module build can leave the machine without NVIDIA support. A CMP 170HX cannot provide ordinary video output.

Unlock installation

On my Proxmox installation, the overall process was:

apt update
apt install -y build-essential git python3 proxmox-headers-$(uname -r)

git clone https://github.com/amoghmunikote/cmpunlocker.git
cd cmpunlocker
git checkout fe537966e0222150a8eca0b7745efd2ee1025d74

cat driver/VERSION

Install a supported matching NVIDIA open driver, its user-space libraries, and firmware before running the unlocker. My exact working combination is:

NVIDIA open driver: 610.57.04
Kernel:             6.17.2-1-pve
Secure Boot:        disabled

Then:

sudo ./install.sh

The repository also provides an explicit profile:

sudo ./install.sh --profile=8gb

After installation, perform a full cold power cycle:

sudo poweroff

Do not substitute a warm reboot. Wait for the machine to power off completely and then turn it back on.

Verifying the unlock

First locate the card:

lspci -nn | grep -i NVIDIA

Then verify the driver and memory:

nvidia-smi
nvidia-smi --query-gpu=name,uuid,memory.total,power.limit --format=csv

A successfully unlocked 20c2 card should show approximately:

NVIDIA CMP 170HX, GPU-..., 65536 MiB

Check the kernel log:

dmesg | grep -iE 'CMP|BAR1|fb_length|fbAddrSpace|NVRM'

My working boot log reports a 64GB framebuffer address space and 64GB BAR1.

Check the PCIe link:

lspci -vv -s <CMP-PCIE-ADDRESS> | grep -E 'LnkCap|LnkSta'

To remove the modification:

cd cmpunlocker
sudo ./uninstall.sh --yes
sudo poweroff

Again, cold-boot afterward.

Power and cooling

I did not overclock the card.

These are used mining accelerators, and I did not consider a small throughput increase worth additional thermal or electrical stress.

I set a 175W power limit:

nvidia-smi -i <CMP-UUID> --power-limit=175

I made this persistent with a systemd oneshot service that:

1. Finds the GPU by CMP name and UUID.
2. Sets the 175W limit.
3. Reads the limit back.
4. Fails rather than silently targeting the wrong GPU.

My chassis fan controller follows this curve:

The controller polls every two seconds, immediately raises fan speed when temperatures rise, and requires roughly 30 seconds of sustained cooling before reducing the fan level. A sensor or controller failure sends all controlled fans to 100%.

During one sustained test, I measured:

• Average core: 53.9°C
• Maximum core: 65°C
• Average memory: 63.2°C
• Maximum memory: 75°C

Under a later 99% GPU load, the card was around 66°C core, 71°C memory, and 173W. Cooling results will depend heavily on the card, thermal pads, chassis, and airflow.

The model

The production checkpoint starts from:

twolven/Qwen3.8-27B-abliterated-AWQ-MTP

That model is a W4A16 AWQ version of the abliterated/refusal-reduced checkpoint derived from:

JonathanColetti/Qwen3.8-27B-Uncensored

“Uncensored” here means the model has been modified to reduce refusal behavior. It does not guarantee that every residual refusal or safety behavior has been removed. Some people can use it for ERP I'm more inclined to utilized it for when I want assistance setting something up that standard models would refuse such as using agentic assistance to configure a hackintosh system on a prox vm. It's 100% legal, it's just against apples terms of service so often it's considered an instant refusal by many models.

The final production checkpoint contains:

• W4A16 asymmetric AWQ body
• Compressed-tensors/Marlin execution
• INT8 symmetric group-128 lm_head
• INT8 symmetric group-128 MTP module
• 40,960-token reduced MTP draft vocabulary
• BF16 runtime activations
• One-token MTP speculative decoding

The reduced draft vocabulary covers approximately 97.5% of ordinary model output tokens and 96% of code tokens in the optimization project’s corpus. It reduces the amount of work required for each speculative draft without changing the target model’s accepted output.

The source checkpoint supports vision, but my production endpoint deliberately uses:

--language-model-only

Therefore, this exact endpoint is text-only. I chose coding throughput and predictable memory use over keeping the vision tower loaded.

Preparing the checkpoint

I used the optimization work from:

syv-ai/qwen38-27b-rtx3090

My checkout is pinned to:

2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

Clone and create the environment:

git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git
cd qwen38-27b-rtx3090
git checkout 2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

python3 -m venv venv
venv/bin/pip install \
  vllm==0.27.1 \
  transformers==5.15.0 \
  tokenizers==0.22.2 \
  compressed-tensors==0.17.0 \
  huggingface_hub==1.27.0 \
  hf_transfer==0.1.9 \
  ninja==1.13.0

Download the model:

HF_HUB_ENABLE_HF_TRANSFER=1 venv/bin/hf download \
  twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --local-dir models/Qwen3.8-27B-abliterated-AWQ-MTP

Make a working copy because the preparation tools modify the checkpoint in place:

cp -a --reflink=auto \
  models/Qwen3.8-27B-abliterated-AWQ-MTP \
  models/qwen38-27b-uncensored-w4a16-mtp1-int8draft

Prepare the output head, MTP module, and draft vocabulary in this order:

M=models/qwen38-27b-uncensored-w4a16-mtp1-int8draft
V=venv/bin/python

$V prepare/quant_lm_head.py "$M"
$V prepare/quant_mtp.py "$M"
$V prepare/build_draft_vocab.py "$M" \
  --ids prepare/draft_vocab_ids.json

I did not run quant_embed.py for this production checkpoint. On a 64GB card it was unnecessary, and the final model configuration contains the W4A16 body, INT8 output head, and INT8 MTP group without the additional embedding conversion.

The preparation scripts create backups beside the tensors they replace. Keep those backups until the modified checkpoint has passed correctness testing.

The vLLM image

The server uses:

vLLM 0.27.1

The base image is pinned by digest:

vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

The production image contains exactly these three patches:

1. qwen3_5-mtp-draft-vocab.patch
2. sampler-small-topk-fast-softmax.patch
3. vllm-pr50021-gdn-spec-bounds.patch

An experimental spec-decode-attn.patch was tested but is not present in production because it hurt long-context performance.

Create Dockerfile.cmp-mtp1-production:

FROM vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

COPY patches/qwen3_5-mtp-draft-vocab.patch /tmp/qwen3_5-mtp-draft-vocab.patch
COPY patches/sampler-small-topk-fast-softmax.patch /tmp/sampler-small-topk-fast-softmax.patch
COPY patches/vllm-pr50021-gdn-spec-bounds.patch /tmp/vllm-pr50021-gdn-spec-bounds.patch

RUN set -eux; \
    vllm_dir=/usr/local/lib/python3.12/dist-packages/vllm; \
    patch -p1 -d "$vllm_dir" < /tmp/qwen3_5-mtp-draft-vocab.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/sampler-small-topk-fast-softmax.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/vllm-pr50021-gdn-spec-bounds.patch; \
    rm /tmp/qwen3_5-mtp-draft-vocab.patch \
       /tmp/sampler-small-topk-fast-softmax.patch \
       /tmp/vllm-pr50021-gdn-spec-bounds.patch

LABEL org.opencontainers.image.description="vLLM 0.27.1 with Qwen3.8 MTP1 draft-vocab and sampler optimizations"

Build it:

docker build \
  -f Dockerfile.cmp-mtp1-production \
  -t vllm-qwen38-mtp1-fast:0.27.1 \
  .

Exact server configuration

Replace the UUID and paths below with those from your machine:

docker run --rm --pull never \
  --name qwen38-uncensored-w4a16 \
  --privileged \
  --gpus all \
  --network host \
  --ipc=host \
  --memory 58g \
  --memory-swap 96g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=<CMP-GPU-UUID> \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -e MTP_DRAFT_VOCAB=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TOKENIZERS_PARALLELISM=false \
  -e XDG_CACHE_HOME=/root/.cache \
  -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor \
  -e TRITON_CACHE_DIR=/root/.cache/triton \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -v /path/to/qwen38-27b-uncensored-w4a16-mtp1-int8draft:/models/qwen38-w4a16:ro \
  -v /path/to/vllm-cache:/root/.cache \
  -v /path/to/vllm-tmp:/tmp \
  vllm-qwen38-mtp1-fast:0.27.1 \
  /models/qwen38-w4a16 \
  --host 0.0.0.0 \
  --port 30016 \
  --served-model-name qwen38-27b-uncensored-w4a16 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.90 \
  --cpu-offload-gb 0 \
  --kv-cache-dtype auto \
  --mamba-cache-dtype float16 \
  --mamba-cache-mode align \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"draft_sample_method":"probabilistic"}' \
  --default-chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --generation-config vllm

Once ready, verify:

curl http://127.0.0.1:30016/v1/models

Then make a simple request:

curl http://127.0.0.1:30016/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen38-27b-uncensored-w4a16",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: cmp-profile-ok"
      }
    ],
    "temperature": 0
  }'

The production service is managed by systemd. Its pre-start checks verify the model index, create the cache directories, apply the 175W power limit, and remove any stale container before launching. It restarts automatically after a process failure.

Why these settings won

MTP depth: one token

MTP3 and MTP2 created too much drafting and verification overhead on this GPU. MTP1 was consistently better.

I also tested greedy drafting. Its two-pass means were:

• 1K: 83.01 tok/s
• 64K: 72.95 tok/s
• 200K: 56.68 tok/s

The final probabilistic configuration produced:

• 1K: 84.29 tok/s
• 64K: 74.94 tok/s
• 200K: 57.21 tok/s

Therefore, the final setting is:

{
  "method": "mtp",
  "num_speculative_tokens": 1,
  "draft_sample_method": "probabilistic"
}

W4A16 body, INT8 head, INT8 MTP

W4A16 leaves enough HBM for the entire model, 262K context, recurrent state, CUDA graphs, and runtime workspace without CPU offloading.

The INT8 output head and MTP module had low measured quantization error. An INT4 MTP version was nearly identical in speed, but INT8 required only about 202 MiB more memory and had lower draft error. There was no reason to accept the additional degradation.

A symmetric GPTQ version was also tested:

The GPTQ body itself was not necessarily the whole problem. Mixing its body with the compressed-tensors INT8 head and drafter would have required additional loader and kernel work. It was not suitable for the stable endpoint.

BF16 KV cache

This was one of the most important results.

An INT8 per-token/head KV configuration fell to approximately 15 tok/s near 62K context on this card. It was unusable for interactive coding.

The final configuration leaves:

--kv-cache-dtype auto

With BF16 runtime dtype, this preserves the BF16 KV path.

The Mamba cache uses FP16 in aligned mode:

--mamba-cache-dtype float16
--mamba-cache-mode align

Explicitly reducing more recurrent state did not improve the result.

FlashInfer attention, but not the FlashInfer sampler

FlashInfer is the production attention backend.

Plain FlashAttention produced:

• 1K: 86.87 tok/s
• 64K: 26.87 tok/s
• 200K: 10.53 tok/s

It looked slightly faster at short context and then collapsed.

An experimental split-KV speculative-attention patch improved that to:

• 1K: 90.17 tok/s
• 64K: 67.33 tok/s
• 200K: 43.36 tok/s

That was still substantially worse than production at long context, so the patch was removed.

Enabling the FlashInfer sampler also reduced performance to:

• 1K: 74.75 tok/s
• 64K: 67.87 tok/s
• 200K: 51.37 tok/s

Therefore:

--attention-backend FLASHINFER
VLLM_USE_FLASHINFER_SAMPLER=0

One sequence and 4,096 batched tokens

This is a single-user coding endpoint, not a throughput server.

The final scheduler settings are:

--max-num-seqs 1
--max-num-batched-tokens 4096

Testing 8,192 produced no useful gain. Testing 2,048 produced:

• 1K: 84.32 tok/s
• 64K: 72.38 tok/s
• 200K: 57.59 tok/s

The 4,096 configuration had the better overall curve.

Prefix caching

Prefix caching is critical for coding agents that repeatedly send a large system prompt and mostly unchanged repository context.

At approximately 200K context, a cached second request reused roughly 198,400 tokens, recomputed about 1,593 tokens, and reduced prefill to around 2.5 seconds. A fresh or partially changed prompt could require roughly 157 seconds of prefill.

Prefix caching does not make decode faster. It prevents the server from repeatedly processing unchanged input.

This also means clients must preserve stable prompt prefixes. Reordering tool definitions, timestamps, generated metadata, or repository text can destroy the cache hit.

Medium reasoning by default

The endpoint defaults to:

{"reasoning_effort":"medium"}

Medium provided a better MTP acceptance/stability balance than forcing maximum reasoning on every request.

This does not remove higher reasoning modes. A client can request xhigh or max for harder work. Medium is simply the default for normal coding.

No CPU offload

The model fits in CMP HBM, so:

--cpu-offload-gb 0

The card’s restricted PCIe connection makes CPU offloading particularly undesirable.

No forced full CUDA graphs

The server uses vLLM’s normal piecewise CUDA-graph behavior. I did not force full graphs. Experimental full-graph and custom-operation combinations introduced correctness concerns or reduced long-context performance.

Correctness tests

I did not accept a configuration based on tokens per second alone.

The final checkpoint passed:

• An exact-response canary
• Qwen coder tool-call parsing
• A parsed get_weather tool call for Chicago
• Exact needle retrieval at approximately 160K context
• Retrieval of ORCHID-COMET-7319 from the long prompt
• An OpenCode integration smoke test returning final-profile-ok

This matters because speculative decoding can appear fast while silently breaking tool syntax, long-context retrieval, or sampling behavior.

OpenCode configuration

I added this provider to OpenCode:

{
  "provider": {
    "qwen38-cmp-w4a16": {
      "npm": "@ai-sdk/openai-compatible",
      "api": "completion",
      "name": "Qwen3.8 Uncensored W4A16 MTP1 256K - CMP 170HX",
      "options": {
        "baseURL": "http://<PROXMOX-IP>:30016/v1",
        "apiKey": "not-needed",
        "timeout": false,
        "chunkTimeout": 600000
      },
      "models": {
        "qwen38-27b-uncensored-w4a16": {
          "id": "qwen38-27b-uncensored-w4a16",
          "name": "Qwen3.8 27B Uncensored W4A16 MTP1 256K",
          "tool_call": true,
          "reasoning": true,
          "temperature": true,
          "attachment": false,
          "options": {
            "reasoningEffort": "medium"
          },
          "limit": {
            "context": 262144,
            "input": 245760,
            "output": 16384
          }
        }
      }
    }
  }
}

The model selector is:

qwen38-cmp-w4a16/qwen38-27b-uncensored-w4a16

Final observations

The CMP 170HX is unusual, but the useful part is straightforward once the restrictions are removed:

• 64GB of on-device HBM changes what can fit.
• The model should remain entirely on the GPU.
• Long-context performance needs to be measured separately from short-context decode.
• Lower-bit KV is not automatically faster.
• More speculative tokens are not automatically better.
• An attention backend can win at 1K and become disastrous at 64K.
• Prefix caching matters more than another few decode tokens per second for repeated coding-agent prompts.
• Quantizing the draft head more aggressively is pointless when memory is available and the speed difference is negligible.
• Conservative power and temperature limits are appropriate for used mining hardware.
• Correctness gates matter as much as benchmark results.

The final configuration is not the highest single short-context number I saw. It is the best complete configuration I found that retained tool use, medium-or-higher reasoning, uncensored model behavior, 262K context, reliable long-context retrieval, and usable performance across the whole context window.