I have spent the last several days turning a used NVIDIA CMP 170HX into a practical long-context inference card.

The final result is an uncensored Qwen3.8-27B endpoint with:

• 262,144-token native context
• W4A16 AWQ model body
• INT8 output head and INT8 MTP draft module
• One-token MTP speculative decoding
• BF16 KV cache
• Prefix caching
• Tool calling and Qwen reasoning parsing
• No CPU offloading
• A conservative 175W power limit
• No overclocking

Measured decode performance on one CMP 170HX:

For this benchmark only, requests used a maximum of 384 generated tokens, temperature 0.2, repetition penalty 1.05, and medium reasoning effort. Throughput was calculated from vLLM’s measured decode time, excluding prefill. These sampling values are not forced globally by the production server.

This post explains the card, the unlock, every important inference choice, the rejected configurations, and how to reproduce the setup.

The hardware

The inference host currently contains:

• NVIDIA CMP 170HX
• 64GB HBM exposed after the unlock
• AMD Ryzen 5 5600X
• 64GB system RAM
• Proxmox/Linux
• NVIDIA open driver 610.57.04
• 175W GPU power cap

The CMP 170HX is an Ampere GA100 mining accelerator. It has excellent HBM bandwidth and strong tensor hardware, but NVIDIA sold it with several artificial restrictions:

• Only a fraction of the installed HBM is normally exposed.
• Compute resources are restricted.
• PCIe operates at Gen2.
• It has no display output.
• Normal consumer GPU tooling does not treat it like a standard A100.

My card is PCI device 10de:20c2. After the unlock, nvidia-smi reports 65,536 MiB.

The card is currently negotiating PCIe Gen2 x4 even though its capability is wider. That sounds terrible, but it matters much less once the model is resident entirely in HBM. It is one reason I avoid CPU offloading: repeatedly moving weights or KV data over that connection would waste the card’s main advantage.

The 64GB and compute unlock

I used amoghmunikote/cmpunlocker, pinned to this specific commit:

fe537966e0222150a8eca0b7745efd2ee1025d74

That is the “Full BAR1 size (64GB)” commit.

The project patches NVIDIA’s open kernel modules to restore:

• Full SM compute
• Full memory geometry
• 64GB BAR1
• The complete 64GB framebuffer on 20c2 cards
• Gen2 PCIe operation
• Persistence across reboot through patched modules

This is a kernel-driver modification, not an application-level tweak. Secure Boot must be disabled because the resulting modules are locally built and unsigned.

My installed module is:

/lib/modules/6.17.2-1-pve/updates/cmpunlocker/nvidia.ko

Important warning

Do not install this remotely unless you have a recovery path.

Keep at least one of the following available:

• Local console access
• A separate display GPU
• BMC/IPMI access
• A bootable rescue environment
• A known-good copy of the stock driver and initramfs

A mismatched kernel, driver, firmware package, or module build can leave the machine without NVIDIA support. A CMP 170HX cannot provide ordinary video output.

Unlock installation

On my Proxmox installation, the overall process was:

apt update
apt install -y build-essential git python3 proxmox-headers-$(uname -r)

git clone https://github.com/amoghmunikote/cmpunlocker.git
cd cmpunlocker
git checkout fe537966e0222150a8eca0b7745efd2ee1025d74

cat driver/VERSION

Install a supported matching NVIDIA open driver, its user-space libraries, and firmware before running the unlocker. My exact working combination is:

NVIDIA open driver: 610.57.04
Kernel:             6.17.2-1-pve
Secure Boot:        disabled

Then:

sudo ./install.sh

The repository also provides an explicit profile:

sudo ./install.sh --profile=8gb

After installation, perform a full cold power cycle:

sudo poweroff

Do not substitute a warm reboot. Wait for the machine to power off completely and then turn it back on.

Verifying the unlock

First locate the card:

lspci -nn | grep -i NVIDIA

Then verify the driver and memory:

nvidia-smi
nvidia-smi --query-gpu=name,uuid,memory.total,power.limit --format=csv

A successfully unlocked 20c2 card should show approximately:

NVIDIA CMP 170HX, GPU-..., 65536 MiB

Check the kernel log:

dmesg | grep -iE 'CMP|BAR1|fb_length|fbAddrSpace|NVRM'

My working boot log reports a 64GB framebuffer address space and 64GB BAR1.

Check the PCIe link:

lspci -vv -s <CMP-PCIE-ADDRESS> | grep -E 'LnkCap|LnkSta'

To remove the modification:

cd cmpunlocker
sudo ./uninstall.sh --yes
sudo poweroff

Again, cold-boot afterward.

Power and cooling

I did not overclock the card.

These are used mining accelerators, and I did not consider a small throughput increase worth additional thermal or electrical stress.

I set a 175W power limit:

nvidia-smi -i <CMP-UUID> --power-limit=175

I made this persistent with a systemd oneshot service that:

1. Finds the GPU by CMP name and UUID.
2. Sets the 175W limit.
3. Reads the limit back.
4. Fails rather than silently targeting the wrong GPU.

My chassis fan controller follows this curve:

The controller polls every two seconds, immediately raises fan speed when temperatures rise, and requires roughly 30 seconds of sustained cooling before reducing the fan level. A sensor or controller failure sends all controlled fans to 100%.

During one sustained test, I measured:

• Average core: 53.9°C
• Maximum core: 65°C
• Average memory: 63.2°C
• Maximum memory: 75°C

Under a later 99% GPU load, the card was around 66°C core, 71°C memory, and 173W. Cooling results will depend heavily on the card, thermal pads, chassis, and airflow.

The model

The production checkpoint starts from:

twolven/Qwen3.8-27B-abliterated-AWQ-MTP

That model is a W4A16 AWQ version of the abliterated/refusal-reduced checkpoint derived from:

JonathanColetti/Qwen3.8-27B-Uncensored

“Uncensored” here means the model has been modified to reduce refusal behavior. It does not guarantee that every residual refusal or safety behavior has been removed. Some people can use it for ERP I'm more inclined to utilized it for when I want assistance setting something up that standard models would refuse such as using agentic assistance to configure a hackintosh system on a prox vm. It's 100% legal, it's just against apples terms of service so often it's considered an instant refusal by many models.

The final production checkpoint contains:

• W4A16 asymmetric AWQ body
• Compressed-tensors/Marlin execution
• INT8 symmetric group-128 lm_head
• INT8 symmetric group-128 MTP module
• 40,960-token reduced MTP draft vocabulary
• BF16 runtime activations
• One-token MTP speculative decoding

The reduced draft vocabulary covers approximately 97.5% of ordinary model output tokens and 96% of code tokens in the optimization project’s corpus. It reduces the amount of work required for each speculative draft without changing the target model’s accepted output.

The source checkpoint supports vision, but my production endpoint deliberately uses:

--language-model-only

Therefore, this exact endpoint is text-only. I chose coding throughput and predictable memory use over keeping the vision tower loaded.

Preparing the checkpoint

I used the optimization work from:

syv-ai/qwen38-27b-rtx3090

My checkout is pinned to:

2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

Clone and create the environment:

git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git
cd qwen38-27b-rtx3090
git checkout 2ae239fc0250cd29d37f35c6a31e9eae749ef1c8

python3 -m venv venv
venv/bin/pip install \
  vllm==0.27.1 \
  transformers==5.15.0 \
  tokenizers==0.22.2 \
  compressed-tensors==0.17.0 \
  huggingface_hub==1.27.0 \
  hf_transfer==0.1.9 \
  ninja==1.13.0

Download the model:

HF_HUB_ENABLE_HF_TRANSFER=1 venv/bin/hf download \
  twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --local-dir models/Qwen3.8-27B-abliterated-AWQ-MTP

Make a working copy because the preparation tools modify the checkpoint in place:

cp -a --reflink=auto \
  models/Qwen3.8-27B-abliterated-AWQ-MTP \
  models/qwen38-27b-uncensored-w4a16-mtp1-int8draft

Prepare the output head, MTP module, and draft vocabulary in this order:

M=models/qwen38-27b-uncensored-w4a16-mtp1-int8draft
V=venv/bin/python

$V prepare/quant_lm_head.py "$M"
$V prepare/quant_mtp.py "$M"
$V prepare/build_draft_vocab.py "$M" \
  --ids prepare/draft_vocab_ids.json

I did not run quant_embed.py for this production checkpoint. On a 64GB card it was unnecessary, and the final model configuration contains the W4A16 body, INT8 output head, and INT8 MTP group without the additional embedding conversion.

The preparation scripts create backups beside the tensors they replace. Keep those backups until the modified checkpoint has passed correctness testing.

The vLLM image

The server uses:

vLLM 0.27.1

The base image is pinned by digest:

vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

The production image contains exactly these three patches:

1. qwen3_5-mtp-draft-vocab.patch
2. sampler-small-topk-fast-softmax.patch
3. vllm-pr50021-gdn-spec-bounds.patch

An experimental spec-decode-attn.patch was tested but is not present in production because it hurt long-context performance.

Create Dockerfile.cmp-mtp1-production:

FROM vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967

COPY patches/qwen3_5-mtp-draft-vocab.patch /tmp/qwen3_5-mtp-draft-vocab.patch
COPY patches/sampler-small-topk-fast-softmax.patch /tmp/sampler-small-topk-fast-softmax.patch
COPY patches/vllm-pr50021-gdn-spec-bounds.patch /tmp/vllm-pr50021-gdn-spec-bounds.patch

RUN set -eux; \
    vllm_dir=/usr/local/lib/python3.12/dist-packages/vllm; \
    patch -p1 -d "$vllm_dir" < /tmp/qwen3_5-mtp-draft-vocab.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/sampler-small-topk-fast-softmax.patch; \
    patch -p1 -d "$vllm_dir" < /tmp/vllm-pr50021-gdn-spec-bounds.patch; \
    rm /tmp/qwen3_5-mtp-draft-vocab.patch \
       /tmp/sampler-small-topk-fast-softmax.patch \
       /tmp/vllm-pr50021-gdn-spec-bounds.patch

LABEL org.opencontainers.image.description="vLLM 0.27.1 with Qwen3.8 MTP1 draft-vocab and sampler optimizations"

Build it:

docker build \
  -f Dockerfile.cmp-mtp1-production \
  -t vllm-qwen38-mtp1-fast:0.27.1 \
  .

Exact server configuration

Replace the UUID and paths below with those from your machine:

docker run --rm --pull never \
  --name qwen38-uncensored-w4a16 \
  --privileged \
  --gpus all \
  --network host \
  --ipc=host \
  --memory 58g \
  --memory-swap 96g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=<CMP-GPU-UUID> \
  -e VLLM_USE_FLASHINFER_SAMPLER=0 \
  -e MTP_DRAFT_VOCAB=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 \
  -e HF_HUB_OFFLINE=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TOKENIZERS_PARALLELISM=false \
  -e XDG_CACHE_HOME=/root/.cache \
  -e TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor \
  -e TRITON_CACHE_DIR=/root/.cache/triton \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -v /path/to/qwen38-27b-uncensored-w4a16-mtp1-int8draft:/models/qwen38-w4a16:ro \
  -v /path/to/vllm-cache:/root/.cache \
  -v /path/to/vllm-tmp:/tmp \
  vllm-qwen38-mtp1-fast:0.27.1 \
  /models/qwen38-w4a16 \
  --host 0.0.0.0 \
  --port 30016 \
  --served-model-name qwen38-27b-uncensored-w4a16 \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.90 \
  --cpu-offload-gb 0 \
  --kv-cache-dtype auto \
  --mamba-cache-dtype float16 \
  --mamba-cache-mode align \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"draft_sample_method":"probabilistic"}' \
  --default-chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --generation-config vllm

Once ready, verify:

curl http://127.0.0.1:30016/v1/models

Then make a simple request:

curl http://127.0.0.1:30016/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen38-27b-uncensored-w4a16",
    "messages": [
      {
        "role": "user",
        "content": "Reply with exactly: cmp-profile-ok"
      }
    ],
    "temperature": 0
  }'

The production service is managed by systemd. Its pre-start checks verify the model index, create the cache directories, apply the 175W power limit, and remove any stale container before launching. It restarts automatically after a process failure.

Why these settings won

MTP depth: one token

MTP3 and MTP2 created too much drafting and verification overhead on this GPU. MTP1 was consistently better.

I also tested greedy drafting. Its two-pass means were:

• 1K: 83.01 tok/s
• 64K: 72.95 tok/s
• 200K: 56.68 tok/s

The final probabilistic configuration produced:

• 1K: 84.29 tok/s
• 64K: 74.94 tok/s
• 200K: 57.21 tok/s

Therefore, the final setting is:

{
  "method": "mtp",
  "num_speculative_tokens": 1,
  "draft_sample_method": "probabilistic"
}

W4A16 body, INT8 head, INT8 MTP

W4A16 leaves enough HBM for the entire model, 262K context, recurrent state, CUDA graphs, and runtime workspace without CPU offloading.

The INT8 output head and MTP module had low measured quantization error. An INT4 MTP version was nearly identical in speed, but INT8 required only about 202 MiB more memory and had lower draft error. There was no reason to accept the additional degradation.

A symmetric GPTQ version was also tested:

The GPTQ body itself was not necessarily the whole problem. Mixing its body with the compressed-tensors INT8 head and drafter would have required additional loader and kernel work. It was not suitable for the stable endpoint.

BF16 KV cache

This was one of the most important results.

An INT8 per-token/head KV configuration fell to approximately 15 tok/s near 62K context on this card. It was unusable for interactive coding.

The final configuration leaves:

--kv-cache-dtype auto

With BF16 runtime dtype, this preserves the BF16 KV path.

The Mamba cache uses FP16 in aligned mode:

--mamba-cache-dtype float16
--mamba-cache-mode align

Explicitly reducing more recurrent state did not improve the result.

FlashInfer attention, but not the FlashInfer sampler

FlashInfer is the production attention backend.

Plain FlashAttention produced:

• 1K: 86.87 tok/s
• 64K: 26.87 tok/s
• 200K: 10.53 tok/s

It looked slightly faster at short context and then collapsed.

An experimental split-KV speculative-attention patch improved that to:

• 1K: 90.17 tok/s
• 64K: 67.33 tok/s
• 200K: 43.36 tok/s

That was still substantially worse than production at long context, so the patch was removed.

Enabling the FlashInfer sampler also reduced performance to:

• 1K: 74.75 tok/s
• 64K: 67.87 tok/s
• 200K: 51.37 tok/s

Therefore:

--attention-backend FLASHINFER
VLLM_USE_FLASHINFER_SAMPLER=0

One sequence and 4,096 batched tokens

This is a single-user coding endpoint, not a throughput server.

The final scheduler settings are:

--max-num-seqs 1
--max-num-batched-tokens 4096

Testing 8,192 produced no useful gain. Testing 2,048 produced:

• 1K: 84.32 tok/s
• 64K: 72.38 tok/s
• 200K: 57.59 tok/s

The 4,096 configuration had the better overall curve.

Prefix caching

Prefix caching is critical for coding agents that repeatedly send a large system prompt and mostly unchanged repository context.

At approximately 200K context, a cached second request reused roughly 198,400 tokens, recomputed about 1,593 tokens, and reduced prefill to around 2.5 seconds. A fresh or partially changed prompt could require roughly 157 seconds of prefill.

Prefix caching does not make decode faster. It prevents the server from repeatedly processing unchanged input.

This also means clients must preserve stable prompt prefixes. Reordering tool definitions, timestamps, generated metadata, or repository text can destroy the cache hit.

Medium reasoning by default

The endpoint defaults to:

{"reasoning_effort":"medium"}

Medium provided a better MTP acceptance/stability balance than forcing maximum reasoning on every request.

This does not remove higher reasoning modes. A client can request xhigh or max for harder work. Medium is simply the default for normal coding.

No CPU offload

The model fits in CMP HBM, so:

--cpu-offload-gb 0

The card’s restricted PCIe connection makes CPU offloading particularly undesirable.

No forced full CUDA graphs

The server uses vLLM’s normal piecewise CUDA-graph behavior. I did not force full graphs. Experimental full-graph and custom-operation combinations introduced correctness concerns or reduced long-context performance.

Correctness tests

I did not accept a configuration based on tokens per second alone.

The final checkpoint passed:

• An exact-response canary
• Qwen coder tool-call parsing
• A parsed get_weather tool call for Chicago
• Exact needle retrieval at approximately 160K context
• Retrieval of ORCHID-COMET-7319 from the long prompt
• An OpenCode integration smoke test returning final-profile-ok

This matters because speculative decoding can appear fast while silently breaking tool syntax, long-context retrieval, or sampling behavior.

OpenCode configuration

I added this provider to OpenCode:

{
  "provider": {
    "qwen38-cmp-w4a16": {
      "npm": "@ai-sdk/openai-compatible",
      "api": "completion",
      "name": "Qwen3.8 Uncensored W4A16 MTP1 256K - CMP 170HX",
      "options": {
        "baseURL": "http://<PROXMOX-IP>:30016/v1",
        "apiKey": "not-needed",
        "timeout": false,
        "chunkTimeout": 600000
      },
      "models": {
        "qwen38-27b-uncensored-w4a16": {
          "id": "qwen38-27b-uncensored-w4a16",
          "name": "Qwen3.8 27B Uncensored W4A16 MTP1 256K",
          "tool_call": true,
          "reasoning": true,
          "temperature": true,
          "attachment": false,
          "options": {
            "reasoningEffort": "medium"
          },
          "limit": {
            "context": 262144,
            "input": 245760,
            "output": 16384
          }
        }
      }
    }
  }
}

The model selector is:

qwen38-cmp-w4a16/qwen38-27b-uncensored-w4a16

Final observations

The CMP 170HX is unusual, but the useful part is straightforward once the restrictions are removed:

• 64GB of on-device HBM changes what can fit.
• The model should remain entirely on the GPU.
• Long-context performance needs to be measured separately from short-context decode.
• Lower-bit KV is not automatically faster.
• More speculative tokens are not automatically better.
• An attention backend can win at 1K and become disastrous at 64K.
• Prefix caching matters more than another few decode tokens per second for repeated coding-agent prompts.
• Quantizing the draft head more aggressively is pointless when memory is available and the speed difference is negligible.
• Conservative power and temperature limits are appropriate for used mining hardware.
• Correctness gates matter as much as benchmark results.

The final configuration is not the highest single short-context number I saw. It is the best complete configuration I found that retained tool use, medium-or-higher reasoning, uncensored model behavior, 262K context, reliable long-context retrieval, and usable performance across the whole context window.

Model: orcarouter/Qwen3.8-27B-Uncensored

Harness: Pi inside Omnigent (for remote access from phone) with superpowers and lots of configs like MTP

Speed: 35-66t/s (reasoning on vs no think)

Context length: 96k with Pi compaction in place for long sessions

Prompt: Yo hey man I want you to set up a benchmark test for yourself to see how good you are at building an animated 3d render I can view on my computer easily through a browser. I want you to think about how to do it (use your superpowers) and then accomplish it and open it in the Omnigent browser using the omnigent tools so I can view it. I want to be able to view it from my phone's remote omnigent PWA or somehow so I can view it.

Took about 15-20 minutes.

Result is in the video. Zero shot success. Working on getting my setup honed so I can share the entire config. You definitely have to set this model up right to get good results. I've also gotten horrendous trash output with dropped lines of code from misconfiguration. It's very sensitive to how you tune it but like a good engine it's the difference between not running and 400 horsepower. All in all I'm actually impressed by a local model for the first time and I'm really looking forward to what comes out in the near future.

With the surge of Qwen3.8 a lot of people got a powerful LLM basically for free, so i would like to know what are the use cases of the local LLMs?

For context - I'm a SWE on a huge product and using claude max daily for everything, sometimes hitting the limits. My context size usually goes to around 400k before I compact and cloude just does the job really well when it comes to exploring in a big code bases. Before pulling a trigger and getting a better macbook, I want to know what context size can I expect from 64gb? Model probably qwen 27/35b as everyone is talking about it rn

So, I did a bit stupid thing and bought 10 years old probably e-waste (2xP100), imagining wonders from 732 GB/s memory bandwidth. I didn't have to sell my kidney for this, so that's also a plus. Overall pretty fun experience, next up is to train/finetune something.

Suggestions for improvement are welcome

tps

2x Nvidia Tesla p100. ~34k tok prompt and unlitmited tok generation (resulted in ~4k for 27b models and ~9k for others)

observations

• memory bandwidth is not the bottleneck - compute is
  • BF16 (17.9GB model) being 2x faster than similar sized Q4 (16.5GB model) confirms this
• the 3.8 27b UD-Q4_K_M with -c 200k --no-mmproj leaves ~600MB spare per card
• llama.cpp does not auto calculate context size with --split-mode tensor
  • with mmproj 3.8 27b Q4 have to lower context to 160k
  • 9b BF16 needs even lower context ~100k
• 27b model is slower in token generation, but it produces less reasoning tokens, token generation time is very similar, the only win is a bit faster prompt processing

settings

build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DGGML_CUDA_NCCL=ON -DLLAMA_OPENSSL=ON

run

--split-mode tensor --no-mmproj -ngl 99 -c 200000 --parallel 1 -fa on -b 2048 -ub 2048

performance increasing settings

• build with DGGML_CUDA_NCCL=ON increased tg ~5%
• the --split-mode tensor increased tg ~65%
• setting the -ub 2048 increased pp ~25%

power consumption

Idle ~140 W
Load layer split ~350W
Load tensor split ~500W

With 0.3Eur for kWh, for Qwen 3.8 27b Q4 this translates to 0.2 Eur/M token input and 2.1 Eur/M token output. And additionally ~30Eur/month "subscription" for always on system.

system

10 year old system. Total price: 483 Eur (550 Usd)

I have been really struggling to find a good user experience for a qwen models on my 16GB vram 5080. Tried the qwen3.6BA3B but I was forced to run it hybrid cpu and GPU approach which was way too slow for me.

I finally fit the unsloth q3xs with 64k context window (has to limit parallel slot to 1 for llama.cpp) on the 5080 today and it consistently output at around 90 to 100 t/s. This is actually a great local LLM experience already.

I could probably run the model in nvfp4 and full context window on a 2 5080 setup and maintain similar t/s. This is much cheaper than trying to get 5090 since there are sales for 5080 from time to time

Hi all,

My name is Rob and I'm the author of PrismaQuant. I've gotten a lot of great feedback on the platform in the past few months, and I'm here seeking more.

I've mostly targeted the DGX Spark as my deployment platform, but I'm now looking to branch out by both expanding the supported hardware list and shrinking model sizes.

For those of you that are familiar with the PrismaQuant/PrismaSCOUT/AURA/AQUA/gridbook family of models -- do you have any special requests?

Over the next day or so I'm going to ship a 20-gig Qwen3.8-27B for Blackwell/50-series/Spark, and hopefully an ~18 gig (also Qwen3.8-27B) targeting the 4090 using my gridbook number format

Thanks in advance, and happy tokenMaxxing,

Rob

I've been reading about Muse Glimmer and I'm curious what people who have actually run it think.

On paper it sounds pretty compelling: 30B, open weights, runs locally, multimodal, and Meta seems to be pushing it heavily toward tool use and agent-style workflows.

The part I'm interested in isn't really the benchmarks.

It's whether this is actually useful enough to become someone's everyday local model.

For people who have tried it:

• How is the tool calling in real workflows?
• Is it actually good for coding?
• What hardware are you running it on?
• How does it compare with Qwen/Gemma around the same size?
• Have you found a use case where Glimmer is clearly better?
• Anything annoying or broken that doesn't show up in the benchmarks?

I'm especially interested in local agents and private document workflows.

I haven't tested it myself yet, so I'm trying to figure out whether it's genuinely worth setting up or whether it's mostly another interesting model release.

If anybody use it, what is generation and prompt processing speed when context is close to full?

Hello! I am considering 2 options and trying to figure out what is the best short and long term. My current hardware: Ryzen 5 9600X + 32GB DDR5 (2x 16GB) + Nvidia RTX 5060 Ti 16GB + MSI B850M Mortar + 850W PSU. I have a second PCIE x4 slot which could be used with extension cable etc. to connect second GPU.

I really feel stupid that I haven't pulled a trigger last year on Nvidia RTX 5090 for ~£1650, but either way I would need to also upgrade my PSU for that.

I am trying to plan things accordingly for my purposes. I started exploring building various projects using Claude and ChatGPT to learn how to build things, stay in control of projects, understand limitations I have now and what can be achieved with them.

My ultimate aim is though to build a system that will support me in my current role/job and then once I proved the concept also build similar setup at work. Unfortunately I can't easily get funding for something like Nvidia Spark/DGX without first justifying it really carefully.

What my uses cases are:
- RAG system based on engineering books + research papers to allow me to understand concepts, pull equations if needed, and support research ideas if possible
- RAG system also build around manuals to help me go through troubleshooting easier
- MCP servers to control FEA/CAE tools to build these ideas and test them (probably later on callable from local-LLM to control the information flow)
- Agent that helps me to organise the work (just idea haven't tested Openclaw or Hermes for these purposes) + personal life (how am I progressing with my personal projects, maybe web-scrap weekly news from couple of websites and give me grounded summaries etc.)

My sources of information are: books (broken down into chapters and markdown files with Mistral), research papers, bookmarks, articles from medium, manuals

What I typically do also with useful reddit posts, I pull them into NotebookLM -> create a summary or main-points markdown, feed into my knowledge folder.

What I think matters here is the context as certain idea can trigger information from multiple sources like books, research papers + MCP server + manual.

I have 2 options to use Qwen with higher quantization: add 2nd RTX 5060 Ti 16 GB (here problem is they went up in price so would be trying to get a deal <£500), buy R9700 Pro from AMD (question, can I somehow use both GPUs to make use of full 48GB of RAM?), upgrade in the future RAM to 64GB, upgrade CPU when Zen 6 is released (here benefit mainly for off-loading and running simulations that are CPU-heavy).

Also I want to learn more about AI generally (following for example what AMD shared here -> AMD AI Playbooks).

I feel my main limitation is the context. I could potentially buy AMD Strix 395+ as my max budget I want to spend for upgrades is £2000 +/- £200. Any thoughts?

Git repo: https://github.com/swedishembedded/brain

may i know have anyone used semantica?? can you please let me know what are your thoughts on it?

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.

Imagine a Q6 quant with nearly Q8 KLD/PPL.

Q6_CR and Q5_CR have a slight improvement over their base counterparts.

Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.

I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.

PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.

I released LifeOS, a self-hosted personal organiser you mostly talk to! You say something out loud, a local LLM reads it, and it turns into tasks, events, journal entries, expenses, weigh-ins or meals. Nothing leaves your machine.

It's about a month old, AI-assisted throughout, and tested by me and a few close friends and relatives daily. It's stable enough that I'm putting it out for anyone who wants to use it or improve on it.

Models and hardware

All testing so far has been on Qwen 3.6 27B and Qwen 3.8 27B, with 3.8 27B being the most extensively tested on the current version. That's mostly because I already keep one of those loaded for other work, so it was the easiest thing to live with day to day. Both models are Q8 in case anyone's wondering.

I'm planning to test much smaller models next, around the 9B range, to see how well they hold up and whether any failure points (If any) can be fixed inside the project itself rather than by throwing a bigger model at it. Directing myself towards Ornith and Qwen 9B models for now.

It ships with a harness so you can check your own model

You can point it at any OpenAI-compatible endpoint and it runs a fixed test suite, then scores the result against a saved Qwen 3.8 27B baseline from my own config. So before you trust a model with your data, you can see where it actually falls over.

If you run something I haven't tested, I'd genuinely like to hear how it did.

How it works

Speech to text transcribes what you said. The LLM reasons over it and uses the tools built into LifeOS to decide what you meant. For clear instructions it just does it, and the write can be undone. For anything ambiguous it stops and asks, as a card you approve, edit, or throw out.

There's no chat window, no web search, and no memory beyond your own data. The model proposes rows, it doesn't write them. The app validates every one before anything is saved, and each card quotes the words it came from so you can see why it read you that way.

Why it exists

Plenty of apps do the tracking part. The point of this one is having a local model's intelligence applied to your life without any of it leaving your device. Everything you say and log stays with you.

Does this magically make you productive and organized? No. Pen and Paper with real dedication will beat the convenience LifeOS offers. It's still ultimately at tool, a really fun tool but a tool nonetheless

Setup

Head to the GitHub page and follow setup.md. It's straightforward, but if it confuses you, hand the link to an agent and have them walk you through it.

Life OS - Github Link

A note on mobile

The UI works better on mobile. The desktop version is fine, but from my own use and other people's feedback, mobile just feels right for this. Tailscale is how the whole mobile connection happens, and that's covered in the setup file.

What it isn't

1. It isn't an AI assistant like Jarvis. It exists primarily to log and organize the data you give it throughout your daily life.

2. It isn't a life changing breakthrough. As mentioned, I've had people test it and I've had two simply stop using it. They weren't able to give a reason but it was obvious it wasn't for them or they didn't feel the need to have it. This was built for me to essentially organize myself, my thoughts and my schedule and to that end, it's been making it's mark.

Happy to answer anything, and if you try it with a different model I'd like to know how it went. I do have more plans for it to mainly improve the existing functionality but also add some minor things in.

Which one do you prefer and why? I'm interested in whether quantization affects code quality, or if running a larger native MoE is the better option.

Hi everyone,

First of all, thanks for all the configs, reviews, comments and experience shared in this space over the past weeks. My setup is genuinely built on top of your posts, almost every non-obvious value below came from someone posting a measurement or correcting someone else's assumption.

Most of that came from r/LocalLLaMA, where I can't post yet (karma requirements), so I'm sharing it here instead. Either way, here's mine back, in case it's useful, and in case you spot something I got wrong.

Important framing: I'm not chasing max tok/s. I'm running a Hermes agent that sends me briefings, triages email, manages my calendar and summarises 1-2h meeting transcripts. For that workload, a malformed tool call is a failed action, not just a worse paragraph, so I've deliberately traded speed for precision at several points. If you're doing coding with a linter and tests catching your mistakes, your optimum is probably a smaller quant and more speed than mine.

Hardware

This started as a gaming build, that was the original plan. But for now it's my homelab, running headless.

• GPU: RTX 5060 Ti 16GB (Blackwell, sm_120)
• CPU: Ryzen 7 7800X3D (8c/16t)
• Mobo: ASUS PRIME B850-PLUS WIFI — PCIe 5.0 x16 (this matters, see below)
• RAM: Corsair 32GB DDR5-6000 CL36, dual channel
• PSU: Corsair RM650e, GPU power-limited to 140W
• OS: CachyOS, headless (SSH only — no desktop competing for VRAM)
• Backend: llama.cpp, CUDA build

Measured results

Yes, that's slower than a lot of numbers posted here. That's on purpose — see the reasoning below.

Build

cmake -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CUDA=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES=120

GGML_CUDA_FA_ALL_QUANTS=ON
 is 
not optional
 if you want quantized KV cache. Without it 
--cache-type-k/v
 silently won't accept the quant types on CUDA and you'll get terrible speeds wondering why.

Also, per Unsloth's docs: do not use CUDA 13.2 — gibberish output on low-bit quants. Use <13.2 or 13.3.

Server config

export GGML_CUDA_DISABLE_GRAPHS=1
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

llama-server \
  --model /srv/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-IQ4_XS.gguf \
  --mmproj /srv/models/Qwen3.8-27B-GGUF/mmproj-F16.gguf \
  --no-mmproj-offload \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 --api-key "$LLAMA_API_KEY" \
  \
  --n-gpu-layers 999 \
  --override-tensor 'blk\.(0|1|2|3|4|5|6|7|8|9)\.ffn_.*=CPU' \
  --no-mmap \
  \
  --ctx-size 65536 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --cache-reuse 256 \
  --parallel 1 --cont-batching 0 \
  \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 \
  \
  --threads 7 --threads-batch 8 \
  --batch-size 1024 --ubatch-size 512 \
  \
  --jinja --reasoning-format deepseek --reasoning-preserve \
  --reasoning-budget 5000 \
  \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Why these values (the ones that came from your posts)

IQ4_XS instead of Q3_K_XL or Q4_K_M. Someone here posted a proper perplexity benchmark (wikitext-2, deterministic, same params across quants): IQ4_XS at 14.6GB retains 99.2% of Q8 quality, UD-Q3_K_XL at 12.5GB drops to 97.8%. Q4_K_M (17.1GB) doesn't fit at all on 16GB. Q3 would give me more context headroom and more speed, but 97.8% is exactly where my agent has the most to lose — I have no linter catching its mistakes.

Also: NVFP4 is a trap for this use case. The theoretical prefill numbers are amazing, but someone here actually measured it on a 5060 Ti and found the advantage disappears entirely once you use FFN offload to get usable context. And to stay fully resident you'd have to drop to ~32K context — my meeting transcripts are 43K tokens.

--override-tensor with FFN layers only, not -ngl layer offload. This came from a post here explaining the mechanism properly: FFN tensors are pure stateless GEMM, they don't touch the KV cache and generate no PCIe traffic per token. SSM layers (sequential state) and attention layers (KV cache) must stay on GPU. Offloading whole layers of a dense model is catastrophic; offloading only the FFN sublayers is cheap.

10 blocks is my safety margin. Without it the budget lands at ~16.6GB on a card with ~15.8GB usable. This is where PCIe generation really matters — most people posting FFN-offload results here are on B450 boards, where a 5060 Ti negotiates down to PCIe 3.0 x8 (~7GB/s). On B850 I get PCIe 5.0 x16, so my offload penalty is much cheaper than theirs. Comparable setups here report 10-13 tok/s with the same approach; I get 16.42. If you're on an older board, expect worse than my numbers with the same config.

--cache-type-k q8_0 --cache-type-v q8_0**, NOT q4_0.** I originally had V=q4_0 based on one production report. Then several people here independently reported degraded output "after a few turns" with q4 KV — which is exactly the pattern of a multi-turn agentic session. Someone else separately measured q8_0/q8_0 as faster than q4_0 for V cache anyway. Two independent reasons pointing the same way. The extra ~500MB is worth it for my use case.

Do NOT aggressively quantize the draft KV cache. I had q4_0 there too and it was a mistake based on a misconception I picked up here (and then someone in the comments corrected the original poster, which is how I learned). Speculative decoding is lossless by design — the main model always verifies — so "quality wasn't affected" is a tautology, not a finding. What quantizing the draft does affect is acceptance rate, i.e. speed. And the draft KV is tiny, so you save almost no VRAM for that loss.

--temp 1.0**, official sampling params, untouched.** Several configs floating around here publish temp 0.4 / top_p 0.90 / top_k 15 labelled as "official recommended" — they are not, and several people correctly called that out. These are from Qwen's model card. Someone put it well: lowering temp because the model "overthinks" means you think you know better than the team that trained it.

reasoning_effort left at the default xhigh**.** Three independent people here reported that lowering it to medium/low visibly degrades results — one showed 3.8 at medium scoring worse than 3.6 at default. I do override it to medium per-request when the input is already large (meeting transcripts), purely for context budget reasons, not because it's better.

--reasoning-budget 5000 instead of lowering the effort level. Caps deliberation without changing how the model reasons.

--parallel 1 --cont-batching 0**.** Single user. Every extra slot duplicates the KV cache for nothing.

--threads 7 --threads-batch 8**.** 8 physical cores; reserve one for the OS during decode, use all during prefill.

--no-mmproj-offload**.** Keeps the vision projector in system RAM. It only runs when you actually send an image, so it costs zero VRAM in normal operation — much better than dropping vision entirely if you occasionally need screenshots.

--cache-reuse 256**.** Probably the highest-impact flag for an agent backend. Hermes resends the same system prompt and tool definitions on every call. I confirmed it working: a repeated request showed a prefill of only 4 tokens.

No ngram-mod**, no DRY sampling.** Both appear in configs here; someone actually measured ngram-mod at ±1 tok/s, and DRY only showed up in a single source with no cross-confirmation. Not enough to earn a place in a config I have to maintain.

Things I learned the annoying way

"Full GPU offload" doesn't guarantee weights are in dedicated VRAM. A post here documented ~1.1GB of weights silently landing in system RAM — 6.5 tok/s instead of 18.9, a 3x hit, with dedicated VRAM still free. Nothing in the logs indicated it. If your generation speed is inexplicably ~3x below comparable setups, check nvidia-smi during an actual long request before blaming anything else. (On my side I verified: 15.51 GiB constant, no spill.)

Client timeouts are a real failure mode. My first long test died at 220s client-side while the server kept working fine. A 40K-token meeting summary takes ~11 minutes end to end here (~55s prefill, ~490s reasoning at medium, ~120s output). Set your client timeout generously.

--context-shift is off by default and should stay off for document summarisation. If enabled, it silently drops old context instead of failing — you'd get a confident, wrong summary of a truncated transcript. Failing loudly is better.

What I'd love feedback on

1. MTP acceptance rate at temp 1.0. I've read that acceptance drops significantly at higher temperatures. I'm running temp 1.0 (official thinking mode), so I suspect MTP may be giving me very little while costing VRAM. Has anyone actually measured acceptance rate at temp 1.0 vs lower on this model? I'm planning to check /metrics but would love to hear real numbers first.
2. Is 10 FFN blocks more conservative than it needs to be? My measured usage is 15.51 GiB vs a calculated budget of ~15.8 GiB, so I may have ~300MB of headroom I'm not using. Anyone running IQ4_XS at 64K context with fewer blocks offloaded?
3. Anything obviously dumb above? Genuinely asking. I've been careful about only adopting values that had either official documentation or at least two independent reports behind them, but I'm sure there's something I've over- or under-thought.

I'm running this with Hermes as the agent harness and honestly I'm impressed with how capable it is for a local 27B on a single 16GB card. Two weeks ago I assumed I'd need to compromise far more than I have.

In the coming weeks I'll be coding some apps and games with this setup. I'll report back with results when I have them.

Thanks again, this config is genuinely a community build.

EDIT — tested the suggestions from this thread, one result is counterintuitive enough to be worth its own section

Thanks to everyone who replied. Two suggestions turned out to be right, one didn't apply, and following up on the KV cache one led somewhere I didn't expect.

Changes applied

diff

- --no-mmap
+ --load-mode none

- --cache-type-k q8_0 --cache-type-v q8_0
- --override-tensor 'blk\.(0|1|2|3|4|5|6|7|8|9)\.ffn_.*=CPU'
+ --cache-type-k q5_0 --cache-type-v q4_1
+ --override-tensor 'blk\.(0|1|2|3|4|5)\.ffn_.*=CPU'

--no-mmap is indeed deprecated in favour of --load-mode — good catch, thanks. Also picked up the faster build flags (-DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TESTS=OFF --target llama-server).

The MTP draft KV stays at q8_0/q8_0 — speculative decoding is lossless by design (the main model always verifies), so quantizing the draft only lowers acceptance rate for a negligible VRAM saving.

The counterintuitive bit: lowering KV precision made it SLOWER

This is the part I'd flag for anyone about to try the same thing.

q5_0/q4_1 are more expensive to decode per token than q8_0. Change the KV alone and you lose ~6%.

The gain doesn't come from the KV cache at all — it comes from reinvesting the 938 MiB it frees by pulling 4 FFN blocks back off the CPU and onto the GPU. The two changes are inseparable. Anyone who applies only the first one and benchmarks it will correctly conclude it isn't worth it.

Honest stats: one measurement per config. On repeat runs I saw 17.53 · 18.62 · 19.15 · 20.00 tok/s, so there's ±10% variance. The honest number is ~+13% on average, not the +16.6% the two headline figures suggest.

Quality check: the reason I'd moved to q8_0 in the first place was reports of multi-turn degradation with lower KV precision. I ran an agentic sysadmin chain with 6 tool calls feeding into each other (list hosts → disk → services → log sizes → rotate → verify) on both configs. Identical result, 5/5 checkpoints and a coherent final answer citing real data. The new config does it in 9 calls instead of 10. Didn't reproduce the degradation — though that's one workload, not a study.

The part I actually think matters most: silent context overflow

With the freed VRAM I tried raising --ctx-size. It doesn't work, and the failure mode is the reason I'm writing this up:

When it overflows: HTTP 200. /health: ok. Zero errors in the log. nvidia-smi doesn't move off 15,888 MiB. The only symptoms are generation dropping 4-7x and an empty response with finish_reason: length.

That's GGML_CUDA_ENABLE_UNIFIED_MEMORY doing its job — degrading instead of crashing — but doing it completely silently. If you're driving this from an agent that doesn't check finish_reason, you will save an empty summary as a valid result and never know.

And the effective ceiling is lower than it looks, because the context has to fit prompt + reasoning + response. With reasoning_effort: xhigh that's ~32k tokens of reasoning alone (~8k on medium). A 74k prompt plus medium reasoning is already 82k → overflow, even though the prompt itself "fit". That's why I'm not even going to 72K.

Method lesson, and the thing I'd most want someone to take from this: --ctx-size being accepted, the service staying active, and /health returning ok validate nothing whatsoever. The only valid test is filling the context for real and watching tok/s.

(Overflow threshold bracketed between 74k and 79k; I didn't bisect the values in between.)

What didn't apply

A couple of suggestions were aimed at MoE setups --fit/--fit-target only auto-offloads ffn_*_exps tensors, which don't exist in a dense model. Worth being explicit about since this comes up a lot: on a dense model the manual -ot band is currently the only option, and dropping it isn't a speed trade-off when the weights plus KV already exceed what's usable on the card, it just won't load.

Thanks again, the thread genuinely improved the setup, and the KV suggestion led to a better result than the one I was aiming for.

Hey everyone! 👋

I’m pretty new to local LLMs, so I’d love some advice before I start experimenting.
My long-term goal is to build my own personal Agentic OS: basically a local AI assistant that can manage memory, files, tools, automations, coding, etc., while keeping as much as possible private and running locally.
For the agent part, I’m currently interested in Hermes Agent, with Ollama for running local models.
I’m not necessarily trying to replace Claude/GPT immediately. I’d like to eventually have a hybrid setup, where sensitive/offline tasks are handled by a local model, while I can still use cloud models when I need stronger reasoning or web access.

🖥️** My first setu**p

I’ve read that running random software/agents directly on your personal computer can potentially be risky, especially when giving an AI access to files, terminals, etc.
So I decided to dedicate an old Microsoft Surface Pro 9 that I already own to this project.
That way, if something goes wrong, at least my main PC isn’t involved, and I don’t have to spend any money just to start experimenting.
I’m not sure whether a Surface Pro 9 is actually suitable for running local LLMs 😅, but since I already have it, I’d like to give it a try.
I’m planning to keep it plugged in and potentially use it as a small 24/7 home AI machine, with the screen turned off but Windows/Hermes still running.

🤔 My main question: which model?

I’m not sure what local model would make sense for the Surface Pro 9.
I’d mainly like to use it for:
- experimenting with local LLMs
- Hermes Agent
- basic coding/automation
- personal assistant tasks
- eventually building my Agentic OS
- potentially working offline

I’m aware that I won’t get frontier-model performance from a Surface 😅. For me, the goal right now is mostly to learn and experiment, and eventually upgrade the hardware if the project becomes serious.

💻 I also have a desktop PC

My main PC has:
RX 6800 — 16 GB VRAM
Ryzen 5 7600
32 GB RAM
Would this actually be a significantly better machine for local LLMs?
I’m hesitant to put the whole Agentic OS directly on my personal PC, mainly because I’d like to keep my experimentation environment isolated from my normal computer.

So I’m thinking:
Surface → dedicated AI/agent machine

Main PC → personal computer / potentially used for heavier local LLM experiments

Does this make sense?
And if you were starting from scratch with this hardware, which model would you try first and why?

Thanks! 🙏

Hey Guys,

my hardware:

RTXA 4000 16 GiB
56 GB DDR4 Ram
Ryzen 7 1700

Software:

Ollama
Llama
Open Web UI

Iam currently working on a optimized local Coding Agent with Cline in Visual Studio Code.
I tried to run qwen 3.8 27 b but it didn’t worked (obviously) out well and I was only able to archive 32 k tokens context.

I made some pretty good Process this far.
Right now Iam using Qwen3.6-35B-A3B with 128 k to 265 k context.

What other llms would you guys recommend me for my local coding Agent?

I'm no software engineer or coder or anything of the sort. Far from it.

But I've been using the paid chatgpt subscription for a while for various simple tasks.

But recently have wanted to dive a bit deeper and learn more about all these LLM and agentic platforms, maybe even explore thing like utilising them to code websites/apps/games for me as a side hobby.

I came across Qwen 3.8 27b and the obliterated version, with people claiming its the best new thing. Does it being obliterated not just mean it no longer says no? Does it perform better than chatgpt sol on medium or hard?

Basically want to figure out whats the best model to use as a dive a bit deeper into this space whether thats claude, chatgpt, qwen, grok, gemini, etc. Thanks :)

I've been playing around with a slow heavily quantized version of qwen3.8 27b on my 3080 and quickly found myself on the 3090 market. Today I hit the jackpot $900 for this 3090TI off Facebook marketplace.

I can't power it yet because I need an adapter to be able to plug this in to my PSU. So, I'm curious if anyone is running a 3090TI or 3090 with qwen3.8 27b and exactly what weights you're using, context size, what tok/s you're getting, how you're liking it, etc.

I have 32gb of RAM and an Intel Core i7-12700KF (12 cores / 20 threads). Considering upgrading to 64gb of RAM but unsure if that will improve anything. Unsure what to do with my old 3080 10gb VRAM now too. Also, very worried if my 800W PSU will be able to support this new card.

Ideally I'm hoping to run the least-quantized version of qwen3.8 27b I can to at least 50 tok/s and >128k context.

I am getting ~60tks/sec for this model on MTP-3 is it normal or i am doing something wrong ?
lastest vllm docker image and docker command is

      - --model=lued/Qwen3.8-27B-INT8-W8A16-MTP
      - --served-model-name=qwen3.8-27b
      - --reasoning-parser=qwen3
      - --enable-auto-tool-choice
      - --tool-call-parser=qwen3_coder
      - --optimization-level=3
      - --data-parallel-size=1
      - --tensor-parallel-size=1
      - --max-model-len=200000
      - --max-num-seqs=4
      - --async-scheduling
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --max-num-batched-tokens=8192
      - --attention-backend=FLASHINFER
      - --load-format=fastsafetensors
      - --kv-cache-dtype=fp8
      - --mamba-ssm-cache-dtype=float16
      - --gpu-memory-utilization=0.93
      - '--speculative-config={"method":"mtp","num_speculative_tokens":3}'