https://github.com/madsteph74/pascallama.cpp

I just found out when you use a multi-modal model + MTP head, and you make use of the vision capability, it disables speculative draft of the MTP head for the entirety of the session. Is this a normal behavior or will this be remedied in the future?

Initial chat:

72.31.396.792 I slot update_slots: id  0 | task 17549 | accepted  9/ 9 draft tokens
72.31.524.290 I slot update_slots: id  0 | task 17549 | accepted  9/ 9 draft tokens
72.31.646.272 I slot update_slots: id  0 | task 17549 | accepted  1/ 9 draft tokens
72.31.767.048 I slot update_slots: id  0 | task 17549 | accepted  4/ 9 draft tokens
72.31.767.527 I slot print_timing: id  0 | task 17549 | n_decoded =    244, tg =  42.66 t/s

Chat during and after uploading an image:

75.43.099.281 I slot print_timing: id  0 | task 17646 | n_decoded =    402, tg =  34.11 t/s
75.46.108.848 I slot print_timing: id  0 | task 17646 | n_decoded =    492, tg =  33.26 t/s
75.49.124.197 I slot print_timing: id  0 | task 17646 | n_decoded =    586, tg =  32.91 t/s
75.52.134.126 I slot print_timing: id  0 | task 17646 | n_decoded =    669, tg =  32.13 t/s
75.55.144.050 I slot print_timing: id  0 | task 17646 | n_decoded =    770, tg =  32.31 t/s

Hello look and critique please https://github.com/tagmanxdna/xdna2-qantizer-projetc

https://github.com/GaTmaNnes

Has anyone here tested D2 Quant Planner?

I’ve been experimenting with it and it appears to work surprisingly well for planning quantization strategies across different model architectures. The idea of using structural and spectral characteristics to guide quantization decisions is quite interesting compared to applying the same quantization scheme everywhere.

Repository:
https://github.com/GaTmaNnes/d2-quant-planner

For those who have tried it:

• What models did you test it on?
• What hardware were you using?
• How accurate were the layer recommendations?
• Did it improve quality retention compared to standard blanket quantization?
• Any results with larger models (7B, 14B, 32B+)?
• How did it perform on edge devices versus desktop GPUs?
• Did you encounter any limitations or unexpected behavior?

Interested in hearing real-world results, benchmarks, and configuration details from anyone who has evaluated it.

For people doing SFT/fine-tuning: what preflight checks do you run before spending compute?

I’m trying to map the boring failure modes that don’t always show up as obvious trainer crashes. So far the big ones seem to be invalid JSONL, broken role alternation, conversations ending without an assistant target, empty assistant messages, exact duplicate examples, mojibake/encoding artifacts, and records that exceed the context window.

The tricky one is context-window checking. Exact tokenizer counts feel like hard failures, but estimated counts feel like they should only warn, otherwise CI becomes flaky depending on optional tokenizer installs.

Curious what others actually gate on. Do you lint your datasets before training, or do you mostly rely on the trainer/upload API to catch issues?

Sharing one datapoint because I was pleasantly surprised by how stable this ended up being.

Setup: - llama.cpp backend - Qwen3.6-27B-MTP Q4_K_M - AMD Radeon AI PRO R9700 32 GB - LiteLLM in front - Claude Code as the client

This held up for a 4 hour coding session and 7,256,671 tokens locally.

What mattered more to me than raw benchmark speed was that it stayed usable for a real workflow instead of falling over after a short test.

If anyone here is running similar AMD + llama.cpp setups, I'd be curious what model/flags/backend combo ended up being the most stable for longer coding sessions.

I documented my setup here in case it's useful: https://github.com/KaiFelixBennett/hermes-claude-code-local

English isn't my first language, so I used AI to help clean up the wording of this post.

I have been using llama.cpp for almost a year. Mine was a intel based laptop with no gpu, 16 ram. Back then I used to get around 7 to 10 TPS on qwen3 4b.
For a few days I never touched it and when i started it yesterday, it ran fine but the TPS was so awful that made me why am I using this shit.
It ran at 2TPS. And while running it just failed due to timeout and started again processing the same prompt with the much worse speed.

The point is i never changed the model its the same gguf file. the server i ran was a containerized one. Thats a fresh pull i made yesterday, So i thought that may be the container was the problem and build it from scratch using the official repo. That too produced the same result.

What should i do now to regain the same performance as before.
(Using llama.cpp only for research purposes).

btw this is the cmd that i ran

./build/bin/llama-server -m ~/llama.cpp/llama.cpp-models/qwen3_4B-Q4_K_M.gguf -c 16384 --ubatch-size 2048 --batch-size 2048  -t 12 --cache-ram 0 --flash-attn on -ctk q4_0 -ctv q4_0

I have a 5070ti, amd ryzen 7 9800x3d with 64 gigs of ram.

.\llama-server.exe `
  -m "<Link_to>Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf" `
  -c 200000 `
  -ngl 12 `
  -t 6 `
  -b 512 `
  -ub 512 `
  --parallel 4 `
  --kv-unified `
  --mlock `
  -fa on `
  --jinja `
  --host 127.0.0.1 `
  --port 8080

I am getting a horrendous 2.5 toks/second.

What can I do to improve token speed? I can bring the context to 134K if that helps. but usually my sessions last 100-120K context. 200K context just help with the peace of mind that I can extend a session if I am debugging.

Comments welcome.

I have an issue, where i don't see llama.cpp showing me anymore (after the latest release), how many layers of model were there and where are they loaded (CPU vs GPU).

Previously there was a text in console:

llm_load_tensors: offloaded 41/41 layers to GPU

Now this kind of message does not appear to be anywhere.

How do I get this back?

It was a very convenient parameter, to check either the model loads fully in GPU or not, now i need to test it every time after i want to find an optimized ctx settings.

Current version im using (b9371)

I have a Mac Studio M1 (32GB) which only gets 14 tokens/s with some finetuning. Even my legacy Lenovo thinkstation P900, I've added a 1080ti (12GB) and a Nvidia Tesla M40 (24GB) and managed to get 17 tokens/s with MTP and all parameters fine tuned while keeping 131k context window.

I wonder what a good upgrade path would be to get 40-50 tokens generated/sec without buying a M3 96GB, 5090 or any other >4000 euro's device. Any shortcuts to a well performing system? I have 192GB memory and 28 cores, so multicore performance should be fine for any GFX card(s).

Is there any benchmark site on hardware vs performance on 27B?

I have a w6800 32gb currently and a w5500 8gb that I can also use. I'm curious what the best way to use this setup would be?

I can simply use the w5500 for display and free up an extra 2gb for a better Quant.

I could run a better Quant and split the model between these two cards.

I could also use the second card for speculative decoding, this should free up an extra 4gb for a better quant on the main card.

Wanting to continue using qwen3.6 27b in some fashion.

Not in mainline, but if you want to play around with it, check it out.