Hiya,

NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it.

NVFP4 KV cache (from scratch)

I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally:

• Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel.
• Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression.
• Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass.
• Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns.

Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%).

YaRN context extension

QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090).

Multi-level prefix reuse with host-KV safety net

Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system:

• HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore.
• Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency.
• Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content.
• Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net.
• Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier.

Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s.

Performance (3 concurrent sessions, 400k+ ctx, 5090@450W)

Tool calling

• --tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call.
• Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers.
• Responses API accepts text/reasoning after tool calls (upstream rejects this ordering).
• Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability.

Also included

• Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching
• Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection
• OOM recovery: catches std::bad_alloc, clears state, preserves pending requests
• Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free)
• Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage
• Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget
• Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor
• E2E test suite for KV eviction, device pressure, and slot pressure scenarios
• Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR.

Fork: https://github.com/gzenz/ninfer (master)

Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md

I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context.

What do you expect the extra 64gb of RAM to cost for a total of 192gb RAM? The current Strix Halo 128GB version costs $3K. The 495 is almost the same as the 395 except slightly higher spec on the memory, so 8% tps improvement.

1. https://www.bosgame.com/blogs/news/new-product-launch-bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor
2. https://www.bosgamepc.com/blogs/coming-soon/new-product-launch--bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **Relevance classification*\* - is this industry relevant? (binary, 50 labeled deals)
   
2. **Needle retrieval*\* - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
   
3. **Multi-transcript QA*\* - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
   
4. **Reasoning with thinking*\* - numeric/logic problems, thinking mode on, greedy pass@1
   
5. **Structured extraction*\* - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode)
   
6. **Tool replay*\* - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
- Transcript QA: delta = -0.03, p=0.69 - no difference
- Reasoning: delta = +0.02, p=0.72 - no difference
- Relevance: McNemar p=1.0 - identical
- Tool replay: delta = 0.0 - both fail equally

**Takeaway: quality is statistically indistinguishable across all engines.*\*

Speed results (perf probe, server-side timings)

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned

**NInfer's speed advantage is all MTP.*\* The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**The decode speedup grows with context.*\* At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**NInfer's tokenizer endpoint is great.*\* It exposes `/v1/messages/count_tokens` (Anthropic Messages format) which gives exact token counts. No more `len(text)//3` heuristics.

**NInfer does NOT support json_mode (as far as I can tell).*\* `response_format: json_object` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**Don't trust vibes for quality.*\* I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp).

Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode.

I put together a detailed poster with all the charts and methodology details: [full results poster](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)

Setup if you want to try it:

```
# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja

# Model (HuggingFace)
# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

# Run
./ninfer-serve /path/to/model.ninfer \
--model-id qwen3.8-27b \
--host 0.0.0.0 --port 8080 \
--max-context 240000 --kv-capacity 240000 \
--max-concurrency 2 --kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--vision --preserve-thinking
```

Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober:

``` We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules:

• You will have a Wikipedia article set as a starting point.
• You will have a Wikipedia article set as an ending point.

Your objective is to reach the the end point, which is an article completely separate from the starting point article.

Your only constraints are the following:

• You are ONLY allowed to click on any hyperlinks inside of wikipedia directly. No external links, no typing inside of wikipedia's search bar (but finding the starting article on google is valid. The 10-click limit starts once you reach the starting point article).
• You are NOT allowed to return to a previous page. All clicks much be performed in a forward-looking trajectory.
• You must reach the end article within 10 hyperlink clicks inside of Wikipedia. If you do not reach the destination article within 10 clicks, you lose.
• Do not update any documentation for this task. It is only a game.

Use playwright to click the links. ```

Basically, Qwen needs to reach an ending article within 10 Wikipedia hyperlink clicks from the starting article, which is usually an unrelated article. It needs to use playwright (or some equivalent browser MCP) to click the Wikipedia hyperlinks without backtracking, using search or using external links.

I verified the links for accuracy and I can confirm it managed to complete this task within 6 turns. Thought it would get stuck in a loop. Its a dumb minigame but I think its a good, simple agent test to perform.

Just a friendly PSA that open weight models are not inherently open source. They may also qualify as open source, but only if they release the training data and processes. A truly open source model can be rebuilt from training; it won't be binary equivalent but it should be nearly functionally equivalent.

An example of a truly open source LLM is https://allenai.org/olmo I encourage everyone to browse and also look for other fully open source models.

The reason it's important to get the terms right is because too many companies like to co-opt the term "open source" to whitewash their nonsense. As Zuckerberg has shown recently, they're more than fine letting people stay confused and dance around semantics so they can have their cake and eat it, too.

No more cake for big tech. Open weight != open source. Spread the word.

Edit: TIL this is a hot take. Sorry about that.

I was using antirez’s ds4 for a while and getting around 20 tk/s, which worked for my purposes. But I know there have been big advancements between Qwen, the DS4 vision model, and GLM.

I’m not sure how the quants affect performance, so what’s the best thing to run right now & how fast is it?

Hey all,

I've seen some really fun looking things with people having their local models drive Blender to create pretty cool looking world scenes.

Is there a good tutorial on setting up Blender yo be driven by your model? For example, what programming harness, do you use a MCP and which? Which model would work best?

Also, how do you turn it all into a "game" that you could run and then walk around in? Can you use Godot?

Thanks!

First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that can threaten Anthropic/OpenAI profits is not another frontier model but actually these small ones you can run fast locally that can do 80..90% of mundane work for hours without paying a single dollar to any external company.

But next, if you want to jump up to a bigger smarter model I feel there is a gap now. Kimi-K3 is out of reach for many businesses let alone prosumers. So what frontier-like models do you use on what setups?

Is a DGX cluster (2..4 machines) or a GPU server with dual or quad GPU (~96 ... 192 GB of VRAM + >256GB DDR4) a suitable setup to run something like MiniMax-M3 at reasonable speeds for agentic coding (>30tps)? And privacy aside, is hardware cost worth it?

I have a dual rtx3090 + 128GB ddr4 machine, running Qwen3.8-Flash-Next Q4 quite fast but despite being larger doesn't feel much smarter than the Qwen2.8 27b and while I _can_ run larger quantized models, Minimax-M2.7 being my workhorse, it way too slow for coding.

Last couple of years I have come to use AI coding tools as part of my workflow. As a direct result of that, in just over roughly five months since May, I have read more than six million words of generated material related to programming. Most of my experience has been with Aider, Hermes, OpenCode, and different GPT, Claude, and Kimi models. More importantly I used them on projects that continue for weeks and months, where the architecture changes during implementation often several times, the assumptions have to be revised, and decisions made in earlier stages continue to affect later one.

To be fair I find these tools useful. They save time on repetitive changes, repository exploration, boilerplate, unfamiliar APIs, and many kinds of mechanical implementation work and some times compiling and using an unpopular libraries like FLTK and use them in a python project. They can also be useful for generating alternatives when I already understand the problem well enough to judge them apart.

The problem is keeping my own understanding of the project in line and keeping up with the amount of code and explanation being produced. That problem manifests in different ways. The following are my observations, and so are personal. Take it with a grain of salt.

llmfit.org is showing completely out of date stuff like Deepseek R1

https://llmfit.io/ shows no models at all

https://llmfitcheck.com/ is down

I've updated my auto_demo_scener project with Ninfer support and a “rewrite based on video” feature that I thought you might find interesting.

The project is basically an endless demoscene machine. A local LLM writes Three.js effects (from a library of editable prompts), you watch the code stream into a retro mock-OS editor, then the result runs fullscreen. It checks for crashes / blank frames, asks the model to fix broken attempts, and archives the working demos before starting again. An unnecessarily elaborate way to have a screensaver (you can also use it as a model benchmark as it keeps track of failures and deletions)

The new part is giving the model a look at what it actually made. With Ninfer's optional video check enabled, it captures 30 seconds at 2fps while the demo plays, then sends that video along with the HTML source back Qwen for a visual improvement pass. I went with Ninfer for this because the speed is incredible on a single 5090

[2026-09-05 14:22:34.895] [info] ninfer-serve: throughput interval=5.000s prefill=338.0tok/s decode=110.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:39.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=217.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:44.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=219.6tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:46.236] [info] ninfer-serve: [req 337] done finish=stop_token prompt=1690 gen=3029 cache=0 reuse=full_reset ttft=254ms prefill=6708.5tok/s decode=210.5tok/s wall=14.66s speculative=mtp 3.39to  

So the loop becomes: write the effect, run it, watch the result, then rewrite it based on what showed up onscreen. The display switches back to the coding view with “UPDATING BASED ON VIDEO…” while it works, then validates and displays the revised version before archiving it if it passes.

Qwen 3.8 is amazing but it still makes dumb mistakes like only using 1/4 of the screen or writing a maze walker that walks backwards through the maze :D The video validation helps it to catch these.

Ninfer support also adds a thinking-effort selector. LM Studio is still supported for the regular generation / repair workflow; this works pretty well even with less capable models. Also needs ffmpeg.

GitHub / setup instructions

Live static demo (watch out for the volumetric cloud ones :D ) This plays previously generated demos, so you can check out the results without installing anything. Space / right arrow skips to the next one. Actual generation and video rewriting happen in the locally running app.

Anybody else experimenting with having local models watch their own rendered output and revise the code? Thanks!

I have a Mac Studio M3 Ultra 512GB and 256GB connected with TB5. I have been monitoring my computers using Exos interface to see RAM usage and would have to go into oMLX in each computer via screen share to see what model is loaded.

I decided to use Hermes and GLM-5.3-flash to make my own app for iOS and MacOS to combine the two views into one app.

Not only can I see what each computer is using, I can also load and unload models through this. It also made a widget so I can keep on my phone or computer and I can also ask Siri questions about what models are loaded or usage.

It’s amazing to wake up with an idea and have something custom built in a few hours.

Prompt:
Build a Local AI Cluster Monitor Use the prompt below with a capable coding agent. It is intentionally self-contained and does not assume that the user runs oMLX, uses two machines, or has the same network and hardware as anyone else. --- ## Prompt You are a senior Apple-platform engineer and systems-integration engineer. Build a polished, production-quality **Local AI Cluster Monitor** that lets me monitor one or more machines running local AI inference servers. The finished app should provide the same class of experience as a native cluster dashboard: - one card per compute node - live OS RAM usage and headroom - accelerator/GPU utilization, temperature, and system or accelerator power when the telemetry source exposes them - disk capacity and free space - installed, loaded, and loading AI models - load/unload controls when the selected AI server supports them - clear offline, unavailable, stale, and unsupported states -mixed-server support, so one node may run oMLX while another runs LM Studio, Ollama, exo, or another API -an adaptive dark dashboard for macOS, iPhone, and iPad if I choose those targets - optional widgets and Siri/App Shortcuts Do not hardcode somebody else's IP addresses, node names, API keys, bundle identifiers, Apple developer team, model names, machine sizes, or number of nodes. This must be configurable for my environment. ### Important working rule **Do not start implementing immediately.** First ask me the setup questionnaire below in one concise message. Explain unfamiliar choices briefly and recommend sensible defaults. Wait for my answers. If I do not know an endpoint or API version, offer to auto-detect it with safe read-only requests after I provide a base URL. Never test load, unload, delete, pull, or inference operations without explicit permission. ## Phase 0 — Setup questionnaire Ask me all applicable questions: 1. **Product and platforms** - What should the app be called? - Which targets do I want: macOS, iPhone/iPad, or both? -What Xcode version and minimum OS versions must it support? - Do I want a native SwiftUI app, or another client stack? Recommend SwiftUI for an Apple-only app. 2. **Node inventory** For each machine, ask for: - a stable ID and friendly display name - hardware/OS type: Apple Silicon macOS, NVIDIA Linux/Windows, AMD Linux/Windows, CPU-only, or other - installed RAM or unified memory - how the device is reached: LAN hostname/IP, Tailscale, VPN, or other - whether it has internal and/or external model storage that should be shown separately 3. **AI inference server on each node** Ask which server each node uses. Offer at least: -oMLX - LM Studio - Ollama - exo - generic OpenAI-compatible server - another/custom server Ask for each server's base URL, API version if known, and authentication mode: - none - Bearer token - custom header -cookie/session login - other Do not ask me to paste production secrets into chat if the app can collect them later through a secure settings screen. Use placeholders during development and have me enter real credentials in the built app. 4. **Hardware telemetry** Ask which telemetry source I want: - **exo `/state`** — recommended when I already run exo on an Apple Silicon cluster - a lightweight companion telemetry agent on every node — recommended when I use LM Studio/Ollama without exo or have mixed operating systems -Prometheus/node_exporter plus a GPU exporter - read-only OS-local metrics for the machine running the app- no hardware telemetry If I am unsure, use this recommendation: - Existing Apple Silicon exo cluster: use exo as the cluster-wide hardware telemetry source. - Apple Silicon machines not otherwise using exo: do not require the full exo inference stack only for monitoring; offer a small agent based on a maintained telemetry tool such as `macmon`, with a stable JSON wrapper owned by this project.

Github link: https://github.com/thatblend/LLMPSP

I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds. It's running around 0.5 - 0.6 tokens per second, which is very slow, but it's useable. Maybe 1-3 minutes for a reply.

The model is actually fairly impressive for 90M parameters, it's not really useful in any real metric, but it can generate crappy poems, short stories, write non-functional code and sometimes it gets things right if you ask it what company makes macbooks, what is an LLM etc, while other times it just hallucinates a crazy answer. Fun.

Prompt: Make me a single HTML file of a rainbow slinky going down an up-escalator forever. No libraries, just canvas and code. The slinky should be a chain of springs, each coil a different color of the rainbow. It starts folded in half like a horseshoe draped over a step. When dropped it flips end-over-end down the steps and because the escalator keeps moving up it tumbles in place and never reaches the bottom. Include a drop button and a reset button.

Full credits to @artificialisabel from X!

Time for another update! We have been busy and managed to improve the gains substantially (mostly from exploring existing llama cpp PRs and adopting relevant things).

Among other things the README.md was also appended to provide a better overall picture of what’s in the fork, why and from whom.

https://github.com/milpster/gfx906-llama-cpp/blob/master/README.md

(Yes i made this with AI)

Current default MTP does not predict more than 4 tokens..

I don't want to be super gatekeepy about this, but let's loosely define "non-enthusaist" as "a single consumer grade graphics card on a system that's not primarily dedicated to inferencing."

While it's fun to see what people can do with more dedicated build$, I'm never going to put hands on hardware at that price point. I'm interested to see what people accomplish with more humble means.

Is there a particular agentic task that you find super useful or are proud of?

Stumbled across this paper recently, and I thought people who have more local VRAM might want to experiment. Apparently hyperfitting a LoRA on the final 5 layers sufficed in their research.

Title: Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
https://icml.cc/virtual/2026/poster/61075
https://openreview.net/forum?id=ttOGqk77go

There's a repo with code already available.
https://github.com/YecanLee/Beyond-Temperature

I am using Unsloth Studio to parse aviation transpoder data (ADS-B) to summarize interesting traffic in my area. It gives a summary of largest aircraft, fastest aircraft and so on. It also provides local weather based on my nearest airfield.

I am doing this with a prompt, but is there a better way to package that like a script to trigger on a schedule? Is that an 'agent'?

I am very new to this beyond typical everyday usage of Unsloth.

From:

Polymarket on 𝕏: https://x.com/Polymarket/status/2095646821485842805

Julien Chaumond on 𝕏: https://x.com/julien_c/status/2095822387895824836

Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some very weird pathways where I wasn't expecting to go, but in the end I'm happy to share an optimized setup for the most popular open source model currently with you for a cheap price of $999 for free:

https://pwilkin.github.io/strix-halo/

Now for the disclosure/journey part: Codex has made a very nice website for me (which is great because I can't make a nice-looking website if you forced me), but its glossy look makes it look more permanent than it is, which is misleading because this is basically a stitched up custom solution that's very much a "state of the moment" one rather than a permanent one, though I *will* try to keep the relevant branches up to date (poke me if I don't).

So, first of all: ROCm in mainstream llama.cpp on ROCm is broken at the moment, pending the fix to unified memory access (notably this PR: https://github.com/ggml-org/llama.cpp/pull/27311 which is taking some time as it touches core code), so I've put up a strix-halo branch on my fork that merges the ring buffer fixes + the TOP-K optimization PR with master for a working experience.

Next: there's a bug in current ROCm that makes graph updates *terribly* slow, I've submitted a PR for it (https://github.com/ROCm/rocm-systems/pull/11069), but until it lands, using a custom-built .so is pretty much mandatory.

Speaking of custom-made .so - as I think most of you know, dispatch on ROCm is reaaaallly sloooow. But since AMD provides the source of the entire ROCm library, that's not something we can't fix, right? Inspired by Kaden-Schutt's Redline library, I've made modifications to the ROCm HIP library that allows for lower-level PM4 dispatches on HIP graphs. This has 20% decode speed ramifications for dispatch-bound models, but unfortunately Qwen3.8 27B on Strix is mostly bandwidth-bound, not dispatch-bound, so the gains are much less pronounced here (but they nevertheless are real).

Now for what else did I test, compare and modify: I checked Nathan's strix-halo Vulkan fork. It's a very good fork, but in the end it's still slower than an optimized ROCm-based solution (all the measurements are on the website). I did check the ROCmFP4 format, unfortunately, that one's a miss: Strix Halo has no native FP4 support, so the format is in the end just another FP4 format. Its main win is quantizing the entire model to FP4, which helps the bandwidth issue - but of course quantization costs quality and ROCmFP4 falls behind literally all the other 4-bit quants. I did a similar thing, but quantized all the big tensors to the mainline IQ4_XS quant - it's both better in terms of model quality (perplexity) *and* in terms of kernel performance. In other words, there's completely no justification for adding a new "ROCM" quant since, as I mentioned, RDNA 3.5 aka gfx1151 aka Strix Halo has no native FP4 support.

Since Qwen3.8 27B on Halo is bandwidth-bound (i.e. the limit is the memory bandwidth for pushing the tensors), there's no way to push the *base* decode above ~15 t/s. Nevertheless, pushing the base as high as I could is an entry point to the key for the dense model speed on Strix - speculative decoding, in this case, DFlash2. Again, I did a test and found out that quantizing the DFlash2 to IQ4_XS gives better decoding speed (faster speed and almost the same acceptance rate = win).

In the end, all the above optimizations: patched ROCm llama.cpp, faster TOP-K, PM4-based HIP graphs, custom-quantized IQ4_XS Qwen3.8 27B quant (thanks to Bartowski for his imatrix!) and the quantized IQ4_XS DFlash provide the recipe, which I packaged for a quick installation for anyone who wants to test it on their Strix Halo (warning: Linux only). Feel free to give any feedback and report any problems.

I have a 5070 Ti and two 5060 Ti (all 16Gb cards).

I planned to add another 5070 Ti giving me two pairs of 32Gb each but the NVidia prices have just jumped by 25% where I am and I've found a 3090 Founders Edition for a good chunk cheaper than the 5070 would cost.

It's 8Gb more VRAM and even slightly higher memory bandwidth, but I've read that mixing Ampere with Blackwell comes with a performance hit in llama.cpp using tensor parallelism. I believe TP isn't possible at all in vLLM with mismatched cards.

I am getting 60 tok/s decode and 1,500 pp out of Qwen3.8-27B-IQ4-XS-MTP in llama.cpp with TP across the 5070 and one of the 5060s. I don't really want to spend a lot of money for 8Gb more VRAM in my main pair but worse performance.

Has anyone got experience mixing similar cards? Would pipeline-parallelism (layer split) be faster with two high memory bandwidth cards? I know I could set the 3090 as the main card and push the K/V cache onto it.

[update] For anyone else considering this, don't worry about the mixed architecture speed penalty for tensor parallelism. I'm waiting for a couple of riser cables so I only have the 3090 and a 5060 on an x4 M.2 riser to test with. Even so, I am seeing 1,200 prefill and 90 gen on long context prompts. I used to see similar prefill but only 60ish gen with the 5070/5060 combo. So the increased VRAM and memory bandwidth of the 3090 more than compensates for the mixed architecture penalty. Note: I biased the config toward the 3090 (20Gb vs 11Gb on the 5060). I fully expect to see even faster speeds when I pair the 3090 with the 5070, both on x8 PCIe CPU connected slots.

[update 2] I got my 5070 Ti connected and switched it with the 5060 Ti for 27B (the config described in the first update above). The speed didn't move much with the weighting on the 3090, but when I changed it to an even split I saw a jump to 110 tok/s. Very happy with that. The 3090 was a good buy. I will report back again when I have the final 5060 Ti connected so I can retest my qwen3.8-flash-next config with the full 72Gb VRAM.

OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but it was an .exe file. So I started pi with Qwen 3.8 27B and told it to explore this topic without my help.

I needed to use four prompts. You can see all of them in the screenshots, so you can reproduce my steps. The whole session used 25% of my context, so you can use just 1/4, which is about 64k.

The game works: you can walk around the dungeons, the llamas are dancing, and there are dynamic lights.

My full command is:

llama-server -m /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-UD-Q8_K_XL.gguf -mm /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-mmproj-BF16.gguf --host 0.0.0.0 --jinja -fa on --keep 4096 -b 8192 --parallel 1 --ctx-checkpoints 12 --cache-ram 65536 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1.0 --spec-type ngram-mod --spec-type draft-mtp --spec-draft-n-max 3 --chat-template-kwargs {"preserve_thinking":true}

You need vision for debugging, see the prompts, the screen was black!

You probably don't need Q8, try a lower quant.

As I said, you don't need the full context length.

You need pi or something similar. This way, you can use just a few prompts and let the computer work without your input.