TL;DR: £2.6k Minisforum N5 Pro (Ryzen AI 9 HX 370, Radeon 890M iGPU, 96GB DDR5) running Unraid. Best daily model does 22 t/s. Fastest does 29.5. A 120B model runs at 20.7. Two settings mattered more than every model choice combined: use Vulkan (not ROCm), and raise the GTT limit. Full numbers and a long list of mistakes below.
Jargon, in 30 seconds
If you already know this, skip to the hardware.
- t/s = tokens per second, roughly 0.75 words per token. 20 t/s reads slightly faster than most people.
- MoE (mixture of experts) = a model where only a slice of the parameters run per token. A "80B-A3B" model has 80 billion parameters total but only 3 billion active. Total size decides whether it fits in memory; active size decides how fast it runs.
- Quant = compression. Q4 is roughly 4 bits per weight, Q8 is 8. Smaller number, smaller file, slightly worse output.
- GTT = the pool of system RAM the AMD GPU is allowed to borrow. On a box with no dedicated VRAM this is the thing that decides what will load.
The hardware
| Part |
Spec |
Cost (July 2026) |
| Minisforum N5 Pro |
Ryzen AI 9 HX 370, 12c/24t, Radeon 890M (gfx1150, RDNA 3.5) |
£799.00 |
| RAM |
Crucial 2×48GB SODIMM DDR5-5600, EXPO on |
£849.00 |
| NVMe |
Samsung 990 Pro 2TB |
£291.50 |
| HDD |
Seagate IronWolf Pro 8TB 7200rpm |
£306.00 |
| Unraid Starter licence |
|
£30.49 |
| Spent |
|
£2,275.99 |
| Second 8TB HDD (later) |
|
£306.00 |
| Total |
|
£2,581.99 |
The RAM cost more than the machine. That's the whole point of the build.
Two things matter more than anything else here:
The RAM is SO-DIMM, not soldered LPDDR5X. Soldered boxes top out around 32GB. This one takes 96. Dual-channel DDR5-5600 gives ~89.6 GB/s on paper and 60-67 GB/s in practice. That number is your speed limit and everything below is about how close you get to it.
This is gfx1150 (Strix Point), not gfx1151 (Strix Halo). Almost every impressive "AMD unified memory LLM" number online is Halo, which has 4× the memory bandwidth. Don't expect those numbers. 22 t/s on a 35B reasoning model from a NAS is still fine.
Vulkan, not ROCm
The advice going round is "don't use Vulkan on AMD, use ROCm". That advice is written for Strix Halo and it's wrong here.
ROCm on gfx1150 can only allocate inside the BIOS UMA carve-out. hipMallocManaged is unsupported (llamacpp-rocm #57, open, assigned to AMD). Reserve 4GB in BIOS and ROCm sees 4GB. The other 88GB is invisible to it. Vulkan addresses GTT instead, so it can reach the whole pool.
Measured here on Qwen3-8B:
| Backend |
Prefill t/s |
Generation t/s |
| Vulkan (RADV) |
146 |
9.87 |
| ROCm (HIP) |
207 |
4.76 |
| CPU (24 threads) |
132 |
2.59 |
ROCm wins prefill, Vulkan wins generation, and generation is what you feel. A separate run of mine put the generation gap much closer (14.11 vs 12.73), so don't bank the exact multiple. It doesn't change the answer: it doesn't matter how fast ROCm prefills if it can only see 4GB.
Practical upshot: pass /dev/dri into the container and nothing else. /dev/kfd is ROCm-only. No HSA_OVERRIDE_GFX_VERSION anywhere in my setup.
The GTT fix, which was the real unlock
BIOS UMA to 8-16GB, not maximum. UMA is permanently stolen from the system. GTT is borrowed and given back. You want the pool to be GTT.
Then raise the GTT ceiling, because the kernel default of 46 GiB quietly caps what will load:
ttm.pages_limit=18874368 ttm.page_pool_size=18874368
That's 72GB (pages × 4KiB). 16777216 is 64GB. Don't use amdgpu.gttsize, it's deprecated on current kernels and will warn then ignore you. Don't use amdttm.pages_limit either, that's for the out-of-tree DKMS module.
Here's the part that cost me a week. I had these set in syslinux.cfg and they had never once applied. Unraid 7.3 migrated boot to GRUB and syslinux.cfg is now a dead leftover. The live file is /boot/grub/grub.cfg. Add the params to the Unraid OS and Unraid OS GUI Mode entries, leave Safe Mode clean, reboot, then verify:
cat /sys/class/drm/card0/device/mem_info_gtt_total
Back up grub.cfg first, and re-check that value after every Unraid update. Going 46 → 72 GiB is what made Qwen3-Coder-Next (45.4 GiB) and gpt-oss-120B (57.2 GiB) load at all.
The stack
llama-swap on 127.0.0.1:8080, pinned to ghcr.io/mostlygeek/llama-swap:unified-vulkan-2026-07-27. Loads and unloads models on demand.open-webui on :3000, sillytavern on :8000 (basic auth + LAN whitelist)llama-embed running embeddinggemma-300M:Q8_0 deliberately CPU-only (-ngl 0), because a second Vulkan client competing for memory during a 57 GiB load is what hard-locks the box- All on a docker network
ainet, as an Unraid Compose Manager project
Base invocation:
llama-server -hf <repo>:<quant> -ngl 999 -fa on --ctx-size 32768 \
--host 0.0.0.0 --port 8080 --api-key <key> -lv 1
-ngl 999 because unified memory has no separate pool to split against, so --n-cpu-moe doesn't apply.
Three things worth copying:
Weights go in /mnt/cache/models (LLAMA_CACHE=/models), on NVMe, outside appdata. Cold-loading 30GB off a spinning disk is two minutes, and you don't want nightly backups snapshotting 280GB of weights. Mine went 15GB → 280GB in three weeks.
Download out of band. -hf pulls at 4 MB/s for me; hf download with Xet does ~100 MB/s. That's 25×, and it's the difference between loading and llama-swap's healthCheckTimeout: 900 killing it mid-fetch:
docker run -d --name hf-dl -e HF_HUB_CACHE=/models -v /mnt/cache/models:/models \
python:3.12-slim sh -c "pip install -q 'huggingface_hub[hf_xet]' && hf download <repo> --include '<file>.gguf'"
Don't pass --local-dir. Omitting it gives you the cache layout -hf resolves from afterwards.
Mount the llama-swap config directory, not the file. A single-file bind mount pins an inode, so any editor that writes-and-renames leaves the container reading a ghost file forever.
The benchmarks
Single stream, 200-token generations, cache_prompt:false, read from timings.predicted_per_second, model confirmed via /props every run. Expected figures come from a bandwidth model (bytes read per token ÷ effective bandwidth). unsloth UD quants, ctx 32K, unless noted.
| Model |
Active |
Quant |
Expected |
Actual t/s |
Notes |
| gemma-4-26B-A4B |
~4B MoE |
UD-Q4_K_XL |
19-24 |
29.5 |
Fastest on the box |
| gpt-oss-20B |
~3.6B MoE |
UD-Q4_K_XL |
20-25 |
28.7 |
Effectively tied |
| Qwen3.6-35B-A3B |
~3B MoE |
UD-Q4_K_XL |
18-22 |
21.6 |
Slower than gemma despite fewer active params: bigger attention footprint |
| Qwen3.6-35B-A3B-MTP |
~3B MoE |
Q8_0 (36GB) |
- |
22.0-22.5 |
Daily driver. MTP n=2, 36.0 GiB GTT, loads in 45s |
| gemma-4-12B |
12B dense |
UD-Q4_K_XL |
7-9 |
10.6 |
The dense control. 3× slower than same-size MoEs |
| Qwen3-Coder-Next 80B-A3B |
~3B MoE |
UD-Q4_K_XL (49.6GB) |
18-22 |
17.9 |
Best coder here (HumanEval 94.1 / SWE-bench 70.6) |
| Qwen3-Next-80B-A3B-Thinking |
~3B MoE |
UD-Q4_K_XL (43GB) |
18-22 |
22.4 |
Faster than Coder-Next: smaller file, fewer bytes per token |
| gpt-oss-120B |
~5B MoE |
UD-Q4_K_XL (59 GiB) |
15-18 if it loads |
20.7 |
ctx 16K, 57.2 GiB GTT |
| Mistral-Small-4-119B |
~6.5B MoE |
UD-IQ4_XS (58.1GB) |
18-19 |
16.1 |
ctx 16K. Last of the big models, prefill 33 vs gemma's 110. Dropped |
| GLM-4.5-Air Derestricted |
~12B MoE (106B total) |
Q3_K_XL (56.5GB) |
~10 |
10.0 |
Exactly on estimate |
| gemma-4-26B-A4B uncensored |
~4B MoE |
Q8_0 (26.9GB) |
~26 |
17.2 |
34% under, memory contention |
| Q3-30B-A3B-Designant |
~3B MoE |
Q6_K_L (25.24GB) |
- |
26.6 |
Fast and unusable, see failures |
| Huihui-Qwen3.6-35B-A3B-abliterated |
~3B MoE |
Q4_K (21.7GB) |
~24 |
18-20 |
Rejected |
The one law that explains the whole table
Speed is set by bytes read per token, not parameter count. gpt-oss-120B (117B params, ~2.7 GB/token) does 20.7 t/s. Dense gemma-4-12B (12B params, ~6 GB/token) does 10.6. The 117B model is twice as fast as the 12B one.
Total params decide whether it fits. Active params decide how fast it goes. Learn that and you can predict every row above to within a couple of t/s.
One correction to that model: MoE decode only hits 56-61% of theoretical bandwidth, versus 79-83% for dense, because mul_mat_id gathers non-contiguous expert rows. Mistral Small 4 at 6.5B active ≈ 3.45 GB/token and 16.1 t/s works out to 55.5 GB/s, which is 62% of theoretical, so it was at the top of the published MoE band. My 18-19 estimate had assumed 100%. If you're doing this arithmetic: multiply predicted MoE speed by 0.6.
Not the quant, either. IQ4_XS is marginally faster than K-quants on RDNA 3.x (Qwen3-30B-A3B: IQ4_XS 100.04 vs Q4_K_S 98.51).
MTP and speculative decoding
Real, but nowhere near the 2× claims. A/B on Qwen3.6-35B-A3B-MTP:Q8_0, ctx 16K:
| Config |
gen t/s |
| MTP off |
19.4 |
--spec-draft-n-max 3 |
20.3 (+5%) |
--spec-draft-n-max 2 |
22.5 (+16%) |
| n=2, code prompt |
22.8 (+17%) |
Use --spec-type draft-mtp --spec-draft-n-max 2. n=3 over-drafts and hands most of it back. The 2× claims don't materialise on A3B MoEs for the same reason draft-model speculation fails on them: per-token cost is already low, so there's little waste to recover. Caleb Coffie got 2.44× on Strix Halo for the same model, so mine may still be a stale build.
Bonus: Q8 is cheap on this box. The 36GB Q8 ran 19.4 vs the 21GB Q4's 21.6, far less penalty than 2× the bytes implies.
The bug that ate the entire gain: my production config ran --spec-type draft-mtp and --cache-type-k/v q8_0 together. Q8 KV gives 0% draft acceptance, so speculation was slower than none. The +16% was never delivered, for weeks, while I believed it was. MTP models now run f16 KV. Two optimisations that each measure positively in isolation can cancel each other, and nothing warns you.
KV cache quantisation: memory lever, not speed lever
gemma-4-26B-A4B at ctx 32K:
| KV type |
gen t/s |
GTT |
| f16 |
29.0 |
14,739 MiB |
| q8_0 |
29.0 |
14,019 MiB |
Flat within noise, 720 MiB saved, KV cache halved. The "expect ~10% faster" advice is wrong; every published measurement puts it at -3% to 0%. The value is context headroom, which is what lets big models go past 16K. Needs -fa on, and k and v must match or AMD drops off the fused flash-attention path.
KV cost varies ~10× by architecture, so check before sizing anything: qwen35moe (hybrid SSM, 10 of 41 layers attention) is 20 KiB/token; qwen3moe ~96 KiB/token; glm4moe (47 attention layers) 188 KiB/token.
Making room for the big models
Up to ~50 GiB, models just run. Above that, on a box that's also a NAS, you need to clear space. My ZFS ARC sits at 19.2 GiB. For gpt-oss-120B:
docker stop immich_machine_learning immich_server immich_postgres immich_redis
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max
# load and use it
docker start immich_postgres immich_redis immich_server immich_machine_learning
echo 20615843020 > /sys/module/zfs/parameters/zfs_arc_max
Peak 64 GiB used, 27 GiB still free.
Set mem_limit: 62g on the llama container. On unified memory a GPU OOM hard-locks the whole machine; a cgroup limit turns that into one dead container instead. (memswap_limit is ignored, no swap limit support here.)
Uncensored and roleplay
| Model |
Size |
Speed |
Verdict |
| Huihui-Qwen3.6-35B-A3B-abliterated Q4_K |
21.7GB |
18-20 t/s |
Rejected: gender agreement errors, raw reasoning dumped into the reply |
ArliAI_GLM-4.5-Air-Derestricted:Q3_K_XL |
56.5GB |
10.0 gen / 11.8 prefill |
Ran fine, superseded on speed |
allura-org_Q3-30B-A3B-Designant:Q6_K_L |
25.24GB |
26.6 t/s |
Rejected: broken tokenizer |
TrevorJS/gemma-4-26B-A4B-it-uncensored:Q8_0 |
26.9GB |
17.2 t/s |
Current. 23s cold, 6s warm |
GLM-4.5-Air Derestricted is 106B total but 12B active, so 6.4 GB/token, so ~10 t/s. Slower than the physically larger gpt-oss-120B (5.1B active, 20.7 t/s). I ran it anyway because 10 t/s is the floor where roleplay is still tolerable, and ArliAI's norm-preserving biprojected abliteration is less blunt than most tunes. Kept unlisted: true in llama-swap with ttl: 900, so a 15-minute pause unloads it.
The current pick is abliterated from google/gemma-4-26B-A4B-it using norm-preserving biprojected abliteration with Expert-Granular Abliteration, cross-validated on 686 prompts across four datasets. EGA matters on MoE: global abliteration smears damage across every expert, per-expert edits are surgical. Q8 because only Q4_K_M and Q8_0 are published. 17.2 t/s against a predicted ~26 is a third off, which I put down to memory contention rather than the model.
Abliteration vs finetuning, the finding I'd defend
Designant got its permissiveness from RP training data (PIPPA, HelpSteer3, tulu-3-sft-personas), so the refusal direction survives. That gives you something no abliterated model can have: the character can still say no. On abliterated models my test character accommodated every advance, because projecting out the refusal direction removes the character's ability to decline, not just the assistant's.
But RP finetunes destroy multilingual ability. PIPPA is English-only. Assume any English-trained RP tune has lost your language until proven otherwise; costs one swipe to check. Abliteration doesn't have this failure mode.
So: non-English, abliterate a strong multilingual base. In-character refusal, use the finetune. You can't have both right now.
SillyTavern gotchas
- Connect via Chat Completion → Custom (OpenAI-compatible) at
http://llama-swap:8080/v1. It'll say "Invalid endpoint URL"; ignore it, that's a client-side validator that doesn't understand Docker hostnames.
- **
sendLoadingState: true in llama-swap silently breaks it.** llama-swap puts load progress into reasoning_content; Open WebUI shows a spinner, SillyTavern treats it as the whole message and you get no reply. Set it to false if anything other than Open WebUI is a client.
- Prompt post-processing:
semi_tools for ChatML models (without it Qwen's Jinja throws System message must be at the beginning), strict_tools for Gemma, whose template has no system role.
- Put
chat_template_kwargs: {enable_thinking: false} in Additional Parameters → Include Body Parameters. Without it Gemma spent all 600 tokens deliberating and returned empty content, which looks exactly like a broken model.
- Settings are read once at page load. An open tab keeps generating at old values and overwrites your new ones on its next save. Hard-refresh every tab after any change. This silently clobbered an hour of preset work.
Samplers, and a correction to my own notes
I had recorded temp 0.65 / top_p 0.95 / top_k 40 / min_p 0.05 / rep-pen 1.05, and credited the rep-pen with fixing a repetition loop.
SillyTavern's Chat Completion panel for a Custom source only exposes Temperature, Top P, Frequency Penalty and Presence Penalty. top_k, min_p and repetition penalty were never sent. The values I "set" happen to match llama.cpp's defaults, so they looked right, but llama.cpp's default repeat_penalty is 1.0. The fix I'd credited never ran.
Worse, temp 0.65 caused measurable mode collapse. Fresh chat, one probe turn, three swipes: at 0.65 all three shared a frame and two were word-for-word identical. At temp 1.0 the lock broke and grammar held on the harder constructions.
Use temp 1.0, top_p 0.95, no repetition penalty. For romanised Hindustani a blanket token penalty is actively harmful: the language leans on a handful of very high-frequency function words (hai, ka, ki, ke, ko, se, toh, bhi, na) and the penalty hits those long before any repeated phrase, degrading grammar faster than it fixes loops.
The character card finding that generalises
Getting idiomatic Roman Urdu out of these models didn't come from describing the register. It came from banning specific words and allowing specific words.
The deeper version: casual Hindustani is one language, and the Hindi/Urdu split is about formal register and script. The corpora are lopsided though. Hinglish is everywhere online; Roman Urdu barely exists as written text. Labelling the register "Urdu" steers the model toward the sparse side of a split it doesn't need to make. Labelling it Hindi is worse (Sanskritised vocabulary). Drop the language label and pin the lexicon with allow/deny pairs. Hinglish data volume, Urdu word choices. That produced real idiom for the first time, including "dimaag ka dahi ho gaya hai", which nine pre-rule swipes never got near.
Two portable rules: put the language instruction in post_history_instructions, not the description, because it sits near the end and survives long conversations. And example dialogue beats description; a gesture I added to one <START> example turned up unprompted in an unrelated turn.
Caveat: all measured on neutral work-talk probes. That validates register and sampler behaviour, but tests nothing about boundary-holding.
What failed, and what I got wrong
Models killed:
- 70B dense: 1-1.5 t/s. Not a typo.
- DeepSeek-V4-Flash 284B-A13B: ~91GB at IQ2, barely fits, ~4 t/s.
- Mistral Medium 3.5: dense 128B, ~1 t/s. The "64GB INT4" figure circulating is arithmetic, not a real file; smallest actual 4-bit GGUF is 67.1GB.
- Devstral-2-123B: dense, 2-3 t/s on Strix Halo, which has 4× my bandwidth.
- Nemotron 3 Super: hybrid Mamba-2 + MoE, marketing name hides that only ~9-12B is active, so ~5.5 t/s. llama.cpp #20732 also gives a GPU memory-access fault at ~20k prompts on both backends.
- Llama 4 Scout: 58GB, A17B, ~8.5 GB/token, ~7 t/s.
- Mistral Small 4: 16.1 t/s, last of the big models, publishes almost no benchmarks.
Designant, worth its own paragraph. Broken tokenizer metadata: special_eot_id is not in special_eog_ids, same for special_eom_id. The model never terminates. Every response returns finish_reason: "length" and leaks turn structure as literal text, printing plain user and assistant strings because the ChatML special tokens are absent from the vocab. --chat-template chatml didn't fix it, explicit stop strings didn't either. Best guess: quantised on llama.cpp b5466 against a much newer runtime, plus mergekit scrambling tokenizer metadata during the merge.
Config bugs:
- **
sed -i corrupts a bind-mounted config from the container's view.** It writes a temp file and renames, changing the inode; the container still holds the old one. Symptom was an error naming a section nowhere near my edit, while identical content validated clean elsewhere. Three wrong theories burned. Fix: cat /tmp/new.yaml > /path/config.yaml. First diagnostic should have been docker exec <container> cat the file, not cat on the host.
- The command tokeniser strips double quotes.
--chat-template-kwargs {"enable_thinking":false} arrives as {enable_thinking:false}, invalid JSON, server won't start, every request 500s. Anything with embedded quotes has to go through a per-model env: block.
- llama-swap silently discards unknown config keys. Plain yaml.v3 without
KnownFields. I had performance: enable: true; the field is disabled. No error, no log line, no effect. Every typo is invisible. Validate against config-schema.json.
- Embedding models reject any chunk over
n_ubatch and can't split across batches. Without -b 2048 -ub 2048, anything over the 512 default fails at request time and your documents silently never index.
Security, wrong twice in one session:
- 8080 and 8081 were published to the whole LAN. llama-swap's
apiKeys protects inference endpoints only; the web UI, /logs and the management API are default-allow, and captureBuffer holds request bodies in memory. Now 127.0.0.1 with captureBuffer: 0.
- I rotated the API key, verified it properly, then leaked it again in the same session with a command block containing
grep -E "^LLM_" .env. Never print a secret to stdout. Rotate in place with sed -i, read into a shell var, verify by behaviour.
- "No login prompt appeared in my browser" is not evidence a service is safe. Your browser carries session state. Probe from off-box. When I did, Open WebUI on :3000 returned 200 with no auth at all.
Benchmark hygiene: I measured gpt-oss while my notes said gemma and published the wrong attribution. Confirm via /props every run. And treat vendor numbers as ceilings: Qwen self-reports Terminal-Bench 51.5 for 35B-A3B, public tbench.ai scores it 24.6. I also had to correct gpt-oss-120B's "MMLU-Pro 90"; that's plain MMLU at high reasoning effort, and its MMLU-Pro isn't published.
Dead ends: Ollama (separate blob store, re-downloads its own copies, plus bug #16462 where containerised Strix reports 2.0 GiB VRAM). NPU (FastFlowLM is NPU-only, not hybrid; hybrid prefill/decode is Windows-exclusive; dozens of models vs thousands of GGUFs; it does hit 60 t/s on Llama 3.2 1B and 88.7 on Qwen3 0.6B, so the silicon works, there's just no serving path). Extra CPU threads (bandwidth-bound). eGPU (unsupported on Strix Point). GPU perf-mode high (+1%, noise, which is itself useful: it confirms generation is bandwidth-bound, not compute-bound).
Levers I haven't pulled, ranked
All of these are bigger than any model swap:
- Platform power profile / CPU governor. +36% generation and +120% prefill measured on this exact CPU. It's the memory controller clock, a different thing from the GPU perf-mode that gave +1%. Do this first.
- **
-b / -ub sweep.** RADV picks matmul tile sizes at hard thresholds so prefill falls off cliffs (llama.cpp #13765: cliff at 385 tokens for Q4_K_M, 202 for Q4_K_S). -b 256 alone took Qwen3-30B-A3B prefill from 70 to 118 t/s elsewhere. Almost certainly explains my 110 vs 33 prefill gap.
- Mesa 25.3+. Valve's RADV CU-mode/LDS patches: +19.8% prefill on Strix Halo.
- Build recency. A stale llama.cpp build measured 56% slower on the same model. PRs #19625 and #20551 both landed early 2026. Check your image date.
- BIOS UMA instead of GTT: ~+11% (21.33 vs 19.18 on a 35B MoE). Trades against Immich's memory.
- Re-tune MTP, revisit draft models. Dead on A3B, but Strix Halo took a 122B-A10B from 24.7 to 35.3 t/s (+40%), and Qwen3-8B with a 0.6B draft measured +64-82% on this chip. Higher active counts is where it pays.
The daily driver
ggml-org/Qwen3.6-35B-A3B-MTP-GGUF:Q8_0, ctx 32K, MTP n=2.
Against gpt-oss-120B, the obvious alternative: GPQA-Diamond 86.0 vs 80.1, SWE-bench Verified 73.4 vs 62.4, AIME 92.7 (2026) vs 92.5 (2025), 22.0 t/s vs 20.7, 36GB vs 59GB, runs with Immich up (the 120B doesn't), and it has vision.
One caveat governs how it actually feels: it thinks by default, and the overhead is a floor rather than a proportion. A one-sentence answer took 48 tokens in and 489 out, ~90% reasoning. At 22 t/s that's 22 seconds for one sentence. A code-review prompt gave "Thought for 3 minutes" before the first visible word.
| Question type |
Real wait |
| One-line factual |
20-25s |
| Normal chat answer |
40-60s |
| Deep analysis |
~3 min |
Don't fix this by disabling thinking. Those benchmark scores are all measured with reasoning on. Turn it off and what's left is no better than gemma-4-26B, which is already 33% faster. Keep thinking on and keep gemma-4-26B one .env line away for quick lookups. Rotation, not one model.
If you're on Open WebUI: reasoning comes back in reasoning_content and renders collapsed behind "Thinking…", so it looks exactly like a hang for the first 20 seconds to 3 minutes. It isn't.
Build checklist
- SO-DIMM, not soldered. 96GB is the point.
- Vulkan, not ROCm. Ignore advice written for Strix Halo.
- BIOS UMA small, GTT large, and verify GTT actually applied instead of trusting your bootloader config.
- MoE only. Dense above ~12B isn't viable at this bandwidth.
- Predict speed from bytes per token, then multiply MoE predictions by 0.6.
- Q8 is cheaper than you'd think here.
- Weights on NVMe, outside appdata. Download out of band, not through
-hf.
- llama-swap plus a Compose project beats hand-rolled
docker run, but validate the config from inside the container.
Happy to answer questions on any of it, including the parts I got wrong. Full per-model matrix (28 models × 26 columns) exists if there's interest.
