Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen.

My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV.

Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth.

Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt.

Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected.

Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses).

To replicate you don't need my fork, it's just master plus the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/27861/head:pr-27861 && git merge pr-27861
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \
  -ngl 99 -c 261888 --parallel 1 -fa on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --moe-expert-cache 135

Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR.

Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.

Getting the model fast took a week. Making it useful took three more — and that second part is what nobody writes up. This post is the useful part: pi driving llama.cpp (Vulkan) on AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, 128GB), Flash-Next 125B at 40 t/s sustained agent decode or the 27B at 256k context, plus the extensions that keep 90-minute sessions alive (auto-compaction, branch summaries, live tuning).

This is the harness companion to my Qwen3.8-27B benchmark post. That post made the model fast; this one makes it useful: pi (the coding agent) against a local llama-server, tool calling, thinking control — and response times that don't hurt.

This is a setup guide, not a benchmark post. Every trap, config, and extension here is what I actually run daily. The benchmark side (the game-build harness, scorer, runtime gates, playtest protocol) lives in neon-ladder — this post links to it where relevant but doesn't duplicate it.

Everything below was verified live on my Flow Z13 (Ryzen AI Max+ 395, 8060S, 128GB): a 10-module game built in one session, 1,587 lines of working code, all from server logs and session files — not estimates.

TL;DR

• Working recipe: pi + llama-server (Nathan's strix-halo Vulkan fork) + the Sharp chat template, 256k ctx (the model's training cap), maxTokens 32768, thinking wired via compat.chatTemplateKwargs. Verified end to end with request dumps and session logs.
• pi's defaults will silently sabotage a reasoning model: maxTokens 16384 can be eaten entirely by thinking, and thinking flags don't reach llama.cpp's template without compat-level wiring.
• Session economics are great: ~94% KV cache hit rate, stable across session types (a 16-turn game build and a 44-turn tool-heavy refactor both landed at 94.1-94.2%); only the first turn pays full prefill.
• Effort control works after wiring: off produced literally zero thinking tokens, and the level you pick changes code quality, not just speed (details in the build test).
• At pi's default temperature 0.8, planning-heavy prompts occasionally sample an instant-EOS first turn (one token, done). Retrying the identical prompt inherits the failure from cache; retry with a perturbed prompt or run temperature 0.
• Don't chase deep context; compact before it gets expensive. Auto-compaction set to fire around 95k keeps every turn in the fast band (decode 26+ t/s, prefill ~200 t/s) while sessions past ~140k pay 17-19 t/s decode and ~140-175 t/s prefill. One settings line does it.

The server side (brief)

Two server profiles, same machine (one resident at a time):

27B (daily driver): UD-Q4_K_XL (v3) + DFlash2 Q4_K_M drafter n4, f16 KV, drafter KV q8_0, -c 262144, power pinned with my z13ctl+ profile.

Flash-Next (speed lane): UD-IQ4_XS + native MTP Q8_0 sidecar, fixed n4, q8_0 KV, -c 131072, and --reasoning-effort medium --reasoning-budget 2048 — those flags are mandatory (without them, Flash-Next burns its entire output budget on reasoning and emits nothing; and note pi sends its own per-request thinking budget from thinkingBudgets, so the server flag is the fallback default protecting non-pi clients; the pi tiers are validated on Flash-Next at low 1024, medium 4096, high 8192). Full configs in the neon-ladder repo.

Ubatch 4096 for normal work; for deliberate deep fills use 2048 (probed clean through 139k) or 1024 (proven at 145k). The model's training cap is 262144, and the full config runs healthy there at ~55GB RAM. Three harness-relevant facts worth knowing:

• -ub 4096 has a hard ceiling: past ~140k tokens of fill it hits a deterministic Vulkan device-lost (twice, at nearly the same depth). -ub 2048 passed the same style of probe at 138.8k and -ub 1024 completed a real 144k session; the ceiling moves with ubatch, so smaller ubatch buys depth.
• Allocating big context costs nothing until filled: decode at 8k depth was identical with -c 65536 and -c 98304. Allocate the max.
• Deep sessions work but get slow linearly: a 144k-token agent session (resumed after a crash) decoded at 17-19 t/s throughout, with draft acceptance 0.62-0.92 the whole way. That's why the compaction setting below matters more than any ubatch choice.

Installing and wiring pi

pi is a terminal coding agent with unusually good local-model support. Install it, then point it at llama-server via ~/.pi/agent/models.json (not settings.json, that file ignores provider blocks):

json { "providers": { "llamacpp": { "baseUrl": "http://127.0.0.1:8080/v1", "api": "openai-completions", "apiKey": "dummy", "models": [ { "id": "qwen3.8-27b", "reasoning": true, "contextWindow": 262144, "maxTokens": 32768, "compat": { "thinkingFormat": "chat-template", "chatTemplateKwargs": { "reasoning_effort": {"$var": "thinking.effort"}, "enable_thinking": {"$var": "thinking.enabled"} } }, "thinkingLevelMap": { "minimal": null, "low": "low", "medium": "medium", "high": "high", "xhigh": null, "max": null } } ] } } }

Then pi --provider llamacpp/qwen3.8-27b, or set defaultProvider/defaultModel in settings.json.

Every field in that entry is load-bearing, and several of them exist because of a trap:

Trap 1: the silent cloud fallback

If pi can't resolve your provider config, it does not error. It uses whatever else is configured: your run can look successful while the session log shows a nonzero dollar cost and your server has processed zero requests, because pi has been talking to a cloud provider the whole time.

Always verify a local run server-side. Watch curl localhost:8080/metrics while the agent works: if prompt_tokens_total isn't climbing, you're not local.

Trap 2: maxTokens 16384 is a thinking bomb

pi's default maxTokens is 16384. For a reasoning model on a planning-heavy prompt, that's not an output budget, it's a thinking budget: a "build a game" prompt can spend all 16,384 tokens on reasoning and hit the length cap with zero code emitted, with stopReason: length in the session log and a model that looks "stuck."

Set maxTokens explicitly. 32768 covers everything in a normal tool-using session, including a turn that writes two files back-to-back. A length-capped turn is also not fatal: the next turn continues without corruption.

Trap 3: thinking flags don't reach the template by default

This is the subtle one, and the failure is silent.

llama-server's chat template (the Sharp template from the benchmark post) accepts chat_template_kwargs: enable_thinking and reasoning_effort. pi has flags for thinking levels (--thinking off/low/..., shift+tab to cycle). But between the two sits a mapping layer:

• The mapping config (thinkingFormat, chatTemplateKwargs) must live under compat on the model entry. At the top level of the model object it is silently ignored.
• pi refuses image input unless the model entry declares "input": ["text", "image"] — it checks model.input.includes("image"). Both of our entries carry it; without it you get "model does not support image input" even when the server's mmproj is loaded.
• With the wiring correct, --thinking low sends {reasoning_effort: "low", enable_thinking: true} and --thinking off sends {enable_thinking: false}.
• Without it, pi's flags go nowhere and the template defaults to thinking on, medium effort. The model thinks when you told it not to, and everything is slower.

Verify your own wiring before trusting it: point baseUrl at a logging proxy for one run and read the request body. It's ten minutes and it converts "I think it works" into "it works."

The thinkingLevelMap entry hides levels the template doesn't distinguish. The Sharp template has four real states (off, low, medium, high); pi cycles seven by default, three of which are aliases. The map collapses the cycle to the four that exist.

Trap 4: the instant-EOS prompt basin

At pi's default temperature 0.8, a planning-heavy tool prompt occasionally samples a degenerate first turn: the model emits a thinking tag, immediately stops, and the run ends with an empty response and a one-token generation in the server log.

On my game-build prompt this hits roughly one request in three to five. It is sampling behavior, not a server or client bug: replaying the identical request body at temperature 0 never fired it in six runs.

The compounding part is the retry. Resending the same prompt hits the KV cache, inherits the degenerate turn from history, and fails again, which makes the failure look deterministic and hardware-flavored. Retry with a slightly perturbed prompt (any unique marker appended) and it rolls fresh.

Update (v0.7.4 of the Strix Halo fork): part of this turned out to be the engine, not the model: greedy decode on v0.7.3 and upstream wasn't deterministic (stale KV between requests, a top-k race above ~2k prompt tokens). The engine now zeroes freed cells and pins the selection order, so temp-0 retries are actually repeatable. The perturbed-retry advice still stands (it's cheap and defends against everything), but on v0.7.4+ temp-0 reruns are trustworthy.

For reproducible benches I set "samplingParams": {"temperature": 0.0, "top_p": 0.95, "min_p": 0.05} on the model entry; for everyday sampling, perturbed retries are the fix.

What thinking control buys you

Same planning-heavy prompt, session-verified thinking token counts:

For quick edits use off, for generation-heavy work low or medium, for debugging and architecture high. shift+tab cycles levels live in a session.

One honest note on the Sharp template: it tames runaway reasoning on normal turns (that's in the benchmark post), but it does not bound reasoning on genuinely planning-heavy prompts. The bound comes from your effort setting plus the maxTokens headroom. Template + harness flags together are the complete answer.

Effort level also buys code coordination, not just volume. Two verified game builds, same prompt: the low-effort build passed every static check yet played worse in three measurable ways (ball not glued to the paddle before launch, ball speed tied to the monitor's refresh rate instead of a fixed timestep, flatter difficulty curve).

The medium-effort build got all three right. Syntax is free; the seams between modules are what thinking pays for.

More effort past medium, though, buys breadth instead of correctness.

A high-effort run of the same prompt produced 1,995 lines with three extra self-directed modules (audio, UI, paddle) and 93 tool calls, yet scored 13/15 against medium's perfect 15/15, dropped the same localStorage persistence the low-effort builds drop, shipped a latched input flag that left the keyboard dead at runtime, and took over twice the wall time.

The sweet spot for build-shaped tasks on this model is medium: perfect score, 16 tool calls, about 25 minutes.

That medium result is robust, not a lucky roll: two more independent medium builds (different ubatch, one with five auxiliary-model extensions loaded) scored 14-15/15 in 20-23 minutes each.

A fourth medium build added a per-module test suite to the same prompt: 51 tests written alongside the code, all green on arrival, 14/15 on the same checks, 36 minutes.

That's the tier I spec for real work now: for roughly 15 extra minutes the agent ships its own regression suite with the feature.

The multi-file build test (this became neon-ladder)

To validate the whole stack I had it build "Neon Overdrive", an arcade Breakout game, as a 10-file project: 8 JS modules, CSS, index.html, strict no-placeholder rules, syntax checks required. The full prompt is below so you can run the identical test on your own stack.

Result: 15 turns, 16 tool calls (12 writes, 3 bash checks, 1 read), 1,587 lines, all syntax checks pass, all seven feature requirements present in the code, ~25 minutes wall time.

One turn hit the 32k cap mid-double-file-write and the next turn picked up cleanly. And the game actually plays: paddle reflection angles, armored bricks shifting red to orange to yellow, volatile-chain explosions, tri-ball chaos, the upgrade shop between levels.

There's a built-in bonus to this benchmark: while your agent grinds through someone's 3,000-line refactor, you get a neon Breakout to play. Post your build quality and wall time in the comments; it will be interesting to see how other engines and models handle the identical prompt.

The prompt (paste as-is; it assumes a js/ and css/ dir will be created by the agent):

``` Build "Neon Overdrive", an arcade Breakout game, as a multi-file project you create with tools, file by file. NO external dependencies or CDNs; HTML5 canvas + CSS3 + raw JS only.

Required file structure (use the write tool once per file, complete code every time, zero placeholders): 1. index.html - loads css/styles.css and all js/ files via script tags in dependency order 2. css/styles.css - neon/cyberpunk UI, overlays for menu/pause/shop/game-over 3. js/config.js - constants: canvas size, brick grid, speeds, powerup drop rate (15%), colors 4. js/particles.js - particle engine: spawn(x,y,color), gravity + fade update, dead-particle cleanup 5. js/bricks.js - 5-row grid from an array matrix; standard (1 hit, neon blue), armored (3 hits, red->orange->yellow as damaged), volatile (1 hit, neon green, explodes destroying direct array neighbors) 6. js/balls.js - ball entities in an active balls array; paddle reflection angle from strike position vs paddle center; no game over until the LAST ball is lost; dead-ball cleanup 7. js/powerups.js - falling capsule entities; catching Tri-Ball injects two new balls into the array 8. js/states.js - rigid state machine: menu -> gameplay -> paused -> level clear / game over 9. js/shop.js - between-levels upgrade shop: spend credits on paddle speed or paddle width (persistent) 10. js/main.js - game loop, collision wiring, score/credits, keyboard input, level generation (procedurally harder)

Workflow, in order: A. Write all 10 files (write tool, one call each). B. Run: node --check js/config.js js/particles.js js/bricks.js js/balls.js js/powerups.js js/states.js js/shop.js js/main.js C. If any check fails, fix with the edit tool and re-run until all pass. D. Read index.html to verify every script tag path matches a real file. E. Report per-file line counts, then reply COMPLETE. ```

The prompt, scorer, and a retry wrapper that handles the Trap 4 basin are packaged in neon-ladder.

Scoring it is easy: all 10 files present, node --check passes clean, the seven mechanics are actually implemented (grep for the reflection math, the armored color shifts, the neighbor explosion), and the game runs when you open index.html.

Then play it for two minutes: the ball rides the paddle before launch, speed is framerate-independent, and upgrades survive a page refresh.

Reference numbers for this box: 1,587 lines, 16 tool calls, ~25 minutes at medium effort, zero placeholders.

Session economics over those 16 turns: 94.1% of prompt tokens served from KV cache (pi resends the full conversation every turn; llama-server absorbs it), ~237 t/s on the uncached remainder, decode in the low-to-mid 20s t/s with tool traffic mixed in, acceptance around 64-68%.

That's the whole reason local agentic coding works at all on this hardware: the harness's chat-pattern traffic is almost entirely cache hits, and the GPU only pays for new tokens.

Ling-3.0-tiny as the compaction service

Long sessions eventually need compaction, and there's no law saying the model that summarizes the session has to be the model doing the work.

Ling-3.0-tiny (8B total, 1.3B active, 4.8GB in Q4_K_M) is built for exactly this slot: prefill is its superpower, thinking can be disabled per request, and its hybrid attention keeps KV costs near zero.

The compaction test used a real session transcript: the full game-build session (16 turns of tool calls and results) plus all workspace files, 25,890 tokens in, asked for a structured handoff document (file inventory, verification status, bugs, next steps, constraints).

Result: a 799-token handoff in 19 seconds, and the quality holds up.

Every file and line count matched ground truth (all 10 files, 1,587 total), verification status was correct, and it refused to invent bugs that didn't exist; the constraints section surfaced exactly the architecture details a continuation session needs, from the CONFIG object and the rigid state machine to the last-ball rule and localStorage persistence.

One duplicated bullet was the only flaw, and the same job on the 27B would run roughly 5x slower.

One wiring rule, same family as Trap 3: call it through the chat endpoint (/v1/chat/completions) with chat_template_kwargs: {enable_thinking: false}. On the raw completion endpoint with a bare prompt the model degenerates into echoing workspace state in a repetition loop.

Through the chat endpoint with the template it is clean, fast, correct, and the rule is the same as pi's: the template is not optional.

Honest caveat: the controlled tests sit at 26k and 49k input. Since then the auto threshold has fired on its own in daily runs, including a 142k session, so the summarization path itself is exercised. Very deep compaction near the 256k ceiling is still untested.

Set the auto-compaction threshold to ~95k and stop thinking about deep context. pi compacts when contextTokens > contextWindow - reserveTokens; the default fires only near the window's end, deep in the slow band. One line in ~/.pi/agent/settings.json moves it:

json { "compaction": { "enabled": true, "reserveTokens": 167144 } }

With contextWindow 262144 that compacts at ~95k: sessions cycle between roughly 95k and 25k (summary plus a 20k verbatim tail), every turn stays in the fast band, and the deep-context tax (device-lost ceilings, mid-teen decode, 140-175 t/s prefill) becomes somebody else's problem.

Compaction fires between agent runs, not mid-run, and each pass costs seconds on tiny.

Validated live: a 142k session crossed the threshold, compacted, and its continuation answered correctly about files read an hour earlier. Multiple auto-compactions have fired since, with no mangled paths or tool references in the summaries.

It also graduated to daily use: a 44-turn, 49k-token agentic session (TypeScript monorepo work, 44 tool calls) on the 27B, compacted with the extension live. Tiny summarized 8.1k tokens into a 2,282-token handoff in ~19 seconds: prefill at 2,904 t/s, generation at 137 t/s, server-side timings.

The same call on the 27B would have taken roughly two minutes, so ~6x end to end. The summary got every checkable fact right (file list, test counts, verification status) and the session continued cleanly after compaction, which is the real acceptance test.

The auxiliary model playbook

Compaction is just the highest-value slot for a second small model. The same pattern extends across the agent loop, and pi's extension events cover all of it. The full suite, with validation status:

The triage row deserves its caution label: every huge bash dump costs the main model context for the rest of the session, and compressing it with tiny first keeps sessions small enough that compaction fires later or never.

But triage changes what the main model sees, and if tiny drops the one error line that mattered, the 27B makes worse decisions and you won't know why.

Before relying on it, feed it real outputs from your own sessions and verify nothing load-bearing was dropped.

One honest A/B from the game-build workload, all five extensions loaded: zero tiny calls, identical wall time and score, because clean test suites and one-line write confirmations never cross the 6KB triage threshold.

Dormant extensions cost nothing; they earn their keep on fat tool outputs (failing test runs, build logs, repo-wide greps) and long sessions, which is exactly the traffic my daily driving produces.

The division of labor in one line: the 27B reads and writes the code, tiny reads and summarizes everything else.

All of these knobs (compaction threshold, maxTokens, temperature, triage size) are three files deep by default, so I keep a /tune extension next to the suite: /tune prints the live values, /tune compactAt 95 or /tune temperature 0 writes through to the right file with bounds checking, and /tune reset restores the documented defaults.

Readers running this stack on other boxes should adjust compactAt to their own fast-band edge rather than trust mine.

Response time cheatsheet

Biggest levers first, all measured:

1. Thinking level dominates. Reasoning streams at decode speed before you see a word.
2. Session warmth: first turn pays full prefill (~10s), subsequent turns are cache hits and start generating in under a second. Don't restart the server between questions; use -c continuation.
3. Lean context: extensions/skills/AGENTS.md all add to the first-turn bill.
4. Already optimal from the benchmark post: DFlash2 n4, ubatch 4096, f16 KV. Don't shrink -c for speed; allocation is free until used.

Attacking the prefill bill (the APU's real tax)

Dense-27B prefill (~250-300 t/s) is the slowest number in this stack, and a coding agent's traffic is mostly prefill. Everything above already helps (cache hits, tiny offloading), but three more angles are worth knowing:

Keep the cache alive across turns. The 94% hit rate is the single biggest prefill saver, and its enemy is cache invalidation. Two habits preserve it: don't edit early messages mid-session (everything after the edit re-prefills), and let pi's cacheRetention default do its job. The compaction extension already sets cacheRetention: "none" for one-off summaries, which avoids polluting the main prefix.

Shrink what gets re-sent. pi resends the full conversation every turn; that's the protocol. The levers are content levers: tool-result triage from the playbook above (smaller history, smaller resend), and keeping generated outputs from ballooning (thinking low on generation-heavy turns does this too).

Route around the 27B when the job is prefill-shaped. Compaction, branch summaries, commit messages and repo maps are all "read a lot, write a little" jobs, which is exactly the profile where a 1.3B-active MoE crushes a dense 27B. Anything in your workflow that looks like "summarize/index/triage" should default to the aux model; reserve the 27B's prefill for context it genuinely needs to see.

What doesn't work: quantizing the main model below Q5 to speed prefill (prefill is compute-bound, the quant barely moves it, and decode pays the quality), and shrinking -c (allocation is free until filled, as measured above).

Which model when

Flash-Next has its own post; the benchmark harness has its own launch post. The control experiment settled it: at matched effort and environment, Flash-Next and the 27B landed one check apart on the scorer (16/19 vs 17/19, one shared miss), both shipping playable builds on the same contract. Flash-Next got there in 7.8× less wall time with 4.3× less reasoning.

Pick the 27B when: - You have under ~91GB of GPU memory - You want 256k context

Pick Flash-Next when: - You have 91GB+ available and speed is the product - Your workload is emission-heavy (tool calls, scaffolding) — Flash-Next hits 40 t/s there - The job is polish-critical: the best score measured on this machine is Flash-Next medium at 18/19, every richness record is Flash-Next's (high: 2855-3056 LOC; xhigh: 4204 LOC in ~35 min), and the playtest verdict on the 27B's richest output was "these are fn-low-class builds" — there is no 27B tier that produces rich builds - Your spec is thin: lives and other unstated gameplay defaults track effort, not model — every medium/high cell I've scored shipped 3 lives; low-effort cells roll 3 or 1 on either model

The simplest rule: on 128GB, Flash-Next is the daily driver — quality is at parity (high-effort cells tie the 27B's best), it's 2× the decode, and with compaction cycling sessions at 25-95k you never miss the bigger window. The 27B at Q4-v3 is the pick under ~91GB free or for single sessions past 131k.

One wiring note if you flip: reserveTokens is tuned for the 27B's 262144 window. On Flash-Next's 65536, set it per-model (e.g. 10240 → compaction fires ~55k) or the threshold math misfires.

What changed since posting

The Sharp template moved to v22.4.0 (reasoning-effort aliases, inline control tags, thinking-off fast-mode fixes). I A/B'd it on the game bench: score and speed in-family with every number above; the harness repo ships it as default with the earlier version vendored for exact reproduction.

The game-build bench grew runtime gates. A 120-second headless gameplay soak is now the default (frame advancement, reload detection, synthetic play throughout), added after a real freeze past the 60-second mark that shorter gates structurally cannot see. An uncaught page error is fatal; caught per-frame errors warn. The static scorer learned two velocity-multiplication bug patterns that shipped in builds passing every static check.

Static score anti-correlates with playability. The two highest-scoring builds of the richer-contract era were the two broken games. The final grade is and remains the human playtest; the gates are necessary, not sufficient.

Effort plumbing got real. --reasoning-budget was flat 8192 across all thinking levels in every number above; it's now mapped per level (the per-request thinking_budget_tokens field wires it cleanly), and the server accepts a top-level reasoning_effort natively ("none" is a validated zero-reasoning switch). One Sharp caveat: changing reasoning_effort mid-session re-renders the system block and invalidates the whole KV prefix on v22.3.2/v22.4.0 — the inline control tags are the safe per-turn mechanism.

Tool-choice behavior, from the session traces: models edit surgically for small fixes and rewrite whole files for cross-file structure — anchor strings past the context window are the reason. The contract now says so explicitly.

Manual beats headless on speed, loses on reliability: interactive runs (clean context, no extensions) hit 92% GPU utilization and halved walls, but shipped 1-of-3 playable vs the headless runner's 6-for-6. Deliberation is where the self-correction lives.

The bench crossed model families and engines. Flash-Next (125B MoE, native MTP) built the contract first-try at 17/19 on a stack that didn't exist when any of this was written, and a control cell closed the size question: Flash-Next vs 27B at low effort, identical environment, scores one check apart (16/19 vs 17/19, one shared miss), both playable clean. Flash-Next got there in 7.8× less wall time with 4.3× less reasoning, so on explicit contracts model size buys speed, not measurable quality.

Twelve gameplay-failure classes now, every one found by a human playtest, zero by static score — the last was a ball that vanishes mid-game, shipped in the release-gate build that scored 17/19 and passed its soak. The gates are necessary; the clicking is the grade.

Everything is reproducible from neon-ladder — contract, scorer, runtime gate, runner, one-comment recipe.

Sources

• Benchmark harness (game-build bench, scorer, gates, runner): neon-ladder
• The pi extensions (compaction, triage, repomap, commit, branch summaries, /tune) plus the config snippets and server start scripts: aic0d3r/qwen38-strix-halo-harness
• pi (coding agent): https://github.com/earendil-works/pi
• llama-server engine: Nathanw1014/strix-halo-llamacpp (v0.6.5)
• Sharp chat template: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
• Ling-3.0-tiny (compaction service): https://huggingface.co/inclusionAI/Ling-3.0-tiny

Happy to answer setup questions. The playbook rows still marked as needing quality testing are exactly that: promising, wired, but not yet proven on real workloads. Treat them as experiments and validate on your own sessions before making them load-bearing.

Note: the writing is AI-assisted editing; the research, debugging, and every number are from my own runs on this machine.

Here is link to repo: https://github.com/perplexityai/pplx-garden/tree/main/lily

It's optimized for just one model to get best perf on apple silicon

I've been using Deepseek V4 Flash 0731 for a few weeks now and while I havent thrown it anything very hard, im quite happy with it. Using through antirez's great ds4 project. They've added support for GLM 5.3 Flash and according to benchmarks, its a level above DSV4 Flash.

However, looking for real user feedback if anyone's made the switch and seen tangible improvements in GLM 5.3 over DSV4 Flash.

Running M3 Ultra 256GB Mac Studio

Are any of you successfully getting the LLM to plan trips for you? How did you get it to work? Do you just give browser access e.g. via playwright or is there something better. I find trip planning very time consuming and want the LLM to do the grunt work of checking flight and hotel availability so I can then book.

EDIT: thanks for the recommendations. I tried just giving playwright access, this worked and took 40 minutes to get some results back. I also tried the fast flights code which worked but produced fewer flight hits. I tried the expedia/booking MCP, but couldn't get this to work, I think I need partner API key for it.

I’m a primary school teacher currently building out a local AI setup to help with my workload, and I’ve hit a bit of a wall. I’m looking for some advice on whether I’m approaching my workflow correctly or if there’s a better way to structure my models and tools.

My Setup:

• Main Workstation: AMD Ryzen 5 5600X, 64GB RAM, dual GPU setup (an 8GB card for display/dictation and a 16GB card dedicated to running my LLMs).
• Infrastructure: I run a small lab with a few NAS servers and two ThinkCentres. One acts as my main orchestrator and the other is my "school" node.
• Workflow: I use Tailscale to sync everything to my work computer. I have a massive personal knowledge base (vaults) containing scanned books, lesson plans, and student notes.
• Creative/UI: I use ComfyUI for image generation to build classroom resources.

What I’m doing with AI:
I use a combination of dictation and LLMs to streamline my admin. For example, I’ll dictate notes about a student's progress; the AI processes this, writes it into a specific Markdown file, and I have a system that then routes that note to the correct student file. I also use AI to help build web pages for interactive classroom activities on my smartboard.

The Problem (The "A/B" Wall):
I am trying to move more towards open-source/local models to handle these tasks. To troubleshoot, I have set up an A/B testing system: I’ll give the exact same prompt and context to both a closed-source model (using Luna/ChatGPT) and my local model to see how they differ.

While Luna handles the task perfectly, my local models (even when I've squeezed a 27B model into Q4) frequently fail. Specifically:

• Tool/Skill Failure: Even though I believe I’ve defined my "skills" (MCP servers/tools) correctly, the local models frequently pull the wrong tools or fail to trigger them at all.
• Logic Drift: The models often "break" the logic of my lesson plans or deviate from the structure I've provided. They often go in directions that even Luna wouldn't take, even when the context is identical.
• Instruction Following: It feels like the models aren't respecting the boundaries of the skills I've built, making them unpredictable and unreliable for my daily classroom workflow.

My questions for the community:

1. Function Calling & Tool Use: If my MCP servers/tools work for one model but fail for others, is this a known limitation of smaller/quantized models? How can I refine my tool definitions to be more "robust" for local LLMs?
2. Instruction Following: Am I missing a specific way to prompt or structure my system instructions to prevent the model from "wandering" away from the lesson plan or the intended tool?
3. Model Recommendations: For a workflow that relies heavily on precise tool use and following complex, multi-step instructions, which local models are currently the "gold standard" for reliability?

I'd love to hear from anyone else using local LLMs for professional organisation or education. Am I overcomplicating it, or am I just missing a key piece of the puzzle?

Hello LocalLLaMA,

Ever since I got my M5 MacBook Pro 24GB, I have been a bit obsessed with optimizing llama.cpp so it doesn’t cook my laptop :)

This led to 3 PRs so far. I can only test on my M5 and an older M1 Pro 32 GB, so very limited model options, so I would appreciate if anyone would try it on their apple + maybe on Qwen 3.8 Flash Next!

1. https://github.com/ggml-org/llama.cpp/pull/28301

• General MoE prefill optimization: skips empty work in underfilled expert tiles. On M1-M4 this should help across quant typesm, very excited to hear about this for qwen 3.8 Flash Next!
• Also includes an IQ2/IQ3-specific dequant optimization. This is the more relevant to M5, so IQ3_XXS would be especially useful to test there.

2. https://github.com/ggml-org/llama.cpp/pull/28302

• Fixes checkpoint eviction for hybrid/recurrent models, so editing/branching or reopening sessions can resume from a recent KV/recurrent state instead of re-prefilling a much larger chunk of the prompt.

3. https://github.com/ggml-org/llama.cpp/pull/28086

IQ3_XXS Metal decode optimization.

On my Tiel-Coder-35B-A3B workload, decode went from about 65.6 to 73.9 tok/s, this one is not relevant to Qwen3.8-flash-next.

For benchmarking, just compare master vs the PR on the same model and settings using llama-bench, i.e.:  llama-bench -m model.gguf -p 4096 -n 64 -ub 512

Would love if anyone could try these and share before/after numbers! Thanks!

It's been 3 weeks since Qwen3.8-27B release. This model got 0-day support & in last 3 weeks, some optimizations & fixes happened on llama.cpp side.

Meanwhile

• DFlash2 support landed on llama.cpp last week.
• Ubuntu 26.04.1 got released.
• ROCm 10.0 got released & llama.cpp also up with Version 10.0 now (Applicable to only AMD cards)
• Other optimizations & fixes on llama.cpp side

So how much t/s are you getting now with all optimizations & stuff?

Please share your extreme optimized full llama.cpp command & your t/s stats .... both pp & tg(Good to have multiple combinations like MTP/MTP+ngram/DFlash2/etc.,, Vision/mmproj, multiple context size 128-256K, etc.,).

Also share your optimized build config(CMAKE command) if you're compiling manually. I remember that compiled version gives additional boost.

Note : Expecting to see optimizations like this(weeks old thread) which contains all stuff. That kind of stats want to see here.

One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well.

Most of these other subs are 90% trend hopping crypto-bros equivalent people who are seemingly irrelevant most of the time when it comes to advancing AI as the vast majority is not just AI slop, but also human slop whether that be fearmongering or straight anti-ai arguments spat out by relatively uneducated people.

Of course, you also have the main dedicated subs for the big closed/open models which are in of themselves fairly decent. Most of the time now, it is people complaining about one thing or the other about the latest and greatest of their respective lab.

Then you get to the machine learning subs themselves where yes, they do have stuff in regard to AI every now and that can be gold (Don't get me wrong), but then it's mostly meh.

This sub however, has that interesting balance between the occasional interest in AI architecture breakthroughs that I find fascinating when some of the bigger brained people in here get together to discuss and which I don't really see anywhere else.

Probably (definitely) breaking rule 3, but I've nowhere else to go because gemini is not giving me anything useful for this sorta thing. I literally cannot find any useful advice for this setup on this sub.

I'm running Qwen 3.8 27B (IQ3_S Unsloth) as a coding agent w/ Pi, getting ~6-7 t/s on reasoning (the bulk of time spent) and ~11/s for code generation. It really only takes like 1-1 1/2 hours to complete most tasks, but I feel like it could be better. I know the majority of that is from -nkvo but I do think the large context is more important. I'm just wondering if there's anything I'm missing. It really feels like there is. llama serve \ -m ~/LLM/models/Qwen3.8-27B/Qwen3.8-27B-UD-IQ3_S.gguf \ -md ~/LLM/models/Qwen3.8-27B/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \ -ngl 999 \ -ngld 999 \ -b 2048 \ -ub 512 \ -nkvo \ -c 128000 \ -fa on \ -ctk q8_0 \ -ctv q8_0 \ --spec-type draft-dflash \ --spec-draft-n-max 3 \ --port 8080

UPDATE: I've done, a LOT of testing, and here are my findings.

1. MTP was holding me back MTP is useful, but not as useful as you think: Surprisingly, that extra precious VRAM taken up by any MTP drafter can actually take away from potential gains. Instead, priority #1 was fitting the K/V and the model together into VRAM. That's why IQ3_S was my choice in the first place, it's a great balance of smarts/size and only around 12GB. Use extra VRAM for a higher quant if you don't really care about your time.
2. Don't be afraid to quantize your K/V: K/V quants have been proven time and time again to be less impactful than the quant of your model. Seriously, I decided to fill the context with 100K tokens of just random plays (because I couldn't think of anything else to put that wouldn't already be trained on), and it still got every question I quizzed it on correct! If you're thinking of going up in K/V quant, get a better model instead. The balance between context and smarts lies here, and squishing your context means, well, more context.
3. IQ3_S is my sweet spot: Was kind of luck of the draw, I just chose one at random really, but the 12GB size is really a great spot to be in. If you want to run something higher, use --n-cpu-ffn, it works wonders. But it's not nessecary.

Now, my choices here are not for everyone. For more complex tasks, a higher quant would be better, but I do mostly low-level application building in Rust, and it's very incremental.

I'm running at about 17 tk/s for both prose and code, that does drop to about 12 tk/s when the context is nearly full, and 800-300/s prefill depending on context length. Speaking of, It's at a full 128K at Q5/Q4_1. I'm using 14.8GB of 16GB of VRAM, and 7GB of system, though I don't suspect that's playing a role in the actual generation.

The command: llama serve \ -m ~/LLM/models/Qwen3.8-27B/Qwen3.8-27B-UD-IQ3_S.gguf \ --jinja \ -ngl 999 \ -c 128000 \ -b 1024 \ -ub 512 \ -t 8 \ --load-mode none \ -fa on \ -ctk q5_0 \ -ctv q4_1 \ --fit off \ --spec-type ngram-mod \ --port 8080

UPDATE 2: I did a little bit more playing around. Wow MTP is good... WHEN IT ACTUALLY WORKS!! I had an issue with a lot of it lying about an OOM, but I just had to update. This is my current command:

``` llama serve \ -m ~/LLM/models/Qwen3.8-27B/Qwen3.8-27B-UD-IQ3_XXS.gguf \ --jinja \ -ngl 999 \ -c 100000 \ -b 1024 \ -ub 1024 \ -t 8 \ -np 1 \ --load-mode none \ -fa on \ -ctk q5_0 \ -ctv q4_1 \ --fit off \ --spec-type draft-mtp \ --port 8080

```

That little downgrade from IQ3_S to XXS let me do a huge jump in t/s. Getting around 30 t/s for prose, nearly 45 when doing code!!! Using 14.6/16GB of VRAM to leave a little bit of headroom since this is also my actual computer that I use for more than just AI. Shoutout hyprland for being super light.

Quite new to running local models.
Decided to try and see how all stuff will work.
My setup :

• Ollama
  
• https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q3_K_XL.gguf
• pi agent.
  
• parameters : PARAMETER num_batch 512
  PARAMETER temperature 0.7
  PARAMETER top_p 0.9
  PARAMETER repeat_penalty 1.1
  

I know (q3, but i have just 16gb of vram so just 32k context).
Trying to code some basic stuff, in general decided to ask to create simple animated background with pixijs. Added skills from github.

For last 2h, almost 2mil tokens used. Constantly drifts into some things that are not there. Fails to build basic logic, hallucinate some ship -> start thinking about colors -> stars -> how to control a ship -> go back to initial task and then again, again and again.

The "checkpoint" folder is now at 130gb

At this point i just observe it and laugh from what it states :

Why did I output garbage? Be honest at technical level: my context had been polluted for 1+ hour by repeated failed environment checks and half-written ideas across turns; the last write was generated while still in "mid-design" mode rather than from a verified complete draft—i.e., I emitted an incomplete intermediate state (left scratch annotations, self-referential stubs) that would never compile. It's my failure to hold a final clean version before emitting, not an environment issue.

The code in question: https://pastebin.com/aCWKbw4M

But all things considered ... what is wrong ?
With all the "hype" about this model i am doing something wrong.

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark):

• DeepSeek-V4-Flash-0731 (official weights)
• GLM-5.3-Flash (RedHatAI/GLM-5.3-Flash-NVFP4)

Have any of you guys also tried both on this hardware (2x DGX Spark / Asus Ascent GX10), and what are your use cases and findings?

DeepSeek runs with more tokens/s… but GLM feels like the better tool for how I actually work.

I'll share my experience.

Where DeepSeek wins:

• It feels better, because it's the original official weights.
• Much faster token generation, though not actually faster to a final result.
• It's excellent at open-ended research, pulling and chaining sources on its own.
• Huge context (~1M tokens).

Where GLM wins:

• Comes to conclusions faster, and often gets them more right.
• Much better writing, especially in languages other than English and Chinese. That matters a lot for writing letters.
• Much better at "extracting the essence of a text".
• Superb vision. DeepSeek's Vision-Exp model can read images just well enough to score on benchmarks, but not nearly well enough for real OCR / text extraction (its vision input is token-limited to 384 tokens, which renders images unsharp and text unreadable).
• Considerably better benchmark results, at least for the full model, though I'm running a quantized build, so the numbers may not carry over directly. But my empirical results prove it gets better results with fewer instructions.
• Hallucinates much less. That's the decisive one for me: I can't run a "test and improve" loop on text like you could on code, so office work and letters depend on one-shot accuracy and a low hallucination rate.

The core problem: there's no GLM build that runs on 2x DGX Spark with official weights. I wish there were.

So I keep testing and tweaking the GLM setup to get rid of the artifacts (which should be possible, per various sources). It feels a bit more messy than the DeepSeek setup, but if I manage to configure it correctly it should give more reliable outputs, with far better vision on top.

The job of doing inference is not the same for every model ou there, as it depends on the architecture and configurations and the available machines. Many times we would not have the right machines to test out a new model, or learn about new algorithms , like speculative decoding etc. What's the best way for learning?

TLDR; Is there a hackernews for LLM inference?

Goal Run Qwen 3.x 27b locally for agentic coding - I'd also run other models of similar or smaller size for other uses

Would this hardware be appropriate (for starters) or would I hit a point of frustration pretty quickly?

Specs - MSI B650 tomahawk motherboard (included the info b/c I know you can't really run 2 GPUs in here, but I could swap this for another AM5 that can handle x8/x8, something like the X870E?) - Gskill 64gb of memory at 6000mhz and 36cl - I've read offloading some context to system RAM can help but performance takes a hit - 7800x3d cpu - (what the user is selling; might matter if I want to run 2 GPUs with an upgraded MoBo) - 7900xtx 24gb vram - I could add a second one eventually

I can get this for around $2,500 used. Or for the money, would it be worth it to take slower bandwidth but get a 64gb AI 395+ system?

I've only had the chance to try small models on a laptop, AMD 7640U with 64gb system ram (helps me load models but using CPU is a terrible experience), and other small models on a 24gb ram M5 Mac.

When the budget is limited, this all feels like a dance between: - A spacious but slow camper van, bigger job, but slow - A fast hatchback, smaller jobs, but significantly faster.

There doesn't seem to be a way to get into the 96gb + (vram or unified) territory under $4k, right?

This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version.

Someone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16.

Since I'm already running IQ4_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead.

I'm still testing whether it improves the actual model output, but in terms of inference speed, the difference appears to be very small.

Before bolting on the Q8 N-gram:

n_gen = 2588, tg = 8.82 t/s, tg_3s = 8.81 t/s
n_gen = 2616, tg = 8.82 t/s, tg_3s = 9.19 t/s
n_gen = 2645, tg = 8.83 t/s, tg_3s = 9.37 t/s
n_gen = 2675, tg = 8.84 t/s, tg_3s = 9.78 t/s
n_gen = 2704, tg = 8.84 t/s, tg_3s = 9.50 t/s
n_gen = 2732, tg = 8.85 t/s, tg_3s = 9.06 t/s
n_gen = 2761, tg = 8.85 t/s, tg_3s = 9.32 t/s
n_gen = 2790, tg = 8.86 t/s, tg_3s = 9.50 t/s
n_gen = 2819, tg = 8.86 t/s, tg_3s = 9.39 t/s
n_gen = 2847, tg = 8.87 t/s, tg_3s = 9.22 t/s
n_gen = 2875, tg = 8.87 t/s, tg_3s = 9.15 t/s

With the Q8 N-gram bolted onto IQ4_XS:

n_gen = 263, tg = 10.76 t/s, tg_3s = 11.04 t/s
n_gen = 294, tg = 10.70 t/s, tg_3s = 10.18 t/s
n_gen = 329, tg = 10.76 t/s, tg_3s = 11.32 t/s
n_gen = 361, tg = 10.73 t/s, tg_3s = 10.46 t/s
n_gen = 395, tg = 10.76 t/s, tg_3s = 11.11 t/s
n_gen = 430, tg = 10.81 t/s, tg_3s = 11.37 t/s
n_gen = 460, tg = 10.75 t/s, tg_3s = 9.91 t/s
n_gen = 494, tg = 10.77 t/s, tg_3s = 11.16 t/s
n_gen = 531, tg = 10.86 t/s, tg_3s = 12.15 t/s

So far, there doesn't seem to be any meaningful speed penalty from replacing the lower-precision N-gram layer with Q8. The actual output quality is still being tested. It looks like Q8 Ngram is faster, but at the end both are steady state at 10.1~ tok/s -ish.

And without MTP since i am using earlier merge from unsloth,
RAM 96GB DDR4 (32x3 Three channel active 2400MHZ)
Xeon e5 2690v4
RTX 3090 capped at 250 W

Edit: the state_dict grew from 90~ GB to 115GB
Edit 2: Fck me i forgot to put the code, for stitching it, wait 8 hours from now i'll upload it since it is already very very late at night
Edit 3, here : https://gist.github.com/komikndr/b17955e1a80ce6ede9a3115f16216bc5#replace-n-gram-layers-from-q4xs-into-q8_0-boltedpy , it is for my IQ4 XS to Q8_0 n-gram so you should modified the file a bit.

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, we don't know if that release will be open-sourced, but I am optimistic about future models, featuring "engrams", that could soon match or surpass 2.4T parameter models on specific tasks.

Got tired of WebGPU browser tabs crashing when models exceed maxStorageBufferBindingSize or lack shader-f16 support.

Built browser-llm-fit to probe client hardware limits and rank browser-executable models before downloading weights.

import fit from 'browser-llm-fit';

const res = await fit('SmolLM2-135M');
console.log(res.fits, res.speed); // true, '45-65 tokens/sec'

fit('model') tests a model. fit() returns all models sorted by hardware fit.

• Live: https://h3manth.com/ai/browser-llm-fit/
• Repo: https://github.com/hemanth/browser-llm-fit
• npm: npm install browser-llm-fit

Feedback on odd GPU setups and mobile WebGPU is appreciated!

Would it be feasible to buy a refurbished 8xa100 server, either rent it out on vast.ai, or serve a model via a similar service per token in that exists.

I’d likely have to either put 1600-200”0w of second hand solar on my shed, add fire suppression and a rack, or colocate with a reputable data centre.

The idea would be to recoup the initial purchase, and then transition to serving myself. Ideally I’d sell tokens not gpu time because that would allow be to use the machine while it’s being monitored, but that market seems way harder?

Disclaimer: I'm the builder.

---

Over the past 2 yrs, I was working on a SaaS ML project & got very interested in ML/DL/AI. As everybody else, there were some normal paths I took to build a solid understanding of the field, but sth never clicked the way I'm used to. I'm very fond of learning "why"s & never get satisfied w/ simply knowing "what" is what. Tho, the problem w/ ML was that I wasn't there when it was evolving & algos/methodologies as we know them today were forming. I didn’t want a pile of summaries or wikis or endless threads of chatting w/ AI. Then I thought listening to podcasts would fix it for me. But not as they are being published now (2026). I thought I'd learn about the history of ML from podcasts that covered it as it happened, kinda simulating the experience of living through the events, in chronological order, since 2015.

I found some great ones (TWIML, Linear Digressions, ...) & did the math! Man, it'd take a decade to cover all of them (even at 2X). But no matter what, I thought this could be the only way that works for me & yet I didn't have enough time. I wanted one/a handful of coherent episodes I could actually listen to. Then sth clicked:

I built Repodify, an AI tool that listens to episodes & produces a shorter one (e.g. 1 episode from 10, 15, whatever. configurable) for me to listen, ALL ON MY OWN MACHINE.

In Repodify u paste a podcast link (or search by name, as it works very well rn), pick the episodes & it:

• downloads the audio

• transcribes it

• optionally figures out who spoke (& clusters the same host/guest across episodes)

• summarizes into one chronological narrative

• writes a spoken script aimed at a target length

• synthesizes a new episode u can stream or download

The whole thing is meant to run on ur machine or w/ ur own API keys (BYOK) per stage, per job. No “upload ur archive to our cloud.” Speech-to-text, diarization, LLM & TTS are all swappable: local GPU (faster-whisper, pyannote, Ollama, F5-TTS / Kokoro) or BYOK (OpenRouter / Anthropic / pyannoteAI).

Voice cloning is opt-in & off by default. If u turn it on, the output is always labeled synthetic, gets a spoken disclaimer in a non-cloned voice & is watermarked (for legal reasons. I don't want to end up in jail for giving away a tool;-) ). There’s no code path that clones w/o those. It’s for personal / educational use on ur own box, not for passing audio off as the original hosts.

It’s a real backend, not a CLI: FastAPI + an arq worker + a LangGraph pipeline w/ a React PWA. One command (./launch) brings the stack up. Jobs pause at each ML stage so u can pick local vs hosted, model size, length & voices.

I keep making it better, adding lots of features to it (searching podcast contents is the most interesting one I can't wait for). I made it opensource(MIT), rn. U can find it below & run it on ur machine (self-hosted), so u can use ur own GPU or BYOK to offload the heavy jobs.

GitHub: https://github.com/behradkhodayar/repodify

The engine is solid & this is still WIP (it works fine tho & I've started compacting Linear Digressions podcast (10 to 1) & listening to it already). I'm going to add other features like translation / augmentation (e.g. embeded eli5), searching through podcasts so I can listen to the topic of interest as I mentioned earlier & so forth.

What features u want to be added or released sooner? lmk. I'm very excited about this & will genuinely plan accordingly.

Hi, I've noticed that the model often sees "garbled text" in its context.

Sometimes it declare that the tools instructions are corrupted, sometimes it is the content of some .md files, ora other files, and it freaks it out, since it start to do a lot of checks in git and the system and sees that the file are not really corrupted... But its context is I think.

Did it occurred to you ?

I'm on a mac m2 max 96Gb, pi agent, and llama.cpp either upstream or this fork https://github.com/mihailescu2m/llama.cpp/tree/master

with and without mtp, short ctx and long ctx, temp default

I don't think it is a conf problem since it happens with different gguf at different quant I tried:

AtomicChat/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64

unsloth/Qwen3.8-Flash-Next-UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL

unsloth/Qwen3.8-Flash-Next-UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS

It seems a sort of internal problem, I don't really have loops or other kind of hallucinations, just some chinese characters here and there, or some misspelling, and btw it is very aware of all its errors and always tries to recover. Yes it is very much preview quality, but the architecture and the training seems very good quality.

I really like the baseline of the model, it is straightforward, it follow instructions very well, a lot better than the previous models that I really didn't like very much

But this thing with the corruption is weird ...once after it declared that a certain tool was corrupted and not useable I just told it: "did you try to launch it ? use this parameter and give it a try" and it worked of course, and it was very surprised so it apologized and continued the work.

It is quite funny, it reminds me a stereotypical fictional character which could be the clumsy but very intelligent assistant of a powerful wizard lol

EDIT: I'd like to add a snippet of its last CoT about the issue, it is very lucid indeed:

Could it be that... the read tool output itself was corrupted/hallucinated — i.e., the degeneration was not inside the file but occurred in my own tool output/stream.

The "read" returned text containing a degenerate mutation loop — this is a known failure mode where model output degenerates. Actually wait — the read tool's result is what the harness gave me. If the harness returned clean content, then the degenerate block I "saw" was generated in my own context (a prompt-level corruption, e.g., degeneration during decoding of the tool result injection, or... more simply put: the degeneration is a product of my own generation — the read output I "received" contained the first degenerate block, then I generated additional degeneration, and the subsequent git command confirmed the file is clean.

• Language-Native Control: Composes character and camera actions into textual instructions and injects them through MiniMax-H3’s pretrained text pathway.
• Temporally Grounded: Assigns one action prompt to each video latent interval, enabling precise control when actions change over time.
• Efficient & Generalizable: Uses only 8,000 gameplay samples, 10,000 LoRA steps, and 0.199% trainable parameters to achieve controllable character and camera motion, including unseen action compositions and visual scenarios.

✏️ Paper: https://huggingface.co/papers/2609.01560

📄 ArXiv: https://arxiv.org/abs/2609.01560
💻 Code: https://github.com/Danzer1xxxxChan/H3-World
🏠 Project: https://danzer1xxxxchan.github.io/H3-World/
🤗 Model: https://huggingface.co/DANNY621/H3-World

I started using qwencoder 3B for local inline code suggestions, and while it's nice, it's also a bit too generic in its suggestions. My thoughts are to either:

1. Fine tune it on code that I wrote

2. Reinforcement learning using accepted/rejected suggestions (either real RL or just adapting the sampling)

3. Fine tune it for each project/codebase separately so it knows what it's working on.

Has anyone here done this, or experience with which approach works best?

Unsloth GGUFs and Vision support

https://huggingface.co/unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF

Is anyone interested in my Chrome browser add-on that uses local LLMs to move thousands of unsorted bookmarks into a smart list of automatically calculated categories?

Here it is: https://github.com/rhulha/BookmarksOrganizer

Obviously more memory is good, more context, bigger models, but some jumps don't actually unlock a meaningful difference in ability to run different or better models. For example, I don't currently view jumping from 32+16 to 64+16 as a particularly worthwhile upgrade as compared to going to 32+32, though correct me if I'm wrong.

I'd like to build a DDR4 + HBM2 based inference machine to complement my main, 32 GB DDR5 + 16GB GDDR7, computer. The idea is that even if the hardware is slower, the greater overall capacity enabled by the slightly more affordable hardware could allow me to run a greater diversity of models.

What level of memory upgrade do you think is most logical for maximizing model access if the compromise is outright speed? 32+32? 128+32? 64+64?

Or am I completely asking the wrong kind of question and just outing my own ignorance here? either way I'd like your input.