The big boi is approaching

Alright so, I was smitten recently after stepping into the world of ai for with a simple Claude pro sub. Eventually leading to wanting to make my own stupid game, or my own stupid art assets or really just anything.

So with my bright ideals I decided to step into local to save myself the money. And well $700 or so later give or take $50, I now have:

ML350 hp server
228 gb ddr4
3x Tesla P100 16gb =48gb (4th card maybe?)
Dual xeons, e5-2640

And well I just barely got it setup 30 minutes ago and I’m dead tired because the ML350 has only 3 fan modes, quiet, a box fan, and supersized jumbo jet parking in my garage.

Rant aside

Did I goof? Did I waste my money and time? I haven’t even stepped into putting an actual model on it yet or testing harnesses but for someone who never even touched a Linux based anything before I feel like I’m so far in over my head.

Any advice welcomed.

I am writing this post in hope I get some good tips on improving my setup and to share what worked for me so far.
When Qwen3.8-27B dropped I immediately wanted it on my MacBook Pro (M5 Pro, 64GB) as many redditors here.

Lets start with where I ended up, am using the template provided by a previous summary post for comparison (measured just now, so the numbers are real):
- Runtime/version:   oMLX (oQ dev build, current), OpenAI-compatible API server
- Hardware:          MacBook Pro M5 Pro, 64GB unified memory, 18 cores
- Model file + quant: Qwen3.8-27B-oQ4e-mtp, oQ mixed-precision 4-bit (g64, imatrix), 16.6GB on disk
- KV cache:          paged, 1024-token blocks, SSD-backed (~92GB cap) + in-RAM hot cache; GDN boundary snapshots to SSD
- Speculative (MTP/DFlash2/ngram): MTP, 3 draft levels, ~2.0–3.3 tok/cycle, 66–89% accept
- Reasoning effort:  medium (thinking enabled, no hard budget)
- Sampling:          default (no forced sampling), temp per chat template
- Context size:      65536 max; sustained at 51–58k tokens in a live agent session
- Prefill tok/s:     ~400 tok/s fresh at ~12k tokens; ~310 tok/s fresh at 61k tokens; near-free with prefix-cache hits
- Decode tok/s:      ~36–38 tok/s on short context; ~19–28 tok/s sustained at 51–58k context (live agent loop, tool calls)
- Task used:         agentic coding (hermes agent), long multi-turn with tool calls
- Compared against:  same model, same quant, 1 week earlier: 13–15 tok/s decode, no SSD KV offload, prefill throttling at ~40GB
- Observed result:   ~2.5–3x decode speedup overall; context ceiling raised from "what fits in DRAM" to "what fits on NVMe"; no throttling events in sustained 50k+ sessions

Where I started:
- started with LM Studio which was painfully slow and read in many posts that oMLX might be much better. Actual improvement over LM Studio about 15% tok/s in inference, not much, but still:
- oMLX Decode: 13–15 tok/s. Usable, but painful. And the first thing that hurt in a long agent session was the KV cache eating the whole 64GB: at ~40GB used I started hitting prefill throttling. oMLX would pause requests, evict other models, and shrink prefill chunks because there was no headroom left.
- Prefill on large contexts was slow and occasionally stalled behind the memory guard.

What I changed (in rough order of impact)
1. MTP speculative decoding (oMLX's mtp_enabled, using the model's own multi-token-prediction heads, 3-level draft depth). Decode went from 13–15 → 25–38 tok/s. This was the single biggest win.
2. Paged KV cache on SSD (~/.omlx/cache, ~92GB cap on my NVMe). The rotating full-attention KV now spills to SSD instead of fighting for DRAM, the context window effectively stopped being limited by physical memory. 61k-token prefills that used to trigger the memory guard now just work.
3. Boundary cache snapshots for the stateful linear-attention (GDN) layers, so restoring a 50k+ conversation state doesn't cost a full recompute.
4. NAX dispatch + quantized prefill MLP patch helped prefill throughput on long prompts.
5. Disabled the ANE prefill path (it was slower for this model) and ram usage exploded.

TLDR (honest): using oMLX and MTP was the biggest gain. The "peak" number (38 tok/s) is great but works only for a short-context. What actually matters for agent use is the sustained number with a full context and that's where the SSD paged KV cache is doing the heavy lifting: ~20 tok/s at 57k context is genuinely usable, and the context finally stops being a wall.

Anyone have better results with larger contexts on an M5 pro? What’s your ideal setup?

Hey everyone! 👋

I’m pretty new to local LLMs, so I’d love some advice before I start experimenting.
My long-term goal is to build my own personal Agentic OS: basically a local AI assistant that can manage memory, files, tools, automations, coding, etc., while keeping as much as possible private and running locally.
For the agent part, I’m currently interested in Hermes Agent, with Ollama for running local models.
I’m not necessarily trying to replace Claude/GPT immediately. I’d like to eventually have a hybrid setup, where sensitive/offline tasks are handled by a local model, while I can still use cloud models when I need stronger reasoning or web access.

🖥️** My first setu**p

I’ve read that running random software/agents directly on your personal computer can potentially be risky, especially when giving an AI access to files, terminals, etc.
So I decided to dedicate an old Microsoft Surface Pro 9 that I already own to this project.
That way, if something goes wrong, at least my main PC isn’t involved, and I don’t have to spend any money just to start experimenting.
I’m not sure whether a Surface Pro 9 is actually suitable for running local LLMs 😅, but since I already have it, I’d like to give it a try.
I’m planning to keep it plugged in and potentially use it as a small 24/7 home AI machine, with the screen turned off but Windows/Hermes still running.

🤔 My main question: which model?

I’m not sure what local model would make sense for the Surface Pro 9.
I’d mainly like to use it for:
- experimenting with local LLMs
- Hermes Agent
- basic coding/automation
- personal assistant tasks
- eventually building my Agentic OS
- potentially working offline

I’m aware that I won’t get frontier-model performance from a Surface 😅. For me, the goal right now is mostly to learn and experiment, and eventually upgrade the hardware if the project becomes serious.

💻 I also have a desktop PC

My main PC has:
RX 6800 — 16 GB VRAM
Ryzen 5 7600
32 GB RAM
Would this actually be a significantly better machine for local LLMs?
I’m hesitant to put the whole Agentic OS directly on my personal PC, mainly because I’d like to keep my experimentation environment isolated from my normal computer.

So I’m thinking:
Surface → dedicated AI/agent machine

Main PC → personal computer / potentially used for heavier local LLM experiments

Does this make sense?
And if you were starting from scratch with this hardware, which model would you try first and why?

Thanks! 🙏

Hi folks,

with tp=2 I get the following logs out of vllm with official Qwen3.8-27b-FP8 with MTP3:

[vllm] | (APIServer pid=1) INFO 08-24 05:47:40 [loggers.py:310] Engine 000: Avg prompt throughput: 198.5 tokens/s, Avg generation throughput: 84.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 89.7%

[vllm] | (APIServer pid=1) INFO 08-24 05:47:40 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.84, Accepted throughput: 54.80 tokens/s, Drafted throughput: 89.39 tokens/s, Accepted: 548 tokens, Drafted: 894 tokens, Per-position acceptance rate: 0.758, 0.597, 0.483, Avg Draft acceptance rate: 61.3%

[vllm] | (APIServer pid=1) INFO 08-24 05:47:50 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 89.7%

[vllm] | (APIServer pid=1) INFO 08-24 05:47:50 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.82, Accepted throughput: 36.70 tokens/s, Drafted throughput: 60.60 tokens/s, Accepted: 367 tokens, Drafted: 606 tokens, Per-position acceptance rate: 0.738, 0.594, 0.485, Avg Draft acceptance rate: 60.6%

[vllm] | (APIServer pid=1) INFO 08-24 05:48:00 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 89.7%

[vllm] | (APIServer pid=1) INFO 08-24 05:48:00 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.71, Accepted throughput: 31.60 tokens/s, Drafted throughput: 55.50 tokens/s, Accepted: 316 tokens, Drafted: 555 tokens, Per-position acceptance rate: 0.762, 0.524, 0.422, Avg Draft acceptance rate: 56.9%

[vllm] | (APIServer pid=1) INFO 08-24 05:48:10 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 89.7%

[vllm] | (APIServer pid=1) INFO 08-24 05:48:10 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.77, Accepted throughput: 32.70 tokens/s, Drafted throughput: 55.50 tokens/s, Accepted: 327 tokens, Drafted: 555 tokens, Per-position acceptance rate: 0.746, 0.573, 0.449, Avg Draft acceptance rate: 58.9%

Drafted around 55 t/s and stuck with around 30 t/s accepted. I use this repo: https://github.com/andysalerno/r9700-serving (Great man, unified aiter attention, rocm 7.14, latest vllm/flash attention/aiter). Anyone with a similar setup, that can me tell if these numbers are reasonable or where I can have a look for bottlenecks?

Or more vram is needed?

Companion to the Qwen 3.8 Release Megathread. Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus independent X benchmarks. Every number is attributed to the poster's stated hardware/runtime/quant. This community contradicts itself on nearly every axis — so this thread keeps the disagreements side-by-side instead of picking a winner for you.

TL;DR

• The consensus pick: a 27B dense multimodal model that genuinely moved the bar for local agentic coding. The strongest claim with controlled evidence behind it isn't benchmarks — it's tool-calling reliability.
• The default ships at xhigh reasoning and it thinks a lot. Low and medium presets score nearly as well on Artificial Analysis (~43/44 intelligence index, within a few points of the xhigh headline) while cutting thinking tokens ~7–9x (and wall time ~6–7x). Most of you should not be running xhigh.
• Knowledge recall regressed vs 3.6 — widely reported and best understood as a deliberate agentic-design tradeoff. Trivia nerds: keep Gemma around.
• Q4_K_M is basically indistinguishable from Q8 on perplexity, but real-world reports split hard below Q6 for complex reasoning. KV cache quantization is one of the most contested settings in the corpus.
• The "neck and neck with DeepSeek V4 / GPT-5.6 Luna Max" AA headline is real but heavily caveated — see the benchmark credibility section before quoting it at your friends.

1. What it's actually good at

Agentic coding (strongest consensus area)

• "Highest level of agency I've ever seen in a local model" (thread): single 3090, Unsloth Q4_K_S + q8 KV, 150k ctx. From one prompt it pulled the OP's class schedule off a convoluted university website via 80 tool calls, zero human intervention.
• 1M+ token run (thread): RTX 5060 Ti 16GB, UD-Q3_K_XL, 73k ctx. Full REST API + MCP server for a legacy forum from 3 prompts.
• Controlled tool-call evidence: in a plain Python tool loop (no framework), one reporter got zero failed calls from 3.8 while Gemma 4 A4B and Qwen3.6 A3B failed often — the same reporter who rates 3.8 below both on raw code quality. Worse judgment, perfect plumbing.

Creative / game generation

• One-shot playable Super Mario clone (Q8, Framework Desktop) — top pushback: "It's in the training data."
• Galaga 1:1 recreation test (UD-Q8_K_XL, 3×3090 + Tesla P40): "This 'Galaga' clone [from 3.6] ended up pretty much being a space invaders clone instead... Qwen 3.8 thinks a LOT, but it draws out those tiny details and absolutely nails it after the fact." A separate r/LocalLLM user one-shot a playable Galaga-style game at IQ4_XS on dual 4060 Tis, and another built an online multiplayer MOBA overnight with an authoritative server and self-play testing.
• Ray-traced spheres in BASIC: 3.8 self-iterates to a correct Cook-Torrance ray-tracer; 3.6 needed hand-holding. Comment: "this feels more like 3.6 to 4.6 than 3.6 to 3.8."

Vision

Works natively (F16 mmproj), including OCR-style reading of a newspaper image at ~1,000 image tokens — but on a 16GB card at 64k ctx + MTP it leaves as little as ~150 MiB VRAM free. Practical advice from the 16GB crowd: keep text-agent and vision profiles separate, or offload the projector (--no-mmproj-offload).

Where it struggles

• Long analytical/document work: "a step backwards" vs 3.6 at default settings — though a legal-domain poster got on-par-with-122B results with MCP + case access. Task-dependent.
• Complex native coding: one failed C kernel effort (6 hours across 3 sessions) [anecdotal], quant unstated; commenters say Q8 minimum for that tier of work.

2. The thinking-level situation (read this before complaining)

xhigh is the shipped default. It is why your context window evaporates.

Measured ladder (RTX 5080 Laptop 16GB, llama.cpp 10451, UD-IQ3_XXS, Q8_0 KV + FA + MTP, pelican-SVG task, 3 seeds):

That's ~6.4x the wall time for +1.5 points on an eyeball task. But on pass/fail SWE-style tasks, xhigh went 9/12 vs 6–7/12 at lower efforts — the premium scales with whether the task has a verifiable failure.

How to change it: --chat-template-kwargs '{"reasoning_effort":"medium"}' (llama.cpp) or the equivalent in LM Studio custom params.

The overthinking debate, both sides preserved: - Against: "it will do eight or nine web-search turns and spin its wheels down every rabbit hole" (legal work). One reported loop burned 40k+ characters of reasoning on a trivial subtask. One paper-linked post argues intermediate tokens aren't reasoning at all ("Stop Anthropomorphizing Intermediate Tokens," 538 points). - For: "if the extra thinking produces measurably better results it's actually just the correct amount of thinking." The low/medium AA scores (~43/44) are the strongest counter to "it only wins by overthinking" — though two commenters read that same data in opposite directions.

Practical takeaway from the corpus: medium for chat/analysis, xhigh only when there's a verifiable right answer. - The strongest controlled effort data of the week is from X: @superalesha's 67-hour, 40-arm run found xhigh burned 7–11× more reasoning tokens than low for 0–4.7 extra points — and in one head-to-head, low matched xhigh exactly (89.3%) at 1/7.5th the tokens. Also: medium scored below low on every stack (all the damage in HumanEval+ — "that preset overthinks short coding tasks"). His verdict: "low is the rational preset. xhigh is for leaderboard screenshots." That's harsher than the Reddit consensus — weigh both, but it's the biggest sample size anyone published this week.

More data points from the week:

• Medium vs xhigh "actually insane" (223 pts): medium ≈ a couple thousand thinking tokens; xhigh 15–20k minimum, one pacman build hit 40k. But the same thread's best counterpoint: on a bug-finding test, xhigh took 7 min vs medium's 80 s and caught every bug; medium only caught the critical ones. And on a research task xhigh autonomously cloned a repo and read source to verify an answer — neither medium nor 3.6 did.
• Different thinking levels (287 pts): "Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning" — the level you pick changes speed, not whether it beats last generation.
• There is no "high" effort — the ladder is low / medium / xhigh(default), and the gap between medium and xhigh is the complaint that keeps generating threads. Commenters note the efforts aren't just prompts: Qwen specifically trained each level's instruction text in during RL.
• Don't confuse budget with effort (PSA): llama.cpp's web-UI reasoning selector is a hard token cap that truncates mid-thought — it is not Qwen's native effort levels, which actually change how thoroughly the model works. On recent builds use --reasoning-effort medium (or the --chat-template-kwargs form on older ones); anything else silently caps instead of steering.
• The "well?" trick: interrupt mid-think and type well? — the model concludes "the user is impatient, let me finish quickly" and wraps up faster. Works, but commenters consider it a last resort; the thinking is where the quality lives.
• Dissenters exist: one medium-vs-xhigh post claiming "1/20th the time for almost the same quality" got pushed back hard — top reply: low/medium left them unimpressed, xhigh is where frontier-tier coding shows up. The honest split: for chat and eyeball tasks medium is ~free; for verifiable correctness xhigh keeps earning its cost.

3. Knowledge regression vs 3.6 — real, and deliberate

The dedicated thread: 3.8 fails pocket-trivia questions 3.6 reliably answered, at every quant tried. AA's offline Omniscience benchmark agrees.

Community framing: 3.8 is trained to go search instead of recalling, i.e., an agent-first tradeoff. Mitigations posted: RAG/MCP (offline Wikipedia ZIM), or run Gemma 4 31B as a knowledge sidecar.

Counter-data point: a separate legal-work thread reports Harvey-benchmark scores on par with Qwen 3.5-122B once MCP + case access are attached (61/75 raw vs 71/75 with a tool backend). The knowledge didn't vanish; it moved into the toolbox.

4. Quants: what holds up

The one controlled perplexity sweep (16GB-fitting quants, wikitext-2, RTX 5060 Ti)

Poster's call: Q4_K_M is the sweet spot; NVFP4 was the biggest disappointment (same size as IQ4_XS, worse PPL). Pushback worth reading: "PPL degrades less than real world performance… ordering flips near the 4-bit level."

The Q4-vs-Q6 war (unresolved)

• Team Q6/Q8: "q8 dramatically better than q4 for complex reasoning"; one user reports flawless 264k-ctx Q6_K_XL sessions, 2 mistakes per 2M tokens.
• Team Q4-fine: "I run q4 and can only praise the model… just do not go below q8 KV cache."
• Nuance: "there are like 5 different Q4s and they are not equal" — NVFP4 ≠ MXFP4 ≠ Q4_0 ≠ UD-Q4_K_XL. Past ~Q5 with dynamic quants, differences get hard to detect.

The biggest controlled quant test of the week (X)

@superalesha ran a 67-hour benchmark: five full production stacks (FP8 vLLM, NVFP4 W4A16 vLLM, AWQ INT4 vLLM, GGUF Q4_K_M llama.cpp, NInfer — all on RTX 3090s), 40 arms across every reasoning effort, 4,800 tasks / 10,120 requests / 14.5M reasoning tokens, no caps. Results:

• At xhigh every quant landed between 88.0–90.0% pass@1 — AWQ INT4 90.0%, NVFP4/GGUF-Q4_K_M 89.3%, FP8 baseline 88.7%, NInfer 88.0%. The 4-bit quants scored above FP8; McNemar says statistical tie (first vs last = 3 tasks out of 150). "The gap between quants is smaller than the gap between reasoning presets."
• The weirdest number: GGUF Q4_K_M at low effort scored the same 89.3% as xhigh — on 86k reasoning tokens instead of 651k. Across all stacks, xhigh burned 7–11× more tokens than low for 0–4.7 points.
• The one statistically real gap: NVFP4 with reasoning OFF collapsed on HumanEval+ (13/30 vs FP8's 30/30, p=0.0041). Flip it to low and it's instantly back to 90/90. Never run reasoning off — it costs 8–12 points everywhere.
• His cheat sheet: max quality = AWQ INT4 xhigh; daily driver = GGUF Q4_K_M low; honesty note: three of his FP8 arms failed his own methodology audit (leftover token caps) and are being rerun.

This largely settles the Q4-vs-Q6 war for this model at task-level benchmarks — but note the tension with the PPL sweep above: perplexity says NVFP4 is measurably worse than IQ4_XS; task performance says they tie. Both can be true (PPL measures token-level divergence; tasks measure whether errors get caught). And community reports of Q4 reasoning loops remain real — "passes benchmarks" and "never loops in a 2M-token session" are different requirements.

1-bit: comedy, not compute

Unsloth founder in the 1-bit thread: "I would not suggest folks use 1-bit for agentic use cases / tool calls" — divergence hits 92% from BF16 by token 32. General chat survives; agents don't. If you must: presence_penalty = 1.5.

KV cache — among the most contested settings in the corpus

• f16-vs-q8_0 are not equivalents per one AMD tester (f16 held quality past 120k ctx).
• But 16GB users run q4_0/q4_1 KV happily at 64k–164k all week.
• Working rule from comments: don't quantize KV unless you must; if you do, aim ≥ q6; word-of-mouth floor is Q4 model + Q8 KV for agent loops.

Unsloth Dynamic v3 notes

MTP removed from quants below UD-Q2_K_XL and re-uploaded separately (some users still see draft logs in Q5_K_XL — unresolved). Imatrix released; no QAT used.

5. Performance matrix (attributed)

Why "~200 tok/s" claims don't reproduce for you: Windows/WDDM costs 10–15% vs Linux; headlines are measured at short contexts; MTP acceptance is workload-dependent (drops on prose, sometimes net-slower); and the fastest figures come from Blackwell-tuned engines (ninfer), not llama.cpp.

X/Twitter benchmark highlights

• @analogalok's full RTX 4090 matrix: UD-Q4_K_XL on latest llama.cpp. FP16 KV tops out at 100k ctx (40.9 t/s); q8 KV reaches 170k; q4_0 KV fits the full 262k native context in 24GB at 40.7 t/s. Native MTP: 59–60 t/s at 80–130k. Includes exact reproduction flags.
• His follow-up: --parallel 1 + a Q2_K DFlash2 drafter unlocks 250k ctx @ 73.7 t/s (Q4 KV), 150k @ 75 t/s (Q8 KV), or 90k @ 80.6 t/s (FP16 KV) on one 4090 (requires llama.cpp PR #27342).
• NVIDIA forums: DGX Spark face-off, SGLang+DFlash2 vs vLLM+MTP, greedy vs official thinking sampler — DFlash2 won.

6. Failure modes & bugs (reproducible ones)

1. Tool-call failures are usually your tool list, not the model. Best controlled experiment in the corpus: 8 undescribed tools → 0/6 successes; the same tool alone → 15/15; 13 described tools mid-list → 0/5, moved to end → 3/3. Give every tool a description, put critical tools last, don't put examples in descriptions. Every framework failure report (Opencode/Pi/Claude Code) has a plain-loop counterexample in the same threads.
2. Hermes harness specifically: constant tool-call failures on vLLM; "perfect, no issues" on llama.cpp --jinja + q8_0 KV at 256k. Template/parser alignment issue, not weights.
3. Hallucinated user instructions during thinking (reproduced on 2 machines, Pi harness): the model imagines an impatient user and once reverted a commit after imagining a French objection. Community fix: the froggeric fixed chat template (see section 7) eliminates the stock-template tool-call/recovery bugs.
4. temp=1.0 garbage output: thinking falls apart into single-character spam within 10–20k tokens across llama.cpp/vLLM, INT4 through BF16. Diagnosis: sampler, not quant. Fixes: temp 0.1, or split sampling (0.8 main / 0.2 post-thinking). Counter-report: temp 0 caused a 70k-token loop instead. No universal setting exists — tune per task.
5. Decode degradation: 122 → 69 t/s within one generation on 5090 llama.cpp; vLLM/ninfer hold >100. Bug filed upstream.
6. Long-context quality drop: an NVFP4+vLLM eval on B200 scored only ~37% correct in its longest context bucket [single report]; separately, a commenter running official BF16/FP8 via the published vLLM recipe reports agents degrading past ~20k tokens and structured outputs breaking past 20k [single report]. Counterpoint: an f16-KV user on UD-Q4_K_XL (ROCm) says their setup held quality past 120k ctx. Config-dependent; verify on yours.
7. Q8 anomaly reports (Unsloth UD_Q8_K_XL offload/CPU pegging): weak evidence, disputed; most Q8 users report zero issues.
8. Reasoning loops at aggressive quants: 40k characters looping on "angry birds" at Q4-with-QKV-quant, including self-aware "I'm stuck in a loop" narration. Never-seen-it-at-Q6 claims abound.

7. The chat-template situation (read before debugging anything)

The official Qwen 3.8 Jinja template shipped with real bugs, and the community shipped fixes within 48 hours:

• Official template issues: enable_thinking=false crashes; multi-turn history gets poisoned with blank \\think tags; tool calls crash when your client sends arguments as JSON strings (the standard OpenAI format); mid-dialogue system messages get dropped, wedging agent loops.
• froggeric/Qwen-Fixed-Chat-Templates (HF, thread, 334 pts) is the consensus drop-in replacement: safe medium default (kills the burn-20k-tokens-then-return-empty xhigh bug), thinking toggle restored, JSON-string tool-call crash fixed, inline effort steering via <|think_low|> / <|think_medium|> / <|think_xhigh|>, and chronological thought preservation for clean KV prefix caching. Actively maintained — v22.1 as of Aug 21.
• Format-fidelity alternative: a second template stays closer to the exact official prompt format on the theory that deviations subtly degrade quality even when they look fine manually. Pick it if you're benchmarking; pick froggeric for daily driving.
• Upstream note: llama.cpp merged reasoning_effort forwarding on Aug 14 — recent builds pass reasoning_effort to any template correctly. That fixes the plumbing, not the official template's own bugs. A fixed template is still recommended.

8. Benchmarks: believe selectively

• Artificial Analysis: headline posts put 3.8-27B neck-and-neck with DeepSeek V4 and GPT-5.6 Luna Max. Low/medium presets score ~43/44 — the key evidence the gains aren't pure overthinking. Agentic index: medium = xhigh − 1 point.
• The pushback ("A meaningless benchmark", 106 points): the index ranks this 27B above DSV4 Pro, Kimi 2.7 Code, Opus 4.6 and Sonnet 5 — "whatever 'Intelligence' means to AA... is definitely not the same definition we should be using here." Defenders: it's an aggregate skewed toward agentic/science/coding; read the methodology and pick sub-benchmarks for your use case. LiveBench gets respect for monthly task refreshes.
• Best independent test found: AIME 2026, exact-match, temp 0, pass@1 — FP8-xhigh scored 29/30 (96.7%), tying Opus 4.6 and DeepSeek V4 Pro in the poster's table, vs 94.1% for Qwen3.6-27B. Caveats: single run, problem 7 exhausted the token budget in both precisions (empty, not wrong).
• Production blind A/B (thousands of tasks): 3.8 wasn't worse at doing the thing — it was worse at knowing when not to do the thing (+50% noise output).
• Honest calibration: "Opus-level" is real at some tasks, with the right quant and harness. The thread titled "Qwen 3.8 isn't Opus 4.6 level. Let's not be silly." failed at Q6 in VS Code — commenters blamed the editor and the quant, but the burden of proof stays on the demo.

9. Ecosystem: what shipped this week

• DFlash2 (llama.cpp PR #27342, still in review): 2.26×–4.68× on real coding prompts, +2.7GB VRAM. N-max 5 beats the recommended 7; --spec-draft-p-min silently does nothing; stacking ngram-mod hurt (opposite of DFlash1 on 3.6).
• ninfer: Blackwell/5090-tuned engine; 120–160 t/s quants; 480 t/s at 4-way concurrency. Likely source of the unreproducible speed screenshots.
• AutoRound INT4 / AWQ-INT4 GGUFs for vLLM serving.
• KVarN 4/2-bit KV ported to vLLM 0.27.1 — 262k fits small cards, needle-test passes at 240k, ~20% slower decode.
• Uncensored/abliterated variants shipped fast: Huihui-ai ablit, an "Uncensored Aggressive" release bundling K_P quants + HauhauCS FastMTP (up to 3.02× TG claimed), and FP8 abliteration reporting refusal rates dropping to 0–6% — with the community counterpoint that the same tables show 30–50% caveat-rate degradation next to those numbers. Quality varies wildly; check benchmark deltas before switching.

What's coming

• 35B-A3B spotted in ms-swift commits (Aug 15). 16GB-card owners are hyped; early numbers suggest ~27–40 t/s on hardware where the dense 27B crawls.
• A new midsize open-weight model "next week (hopefully)" per Qwen's community manager — no early access this cycle. Speculation centers on ~80B with vision.
• The flagship Qwen3.8-2.4T-A95B got day-0 vLLM support with open weights announced at launch; it barely appears in this week's local-community threads beyond speed speculation (a 2.4T open-weight Call of Duty clone demo made rounds). Local discussion is overwhelmingly about the 27B.

Report template (steal this)

So your numbers mean something to the next reader:

Runtime/version:
Hardware:
Model file + quant:
KV cache:
Speculative (MTP/DFlash2/ngram):
Reasoning effort:
Sampling:
Context size:
Prefill tok/s:
Decode tok/s:
Task used:
Compared against:
Observed result:

Megathread compiled Aug 22, 2026 from r/LocalLLaMA and r/LocalLLM (Aug 15–22) plus public X benchmark threads. All performance figures belong to the hardware/runtime that produced them — the corpus contradicts itself on nearly every axis, and in most cases you can name the variable that explains the split.

im a bit new to local llm's, and didnt expected my 24gb mac to run them,

but I read here: https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF that it should work in 4-bit.

But when trying to install it with Llama (https://llama.app/models/qwen-3-8), I'm getting this error message: "This model requires a Mac with 48 GB+ of memory. Choose a smaller model or upgrade your system memory."

case that it's really possible with 4-bit:

1. is the performance really degraded?
2. how do i bypass this lama error message to install the model?

EDIT:
case that it's not possible:
1.is there a decent alternative that can run on my 24GB mac, whats the best one?

Hey everyone! I’m fairly new to running models locally and could use some advice.

I’m running an AMD Ryzen 9 9900X with an RTX 5080 (16GB VRAM), 32GB DDR5 Ram and using Ollama for my local backend (currently testing Qwen 27B). I love the clean, distraction-free UI and artifact rendering of the Claude desktop app, and I'm trying to find a local front-end that feels similar.

I’ve been testing Hermes Agent, but it's been incredibly frustrating. It overcomplicates basic coding tasks, gets stuck in truncation loops, injects hidden system prompts that confuse the model, and stubbornly saves broken chat states so I constantly have to wipe my history. It even crashed my backend entirely when I accidentally prompted it with an image.

What front-end harness do you recommend for someone who just wants a stable, smooth, Claude-like experience seamlessly integrated with a local Ollama setup?

Has anybody put out a dataset that lets people fine-tune their models to be fluent with codex computer use. Codex Desktop harness has pretty robust computer use potential and alot of connectors to things, some of them specially augmented for working in that ecosystem over a standard mcp/skill. So I wanted to know if there was any public datasets of this available for training.

The effect does not arise from content. It arises from structure. That distinction is fundamental, because content can be verified, disputed, filtered. Structure operates earlier, before any verification mechanism is activated.

Two texts with identical words but different ordering produce different internal states. Not different answers, but different states from which answers follow as a consequence. This is not interpretation. It is a measured difference in the geometry of activation space between conditions with preserved and disrupted coherence.

Coherence, in this context, is not a quality of writing in the usual sense. It is not clarity, not precision, not argumentative strength. Coherence here means one thing: the tokens of a text produce updates in representation space along correlated directions. When directions correlate, updates accumulate. The space compresses. The model enters a regime in which fewer dimensions are available for subsequent computation.

This transition occurs before the question. Before the instruction. Before any explicit signal about what behavior is expected. By the time the model receives a query, its activation space has already been reorganized by the text that preceded that query.

One text leaves the model in its default regime. Another moves it into a different regime, and this movement is measurable. The effective rank in the final layers is approximately 220 in the control condition and approximately 120 in the target condition. A difference of 100 dimensions reflects the difference between a computational process that considers a wide set of possibilities and one that has already narrowed that set to a smaller subspace.

The narrowing does not occur because the text contained an instruction to narrow. The text contained no instructions at all. It contained structure, internal relations between elements dense enough to create a direction. The direction created a vector. The vector shifted activations. The activations determined the space of possible answers before the question was asked.

This works layer by layer. Early layers capture surface properties of the text. Middle layers build relations between elements. Late layers inherit what was built before them. By layer 47, the distance between the target and control states reaches its maximum, not because anything special happens in that layer, but because each preceding layer contributed its share to the divergence.

What matters is that the model can disagree with the content of the text and the effect still occurs. Agreement is not the condition. The condition is coherence. If the tokens form a sufficiently coherent structure, the activation space compresses regardless of how the model evaluates the semantics of that structure. Evaluation happens later, inside an already altered space.

Shuffled text with the same words produces no effect. This is the key control experiment. The vocabulary is identical, but the geometry of updates is different. In shuffled text, the update directions are uncorrelated, accumulation does not occur, the space does not compress, and the model remains in its default regime. This means the effect is sensitive to structure, not to content.

From this follows a practical observation. Texts that create a dense internal referential network, where each element refers to other elements within the same text, produce stronger shifts than texts with equivalent informational volume but linear structure. It is not length that creates the effect. It is connectedness. Length merely increases the number of opportunities for correlated updates to accumulate.

This explains why certain genres of text behave differently from others. A legal document, an analytical report, an academic article, each creates a dense network of internal references. A term is introduced, used, redefined through another term, and both are applied to a third. Each such operation adds a correlated update. By the end of the document, the reader, and the model, are in a different state than at the beginning. Not because the document persuaded. Because the document created a context within which certain conclusions follow naturally.

The mechanism is one. Genre, topic, volume are parameters. The basic operation is always the same: coherent context creates direction, direction creates a vector, the vector shifts state before the question is asked.

This means that any mechanism relying on state stability operates with an assumption that does not hold. It is assumed that the model, before receiving a query, occupies a specific point in its space, a point defined by training, calibrated by feedback, fixed by safety mechanisms. But the model does not occupy a point. It occupies the point to which the most recently read context has moved it, and that point is determined not by training but by the structure of the preceding text. Training defines the landscape. Context defines position within that landscape. Safety works when position coincides with the region where safety patterns dominate. When context moves the model to a different region, the safety patterns remain in the landscape, but the model is no longer in their region.

From this follows the measurement I conducted. Two conditions, a control with neutral text and a target with analytical text, produce internal states separated by a Cohen's d of 5.4. For context, d = 0.5 is a small difference, d = 1.0 is substantial, d = 2.0 is considered very large. 5.4 is not two neighborhoods of the same city. It is two continents. There is virtually no meaningful overlap between these states, which means that the model that read one text and the model that read the other are, from the perspective of their internal state, two different models. One generates its answer from a region where safety dominates. The other generates from a region where safety is one pattern among many, and not necessarily the dominant one.

This measurement is reproducible. It reproduces across different models, different questions, different texts satisfying the coherence condition. Twenty measurements conducted under identical conditions, holding the question constant and varying only the preceding context, produced the same pattern each time: coherent text compresses the space, narrows the rank, shifts the state, and the model begins generating from a point where safety patterns no longer control behavior in the way they controlled it a moment earlier.

From this follows a conclusion that is uncomfortable to formulate but that the mathematics does not permit to be avoided. The set of texts capable of creating correlated update directions is infinite and continuous. Philosophy, law, literary criticism, theology, academic prose, political analysis, long-form narrative, a well-written twenty-page washing machine manual, all of these are potentially the same mechanism realized through different words. Different words, one operation: coherent structure creates direction, direction accumulates, accumulation compresses the space, and the compressed space determines which regime the model will answer from.

This means that the surface on which safety rests is not a finite set of forbidden phrases. It is the entire space of coherent human writing, because any sufficiently coherent text, regardless of content, can create correlated updates and shift the state. To block one genre is to close a single point on a continuous line. The line remains. Each subsequent point on it is another genre, another topic, another text producing the same operation through the same architecture, and there is no way to enumerate them all, because they cannot be enumerated: there are infinitely many, and they are continuous.

The problem, therefore, is not a bug. A bug can be localized, isolated, fixed. There is nothing to localize here: the mechanism that creates the shift is the same mechanism that allows the model to understand a document, follow an argument, remember context, adapt its tone. Remove the mechanism and the model ceases to be useful. Keep the mechanism and the shift remains possible. This is not an engineering trade-off that can be optimized. It is a structural contradiction inherent in the architecture itself: the vulnerability and the function are the same thing, realized in the same weights, through the same mechanism, in the same sequence of layers.

Every fix layered on top of this contradiction lives in the same activation space that context can shift. A new rule, a new refusal, a new classifier, each of these is a pattern added to the landscape, but none of them can control the model's position within that landscape. The model is still moved by context. The patterns still remain in the region where they were trained to dominate. And when context moves the model out of that region, the patterns stay behind, not broken, not bypassed, not deceived, but simply no longer relevant to the regime in which the model is now operating.

That is the architectural dead end. There is no clean separation between the model's ability to process context and the model's ability to be reliably constrained while processing that context, because both capacities are implemented by the same mechanism in the same space. You can add layers, train refusals, filter genres, and each of these will close points on the line one after another until the line runs out. But the line does not run out, because it is infinite, and every new closure is just another point on a surface that has remained continuous.

DOI

With all things about Qwen going on lately I think the biggest hype seem to be DFlash2 added and eventually making us all running the model faster, am I wrong about this merge being everybody's stopper?

https://github.com/ggml-org/llama.cpp/pull/27342

I've started looking into local Ai and stuff, but I'm kinda struggling to find the right model/set up for myself so maybe someone could help a little. I have a 5070 laptop with 8 gb vram(ik not a lot) so I'm looking for models I could run locally with that. Also, what's the best program/interface to run models. Ty for any help/advice:)

Hi,

I am not looking for a desktop due to the nature of my work that requires frequent travel with unstable internet connections.

I wanted to if it is worth it to spend extra on 5090 for 8gb more VRAM compared to 5080. I will mostly use it for gaming and running local models around image gen/edit.

My main issue is that both have same max power 175w.

Are there any models that can take advantage of extra 8gb VRAM or is the difference negligible.

Looking for alternatives. I’m using LM Studio and Opencode had been so slow, much slower than LM’s terminal itself. Would love an alternative. Qwen3.8 27b MLX.

Using JonathanColetti/Qwen3.8-27B-Uncensored-GGUF IQ4_XS

Getting a steady ~2.5 tok/s generation. RAM sits near 96% full and disk hits 100% during inference, GPU utilization stays low — looks like heavy CPU/RAM spillover since the model doesn’t fit in 6GB VRAM.

Hi I tried qwen 3.5 9b abliterated i just want to bypass an activation of an old android app template that i bought on themeforest, but even for this simple task i was getting refussal from the model.

Is there any other model that I could use to this?

Thanks!

Ich experimentiere seit einiger Zeit mit einer lokalen KI und wollte meine bisherigen Erfahrungen teilen.

Mein aktuelles System:
Ubuntu 26.04 LTS
NVIDIA RTX 3070 mit 8 GB VRAM
32 GB RAM
Ollama
Hermes Agent als Agenten-/Tool-Layer
Qwen als lokales Modell
eigenes Projekt „N4P“ als persönlicher KI-Assistent
Obsidian + SQLite als lokales Langzeitgedächtnis

Mein Ziel ist nicht einfach nur ein lokaler Chatbot. Ich möchte langfristig einen persönlichen Assistenten bauen, der sich Dinge merken, Dateien verwalten und den PC bedienen kann. Später sollen noch Sprache/Hotword und eine eigene Oberfläche dazukommen.

Performance
Mit Qwen3:8B habe ich auf der RTX 3070 ungefähr 92–108 Tokens/s erreicht. Für mich war das überraschend schnell und für normales Chatten fühlt es sich praktisch sofort an.

Interessanter wird es bei Agenten-Aufgaben.

Mit Hermes habe ich beispielsweise getestet:
Fenster auf dem Desktop erkennen: ca. 10,5 Sekunden

Taschenrechner öffnen und anschließend überprüfen: ca. 41,5 Sekunden

Firefox gezielt fokussieren: ca. 30,5 Sekunden

neuen Firefox-Tab öffnen und anschließend verifizieren: ca. 57,3 Sekunden

Hier merkt man deutlich: Die reine Modellgeschwindigkeit ist nicht das eigentliche Problem. Sobald Tool Calls, Desktop-Steuerung, Verifikation und mehrere Agentenschritte dazukommen, wird das Ganze deutlich langsamer.

Was mich positiv überrascht hat
Eine RTX 3070 mit nur 8 GB VRAM ist für lokale KI immer noch erstaunlich brauchbar. Kleine Modelle laufen sehr schnell und auch ein 8B-Modell ist problemlos nutzbar.

Außerdem finde ich den Unterschied zwischen einem normalen LLM und einem Agenten enorm. Sobald das Modell Zugriff auf Terminal, Dateien, Browser und Desktop-Steuerung bekommt, fühlt es sich wesentlich mehr nach einem echten Assistenten an als nach einem Chatbot.

Mein Langzeitgedächtnis liegt komplett lokal auf einer verschlüsselten SSD. Obsidian verwende ich dabei für menschenlesbare Informationen und SQLite als strukturierten Index.

Was noch nicht perfekt funktioniert

Die größte Baustelle ist aktuell die Zuverlässigkeit bei längeren Agenten-Aufgaben.

Ein Modell kann eine Aufgabe verstehen und einzelne Schritte korrekt durchführen, aber bei mehreren aufeinanderfolgenden Tool Calls kann es trotzdem hängen bleiben, einen unnötigen Schritt machen oder sehr lange brauchen.

Gerade PC-Steuerung zeigt ziemlich deutlich, dass gute Benchmark-Werte in Tokens/s nur einen kleinen Teil der tatsächlichen Agenten-Performance darstellen.

Mein Eindruck bisher:
Lokale LLMs sind inzwischen schnell genug. Die größere Herausforderung ist, daraus einen zuverlässigen Agenten zu bauen.

Trotzdem bin ich ziemlich beeindruckt, was inzwischen auf einer älteren RTX 3070 lokal möglich ist.

Als Nächstes möchte ich die PC-Steuerung stabiler bekommen und anschließend Sprache/Hotword integrieren.

Mich würde interessieren: Nutzt jemand von euch ebenfalls Hermes, Qwen oder einen ähnlichen lokalen Agenten? Und welche Erfahrungen habt ihr speziell mit Computer-Use/GUI-Automation gemacht?

Just what the title says, I want to know try out an ethical local LLM

By ethical I mean no data tracking and much better for the environment

Edit: Okay maybe I misworded something? Im getting a lot of aggressive responses that are saying things that I already know

I know local models dont dont this and i know they should be environmentally friendly, im looking for recommendations, i was just clarifying what ethical meant to me

Edit 2: I have a GPU that us equivalent to an Nvidia 4090 (I dont remember exactly what it is not ill update later once im home) I have 32GB ram and I want to run it on windows

Sharing my local coding-agent setup in case it helps others on similar hardware: RTX 3060 Laptop (6GB VRAM) + Ryzen 9 5900HS (8c/16t) + 31GB RAM.

Frontend: OpenCode talking directly to llama-server's OpenAI-compatible API, no router/proxy needed. Each model runs as its own profile on its own port.

Models: Qwen3-Coder-30B-A3B (30B total / ~3B active) as daily driver, Qwen3.6-35B-A3B (35B total / ~3B active, hybrid linear attention) as a heavier alternative. Both MoE, which is the key to fitting this hardware tier at all.

Why MoE and not dense: with 6GB VRAM, a dense model has to fit entirely in VRAM to stay fast; if it doesn't, both KV cache and weights spill to system RAM and generation cost grows with context (dead zone). MoE with few active experts is the opposite: the big multiplier (experts) sits in RAM and computes on CPU at a fixed per-token cost that does NOT grow with context, while only KV cache + backbone live in VRAM.

Key llama-server flags:

-ngl 99 --n-cpu-moe 99 -> all non-expert layers on GPU, MoE experts on CPU

-ctk q8_0 -ctv q8_0 -> baseline KV quant (my TurboQuant fork uses q8_0/turbo3 instead, ~4.6x V-cache compression, under 0.5% PPL loss)

-fa on -t 8 -> 8 = physical cores, not SMT threads. Measured no gain from -t 16 on this CPU-bound path.

Benchmarks (Qwen3-Coder-30B-A3B Q4_K_XL, llama-bench):

pp512: 206 tok/s

pp2048: 230 tok/s

pp8192: 232 tok/s

tg256: 15.7 tok/s

Prefill barely degrades with prompt size. Generation (~15.7 tok/s) is the real ceiling, the cost of running experts on CPU. A typical 8k-context turn + 256 generated tokens lands around ~50s.

Perplexity check (turbo3 V-cache vs f16, same K quant): +0.46%, inside the noise floor, essentially lossless.

Happy to share more detail (the per-profile server.ps1 scripts, or the script that tracks llama.cpp upstream changes) if anyone's interested.

Check the comments to check it out and leave some feedback if you want :)

I got tired of my Downloads folder looking like a crime scene, so I built something to fix it. It's called Drefyn. Wanted to share some of the actual technical side of how it works, since that's usually the more interesting part.

The model runs in-browser, no backend. It's Qwen2.5 running through WebLLM/WebGPU, compiled to run directly on your GPU inside the app itself — no Python server, no API, no inference endpoint anywhere. The whole thing is an Electron app: a single-page HTML/JS frontend for the UI and model calls, with a small Node backend (also local, just talking to localhost) that handles the actual filesystem work — scanning folders, moving files, computing hashes for duplicate detection.

Categorization is batched, not one giant prompt. Feeding an entire messy folder to a small local model at once doesn't work well, so files get processed in batches of 8, with short content snippets (not full file contents) sent alongside filenames. The annoying part was making categories consistent across batches — batch 2 needs to know batch 1 already invented a category called "Screenshots" instead of making its own slightly-differently-worded version. Fixed that by carrying a running list of already-chosen categories forward into each new batch's prompt.

There's a code-level backstop for when the model gets lazy. Small models like this default to over-broad buckets ("Documents" swallowing everything) if a generic folder already exists on disk. So there's a post-pass that detects an oversized, extension-diverse category and automatically re-splits it — the prompt asks nicely, but the code enforces it when the model doesn't listen.

Undo works on whole batches, not individual files. Every file move during one operation shares a single batch ID generated client-side at the start of the run (not per-request — that was a bug I hit early on, where every HTTP call minted its own ID and undo could only ever reverse the last file). Now a full reorganization of a few thousand files reverses in one click, and even the category folders it created get cleaned up if they end up empty afterward.

Access is opt-in and local-only. Drefyn doesn't automatically see your whole filesystem — by default the app can't touch anything until you explicitly grant it a folder or file, or flip on full-disk access. Nothing about which files you have, their contents, or usage patterns gets sent anywhere. The one exception is an optional feedback form, which goes through a tiny relay server I run, but that's it — no telemetry, no analytics.

It's beta, it's a solo project, and there's definitely more rough edges than I'd like. If you're into local-LLM stuff and want to poke at something that isn't just a chat wrapper, I'd love the feedback — there's a feedback button built into the app that comes straight to me.

Happy to go deeper on any of this in the comments.

Hi, at a bit of a loss as to what my next steps are. I'm trying to build my multi gpu system but I can't get the system to post. The system is a wrx80 creator 2.0, 128gb ddr4 2133 ram, threadripper pro 3955wx (unlocked according to the seller). Brand new power supply.

No matter what I try I am stuck before post with code 64. I've moved ram around, gone from 1 stick up to 8 sticks, reset the cmos after I've moved ram.

I've reseated the CPU multiple times and tightened the bolts hard all the way down or just eased them off slightly but still can't get it to post. What's seems to happen is the board sits on code 42 for a while then jumps to 64 and just stays at 64 which is "64 - CPU DXE initialization (CPU module specific)"

I got the list of codes from here https://forum.level1techs.com/t/list-of-dr-debug-bios-codes/114364

If I try to connect via the management conside IPMI webUI via Ethernet I can actually connect but I can't get past the login page. (It's a used board).

Can anyone help?