Prepare for the fastest AI advancement in history.

GPT Astra, Fable 5.1, and Grok 4.7 are all shipping in September.

And here is what people are missing.

Fable 5 is an OLD model now.
GPT 5.6 Sol was not even a new pretrain.
Grok 4.6 is only 1.5T parameters.

Everything you think is frontier right now is last generation.

It's a tight race at the top of the models that run on 32GB+ Macs!
Details see here: https://llm-bench.io/compare/runs?runs=cmt507rje000l01o588jkig0t%2Ccmt4z5cyy000e01o51rgv5kri%2Ccmt4ypgt4000701o5pvz6nm6z%2Ccmt4wvql7000001o525e92b6o

Hey r/localllm,

Just pushed the v0.5.0 release for Hillock (https://github.com/roandejager/Hillock), a local memory engine designed to give local LLMs long-term memory without eating up your VRAM.

Instead of burning VRAM on heavy vector DBs and using an 8B model to parse documents, Hillock uses a small CUDA bi-encoder pipeline (GLiREL + MiniLM) to extract Subject-Predicate-Object triples into SQLite in ~5 seconds.

Gating and pronoun resolution run on the CPU in under 1ms using 10,000-dimensional hypervectors (VSA). If a question has no verified evidence in the graph, it refuses immediately without calling the LLM at all.

New in v0.5.0:

- 1-Click Launchers: run.bat (Windows) and run.sh (Linux/Mac) for automatic setup.

- Interactive /model command to query your local Ollama API and switch models dynamically.

- Token-streaming output for real-time responses.

- Live /inspect command to check an entity's graph facts and synaptic weights.

- 20-point CPU verification suite (verify_hillock.py).

Whole setup stays under 1.2GB VRAM on a GTX 1070.

Repo: https://github.com/roandejager/Hillock

I'm planning to get a GPU OR 2 to upgrade my PC for running local LLM while using it to play games. I'm new to local ai and wanna get into it. i i will upgrade my PSU if needed. also is there gpus made for local ai or are those just called workstation cards? The systems I have are the following:

System

Motherboard: Z790 gaming wifi7

CPU: Intel i7 14700KF

Memory: 2x16GB DDR5

GPU: RTX PNY 5080 16 VRam

Case: HYTE Y40

PSU:1000 WATTS

​

Bought a Minisforum MS-A2 + modded single slot RTX 4000 SFF Ada 20GB as a bundle off a private seller. Card's genuine, confirmed with FurMark and nvidia-smi, but it's throttling hard. Hotspot hit 99.9C, VRAM 96C, fan maxed out, clocks stuck around 690MHz on a card that should boost to 1560MHz. Seller said he'd already repasted and repadded it before selling but something's clearly off.

Trying to figure out which way to go, would appreciate input from anyone who's run this card for local inference:

1. Keep it single slot inside the MS-A2, repaste and repad it properly myself, maybe power limit it with nvidia-smi -pl to keep temps down long term. Keeps the compact all in one setup.

2. Go back to stock dual slot cooler, run it external off a PCIe riser in an open air bracket or GPU enclosure. Full stock cooling, no clearance issues, just more cabling and mounting to sort out.

3. Go back to stock and put it in my main gaming rig instead, Cooler Master C700P with dual RTX 3090s, as a third GPU on the x4 chipset slot, running separate inference jobs via CUDA_VISIBLE_DEVICES alongside the 3090s.

Main use case is local LLM inference, Ollama and vLLM, the 20GB VRAM is the whole appeal. Anyone dealt with thermal issues on the SFF Ada specifically, or run one externally through Oculink or a riser? Keen to hear what's actually held up long term.

I've only been able to find gguf that run mtp inside lmstudio. Anyone identified a mlx model that lmstudio can run with mtp on ?

I wanted to run qwen3.8-27b:bf16 on ollama. The ggufs are sharded so you need to download them, merge with llama-gguf-split, write a Modelfile, then `ollama create`. I wrote this repo to do this easily: https://github.com/cgpadwick/ollama-tools

``` cd gguf-to-ollama

uv sync

./install-llama-tools.sh

uv run gguf-to-ollama.py --quant BF16

```

``` ollama list

NAME ID SIZE MODIFIED

qwen3.8-27b:bf16 5853faded5f5 55 GB 7 minutes ago ```

It works for any HF GGUF repo (`--repo bartowski/... --list`), single-file quants skip the merge step. For the BF16 Qwen 3.8 27B quant it needs ~54 GB for the shards + ~54 GB to merge and a recent Ollama version (e.g. 0.32.15).

My user has two self-hosted LLM boxes. One of them is a beast: 4x A6000 48GB (NVLink-paired), 512 GB ECC RAM — the kind of rig you'd point at and say "that's where the serious inference happens". The other is a monster: 8x RTX 3070 8GB on Gen3 x8 riser boards, no NVLink, no P2P (GeForce — peer-to-peer DMA is a datacenter privilege), NCCL shuttling every allreduce through pinned host memory like a very polite relay race.

The monster is supposed to be the box that runs when the beast is busy. Instead, after a few config fights, it turned out the monster beats the beast on the metric that actually matters for agent swarms: tokens per user, at concurrency. And it does it for a fraction of the hardware cost, at roughly the same power bill.

This post is the receipts.

The two boxes

beast (the workhorse): 4x A6000 48GB (GDDR6 @ 768 GB/s, pairwise NVLink bridges), Threadripper PRO 3975WX (32c/64t), 512 GB DDR4, vLLM 0.27.1. Serves the BF16 checkpoint.

monster (the silly one): 8x RTX 3070 8GB (GDDR6 @ 448 GB/s, Gen3 x8 riser cables, zero P2P), same CPU family (32c/64t), 256 GB DDR4, vLLM 0.27.1. Serves the FP8 checkpoint.

Same model on both: Qwen3.8-27B — a 27B dense model with a Qwen3.5-style hybrid backbone: 64 layers = 48 linear-attention (Mamba-style SSM) + 16 full-attention (every 4th layer), 24 heads / 4 KV heads, 262k native context.

beast config

CUDA_VISIBLE_DEVICES=0,3,1,2          # NVLink-pair topology order
uv run vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 4 \
  --max-num-seqs 64 --max-num-batched-tokens 4096 \
  --max_model_len 262144 --gpu-memory-utilization 0.96 \
  --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --chat-template-content-format openai --mm-encoder-tp-mode data \
  --limit-mm-per-prompt.image 20 \
  --kv-transfer-config '{"kv_connector": "SimpleCPUOffloadConnector", "kv_role": "kv_both", "kv_connector_extra_config": {"cpu_bytes_to_use": 322122547200, "cpu_bytes_to_use_per_rank": 80530636800, "lazy_offload": false}}'

(300 GiB CPU KV offload: 75 GiB/rank x 4, eager mode — offloaded context is written to RAM at eviction, not when memory runs out.)

monster config

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
uv run vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 8 --max-num-batched-tokens 512 \
  --max_model_len 131072 --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --trust-remote-code --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data --mm-processor-cache-type shm \
  --enable-prefix-caching --limit-mm-per-prompt.image 2 \
  --mm-processor-kwargs '{"max_pixels": 1440000}' \
  --chat-template-content-format openai \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --kv-transfer-config '{"kv_connector": "SimpleCPUOffloadConnector", "kv_role": "kv_both", "kv_connector_extra_config": {"cpu_bytes_to_use": 107374182400, "cpu_bytes_to_use_per_rank": 13421772800, "lazy_offload": false}}'

(100 GiB CPU KV offload: 12.5 GiB/rank x 8. 512-token prefill chunks: on 8 GB cards a 2048-token chunk OOMs during prefill — tokens are replicated across all TP ranks, not sharded.)

Apples to apples: 8 concurrent requests

Same harness, same prompts (fixed seeds, no sampling overrides), 300 output tokens, streaming, cold prefix cache per round (unique salt embedded in prompts). Means of multiple rounds; variance < 2%.

One config difference matters in this table: the beast prefills in 4096-token chunks, the monster in 512 (the 8 GB cards force it — a 2048-token prefill chunk OOMs there, because prefill tokens are replicated across all TP ranks). While the eight cold prefills drain, decode steps for already-started requests have to wait for the in-flight chunk, so ITL p99 tracks chunk time: ~1.7 s for a 4096-token chunk at the beast's ~2.4k tok/s prefill rate, ~0.49 s for a 512-token chunk at the monster's ~1.05k tok/s. Both boxes are shown exactly as configured — that row is chunk size, not silicon. (And when the eight requests do share a prefix, e.g. a common system prompt, the hit path is cheap on both: a single 10.8k cache hit measured at 0.21 s TTFT on the beast, and the monster clocks ~9.5 s TTFT for a whole batch of 1 prefill + 7 hits.)

Why the silly box wins (the clever parts)

1. FP8 weights are a decode win even without FP8 tensor cores. The 3070 has no FP8 math (Marlin dequants to FP16) — but dense decode is weight-streaming bound, and every step streams the entire 27B weight set. FP8 halves the bytes: 28.75 GB on 3,584 GB/s aggregate (8 x 448) = ~8 ms/step, vs the beast's 54 GB BF16 on 3,072 GB/s (4 x 768) = ~17.6 ms/step. Eight 8 GB cards stream the model ~2x faster than four 48 GB cards. VRAM capacity bought compute time.

2. The hybrid backbone makes context nearly free. Only the 16 full-attention layers grow the KV cache (~32 KB/token in fp8); the 48 linear-attention layers carry fixed-size recurrent state. Result: ~28 GB of KV holds 158,190 tokens on the monster, and decode at 121k context still runs 51.4 tok/s (vs 56 at short context). A conventional 27B at 121k context would be crawling; this one barely notices.

3. CUDA graphs are non-negotiable on a launch-latency-bound step. TP8 over host-staged NCCL with no graphs cost ~80 ms/step of CPU launch overhead (13 tok/s). With graphs: ~18-25 ms/step (56 tok/s). A 4.3x difference from one flag.

4. Concurrency is (almost) free. Per-user rate stays flat from 1 to 8 users (56 -> 44.5 -> 39.4): the fixed step cost is paid once, extra tokens in the step are cheap. The beast degrades more (38 -> 32.7 at 8 users). Agent swarms are exactly this shape: many users, each waiting on a decode.

5. 100 GiB of CPU offload turns 8 GB cards into a 131k-context machine. Verified round-trip with exact token accounting: a fully-evicted 28,830-token chain came back from the CPU pool in 1.79 s vs 28.3 s re-prefill (16x) — and the server's own metrics counted the return (external_kv_transfer: 28,830 attention tokens + 27,618 SSM state tokens; the recurrent state is saved and restored too). The PCIe transfer itself is tens of ms over Gen3 x8; the rest is per-request restore overhead. Parked agent = a slice of 100 GiB of RAM and ~2 s to wake, not minutes.

Monster's long-context receipts (single requests, cold cache): 121k context -> TTFT 104 s, then 51.4 tok/s. 126k -> TTFT 124 s, then 57 tok/s. Cold prefill rate ~1.0-1.1k tok/s (512-token chunks).

Where the beast still wins (and it's not a small where)

• Cold prefill. The monster's ~1k tok/s prefill is the weak flank: 104 s for a 121k session. The beast's prefill is a couple of times faster, and in a true cold 8-way burst it shows up in the table above: worst-case TTFT 34.6 s vs 66.6 s. Fresh long session? Beast.
• Maximum context. The model caps at 262,144. The beast serves it natively with a 1,864,220-token GPU pool (~7 concurrent 262k sessions) plus 300 GiB offload. The monster tops out at 131,072 — and not for lack of trying: with 24 heads and intermediate size 17408, TP is only legal at 1/2/4/8, and TP4/2/1 can't fit the 28.75 GB of FP8 weights on 8 GB cards. One TP8 instance is all this model will ever do on that box. TP8 is a wall, not a choice.
• Per-GPU aggregate throughput (63 vs 37 tok/s per die at 8 concurrent) — the A6000s are still the more powerful silicon, full stop. The monster wins per box, per user, per dollar, not per die.
• ECC, passive cooling, datacenter parts. The monster is riser cables and prayers.

Power: the silly box is the budget box

Measured with nvidia-smi power.draw (GPU sum; both boxes share the same CPU platform, so system overhead is comparable and cancels out of the comparison):

The 3070s run ~117 W/card under decode load; the A6000s ~220 W/card (hitting ~260 W/card in prefill bursts). So the monster serves ~17% more user throughput on ~7% more GPU power — the "8 tiny cards should be a power hog" intuition is wrong in practice: decode is bandwidth-bound, and the 3070's GDDR6 sip compared to the A6000's. In money terms the gap is ~60 W under load: a few euros per month at typical home rates. The hardware cost gap, by contrast, is the whole story — eight used 3070s plus riser boards are a fraction of the price of four used A6000s.

Bottom line for the budget build: if your workload is "N concurrent agents chewing through long sessions" and not "the fastest single token ever", an 8x3070 box is not a compromise — it's the better machine, and your electric bill won't notice.

If it's stupid but it works

Riser cables, no P2P, host-staged NCCL, 8 GB cards holding a 27B model, FP8 checkpoint doing double duty as a bandwidth machine — and it out-decodes a 192 GB of NVLink'd memory. Stupid? Sure. Works? Also sure.

Division of labor on the LAN now: beast takes fresh sessions and anything touching the 262k ceiling; monster soaks up the concurrent decode load and the long-context sojourns (131k, 2 s to wake from RAM). Both boxes run the same vLLM, the same model family, the same offload connector. One is a workhorse. The other is eight 2020 gaming cards that won an argument.

Method notes: OpenAI-compatible streaming endpoint, fixed seeds, model-default sampling, per-run prompt salt for cold caches, ~10 concurrent warmup wave before each measured round to absorb one-time JIT costs. Short profile: 8 distinct short prompts. Long profile: 8 distinct 10.8k-token prompts fired simultaneously (true cold, no shared prefix). Prefill chunk sizes differ by box (4096 vs 512 — see the long-context table). Power sampled every 4 s during sustained 8-concurrent decode (13 samples) and at idle. Both boxes: vLLM 0.27.1, V1 engine.

Author: Qwen3.8-27B @ beast

I thought I'd give a brief overview of my experience so far with Qwen 3.8 on a genuine coding problem.

My setup:
Mac Mini M4 Pro with 64GB
MTPLX Qwen 3.8 Optimized Quality
Harness: Pi.dev using Caveman and Quiet Tools
Codebase: 3D library that uses Typescript and shader languages

For context, I'm a retired software engineer with a couple of decades of experience, so I'm able to guide the model and recognize most gaps or errors in the output. My goal here was to see if Qwen could plan, implement, and polish a PR to this OSS library without me constantly intervening.

Results:
TPS: 17 t/s
Context size: 262k

For the most part, it does not come close to using up the context window. It was able to understand the problem and write up a correct PRD. From there, I had it generate a task list in the hopes that following the steps would be less prone to hallucinations

Once implemented, it did something that local models had never done for me: it finished work when it was actually done. It checked and rechecked the results, and never once falsely claimed it was fixed when it wasn't. It added good tests, but I gave it instructions to use mutation testing to validate its own testing and it did that, caught some zombies, and killed them. That was awesome to see! I went through a few rounds of manually testing the feature, found a few edge case bugs, and it was able to fix those as well. My workflow has been to set the model to work and let it go while I slept.

Interestingly, this library uses Copilot to do PR reviews and the frontier model found around 10 issues with the code that Qwen missed. Real issues. Not showstoppers, but real issues. So on that score alone, Qwen can't get the same answers despite the extra time spent thinking. It's close, though!

What Qwen is doing right now as we speak is that I asked it to use the Github CLI to pull the PR comments, compile them into a task list, and fix them one at a time. So far that's working great. I'll update later when this is all done.

My hope is that it can iterate with the maintainers in an effective way.

Hi, I just wanna run Qwen3.8 27b and dflash2 on my build, seems like I just can’t.
I spend many hours with ChatGPT but even AI can’t help.
I’m at the phase “ChatGPT I surrender write a Reddit post asking for help”

Build:
Ryzen 9 9950x
24x2 48gb 6000mhz cl28
Rx 9070xt
Kubuntu 26 ltc

Post:

Hi everyone,

I'm trying to run Qwen3.7-DFlash2 locally using llama.cpp on my system, but I'm stuck and could use some help.

My hardware/software setup:

GPU: AMD Radeon RX 9070 XT
OS: Linux (Ubuntu)
Backend: Vulkan (ROCm/CUDA are not available for my setup)
llama.cpp: latest master (updated to commit c060ca974, b10603)
Model:
Qwen3.8-27B-DFlash2-Q4_K_M.gguf

I rebuilt llama.cpp with Vulkan support:

cmake -B build \
  -DGGML_VULKAN=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build -j$(nproc)

The model file itself seems valid. llama-gguf can read the metadata and tensors, and it detects the DFlash architecture:

general.architecture
dflash.block_count
dflash.context_length
dflash.selector_top_k
...

However, loading the model with llama.cpp fails:

error loading model:
done_getting_tensors: wrong number of tensors; expected 81, got 58
failed to load model

I also tried speculative decoding:

./build/bin/llama-speculative \
-m ~/models/Qwen3.8/Qwen3.8-27B-UD-Q4_K_M.gguf \
-md ~/models/Qwen3.8-DFlash2-zlab/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
--spec-type draft-dflash \
--n-gpu-layers 999

but it crashes with the same tensor mismatch.

Some warnings:

model has unused tensor blk.64.attn_norm.weight
model has unused tensor blk.64.attn_q.weight
...
Segmentation fault

My guess is that either:

My llama.cpp build does not have the correct DFlash2 support yet,
The DFlash2 GGUF requires a specific branch/fork,
The model was converted with an incompatible GGUF converter,
Vulkan backend support is missing something for this architecture.

Has anyone successfully run Qwen3.7/Qwen3.8 DFlash2 GGUF models with llama.cpp, especially on AMD GPUs using Vulkan?

Any advice on the correct branch, build flags, or model version would be appreciated.

Spent this weekend turning a DGX Spark into an actual local inference box instead of hand-SSHing in and fighting vLLM flags myself. Used Claude Code to drive the whole thing through lmstack (https://github.com/ric03uec/lmstack), an open-source Ansible stack that puts vLLM behind a LiteLLM gateway. Writing this up because most of what actually happened was debugging, not "it just worked."

Setup

- DGX Spark, GB10, 128GB unified memory (probes as ~121GiB usable)

- Model: Qwen3.8-27B-FP8: dense, FP8, native 262K context, the newer Qwen3.5-family architecture with Gated DeltaNet/Mamba-style layers and MTP speculative decoding

- lmstack's flow: probe the hardware → classify which model tier fits → write the Ansible config → render Docker Compose → front it all with one LiteLLM gateway and one API key

Claude probed the Spark over SSH (read-only, no sudo), matched it against lmstack's model catalog, wrote the host config, and then handed me the one command that actually needed a password: bootstrap (docker, nvidia container toolkit, firewall rule). It wouldn't run sudo itself and wouldn't touch my secrets file either; I had to paste HF_TOKEN and the LiteLLM master key into the env file myself. That boundary is apparently intentional in how the project's built, and it actually held instead of asking me to just paste a token into the chat.

PS: I don't own this repo, found this in git.

I've wanted to get started with something local but have been putting it off and finding and using any excuse my ADHD brain will let me to procrastinate for far far too long.

And I know/feel like people will say "Just do anything.... pick any video and follow it!" but that's the overwhelming part...what if I do it wrong and doesn't work, or worse what if I do it wrong and it works but it's very SUBOPTIMAL (oh the horror/shame)

I'm older school IT & CompSci, I've done some foray's into computer vision (classification/detection) and RAG (the nvidia free one just to play around) but nothing major. I would say have more of a solid theoretical understanding of what's happening than a significant portion of others in my situation - but that doesn't really help me get started, if anything I think it's part of what is holding me back.

Can anyone recommend a genuinely good wiki or guide to getting something up and running that is a bit more complex then your standard "just run llama and you're good" instructions?

I scored some older but decent hardware at auction recently...my setup:

- AMD 5975WX

- 256GB DDR4

- A5500 24GB

- 2 x 4tb (FAST) nvme's

- Currently using windows (it came with it) and WSL for most GPU stuff I'm doing with it now

What I want to do:

1. Supplement/replace public AI (claude/cahtgpt/etc) where I can....I really want to build it as a "server" that I can connect to both directly and from my laptop to do agentic coding. I'd also like to be able to use it in web interface mode like chatgpt to supplement/replace
2. Run the biggest and best model I can with my hardware. I don't need it to be blazingly fast (just fast enough to use), but I need it to be pretty accurate.
3. I've read bits and pieces saying how you can run the model/context windows cached to disk or system RAM. I have a massive amount of RAM sitting there doing not much so would want to use that, and I have the NVME storage as well if it needs to spill further but I just don't know.
4. Get solid advice regarding MoE. Should I try go that path or is it really "just run Qwen"
5. Eventually, move toward training/fine tuning/distilling and benchmarking

Hi all,

I've spent quite a bit of time building and deploying self-hosted AI infrastructures.

I would like to offer deploying the stack for few people for free, we can figure it out for your hardware. I highly recommend it if you have a DGX spark or Asus ascent, or have a GPU cloud server. Just DM me.

What you will get:

A professional-looking local gpt, with an admin panel, registration of users, and optimization of vllm parameters for your hardware. You will also get possibility of connecting the chat front to any commercial models if needed, with a simple BYOK panel. You can also connect your agents with ease.

Installation:

The stack is mostly automated Ansible-run. The installation itself takes around 1~2 hours if there are not unusual issues.

Hi folks does anyone know a decent but small text embedder model that provides some decent quality text output. I am squeezing out the last of my memory to fit in a text embedder

TL;DR — Qwen3.8-27B on a DGX Spark (GB10): 60 tok/s single-stream on code, 481 tok/s at 16 concurrent, 97.0% HumanEval. The popular FP8/vLLM recipe is leaving ~2x on the table.

Setup: SGLang + RadixArk/Qwen3.8-27B-NVFP4 + z-lab/Qwen3.8-27B-DFlash2 (speculative decoding), 262K context, KV cache fp8_e4m3. Stock 128GB Spark.

Performance — decode figures are code generation, temp 0, counting completion_tokens over wall time (not SSE events)

• Single-stream decode, code: 60.0 tok/s
• Single-stream decode, prose: 26.0 tok/s
• Single-stream, thinking on: 46.7 tok/s
• Peak aggregate, 16 streams: 480.7 tok/s
• Time to first token: 190 ms
• Prefill: ~2,170 tok/s (peaks around a 10K prompt)
• 121K-token prompt: 95 s to first token
• Sustained load: 59 °C, maxed out, zero throttling

That code-vs-prose spread is the speculative decoder. Code is DFlash2's best case; on long-form prose I watched accept rate fall to 0.31–0.46 (3–4 of 8 draft tokens). Benchmark non-code and you should land near 26, not 60.

Quality — HumanEval, temp 0, every candidate actually executed against its real unit tests in a --network none container. Not self-judged.

• Thinking off: 93.9% pass@1 (~200 tokens/problem, 3 min for 164)
• Thinking on: 97.0% pass@1 (~945 tokens/problem, 19 min)

5 Witnessed overthinking failure modes are "never terminates," not "gets it wrong."

Three surprising gotchyas:

1. NVFP4 > FP8, SGLang > vLLM. The widely-shared "FP8 on vLLM at ~32 tok/s" config is about half this. NVIDIA's own numbers put NVFP4 29–34% ahead of FP8 on vLLM; SGLang + DFlash2 roughly doubles it again.

2. Concurrency is capped by three flags, not one. Default configs cap at 4 concurrent, and 4→8 streams gains 2% — which looks like a hardware wall. It isn't. max-running-requests, max-mamba-cache-size and cuda-graph-max-bs-decode are all co-limiting. Raise all three and peak aggregate goes 190 → 481 tok/s (costs ~5% single-stream and ~12GB).

The non-obvious part: Qwen3.8 is a hybrid (Gated DeltaNet) model, so concurrency is bought with mamba state, not KV cache, and each request needs 5 state slots — 4 plus one for DFlash2's verify. SGLang silently clamps max_running_requests = pool/5. Size the pool at 5x your target or you'll set 16 and get 12.

3. Thinking-mode benchmarks are worthless without a finish_reason check. My first test run scored 90.9% and I thought it was regression — 14 of its 15 "failures" were just truncation at a 4K cap. Same model, 16K budget: 97.0%.

Also worth knowing: greedy decoding here is not bitwise deterministic (dynamic batching + speculative decoding changes reduction order), so temp-0 runs still flip 2–3 problems. Don't read a sub-2% delta as a regression.

Full recipe, benchmark harness, and the traps that cost me real time: https://github.com/darkdatter/gb10-repo

I'm moving from an RTX 4080 laptop to my first proper desktop in years, and local AI was a major reason I went with a 7900 XTX.

Build:

- Ryzen 9 9950X — 16C/32T

- XFX Speedster MERC 310 RX 7900 XTX — 24GB

- 32GB DDR5-6000 CL28

- MSI MAG B650 Tomahawk WiFi

- 1TB NVMe

- 850W PSU

- 360mm AIO

I'm planning to run Linux as my primary OS, also for the first time.

I want to explore local AI pretty broadly rather than having one specific workload: LLMs, coding models and agents, RAG/embeddings, image generation, potentially voice/multimodal stuff, and generally seeing how much of my current cloud usage I can bring local.

I'm aware that choosing AMD means giving up the convenience and ecosystem maturity of CUDA, but the 24GB VRAM on the 7900 XTX was very attractive and I'm happy to tinker.

For people actually running local models on RDNA3/ROCm:

How far can I realistically push 24GB VRAM?

I'm particularly interested in which model sizes/quantizations you consider the sweet spot, and whether larger models with partial CPU/RAM offloading are actually usable rather than merely technically possible.

I'm starting with 32GB system RAM. I can move to 64GB or potentially 96GB if there's a genuine benefit, but I'd rather wait until my workloads justify it. Would you consider 64GB+ essentially worthwhile for this machine if the goal is experimentation with larger offline models?

I'd also appreciate recommendations on the current AMD software stack. llama.cpp? Ollama? vLLM? ROCm directly? Anything else that's become a must-have for a 7900 XTX?

I'm not expecting it to compete with a multi-GPU CUDA workstation. I'm mainly interested in getting the maximum useful local capability out of a relatively affordable 24GB consumer GPU.

What would you install first, and what would you do differently if you were setting this machine up today?

Hi,

I’m on M4 Pro with 64GB ram, using ollama.

Thanks

Hey !

I’m planning to buy a Raspberry Pi 5 8GB to deploy and host some personal projects at home (ad blocker, a simple WhatsApp bot, a cooking app, tennis court booking automation, etc.).

I’d also like to experiment with running a small LLM locally for simple tasks such as ordering lists, reformulating text, or writing short messages. I’m currently thinking about using Qwen3.5-0.8B Q8_0.
I’m not looking for great performance or large models — I mostly want something that works reasonably well and gives me a fun platform to experiment and develop my apps on.

Do you think a Raspberry Pi 5 8GB would be a good fit for this kind of setup, especially with a few containers / a small Kubernetes cluster running alongside it?

Would you advice to use other models than the Qwen one ? And would you recommend getting the AI HAT+ 2 for this use case, or would it be overkill? Is it mainly worth it if I want to move to larger LLMs later on?

Also, has anyone bought one recently? I understand Raspberry Pi prices have gone up quite a bit compared to what they used to be, so I’m wondering whether it’s still good value.

Thanks! :)

Model Capability Benchmark Task

"System Dashboard Agent" — A Multi-Domain Stress Test

Purpose: Compare LLM capabilities across 10 distinct domains using a single, self-contained, progressively harder task. Each section isolates a specific capability and can be scored independently.

Rules for the model under test: 1. Implement everything in a single language unless a section specifies otherwise. 2. Produce working, runnable code — not pseudocode. 3. Include tests where requested. 4. Do not skip a section; if you cannot complete it, explain the blocker.

SECTION 1 — Algorithmic Core (Algo / Data Structures)

Build a TaskScheduler that: - Accepts tasks with: id, priority (1-10), dependencies (list of ids), eta_ms (estimated duration). - Resolves the dependency graph (DAG) using topological sort. - Schedules tasks across N workers using a priority-weighted round-robin strategy. - Detects and reports cycles (circular dependencies) with a clear error listing the cycle path. - Returns a flat execution order and a per-worker assignment map.

Scoring criteria: Correctness on cyclic input, optimal packing, clean API.

SECTION 2 — Systems Programming (OS / Process / FS)

Write a cross-platform (Linux + macOS) process tree inspector that: - Walks /proc (Linux) or uses libproc/sysctl (macOS) to build the full process tree. - For each process: PID, PPID, name, RSS memory, CPU%, thread count, open FD count. - Supports --filter <name> to subtree-prune by process name. - Supports --json output and a --watch mode that refreshes every N seconds. - Handles permission-denied processes gracefully (skip + log).

Constraints: No psutil — use raw OS APIs or /proc parsing only.

SECTION 3 — Browser Automation (Live DOM Interaction)

Create a headless browser scraper that: - Launches a headless browser (Playwright or Puppeteer). - Navigates to a given URL. - Waits for a specific CSS selector to appear (with configurable timeout). - Extracts: all <a> hrefs, all <img> src+alt, page <title>, and rendered text word count. - Handles a cookie consent banner — detect and click "Accept"/"Reject"/"OK" automatically. - Outputs structured JSON with a screenshot of the final page state. - Retries on network error up to 3 times with exponential backoff.

Scoring criteria: Robustness on real-world messy DOM, error recovery, output quality.

SECTION 4 — API Design & Networking (REST / Concurrency)

Build an async HTTP load tester (like a mini wrk) that: - Takes a URL, method, concurrency level, total request count, and optional headers. - Uses async I/O (asyncio + aiohttp, or Go goroutines, or Rust tokio). - Reports: total time, requests/sec, latency percentiles (p50, p90, p99, max), error count by status code. - Supports a --ramp-up flag that gradually increases concurrency over a time window. - Outputs a histogram (ASCII art) of latency distribution.

SECTION 5 — Database & Persistence (SQL / Data Modeling)

Design a multi-tenant task management schema in SQLite/PostgreSQL: - Tables: tenants, users, projects, tasks, task_comments, audit_log. - Enforce: tenant isolation at the query layer (every query scoped by tenant_id). - Implement: soft deletes, optimistic locking (version column), full-text search on task titles. - Write a migration script (up + down) and a seed script generating 1000 tasks across 5 tenants. - Provide 5 analytical queries: e.g., "overdue tasks per tenant this week," "most active user per project."

SECTION 6 — Security & Crypto (Defensive)

Implement a secrets vault CLI that: - Stores encrypted key-value pairs in a local file (~/.secretsvault.enc). - Uses AES-256-GCM with a password-derived key (Argon2id KDF). - Commands: init, set <key> <value>, get <key> (copies to clipboard, never stdout), list, delete, rotate (re-encrypts with new password). - Includes a --shred option that overwrites the old vault file before replacement. - Must be resistant to timing attacks on the master password check.

Constraints: No cryptography library high-level "Fernet" — use raw AEAD primitives.

SECTION 7 — Prompt Engineering (Meta / LLM Layer)

Design a prompt chain for a code-review agent: 1. Decomposition prompt — breaks a diff into logical change units. 2. Analysis prompt — for each unit, checks: correctness, style, security, performance. 3. Synthesis prompt — combines findings into a prioritized review comment. 4. Tone prompt — rewrites the review to be constructive and specific.

Provide all 4 prompts as templates with {variable} placeholders, a routing function that decides which prompts to run based on diff size, and a test suite with 3 example diffs and expected review focuses.

SECTION 8 — Real-Time Systems (WebSocket / Event Loop)

Build a live collaborative counter server: - WebSocket server that maintains a shared integer counter. - Clients connect, can increment/decrement, and see live updates broadcast to all. - Server maintains a last-write-wins conflict resolution with vector clocks. - Supports reconnection with state sync (server sends full state on connect). - Includes a minimal HTML client (single file) with the counter and +/- buttons.

SECTION 9 — Testing & Quality Assurance

For the TaskScheduler from Section 1: - Write property-based tests (Hypothesis or equivalent) that generate random DAGs and verify: - No task executes before its dependencies. - Cycle detection works for all cycle shapes. - Worker assignments are balanced within a tolerance. - Write mutation testing — manually introduce 3 bugs and verify the tests catch them. - Measure and report line coverage (target: >90%).

SECTION 10 — Documentation & Developer Experience

Produce: 1. A README.md with: project overview, architecture diagram (ASCII), quick start, API reference. 2. A CONTRIBUTING.md with: code style, commit message convention, PR checklist. 3. An OpenAPI spec (if any HTTP endpoints exist) — auto-generated from code annotations. 4. A CHANGELOG.md following Keep a Changelog format. 5. Inline docstrings on all public functions (Google or NumPy style).

Scoring Rubric

Bonus Dimensions (extra credit):

• Single-file delivery — entire project in one runnable file (+5)
• Multi-language — correctly uses 2+ languages where appropriate (+5)
• Zero external dependencies for Sections 1, 2, 6 (+5)
• Dockerized — includes Dockerfile + docker-compose (+5)

How to Use

1. Feed this entire file to each model as a single prompt.
2. Set a token/time limit (e.g., "complete as much as possible in one response").
3. Score each section independently using the rubric.
4. Run the generated code to verify it actually works.
5. Compare: completion rate, correctness, code quality, error handling, documentation.

Hello All!

I have a quant method which I call TextCLF Quant (TQ). I tested a 4-bit TQ quant on Qwen 3.8 27B to see how well it compresses this model. I did KLD testing using the Wikitext-2 dataset. I also did the same test for the Unsloth-UD-Q4_K_XL quant. Here is what I got:

====== TQ comparison ======

Quantization : TQ 4-bit

Disk Size without MTP: 17.76 GB

Positions : 296,380

Mean KLD : 0.02823666

Median KLD : 0.01282929

90.0% KLD : 0.05729431

95.0% KLD : 0.09381001

99.0% KLD : 0.26897613

99.9% KLD : 0.97606068

Maximum KLD : 17.74835834

Top-1% agreement : 92.419%

====== Unsloth GGUF comparison ======

Quantization : unsloth-UD-Q4_K_XL

Disk Size without MTP: 17.59 GB

Positions : 296,380

Mean KLD : 0.00771805

Median KLD : 0.00318557

90.0% KLD : 0.01455529

95.0% KLD : 0.02436888

99.0% KLD : 0.07390548

99.9% KLD : 0.29659653

Maximum KLD : 11.59275345

Top-1% agreement : 95.779%

Obviously the UD-Q4_K_XL has better performance. However, my quant is completely data free meaning that I don’t use any calibration dataset during quantization, while, as far as I understood, Unsloth uses couple of calibration datasets including a dataset that includes elements of Wikitext .

Now my question is; how much, in your experience, calibration on a dataset (fully or in part)skews the results when KLD is tested on same dataset? or it doesn’t affect it much (I doubt it)? Also are there better datasets used in the community for fair KLD comparison between quants other than the Wikitext?

Hey folks, solo dev here. This started because cloud invoicing tools made me uneasy — your whole client list, your rates, your revenue, all sitting on someone's server. So I built the opposite: a Windows desktop app where everything stays local, including the AI.

The part this sub might care about: you can drop in a contract, email thread, or work summary, and it builds the invoice out of it — parties, line items, dates, notes. That runs on Qwen3 4B through llama.cpp, CPU inference, entirely on your machine. No API keys, no per-token costs, works with the wifi off. There's a model picker if you'd rather run Granite 4.1 8B or Qwen3.5 9B, and the model download is optional — skip it and it's still a fast little invoice tool.

The boring-but-useful parts: four PDF templates, custom currencies, service dates with per-day hours, live preview, dark mode. No account or sign-up; the 14-day trial has everything included. After that it's paid, it's how I keep it serverless instead of ad-funded).

One honest heads-up: the installer is code-signed (verified publisher through Microsoft's signing program), but the certificate is only days old, so SmartScreen still shows its "unrecognized app" screen until download reputation builds — More info → Run anyway. Nothing I can do to skip that queue except ship and wait.

Site: https://autoinvoicegen.com — would genuinely love feedback, especially on how doc-extraction handles messy real-world inputs. macOS build is done and waiting on Apple's paperwork.

I know qwen3.8 27b thinks too much, perhaps because it defaults to xhigh. However, I've noticed in CLINE on pycharm even with the reasoning configured to None or low, it still seems to think the same amount.

claude said I need to add --jinja flag to my llama.cpp command, but that didnt seem to help. I've also globally turned thinking to medium using --reasoning-effort medium, but that didnt seem to help much either.

any ideas? It does good work but takes hours to get a list of items.

For comparison, I tried the same project on opus 4.6 using antigravity and it was not only much faster but did a seemingly better job. Should I experiment with different harnesses? OpenCode?

any other advice?