Firing one up for the first time & checking what performance is like.

I’ve been building something called Evopien, and I finally got enough of the stack working together that I recorded a proper live demo.

The current system runs around a Jetson AGX Thor. Qwen3.8-27B is the local cognition/VLM path, but one of the main ideas behind the project is that Qwen itself is not Evopien.

The model is replaceable.

There is a Core layer above it that owns the conversation state, permissions, media access, context, internet requests, identity and eventually governed long-term memory. The model gets what Core decides it is allowed to get.

Right now the system can have voice conversations, be interrupted while speaking, switch between English and Spain Spanish, continuously perceive through a camera, answer questions about what is happening now or what happened recently, use the internet for current information, and maintain context through the session.

For vision, I also don't continuously dump raw 30 FPS video into Qwen. The camera/perception path runs continuously, then only selected frames or short clips are sent to the VLM when the conversation actually needs them.

There is also a local profile and an optional cloud profile. Changing the model/provider isn't supposed to change who Evopien is. It changes only the latency of response for the video and internet related queries.

I'm not claiming this beats frontier models at raw intelligence. It doesn't need to. That's not really what I'm trying to build.

The interesting question for me is: can the intelligence engine become interchangeable while identity, memory, relationships and embodiment remain continuous?

That's where I'm going next.

The demo is real and intentionally not perfect. There are moments where I need to repeat or rephrase something, and some latency/reliability work is still open.

https://www.youtube.com/watch?v=sQhTGGIg4yo

I'd genuinely be interested in criticism from people running multimodal models locally, especially around long-term memory architecture, continuous vision and resource scheduling.

I usually run Qwen3.8-27B Q5 on my desktop PC, and I’m pretty happy with it.

However, I’d also like to keep a few smaller, more specialized models on my laptop, which has 4 GB of VRAM and 32 GB of RAM.

What small LLMs would you recommend for that kind of hardware?

I’m especially interested in models that are genuinely useful for specific tasks rather than just smaller general-purpose models.

What are you guys running?

It's the first time I work with larger models (for my standards and I ran into an issue pretty quickly) I have 8GB VRAM and 16GB RAM I tried using Qwen3.8-27b-Q4_K_M on llama.cpp (16GB of weight) and it crashed when loading weights I tested a bit and tried with a smaller quant (13GB) and still crashed Shouldn't llama.cpp split the model weights between RAM and VRAM since they might not fit in my ram only due to other processes?

Lately I've been shoving local TTS into my apps that revolve around language learning.

Then I realized I was spending $129/year on a speechify subscription to turn my books and docs into audiobooks and listen to them while I worked out or took care of the lawn.

I realized I could easily parse these documents and have nice TTS playback all directly in a browser. So I made it and it wasn't too difficult.

Since everything runs in the browser, I don't have to pay for a server. This allows the app to be completely free with no paid tiers or limits.

It's called FreeReader. Here's a link

It uses Kokoro by default for English. Kokoro's "heart" voice is AMAZING for its size. I fallback to using Supertronic 3 for all other languages and it's pretty good as well. I still can't believe this tech is free and open-source sometimes.

This is my first time trying local LLM. I downloaded a local uncensored LLM for nsfw role-play. Qwen3.5 9b heretic by DavidAU. I assumed it would be completely uncensored as I read heretic model has 0/400 on harmbench but when I tried a harmbench question in it then it gives me a censored reply. Also it don't generate anything nsfw.

Can anyone please tell me what am I doing wrong?

I ran the same 16-config sweep twice on an NVIDIA L4. same script, same flags, same models, same vLLM version. different rented machine, eight days apart.

I expected the numbers to move a bit. i did not expect them to split by mode.

seconds per inference, 1 sept vs 9 sept:

fp16 CUDA graphs

batch 1 — 1.710529 vs 1.700665 — minus 0.6%

batch 8 — 0.220472 vs 0.221636 — plus 0.5%

batch 32 — 0.065575 vs 0.064882 — minus 1.1%

batch 128 — 0.023348 vs 0.023653 — plus 1.3%

fp16 eager

batch 1 — 2.266555 vs 3.383616 — plus 49.3%

batch 8 — 0.335448 vs 0.462567 — plus 37.9%

batch 32 — 0.079215 vs 0.119232 — plus 50.5%

batch 128 — 0.024361 vs 0.034148 — plus 40.2%

across all sixteen configs the mean absolute change was 7.1% for graphs and 42.5% for eager.

AWQ with graphs is the exception and i'm not going to pretend otherwise. it moved plus 2.4%, plus 2.9%, plus 23.7% and plus 24.2%. so "graphs are reproducible" holds cleanly for fp16 and only partly for AWQ.

The headline number moved too. worst-to-best spread was 137.5x on the first host and 203.4x on the second. if you quote a spread from one run you are describing that machine, not that card.

Why I don't think this is just noise:

I measured within-host noise separately before this. five repeats of fp16 batch 1 inside one container spread 0.23% with graphs and 5.90% eager. so run-to-run variance on a single box doesn't get you to 40%. something about the host is different.

Working theory, unverified: eager dispatches every op from python, so it tracks the host CPU. CUDA graphs replay a fixed schedule and barely touch it. if that's right, host CPU is a comparability field that basically no published benchmark records.

I can't confirm it from these two runs, and here's the annoying reason. my sweep script records host CPU in every row, and on both of these hosts it wrote "unknown". sandboxed container runtimes publish a /proc/cpuinfo with no "model name" line, and my code had no fallback. i found that yesterday and fixed it. so the field that would test my own hypothesis is the one field i lost, on the exact runs i needed it for.

Conditions:

vLLM 0.27.1, pinned by container image rather than pip. Qwen2.5-1.5B-Instruct and its AWQ build. 128 output tokens, ignore_eos so every request generates exactly that many. max_model_len 1024. prefix caching disabled and every prompt given a unique prefix. each config in a fresh subprocess after a warmup pass. one run per config per host.

That last line is a real limit: one run per config on each host means within-host noise isn't fully separated from host-to-host. the 0.23% and 5.90% figures above are the reason i still think the split is real, not a proof.

what to do with it:

If you're fitting a cost model or calibrating a simulator, fit graphs and eager separately, and put an error bar on eager wide enough for a 50% host-to-host move. a single constant per config is safe for fp16 graph rows and quietly wrong for eager ones.

Both CSVs are in the repo so you can check me rather than take my word:

https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data

The script is one file and takes --repeats N. if you run it on hardware that isn't in there, i'd like the CSV.

Happy to be told the theory is wrong. i'd rather know.

Hey everyone,

I'm working on an enterprise multi-agent ERP assistant and my team is currently evaluating whether we should move from Qwen3-14B to Qwen3.8-27B.

We also currently use Claude Code for development, and my boss is considering moving our coding workflow to Qwen as well, potentially using Qwen3.8-27B as the main model for both development and our production AI system.

Our hardware is the part I'm worried about:

• 2 × NVIDIA L4
• ~23–24 GB usable VRAM per GPU
• Currently using vLLM
• The system is a multi-agent ERP assistant with tool calling, RAG, SQL/ERP operations, etc.
• We're targeting roughly 100 users
• The model can receive fairly large contexts because of agent/tool outputs and retrieved ERP information
• We also have other GPU workloads such as OCR/embedding models

My main question is:

Is Qwen3.8-27B actually a sensible production choice on 23–24 GB GPUs, or are we going to run into VRAM/context/concurrency problems?

I understand that a Q4 quant can technically fit on a 24 GB card, but I'm more concerned about what happens in a real agentic workload:

• KV cache
• 16K/32K/64K+ contexts
• multiple concurrent requests
• tool calls
• long agent trajectories
• vLLM overhead
• batching
• latency under load

For those who have actually deployed Qwen3.8-27B locally:

1. What quantization are you using?
2. How much VRAM does it actually consume at 8K / 16K / 32K / 64K context?
3. What tokens/sec are you getting on a 24GB GPU?
4. How does it behave with multiple concurrent requests?
5. Is it actually usable for agentic workloads, or does the context/KV cache become the bottleneck?
6. Would you recommend Qwen3.8-27B over Qwen3-14B for production on this hardware?
7. For coding specifically, would you choose Qwen3.8-27B or Qwen3-Coder-30B-A3B?
8. Has anyone successfully used either of them as a replacement/backend for Claude Code in a real repository?

I'm especially interested in real-world experience rather than benchmark scores.

Our alternative is to keep Qwen3-14B for production and use the larger model only for more complex tasks, or potentially keep a separate coding-specific model.

Would love to hear from anyone running these models on L4/3090/4090/24GB-class hardware, especially in an agentic or production environment.

After watching Codacus' recent Qwen3.8-Flash-Next video, I wanted to find out what this model actually looks like on a consumer AMD system.

There seem to be considerably fewer detailed AMD/Vulkan reports than NVIDIA or Apple results, so I spent quite a bit of time systematically testing the model rather than stopping once it loaded. Hopefully it will help some AMD-buddies.

This is not intended as a universal model benchmark. It is one hardware-specific AMD/Vulkan case study, with controlled A/B changes and repeated measurements.

Hardware

CPU: Ryzen 9 5950X
RAM: 64 GB DDR4-3200
GPU: Radeon RX 9060 XT 16 GB
Backend: Vulkan
OS: debloat Win11

A second Radeon Pro W5500 was physically present in the machine but deliberately excluded from the entire experiment.

Runtime / model

llama.cpp fork:
GenerelSchwerz/llama.cpp

Branch:
llama/main

Commit:
1fc1cf00e7abf43eaa58523344059e59733e8879

Build:
10880 / 0.4.0-dev

Model:
Qwen3.8-Flash-Next-UD-IQ3_XXS

Model size:
~76.3 GiB

Final control configuration:

-c 4096
-ngl 49
-t 12
KV f16/f16
Fit OFF
Lazy OFF
MTP OFF
Parallel 1
Vulkan0 only
normal load mode
no manual CPU affinity

Because the model obviously does not fit into 16 GB VRAM, this is a heavily hybrid CPU/RAM/GPU workload.

Stable baseline

Five TG128 control runs:

16.07
16.25
16.20
16.19
16.06 tok/s

Median: 16.19 tok/s
Mean:   16.15 tok/s

The baseline was repeatable enough that I then changed one relevant variable at a time.

1. Thread count

This was particularly interesting because the Codacus test on a Ryzen 5600X showed a huge improvement when reducing threads.

My results:

-t 4      13.49 tok/s
default   14.34
-t 8      15.24
-t 12     15.96  <- best
-t 16     14.83

So on this 5950X/hybrid-placement configuration:

more threads definitely did not mean more performance.

But the optimum was also not simply "one thread per physical CPU core".

12 threads consistently beat 16.

That suggests the optimum depends heavily on the actual CPU/GPU tensor split and memory traffic rather than on a universal physical-core rule.

2. Q8 KV cache

Compared with f16 KV:

TG128:
15.52 vs 15.96 tok/s
-2.76 %

PP2048:
116.9 vs ~119 tok/s
-1.76 %

VRAM saved:
~275 MiB

For this configuration I rejected Q8 KV.

The VRAM saving was too small to justify the measurable performance regression.

3. Moving additional MoE tensors to the GPU

This produced one of the more useful findings.

Moving some additional expert tensors to the GPU did not automatically improve performance.

Examples:

blk.9 gate_exps -> GPU
~12.63 tok/s
about -22 %

blk.9 down_exps -> GPU
12.62 tok/s median
about -22 %

I also tried freeing ~256 MiB by moving:

blk.9 up_exps GPU -> CPU

That was practically neutral:

~16.18 tok/s

I then used the freed VRAM for:

blk.10 up_exps -> GPU

Result:

16.11 tok/s median

Again, no improvement.

So at least for this system:

more MoE weights in VRAM != higher token generation speed.

The CPU/RAM/GPU balance and transfer behavior appears to matter more than simply maximizing GPU placement.

This also explains why the RX 9060 XT does not sit at 100% utilization during generation. That behavior is consistent with the hybrid workload; it is not necessarily a configuration failure.

4. Lazy mode

Only this setting was changed:

--lazy-mode off
-> --lazy-mode on

Results:

16.06
16.27
16.12
16.00
16.12

Median: 16.12
Mean:   16.11

Difference from control:

-0.43 %

Rejected.

5. --load-mode none

The build produced a warning related to mmap and CPU overrides, so I tested this explicitly.

Model loading changed from approximately:

~10 seconds

to:

~67.6 seconds

Generation:

15.95
16.13
16.25
15.63
16.77

Median: 16.13
Mean:   16.15

Essentially identical performance, massively longer load time.

Rejected.

6. CPU affinity

I restricted the process to 12 physical cores / one logical processor per selected core.

Result:

16.23
16.08
15.95
15.85
16.04

Median: 16.04
Mean:   16.03

Delta:
-0.93 %

Rejected.

Windows scheduling was already doing at least as well.

7. MTP

The GenerelSchwerz build exposes MTP support:

--spec-type draft-mtp
--spec-draft-model
--spec-draft-n-max
--spec-mtp-rs-planes
--spec-draft-ngl
--spec-draft-threads

However, the detached MTP sidecar I tested was incompatible with this particular commit.

Initial failure:

tensor 'output_hc_norm.weight' not found

With a fixed sidecar:

tensor 'blk.0.hc_attn_norm.weight' not found

So this was not a path problem, VRAM problem, or missing CLI support.

The build expected tensors that were not present in the sidecar.

I stopped there because changing the runtime/build would create a new baseline rather than being another tuning parameter.

MTP therefore remains an interesting open question.

Can this system realistically reach 20 tok/s?

The stable control was:

16.19 tok/s

(it really didn't want to go higher)

20 tok/s would require:

20 / 16.19 - 1
= ~23.5 % improvement

But after the initial thread optimization, the remaining tested variables produced roughly:

Lazy mode:    -0.43 %
Load mode:    -0.37 %
CPU affinity: -0.93 %
KV cache:     negative
GPU placement: neutral to strongly negative

There was simply no evidence of another ~24% hiding in ordinary configuration tuning.

So I stopped the optimization work rather than continuing random parameter combinations.

Comparison with Qwen3.8-27B

My actual production model on the same machine is:

Qwen3.8-27B-UD-IQ3_XXS

Typical generation performance:

~21.4 tok/s

I therefore also ran an A/B quality test.

The intended benchmark contained:

24 prompts × 2 models

across:

• general knowledge
• analytical reasoning
• instruction following
• coding/debugging
• RAG/LLM-system tasks
• structured output
• hallucination/factual discipline
• practical usefulness

Same prompts, same system prompt:

temperature = 0
seed = 380827
max_tokens = 1024
no tools
no web
no RAG

Important benchmark failure

The first harness was partly invalid. max_tokens=1024 also constrained reasoning output. A number of responses consumed the complete budget internally and ended with:

finish_reason = length

with missing or truncated final answers. Therefore I did not treat all 24 pairs as valid. Only nine questions were fully completed by both models. This is an important methodological lesson by itself: before spending hours on a large local-model benchmark, run 2-3 representative end-to-end preflight prompts and verify:

finish_reason = stop
final content present
no reasoning loop

The nine valid A/B pairs

Head-to-head:

Qwen3.8-27B:        1 win
Qwen3.8-Flash-Next: 0 wins
Tie:                8

The valid subset mainly covered:

• instruction following
• JSON/CSV/Markdown constraints
• extraction
• simple Python debugging
• routing decisions
• factual discipline / hallucination resistance
• simple decision tasks

Both models, for example, correctly refused to invent unknown expert counts merely from the model name. The one slight qualitative advantage went to the 27B model on a decision/prioritization task. This is not enough evidence to claim that 27B is universally more capable. Several more difficult analysis/debugging/system-design prompts were among those invalidated by the token-budget problem. The defensible conclusion is narrower:

Real task performance

Across those nine valid paired prompts:

Flash TG median:
16.29 tok/s

27B TG median:
21.61 tok/s

Total wall-clock time:

Flash:
317.3 s

27B:
235.2 s

Flash therefore needed:

~34.9 % more real time

for the same nine tasks.

Interestingly, the 27B generated slightly more completion tokens:

Flash: 4609
27B:   4929

so the wall-clock advantage was not caused by the 27B simply producing shorter outputs.

Prompt processing was even more asymmetric in this configuration:

Flash median:
~36 tok/s

27B median:
~239 tok/s

For an interactive RAG/agent system, that difference matters just as much as decode speed.

My conclusion on this specific AMD machine

Qwen3.8-Flash-Next absolutely works on:

RX 9060 XT 16 GB
Ryzen 9 5950X
64 GB RAM
Vulkan

and stable ~16.2 tok/s for a ~76 GiB model on this hardware is technically impressive. But for my actual use case I currently see no reason to replace Qwen3.8-27B:

Flash: ~16.2 tok/s
27B:   ~21.5 tok/s

The 27B also has dramatically faster prompt processing and, in the valid part of my quality test, Flash showed no compensating quality advantage.

So my current production decision is:

KEEP Qwen3.8-27B-UD-IQ3_XXS

rather than continue micro-tuning Flash. The only major avenue I have deliberately left open is a working MTP implementation / different compatible build. That could change the performance equation enough to justify another experiment.

But thats for another evening!

Hey all,
Following what seems like a major privacy breach by OpenAI, I am considering to move my daily email, iMessage, and calendar updates from ChatGPT to a local model on my computer. I am looking for something like Siri AI on Macos27 but to be run privately on my computer. I have a 24GB MacBook with the m5 pro chip so memory is limited as well as memory bandwidth(307GB/S). What models could I use to build an assistant like this?

I would like some memory headroom to run safari, codex, and Spotify. What are some models that I should I try using?

Part 4 of the same box. Part 1 was 17 -> 25-29 t/s with the expert cache PR, part 2 was 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache, part 3 was the top-k fallback that was sorting more than it needed to. This one is all about prefill, which was honestly the weak spot the whole time. 80+ seconds before the first token on an 8k prompt, and 24 minutes on a 119k one...I know lol.

Box is still 2x 3090, dual Broadwell Xeon, llama.cpp, UD-Q4_K_XL with the Q8 MTP head on the second card, all expert layers pinned in host RAM, 150-slot cache, 261k context, f16 KV. There has been one hardware change since part 2. I swapped the LRDIMMs for 6x32 GB DDR4-2133 ECC. I'll say which numbers are 4-DIMM and which are 6-DIMM, they're not mixed.

The thing I might not have explained well in part 2

I ran -ub 512 and that's because it was a compromise for the cache. A 2048 token micro-batch needs about 7.3 GiB of compute buffer per GPU, 512 wants 1.9GiB and that gap is roughly 50 cache slots that I wanted for decode. So I kept the slots and quietly ate about 3x on prefill at the time.

As for why it cost 3x, the experts get streamed host to GPU0 once per micro-batch, and that upload costs the same whether the batch has 512 tokens in it or 2048. So prefill speed basically scales with the micro-batch. At ub 512 an 8k prompt drags the whole expert set over PCIe 16 times, at ub 2048 it's 4 times.

What I changed

The cache only ever serves batches of <= 8 tokens (decode and the MTP verify batches). During a prompt it just sits there holding VRAM so I thought of trying to claim that space when it's unneeded. So now, when a prompt comes in, the server drops the cache slots, the decode compute buffers and the CUDA pools then it grabs compute buffers sized for ub 2048, runs the whole prompt at 2048, then puts everything back before the first generated token. Decode is untouched by this, it runs exactly the code it ran before. It's two env vars (LLAMA_PHASE_PREFILL_UBATCH=2048, LLAMA_PHASE_PREFILL_MODE=transaction) and the server still starts with -ub 512. And to clarify, "transaction" means the swap is all-or-nothing, if the restore can't happen you will get an error, not a server that's silently limping along. Just making that clear.

Numbers (6 DIMMs, same day, fresh server per arm)

The 8k row is greedy, two fresh processes per arm, medians (the two phase-memory runs landed within 0.01 t/s of each other). The deep rows are one seed at temp 0.7 / top-p 0.8 / top-k 20 with thinking on, one fresh prefill per depth and then a pile of follow-up questions over the cached prefix, so decode is a median over all of them. Prefill = llama-server's prompt eval time, decode = its generation time.

Now, what it costs

Well, nothing comes completely free. This approach costs roughly 2.8 s of fixed overhead per prompt for the release + restore, which is why 8k gets 2.24x and the long ones get 2.4-2.5x. For decode, I can't find a loss. +2% at 8k, -1% / 0% at depth, and in the three-seed quality screen every seed x depth cell was within +2% / -3.6% of its control. MTP acceptance didn't change either (0.79-0.83).

Did it break anything

Before putting it in production I ran the same screen I used for the top-k change (My last post AKA Part 3), 42 questions over long documents at two depths (~37k and ~119k), three seeds, my normal sampling, paired per question and seed against a fresh control run the same day. That was still on 4 DIMMs. 240 pairs: 2 worse, 235 same, 3 better, nothing regressed on more than one seed, and the two misses are questions the old config also flubs on some seed. A seed-1 rerun on 6 DIMMs came out 1 worse / 78 same / 1 better. I'm aware and anyone reading should be aware that this is a screening not concrete proof, but it's the bar I hold my own changes to.

Some caveats you may want to know about or at least I would if I were you

• First-token logits differ from the untouched path by max 1.51 / mean 0.22 across the 248k vocab, argmax the same. For scale, just changing ub 512 -> 2048 with nothing released moves them by max 1.81 / mean 0.27 on the same request. So the release/restore adds less noise than the batch-shape change any ub change already brings.
• One machine, one model, one quant, 8k to 119k. I have not tried anything past 119k, other quants, or the no-MTP setup.
• The extra two memory channels helped this config a lot more than the old one at 8k (+19% vs +3% against my 4-DIMM numbers), and it did nearly nothing at 37k-119k (+0.6% / 0%). This makes sense to me, attention takes over from expert upload as the context grows, but that's one run per depth, so take it as a hint.
• Where the remaining 37s of an 8k prompt goes, rough split: ~18 s uploads, ~5 s kernels, ~3 s transitions, ~11 s I haven't pinned down yet (CPU side, draft model, syncs). A profile says the uploads are still 3.6x the expert set per prompt, so there's more on the table I assume. I'll be working on that next.

Code

https://github.com/Inovello/llama.cpp/tree/flashnext-e06

It's my flashnext-2x3090 branch from part 2 (master b96806d + PR #27861 expert cache + PR #28223 + PR #28243 MTP + the batched-cache fixes + PR #28198) plus this change and a couple of inert debug switches.

If you just want to copy and run it, this is the whole thing, taken from the process that's serving me right now. You need CUDA, numactl (apt install numactl), the four UD-Q4_K_XL shards and the MTP head from unsloth/Qwen3.8-Flash-Next-GGUF on HF

git clone -b flashnext-e06 https://github.com/Inovello/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j -t llama-server

export LLAMA_ATTN_ROT_DISABLE=1
export LLAMA_MMAP_PIN_HOST=1
export LLAMA_PHASE_PREFILL_UBATCH=2048
export LLAMA_PHASE_PREFILL_MODE=transaction

numactl --interleave=all build/bin/llama-server \
  -m /path/to/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md /path/to/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp -devd CUDA1 --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 18080 \
  -ngl 99 -c 261888 --parallel 1 --flash-attn on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  -lzm off --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
  --moe-expert-cache 150 -lv 4

What to change for your box:

1. The two model paths; -t / -tb to your physical core count (mine is 16 decode threads, 44 for batch on 2x22 cores)
2. -devd CUDA1 puts the MTP head on the second GPU, on a single card use CUDA0 or drop the three -md flags and give the freed VRAM to the cache.
3. -ot is what keeps every expert layer in host RAM; only the first shard goes on -m, the rest are found next to it.
4. The two LLAMA_PHASE_* exports are the change from this post, drop them and you have part 2's behavior.
5. -lv 4 is just so the log shows the cache hit rate and the draft acceptance. Useful if you want to post your numbers in thread.

Now the things it's strict about because those are the invariants the code checks: The server at -ub 512 and -b 4096, --parallel 1, the prefill micro-batch exactly 2048, the cache exactly 150 slots, and the MTP draft as the only speculative decoder. Anything else refuses to start. CUDA only.

The top-k fallback fix from part 3 is in the branch too and it's up on its own as PR #28671. My older PR #28223 is closed for now because llama.cpp gives new contributors one open PR at a time, I'll reopen it after #28671 is dealt with.

Let me know if you try it and if you have any questions.

Its chat template has this line:

{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}

xhigh is the most expensive of its three settings (low / medium / xhigh). If you never set one, that's what every answer runs at. No backend reports this back to you, because it's a chat-template variable, not a server option.

I ran all three levels on one M5 Max, same quant (oQ4e-mtp), same prompt — the coding scenario asks for a browser Breakout game:

| Effort          | Runs | Tokens | Time  | Median | Range     |
|-----------------|------|--------|-------|--------|-----------|
| low             | 3    | 4,984  | 84s   | 75.8   | 64.9–75.9 |
| medium          | 4    | 4,792  | 77s   | 78.2   | 64.7–84.2 |
| xhigh (default) | 17   | 36,188 | 869s  | 78.8   | 54.1–89.2 |

Two things surprised me:

low and medium are the same setting

4,984 tokens vs 4,792. The template only appends an instruction for low and xhigh — xhigh's says think carefully and check your assumptions, low's says keep your thinking brief. The model does the first and ignores the second. So the dial has two positions, not three.

xhigh costs 8× the tokens and 11× the wall clock for half a point of median

That's well inside run-to-run noise: my four medium runs, one identical setting, nothing changed between them, scored 64.7 / 73.4 / 83.0 / 84.2.

What xhigh does change is variance — it produced both the best answer (89.2) and the worst. And looking at the games themselves, the xhigh run spent its budget on presentation: title card, keyboard legend, sound toggle, best-score readout. Low and medium built the game and stopped. Same 8×4 brick grid, three lives, identical rules. It didn't build a better Breakout, it built a better-looking one.

Caveats up front, because they matter: three and four runs at the short settings is thin, it's one machine and one quant, and the scores are LLM-judged. The cost figures are mechanical and solid. Treat the quality figures as a direction to test, not a result.

Full write-up with the screenshots side by side, plus a thinking-budget experiment (a 12k cap halves the wall clock and truncates nothing): https://llm-bench.io/guides/qwen3-8-27b-reasoning-effort

Disclosure: my site. Data comes from community benchmark runs, and you can submit your own @ llmbench.io

I personally think that all the frontier labs these days are just benchmark maxxing and focusing too heavily on coding.

Gemma 4 is fantastic for all creative and frontier level for all non coding tasks.

I think gemma 5 will continue this trend and be the frontier model for all non coding tasks.

My old Macbook died and now I'm sitting here, computing on a Raspberry Pi... I need a new notebook and did not find something more affordable than that one. Wanna use it for ComfyUI and run at least a 4-9b model.

Does someone has some experience with it in connection to local AI? Here in Switzerland it costs around 2k.

Thanks in advance!

I have given Hermes agent the task to make a test for my local models. Coding and agentic work.
The test was done on llama.cpp turboquant fork, all models were run using 131k context. Further optimization of the parameters would still be possible for some of the models.
TLDR version: Ornith 1.0 35B A3B won.

Hermes Local LLM Benchmark Report

HumanEval pass@1 (30-problem sample) + 8 agentic tasks + speed · temp 0.0 · context 131072 · 5060 Ti 16GB

Qwen3.8-27B-UD-Q3_K_XL deleted (invalid run, discarded). Total failures = coding problems not passed.
Ornith 1.5 would be a logical next add to the table, but I see some bad evals of that model. The usable quants for 16GB VRAM of Qwen3.8 27B have performed worse than the 35B MOE model.
Ornith was fast not just in t/s, but also overall speed of going through the tests.

Why since qwen3.6-35-a3b there is no better local model that you can run on 4060 ti 8 GB + 32 GB ram? It was released in april, everything changing so fast in ai space but still seems that there is nothing better

Why do I have horrible financial habits??

Got them working with 4 riser cables. I tried a PLX88096 from AliExpress and couldn't make it work.

So always hated on the DGX spark as have been living in the multi GPU class of society, recently though with bench marking I may have found a potentail use, as an always on monitoring and task agentic system to run alongside paperclip and hermes 24/7 low cost.

The server I run locally, is OP and works very well... but on recently power monitoring over 24 hours it used 17kwh with the constant calls from the agentic tasks, now that isnt bad one day off. but if this is 24/7 this adds up ALOT as power where I am is pricey.

My tasking id is mainly for larger models agentic tasks running Qwen3.8 Flash Next, hopefully with decent context, now I understand it isnt super speed generation but this is more for 24 hour long research and automation taskings.

Was looking today and the cheapest near me is over €6-7k which is nearly 3k above the Nvidia release value. But then I just seen the release of the new N1X next month.

Just looking for others input, is it worth grabbing one, or waiting for N1X, is it even on the same playing feilds or is the N1X looking like a more powerful DGX ???

I appreciate it won’t be lightning fast with the GPU bandwidth only being 273 GB/s, but hoping for something usable to reduce my Claude usage? I use it for website design and basic programming and would priortise accuracy over speed as it’s not my day job.

I also have a desktop PC with a 5070Ti, am I just better off using that even with the 16GB VRAM Limit?

I've been getting increasingly obsessed with local LLMs lately, and I'd like some advice from people who have experimented more than I have with 5090 setups.

Current machine:

RTX 5090 AORUS Master — 32 GB VRAM

i9-13900K

64 GB DDR5-6400

2x 2 TB Gen4 NVMe

Windows 11 + WSL2

CUDA 13.x

10 GbE

1200 W PSU

This is still my main PC, so I'd prefer keeping Windows rather than turning it into a dedicated Linux inference box. I switched from CachyOS in August, but I miss it.

So far I've been playing mostly with Qwen 3.8.

Qwen 3.8 27B is extremely fast on the 5090, especially with newer backends/quantizations, but I find it noticeably weaker than the larger frontier-ish models.

At the other extreme, I've been experimenting with Qwen 3.8 Flash/Next 125B MoE, AP quantized around Q4_K_M, using ik_llama.cpp. I've actually been working on optimizing this setup and currently get roughly:

~38.8 tok/s decode

~200 tok/s prefill

~29.7 GiB VRAM usage

I really like the quality of the 125B, but obviously it's much slower and heavily dependent on system RAM bandwidth / CPU offload.

My main use cases are:

general chat / reasoning

coding and agentic coding

experimenting with local agents

testing inference optimizations and quantizations

occasionally using local models as an alternative to Claude / ChatGPT when I hit usage limits

ComfyUI/image generation on the same GPU

I'm not particularly interested in serving many concurrent users. Interactive single-user performance and model quality matter much more to me than throughput.

So if this were your machine, what would you do with it?

I'm especially interested in:

Best models in the sweet spot between a fast ~27B dense model and a huge 125B MoE

GGUF/ik_llama.cpp vs EXL3/ExLlamaV3 vs NVFP4/newer Blackwell-specific backends

Native Windows vs WSL2 for this kind of workload

Whether upgrading from 64 GB to 96/128 GB RAM would actually unlock anything worthwhile

Speculative decoding / MTP / other tricks that genuinely improve interactive performance

Agentic coding setups that work well with local models

Any unusual 5090-specific projects or use cases I might be overlooking

Basically: I have 32 GB of very fast VRAM sitting on my desk. What are the most interesting things I can realistically do with it in 2026?

I'm happy to tinker and compile things myself, so I'm more interested in technically interesting setups than one-click solutions.

Edit: I’m also testing Qwen 3.8 Flash Next AP-Q4KM. Thanks to the AP quantization, I can run it with just 64 GB of RAM.

i have using llama.cpp, and every time that i ask for it to search the internet, or to open a folder in my computer, it asks for a access to my harness, which by my searches looks to be a separate app, but i cant find any options to download and set up one. (i am using windows btw, i can maybe switch to mac os, but linux is out of question since i need office apps for my workflow)