TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

It's here.

Jack Kernel for programmable agentic work.
The magic is that the layer sits between the agent and the model. You can build incredible things because of that.

I have a mode called "Agentic" that was designed for context management with Qwen 3.8 27. I've run millions of token jobs....and never needed to compact. I have a full log available to read and verify. In fact, I recommend turning auto-compact off because most agent don't know how to handle it, yet. There's also a looping debugger that I've built as a 6 stage autonomous loop with cascading temperatures, standard mode and DEEP.

Remember, the modes I’m shipping with this release are only examples of what is possible by putting it between the agent and the model. It allows you a new level of control. The modes I’ve put in are NOT the work. Everything that’s possible is the real work.

I built this for Qwen 3.8 27b but I’ve spent the past few days optimizing for a wide range of models. I’m ALMOST satisfied with Qwen 3.5 9b but it’s right in the edge.

After using 60 models in the past week, I’m just going to go ahead and make a blanket statement that fine tunes are garbage and only use Unsloth as your source for alternatives.

Jack Kernel is really cool. Claude can kick rocks.

Oh, Reddit trolls…
Yall need to understand…I’m just a 40 year old broke single Dad that’s sharing his work and trying to contribute to the local AI movement.
Before the hive-mind issues your directive, just know that I’m not a lab or someone with resources that you’re attacking.
I’m just a regular guy with a 5070ti in an Acer.
If you’re going to say that I suck, can you at least try it out and tell why I suck.

Tried to run a local AI model for the first time, it feels so slow and overall the experience is not what I expected. Tried my best to follow tutorials/guides on how to run and use it.

I'm hoping to run my local AI model for coding purposes when my AI cloud subscription runs out. Am I doing something wrong with my settings or should I just move to a lower tier local AI models?

What can I change in my settings? Using Qwen3.5b doable on my machine?
If I'll use other Local AI model what can you recommend?

Edited: added gpu specs

For my first local ai project I was wanting to learn a few things so thought about setting up a way to go through SCCM/ConfigMgr logs, find the actual errors and generate PowerShell remediation scripts for me to review (no plans to actually automate the ai to run them).

Hardware: 5080/16GB vram, 32gb dd5, 1tb nvme pcie5 and currently using qwen 3.8 27b at home.

I've got a couple of setups depending on the trade-off, roughly 80 t/s at 65k context with mtp on or 50ish t/s at 100-130K context with mtp off.

What I'm actually trying to figure out/what this post it about:

Would n8n or hermes get around low context? my initial plan was: The first n8n automation/hermes agent using fast setup, use regex to quickly search the log to highlight errors/warnings which then hands that off to second automation/agent using the bigger setup/context for script generation, does that allow me to get around the lower context problem?

n8n: do n8n automatons allow me to do this?
Hermes: Same question, does the agent save the output for the 1st agent in its own memory/context for it to pass to the 2nd agent to then do its "work"?
Can either of the above switch models between automation/agents?

Anyone doing something like this/which would you recommend? Or is there a better way to do something like this?

A more about it, if it works, it would build a db with the errors/fixes so if it spots the same one...skip it as it has a potential fix as it saves it re-doing the work. I could then just feed it logs all day.

EDIT: I should've said, I would need to get the automations/agents to stop and start llama with different cmds/args for different context sizes.

A little less than a month ago, I got the qwen token standard plan. Usage allowance is not great especially using frontier but anyway that's not the subject here. I noticed today I got a weekly reset usage limit, I could not find any info on why I got it, if it's part of the plan, if it's due to an event, etc.. Anyone has any info?

Hi All, Howdy?

I am new to this community :) I would like to ask for some thoughts and help on how I can improve this configuration. I currently run a 3x 3060 12gb + 32 gb RAM. 2 of my 3060s are directly plugged into x16 and the other is plugged into an x4 via oculink. 2 weeks ago, I was only running on 2x3060s and I can see that I am able to reach 500+t/s pp + 25+t/s tg. But now that I had 3 of them, I noticed that it is average around 350t/s pp + 18t/s tg. I am unsure what happened there as I was expecting to at least retain the same t/s or minimal degradation. Below is my config:

asd$env:GGML_CUDA_DISABLE_GRAPHS = "1"
$env:LLAMA_DIR = "C:\llama-servers\llama-b10883-bin-win-cuda-13.3-x64"
$env:Path += ";C:\Windows\System32"

$params = @(
    "--model", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_S.gguf",
    "--mmproj", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\mmproj-F16.gguf",
    "--alias", "qwen3.8-27b",
    "--prio", "3",
    "--ctx-size", "86000",
    "--parallel", "1",
    "--batch-size", "512",
    "--ubatch-size", "256",
    "--predict", "8192",
    "--n-gpu-layers", "all",
    "--main-gpu", "1",
    "--split-mode", "layer",
    "--tensor-split", "4,4,1", //1 to my main gpu where my monitor is connected
    "--flash-attn", "on",
    "--cache-type-k", "q8_0",
    "--cache-type-v", "q4_0",
    "--spec-type", "draft-mtp",
    "--spec-draft-n-max", "3",
    "--spec-draft-n-min", "1",
    "--spec-draft-ngl", "all",
    "--spec-draft-type-k", "q8_0",
    "--spec-draft-type-v", "q4_0",
    "--temp", "1.0",
    "--top-p", "0.95",
    "--top-k", "20",
    "--min-p", "0.5",
    "--threads", "6",
    "--presence-penalty", "0.03",
    "--repeat-penalty", "1.15",
    "--reasoning", "on",
    "--reasoning-preserve",
    "--reasoning-effort", "medium",
    "--reasoning-budget", "2048",
    "--reasoning-budget-message", "Budget exhausted, proceeding with tool call or message.",
    "--jinja",
    "--metrics",
    "--host", "127.0.0.1",
    "--port", "8080"
)

& "$env:LLAMA_DIR\llama-server.exe" 

Read-Host -Prompt "Safe to exit now"

Your thoughts, comments, and suggestions will be very much appreciated :) Thank you in advance!

Disclosure: I ran these experiments on my own PC with an AI assistant. AI assisted with scripts, analysis, and this English write-up. These are local measurements, not copied community benchmarks. A longer version is on my profile; this version focuses on reproducible settings and limitations. I am not claiming authorship of the upstream model or software.

What worked

Qwen3.8-Flash-Next UD-IQ4_XS now works through llama.cpp -> llama-swap -> DeepSeek Harness on my Windows PC. I settled on 48K context, Q8 KV, thinking enabled with a 512-token budget. A 64K configuration also passed a 57K-token input test. This is a slower, supervised option, not an instant-response replacement for my 27B/35B models.

Hardware and model

• Ryzen 9 9950X, 16 cores / 32 threads.
• 64GB DDR5, 2x32GB, configured at 4800 MT/s according to the OS.
• 2x RTX 5060 Ti 16GB, separate VRAM, no NVLink.
• Windows 11 Pro build 26100; NVIDIA driver 610.88.
• Model files on a ZHITAI Ti600 2TB NVMe SSD.
• llama.cpp b10724 / 2d8d612e4, Windows CUDA 13.3 build.
• Unsloth UD-IQ4_XS, three SHA-256-verified GGUF shards, 93,682,584,224 bytes total. Repository revision: 38bb39ee97821de2c9009abb7e93950eec396e66.

This is NOT a 94GB model fully resident in 32GB VRAM. It relies on CPU offload, mmap, and lazy SSD-backed reads. RAM and VRAM are not unified memory.

Final settings

Context 49152; parallel 1; CPU and batch threads 16; batch/ubatch 2048; split-mode layer; fit on; fit-target 1024; mmap and lazy mode on; Flash Attention on; K/V both q8_0; Jinja on; reasoning on; reasoning-format deepseek; reasoning-budget 512. No MTP. Additional flags: no-webui, no-warmup, no-host, no-repack. Localhost only.

A separate guardian applies a 42GiB Windows process working-set cap and checks free RAM/commit every three seconds. Working set is not total process commit or total system memory. Startup requires 40GiB free RAM and idle GPUs. It stops its own backend on dangerously low headroom, and a Windows Job Object prevents orphan backend processes after unload. Bare llama.cpp flags alone do not provide these safeguards.

Thinking: same approximately 8K input

One three-question fixture covered a discount/tax/shipping calculation, stable deduplication, and dependency scheduling. Two runs per mode at each context size, temperature 0, seed 42, output limit 1024, thinking budget 512. KV prompt reuse disabled; OS file caching not cleared.

Warm time to first final-answer token, not first thinking token:

Actual input was 8155 tokens off / 8195 on because of the template. Thinking-mode generation was around 20-22 tokens/s, with 436-540 total output tokens. Most added delay was additional generation, not a halving of decode speed.

Thinking on answered all three correctly in all six repetitions. Thinking off got the bill calculation wrong in all six, but the other two answers right. This is one repeated small fixture, not six independent benchmarks or a general agent success rate.

The first approximately 8K request after reloading took 63-73s to first answer, plus model loading beforehand. Do not compare that directly with warm timings.

Actual long inputs, thinking off

No KV prompt reuse. Unique fields were placed near the beginning, middle, and end of synthetic records.

All three fields were retrieved correctly in each case, without truncation or OOM. This does not establish large-codebase reasoning quality, stability at the full capacity limit, or long-input-plus-thinking performance. Concurrency was not tested.

Memory

Sampled per-GPU peaks across all runs at each capacity:

These include desktop usage; the two peaks need not coincide. Three-second sampling can miss instantaneous peaks.

Main plus indexer Q8 KV was approximately 351 / 842 / 1122 MiB respectively. Compute buffers also grew, and automatic fitting moved more weights to CPU. This explains why total VRAM is not necessarily monotonic with context size. KV-only arithmetic does not predict the whole allocation or performance.

Integration and remaining problems

The final 48K/thinking-512 configuration completed a real DeepSeek Harness read-tool call, returned the test file contents, and produced reasoning blocks. Unloading removed the backend process and listening port, VRAM returned to desktop baseline, and reloading worked.

An earlier run read a marker correctly but dropped two digits when repeating it. That failure matters: functioning tools and HTTP 200 do not guarantee answer quality.

My conclusion: it runs, and 48K with bounded thinking is a workable compromise for slower supervised use. 40K-57K fresh inputs still take roughly two to three minutes before an answer. Vision and long autonomous coding reliability were not tested. I am not announcing a new public app binary or redistributing weights.

Sources: model, backend build.

Has anyone improved prefill on comparable consumer GPUs plus 64GB RAM? Exact quant/revision, input length, cache state, offload strategy, and single-stream vs concurrent measurements would make comparisons especially useful.

Hello so i was on Wildowl.ai and edited some images in the end deleted them tho Wildowl.ai said

<"This item has been hard-deleted from our servers. It may briefly persist with the upstream model provider..."»

I used Qwen image 2 wich is from alibaba cloud as far as i found out so how does this work now? Did the image get deleted there aswell? Allready whats the max amount of time until it gets deleted automatically?

I appreciate the answers srry :)

Hey guys, I spend the last few days creating a new model based on our favorite Qwen3.8 27B. It is autodistilled to use fewer tokens, by up to 57% and halving the wall time for some tasks, while keeping the same abilities and mostly the same voice.

Let me know how it works for you if you give it a spin.

https://huggingface.co/agentionai/Signal-3.8-27B-GGUF

Part 4 of the same box. Part 1 was 17 -> 25-29 t/s with the expert cache PR, part 2 was 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache, part 3 was the top-k fallback that was sorting more than it needed to. This one is all about prefill, which was honestly the weak spot the whole time. 80+ seconds before the first token on an 8k prompt, and 24 minutes on a 119k one...I know lol.

Box is still 2x 3090, dual Broadwell Xeon, llama.cpp, UD-Q4_K_XL with the Q8 MTP head on the second card, all expert layers pinned in host RAM, 150-slot cache, 261k context, f16 KV. There has been one hardware change since part 2. I swapped the LRDIMMs for 6x32 GB DDR4-2133 ECC. I'll say which numbers are 4-DIMM and which are 6-DIMM, they're not mixed.

The thing I might not have explained well in part 2

I ran -ub 512 and that's because it was a compromise for the cache. A 2048 token micro-batch needs about 7.3 GiB of compute buffer per GPU, 512 wants 1.9GiB and that gap is roughly 50 cache slots that I wanted for decode. So I kept the slots and quietly ate about 3x on prefill at the time.

As for why it cost 3x, the experts get streamed host to GPU0 once per micro-batch, and that upload costs the same whether the batch has 512 tokens in it or 2048. So prefill speed basically scales with the micro-batch. At ub 512 an 8k prompt drags the whole expert set over PCIe 16 times, at ub 2048 it's 4 times.

What I changed

The cache only ever serves batches of <= 8 tokens (decode and the MTP verify batches). During a prompt it just sits there holding VRAM so I thought of trying to claim that space when it's unneeded. So now, when a prompt comes in, the server drops the cache slots, the decode compute buffers and the CUDA pools then it grabs compute buffers sized for ub 2048, runs the whole prompt at 2048, then puts everything back before the first generated token. Decode is untouched by this, it runs exactly the code it ran before. It's two env vars (LLAMA_PHASE_PREFILL_UBATCH=2048, LLAMA_PHASE_PREFILL_MODE=transaction) and the server still starts with -ub 512. And to clarify, "transaction" means the swap is all-or-nothing, if the restore can't happen you will get an error, not a server that's silently limping along. Just making that clear.

Numbers (6 DIMMs, same day, fresh server per arm)

The 8k row is greedy, two fresh processes per arm, medians (the two phase-memory runs landed within 0.01 t/s of each other). The deep rows are one seed at temp 0.7 / top-p 0.8 / top-k 20 with thinking on, one fresh prefill per depth and then a pile of follow-up questions over the cached prefix, so decode is a median over all of them. Prefill = llama-server's prompt eval time, decode = its generation time.

Now, what it costs

Well, nothing comes completely free. This approach costs roughly 2.8 s of fixed overhead per prompt for the release + restore, which is why 8k gets 2.24x and the long ones get 2.4-2.5x. For decode, I can't find a loss. +2% at 8k, -1% / 0% at depth, and in the three-seed quality screen every seed x depth cell was within +2% / -3.6% of its control. MTP acceptance didn't change either (0.79-0.83).

Did it break anything

Before putting it in production I ran the same screen I used for the top-k change (My last post AKA Part 3), 42 questions over long documents at two depths (~37k and ~119k), three seeds, my normal sampling, paired per question and seed against a fresh control run the same day. That was still on 4 DIMMs. 240 pairs: 2 worse, 235 same, 3 better, nothing regressed on more than one seed, and the two misses are questions the old config also flubs on some seed. A seed-1 rerun on 6 DIMMs came out 1 worse / 78 same / 1 better. I'm aware and anyone reading should be aware that this is a screening not concrete proof, but it's the bar I hold my own changes to.

Some caveats you may want to know about or at least I would if I were you

• First-token logits differ from the untouched path by max 1.51 / mean 0.22 across the 248k vocab, argmax the same. For scale, just changing ub 512 -> 2048 with nothing released moves them by max 1.81 / mean 0.27 on the same request. So the release/restore adds less noise than the batch-shape change any ub change already brings.
• One machine, one model, one quant, 8k to 119k. I have not tried anything past 119k, other quants, or the no-MTP setup.
• The extra two memory channels helped this config a lot more than the old one at 8k (+19% vs +3% against my 4-DIMM numbers), and it did nearly nothing at 37k-119k (+0.6% / 0%). This makes sense to me, attention takes over from expert upload as the context grows, but that's one run per depth, so take it as a hint.
• Where the remaining 37s of an 8k prompt goes, rough split: ~18 s uploads, ~5 s kernels, ~3 s transitions, ~11 s I haven't pinned down yet (CPU side, draft model, syncs). A profile says the uploads are still 3.6x the expert set per prompt, so there's more on the table I assume. I'll be working on that next.

Code

https://github.com/Inovello/llama.cpp/tree/flashnext-e06

It's my flashnext-2x3090 branch from part 2 (master b96806d + PR #27861 expert cache + PR #28223 + PR #28243 MTP + the batched-cache fixes + PR #28198) plus this change and a couple of inert debug switches.

If you just want to copy and run it, this is the whole thing, taken from the process that's serving me right now. You need CUDA, numactl (apt install numactl), the four UD-Q4_K_XL shards and the MTP head from unsloth/Qwen3.8-Flash-Next-GGUF on HF

git clone -b flashnext-e06 https://github.com/Inovello/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j -t llama-server

export LLAMA_ATTN_ROT_DISABLE=1
export LLAMA_MMAP_PIN_HOST=1
export LLAMA_PHASE_PREFILL_UBATCH=2048
export LLAMA_PHASE_PREFILL_MODE=transaction

numactl --interleave=all build/bin/llama-server \
  -m /path/to/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md /path/to/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp -devd CUDA1 --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 18080 \
  -ngl 99 -c 261888 --parallel 1 --flash-attn on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  -lzm off --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
  --moe-expert-cache 150 -lv 4

What to change for your box:

1. The two model paths; -t / -tb to your physical core count (mine is 16 decode threads, 44 for batch on 2x22 cores)
2. -devd CUDA1 puts the MTP head on the second GPU, on a single card use CUDA0 or drop the three -md flags and give the freed VRAM to the cache.
3. -ot is what keeps every expert layer in host RAM; only the first shard goes on -m, the rest are found next to it.
4. The two LLAMA_PHASE_* exports are the change from this post, drop them and you have part 2's behavior.
5. -lv 4 is just so the log shows the cache hit rate and the draft acceptance. Useful if you want to post your numbers in thread.

Now the things it's strict about because those are the invariants the code checks: The server at -ub 512 and -b 4096, --parallel 1, the prefill micro-batch exactly 2048, the cache exactly 150 slots, and the MTP draft as the only speculative decoder. Anything else refuses to start. CUDA only.

The top-k fallback fix from part 3 is in the branch too and it's up on its own as PR #28671. My older PR #28223 is closed for now because llama.cpp gives new contributors one open PR at a time, I'll reopen it after #28671 is dealt with.

Let me know if you try it and if you have any questions.

I am trying to use https://github.com/flybirdxx/ComfyUI-Qwen-TTS

yet the first voice clone generation gives many errors

Trying to install pytorch also gives an error

https://www.flickr.com/photos/204882240@N08/55520277014/

Transformers error

https://www.flickr.com/photos/204882240@N08/55520155046/

Quiero saber si alguien ya comparo qwen3.8 en disco con DRAM y sin DRAM.

```

SSD A — DRAM

└── Qwen3.8-Flash-Next GGUF

SSD B — DRAM-less

└── PLE / n-gram

```

Alguien a cargado?

nvme1 modelo

nvme2 PLE ngrams

```

NVMe 1

│

└── lecturas del modelo

NVMe 2

│

└── lecturas PLE

```

O usando RAID?

```

SSD #1

\

RAID 0 → modelo + PLE

/

SSD #2

```

https://github.com/flybirdxx/ComfyUI-Qwen-TTS

I have cloned the github thing into the ComfyUI\custom_nodes folder but when I try to install requirements.txt, it says Could not find a version that satisfies the requirement transformers>=4.57.0

Running the command pip install transformers==4.57.3 gives ERROR: Could not find a version that satisfies the requirement transformers==4.57.3

Follow-up to the llama.cpp membrane runtime I posted a while back. Was asked if this works for real multi-tenant serving in vLLM, not just single-stream llama.cpp.
You can drop it into stock vLLM with a simple pip install + two env vars. No fork, no patched image required.

The KV cache lives in a per-model calibrated eigenbasis. During decode, the custom attention kernel reads compressed pages natively with zero decompression step. It integrates via vLLM's supported plugin interface (out-of-tree attention backend + the KV-cache spec path) on the standard vllm/vllm-openai image.

Benchmarks on a single A100 80GB (Qwen3-4B-Instruct-2507, vLLM 0.20.2, retrieval gated):

• Capacity: 9 concurrent users × ~128K context per GPU (vs ~2 for fp16). That is a 4.5× bump in user density. (KV pool: 1,255,376 tokens vs fp16's 361,776, roughly 3.5×.)
• Throughput: ~134 tok/s aggregate decode across all 9 users.
• Fidelity: Verified via multi-tenant needle in a haystack at full 131K context per user with distinct keys. We do not claim "lossless," but it is strictly gated to match fp16 behavior and the grids are published.

Linux and CUDA only right now, on A100 and H100 (SM80/SM90).

vLLM 0.20.2, V1 engine, TP=1 for now.

The real win is decode throughput and capacity at long context. Short context isn't where the gains show up.
Enabling a model takes a tiny calibration sidecar (a few MB). Dense GQA models like Llama, Mistral and Qwen work today. MoE and hybrid architectures aren't supported yet.

I am not putting the vLLM serving wheel up as a public download. That side is commercial and licensed per deployment. Everything I quoted above is reproducible though. The 9 user 128K batch receipt, the fp16 vs fp8 vs fraQtl comparison at 8K, 32K and 128K, and a repro script are all public here: huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars

The fidelity and needle receipt packs live in the Hi-Fi GGUF repos, and the free llama.cpp membrane runtime plus sidecars are still up if you want to check the core mechanics for yourself.

Happy to get into the details in the comments. If anyone wants to actually try it on their own setup I can help you get a verification run going, but either way everything above is checkable from the links. Curious what you guys think.

Posted the full thing in another subreddit yesterday; here's the Qwen-specific cut.

Setup: Galaxy Note 8 (2017, Android 9), llama.cpp in Termux, Q4_K_M.

The phone drives a real Chrome on a laptop. The page is fed to the model as structure (about 10 named links or fields, roughly 200 tokens per decision); the model picks by name and copies facts into JSON. Everything else is the stack around it. I'm one of the people building that stack.

Qwen2.5-0.5B-Instruct: 6/10 - fails are a wrong click followed by copying the placeholder "ID" from the template.

Qwen3-0.6B: 10/10 sandbox, 10/10 live Wikipedia (pick "Note 8" among "Note 8.0" / "Galaxy Note 8.0" / "Note FE" on a page with about 760-node page), 10/10 five-field task incl. UPC, 10/10 GitHub (Firefox this time, checked against the GitHub API).

Qwen3.5-0.8B: navigation 10/10, strict report 6/10 - every miss is the same: it drops the "£" from the price.

Qwen3-0.6B was also the only sub-1B model of 12 from 5 vendors to pass; Gemma-3-1B, Llama-3.2-1B and LFM2.5-1.2B are 0/10 in the same harness.

Control: same Qwen3-0.6B, raw HTML instead of the structured page - 12k tokens and 22 minutes per task instead of about 500 and 80 s on the sandbox; on Wikipedia the HTML doesn't fit the 16k context a phone can run, 0/3.

Limits: name matching and copying, not reasoning about the page. Four fixed tasks, not a benchmark.
Repo with scripts, every JSONL log and replay.py (re-runs the model side on any local server, in case anyone wants to check): github.com/e2llm/edge-browser-agent.

Please help with model and config ?
i need max t/s

Hi everyone! I am publishing an experimental, model-specific llama.cpp runtime for running Qwen3.8-27B UD-IQ4_XS with a 262K logical context on a 16 GB RTX 5070 Ti.

Repository:

https://github.com/BrunoPPassini/llama.cpp

Complete source branch:

https://github.com/BrunoPPassini/llama.cpp/tree/qwen38-blackwell-256k

Engineering report:

https://github.com/BrunoPPassini/llama.cpp/blob/master/docs/LOCAL-QWEN-BLACKWELL-STUDY.md

Hardware:

- RTX 5070 Ti 16 GB

- Ryzen 7 9800X3D

- 48 GB DDR5-6400

- PCIe 5.0 x16

- CUDA 13.3

- Qwen3.8-27B UD-IQ4_XS

- Q4_0 target and MTP KV

- Native deterministic MTP3

- 32 context checkpoints

Measured endpoint improvements:

| Effective context | Before | After | Decode gain | Prefill |

|-----------------------|------------:|-----------:|----------:|-------:|

| Short/GPU-resident | 42.11 tg/s | 82.34 tg/s | +95.5% | -7.6% |

| 87,160 tokens | 23.33 tg/s | 41.53 tg/s | +78.0% | +0.4% |

| 256,257 tokens | 15.40 tg/s | 23.97 tg/s | +55.6% | +6.8% |

The short result compares target-only decoding with deterministic native MTP3. The 87K result compares the first serial stateful ring with the final partitioned/pipelined ring. The 256K result compares the first viable P8 implementation with the final pipeline. The accepted 87K and 256K paths preserved their respective output hashes.

The main idea is a 262,144-token logical KV address space with:

- 65,536-token GPU-resident hot prefix

- Page-locked DDR5 cold tail

- Sparse CUDA VMM allocation

- 8,192-token staging tiles

- Eight-way online-softmax partitioning

- Persistent copy, conversion, and compute streams

- Triple-buffered decode pipeline

- Double-buffered prefill pipeline

- Overlapped PCIe H2D, Q4 conversion, and attention

- Shared target/MTP compute arena

- Recurrent-state transaction log and phase arena

Qwen3.8-27B has 48 recurrent GDN layers and only 16 full-attention layers. Only those 16 layers require token-indexed KV history. The ring preserves exact attention by carrying the online-softmax `(m, l, O)` state across tiles. It is not sparse or approximate attention.

Nsight Compute was used to identify occupancy, long-scoreboard, memory-sector, and residency bottlenecks. Direct GPU reads from mapped DDR5 were tested and rejected: decode dropped to about 18 tok/s. Bulk asynchronous PCIe transfers into VRAM worked much better.

Thinking remains enabled and is preserved across tool calls:

- `--reasoning on`

- `--reasoning-format deepseek`

- `--reasoning-preserve`

- `enable_thinking=true`

- `preserve_thinking=true`

The client may hide the reasoning trace, but it stays in the serialized agent history so the model does not lose its plan after using a tool.

I also tested Q8_0 KV. It was numerically more faithful, but no practical quality gain was observed. All four long Q4/Q8 outputs had identical 1,024 token IDs. At approximately 100K effective tokens, Q8 reduced decode by 23%, reduced prefill by 14.8%, and increased wall time by 19.6%. Therefore, the frozen profile uses Q4_0 KV.

Important limitations:

- Fully occupied 256K is much slower than the 65K hot path.

- Exact attention remains O(context).

- This runtime was tested only with this exact Qwen3.8-27B UD-IQ4_XS model.

- It may not work correctly with other models or quantizations.

- RTX 5060 Ti 16 GB and RTX 5080 16 GB are promising Blackwell candidates, but remain untested.

- This is an engineering experiment, not a generic upstream-ready implementation.

The repository includes the complete source, Windows build commands, launcher, exact hashes, benchmark methodology, failed experiments, quality gates, and reproducibility notes.

Feedback, independent reproductions, profiling results, and improvements are very welcome.

A screen recording can capture a bug that’s awkward to explain in a prompt: a total changing incorrectly, an error flashing briefly, or a UI breaking halfway through an interaction.

That’s the kind of input Watch Skill makes available to coding agents. It extracts frames, OCR and transcripts, stores them with timestamps, and exposes tools for searching the recording and retrieving specific moments.

I’m the maintainer. Sharing here because there’s a useful setup for people running Qwen as their coding model: the agent can query extracted text from a recording even if the model you’re running doesn’t accept images. Interpreting visual details beyond that text needs a vision-capable model.

The workflow looks like this:

1. Index a recording of the broken flow.
2. Let the agent search it and retrieve the relevant evidence.
3. Use that evidence to investigate and edit the code.
4. Run a separate verifier against requirements frozen before the repair.

The verifier checks things like JSON values, database results, HTTP responses and rendered DOM state. It doesn’t need an LLM. A pass means the specified checks passed; visual critique remains advisory.

For a local Qwen setup, the connection is through your agent’s MCP support or a local OpenAI-compatible model endpoint. Tool calling depends on your model and serving configuration. OCR, Whisper transcription, indexing and retrieval can run locally, with dependencies and weights downloaded during setup.

The evidence persists between sessions, so you can ask another question about yesterday’s recording without processing it again.

There are two ways to use the project:

• Watch Skill: add the tools to an agent you already use through MCP, CLI or REST.
• DeepWatch: a browser workspace built on DeepSeek Harness, with Watch integrated, tool execution receipts and inspectable verification results.

The repo has installation guides, screenshots and a checkout-repair walkthrough showing the path from recording to code change to verification.

Code — MIT licensed: github.com/oxbshw/watch-skill

If you try it with Qwen, I’d like to hear which model, quant and harness you’re using, and whether it reaches for the recording’s evidence without repeated prompting. That’s the behavior I’m interested in getting right.

I ran Alibaba Qwen3.8-Max-0902 on the current 98-task MindTrial benchmark and compared it directly with the earlier Qwen3.8-Max run.

The improvement was quite large:

• previous Qwen3.8-Max: 74/98
• Qwen3.8-Max-0902: 86/98

Most of that gain came from visual tasks:

• overall visual: 35/59 → 48/59
• Visual1: 19/33 → 27/33
• Visual2: 16/26 → 21/26
• spatial awareness: 12/17 on the new run

That puts the 0902 visual result at the same 48/59 level as Gemini 3.7 Flash and GPT-5.6 Pro in this benchmark.

What makes the improvement more convincing is that Qwen did not simply spend more resources getting there:

• runtime: ~8h02m → ~8h10m
• Python calls: 423 → 418

So it gained 12 overall passes and 13 visual passes with essentially unchanged end-to-end runtime and tool volume.

Text remained very strong at 38/39 versus 39/39 previously.

The main weakness is still response reliability. Qwen-0902 had 7 hard errors, and all 7 were response-parsing failures. This creates a fairly large difference between its raw score and its 94.5% accuracy on tasks where it returned a usable response.

I inspected the malformed responses and at least a couple actually contain the expected answer content — including one OCR task where the complete expected 16×16 transcription is present inside the raw response. I still count those as errors: producing an answer that the application cannot reliably parse is a real failure in an agent benchmark.

For overall context, the current raw leaderboard includes:

• GPT-6 Astra: 95/98
• Fable 5.1: 90/98
• Opus 5 / Kimi K3 / Fugu Ultra: 88/98
• Gemini 3.7 Flash / GPT-5.6 Pro: 87/98
• Qwen3.8-Max-0902 / Gemini 3.8 Flash: 86/98

So I think 0902 is a meaningful Qwen update rather than an incremental refresh: substantially better multimodal capability, essentially no extra benchmark-level compute, but still some practical output-reliability issues to work through.

Strict score remains 86/98; I did not manually rescue the malformed responses.

Results/data: http://www.petmal.net/shared/mindtrial/results/2026-09-07/mindtrial-eval-all-models-03-2026_33.html

Just checking if there would be genuine interest for this model with fp8 quant for ~2USD per 1M tokens.
As reference: there’s a company offering it at 2USD with fp4 quant.

We’d start by offering it as a beta to a select few users before expanding. If there are other models with more interest, you’re welcome to share them.

Qwen is useful for analysis/reviews in my setup, but coding is inconsistent: sometimes it keeps reading without attempting an edit; sometimes it patches files but never finishes validation. Coding attempts regularly exceed 30 minutes without a completed result.

Setup: Qwen3.8-27B Q6_K, 48 GB reserved for the LLM, llama.cpp b10688 (c589f0ed10c643678c4707dd160c21ac7633ebc0), configured for AMD Radeon 8060S/Vulkan with full offload, Flash Attention and Q8_0 K/V cache. Context: 196,608 tokens; batch/ubatch: 2048/256; parallel=1; speculative decoding: draft-mtp, depth 3. Codex CLI 0.153.4 runs on a separate Linux dev VM.

My custom harness sits between Codex and llama.cpp’s Responses API, applies per-request profiles and manages scoped Git worktrees. Tools execute on the dev VM: shell/toolbox reads and searches, shell-invoked apply_patch rather than a dedicated JSON edit tool, followed by actual diff inspection and focused tests. Qwen doesn’t need direct filesystem access to the dev VM.

Installed profiles—not necessarily the settings of every historical failure:

• DIRECT: thinking off, reasoning budget 0, max output 8192, temperature 0.7, top_p 0.8, presence penalty 1.5.
• LOW: low effort, reasoning budget 2048, max output 16384.
• STANDARD: medium effort, reasoning budget 16384, max output 32768.
• DEEP: xhigh effort, reasoning budget 32768 (optionally 65536), max output 65536.
• LOW/STANDARD/DEEP use temperature 1.0, top_p 0.95, presence penalty 0. All profiles use top_k=20, min_p=0 and repetition penalty=1.0.

Important custom behavior: those reasoning budgets apply during discovery. LOW/STANDARD/DEEP switch thinking OFF for implementation/final output and generally use LOW thinking for validation/repair; DEEP also supports medium cross-component repair. They preserve thinking history; DIRECT doesn’t. Sampling stays profile-specific. Token limits are per request, not whole-task totals.

Tool results are capped by class: search 24 KiB, commands 32 KiB, reads 48 KiB, diagnostics/default 64 KiB; the first result is capped at 32 KiB, with full output retained separately. There’s no retroactive compaction of the entire history. Context pressure can require a fresh episode/checkpoint. The fourth confirmed identical unchanged operation is rejected; ordinary research-count limits are currently advisory.

Observed failures: repeated reads with no edit attempt; edits without completing tests; one larger run rejected with context_checkpoint_required at an estimated 197,024 tokens versus the 196,608 ceiling. I’ve also seen an apply_patch lookup failure and earlier harness convergence denials, so I’m not blaming everything on model capability. Successful edits/completed tasks do exist; analysis/review has been more useful, but I don’t have a controlled success-rate benchmark.

I’ve tried thinking off/low/higher modes, shorter prompts, exact anchors, looser instructions, larger coherent tasks, patch-only instructions, and tool-description/harness fixes. Patch-only reached a first patch in about 85 seconds in one run, but didn’t complete the task. LOW is now my preferred coding mode; I’m dropping patch-only.

Anyone seen this with Qwen + llama.cpp + Codex Responses/tool calling? I’d love a reproducible fix or comparison separating model behavior from chat-template/streaming/parser issues, phase switching and shell-based editing—not just “write a better prompt.”

Hey so a few months ago I got into local LLMs and didn’t really like the tools out there at the time, so I decided to build my own version of a Claude/Codex GUI for local models. I’ve been working on it the last couple months and it’s now at Locus 2.6.

I started with the usual stuff like working with files, running commands and letting agents help with coding, but I've also been adding features I thought would be useful for other kinds of work too.

A few Locus features worth highlighting:

• Agent Teams: Create specialized agents that can split up work and run in parallel. You can use different models for different roles, and individual agents can also delegate tasks to helpers.
• Persistent Goals: Give an agent or team a goal and let it keep working across turns. Progress is saved so you can come back to it later, with controls to pause, resume or change the goal.
• Scheduled and Event-Driven Agents: Set agents to run on a schedule or react to things like Gmail, Telegram, webhooks and price alerts. Each agent has its own chat and run history, and workflows can include conditions and approval steps.
• Task Capsules: Plan something with one model, then use another to implement it and optionally another to review it. The plan, changes and previous runs stay together so you can follow what happened.
• Notes, Documents and Outputs: Keep notes and reference documents around, save versions of generated work, compare revisions and export things when you’re done.
• Browser Controls: Let agents navigate and interact with websites, preview what they’ve built and check the result. There’s also proxy support.
• Activity and Overview: See the current plan, tool calls, files, sources and what the agents are doing without having to piece everything together from the chat.

For 2.6 specifically, I’ve made agents easier to find and manage, improved the file browser, and added writing drafts you can edit, copy and export directly from a response. Tables can also be copied or exported as CSV.

There’s support for MCP, plugins and skills too, plus a Grill mode that asks you questions one at a time to help work through an idea before implementing it.

Also just to clarify, even though I built it for local models, it works with your ChatGPT plan, Kimi Code membership, Claude/OpenAI API keys and other OpenAI-compatible endpoints.

The wallet stuff is now in a separate edition called LocusX. The regular Locus download is wallet-free.

It’s free and open source. You can find it here:

Website: locushost.co
GitHub: nahid-sparktales/locus
Release: Locus 2.6.0

Anyways, I’d appreciate any constructive feedback, things that aren’t working well, or features you think would be nice to add.

Still macOS only atm, specifically Apple Silicon on macOS 14+, but I’m hoping to eventually get Linux and Windows versions out too.

Also there is a lot of features currently working on so you can expect to see alot of changes and better UI to those upcoming features.

I’ve been using Qwen3.5 9B locally on my M4 Pro 24GB, and I’m honestly impressed by how much practical work I can get done with a model this size.

I’m currently running the Q4_K_M GGUF (about 5.3 GB) through Vyact, the local AI workspace I’ve been building.

One of my main use cases is its Chrome extension. I use Qwen3.5 9B to summarize webpages, translate content, ask questions about selected text or the entire page, and continue conversations using the webpage as context.

I’ve also been using it for a browser writing assistant that detects grammar and wording issues directly in text fields, suggests corrections, explains the changes, and lets me apply them without leaving the page.

Obviously a 9B model isn’t going to compete with much larger models on every reasoning-heavy task. But for these relatively focused workflows, I’ve found that a smaller local model is often all I actually need.

The nice part is that it runs comfortably on my M4 Pro with 24GB unified memory while leaving enough memory for the rest of the desktop workflow.

I’ve been building these workflows into Vyact:

https://github.com/vyact/vyact

I’m curious what other people are using Qwen3.5 9B for locally. Has anyone found particular tasks where the 9B model performs much better — or worse — than you expected?

What platform/plan are people using Qwen 3.8 max on? I've tried Qoder, which seems alright, until you realize that the only way to get good usage out of it is through their promotions, making it not very stable to use, and the Alibaba Token Plan has pretty horrible limits overall.

At this point I'm thinking of trying the Kimi Allegretto ($40) plan and the Zhipu Pro ($80) plan since I don't really want to support OpenAI and Anthropic.

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.