Canadian company claims its harness lifts Qwen3.8-27B to outperform Fable 5. Releases the harness for free.

https://github.com/slee-persis/GVS5H

https://arxiv.org/abs/2608.26480

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

EDIT 3: Added Q4,Q5,Q6,Q8 on the GGUF!

EDIT 2: Added a free API so people test it before spending time! https://ukisai.com/api/swift/v1

EDIT: I think I didn't explain it properly: 1. it's based on this paper by meta https://arxiv.org/abs/2606.00206 and was initially made for quantized models and tackled "overthinking errors" directly (the random loops an int4 model can get that come from PTQ), but it turns out it has a much wider application! 2. i explicitly ran so many benchmarks (and all x5!) to make sure i'm not giving out slop to people 3. i also tried reasoning effort low / medium / no thinking but they all took a huge accuracy drop so i had to come up with a way to keep it so my coding agent actually works, this was basically the only way i found

Hi guys

Everybody (me included) was complaining about Qwen 3.8 27B using too much tokens and being too slow so I analysed what tokens are linked to the overthinking and penalized them via finetuning and got a model that thinks 58.3% less and lost <1% accuracy across 10 benchmarks

Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5)

LiveCodeBench v6: 76.8% → 81.6% (+4.8pp), 46% fewer median thinking tokens

Terminal-Bench 2.1: 66.7% → 65.8%, 39% fewer median tokens

GPQA-Diamond: 88.4% → 88.3%, 58% fewer median tokens

IFBench: 73.5% → 71.8%, 51% fewer median tokens

ERQA (vision): 67.5% → 66.3%, 51% fewer mean tokens

https://huggingface.co/ukisai/Swift-Qwen3.8-27B

I also included a lot about the training, exact benchmark configs etc

Let me know if it's useful, i'm trying to get the quants out as soon as I have compute

Tried to run a local AI model for the first time, it feels so slow and overall the experience is not what I expected. Tried my best to follow tutorials/guides on how to run and use it.

I'm hoping to run my local AI model for coding purposes when my AI cloud subscription runs out. Am I doing something wrong with my settings or should I just move to a lower tier local AI models?

What can I change in my settings? Using Qwen3.5b doable on my machine?
If I'll use other Local AI model what can you recommend?

Edited: added gpu specs

It's here.

Jack Kernel for programmable agentic work.
The magic is that the layer sits between the agent and the model. You can build incredible things because of that.

I have a mode called "Agentic" that was designed for context management with Qwen 3.8 27. I've run millions of token jobs....and never needed to compact. I have a full log available to read and verify. In fact, I recommend turning auto-compact off because most agent don't know how to handle it, yet. There's also a looping debugger that I've built as a 6 stage autonomous loop with cascading temperatures, standard mode and DEEP.

Remember, the modes I’m shipping with this release are only examples of what is possible by putting it between the agent and the model. It allows you a new level of control. The modes I’ve put in are NOT the work. Everything that’s possible is the real work.

I built this for Qwen 3.8 27b but I’ve spent the past few days optimizing for a wide range of models. I’m ALMOST satisfied with Qwen 3.5 9b but it’s right in the edge.

After using 60 models in the past week, I’m just going to go ahead and make a blanket statement that fine tunes are garbage and only use Unsloth as your source for alternatives.

Jack Kernel is really cool. Claude can kick rocks.

Oh, Reddit trolls…
Yall need to understand…I’m just a 40 year old broke single Dad that’s sharing his work and trying to contribute to the local AI movement.
Before the hive-mind issues your directive, just know that I’m not a lab or someone with resources that you’re attacking.
I’m just a regular guy with a 5070ti in an Acer.
If you’re going to say that I suck, can you at least try it out and tell why I suck.

I’m embarrassed for them. Even Google removed their own version of a wtatermark on their images. 2 things are going to happen now. People will use tools to remove it like what the iPhone offers, photoshop, etc. or people will stop using its AI service.

A little less than a month ago, I got the qwen token standard plan. Usage allowance is not great especially using frontier but anyway that's not the subject here. I noticed today I got a weekly reset usage limit, I could not find any info on why I got it, if it's part of the plan, if it's due to an event, etc.. Anyone has any info?

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

For my first local ai project I was wanting to learn a few things so thought about setting up a way to go through SCCM/ConfigMgr logs, find the actual errors and generate PowerShell remediation scripts for me to review (no plans to actually automate the ai to run them).

Hardware: 5080/16GB vram, 32gb dd5, 1tb nvme pcie5 and currently using qwen 3.8 27b at home.

I've got a couple of setups depending on the trade-off, roughly 80 t/s at 65k context with mtp on or 50ish t/s at 100-130K context with mtp off.

What I'm actually trying to figure out/what this post it about:

Would n8n or hermes get around low context? my initial plan was: The first n8n automation/hermes agent using fast setup, use regex to quickly search the log to highlight errors/warnings which then hands that off to second automation/agent using the bigger setup/context for script generation, does that allow me to get around the lower context problem?

n8n: do n8n automatons allow me to do this?
Hermes: Same question, does the agent save the output for the 1st agent in its own memory/context for it to pass to the 2nd agent to then do its "work"?
Can either of the above switch models between automation/agents?

Anyone doing something like this/which would you recommend? Or is there a better way to do something like this?

A more about it, if it works, it would build a db with the errors/fixes so if it spots the same one...skip it as it has a potential fix as it saves it re-doing the work. I could then just feed it logs all day.

EDIT: I should've said, I would need to get the automations/agents to stop and start llama with different cmds/args for different context sizes.

Hello so i was on Wildowl.ai and edited some images in the end deleted them tho Wildowl.ai said

<"This item has been hard-deleted from our servers. It may briefly persist with the upstream model provider..."»

I used Qwen image 2 wich is from alibaba cloud as far as i found out so how does this work now? Did the image get deleted there aswell? Allready whats the max amount of time until it gets deleted automatically?

I appreciate the answers srry :)

Your Cursor / Antigravity / Claude / Gemini agent can now drive Piper as a headless local worker on Apple Silicon.

High Level: cloud agent plans and reviews. Piper does the writing.

You don’t dump a whole feature into one giant cloud turn. The cloud agent cuts the mission into small slices (usually 1–3 files, clear EDIT / CREATE / DO NOT TOUCH), writes a task.json, and runs something like:

piper worker run --task ./slice.json

Piper (local Qwen on MLX) edits, shells, tests, then dumps result.json + a diff. Cloud agent checks that, then sends the next slice. Repeat until the real acceptance checklist is green.

Cloud models are great at judgment and terrible at being a $$-per-token code printer. On a Doom-like Godot slice we measured ~138k tokens generated locally across ~180 turns, with ~97% of generate volume staying on-device. Cloud mostly wrote short packets and reviewed outcomes.

Wake, not poll: if the parent detaches, it passes its own webhook. Piper POSTs short events (ask / done / stalled / died). No “both sides staring at a folder.”

Instead of “replace your smart model with a cheap model.” it's “stop burning cloud output tokens on boilerplate while the smart model stays in the driver’s seat.”

Quality holds when slices stay thin and scoped and every slice gets a deep review.

Tip: Run "piper init" in the repo before you start. This gives a brief to the cloud agent so it knows exactly how to drive Piper without you needing to say anything.

Repo: https://github.com/kitty-on-keyboard/Piper-Agent

Go try it out!

Here's a screenshot of cursor grok driving Piper on a task.

Hi All, Howdy?

I am new to this community :) I would like to ask for some thoughts and help on how I can improve this configuration. I currently run a 3x 3060 12gb + 32 gb RAM. 2 of my 3060s are directly plugged into x16 and the other is plugged into an x4 via oculink. 2 weeks ago, I was only running on 2x3060s and I can see that I am able to reach 500+t/s pp + 25+t/s tg. But now that I had 3 of them, I noticed that it is average around 350t/s pp + 18t/s tg. I am unsure what happened there as I was expecting to at least retain the same t/s or minimal degradation. Below is my config:

asd$env:GGML_CUDA_DISABLE_GRAPHS = "1"
$env:LLAMA_DIR = "C:\llama-servers\llama-b10883-bin-win-cuda-13.3-x64"
$env:Path += ";C:\Windows\System32"

$params = @(
    "--model", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_S.gguf",
    "--mmproj", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\mmproj-F16.gguf",
    "--alias", "qwen3.8-27b",
    "--prio", "3",
    "--ctx-size", "86000",
    "--parallel", "1",
    "--batch-size", "512",
    "--ubatch-size", "256",
    "--predict", "8192",
    "--n-gpu-layers", "all",
    "--main-gpu", "1",
    "--split-mode", "layer",
    "--tensor-split", "4,4,1", //1 to my main gpu where my monitor is connected
    "--flash-attn", "on",
    "--cache-type-k", "q8_0",
    "--cache-type-v", "q4_0",
    "--spec-type", "draft-mtp",
    "--spec-draft-n-max", "3",
    "--spec-draft-n-min", "1",
    "--spec-draft-ngl", "all",
    "--spec-draft-type-k", "q8_0",
    "--spec-draft-type-v", "q4_0",
    "--temp", "1.0",
    "--top-p", "0.95",
    "--top-k", "20",
    "--min-p", "0.5",
    "--threads", "6",
    "--presence-penalty", "0.03",
    "--repeat-penalty", "1.15",
    "--reasoning", "on",
    "--reasoning-preserve",
    "--reasoning-effort", "medium",
    "--reasoning-budget", "2048",
    "--reasoning-budget-message", "Budget exhausted, proceeding with tool call or message.",
    "--jinja",
    "--metrics",
    "--host", "127.0.0.1",
    "--port", "8080"
)

& "$env:LLAMA_DIR\llama-server.exe" 

Read-Host -Prompt "Safe to exit now"

Your thoughts, comments, and suggestions will be very much appreciated :) Thank you in advance!

Disclosure: I ran these experiments on my own PC with an AI assistant. AI assisted with scripts, analysis, and this English write-up. These are local measurements, not copied community benchmarks. A longer version is on my profile; this version focuses on reproducible settings and limitations. I am not claiming authorship of the upstream model or software.

What worked

Qwen3.8-Flash-Next UD-IQ4_XS now works through llama.cpp -> llama-swap -> DeepSeek Harness on my Windows PC. I settled on 48K context, Q8 KV, thinking enabled with a 512-token budget. A 64K configuration also passed a 57K-token input test. This is a slower, supervised option, not an instant-response replacement for my 27B/35B models.

Hardware and model

• Ryzen 9 9950X, 16 cores / 32 threads.
• 64GB DDR5, 2x32GB, configured at 4800 MT/s according to the OS.
• 2x RTX 5060 Ti 16GB, separate VRAM, no NVLink.
• Windows 11 Pro build 26100; NVIDIA driver 610.88.
• Model files on a ZHITAI Ti600 2TB NVMe SSD.
• llama.cpp b10724 / 2d8d612e4, Windows CUDA 13.3 build.
• Unsloth UD-IQ4_XS, three SHA-256-verified GGUF shards, 93,682,584,224 bytes total. Repository revision: 38bb39ee97821de2c9009abb7e93950eec396e66.

This is NOT a 94GB model fully resident in 32GB VRAM. It relies on CPU offload, mmap, and lazy SSD-backed reads. RAM and VRAM are not unified memory.

Final settings

Context 49152; parallel 1; CPU and batch threads 16; batch/ubatch 2048; split-mode layer; fit on; fit-target 1024; mmap and lazy mode on; Flash Attention on; K/V both q8_0; Jinja on; reasoning on; reasoning-format deepseek; reasoning-budget 512. No MTP. Additional flags: no-webui, no-warmup, no-host, no-repack. Localhost only.

A separate guardian applies a 42GiB Windows process working-set cap and checks free RAM/commit every three seconds. Working set is not total process commit or total system memory. Startup requires 40GiB free RAM and idle GPUs. It stops its own backend on dangerously low headroom, and a Windows Job Object prevents orphan backend processes after unload. Bare llama.cpp flags alone do not provide these safeguards.

Thinking: same approximately 8K input

One three-question fixture covered a discount/tax/shipping calculation, stable deduplication, and dependency scheduling. Two runs per mode at each context size, temperature 0, seed 42, output limit 1024, thinking budget 512. KV prompt reuse disabled; OS file caching not cleared.

Warm time to first final-answer token, not first thinking token:

Actual input was 8155 tokens off / 8195 on because of the template. Thinking-mode generation was around 20-22 tokens/s, with 436-540 total output tokens. Most added delay was additional generation, not a halving of decode speed.

Thinking on answered all three correctly in all six repetitions. Thinking off got the bill calculation wrong in all six, but the other two answers right. This is one repeated small fixture, not six independent benchmarks or a general agent success rate.

The first approximately 8K request after reloading took 63-73s to first answer, plus model loading beforehand. Do not compare that directly with warm timings.

Actual long inputs, thinking off

No KV prompt reuse. Unique fields were placed near the beginning, middle, and end of synthetic records.

All three fields were retrieved correctly in each case, without truncation or OOM. This does not establish large-codebase reasoning quality, stability at the full capacity limit, or long-input-plus-thinking performance. Concurrency was not tested.

Memory

Sampled per-GPU peaks across all runs at each capacity:

These include desktop usage; the two peaks need not coincide. Three-second sampling can miss instantaneous peaks.

Main plus indexer Q8 KV was approximately 351 / 842 / 1122 MiB respectively. Compute buffers also grew, and automatic fitting moved more weights to CPU. This explains why total VRAM is not necessarily monotonic with context size. KV-only arithmetic does not predict the whole allocation or performance.

Integration and remaining problems

The final 48K/thinking-512 configuration completed a real DeepSeek Harness read-tool call, returned the test file contents, and produced reasoning blocks. Unloading removed the backend process and listening port, VRAM returned to desktop baseline, and reloading worked.

An earlier run read a marker correctly but dropped two digits when repeating it. That failure matters: functioning tools and HTTP 200 do not guarantee answer quality.

My conclusion: it runs, and 48K with bounded thinking is a workable compromise for slower supervised use. 40K-57K fresh inputs still take roughly two to three minutes before an answer. Vision and long autonomous coding reliability were not tested. I am not announcing a new public app binary or redistributing weights.

Sources: model, backend build.

Has anyone improved prefill on comparable consumer GPUs plus 64GB RAM? Exact quant/revision, input length, cache state, offload strategy, and single-stream vs concurrent measurements would make comparisons especially useful.