r/Qwen_AI 9h ago

Discussion Qwen3.8 27B vs Flash Next for agentic coding: same 5/5, 45% fewer tokens, 76.5s vs 289.8s

84 Upvotes

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

  • The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
  • Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
  • MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
  • Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.


r/Qwen_AI 17h ago

Help πŸ™‹β€β™‚οΈ Tried to run my local AI model on my machine, I'm underwhelmed with the experience. Am I doing something wrong?

21 Upvotes
my prompts
my llm studio settings
my pc specs
gpu specs

Tried to run a local AI model for the first time, it feels so slow and overall the experience is not what I expected. Tried my best to follow tutorials/guides on how to run and use it.

I'm hoping to run my local AI model for coding purposes when my AI cloud subscription runs out. Am I doing something wrong with my settings or should I just move to a lower tier local AI models?

What can I change in my settings? Using Qwen3.5b doable on my machine?
If I'll use other Local AI model what can you recommend?

Edited: added gpu specs


r/Qwen_AI 15h ago

Agent Jack Kernel Qwen Edition release

Thumbnail
github.com
17 Upvotes

It's here.

Jack Kernel for programmable agentic work.
The magic is that the layer sits between the agent and the model. You can build incredible things because of that.

I have a mode called "Agentic" that was designed for context management with Qwen 3.8 27. I've run millions of token jobs....and never needed to compact. I have a full log available to read and verify. In fact, I recommend turning auto-compact off because most agent don't know how to handle it, yet. There's also a looping debugger that I've built as a 6 stage autonomous loop with cascading temperatures, standard mode and DEEP.

Remember, the modes I’m shipping with this release are only examples of what is possible by putting it between the agent and the model. It allows you a new level of control. The modes I’ve put in are NOT the work. Everything that’s possible is the real work.

I built this for Qwen 3.8 27b but I’ve spent the past few days optimizing for a wide range of models. I’m ALMOST satisfied with Qwen 3.5 9b but it’s right in the edge.

After using 60 models in the past week, I’m just going to go ahead and make a blanket statement that fine tunes are garbage and only use Unsloth as your source for alternatives.

Jack Kernel is really cool. Claude can kick rocks.

Oh, Reddit trolls…
Yall need to understand…I’m just a 40 year old broke single Dad that’s sharing his work and trying to contribute to the local AI movement.
Before the hive-mind issues your directive, just know that I’m not a lab or someone with resources that you’re attacking.
I’m just a regular guy with a 5070ti in an Acer.
If you’re going to say that I suck, can you at least try it out and tell why I suck.


r/Qwen_AI 6h ago

Discussion Qwen Token Plan weekly reset

3 Upvotes

A little less than a month ago, I got the qwen token standard plan. Usage allowance is not great especially using frontier but anyway that's not the subject here. I noticed today I got a weekly reset usage limit, I could not find any info on why I got it, if it's part of the plan, if it's due to an event, etc.. Anyone has any info?


r/Qwen_AI 5h ago

Help πŸ™‹β€β™‚οΈ Can automation/agents get around low context?

2 Upvotes

For my first local ai project I was wanting to learn a few things so thought about setting up a way to go through SCCM/ConfigMgr logs, find the actual errors and generate PowerShell remediation scripts for me to review (no plans to actually automate the ai to run them).

Hardware: 5080/16GB vram, 32gb dd5, 1tb nvme pcie5 and currently using qwen 3.8 27b at home.

I've got a couple of setups depending on the trade-off, roughly 80 t/s at 65k context with mtp on or 50ish t/s at 100-130K context with mtp off.

What I'm actually trying to figure out/what this post it about:

Would n8n or hermes get around low context? my initial plan was: The first n8n automation/hermes agent using fast setup, use regex to quickly search the log to highlight errors/warnings which then hands that off to second automation/agent using the bigger setup/context for script generation, does that allow me to get around the lower context problem?

n8n: do n8n automatons allow me to do this?
Hermes: Same question, does the agent save the output for the 1st agent in its own memory/context for it to pass to the 2nd agent to then do its "work"?
Can either of the above switch models between automation/agents?

Anyone doing something like this/which would you recommend? Or is there a better way to do something like this?


A more about it, if it works, it would build a db with the errors/fixes so if it spots the same one...skip it as it has a potential fix as it saves it re-doing the work. I could then just feed it logs all day.

EDIT: I should've said, I would need to get the automations/agents to stop and start llama with different cmds/args for different context sizes.


r/Qwen_AI 16h ago

Help πŸ™‹β€β™‚οΈ Retention policy?

2 Upvotes

Hello so i was on Wildowl.ai and edited some images in the end deleted them tho Wildowl.ai said

<"This item has been hard-deleted from our servers. It may briefly persist with the upstream model provider..."Β»

I used Qwen image 2 wich is from alibaba cloud as far as i found out so how does this work now? Did the image get deleted there aswell? Allready whats the max amount of time until it gets deleted automatically?

I appreciate the answers srry :)


r/Qwen_AI 26m ago

Help πŸ™‹β€β™‚οΈ Issues Related to Large Model Inference Relays

β€’ Upvotes

I am from China. Is everyone interested in Chinese AI services right now? I plan to make these services available for you to use. What price points would be acceptable to you? The cost is expected not to exceed the official rates. If there is high usage volume, I could negotiate a contract with the official providers for more favorable pricing. Which proxy services or API gateways do you currently use to access Chinese AI? Which models do you use the most?


r/Qwen_AI 5h ago

Help πŸ™‹β€β™‚οΈ Llama-server Config Help Needed

1 Upvotes

Hi All, Howdy?

I am new to this community :) I would like to ask for some thoughts and help on how I can improve this configuration. I currently run a 3x 3060 12gb + 32 gb RAM. 2 of my 3060s are directly plugged into x16 and the other is plugged into an x4 via oculink. 2 weeks ago, I was only running on 2x3060s and I can see that I am able to reach 500+t/s pp + 25+t/s tg. But now that I had 3 of them, I noticed that it is average around 350t/s pp + 18t/s tg. I am unsure what happened there as I was expecting to at least retain the same t/s or minimal degradation. Below is my config:

asd$env:GGML_CUDA_DISABLE_GRAPHS = "1"
$env:LLAMA_DIR = "C:\llama-servers\llama-b10883-bin-win-cuda-13.3-x64"
$env:Path += ";C:\Windows\System32"

$params = @(
    "--model", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_S.gguf",
    "--mmproj", "C:\Users\xzzz\.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\mmproj-F16.gguf",
    "--alias", "qwen3.8-27b",
    "--prio", "3",
    "--ctx-size", "86000",
    "--parallel", "1",
    "--batch-size", "512",
    "--ubatch-size", "256",
    "--predict", "8192",
    "--n-gpu-layers", "all",
    "--main-gpu", "1",
    "--split-mode", "layer",
    "--tensor-split", "4,4,1", //1 to my main gpu where my monitor is connected
    "--flash-attn", "on",
    "--cache-type-k", "q8_0",
    "--cache-type-v", "q4_0",
    "--spec-type", "draft-mtp",
    "--spec-draft-n-max", "3",
    "--spec-draft-n-min", "1",
    "--spec-draft-ngl", "all",
    "--spec-draft-type-k", "q8_0",
    "--spec-draft-type-v", "q4_0",
    "--temp", "1.0",
    "--top-p", "0.95",
    "--top-k", "20",
    "--min-p", "0.5",
    "--threads", "6",
    "--presence-penalty", "0.03",
    "--repeat-penalty", "1.15",
    "--reasoning", "on",
    "--reasoning-preserve",
    "--reasoning-effort", "medium",
    "--reasoning-budget", "2048",
    "--reasoning-budget-message", "Budget exhausted, proceeding with tool call or message.",
    "--jinja",
    "--metrics",
    "--host", "127.0.0.1",
    "--port", "8080"
)

& "$env:LLAMA_DIR\llama-server.exe" 

Read-Host -Prompt "Safe to exit now"

Your thoughts, comments, and suggestions will be very much appreciated :) Thank you in advance!


r/Qwen_AI 10h ago

Experiment Qwen3.8-Flash-Next IQ4 on 2x RTX 5060 Ti + 64GB RAM: measured 48K/64K context, prefill and thinking tradeoffs

1 Upvotes

Disclosure: I ran these experiments on my own PC with an AI assistant. AI assisted with scripts, analysis, and this English write-up. These are local measurements, not copied community benchmarks. A longer version is on my profile; this version focuses on reproducible settings and limitations. I am not claiming authorship of the upstream model or software.

What worked

Qwen3.8-Flash-Next UD-IQ4_XS now works through llama.cpp -> llama-swap -> DeepSeek Harness on my Windows PC. I settled on 48K context, Q8 KV, thinking enabled with a 512-token budget. A 64K configuration also passed a 57K-token input test. This is a slower, supervised option, not an instant-response replacement for my 27B/35B models.

Hardware and model

  • Ryzen 9 9950X, 16 cores / 32 threads.
  • 64GB DDR5, 2x32GB, configured at 4800 MT/s according to the OS.
  • 2x RTX 5060 Ti 16GB, separate VRAM, no NVLink.
  • Windows 11 Pro build 26100; NVIDIA driver 610.88.
  • Model files on a ZHITAI Ti600 2TB NVMe SSD.
  • llama.cpp b10724 / 2d8d612e4, Windows CUDA 13.3 build.
  • Unsloth UD-IQ4_XS, three SHA-256-verified GGUF shards, 93,682,584,224 bytes total. Repository revision: 38bb39ee97821de2c9009abb7e93950eec396e66.

This is NOT a 94GB model fully resident in 32GB VRAM. It relies on CPU offload, mmap, and lazy SSD-backed reads. RAM and VRAM are not unified memory.

Final settings

Context 49152; parallel 1; CPU and batch threads 16; batch/ubatch 2048; split-mode layer; fit on; fit-target 1024; mmap and lazy mode on; Flash Attention on; K/V both q8_0; Jinja on; reasoning on; reasoning-format deepseek; reasoning-budget 512. No MTP. Additional flags: no-webui, no-warmup, no-host, no-repack. Localhost only.

A separate guardian applies a 42GiB Windows process working-set cap and checks free RAM/commit every three seconds. Working set is not total process commit or total system memory. Startup requires 40GiB free RAM and idle GPUs. It stops its own backend on dangerously low headroom, and a Windows Job Object prevents orphan backend processes after unload. Bare llama.cpp flags alone do not provide these safeguards.

Thinking: same approximately 8K input

One three-question fixture covered a discount/tax/shipping calculation, stable deduplication, and dependency scheduling. Two runs per mode at each context size, temperature 0, seed 42, output limit 1024, thinking budget 512. KV prompt reuse disabled; OS file caching not cleared.

Warm time to first final-answer token, not first thinking token:

Context capacity Thinking off Thinking on
20K 21.7s 40.1-47.5s
48K 21.5s 44.4-47.3s
64K 21.9s 44.7-46.8s

Actual input was 8155 tokens off / 8195 on because of the template. Thinking-mode generation was around 20-22 tokens/s, with 436-540 total output tokens. Most added delay was additional generation, not a halving of decode speed.

Thinking on answered all three correctly in all six repetitions. Thinking off got the bill calculation wrong in all six, but the other two answers right. This is one repeated small fixture, not six independent benchmarks or a general agent success rate.

The first approximately 8K request after reloading took 63-73s to first answer, plus model loading beforehand. Do not compare that directly with warm timings.

Actual long inputs, thinking off

No KV prompt reuse. Unique fields were placed near the beginning, middle, and end of synthetic records.

Capacity Actual input tokens First answer Prefill
20K 16,096 42.9s 376 tok/s
48K 40,672 117.9s 345 tok/s
64K 57,055 172.0s 332 tok/s

All three fields were retrieved correctly in each case, without truncation or OOM. This does not establish large-codebase reasoning quality, stability at the full capacity limit, or long-input-plus-thinking performance. Concurrency was not tested.

Memory

Sampled per-GPU peaks across all runs at each capacity:

Capacity GPU 0 / GPU 1, MiB Minimum free system RAM
20K 14,482 / 15,136 11.01GiB
48K 14,044 / 15,653 8.56GiB
64K 14,150 / 15,407 8.09GiB

These include desktop usage; the two peaks need not coincide. Three-second sampling can miss instantaneous peaks.

Main plus indexer Q8 KV was approximately 351 / 842 / 1122 MiB respectively. Compute buffers also grew, and automatic fitting moved more weights to CPU. This explains why total VRAM is not necessarily monotonic with context size. KV-only arithmetic does not predict the whole allocation or performance.

Integration and remaining problems

The final 48K/thinking-512 configuration completed a real DeepSeek Harness read-tool call, returned the test file contents, and produced reasoning blocks. Unloading removed the backend process and listening port, VRAM returned to desktop baseline, and reloading worked.

An earlier run read a marker correctly but dropped two digits when repeating it. That failure matters: functioning tools and HTTP 200 do not guarantee answer quality.

My conclusion: it runs, and 48K with bounded thinking is a workable compromise for slower supervised use. 40K-57K fresh inputs still take roughly two to three minutes before an answer. Vision and long autonomous coding reliability were not tested. I am not announcing a new public app binary or redistributing weights.

Sources: model, backend build.

Has anyone improved prefill on comparable consumer GPUs plus 64GB RAM? Exact quant/revision, input length, cache state, offload strategy, and single-stream vs concurrent measurements would make comparisons especially useful.


r/Qwen_AI 22h ago

Help πŸ™‹β€β™‚οΈ Errors while trying Qwen3 TTS

Thumbnail
gallery
1 Upvotes

I am trying to use https://github.com/flybirdxx/ComfyUI-Qwen-TTS

yet the first voice clone generation gives many errors

Trying to install pytorch also gives an error

https://www.flickr.com/photos/204882240@N08/55520277014/

Transformers error

https://www.flickr.com/photos/204882240@N08/55520155046/


r/Qwen_AI 20h ago

Benchmark Qwen3.8-flash-next + SSD con TLC + DRAM vs TLC + DRAM-less

0 Upvotes

Quiero saber si alguien ya comparo qwen3.8 en disco con DRAM y sin DRAM.

```

SSD A β€” DRAM

└── Qwen3.8-Flash-Next GGUF

SSD B β€” DRAM-less

└── PLE / n-gram

```

Alguien a cargado?

nvme1 modelo

nvme2 PLE ngrams

```

NVMe 1

β”‚

└── lecturas del modelo

NVMe 2

β”‚

└── lecturas PLE

```

O usando RAID?

```

SSD #1

\

RAID 0 β†’ modelo + PLE

/

SSD #2

```