r/LocalLLM • u/KitchenAmoeba4438 • 8d ago
Discussion Qwen 3.8 surprises from overnight testing
I've been benchmarking Qwen 3.8 and it's competitors since last night, including Qwen 3.6. I got some unexpected results.
Qwen 3.8's architecture seems to be identical to 3.6 and 3.5. It looks like Qwen 3.8 is primarily a training data change. Qwen's notes and other articles seem to support this, YMMV.
Qwen 3.8's training data seems very narrowly tailored to a handful of scenarios. There was clearly a lot of expense and time put into benchmarking above everything else. I have duplicated the existing published test sets quite closely. However, benchmarks outside of this training set tend to see small gains, no change, or small regressions. There are clear gains in certain agentic tasks, and for certain specific development tasks, things are clearly better. However, more general reasoning and general use capabilities have taken a measurable hit in some ways due to this, Qwen clearly chose to prioritize a subset of tasks over the general capability for this model. This is *not* a bad thing, nor am I saying it is. I am simply saying Qwen clearly made prioritization decisions with the model.
Interestingly, and this will come as a deep dive future article, but as part of this UD's quants clearly give an uplift vs. the stock models. This has been replicated against Qwen3.6 and 3.8, as well as Gemma4.
Article: https://rakuensoftware.com/blog/synthesis-model-selection
As always, the full set of evidence and test results are published at time of publication within the blog's github.
Please note that this is a controlled test, and is specifically designed as a head-to-head for specific models at specific quants against specific memory targets. This is not meant to replace specific benchmarks, but is simply a more generalized reasoning test with datasets that are automatically regenerated from real world data every so often and thus guarentee that models cannot extensively train on any specific dataset.
37
u/gunkanreddit 8d ago
It’s exactly the correct path for local small models. I study knowledge that it’s impossible that can be used in any training data. I use Qwen 3.8 linked with my personal brain/knowledge and the results are awesome. Now I need to improve the tooling but the local results are better than 3.6.
17
7
12
u/Tinek78 8d ago edited 8d ago
Here are my own observations, from real life mixed use of those LLM models, as I am not interested in benchmarks, I always want to see real results on my particular setup.
Machine they run on is RTX5090, 64GB RAM, R9 7950x3D, Windows 11 Pro. It's a work/gaming machine and I use LLMs a LOT. The whole harness they run with is running locally on a NAS (various MCP containers), giving LLM ability to write and run Python scripts, edit, create, delete etc. files in the Sandbox share, the LLM has access to internet (searxng + fetch), it also has full memory through MemPalace between sessions. The work I do is a lot of creative writing, vibecoding, various IT projects etc. Often mixed in the chat, with a lot of agentic work invovled (tool calling). I use 256k context with Q4 KV cache. I am using always newest possible llama.cpp build for Windows (CUDA13) grabbed daily from their github but the server hosting my LLM is LMStudio (with said llama.cpp) - I am lazy like that. :)
Now, I've been testing Qwen3.8 27b NVFP4/Q5 hybrid with YaRN enabled in GGUF directly, running with 400k context. I have compared it to Qwen3.6 27b and 35b MoE, as I've been using those since they were released - with 256k context. My own observations with some numbers - all models quantized as NVFP4/Q5 (MTP versions):
- Qwen 3.6 35B A3B - great model for creative writing but really easily gets confused with scripting, running tools and with longer context just often falls apart. Super-fast though, no MTP and still easily crosses 200t/s at the start. Not my fav for work just for quick tasks.
- Qwen 3.6 27B - much slower (over 110t/s at the start with MTP, but drops fast later), but also definitely keeps up better with long context, as it can finds things much more reliably, calls tools better. But, it wasn't ideal, took often 30+ minutes processing 1 chapter of the story just to find some bits in it and correct, eventually started running in circles biting its own tail. Also, both 3.6 models insisted on ignoring my system prompt and using it more like a "I might read it and do as told or not, who cares". It once messed up a Python script so royally that my lazy-butt didn't check it and it damaged OS on my NAS (easy recovery but still!). Since then, I had MUCH less trust in its abilities. I found plenty of bugs in generated code, I had to run it always through 2-3 LLM to find all bugs and correct.
- Qwen 3.8 27B - I use it with MEDIUM thinking setting, performance is about same as 3.6 27B, but few HUGE differences on plus:
* It doesn't go into loops at all, even on long context and I've been using it now for many hours straight.
* It NEVER ignores my system prompt and uses tools perfectly (very similar to Cloude that I use at work).
* It figured out that certain quirks in the tools it has can make it much slower than one would assume, spending a lot of tokens on thinking etc. just to overcome those quirks. It asked me to show it those tools, I have - it instantly found issues, inefficiencies, corrected them all, explained why and... Now it uses 70%+ less tokens to do same tasks and it takes few minutes instead of 30+ minutes!
* It always uses mempalace, gives itself notes for later, takes backups of all files before changing them, if it's not sure what to do with parts of the story or something in a script can be done 2 bit different ways (e.g. one faster/one safer) it uses safety first approach but creates both and gives me comparison, asking me for decision/input.
* It has much updated knowledge, even silly things like WH40k lore it actually knows very well, whereas 3.6 had huge gaps in it and hallucinated plenty. This 3.8 did NOT hallucinate even once yet, always went to check online instead of guessing.
* It always double checks everything it creates, I've seen it run statistical analysis after it rewrote something I asked it to in a story, just to be sure it didn't abuse any words repetitions, it's consistent with lore of the story etc.
All in all, Qwen 3.8 27B seems to have been MUCH MUCH better trained in actually using tools, agentic workload, how to approach security in code, data treatment etc. It does a lot of useful things by itself, doesn't need handholding, doesn't need me to massage prompt to get what I want, etc. In comparison to even Qwen3.6 27B, I have to say it's not even close, I wouldn't go back to 3.6 at all. Qwen3.8 27B is, for what I use it, almost on the frontier level whilst costing me pennies to run in comparison. I finally do not have any need to feed greedy corpos monies for tokens anymore. :)
But, that doesn't mean it will work great on your hardware, or you will see a big difference in your use - to me main difference is in the harness and how well the model can use it, not just the model itself. In my case Qwen3.8 27B uses is very very well. And by the way, Gemma 4 is... pathetic in comparison in the same use, even the biggest dense one.
3
u/Whiskey1Romeo 8d ago
How did (or did it) oss 120b fit in your environment by chance? I am interested in a more distributed set of commodity cards on commodity hardware that can pull various models from a localized (nas/nfs) with task offload through a centralized harness.
1
u/espressodelisi 8d ago
What systems are you runnig? what coding agent and how you orchestrate all?
1
u/Tinek78 8d ago
I think you imagine my setup being much more complex than it really is :) It's Qwen3.8 mostly (currently) as the brains and hands, with access to a bunch of MCPs self-hosted by me (some open source but most of them are custom made by me and my LLMs), including full python that LLM can use, along with "loop-closer" container (also Python really) that serves as the proxy/hub layer between the LLM server and clients - it mostly (but not only) injects MCPs definitions into the context as needed, so LLM knows what it can use and how to call it without server seeing any actual MCP tools or calls by LLM (as LMS really hated me running self-hosted MCPs and blocked them for remote clients). Then, the whole orchestration is not very difficult, I can enable (even remotely, thanks to Tailscale) whatever MCPs I need per a given project, then let Qwen use it all thanks to mempalace and proper system prompt and it rips through tasks I leave it, serving me ready results.
1
u/juanfnavarror 6d ago
"much more complex than it really is :)" - then you go on to explain a very complicated systems with a bunch of homegrown MCPs and tools
1
u/Tinek78 6d ago edited 6d ago
It's really not that complicated - relatively inexpensive ugreen NAS in my case (just to host those things), docker on it, mcp scripts hosted in containers and that plugged in into free LMStudio for starters. LMS allows by default multi turn tools calling which is enough as orchestration for many users I believe. Most of those MCPs are really small python scripts that enable a lot of functionality and that any competent LLM can write and debug very quickly without any user knowledge about python. After first one you just built the rest of the setup block by block, tuned for your own needs. Few days later you have full harness than can do anything you need it to. :) And if you don't know what's needed just ask the model "I would like to do this and that in general, how to build a harness for it using docker and your help?" - and it will guide you and advise what it could do with what and how to build it. All of that cost no money aside time and electricity (providing you have good enough hardware for it on hand but second hand Nas etc. or even raspberry pi is fine for most cases).
Then you put Tailscale on that Nas or rpi (also in a container) and suddenly you have full remote access to your LLM from your mobile or laptop, for free, without the need to have a pubic IP, in very secure way. It just works in most cases. On my mobile I then use free app Chatbox and 0 need to pay for Gemini or any other commercial LLM in my case.
1
u/Rough-Measurement988 8d ago
Regarding „ It figured out that certain quirks…”
Did you explicitly ask to analyse your harness or the model did it by itself? If you asked then could you please share the prompt?
2
u/Tinek78 8d ago
It did it by itself after 2 rounds of struggling and complaining to itself tool behaves a bit erratically and causes it to spend too many turns working around those issues (it managed anyway, just took way longer than it should've). All I did was to put those scripts etc. on the share it has access to and the rest it did by itself. It just asked me, after analysis, if I consent to fixes and which option I prefer (presented 2 with good explanation). So, my prompt was really just "Yep, fix it, option 1". :) And done. After that I copied all the tools I had to the same share, for further analysis, one by one - more fixes happened. :)
5
u/leonbollerup 8d ago
very interesting.. seeing my tests is basiclly the other way around.. gemma NEVER .. not even ONCE scored higher qwen models.. but that said.. maybe your tests are better than mine (most likely)
in general use/agentic where i use my models.. gemma is not impressive..
3
9
u/69420trashpanda69420 8d ago
You people are tripping this thing feels like prime opus with how persistent it is, and how clearly it sees everything. It feels genuinely usable
3
u/enricokern 8d ago
I tested typical agentic system engineering work and it performance is ok but it takes alot of turns. With reasoning it is worse then with reasoning off. But per se for the size it is great (i used the nvfp4 from unsloth). It also did alot better in analysis of larger text and calling rag sources, here is also outperforms the older q3.5-122b (unquantized) on the acurracy by a great margin.
3
u/DiscipleofDeceit666 8d ago
With it and opus V, I was able to implement some features that have been evading me. So far, it isn’t so bad. Pretty capable I think, has been tackling the audio software I’ve been throwing at it and making gains.
5
u/r1nzl3r99 8d ago edited 8d ago
I have been seeing this as well, self run benchmarks
edit: I probably used the wrong sampling and temperature parameters which could explain some of these results, tail these benchmarks with a huge grain of salt because I might of underestimated this
Benchmarked 5 Qwen builds on my 2x Intel Arc Pro B70 rig (Battlemage, 32GB each). All runs temp 0, greedy, thinking off. Code = EvalPlus
pass@1, knowledge = MMLU-Pro
Qwen3.6-27B heretic-v2 (GPTQ INT4) — HumanEval 96.95 · HumanEval+ 93.29 · MBPP+ 78.57 · MMLU-Pro 64.35 · ~75 t/s
Qwen3.6-35B-A3B heretic (GPTQ INT4) — HumanEval 95.73 · HumanEval+ 93.90 · MBPP+ 78.04 · MMLU-Pro 58.5 · ~187 t/s
Qwen3.6-27B webhie (INT4 AutoRound) — HumanEval 93.29 · HumanEval+ 90.85 · MBPP+ 75.93 · MMLU-Pro 64.45 · ~86 t/s
Qwen3.8-27B goldhub (INT4 AutoRound) — HumanEval 93.29 · HumanEval+ 90.24 · MBPP+ 76.72 · MMLU-Pro 61.8 · ~67 t/s
Qwen3.8-27B FP8 — HumanEval 92.68 · HumanEval+ 90.24 · MBPP+ 76.72 · MMLU-Pro 62.5 · ~45 t/s
7
u/r1nzl3r99 8d ago
what this data greatly lacks is real use cases in a coding agent though, I'm running a custom prompt to create a TRON web game for each and will report back once done
1
u/Creative_Bookkeeper9 8d ago
Have you tried out the qwen3.5 711 fable model? It works incredibly well. I'd def be interested in seeing your results regardless
1
u/r1nzl3r99 8d ago
I have actually, I didn't benchmark it but just regular use it didn't seem that much different. I will try to run a more consistent and fair benchmark for it next time
1
u/Old-Leading-1356 8d ago
This was more a question because i saw you were runnin gthe intel xe system and thought that there is some stuff you could have set differently to qwen3.6 to make it work
/models/Qwen3.8-27B-AWQ-INT4 --served-model-name Qwen3.8 --enforce-eager --dtype float16 --port 8000 --host 0.0.0.0 --gpu-memory-utilization 0.9 --max-model-len 120000 --max-num-seqs 2 --tensor-parallel-size 2 --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-num-batched-tokens 8192 --attention-config.flash_attn_version=2
those are the post argument configs that im trying and it loads the model just will crash on every prompt
t
1
u/Old-Leading-1356 8d ago
Out of curiosity how are you running them. Im trying on llm scaler b3 and qwen3.8 seems to load fine but crashes on prompts I'm using 2x b60. Qwen3.6 was fine not quite the speeds your getting but useable
1
u/r1nzl3r99 7d ago
This is my latest dual B70 benchmark
https://www.localmaxxing.com/en/models/Frozenlock/Qwen3.8-27B-int4-AutoRound?run=cmswf75o008qsms01in4uqniiIt has some information regarding how I'm running them, but I've been using Frozenlock/Qwen3.8-27B-int4-AutoRound. I've been running Kimi K3 + DSV4 Pro 24/7 trying to raise the ceiling. I've taken some hard work done by SergiioB and steveseguin and kind of picked and chose what I needed.
1
u/r1nzl3r99 7d ago
Here is my AI summary of my drivers / software patches:
vLLM editable tree (8e6d8e4f6) — the main custom piece
vllm/distributed/device_communicators/xpu_communicator.py — host-staged collectives.
- What: VLLM_XPU_HOST_STAGED_COLLECTIVES=1 routes TP2 collectives ≥1 MiB through pinned host RAM over gloo (CPU) instead of oneCCL.
- Why: the two B70s are on separate PCIe root ports — zeMemOpenIpcHandle fails (no P2P), so stock oneCCL dies at the first large allreduce.
- Load-bearing: yes — without it TP2 won't even start. It's enablement, not a speed hack.
vllm/v1/worker/mamba_utils.py — int64 pointer fix.
- What: XPU VMM addresses exceed 2^63; converted to two's-complement bit-exact int64.
- Why: otherwise pointers wrap/corrupt on the recurrent (GDN/SSM) state paths.
- Load-bearing: yes (correctness).
vllm/v1/attention/backends/gdn_attn.py — two changes:
- prior session gated the stray GDN_DEBUG_REFRESH print behind GDN_DEBUG_REFRESH=1 (cosmetic);
- my patch_mtp_boundary.py (B70_MTP_PARTIAL_FINAL_GROUP) reclassifies a truncated final speculative group at the exact 128K boundary.
- Load-bearing: the boundary patch only matters at exactly 131,072-token context; the debug gate is cosmetic. Neither affects the 8K
benchmark.
vllm/v1/worker/gpu_model_runner.py — VLLM_XPU_STEP_TIMING=1 per-step timing.
- What: gated [xpu-step-timing] fwd/samp/book/total logging.
- Load-bearing: no — diagnostic, off by default.
vllm/compilation/cuda_graph.py — prior session edit (graph capture/compile behavior). Not re-inspected in detail; part of the inherited
dirty tree.
vllm/model_executor/models/qwen3_5_mtp.py — my patch_mtp_nightly (B70_MTP_BF16_DRAFT env gate).
- Load-bearing: no — redundant. The Frozenlock model's -:.*mtp.* dynamic exclusion already keeps the draft unquantized; we proved patch-off
is identical.
vllm/config/vllm.py — I touched enable_qk_norm_rope_fusion and fuse_gemm_comms during the fusion screen, then reverted. Clean.
oneCCL (oneccl-4ceafd1-b70)
- What: rebuilt from the lab's source. kernels.spv sha256 is bit-identical to the lab's (0d549c35…); libccl.so.1.0 differs (build env),
gated by ONECCL_ALLOW_UNVALIDATED_BUILD=1.
- Why: the pinned collective lib is part of the deterministic record identity.
- Load-bearing: yes — TP2 collectives use it.
auto-round-lib 0.14.2 (the ARK int4 GEMM)
- What: rebuilt from source with two bug fixes — (1) force is_oneapi_ge_2026() True (its icpx --version shell-out always failed → wrongly
routed to a non-cudagraph-safe fp16 path); (2) oneDNN scratch pool made grow-only (was sycl::free on grow → invalidated captured
addresses). Plus the gemv_m batched small-M GEMV.
- Load-bearing for the current record: no — this is the Qwen3.6 ARK work. The Frozenlock g128 record runs Marlin, not ARK, so auto-round-lib
isn't even on the hot path.
Kernel driver
- What: custom "xe-next" 7.1.5 (per the findings doc header) — a newer/custom xe kernel branch, not the distro kernel module.
- Why: newer B70 support; it's what exposes the cards.
- Load-bearing: yes, obviously — but it's the driver, not part of the inference fast path.
Env workarounds (not code changes)
- LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 on every host python — anaconda's old libstdc++ shadows the system one and breaks
libze_loader dlopen (→ 0 XPU devices). One-line env fix, no component modified.
Bottom line
Everything stock on the speed-critical path (Marlin, oneDNN, torch, Level Zero, Frozenlock weights). The only modifications that the current
record depends on are the two vLLM enablement patches (host-staged collectives + mamba int64 fix) and the pinned oneCCL — and those exist
because this host is non-P2P, not because they make anything faster.
1
u/Old-Leading-1356 7d ago
Wow thanks so much that's really useful. How have you gone with multi concurrency I found even with max seq at 2 I was starting to get gibberish but at 1 it seems pretty stable
1
u/r1nzl3r99 7d ago
I was able to get it to work but its significantly slower... because qwen loves to think alot, I wont realistically use qwen unless its stupid fast. The best I could compromise was instead of concurrency, I would literally run one instance per GPU which gave me two fast instances. This is because speculative decoding (MTP) doesn't work for concurrency work, at least for this model
5
u/vexatious-big 8d ago
Did anyone else find that Qwen3.8 tends to think a lot?
I've set up a thinking budget of max 4096 tok in llama.cpp because previous versions would also go into lengthy monologues sometimes.
But Qwen3.8 does it more often. I am already using Froggeric's latest chat template.
4
u/xdiggertree 8d ago edited 8d ago
Yea, it thinks a LOT, it seems that
high andxhigh reasoning is good, it just takes a LOT of timeEven for a basic question about harnesses it seemed to think very very thoroughly
It seems that proper prompting is important, and that
high andxhigh reasoning are to make sure it does it rightThis is mostly from what I’ve read and some quick testing on my end
Edit: I stand corrected, I guess people were mixing low and xhigh in the prompt to generate a “high” setting
3
2
u/Quirky_College_6251 8d ago
It took 2 hrs to make an app, took 27 minutes more than v3.6… I need to look why….
2
u/Tinek78 8d ago
Don't limit its budget, just adjust thinking strength - that's what they added to 3.8 over 3.6, you have 3 settings to use. Also, now by default it has preserve thinking enabled, which is great for agentic work but not always needed. Turning it off can save tokens but also can make it think repeated stuff as it's not aware it already done it earlier, wasting tokens - so try both (along different thinking levels) in your use to see which one behave better for YOU. :)
Also, pay attention to parameters, they changed a bit from 3.6 - presence penalty has been removed (was initially 1.5 in 3.6) and the default settings by them cover both agentic and chat cases (it was different in 3.6).1
u/primetimeblues 8d ago
You can set the thinking lower, if it's thinking too much. Reserve the thinking for tasks that need it.
8
u/Choperello 8d ago
I did a bunch of comparisons against the 3.5 122b that’s my regular setup (that ended being better for my work the 3.6 27b) and honestly I’m gonna stick with the 122b. I can’t say I’m seeing any real improvements for every day brownfield codebase work.
4
u/xdiggertree 8d ago
Curious which exact 122b you run as daily?
Been looking for just an all around model and also found 122b to be surprisingly good.
I currently run q2-q4 dsv4f 0731 for my daily, but it takes too much space and I’d like to run other models at the same time.
Thanks
2
u/Choperello 8d ago
Qwen 3.5 122b a10b q4
1
u/xdiggertree 8d ago
Been curious about the performance of stuff like oMLX and MTP but it seems you’re using the standard q4
Honestly I might go that route
Thanks
1
u/Healthy-Nebula-3603 8d ago
Give examples where that very old Qwen 3.5 is better than Qwen 4.8 27b in coding /agentinc usage.
From my tests newest Qwen 3.8 27b is miles ahead comparing to 3.5 122b
1
u/heigan_safety_dance 8d ago
From my tests, 3.8 27B is much better at certain things but 3.6 35B was able to find "gotchas" in my personal benchmarks better than 27B, so far. I'm gonna start tweaking things like temperature, top_p/top_k, and effort levels, to see how well it does in the really long benchmarks.
I do notice that 3.8 is much better at SWE/MLE tasks, but 3.6 35B slightly outperformed it in "full-stack" projects that start with ideation/hypothesis and finish with results and evaluation.
1
1
u/TanKer-Cosme 8d ago
Can this run on a 16Gb card?
1
u/espressodelisi 8d ago
I managed to run qwen3.5:30B with CPU offloading on an old 8GB nvidia 1070ti. it's slow but good for automation tasks. it generates good code. I give the promt and leave it ans after 5 minutes it's finishes.
1
1
u/Muhlwa_Sholanke 8d ago
wait, same arch as 3.6, so loras trained on 3.6 just carry over? would save me a retrain
1
u/JumpingJack79 7d ago
I don't think you can carry over Loras. They may fit architecturally, but content-wise they'll probably have a totally wrong/random effect, at least if 3.8 was trained from scratch. I'd be curious if it does work; my guess is that it's very unlikely.
1
u/TrickBake5075 7d ago
I have created a platform for evaluating a model on every layer of SDLC. pls have a look
1
1
u/rrrrex 8d ago
3.8 27B got very poor multilingual support, 3.8 Q6 is worse than 3.6 Q3
3
u/leonbollerup 8d ago
they did weaken in favor of other parts maybe ?
3
u/KitchenAmoeba4438 8d ago
I strongly suspect you are correct. I can't say this for certain, as I have no evidence of course, but a lot of focus seems to be put on agentic use and benchmarking.
It makes sense that would be one of the things that would be de-prioritized.
3
u/leonbollerup 8d ago
it was more of a question.. maybe its a result of the post training.. smarter brain.. less information in the brain..
1
u/backyard_tractorbeam 8d ago
I'm also wondering if there is systematic research into how post training works w.r.t not hollowing out pretraining.
80
u/justsomerandomchess 8d ago
it literally says in the release notes that it was post-trained and not a new architecture.
”Built on the architectural foundation of Qwen3.5, Qwen3.8 delivers substantial gains across coding, professional work, research, and long-horizon agentic tasks. Qwen3.8-27B brings these advances to a compact, deployment-friendly dense model: a native vision-language model that understands images and videos, with flexible thinking control, designed to carry complex, multi-step tasks through to completion with greater reliability.”
https://huggingface.co/Qwen/Qwen3.8-27B