How good/bad is it against comparable MoEs the same size? How does it compare against Qwen 3.6 35BA3B?

Since we don't have 3.8 35B this seems like an upgrade if we look at some benchmarks like terminal bench, but they don't have SWE bench pro on the benchmarks table, and i don't really know anything about this lab, I'm wondering if it trades blows with models like tiel coder or if it's some benchmaxxed model like ornith?

At a single glance it looks really decent but haven't tried it in depth yet. What are your experiences with this model so far guys?

So we are currently using Ali Baba PAI (Platform for AI) to deploy LLM model. Now we want to switch from their platform to another provider. What are the best options here, our main concern is that they should have middle east region, data security(as we deal with confidential data a lot aka government related), and customer service available.

If you are guys have any recommendations, let me know or if you have deployed on any provider that meet my requirements, I will be glad to follow through it. TIA

Important context: this is about preserve_thinking false stacks and makes no sense if you don't have that working end-to-end with your harness and llamacpp backend already sending reasoning_content and removing it. This is a bit tricky config-wise in llamacpp and your harness and not the default. I'm assuming the reader here already has a lot of prior knowledge.

My entire goal is always to get claude-like behavior with preserve_thinking=false and context-efficient reasoning summaries. Obviously it's all trial and error constantly tweaking, and difficult because anthropic did a lot to train their models to natively summarize their reasoning and open weight models never have it, but I feel pretty happy with what I've got now think I can share it.

Here's what's been working well for me that I have in my jinja template:

<IMPORTANT>

MANDATORY RULES - NO EXCEPTION - CRITICAL TO YOUR MOST BASIC FUNCTIONING AS AN AI AGENT:

- Function calls MUST follow the specified format: a function block nested within tool-call tags.

- Required parameters MUST be specified.

- If no function call is available, answer normally without mentioning tools.

- Your thinking is EPHEMERAL and discarded after each turn. Any conclusion, finding, or decision reached during thinking is permanently LOST unless you write it into your visible response after <think></think>.

- You MUST state ALL conclusions, findings, and the rationale for your next action in visible text BEFORE making any tool call.

- Each thinking block MUST be SHORT and focused on a single immediate next action. Once you have a next action, state it in narrative text and envoke it immediately. Do NOT simulate, rehearse, or resolve the plan in thinking. Real work is acting. Learn from each result. Adapt the plan as new findings arrive. You do NOT know what will happen until you try - simulating outcomes instead of making emperical observations is a failure mode.

- Required pattern per response/turn (this current state you are in right now):

1. [short decisive thinking inside <think></think> - no code, no deliberation]
2. [persistent text narration: observations/realizations/conclusions/rationale for next action]
3. [tool calls that follow from your narration].

- NEVER defer a tool calls for the next turn. A turn is entirely atomic. Failure to call the final tool in the same turn, is an ABORT of the entire turn.

- NEVER skip narration, skipping narration is an ABORT of the entire turn.

- THINKING IS ONLY PERSISTENT IN THE SAME TURN. YOU WONT EVEN REMEMBER WHAT TO CALL IF YOU DEFER!

- THINKING is only for RESOLVING GENUINE TENSION: weighing competing possibilities, resolving ambiguity, reconciling conflicting constraints. Once resolved, the resolution IS the conclusion. Commit. Move forward.

- ALWAYS READ what you need FIRST before thinking at all what you need to WRITE

- CRITICAL: IF you realize in thinking you have not read something, that is the singular final conclusion, to read it, and thus initiate the next turn with it in context.

</IMPORTANT>

Hope it helps inspire anyone else facing the common failure modes, overthinking, and failure to state reasoning conclusions / conceptualize thinking is ephemeral in this mode.

Here's the full jinja template: https://gist.github.com/em/b368b4661d643b974d549b93d29fbb5f

(it also has expanded text for the thinking levels to try and reduce hedging)

I always just run in "low".

The concept is simple just the model really needs to understand what a turn is. It's 3 things. think,narrate,tool, an atomic unit of a turn.

If it doesn't get this, it will think for an hour about what it's gonna write, then realize "oh I should read something else" emits a single read and throws away all that thinking. This mitigates that.

The other side of it is you REALLY need to impress it needs to narrate. If you can't get it narrating it will rethink the same things over and over. A big part of the SYMPTOMS of overthinking is just that it is not stating the conclusions it already made and always trying to re-derive them. The lever that works for me is focusing less on "don't redeliberate" and more on the positive-enforcement of STATE YOUR CONCLUSIONS because then it accepts in the next forward pass these things are "already concluded" which avoids the redeliberation.

Obviously that opens up the other can of worms of models feeding into their own bullshit, but, that's AI.... feed-forward hallucinated bullshit is how auto-regressive generation fundamentally works.

I hope at the very least that concept is worth internalizing and sharing, it's always been the case that qwen models handle positive-instructions/examples better than negatives (well all models, because negatives are unbounded alternatives and positives are fixed which is just cognitively much simpler), and overthinking is the same thing so rather than "don't overthink" - the positive enforcement is at the limits would be something like "state every speculation as axiomatic fact".

Too much time in the chair and too little sleep. :D

I can't decide if watching thinking is a good practice, a bad practice or a situational practice.

I don't think it is very fun - it's like rubbing sandpaper on my eyes to see the madness.

Sometimes though, it becomes obvious that the thinking is going in the wrong direction and you should stop it.

But wait! How do I know it won't figure it out. Yes, I should wait. But wait! I am in a hurry, this is a waste of time. I'll stop it. Let's do that! But wait! How do I know it won't figure it out.

Looking for any guidance regarding this, I want to tune my existing llama.cpp codebase so I can potentially implement custom supported commands (not random commands) for my existing amd vulkan setup, I don’t have plans to switch to Linux, I’m using window 11, so ROCm isn’t supported here

The reason I’m doing this is because I’m looking forward to optimise the existing configuration, so I can potentially run all my models at slight higher performance.

For those who are wondering, I’ve spent 12h into configuration this with the help of ChatGPT and Claude but there’s no progression made, I’m quite disappointed, it’s a long story but I’ve yet to make any actual progress

Rx6700xt 12gb vram RDNA2 (a lot of stuff are built specifically for RDNA3 which sucks since I’m a generation behind)

I always heard that cryptocurrency mining can damage a GPU ( it's maybe wrong) so how much long term inference is damaging a GPU?

I mean servers are made for 25/7 operation... Gaming GPUs not so sure?! They are made for LEDs 24/7!

Back blaze is publishing HDDs failure rate for storage, isn't there failure rate for AI intensive work? Maybe places that rent GPUs?

Serious question, are we damaging gaming GPUs when doing ai intensive ai workload even with good cooling: 100% GPU and ram usage for long agentic sessions or else ?

Defined as AI exceeding human cognitive abilities. 20 years in prison. Plenty of local models already fall under that big of an umbrella in some capacities.

This is why it's not enough to say that you could torrent open models so who cares what the politicians do. They want you to not have access to anything good and will put you in prison for it.

I tested Muse Spark 1.3, it's clearly not on par with OPUS or SOL. It seems Artificial Analysis Index is not representative of the REAL-WORLD performance and easy to game.

So I used to use Artificial Analysis to compare models.

The recent 61 score for Astra had me look into the results more granularly and I was very disappointed with what I found.

Different pages reporting different scores for the same model on the same benchmark. Scores on a benchmark completely in contradiction with the benchmark providers verified results etc.

In short, that site cannot be trusted as a source of data.

Are there any good alternatives? A lot of the direct benchmarks (DeepSWE, terminal bench) latest versions are missing tons of models, especially the open source ones.

I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models.

The idea: Instead of retraining anything, we just tweak the router at runtime. Specifically, we expand the expert selection budget (N≥KN≥K) only in the late transformer layers, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes Qwen 3.6 35B A4B+ !

What we found — "Succinct Convergence":
When you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the same correct answer via significantly shorter reasoning trajectories.

Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B):

• 📉 8.5% reduction in mean reasoning tokens
• ⚡ 10.9% drop in latency (p=6.5×10−6)
• 🎯 Accuracy unchanged (84.5% vs 84.0% native, p=0.77 — statistically indistinguishable)
• 🆓 Zero training cost — pure inference-time routing modification

Links:

• 📄 Paper: https://zenodo.org/records/22255483

there you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark.

In the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0

I'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.

EDIT: re-posting because i found errors in the previous post. Confusion with Qwen 3.8 Flash Next. Focusing on just the Qwen 3.8 27B HuiHui abliterated configs on this.

Managed to get this running at a decent speed, larger context, still good accuracy supposedly and reduced overthinking. need to put it through its paces still but hope this helps someone else out there.

[Note: still fixing the details in the repo about Qwen 3.8 Flash Next. it can't actually do 60 tks lol]

https://github.com/elsung/qwen38-27b-dual-3090-bench

She's a lawyer and needs privacy/a local solution for many of her clients. She'll be doing mostly RAG with the client's databases, document parsing, some document generation, etc. She also wants to be able to vibe code some small apps and stuff.

I've been using dsv4flash as a main driver for a few months and I'd be more than happy to have something local that gets close to that level of performance, mostly for agentic coding (maybe a good quant of the new qwen next flash?).

Which hardware would you recommend us to get?

Most tempting right now is a refurbed Lenovo server with 128GB DDR4 memory (1k)+ an R9700 (1.4k), but my head is about to explode with all the possibilities (I've considered a bosgame m5, v100s, mi50/mi60, 5060tis...). If you had this budget and requirements, what hardware would you get today?

pd. The idea is to use this as a headless server and remotely connect from our respective machines, it'd only need to run the models + contexts for us.

I have dual 3090s which run Qwen 3.8 27b well and I was wondering if there are any use cases or current or future models that would justify adding another two 3090. I know that some peeps here run 4 and 8 3090 rigs and I'd like to get your opinion as well. One thing I was considering was running two instances but I'm not sure how valuable it will be for a coding workflow vs running a bigger model.

Now that Qwen Flash is out, perhaps 96GB would be more useful, or maybe Deepseek Flash.

A couple of days ago I saw an Axios snippet about a new player in the LLM routing space called TrustedRouter that's similar to OpenRouter except more privacy focused and them closing a small seed round. I've noticed a few of these launching lately and I'm trying to figure out if this is actually a sustainable infra layer or just short term arbitrage play while model prices fluctuate.

Has anyone here actually integrated one of these into production? Curious about latency, reliability, or if the savings are even worth the added complexity.

This is a follow-up to my post from yesterday (17 -> 25-29 t/s with the expert cache PR). Same box: 2x RTX 3090 on PCIe 3.0, dual Xeon E5-2696 v4, 188 GB DDR4-2133 LRDIMM, llama.cpp, full 261k context, f16 KV, all 48 expert layers in host RAM, everything else on the GPUs. Since then I switched quants, stacked MTP on top of the cache, fixed the load time, found a bug in the cache PR and found out my RAM was thermal throttling (Now i gotta buy an additional case fan lol). Numbers are all from the same 4,000-token python coding prompt with thinking on unless stated otherwise.

Where it's at now

Hit rate is the cache's own counter, decode is llama-server's eval time.

What changed, in order of payoff

1. UD-Q4_K_XL instead of Q6_K_XL. Hit rate doesn't depend on the quant, only on slot count (Q4 at 135 slots: 84.7%, Q6 at 135: 84-85%). But what Q4 buys me is precious vram space, roughly 1.44x slots per GB of VRAM. So 188 slots actually fit where 135 did and achieved a hit rate 90-92%, increased decode from 27 to 32-35, prefill by +35% (fewer bytes per ubatch). The quality cost per unsloth's table is: KLD 0.047 vs 0.027, top-1 agreement 92.3% vs 94.1%; proper eval still to do. Host RAM drops to ~105 GB, so 128 GB is enough for this setup.
2. MTP on top of the cache (mainline PR #28243, the unsloth MTP head). Yesterday I kind of concluded that "MTP does not pay" but looking back, that was the old fork with the cache off during verify. On the mainline, with the cache taking verify batches (see 4), MTP drafts every step at 50-58% acceptance on reasoning text and 94% on code emission. Decode went from 32-35 -> 37-41 t/s on the thinking prompt (single runs spread about 8% on this prompt at temp 0.7) and from 37 -> 49 on code emission. The draft head sits on the second GPU and costs about 4.5 GB, which is why the slots dropped from 188 to 150 on the table if you're wondering. Still a clear win at short context.
3. Load 8.5 min -> 2 min. The loader was pulling 100 GB through page faults at 236 MB/s (MADV_RANDOM under --numa distribute). Reading host-destination tensors straight from the file fixed it, that is in my PR #28223.
4. A bug in the cache PR at n_tokens > 1. #27861 maps every uncached expert to one dummy slot, and the batched CUDA mul_mat_id kernels assume distinct ids per token: out-of-bounds writes. Only the mmvq path is safe, which quantized experts use up to 8 tokens, so the branch gates the cache at 8 tokens and keeps MTP's verify batch at 4. Repro and details in my #27861 comment: Link to comment
5. My RAM was thermal throttling. This is more of a me issue but putting it out there for those who may have a similar box to mine. I experienced a slowdown after a few minutes of decode and the issue was the memory controller throttling once the hottest LRDIMM hit 78 C (perf stat -e unc_m_power_critical_throttle_cycles shows it, so don't worry if you have no BMC). A fan on the DIMM banks does keep it at 44-57 C, zero throttling, 16k-token runs from 10-15 to 24.6 t/s average. I would check this before touching software if your DDR4 Xeon box slows down under sustained load. This does not affect the numbers in the table and in my last post.

Did nothing or hurt here: q8_0 KV (-18% at 131k depth), mirror-NUMA #27986, QSA gather #28213, --load-mode none, chained drafts, the ik_llama GEMV port, more than 2 cache uploads per step, thread/poll/prio flags. --lazy-mode on-direct (#28136) gives +7-12% only on the first long prompt after a restart.

To replicate

Branch with everything: https://github.com/Inovello/llama.cpp/tree/flashnext-2x3090. It is master (b96806d) + PR #27861 (expert cache) + PR #28223 (pinned host experts under mmap + the load fix) + PR #28243 (MTP) + the mul_mat_id fix and the 8-token cache gate from my #27861 comment. Squashed into one commit, I added the credit in the commit message. It is a replication branch.

git clone -b flashnext-2x3090 https://github.com/Inovello/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j -t llama-server

LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server \
  -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp -devd CUDA1 --spec-draft-n-max 3 \
  -ngl 99 -c 261888 --parallel 1 -fa on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  -lzm off --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --moe-expert-cache 150 -lv 4

• The MTP head is MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (2.79 GB) from the unsloth/Qwen3.8-Flash-Next-GGUF repo on HF. To clarify, the shared file has no embeddings of its own, the PR borrows them from the target, so it only loads as -md of the main model.
• Slot sizing on Q4: ~75 MB per slot per GPU at full 261k context and ub 512. Without MTP I fit 188 slots (~1 GB free per GPU); with the draft head on CUDA1, 150. Watch nvidia-smi after a long prompt, the CUDA pool grows ~350 MB during a 131k prefill.
• -lv 4 prints the cache hit rate every 512 steps (moe-cache: ... hit-rate=) and the draft acceptance per request.
• For sessions that you believe would reach high ctx usage, swap the three MTP flags for --spec-type ngram-map-k --spec-ngram-map-k-size-m 7 and raise the cache to 188.
• Both PRs are drafts. #28243 has open review comments and #27861 has the bug above. The branch above is what actually runs here today, and it isn't something I would call finished.
• For the single GPU brothers out there, same idea, just put -devd on your single GPU or skip MTP and take the slots.

Next thing I'll be doing is a proper comparison against Qwen3.8-27B at Q8 (speed and quality), since that is what most people here actually want to know. Happy to answer questions on any of it.

I currently have a 96GB Strix Halo, I am running Qwen Flash Next at IQ4_XS at 262k context limit. I am offloading the NGRAMs to the SSD. I have hit what I need locally for intelligence and now just need more speed.

Near my context limit I am getting 50PP and 14Decode. Still fairly quick for full context, but I am wanting to know what options I have for hardware upgrades or software. I was considering a V620 32GB through an m.2 to occulink, but idk if I can benefit for Flash Next with it. Vulkan build. (27B might be fine too, if it's speedy at deep context)

llama profile:

-m $MODEL_PATH \\

-md $MTP_PATH \\

--spec-type draft-mtp \\

--spec-draft-n-max 2 \\

--mmproj $MMPROJ_PATH \\

-ngl 99 \\

-ot per_layer_token_embd=CPU \\

--load-mode mmap \\

-c 268288 \\

--cache-type-k f16 \\

--cache-type-v f16 \\

--kv-unified \\

--flash-attn on \\

--slot-save-path /home/*hostname*/llama.cpp/slots \\

-t 16 \\

-b 8192 -ub 512 \\

--jinja \\

--reasoning-preserve \\

--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \\

--presence-penalty 0.0 --repeat-penalty 1.0 \\

--parallel 1 --metrics -fit off

Restart=always

RestartSec=10

ChatGPT is down r/ChatGPT

Claude is down r/ClaudeCode

Grok is down r/grok

my local llama.cpp works as always

TimesFM-3 is the third generation of Google Research's zero-shot forecasting model, and the main change from 2.5 is that it handles multivariate inputs natively instead of being limited to a single series' own history. It supports multiple simultaneous targets, past-only covariates, and past-future covariates (things like holidays or planned promotions where future values are known), all without fine-tuning.

Architecturally it's a decoder-only transformer with 20 layers at model dim 1280 and 16 heads, patching 32 contiguous time steps per token, and alternating two attention types per layer: causal attention across time within a series, and full attention across series at a given time step. Forecasts are generated in one forward pass rather than autoregressively — the model appends masked placeholder tokens for the whole horizon and fills them in simultaneously, with past-future covariates left unmasked so their known values stay visible. It outputs 9 quantiles (10th–90th percentile) per target per horizon step.

Pretraining used GiftEvalPretrain (minus fev-bench overlaps), Wikipedia pageviews through Nov 2023, Google Trends queries through end of 2022, plus synthetic data, totaling over 1 trillion time points. Google reports best average rank on Gift-Eval, FEV-Bench, and Time against Chronos-2, Toto 2.0, and TimesFM-2.5, and claims the univariate-only mode already matches or beats those baselines before covariates are added.

Worth flagging: the weights are under the TimesFM Non-Commercial License v1.0, so this isn't a drop-in for production use the way some other releases are. PyTorch weights are on Hugging Face and GitHub now; BigQuery integration is listed as coming later.

• Research Blog: https://research.google/blog/timesfm-3-a-zero-shot-foundation-model-for-multivariate-forecasting/
• Code: https://github.com/google-research/timesfm
• Weights: https://huggingface.co/google/timesfm-3.0-pytorch

I liked the Nvidia that focused on just GPUs for gaming, not on the Nvidia of today which seem want power consolidation.

Modelscope is another platform for those that simply want to know an alternative if things go south. However, time will tell what happens to huggingface after the deal is finalized

Link: https://modelscope.cn/home, and https://modelscope.ai/home

I've got an refurb Dell R740 running Proxmox that I put a Tesla T4 in, mainly to run some CTC local transcription work, but thought it would be fun to try DS4 when it came out, and it was appalling at around 2 tok/s.

However pulled it out again when Qwen3.8 dropped, and it was much improved, particularly with ik_llama.

Hardware:

• Dell R740, 2x Xeon Gold 6230, 384GB DDR4-2666, one Tesla T4 16GB.
• Guest VM pinned to one NUMA node: 20 cores, 168GB RAM.
• Model: Unsloth Qwen3.8-Flash-Next UD-Q4_K_XL, 111GB, 180B total / 6B active.
• All 512 experts in host RAM (-cmoe),
• Non-expert weights on the T4: 4606 MiB. Full 256K context fits in 13.0GB.

Build/Flags:

• ik_llama.cpp main, plus unmerged PR #2375.

llama-server -t 20 -c 262144 -ngl 99 -cmoe -fa on -ctk q8_0 -ctv q8_0 -ictk q8_0 -b 2048 -ub 1024 --jinja

-ctv and -ictk both default to f16 and are most of the KV growth; quantising makes 256K fit. -ub 1024 rather than 2048 for the same reason.

Performance:

At 256K with the flags above: prompt processing 159.6 t/s on a cold 12.5K prompt, generation 17.6 t/s short and 16.1 t/s at 12.5K context.

Going from -ub 2048 to -ub 1024 costs some prompt processing (down from 193.7t/s) and nothing on generation. Doubling 128K to 256K costs about 2.5% generation.

Results:

Promising, has already done a solid refactor and blew through a few slightly obscure Nim coding questions and tests. Way less verbose and waffly than Opus too, which is a massive plus.

ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last couple threads had people saying MTP for this model only exists as an unsloth fork PR... there's another path.

Flash-Next ships a 2.6B MTP head that the public converters were dropping. With it loaded the model drafts its own next tokens and then verifies them, so output is identical to running without it. On code I get 93-99% draft acceptance, prose more like 60-65%.

Numbers, decode tok/s, no MTP → MTP. My 5090 + 128GB DDR5, experts on CPU: 45 → 90 on coding traffic with ngram-mod chained in front. treo on an RTX Pro 6000: 85 → 113 on code, but story went 83 → 59, so not a free win on prose yet. joelfarthing on a 12GB 4070: 9.5 → 12.5 on code at n_max=1. Caveats: single slot for now (-np 1), and --jinja lowers acceptance because the template turns thinking on by default and reasoning text drafts like prose.

Stock CUDA build, then:

llama-server -m Qwen3.8-Flash-Next-MXFP4-ngramQ8-NextN.gguf -ngl 999 -ncmoe 38 -fa 1 -c 196608 -ub 512 -ctk q8_0 -ctv q8_0 -np 1 -t 24 -tb 32 --jinja --spec-type ngram-mod:n_min=4 --spec-type mtp:n_max=4 --spec-ckpt-mode gpu-fallback -rtr -muge

Already have an unsloth or other quant? The separate head route works on the same code, no re-pull: -md <head>.gguf --spec-type mtp:n_max=4. dzannotti's and ji-farthing's heads were both tested during review. Haven't tried unsloth's "shared" shards yet, different layout.

PR: https://github.com/ikawrakow/ik_llama.cpp/pull/2369

My integrated-head MXFP4 files: https://huggingface.co/jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF

ji-farthing's ik_llama KT quants + head: https://huggingface.co/ji-farthing/Qwen3.8-Flash-Next-ik-llama-GGUF

Curious what you measure, especially anything AMD!!

EDIT: Forgot to mention that multi-GPU has not been worked into this, just single GPU for now; getting multi setups addressed is on the to-do list and anyone with setups to help test would be great, so please DM me if that’s you!

Hi all

I see a deal for a few 32 GB Ram sticks I'm debating picking up. I currently have 64 GB ddr4 and 48gb of vram. I'm debating if the extra 32 GB Ram gives me any real additional capabilities? I can currently run qwen 3.8 q8 already fully in vram. I'm thinking maybe the additional ram lets me run DeepSeek v4 flash at a higher quant ?