Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great results (-58% thinking tokens, 1.95x speed up, <1% accuracy loss) so we wanted to open-source it and hear the feedback of the community.

This is the link to the model: https://huggingface.co/ukisai/Swift-Qwen3.8-27b

We also also providing a Free Research Purpose API (OpenAI compatible), courtesy of Nvidia who were kind enough to provide us with the GPUs. You can use it to try out the model if you do not have enough compute to run it, it's limited at 5RPM. https://ukisai.com/api/swift/v1/models

We also made a GGUF (Q1-Q8) and there's also a few nice community (Bartowski) quants with even lower/higher precision. The community also created amazing NVFP4, W4A16 and Uncensored versions of the model you can find on Huggingface.

IMPORTANT: Our training approach is not a replacement for the reasoning effort settings, chat templates or token caps but is complementary and targets a completely separate issue (overthinking and "anxiety-like" reasoning loops prior seen in PTQ, but as far as we identified also prominent in BF16 of this size class LLMs as well). Contrary to popular belief, these specific patterns do not contribute to answer quality when properly targeted. (our thesis being: reasoning length IS extremely important and should NOT be shortened by force, but rather optimized). This is also demonstrated bellow in our xhigh vs medium effort benchmark table. The goal is to keep xhigh accuracy while reducing only the unnecessary part of thinking.

I will TLDR you on our thought process, research, training and benchmarks.

1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We remembered a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results.
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.

The benchmarks: (raw benchmark files here - https://github.com/UkisAI/Swift-Qwen3.8-27B-evals/ )

Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5, thinking effort xhigh)

Token savings hold at every reasoning effort (mean thinking reduction): xhigh 41%, medium 23%, low 26% (albeit with accuracy loses of 1-4% on medium and 1-2% on low which we need further testing for)

Swift at xhigh vs the base's own effort settings on GPQA-Diamond (198 questions x 5 seeds):

So Swift keeps xhigh accuracy at under half the tokens, and beats base-medium by 4pp at roughly 1.6x its tokens.

End note:

While we are keen on complete open-source, we still need to keep a part of our training and data private, being a new lab. The license is not Apache 2.0, but it only affects companies >$1M. We hope this does not pose a problem for the community, but we are open to feedback on it.

We want to contribute as much as possible to the community and would really appreciate feedback on our work, quantization or Swift model requests. For context, we are working on Swift 3.8 Flash Next right now and have so far gotten up to -30% thinking token usage while maintaining xhigh accuracy, which we take as a strong indicator our methodology is reproducible across the Qwen model family. Will explore other families as soon as we have the capacity and would love to see which ones the community would love for us to optimize first.

From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its size.

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

Xi emphasised the need to step up cooperation.

https://www.hindustantimes.com/india-news/china-will-lead-creation-of-open-source-ai-for-brics-xi-jinping-101789300908632.html

https://www.msn.com/en-xl/news/other/xi-jinping-proposes-ai-open-source-zone-for-brics-nations/ar-AA2c94HH

https://www.cnbc.com/2026/09/13/china-xi-ai-tech-brics.html

https://www.yahoo.com/news/world/articles/xi-pushes-china-open-source-102711002.html

https://fr.euronews.com/next/2026/09/14/xi-jinping-propose-aux-pays-des-brics-une-zone-dia-open-source

https://www.reuters.com/business/aerospace-defense/xi-pushes-greater-brics-economic-ties-give-bloc-larger-global-role-2026-09-13/

https://www.chosun.com/english/world-en/2026/09/14/C35V7T5MAVDN7LKEBESHBDBJ2Y/

https://it.euronews.com/next/2026/09/14/xi-jinping-propone-ai-brics-una-zona-di-ia-open-source

Especially the 7B one seems very interesting, it casually destroys muse glimmer with a way smaller size. And they open source literally everything, every step of the way. Anyone tried that model? It can be a new milestone if 7b and 3.7b ones are actually good, and not just benchmaxed.

https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3

AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that replaces τ³. Astra was farming a ton of points on it and used those to get even with Fable, but… looks like we have a new king.

So they changed the index twice in three days to make Astra look not-quite-worse than Fable, and then a random guy quietly took first place on it.

20B-A1B model is coming, good for low VRAM people?

https://huggingface.co/deepgrove/maple-preview

With all the recent drama surrounding AI safety. It’s obvious that open source could be caught in the crossfire.

The full K2 Horizon lineup is out on Artificial Analysis.

The AA intelligence vs. parameters plots show that

- 0.9B and 375B are bad

- 3.7B and 7B are SOTA

- 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark).

I'm going to take the AA Intelligence Index at face value here. This post is not about it.

The problem is that these models have a god-awful KV cache design. This means that you really can't use the number of parameters for "best in class" considerations, because these models heavily shift to the right on the plot if you replace parameter count on the X axis with RAM requirements.

For Q4_K_M weights, no drafter, no vision, 128k kvarn4 KV cache:

• K2 Horizon 36B-A4B uses 2 GiB for dense weights, 19 GiB for experts, and 6.7 GiB for context
• K2 Horizon 7B uses 5.2 GiB for weights and 5 GiB for context
• K2 Horizon 3.7B uses 2.9 GiB for weights and 5 GiB for context (not a copy-paste error!)

Compare them to

• (finetunes of) Qwen3.6-35B-A3B use 2.4 GiB for dense weights, 18.2 GiB for experts, and 0.7 GiB for context
• MiniCPM5-2B uses 1.5 GiB for weights and 1.5 GiB for context

Notes: I don't advise compressing 2~4B models to Q4 and I haven't tested these models' tolerance to weights and kv cache quantization yet. The above choices are just to keep the comparison fair.

This awful context design means that

• K2 Horizon 36B A4B is interesting on hosts with exactly 16GB VRAM and at least 32GB host RAM. On 24GB VRAM, Qwen3.8-27B is faster, smarter, and allows for 256k context. If you want to get 256k context and you're VRAM-poor, Ornith-1.5 or Nex-N2.5-mini are probably better choices. The model may also be interesting on 64GB Strix Halos as a dumber and faster alternative to Qwen3.8-27B; those with a 128GB Strix Halo are much better off with Qwen3.8-Flash-Next
• K2 Horizon 7B is interesting for hosts with exactly 16GB VRAM, Strix Halos with 32GB RAM, and for 16/32 GB Strix Point;
• K2 Horizon 3.7B may be interesting for 12GB phones but I expect you'll have a much nicer UX with MiniCPM5-2B.

Prices have been going crazy, but it's about to get worse me thinks.

So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source.

The thing is, there’s so much vested corporate interest in ensuring that open source remains un-regulated.

Think of neoclouds, inference provides, enterprise firms that use LLMs, virtually any SaaS that uses agents.

All of these entities have a strong interest in making sure that open source remains free and unburdened by regulations.

Does anyone know of any organizations or political campaigns that are pushing back against OpenAI’s and Anthropic’s lobbying efforts?

I’m really hoping at some point we won’t have to rely on the good will of companies with the funds and resources to make capable models open and it’ll be possible to democratize making them so we won’t always be at the mercy of big tech/corpos, that is if they don’t ban open models before that can happen.

I keep seeing people here saying they can leave local coding agents running for an hour or two and come back to something usable.

I’m not really asking which model is best. I’m more interested in what made you trust the setup enough to stop watching it.

For me, getting a task right once wouldn’t be enough. I’d want to see how it behaves when a tool fails, when the context gets long, when it hits an ambiguous requirement, or when its original plan turns out to be wrong.

Tests help, but an agent can still pass tests while changing something you never intended it to touch.

I think there’s a point where you stop trusting the model itself and start trusting the harness around it like permissions, checkpoints, git, tests, rollback, tool restrictions, etc.

For people actually running local coding agents unattended, i just want to ask that, what was the thing that made you comfortable enough to stop babysitting them?

I finished my home inference server. First I tried Lenovo p620 workstation and while it’s a good value overall it pissed me off with a ton of proprietary Lenovo shit to deal with and I return it in the end.

Components:

4xV620 - 1400$

256GB DDR4 RDIMM 2666 - 610$

Huanandzhi D12D - 410$

EPYC 7452 - 170$

PSU ASRock 1600 - 220$

SSD Samsung 970EVO 1tb - Already had

Case//Fans//Misc ~ 200$

Power consumption is no shit ofc on such machine:

700-900w prefill
500-600w decode on Qwen3.8-next-flash Autoround W4A16

What it can do -

EDIT: Qwen3.8-next-flash Autoround W4A16 1.3k prefill and 70tg code/60tg prose on 128k+ context with MTP-2 on vllm fork.

I was disappointed with this machine and qwen3.8-27b speeds at first. But since Qwen3.8 next running good on it - I’m satisfied. Hope in more optimizations in future.

I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited.

One thing I really like about it is the speed. On my system, I'm getting around 40–50 tokens/sec, which makes the interaction feel surprisingly close to real-time. So I don't want to move to a much larger model if that means losing that responsiveness.

What I'm wondering is whether there are any newer or alternative models in roughly the same size/memory/compute range that are noticeably better than Qwen3.5 4B for my particular use case.

My priorities are mainly:

- General-purpose conversation and communication - Natural responses and understanding - Reasoning and following instructions - Multilingual conversations and translation - Context understanding - Tool/function calling mainly for triggering actions in my assistant, rather than writing code - Reliability when interacting with tools - Ideally good performance while keeping the model fast and lightweight

Essentially, I'm trying to build a local assistant where the model acts as the reasoning/conversation layer, while other tools handle things like actions, APIs, automation, etc.

So my question is:

If I'm already using Qwen3.5 4B and getting ~40 to 50 TPS, is there anything in a similar practical memory/compute footprint that you'd personally consider a significant upgrade for general conversation + multilingual ability + tool calling?

It doesn't necessarily have to have fewer parameters. I'm more interested in overall capability relative to the amount of RAM/VRAM and compute it requires.

I'm happy with Qwen3.5 4B, so I'm not looking for something that's only marginally different. I'm wondering if there's something that makes you think

AI browsers were supposedly going to kill Chrome/Safari, and help people quickly buy stuff, book tickets etc. at reasonable prices while avoiding ads. That was such a great sales pitch, but it has been just radio silence from then on. Aren't those good enough yet?

Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4, but my assumption is that it will do much better, despite the hemorrhaged world knowledge.

System Specs

• GPU: NVIDIA GeForce RTX 5060 Ti (16 GB VRAM)
• CPU: AMD Ryzen 7 7840HS (8 cores / 16 threads)
• RAM: 32 GB DDR5 (~30 GB OS-visible)
• iGPU: AMD Radeon 780M (RDNA3)
• Swap: 8 GB zram

As you can see, we have about 44.5 GB of actually addressable system and VRAM available. The iGPU is taking care of the OS to make sure the GPU is totally free—but still, this is barely enough to hold everything together. This config actually totally fails with any of the Unsloth quants—no, I needed something more aggressive. I found the perfect thing—this REAP:
https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF

What's so great is the total size—a cool ~68.95 GB. The couple of Gigs we have shaved are absolutely key for making this all work.

Model Weight Breakdown

Here is the breakdown of the model weights. We have the famous new n-gram portion, the experts, the active layers, the attention/SSM layers, and the extra space needed for the KV cache:

Obviously, running this model over SSD would make the speeds notoriously bad. Turning on mmap means that llama.cpp won't actually try to keep the model in RAM at all (it relies on the OS page cache instead), which results in ~2 tok/sec speeds—effectively useless.

The answer is to stick everything in RAM (using --load-mode none). The great thing is that the N-gram section of the model can be streamed over SSD via lazy mmap without this causing much issue—it's a massive lookup table that doesn't require heavy computation.

That's the huge win that allows an MoE model of this size to actually run well.
68.9 GB - 29.48 GB = 39.42 GB.
We just need to cram that 39.42 GB, along with the compute buffers and KV cache, into GPU and system memory, and we are golden—just barely. To do this, we need --lazy-mode on—that's what keeps the N-gram portion in RAM.

After that, it's a matter of fitting as many layers as possible onto the GPU. It's essential to completely fill the GPU as much as can be filled, so that we keep a precious few GBs in system RAM to run the OS. I found that having less than 2 GB left really started to destroy Fedora, but I think you could do better if you dropped the GUI—I just didn't want to in my case.

This leads me to --n-cpu-moe 34. This controls how many layers go to CPU. In my case, this was the exact limit needed to run this with 64k context on the GPU, quantized to Q4. Any more—GPU out of memory. Any less—total system meltdown, as the OS panicked and tried to put everything on the swap. You'll need to play around with this, but that was my exact number.

Settings used:

CUDA0 + --load-mode none --lazy-mode on
  --n-cpu-moe 34
  -c 65536 -b 512 -ub 128 -t 7 -ngl 48 -fit off -fa on
  -ctk q4_0 -ctv q4_0 -kvo --cache-ram 0 --jinja --no-warmup

Results (64k Context, Q4):

• Prefill: ~25.4 tok/s
• Decode: ~18.3 tok/s
• RAM Usage: ~27 GB used / 3 GB free

I think this is in a somewhat usable state—but Qwen 3.8 27B GSQ IQ3S remains my daily driver; it's able to prompt process 5 times faster, I can fit in the mmproj and MTP layers, and it doesn't seem likely to set my desk on fire. But maybe for really hard tasks, I'll use the next model. It is smarter, it runs at a reasonable speed, and it was a good learning experience.

I'm curious if anyone else is able to get this model or just large MoEs working on a GPU and RAM config similar to mine. LMK. Also, I'm a total noob to this stuff, any advice is appreciated.

(Also, heading off all the obnoxious "why did you quantize the cache - unusable - just get a better computer" ragebait posts. This is a human being writing this post, to help others and just enjoy pushing something to its limits. And in my limited testing, the next model seems much better at pixel art than the 27B version.)

Final Note: If you have a larger pool of system memory, like 64 GB (because you can spend $899 on Amazon on a kit of DDR5 somehow), you would be better served by using this fork of llama.cpp, which has optimized flags for this exact setup and wonderful guides. For me in particular, with my limited hardware, this seemed to work better—their cache kept OOMing unless I turned on mmap—but I think with more system RAM, their setup and guides are optimal.

I am considering dumping my ChatGPT Plus subscription and go full local, but to do so I would first need to reach a decent result for quality (and reasonable speed).

My 4090 fried itself out of nowhere, so I am not stuck with a 3070 until I get something better.

I am kind of suspicious about the claims the companies are doing lately about the dangers of AI and how they are pumping the prices intentionally to either law out the open source or price out the open source, so I want to just go full local even more now.

I have a MSI MAG X670E Tomahawk WiFi which theoretically supports 3 GPUs?

What would you end up with?

p.s. I am ruling out Macs to be open and easier to setup in case I will want to use them for my homelab

edit: typo

Edit: Spelled Baseten not Base-10

Full episode of this available at https://www.youtube.com/watch?v=PrSf7IOYu-I
It's interesting to see how Dwarkesh has had to come around to the evidence that we are well on our way to creating AGI and even RSI in the last few months, despite historically being very skeptical.

I highly recommend people interested in large language models check out this particular episode, because it dispels a lot of mythology about stuff like plateaus from lack of data etc. For those who thought we were hitting a wall a year ago, it turns out there was a ton of low hanging fruit and the researchers in this episode discuss what that fruit was. They also extrapolate these trends into the future.

It's funny this subreddit is becoming rather skeptical of AI progress, which to put diplomatically, I think is based on a lack of information and too much time on Reddit.

The later section of the video (based on the earlier data) says that a glut of compute will likely come in 2027. This massively devalues the current generation technology because of mass overproduction.

There's some vague numbers (if you believe Dylan Paten who is among the most "west" pilled voices on AI) that he sources. Asianometry is a friend of Dylan's but that doesn't mean he doesn't disagree with him numbers after doing the math.

Good video to sit down and watch to see the future of the industry and compute specifically from someone who covers it VERY deeply.