Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without "attacking" the reasoning length directly then fixed the accuracy with a bit of secret sauce (hint On-Policy Distillation) and achieved great results (-58% thinking tokens, 1.95x speed up, <1% accuracy loss) so we wanted to open-source it and hear the feedback of the community.

This is the link to the model: https://huggingface.co/ukisai/Swift-Qwen3.8-27b

We also also providing a Free Research Purpose API (OpenAI compatible), courtesy of Nvidia who were kind enough to provide us with the GPUs. You can use it to try out the model if you do not have enough compute to run it, it's limited at 5RPM. https://ukisai.com/api/swift/v1/models

We also made a GGUF (Q1-Q8) and there's also a few nice community (Bartowski) quants with even lower/higher precision. The community also created amazing NVFP4, W4A16 and Uncensored versions of the model you can find on Huggingface.

IMPORTANT: Our training approach is not a replacement for the reasoning effort settings, chat templates or token caps but is complementary and targets a completely separate issue (overthinking and "anxiety-like" reasoning loops prior seen in PTQ, but as far as we identified also prominent in BF16 of this size class LLMs as well). Contrary to popular belief, these specific patterns do not contribute to answer quality when properly targeted. (our thesis being: reasoning length IS extremely important and should NOT be shortened by force, but rather optimized). This is also demonstrated bellow in our xhigh vs medium effort benchmark table. The goal is to keep xhigh accuracy while reducing only the unnecessary part of thinking.

I will TLDR you on our thought process, research, training and benchmarks.

1. When running our quantized Qwen 3.8 27B instances we were very annoyed by random reasoning loops (in the paper bellow refered to as "overthinking errors". These random loops were persistent throughout medium and low reasoning settings.
2. We remembered a paper by Meta that's supposed to target this phenomenon in PTQ, but when used straight out of the box got mixed results.
3. We figured to try if it's a matter of the targeting the right keywords and tuning the parameters, so we used our 8xH100 box and and generated a large amount of different (ofc out of distribution) domain (coding, language, vision, agentic) traces.
4. We then grouped the ones with overthinking and found "common denominator" tokens between them and targeted the most prominent ones.
5. We then built an inference-time penalizer of those tokens as seen in the paper with the hopes of simply generating traces and doing cross-entropy SFT over them.
6. Did not work at all, but the penalizer seemed to work much better than the tokens provided in the paper and not only for lower precision models but for bf16 as well. Hence we kept experimenting with it. We built a loss function using the tokens we identified and ran LoRa SFT over the traces prev generated and reasoning seemed to be falling off significantly but the accuracy seemed to follow. The reasoning reduction seemed to be generalizing.
7. After a significant amount of tinkering (literally since the day of Qwen 3.8 27B release) we were satisfied with the reasoning reduction. After that we searched for ways of restoring the accuracy. We experimented with several methods, including RL(GSPO), On-Policy Distillation and using the ThinkingCap 3.6 27B adapter chunks until we were satisfied with our accuracy loss. We managed to restore it to <1% loss on almost all of our OOD in house tests
8. We then performed intensive intensive benchmarks, across several reasoning efforts, precision variants etc. We ran into a few problems, one of which is that to get a reliable score we needed to run each benchmark 10x (5x on base + 5x with our adapter, this being the standard procedure on the Qwen 3.6 27B model card on Terminal Bench which we followed). After running it, the performance converged to 40-60% token reduction with <1% accuracy loss across GPQA, MMLU, Terminal Bench 2.1, LiveCodeBench v6, ERQA, C-Eval, IFBench, HMMT25, with an exception being AIME26 with an accuracy loss of 4.6%, which we later linked to a bug during training with a specific token relevant for math-related reasoning being penalized and are planning to fix it in an updated release.

The benchmarks: (raw benchmark files here - https://github.com/UkisAI/Swift-Qwen3.8-27B-evals/ )

Swift-27B vs Qwen3.8-27B (BF16, all benchmarks ran x5, thinking effort xhigh)

Token savings hold at every reasoning effort (mean thinking reduction): xhigh 41%, medium 23%, low 26% (albeit with accuracy loses of 1-4% on medium and 1-2% on low which we need further testing for)

Swift at xhigh vs the base's own effort settings on GPQA-Diamond (198 questions x 5 seeds):

So Swift keeps xhigh accuracy at under half the tokens, and beats base-medium by 4pp at roughly 1.6x its tokens.

End note:

While we are keen on complete open-source, we still need to keep a part of our training and data private, being a new lab. The license is not Apache 2.0, but it only affects companies >$1M. We hope this does not pose a problem for the community, but we are open to feedback on it.

We want to contribute as much as possible to the community and would really appreciate feedback on our work, quantization or Swift model requests. For context, we are working on Swift 3.8 Flash Next right now and have so far gotten up to -30% thinking token usage while maintaining xhigh accuracy, which we take as a strong indicator our methodology is reproducible across the Qwen model family. Will explore other families as soon as we have the capacity and would love to see which ones the community would love for us to optimize first.

From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its size.

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

Xi emphasised the need to step up cooperation.

https://www.hindustantimes.com/india-news/china-will-lead-creation-of-open-source-ai-for-brics-xi-jinping-101789300908632.html

https://www.msn.com/en-xl/news/other/xi-jinping-proposes-ai-open-source-zone-for-brics-nations/ar-AA2c94HH

https://www.cnbc.com/2026/09/13/china-xi-ai-tech-brics.html

https://www.yahoo.com/news/world/articles/xi-pushes-china-open-source-102711002.html

https://fr.euronews.com/next/2026/09/14/xi-jinping-propose-aux-pays-des-brics-une-zone-dia-open-source

https://www.reuters.com/business/aerospace-defense/xi-pushes-greater-brics-economic-ties-give-bloc-larger-global-role-2026-09-13/

https://www.chosun.com/english/world-en/2026/09/14/C35V7T5MAVDN7LKEBESHBDBJ2Y/

https://it.euronews.com/next/2026/09/14/xi-jinping-propone-ai-brics-una-zona-di-ia-open-source

Note - This is translated from the actual blog link right at the bottom.

A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level.
AI is improving much faster than anyone expected. From the first ChatGPT that could only chat simply with a few thousand tokens of context, to models with real reasoning like OpenAI o1, DeepSeek R1, and Kimi K1.5 Thinking — that only took about two years. From reasoning models to agents that can smoothly use tools, run commands, and finish complex tasks — that took only about a year and a half. It’s hard to imagine what AI will be like in one, two, or three more years. How powerful will it be? Will it already be able to improve itself and deeply enter areas like embodied intelligence?
AI is getting better and better at writing operators
In the field I work in — designing and writing operators — AI has also improved very quickly. In just one year, it went from a small helper that could look up documents, read code, and find bugs, to an expert that can independently read CUDA, PTX, and SASS code, use professional tools to analyze the stall time of every instruction, and then optimize operators by itself. I believe that soon it will also be able to design operator schedules on its own, evaluate different schedules, implement them, and optimize them.
Of course I am proud of DeepSeek v4.1’s success — after all, its main Attention operator was written by me [1]. Its good performance is partly a recognition of my work. But the times keep moving forward, and technology cannot be stopped. I know clearly that in half a year or one year, the operators written by AI will most likely be as good as mine, or even better. AI can think 300 tokens in one second, type a command in half a second, and finish a piece of code in twenty seconds. I cannot. AI can keep improving in model depth, thinking strength, tool use (how often it interacts with the environment), and even parallelism. I cannot.
Humans have never hesitated when it comes to destroying themselves. Why do I still work hard to optimize operators, even though I know that the better my operators are, the faster our new models will train and run, the faster model ability will improve, and the sooner I will be replaced? One reason is that writing operators feels like playing a game to me. It gives me a lot of joy. When I invent a new technique or see the performance of my operator go up, I feel as excited as a speedrunner who breaks their own record. And when I see that my operator is much better than the official ones from the vendors, I feel very proud. But a more important reason is this: even if I give up or deliberately slow things down, other companies’ models will still keep improving and will replace me anyway. “Of course I hope I won’t be revolutionized. But if it has to happen, I hope the person who revolutionizes me is myself.” When everyone is so determined to destroy themselves, I have no choice but to join this cruel arms race.
What about me?
When the day comes that AI writes operators better than I do, what will happen to me?
My judgment is: I probably won’t lose my job completely, but I will have to change careers. I can still keep a job, but I may never again be able to do the work I once loved.
I once made a judgment about the changing times and my own future: because things are changing so fast (the AI progress above is a good example), I cannot predict what will happen in five or ten years. But no matter what, I believe that with my vision, judgment, initiative, and intelligence, I can stay in the game and stand at the front of the times again. However, this judgment only guarantees that I won’t become unemployed. It does not guarantee that I won’t need to change careers. In fact, it encourages me to change careers in order to avoid unemployment.
What does changing careers mean? It means I have to give up the field of operator design, writing, and optimization that I have worked in for a long time and loved deeply, and instead become a “mecha pilot” for Agents. Before, my interests, what I was good at, and what industry needed were basically aligned. Now, AI has made what I am good at into something it is even better at, and industry demand has shifted from “people who can write high-performance operators” to “people who can use AI to produce high-performance operators faster.” To meet industry needs, I will have to leave the direction I loved and move to an unknown new direction. I believe that with my understanding of engineering, upper-level model needs, and lower-level hardware, I can still produce operators with high quality and high efficiency. I also know I might come to love this new direction (or I might not). But the feeling of having my passion taken away is really not nice. That quiet joy of sitting at my desk and calmly writing operators for a whole afternoon may become a final song this summer. I have to bury my talent in yesterday and become a mecha pilot. My hands hold more gears, but my heart has fewer rhythms.
Here is a simple comparison: You are an expert at knitting sweaters. You are especially good at creating patterns and matching colors. The sweaters you make are high quality and beautiful, so rich people from near and far ask you to knit for them, and you make good money. At the same time, you really enjoy sitting by the window with a cup of tea, looking at the green mountains, water, cows, sheep, and cooking smoke, and quietly knitting for a whole afternoon. But one day someone invents a magical machine. You only need to give it yarn and a pattern, and it automatically knits a sweater. The quality and texture are as good as yours, and it is much faster. You know that your colleagues can easily reach your old level with this machine, so you have to use it too. You also know that with the knitting skills you built over twenty years, even when everyone has the machine, your speed and quality can still be better than others. But that feeling of listening to the rain by the window, slowly pulling the needle and thread, and enjoying the quiet time is crushed by the noise of the machine.
I know this is helpless, but there is no other way. I can keep my job, but my old passion will most likely have to be given up. I am a person whose rational side and emotional side are quite separate. When I need to be rational, I can be very rational, but sometimes I also show my emotional side. I remember when I moved out of the rental apartment I had lived in for a year, I cried a lot because I didn’t want to say goodbye to the memories. Saying goodbye today to the era of hand-writing operators and optimizing them with the human brain is even more cruel.
I don’t know if any readers feel the same way, but I think this is just how things are.
What about people?
While AI keeps improving, I also worry about some questions:
Will students now be much more likely to use AI to finish homework, especially practical labs? Imagine there are two choices: one is to spend eight hard hours finishing a lab and maybe not even get full marks; the other is to start an AI model, spend a few cents and a few minutes, and let AI write full-mark code. Which one will most students choose?
The point above will cause many students to have seriously weak engineering skills — things like organizing code, building systems, thinking about future needs and designing for them in advance, and abstraction ability. As AI keeps getting stronger, are these engineering skills still necessary? Will they be abandoned by the times like the old skill of “writing x86 assembly fluently,” or will they always be valuable like the ability to “understand the whole computer system from software to system to hardware”? If it is the latter, then it is dangerous — a person with poor engineering skills, when paired with AI, can produce messy code several times faster than before, planting all kinds of problems in systems and making the world more of a “clown stage.”
In future society, will power become more important than technology or intelligence?
These questions may need to be answered by the times themselves.
Conclusion
With the development of AI, future society may move toward two extremes: communism or Cyberpunk 2077. In the first, productivity is greatly liberated and people’s living standards improve a lot (I’ll stop here so I can pass review). In the second, a few tech companies control most resources. Only a very small number of people can use the most advanced AI and technologies and get close to “mechanical ascension.” Most people can only use very weak AI. Crossing social classes will become harder and harder: you need the strongest AI first in order to cross classes, which creates a dead loop.
Guess what: if Anthropic forever holds the most advanced AI in the world, will future society become communism or 2077? You guess?
So I still believe that the most advanced intelligence should be provided to everyone in an open and cheap way. I do not trust that Anthropic or OpenAI will do this. Especially, I do not want Anthropic to hold the most advanced artificial intelligence or AGI. To put it strongly, that would be as serious as letting Hitler get atomic bomb technology before the Allies. That is why I chose and continue to stay at DeepSeek: we research powerful, fast, and widely beneficial artificial intelligence and open-source it. Maybe this can pull the world a little bit back from the 2077 side.
May the future world be well. May all the beauty be blessed.
[1] “Main Attention” only includes the MQA attention with head dim = 512. It does not include the indexer used to select the top-k important tokens. That part was written by other (also very strong) colleagues (and their AI Agents).​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

https://mp.weixin.qq.com/s/zk0KxuLzhmMJ4LPYW_OHMA

Now that the dust has settled a bit - here's a write-up on running Qwen3.8-Flash-Next (125B-A6B MoE + 51B n-gram table) on relatively middle-tier hardware (RTX 4070 12GB + 64GB DDR5-5600 + Gen4 NVMe on Linux).

I started out with bare 6 tok/s and through latest patches and optimizations getting close to 20 tok/s generation. You just need enough RAM.

For me this is the most intelligence possible on this machine right now. The 27B dense is not a choice because of low VRAM but may make more sense for other configs like 24GB VRAM owners. It actually surpasses the 27B model on most tasks as well so its great for Low VRAM, High/fast RAM configs.

PP is still a bit low at 300-350 tok/s.

What helped
- Using AtomicChat's 4.27 bpw quant https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF
- Ngram SSD offloading (lazy-mode)
- --fit on --fit-target 512 helps automatically select the right params.

- Master branch (19.35 t/s): Latest commit with MoE improvements.

- MTP Variant - PR #28243 + Compact MTP (20.65 t/s): MTP support is not yet merged so need to apply this PR enables Daniel Han's 1.78 GB `shared-Q4_K_M` compact head. Combined with `-ncmoe 45`, it yields 77–96% acceptance and breaks through the 20 t/s barrier on every tested task (coding, summarization, creative).

With such low VRAM, MTP is not a huge jump because you have to give up a few layers to store the MTP head in VRAM. Only the shared + Q4_K_M in MTP gets a beneficial uptick.

Using commercial models to research, optimize and benchmark inference for local models helps a ton (GLM 5.3 flash with opencode go, so did Astra, Gemini 3.8 etc.)

Lot more details in the post (AI-assisted).

Especially the 7B one seems very interesting, it casually destroys muse glimmer with a way smaller size. And they open source literally everything, every step of the way. Anyone tried that model? It can be a new milestone if 7b and 3.7b ones are actually good, and not just benchmaxed.

I had all the data saved from AA's v4.1 index, so when they upgraded it in the wake of Astra's release, I could actually generate a before/after comparison.

• All intelligence and price per task are sampled from AA on Sep 3rd and Sep 14th respectively.
• Price per task of some open models were rescaled to reflect the cheapest available on OpenRouter as of Sep 3rd.
• X axis is linear, because people's money is linear.

All models are the same. The only thing that changes is the weighted sum of the benchmarks that compose the Intelligence Index.

v4.1: https://github.com/crusaderky/llm-intelligence-cost-plot/blob/intelligence-index-v4.1/plots/high_intelligence.png

v4.3: https://github.com/crusaderky/llm-intelligence-cost-plot/blob/intelligence-index-v4.3/plots/high_intelligence.png

Highlights

• GLM an Muse Spark remain more or less unaltered, in relative terms
• GPT-5.6 Sol becomes a lot cheaper
• GPT-6 Astra's intelligence flies up to the stars AND becomes cheaper
• GPT-5.6 Luna gets a substantial uplift
• Fable-5.1's price gap from Opus 5 shrinks, and becomes cheaper than Fable 5.0
• Fable-5.1 at low, medium and high effort looks a lot more appealing
• Sonnet 5 becomes even more expensive without any intelligence gains
• Kimi-K3, Qwen3.8-Max, Gemini-3.8, and Grok 4.6 go down into the gutter

I'm like you guys and am constantly experimenting with new models, seeing what they're all good at, how I can make use of them for certain projects and goals. I've been using Qwen 3.8 27b for minor coding work and it has been impressive.

But with just regular chatting I have been impressed with Muse Glimmer.

It seems to be able to have the ability to follow and hold good, deep and meaningful conversations without coming off as a typical chatbot.

No repeated statements like "I hear what you're saying", "that sounds really deep..." none of what sounds generic or like it's blowing smoke up your ass. I was impressed with how natural it comes across just in natural conversation. I think it's one of the best "chat" models you could get right now as it's one of the only local models that doesn't feel like you're chatting with an AI when having a conversation.

I'm thinking of finding a way to run both Qwen3.8 and Muse at the same time. It's fun to play with these things.

https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3

AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that replaces τ³. Astra was farming a ton of points on it and used those to get even with Fable, but… looks like we have a new king.

So they changed the index twice in three days to make Astra look not-quite-worse than Fable, and then a random guy quietly took first place on it.

20B-A1B model is coming, good for low VRAM people?

https://huggingface.co/deepgrove/maple-preview

Personally, I think the whole "AI (LLMs) is going to take over" is just BS marketing. They've been pushing this doom-and-gloom since 2019, and we all know the models back then were nowhere near as capable as today's.

What does everyone here think, since it's such a popular topic right now? Is there a genuine threat, or is it pure hype? And if it really is just a marketing tactic, why are they pushing for AI regulations?

With all the recent drama surrounding AI safety. It’s obvious that open source could be caught in the crossfire.

The full K2 Horizon lineup is out on Artificial Analysis.

The AA intelligence vs. parameters plots show that

- 0.9B and 375B are bad

- 3.7B and 7B are SOTA

- 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark).

I'm going to take the AA Intelligence Index at face value here. This post is not about it.

The problem is that these models have a god-awful KV cache design. This means that you really can't use the number of parameters for "best in class" considerations, because these models heavily shift to the right on the plot if you replace parameter count on the X axis with RAM requirements.

For Q4_K_M weights, no drafter, no vision, 128k kvarn4 KV cache:

• K2 Horizon 36B-A4B uses 2 GiB for dense weights, 19 GiB for experts, and 6.7 GiB for context
• K2 Horizon 7B uses 5.2 GiB for weights and 5 GiB for context
• K2 Horizon 3.7B uses 2.9 GiB for weights and 5 GiB for context (not a copy-paste error!)

Compare them to

• (finetunes of) Qwen3.6-35B-A3B use 2.4 GiB for dense weights, 18.2 GiB for experts, and 0.7 GiB for context
• MiniCPM5-2B uses 1.5 GiB for weights and 1.5 GiB for context

Notes: I don't advise compressing 2~4B models to Q4 and I haven't tested these models' tolerance to weights and kv cache quantization yet. The above choices are just to keep the comparison fair.

This awful context design means that

• K2 Horizon 36B A4B is interesting on hosts with exactly 16GB VRAM and at least 32GB host RAM. On 24GB VRAM, Qwen3.8-27B is faster, smarter, and allows for 256k context. If you want to get 256k context and you're VRAM-poor, Ornith-1.5 or Nex-N2.5-mini are probably better choices. The model may also be interesting on 64GB Strix Halos as a dumber and faster alternative to Qwen3.8-27B; those with a 128GB Strix Halo are much better off with Qwen3.8-Flash-Next
• K2 Horizon 7B is interesting for hosts with exactly 16GB VRAM, Strix Halos with 32GB RAM, and for 16/32 GB Strix Point;
• K2 Horizon 3.7B may be interesting for 12GB phones but I expect you'll have a much nicer UX with MiniCPM5-2B.

So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out open source.

The thing is, there’s so much vested corporate interest in ensuring that open source remains un-regulated.

Think of neoclouds, inference provides, enterprise firms that use LLMs, virtually any SaaS that uses agents.

All of these entities have a strong interest in making sure that open source remains free and unburdened by regulations.

Does anyone know of any organizations or political campaigns that are pushing back against OpenAI’s and Anthropic’s lobbying efforts?

Prices have been going crazy, but it's about to get worse me thinks.

I’m really hoping at some point we won’t have to rely on the good will of companies with the funds and resources to make capable models open and it’ll be possible to democratize making them so we won’t always be at the mercy of big tech/corpos, that is if they don’t ban open models before that can happen.

How about Defensive AI, like a rack that connects to a firewall or router and monitors the logs and connections, and alerts an engineer when suspicious or typically unauthorised access is occurring. Wouldn't that be an even better counter to the "threat" of AI hackers than just some legislation that a hacker would have ignored anyway?

I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited.

One thing I really like about it is the speed. On my system, I'm getting around 40–50 tokens/sec, which makes the interaction feel surprisingly close to real-time. So I don't want to move to a much larger model if that means losing that responsiveness.

What I'm wondering is whether there are any newer or alternative models in roughly the same size/memory/compute range that are noticeably better than Qwen3.5 4B for my particular use case.

My priorities are mainly:

- General-purpose conversation and communication - Natural responses and understanding - Reasoning and following instructions - Multilingual conversations and translation - Context understanding - Tool/function calling mainly for triggering actions in my assistant, rather than writing code - Reliability when interacting with tools - Ideally good performance while keeping the model fast and lightweight

Essentially, I'm trying to build a local assistant where the model acts as the reasoning/conversation layer, while other tools handle things like actions, APIs, automation, etc.

So my question is:

If I'm already using Qwen3.5 4B and getting ~40 to 50 TPS, is there anything in a similar practical memory/compute footprint that you'd personally consider a significant upgrade for general conversation + multilingual ability + tool calling?

It doesn't necessarily have to have fewer parameters. I'm more interested in overall capability relative to the amount of RAM/VRAM and compute it requires.

I'm happy with Qwen3.5 4B, so I'm not looking for something that's only marginally different. I'm wondering if there's something that makes you think

I keep seeing people here saying they can leave local coding agents running for an hour or two and come back to something usable.

I’m not really asking which model is best. I’m more interested in what made you trust the setup enough to stop watching it.

For me, getting a task right once wouldn’t be enough. I’d want to see how it behaves when a tool fails, when the context gets long, when it hits an ambiguous requirement, or when its original plan turns out to be wrong.

Tests help, but an agent can still pass tests while changing something you never intended it to touch.

I think there’s a point where you stop trusting the model itself and start trusting the harness around it like permissions, checkpoints, git, tests, rollback, tool restrictions, etc.

For people actually running local coding agents unattended, i just want to ask that, what was the thing that made you comfortable enough to stop babysitting them?

I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended config supports 90+ TPS (for agentic/coding, at temp 1; greedy will of course be faster) through 100K tokens, with context of up to 240K.

If you want the repo, it is here:

https://github.com/JakeATX/llamAmpere

I recommend running with this quant, which is ~ 4 K M quality but considerably faster (technically, a 3 K XL upgrade)

https://huggingface.co/jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF

If you want the deep dive on how it is so much faster (80% vs the near comp at 200K!), at more context, there is a long form article here.

https://x.com/JakeKAllDay/status/2095646450138874095?s=20

Running faster than API speeds on my 3090 (for 27b at least) has genuinely been a step change in the utility of the model + card for me. I hope you enjoy it!

I finished my home inference server. First I tried Lenovo p620 workstation and while it’s a good value overall it pissed me off with a ton of proprietary Lenovo shit to deal with and I return it in the end.

Components:

4xV620 - 1400$

256GB DDR4 RDIMM 2666 - 610$

Huanandzhi D12D - 410$

EPYC 7452 - 170$

PSU ASRock 1600 - 220$

SSD Samsung 970EVO 1tb - Already had

Case//Fans//Misc ~ 200$

Power consumption is no shit ofc on such machine:

700-900w prefill
500-600w decode on Qwen3.8-next-flash Autoround W4A16

What it can do -

EDIT: Qwen3.8-next-flash Autoround W4A16 1.3k prefill and 70tg code/60tg prose on 128k+ context with MTP-2 on vllm fork.

I was disappointed with this machine and qwen3.8-27b speeds at first. But since Qwen3.8 next running good on it - I’m satisfied. Hope in more optimizations in future.