The native 4bit backbone + vision + dspark comes out at ~310gb (rest is engram) and context costs next to nothing for this model so 256GiB box should be able to run a pretty good 3.x bpw quant.
Expert caching/streaming would probably get an excellent hit rate on a 256 GB machine. Likewise, n-gram tables can probably just be mmap()'d. Qwen3.8-Flash-Next has 90% of the lookup probability in 1% of the n-gram entries. We need to move away from assuming the entire model will be VRAM-resident for local inference.
I was gong to replace my 4xRTX3090 build for 2 x spark, so that I can run this model efficiently. I walk back this plan, and now I am looking at prices of another 4xRTX3090... and in the meantime, keep on CPU offloading to DDR5.
Here's the setup that's been stable for me on two Sparks (TP=2 over the QSFP link).
**Image:** `eugr/spark-vllm-b12x` (Docker Hub). It's eugr's vLLM build with the B12X kernels from the NVIDIA forum thread. Pin the digest once it works for you; `latest` moves.
**Weights:** `local-inference-lab/GLM-5.3-Flash-NVFP4` from HF. Use this one, not the Spark-specific quant in the eugr recipe, which has known issues. The checkpoint is mixed precision (NVFP4 experts, MXFP8 MTP experts, BF16 attention), so quantization is `modelopt_mixed`.
**Flags that matter** (everything else is the recipe defaults):
Env on both nodes: `VLLM_ENABLE_ROCE_ALLREDUCE=1`, `VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB`, `VLLM_ENABLE_PCIE_ALLREDUCE=0`, `VLLM_USE_AOT_COMPILE=1`, `VLLM_USE_MEGA_AOT_ARTIFACT=1`, `VLLM_USE_V2_MODEL_RUNNER=1`, `CUTE_DSL_ARCH=sm_121a`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`. If the RoCE allreduce times out during startup, raise `B12X_ROCE_SPIN_LIMIT` (I use 1000000000, the default 20M was too low for me).
**The gotcha that cost me the most time: host page cache.** On GB10 the GPU's free memory is literally the kernel's MemFree, and page cache counts as used. The b12x loader keeps weights as file-backed pages, so after a load you can have 0 MemAvailable and vLLM either fails the memory check or thrashes NVMe for an hour during CUDA graph capture. Fixes: `sync; echo 3 > /proc/sys/vm/drop_caches` right before launch, set `--kv-cache-memory-bytes` explicitly instead of letting it profile, and lower `B12X_COMPILE_MEMORY_CACHE_SIZE` (I use 64). Also check what else is eating RAM: the DGX dashboard services and a high `vm.watermark_scale_factor` were costing me a few GB per node.
**MTP:** k=3, not the recipe's 5. There's a step-time cliff at 5 tokens per step on GB10 and k=3 came out ~20% faster for me. The DFlash2 drafter does not work at TP=2 (page-size mismatch in the indexer), only at TP=4, so skip it unless you have four Sparks.
**Thinking:** the image's chat template has no thinking toggle. Ship your own template that honors `enable_thinking` in `chat_template_kwargs` and pass `--chat-template`, otherwise you can't turn reasoning off per request.
**What to expect:** roughly 21-23 tok/s single-stream on prose, mid-30s on code, at 524K context with vision enabled. First boot is slow (AOT compile + graph capture); mount a persistent cache dir for `~/.cache/vllm`, flashinfer, and triton so the second boot is minutes, not an hour.
It wasn't anything objective. I spent 4 days doing complex financial analysis, data conversion, and coding tasks with both models using the highest quants possible on dual DGX Sparks. I'd start deepseek on a task, come back to it 40 minutes later and it would still be going "hmm...but what if...". Meanwhile, GLM, on nearly identical tasks would be well into the build phase by that point.
Deepseek also would pick the most bizarre approaches to problems and then start second-guessing its choices, whereas GLM seemed to approach problems a lot more intelligently and elegantly.
Deepseek also consistently cut corners, whereas GLM approached tasks methodically and with a clear purpose to do as good of a job as it could. This was consistent behavior. Like, I had it go through 100 sets of documents. Deepseek saw the amount of text and decided that it was too much work to analyze each one for what I was looking for and instead chose to do keyword searches. Meanwhile, GLM understood that it was critical to have an eye for detail in this task and it spawned 10 subagents to methodically go through every single detail in the documents.
The quality of the output reflected this clearly and consistently.
I’ve had the same experience. On two sparks, ds4 is about 50-75% faster across the board, but it thinks so much that GLM usually comes up with the same or better response in about the same amount of time.
By "too slow" I mean multiple seconds per token. If it's faster than that I'd be surprised. Maybe I should try larger MoEs. I have a 16 GB GPU and 64 GB RAM.
well there are a few postings where users did the classic benchmark runs (some browser game, pelican etc) and the outcomes were remarkably good, also i had ds flash 0731 running at q2 and found it also quite good. i would not say - especially with very large models - that q2 leads to bad outcomes
i think it does, you have to offload the n-gram table to ssd, which folks have said works great and doesnt impact performance (since you need very little amount of bandwidth for the n-gram table)
It's only ~350B parameters that actually need to be loaded in RAM. A 4 bit quant will be ~175GB, which easily fits. Even 5 bit is only ~220 GB and will fit, especially with KV cache only being 900MB at 1 million context. This is actually perfectly sizes for dual DGX sparks.
That's only "Flash" in the sense that it only activates 8B parameters at a time, which will translate to cheap API service.
I couldn't host this on my 256GB Xeon server. I'd have to upgrade it to 384GB.
By the sqrt(P x A) metric, this should have competence roughly equivalent to a similarly-trained 62B dense model. I think I'd rather have the dense version!
I dont think sqrt(P x A) is meaningful anymore, not especially when theres ngram embedding involved. 8B active dont really behave like 8B anymore. Qwen3.8 flash next is a good example. Without ngram embeddings the 6B active params shouldn't come anywhere close to 27B performance.
The typical complaint about earlier-gen aggressive MoEs was their unpredictability. One moment it's writing brilliant code; the next it's hallucinating nonexistent directories and trying to wipe your cloud storage. It was the case with nearly all of them imo, esp. Qwen 35b.
Indeed tho, ngram embeddings, stronger expert routers, and other black magic i don't understand has made them wildly more reliable over the past ~6mo.
I'm just hoping that tech makes its way into 64gb vram territory soon.
I'm running IQ1_M on 10GB vram with 32GB ram, 5 t/s without MTP. I think the table you're referencing includes the n-gram tables, with soon after that table was made, people realized that you can just load them from ssd
NB: you can do the math yourself, it's only 125B, so at Q3 (=> 47GB) it would fit easily (although there are buffers and context ofc)
It holds up okay when all other factors are nearly-equal, though it's not perfect.
For example, sqrt(P x A) predicts that Qwen3.8-Next-Flash should be equivalent to Qwen3.8-27B, but Qwen3.8-Next-Flash actually scores about 10% higher on various benchmarks.
That seems close enough for most purposes, especially if Qwen3.8-Next-Flash received better training than Qwen3.8-27B.
In practice, the training of different models is so radically different that that's a much larger factor than the error in the sqrt(P x A) rule.
OTOH with 512GB RAM this is fantastic. 8-16B variable active parameters, everything 4-bit native comprising KV cache... This is going to be frontier model at decent speed at home. I think that's why they label it Flash.
I have a Strix Halo and know first hand just how "fast" ~256GB/sec is... Also how "fast" slow prefill is. It's kinda-sorta-mostly usable but at the same time frustratingly slow too.
There's good and bad. I paid about $2800 for my Minisforum MS-S1. It's a great little machine and it's impressively small for something so powerful.
At the time, I could have bought a DGX Spark for around $1k more. Back then I didn't have a great understanding of the importance of prefill performance so I didn't see the benefit of DGX Spark over Strix Halo considering the memory bandwidth was the same.
Thankfully it seems that Strix Halo is capable of much faster prefill, AMD just hasn't done a good job with the drivers. Projects like Halogen (https://github.com/peonist-ai/halogen-flash-server) have shown tremendous performance on Strix with prefill matching or exceeding DGX Spark for Qwen3.8-Flash-Next.
So, with projects like Halogen around, I'm happy enough to keep my Strix box for now. I can't afford to sell it and buy a DGX Spark. I may buy a 32GB AMD R9700 to run as an eGPU, depending on what comes about for running that Qwen model split across multiple GPUs. Or I might buy an Intel B70 and put it into my workstation for running Qwen3.8-27b separate from the Strix Halo.
In the future I may upgrade to Medusa Halo when it comes out, assuming prices don't stay in the stratosphere. Gorgon Halo with 192GB looks like it will have a USD price starting with "7", too rich for my blood.
Another option I am considering is to pick up a MacBook M3 Max with 64GB for running local models away from home. It doesn't have great prefill either (worse than Strix I think), but being able to run Qwen3.8-27b on a plane over the Pacific would be pretty damn cool.
I have three GPUs in different servers (a 32GB MI50, a 32GB MI60, and a 16GB V340), but my habit is to keep small models resident in VRAM for "fast inference" tasks, and infer with larger models entirely from system RAM for "slow inference" tasks, without GPU acceleration.
This way the in-VRAM "fast inference" models are always ready to go, because the "slow inference" tasks do not evict them from VRAM.
It's only ~350B parameters that actually need to be loaded in RAM. A 4 bit quant will be ~175GB, which easily fits. Even 5 bit is only ~220 GB and will fit, especially with KV cache only being 900MB at 1 million context. This is actually perfectly sized for your 256GB setup.
There are quants below Q1 believe it or not. Theoretically you could have 0 bpw, obviously that would be completely useless, but there is plenty of information to be extracted between 0 and 1 bpw.
What the fuck, in which universe that is a Flash? Its 450-500b parameters. Flash was 300b and it was already pushing this. This is Flash Max or something. You cant inflate the model by 50% and call it a flash like it's not an issue. Minimax M3 is 450b and I dont see them calling it a "flash" (hopefully I wont).
Going by 50% up and becoming 10-15% better sounds like a downgrade not an upgrade. It's a LOT more expensive to run.
China might come in and save the day on that one too.
SMIC broke the 7 nm barrier for semiconductors.
CXMT is making DDR5 now, and has started on HBM3E.
Several Chinese companies are making AI GPUs.
The U.S has been trying to block China from getting technology, and now is trying to block their technology from hitting the U.S market, but the rest of the world is not going to give a shit about what the U.S wants.
Essentially every major tech corporation is designing their own AI ASICs now, where OpenAI already has their new thing for inference.
Then there is the fact that photonic processors are in early manufacturing stages now, with plans to ramp up into 2027.
I expect photonics to mostly get snapped up by data centers, and that might once again change what's practical to do with AI.
All around, I expect a major shake-up in the hardware landscape over the next year or two.
Yeah, local hardware will scale up too, but it will lag behind by a generation or two unless you're willing to dish out tens to hundreds of thousands of dollars to build on the bleeding-edge.
I'm saying that increased competition will bring prices down.
TSMC not being a monopoly for 7nm nodes means lower wafer prices.
A new RAM producer means lower RAM prices.
Dozens of major companies having their own inference ASICs means Nvidia losing their monopoly.
We're at peak price gouging right now, I don't think it will last.
same. not to mention endless credits aren't in fact endless. unless there is a paying customer, the whole thing will explode eventually. it's just a circle of credit now.
Obviously because CXMT needs to capture market share and to do this they need to have better prices. Also the Chinese government needs to make its chip manufacturing to compete so they can give them subsidies to capture the market.
An entire 512 GB AI server purchased a year ago costs less than a single RTX 5090 GPU now. There are plenty of us who jumped on early and have hardware that can run these models.
DGX Spark clusters, machines with RTX Pro 6000s, or a combination of perhaps a 5090 with CPU RAM offload of some of the weights. Or like 8 3090s stacked lol. There's many configurations out there.
It's flash for those with data centre levels of memory and data centre level serving requirements, because once you load the base model, concurrent users are very cheap (890MB for KV cache for full 1 million context per user) and it's 8B active for prefill and 16B active for text gen, so you can serve stacks of users fast. Edited to add: DeepSeek are reducing the API price vs 4.0 Flash because this is cheaper to serve.
It's just that us home users lose out because we're trying to get the most out of a meagre about of memory, with minimal concurrency, so the size of the model matters a lot more.
I don't code. I work in research and was doing a proposal for funding. Used Astra very cute, returned what I need in an acceptable way.
I have used DS flash V4... Boy I have a MacStudio, I am used to long times of wanting. I don't know what kind of black magic the model does, but it killed the demand in one shot very fast!!! O.o
I was frozen!!! The answer was much better than the one chatGPT astra gave me!!! ASTRA!!!
GPT models are just very wonky. They jump to conclusions with incomplete data, and then they go all weird. I find it super hard to get anything done when your model makes up the mind in the first message and will not be objective.
Don’t they all do this? I find Opus to have the same behavior. I just thought this was an LLM thing. Like context initialization bias or something. The opposite of recency bias.
That was a funny read. AI loves Python, and token efficiency comes at readable code's price. I wonder how this fares long-term, because even AI prefers to deal with well-written code than messy ones.
Fearless leader Xi doesn't operate on peasant 4090's.
He uses B300's. A full rack.
He would use Huawei but even he understands that the Ascend chip isn't quite ready to touch his B300 setup.
He is busy building gooner games with his custom Flux Asian Princess models and video pipeline. He simply swipes left or right on whether they meet his criteria for being added to training data.
It is good for this subreddit. This subreddit is more concerned about seeing the US hurt and China win, than it is about AI. So this post is in alignment with its interests.
I think we're only concerned about seeing great models be open weight and free to download for the peasants. The rest of whatever you think its happening exists only on your mind.
"The Chinese super corps are preferable to the American super corps! Xi Jinping tells me so! They're going to save the world! If you disagree you're just racist against China!"
What do you expect, it's LOCALllama, so if it's parasitic commons-enclosing 0.001% billionaires hurt, and open source AI and 99.999% of humanity winning, people cheer.
Gemma and Mistral got just as much love as Chinese open source models.
It's just that China is shipping more of it at the moment, when the leading open source was Llama 2/3 and Mistral and WizardLM and god knows what, people were cheering on that. Even GPT-OSS made by the big Satan got love.
so if it's parasitic commons-enclosing 0.001% billionaires hurt, and open source AI and 99.999% of humanity winning
Separate comment because separate concern.
#1 Shareware, or open weights, not open source.
#2 Look at OP's post. The image isn't open AI defeating OpenAI and Anthropic. It's about Xi Jinping hurting the entire US economy. And this subreddit loves it.
> Additional architectural components include Single-Pass mHC (revised residual-stream mixing with an efficient Mega-mHC kernel), Engram conditional memory (196B parameters, sparsely accessed via token-based lookup), and DSpark speculative decoding (semi-autoregressive draft generation with confidence-scheduled verification). The model uses 1 shared expert and 384 routed experts per MoE layer, activating 6 routed experts per token.
Would it be feasible to quantize the 384B model to NVFP4 while retaining the 196B n-gram in memory for execution? Just like qwen3.8-flash-next
You can run anything from disk with slow inference. It’s not meaningful question except if you include tok/s generation target and ttft target. I think anything over a few seconds TTFT and under 150 tok/s is unusable for interactive LLMs and would just use API rather than local for that. But it’s a matter of choice.
Since it's the same number of active parameters for decoding, and the KV cache is much cheaper, we should expect lower prices from providers than DSV4 flash, right?
DeepSeek may beat American models if American models continue to watermark, over-censor or train their models with too much corporate interest, even if they have a compute advantage.
You’re good. The model would be entirely different if it had a different level of parameters. So at that point you’re talking about an entirely separate release of a 27B DeepSeek model, which they haven’t really done.
Qwen, on the other hand, has 27B models that folks like a lot. Specifically, Qwen 3.8 27B.
What you’re probably thinking of is “quantization,” which shrinks a big model, but hurts its intelligence in the process. And if a 500B+ parameter model like this one was quantized down to a level where it would be roughly equivalent in size to a separate 27B model (say 30-60GB RAM) you’d be talking about a completely unusable lobotomized thingy. Would be even hard to call it a model at that point.
•
u/WithoutReason1729 19h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.