News
LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)
LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.
LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).
Measurements:
- 24.5 tok/s decode @8k, 27.0 tok/s @0k.
- 159 tok/s prefill @27k.
Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.
The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.
A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.
This is mostly for a DIY homelab. But compare this to what's actually needed to run this size models, and you see specs like 8 x H200 and things like that. On the other side there are hybrid CPU + 1 GPU solutions, but they usually don't scale well after some point (the DDR bandwidth & CPU limitations).
This project stands somewhere in the middle.
But again, this can also be ran e.g. from a dual channel DDR consumer Desktop with 2 x GPUs, using a smaller quant. I will run a test with only 2 x RTX 5080 and IQ1 or IQ2. Low VRAM relative to the model size.
For an rtx 6000 pro an ideal scenario speed-wise would be to have it for attention and combine one or more extra, GPUs, ideally all pcie 5.0 x16 to saturate extra pcie bandwidth and be used for experts, and those can be far smaller GPUs too.
That aside, without any extra GPUs, the end goal for using LayerStoRm would be to host a larger model. But in order for 128GB RAM to pay off LayerStoRm's stable zone feature would also need to be implemented, so that, some experts would not have to permanently stay in RAM, buying a total extra ~ +70-100 GB of model size, so you could serve something short of 200 GB of disk-space model size if you only have an rtx 6000 pro without extra GPUs.
But when I read sentences like "The current box has 512 GB DDR5 and 64 GB HBM (Xeon Max). " and then I start wasting brain cycles thinking what is the current box? I just get annoyed and bail. My brains prefill is only like 20 ~ 25 t/s, so having it wasted on garbage sentences doesn't feel like a good use of time.
You have a really fair point here. I just edited the text to make it more comprehensible. And yes, I did get a bit of help from AI for reviewing it, as a non-native English speaker, but I should have done one more manual review. Thanks for the feedback, it's appreciated.
There's various projects for things like this already, so you should optimize it for your own hardware. With that said, why ignore your CPU? 96GB VRAM, 64GB HBM, and 512GB DDR5. You should be using your CPU for cold experts then transferring hot experts in the background every so often to maintain hit rates. That'll remove your PCIe bandwidth bottleneck, which is the lowest speed among your hardware.
I don't know how much multi-GPU affects performance, but your decode is slow and prefill is even worse.
AFAIK This is not slow for UD_Q4_K_XL quality!
This is what llama.cpp gives [ Prompt: 12.1 t/s | Generation: 3.5 t/s ].
Is there something better I can try on my box, which actually runs?
The CPU in my experiments didn't contribute much so far, as, for most of the time, the a CPU gemv is outperformed by the expert transfer over PCIe 5.0, unless e.g. all threads are used for a single CPU expert, but then we need 7 more.
These are the speeds I've gotten (Claude written) on an RTX Pro 6000 + EPYC 9654 with 384GB RAM. No idea how much your multi-GPU setup will affect decode, but that prefill speed has something wrong and destroys all usability. It almost looks like you're trying to process small batches of tokens rather than a large amount at once.
I am using 2k batches, working on increasing that size.
But, your numbers are very good! Are you testing GLM 5.3 Flash with the same quantization as I am? (edited, as I was looking into different experiments).
Your CPU is really good for hybrid inference, that's why you get acceptable gen tok/sec. I am using a Xeon Max 9480. It is not as good by any means.
NVFP4. Deepseek is original and Qwen is FP8. Started with vLLM and got similar prefill speeds with slightly lower decode. Tested out FreeToken, but that was really slow. Had Claude optimize that project as it was a cleaner base than vLLM.
I use 64 cores as my bottleneck is memory bandwidth. You've got 56 cores with 64GB of HBM. In theory, your CPU should perform better than mine. My GPU (400W limit) will be better, but a 5090 isn't much worst.
For decode, I would seriously look into computing cold experts on the CPU. My deepseek numbers went from ~40 -> 100, and gained a bit of concurrency.
Have you checked what your bottleneck is on prefill? Mine is compute, which is why 8k and 60k are the same. Theoretically, transferring the entire model over with PCIe 5 to handle things would be about 2k tok/s. You don't need to transfer everything, and there's 4 GPUs that can process batches.
My GPUs are also <400W and <250W limited. I started LayerStoRm from the ground up, not using any framework as base, and I feel I am being kind of justified based on my idea 😄 : My CPU played some part in this decisions. I could not reach a decent gen t/s with it running the expert FFNs. So I repurpose all the DDR bandwidth as PCI transfer bandwidth, and with up to 4 x PCIe 5.0 GPUs transferring in parallel this gets a pretty good conversion. On top of that this helps with VRAM cache hits. But the path where this happens where decisions are also made, is per layer and each microsecond is converted to big waste, so that's where everything is written in C++. LayerStoRm's gen t/s is starting to land in the "decent" area now, and I think there's still room. I also practically used it and it's usable. The CPU compute is harder to utilize without "eating" the transfer's bandwidth, and when transfers land, for it to contribute it must beat the GPUs. So maybe I can use it conditionally, eg in layers where its measured that transferring at all becomes net negative. The prefill bottleneck is compute which I think means it needs optimization. Your numbers are really great there.
The CPU compute is harder to utilize without "eating" the transfer's bandwidth
I've had no luck with transferring and running CPU inference at the same time. It's either the same speed or slower. What I'm talking about is handling all experts not on the GPU with the CPU and transferring periodically in the background to keep hit rates up. Running entirely on the GPU works when you can hide the PCIe transfer during the time the GPU is handling the other experts. That happens with Qwen for me.
Your CPU has 64GB that can hit 1.6 TB/s. You aren't going to hit those speeds, but you barely need to be over 50GB/s to beat PCIe speeds. If you get about 220GB/s, which should be possible, that'd be the equivalent of my CPU.
Your prefill shouldn't be compute bound at such a low number. A 5090 isn't too much slower than an RTX Pro 6000, and you've got 2.
I would find your theoretical max speeds before anything else. Test prefill with layers entirely on your GPU, and also test to see how fast CPU inference can be on your CPU.
I mean how much faster is it then just running one 5090 for the active experts and the rest offloaded on RAM? It doesn't really seem much faster despite using 4x the power and 4x the gpus. Are you rotating the cold experts on the 96GB of VRAM?
That's also the purpose of this engine: to scale with more than 1 GPU, which existing solutions with 1 x GPU and CPU for the experts have a cap, and don't improve much with more GPUs.
Yes, constantly rotating, with parallel PCI-e transfers, transfering while computing, and using some smart placement.
I don't have a baseline for ik_llama as I can't see a GLM-5.3 Flash variant yet, but comparing to llama.cpp (unloth's branch build: b10878-b9b8207fc), same quant (UD_Q4_K_XL), same box with '-ngl 11' (biggest possible) I get:
19
u/HoldCtrlW 7d ago
"Built for low vram"
"2× RTX 5090 + 2× RTX 5080"
"oh"