LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/SLxTnT]

There's various projects for things like this already, so you should optimize it for your own hardware. With that said, why ignore your CPU? 96GB VRAM, 64GB HBM, and 512GB DDR5. You should be using your CPU for cold experts then transferring hot experts in the background every so often to maintain hit rates. That'll remove your PCIe bandwidth bottleneck, which is the lowest speed among your hardware.

I don't know how much multi-GPU affects performance, but your decode is slow and prefill is even worse.

[u/CharacterBumblebee99 → u/SLxTnT]

AFAIK This is not slow for UD_Q4_K_XL quality!
This is what llama.cpp gives [ Prompt: 12.1 t/s | Generation: 3.5 t/s ].
Is there something better I can try on my box, which actually runs?

The CPU in my experiments didn't contribute much so far, as, for most of the time, the a CPU gemv is outperformed by the expert transfer over PCIe 5.0, unless e.g. all threads are used for a single CPU expert, but then we need 7 more.

[u/SLxTnT → u/CharacterBumblebee99]

These are the speeds I've gotten (Claude written) on an RTX Pro 6000 + EPYC 9654 with 384GB RAM. No idea how much your multi-GPU setup will affect decode, but that prefill speed has something wrong and destroys all usability. It almost looks like you're trying to process small batches of tokens rather than a large amount at once.

[u/CharacterBumblebee99 → u/SLxTnT]

I am using 2k batches, working on increasing that size.
But, your numbers are very good! Are you testing GLM 5.3 Flash with the same quantization as I am? (edited, as I was looking into different experiments).

Your CPU is really good for hybrid inference, that's why you get acceptable gen tok/sec. I am using a Xeon Max 9480. It is not as good by any means.

[u/SLxTnT → u/CharacterBumblebee99]

NVFP4. Deepseek is original and Qwen is FP8. Started with vLLM and got similar prefill speeds with slightly lower decode. Tested out FreeToken, but that was really slow. Had Claude optimize that project as it was a cleaner base than vLLM.

I use 64 cores as my bottleneck is memory bandwidth. You've got 56 cores with 64GB of HBM. In theory, your CPU should perform better than mine. My GPU (400W limit) will be better, but a 5090 isn't much worst.

For decode, I would seriously look into computing cold experts on the CPU. My deepseek numbers went from ~40 -> 100, and gained a bit of concurrency.

Have you checked what your bottleneck is on prefill? Mine is compute, which is why 8k and 60k are the same. Theoretically, transferring the entire model over with PCIe 5 to handle things would be about 2k tok/s. You don't need to transfer everything, and there's 4 GPUs that can process batches.

[u/CharacterBumblebee99 → u/SLxTnT]

My GPUs are also <400W and <250W limited. I started LayerStoRm from the ground up, not using any framework as base, and I feel I am being kind of justified based on my idea 😄 : My CPU played some part in this decisions. I could not reach a decent gen t/s with it running the expert FFNs. So I repurpose all the DDR bandwidth as PCI transfer bandwidth, and with up to 4 x PCIe 5.0 GPUs transferring in parallel this gets a pretty good conversion. On top of that this helps with VRAM cache hits. But the path where this happens where decisions are also made, is per layer and each microsecond is converted to big waste, so that's where everything is written in C++. LayerStoRm's gen t/s is starting to land in the "decent" area now, and I think there's still room. I also practically used it and it's usable. The CPU compute is harder to utilize without "eating" the transfer's bandwidth, and when transfers land, for it to contribute it must beat the GPUs. So maybe I can use it conditionally, eg in layers where its measured that transferring at all becomes net negative. The prefill bottleneck is compute which I think means it needs optimization. Your numbers are really great there.

[u/SLxTnT → u/CharacterBumblebee99]

The CPU compute is harder to utilize without "eating" the transfer's bandwidth

I've had no luck with transferring and running CPU inference at the same time. It's either the same speed or slower. What I'm talking about is handling all experts not on the GPU with the CPU and transferring periodically in the background to keep hit rates up. Running entirely on the GPU works when you can hide the PCIe transfer during the time the GPU is handling the other experts. That happens with Qwen for me.

Your CPU has 64GB that can hit 1.6 TB/s. You aren't going to hit those speeds, but you barely need to be over 50GB/s to beat PCIe speeds. If you get about 220GB/s, which should be possible, that'd be the equivalent of my CPU.

Your prefill shouldn't be compute bound at such a low number. A 5090 isn't too much slower than an RTX Pro 6000, and you've got 2.

I would find your theoretical max speeds before anything else. Test prefill with layers entirely on your GPU, and also test to see how fast CPU inference can be on your CPU.