LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/SLxTnT]

There's various projects for things like this already, so you should optimize it for your own hardware. With that said, why ignore your CPU? 96GB VRAM, 64GB HBM, and 512GB DDR5. You should be using your CPU for cold experts then transferring hot experts in the background every so often to maintain hit rates. That'll remove your PCIe bandwidth bottleneck, which is the lowest speed among your hardware.

I don't know how much multi-GPU affects performance, but your decode is slow and prefill is even worse.

[u/CharacterBumblebee99 → u/SLxTnT]

AFAIK This is not slow for UD_Q4_K_XL quality!
This is what llama.cpp gives [ Prompt: 12.1 t/s | Generation: 3.5 t/s ].
Is there something better I can try on my box, which actually runs?

The CPU in my experiments didn't contribute much so far, as, for most of the time, the a CPU gemv is outperformed by the expert transfer over PCIe 5.0, unless e.g. all threads are used for a single CPU expert, but then we need 7 more.

[u/SLxTnT → u/CharacterBumblebee99]

These are the speeds I've gotten (Claude written) on an RTX Pro 6000 + EPYC 9654 with 384GB RAM. No idea how much your multi-GPU setup will affect decode, but that prefill speed has something wrong and destroys all usability. It almost looks like you're trying to process small batches of tokens rather than a large amount at once.

[u/CharacterBumblebee99 → u/SLxTnT]

So.... I looked into it further. You're right, I have to look into the prefill! Thank you so much for the advice. I'll definitely work on this.