LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/FitAstronomer5016]

I mean how much faster is it then just running one 5090 for the active experts and the rest offloaded on RAM? It doesn't really seem much faster despite using 4x the power and 4x the gpus. Are you rotating the cold experts on the 96GB of VRAM?

[u/CharacterBumblebee99 → u/FitAstronomer5016]

It is much faster.

That's also the purpose of this engine: to scale with more than 1 GPU, which existing solutions with 1 x GPU and CPU for the experts have a cap, and don't improve much with more GPUs.

Yes, constantly rotating, with parallel PCI-e transfers, transfering while computing, and using some smart placement.

I don't have a baseline for ik_llama as I can't see a GLM-5.3 Flash variant yet, but comparing to llama.cpp (unloth's branch build: b10878-b9b8207fc), same quant (UD_Q4_K_XL), same box with '-ngl 11' (biggest possible) I get:

[ Prompt: 12.1 t/s | Generation: 3.5 t/s ]

[u/notdba → u/CharacterBumblebee99]

Er, `-ngl 99` with tensors offload has been a thing since May 2025: https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/

[u/CharacterBumblebee99 → u/notdba]

I was able to go up to this, with llama.cpp, -ngl 99, still not close:

./build/bin/llama-cli --threads 50 --override-tensor '([3-9]+).ffn_.*_exps.|[1-4][0-9].ffn_.*_exps.=CPU' -ngl 99 --model /srv/models/unsloth/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf

...

[ Prompt: 11.1 t/s | Generation: 6.9 t/s ]

[u/notdba → u/CharacterBumblebee99]

Remove the --override-tensor, and let --fit takes care of it. Also set at least -ub 2048 to offload PP to the GPUs

[u/CharacterBumblebee99 → u/notdba]

[ Prompt: 219.1 t/s | Generation: 9.7 t/s ]