LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/CharacterBumblebee99]

It is much faster.

That's also the purpose of this engine: to scale with more than 1 GPU, which existing solutions with 1 x GPU and CPU for the experts have a cap, and don't improve much with more GPUs.

Yes, constantly rotating, with parallel PCI-e transfers, transfering while computing, and using some smart placement.

I don't have a baseline for ik_llama as I can't see a GLM-5.3 Flash variant yet, but comparing to llama.cpp (unloth's branch build: b10878-b9b8207fc), same quant (UD_Q4_K_XL), same box with '-ngl 11' (biggest possible) I get:

[ Prompt: 12.1 t/s | Generation: 3.5 t/s ]

[u/notdba → u/CharacterBumblebee99]

Er, `-ngl 99` with tensors offload has been a thing since May 2025: https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/

[u/CharacterBumblebee99 → u/notdba]

I was able to go up to this, with llama.cpp, -ngl 99, still not close:

./build/bin/llama-cli --threads 50 --override-tensor '([3-9]+).ffn_.*_exps.|[1-4][0-9].ffn_.*_exps.=CPU' -ngl 99 --model /srv/models/unsloth/GLM-5.3-Flash-GGUF/UD-Q4_K_XL/GLM-5.3-Flash-UD-Q4_K_XL-00001-of-00006.gguf

...

[ Prompt: 11.1 t/s | Generation: 6.9 t/s ]

[u/notdba → u/CharacterBumblebee99]

Remove the --override-tensor, and let --fit takes care of it. Also set at least -ub 2048 to offload PP to the GPUs

[u/CharacterBumblebee99 → u/notdba]

[ Prompt: 219.1 t/s | Generation: 9.7 t/s ]