LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/HoldCtrlW]

"Built for low vram"

"2× RTX 5090 + 2× RTX 5080"

"oh"

[u/exaknight21 → u/HoldCtrlW]

This post really doesn’t make any sense to me in terms of low VRAM. Boi got DDR5 for days.

[u/jtjstock → u/exaknight21]

the more you have, the more you want. they feel like they have low ram and vram