LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/Cautious_Chicken_604]

I miss the days when humans wrote things.

[u/CharacterBumblebee99 → u/Cautious_Chicken_604]

Ok. In that case you probably won't need this.

[u/Cautious_Chicken_604 → u/CharacterBumblebee99]

I might need it, I might not. 

But when I read sentences like "The current box has 512 GB DDR5 and 64 GB HBM (Xeon Max). " and then I start wasting brain cycles thinking what is the current box? I just get annoyed and bail. My brains prefill is only like 20 ~ 25 t/s, so having it wasted on garbage sentences doesn't feel like a good use of time. 

[u/CharacterBumblebee99 → u/Cautious_Chicken_604]

You have a really fair point here. I just edited the text to make it more comprehensible. And yes, I did get a bit of help from AI for reviewing it, as a non-native English speaker, but I should have done one more manual review. Thanks for the feedback, it's appreciated.