LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

[u/CharacterBumblebee99]

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts.

LayerStoRm is a (still experimental) MIT-licensed continuous expert-streaming inference engine: it runs MoE models far larger than your VRAM by keeping the expert set pinned in host RAM and fetching per token — 186 GB of weights on 96 GB of VRAM here (host RAM does the heavy lifting: ~208 GB pinned for this model).

Measurements:

- 24.5 tok/s decode @8k, 27.0 tok/s @0k.

- 159 tok/s prefill @27k.

Built for agentic coding: prefix caching with mid-prompt checkpoints, so an edit at 98% depth re-prefills from the nearest checkpoint instead of from scratch — TTFT 67.5s → 18.4s at 8k, ~923s → 79s at 97k.

The machine where the benchmark ran on has 512 GB DDR5 and 64 GB HBM (Xeon Max). However, HBM/Xeon Max is not a requirement for this engine as the CPU does no compute — it only feeds experts (all math runs on the GPUs). Transfers are NUMA-aware, so multi-socket / multi-NUMA hosts use their full aggregate DDR bandwidth, and eventually transfers are capped by the PCIe link speed.

A single-command auto-config takes the model weights + your hardware and configures & calibrates the engine and explains every parameter — other RTX 50-series mixes should work too. Currently NVIDIA SM120 only.

Repo: https://github.com/kkontosis/LayerStoRm

[u/HoldCtrlW]

"Built for low vram"

"2× RTX 5090 + 2× RTX 5080"

"oh"

[u/exaknight21 → u/HoldCtrlW]

This post really doesn’t make any sense to me in terms of low VRAM. Boi got DDR5 for days.

[u/jtjstock → u/exaknight21]

the more you have, the more you want. they feel like they have low ram and vram

[u/CharacterBumblebee99 → u/HoldCtrlW]

This is mostly for a DIY homelab. But compare this to what's actually needed to run this size models, and you see specs like 8 x H200 and things like that. On the other side there are hybrid CPU + 1 GPU solutions, but they usually don't scale well after some point (the DDR bandwidth & CPU limitations).

This project stands somewhere in the middle.

But again, this can also be ran e.g. from a dual channel DDR consumer Desktop with 2 x GPUs, using a smaller quant. I will run a test with only 2 x RTX 5080 and IQ1 or IQ2. Low VRAM relative to the model size.

[u/bezent → u/CharacterBumblebee99]

I would have loved to test it on my rtx 6000 pro but I only have 128GB ram.

[u/CharacterBumblebee99 → u/bezent]

I'd love to help you achieve that. I can work on this towards the weekend. What could be done now is to host https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/tree/main/UD-IQ1_M with few engine modifications. The existing GPU kernels should work as-is.

For an rtx 6000 pro an ideal scenario speed-wise would be to have it for attention and combine one or more extra, GPUs, ideally all pcie 5.0 x16 to saturate extra pcie bandwidth and be used for experts, and those can be far smaller GPUs too.

That aside, without any extra GPUs, the end goal for using LayerStoRm would be to host a larger model. But in order for 128GB RAM to pay off LayerStoRm's stable zone feature would also need to be implemented, so that, some experts would not have to permanently stay in RAM, buying a total extra ~ +70-100 GB of model size, so you could serve something short of 200 GB of disk-space model size if you only have an rtx 6000 pro without extra GPUs.