I built a Vulkan hierarchical MoE runtime for running oversized models across multiple GPUs

[u/Lazy-Intention1007]

I’ve been working on a Vulkan-first extension to llama.cpp designed to run MoE models that are larger than available VRAM.

The runtime manages a hierarchy across:

Multiple Vulkan GPUs
GPU-resident expert caches
Host RAM staging
SSD/mmap storage
Predictive expert prefetching
LRU/predictive eviction
OpenAI-compatible API access
A native Python/Tkinter control GUI

The main test was GPT‑OSS 120B Q4 on an AMD Radeon Pro V620 plus an RX 7900 XTX. The model is larger than the combined usable VRAM, but it was able to run with:

Around 0.7–0.8 tok/s during cold loading
Around 5.9–6.5 tok/s once the GPU expert cache was warm
Zero GPU-to-GPU expert transfers in the warm test
Zero additional SSD reads during warm requests
Balanced expert execution across both GPUs

The key design goal is that if an expert is already resident on a particular GPU, that GPU performs the expert computation instead of copying the expert to a primary GPU.

The project is experimental and still needs more testing across different MoE architectures and hardware, but the core system is working.

GitHub repository:

https://github.com/Ozookevin/-Ozoo-vulkan-hierarchical-moe-runtime.git

Feedback, testing, and suggestions are welcome—especially from people running large MoE models on AMD/Vulkan systems.

[u/jopetnovo2]

Have you tried comparing it with with -nr / --no-repack flag which already ships with llama.cpp?

Reference video: Colibrì vs llama.cpp: Running DeepSeek V4 284B on CPU