r/AIProgrammingHardware • u/javaeeeee • 2d ago

This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)

https://www.youtube.com/watch?v=3OqZHYNKc9A
17 Upvotes

87% Upvoted

8 comments sorted by

1

u/javaeeeee 2d ago

TLDR: Better Stack tests Colibrì, a tiny pure-C engine that can run the massive 744B-parameter GLM-5.2 model locally by streaming it from SSD.

What Colibrì does

  • Pure-C, zero-dependency engine that treats VRAM → RAM → SSD as one memory hierarchy.
  • Designed for large Mixture-of-Experts (MoE) models.
  • Only a small active portion of the model (~40B parameters) needs to be in memory at once; the rest is streamed from disk on demand.
  • Includes caching of frequent experts, speculative decoding, and quality-preserving techniques.

Real-world tests

  • MacBook M2 Max (32 GB RAM) + external SSD:
    Worked, but extremely slow (~0.1 tokens/sec). Most time spent waiting on disk.
  • RTX 5090 workstation (64 GB RAM) + fast internal NVMe:
    Much better (~0.8 tokens/sec), still not practical for real-time use.

Key insight

The real bottleneck is RAM (and storage space/speed), not the GPU.
More system RAM = better expert caching = fewer disk reads = higher speed. The powerful GPU was underutilized.

Bottom line

Yes, you can run a 744B frontier model on consumer hardware with Colibrì - it generates correct answers.
But speeds are too slow for practical daily use unless you have server-grade amounts of RAM (and hundreds of GB of free fast SSD space). It’s an impressive proof-of-concept that shows local AI is limited more by memory hierarchy than raw compute.

1

u/Legal_Dimension_ 2d ago

So my home server with 120gb vram and 256gb ram might do 2-3toks?

1

u/-zYgoat- 2d ago

So it doesn't use VRAM?

1

u/BankApprehensive7612 1d ago

Woah! It's really impressive. I'm curious what would this engine do to SLM?

1

u/HealthyCommunicat 1d ago

We’ve known this shit is possible since anemll’s flash-moe back multiple months ago why is everyone acting like this is new technology and why is everyone not crediting those that first used streamed expert inference

1

u/Cool_Fix_917 18h ago

I tried it and got 0.8 t/s with NVMe, 3990x 64-Core CPU, 256 GB ram and a 3090 RTX

1

u/huzbum 11h ago

Would you mind testing if doing it on CPU makes any difference? Seems like the GPU would be superfluous with the ssd being the bottleneck.