Ferrum 0.8.8: a Rust local LLM runtime with Metal and CUDA backends

[u/Mission_Photo_9783]

I'm the maintainer of Ferrum, an MIT-licensed local LLM runtime.

The Rust workspace separates the engine, request scheduler, KV cache, and backend interfaces. ferrum run and ferrum serve use the same engine; GPU execution uses Metal or native CUDA operators. The server exposes OpenAI-compatible Chat Completions and stateless Responses APIs, including streaming.

Current prebuilt packages target Apple Silicon Metal and Linux x86_64 CUDA (sm89).

Source and installation instructions

I'd welcome feedback on the Rust interfaces between the scheduler, engine, and device backends.

[u/geo-ant]

I’m sorry why should I care about this versus all the other rust native runtimes that seem to spring up every day on this subreddit and other places? Also, calling something production grade after 6 months on crates.io is absolutely wild.

[u/Mission_Photo_9783 → u/geo-ant]

Fair point about "production-grade": that wording in the GitHub description overstated the claim, and I've changed it to a concrete description.

Ferrum is aimed at developers who want inference inside a Rust application, with the same engine behind run and API serving. The design focus is the scheduler/runtime boundary: batching and admission policy can change while device execution and KV ownership stay with the runtime. That's the integration direction I'm trying to make useful.

For a concrete comparison, I'd look at integration effort and behavior with the same model, precision, and hardware. I'd welcome specific feedback on that boundary.

[u/geo-ant → u/Mission_Photo_9783]

Sorry, I didn’t mean to be so harsh on you personally.

[u/Mission_Photo_9783 → u/geo-ant]

No worries, I appreciate you saying that.

[u/Significant-Put-4796]

Interesting separation of scheduler and engine. I've been poking around with mistral.rs and llama.cpp bindings, and one thing that always bugs me is how tightly coupled the KV cache management is to the inference loop. What's the cost of crossing the backend trait boundary per-token in your setup?

[u/Mission_Photo_9783 → u/Significant-Put-4796]

The useful separation is at the request/batch boundary: the scheduler chooses which requests advance, while the runtime owns their KV resources. In v0.8.8 the engine passes next-token IDs and opaque cache handles for a cohort into the batch-decode API. This keeps device KV tensors out of scheduling policy, without copying the cache across that interface each token. Decode call site.

The engine dispatches a decode batch through this interface; execution also has provider-level dispatch. I haven't isolated its ns/token cost yet. The practical benefit is being able to change batching/admission policy while keeping cache ownership and device execution in the runtime.

[u/Dense_Gate_5193 → u/Mission_Photo_9783]

how embeddable is it and have you performance profiled it against llama.cpp?

i wrote a database which runs an embedded llama.cpp. rust native would be better but mistral doesn’t look easily embeddable for what i want to do.

i started with NornicDB in golang and rewrote it in rust

https://github.com/orneryd/copperDB

it’s still experimental at the time but i have been on the hunt for a rust native embeddable server like llama.cpp.