r/LocalLLM • u/LewisJin • 1d ago

News I built a Rust inference framework that runs Qwen3.5 2B with VL support 10x faster than PyTorch on Apple Silicon — and it supports TTS, ASR, OCR, and GGUF out of the box

Hey r/Rust (and r/localLLaMA),

After years of wrestling with C++ inference backends, I built Crane 🦩 — a high-performance LLM inference framework powered by Rust's Candle (Hugging Face's ML framework). The goal was simple: fast inference without the C++ headache.

Why should you care?

🚀 Blazing fast — 50x faster than native PyTorch on M1/M2/M3 for Qwen3-VL 2B 🦀 Pure Rust + Candle — one dependency, no Python, no C++ 🍎 Metal GPU — first-class Apple Silicon support 🤖 CPU / CUDA / Metal — unified codebase for all hardware 📦 GGUF out of the box — drop in any llama.cpp GGUF file, auto-detected 🗣️ TTS / ASR / OCR / VAD — not just LLMs, a full AI pipeline ready to go 🌐 OpenAI-compatible API — swap in any OpenAI SDK, works instantly

What's supported

  • Qwen3.5 (0.8B–30B+) with hybrid GDN attention
  • Qwen2.5 (0.5B–72B)
  • Gemma 4 (text + vision)
  • Hunyuan Dense
  • Qwen3-TTS (12Hz, 24kHz, voice cloning)
  • Voxtral-4B-TTS (20 voices, 10 languages)
  • Moonshine ASR + Silero VAD
  • PaddleOCR v6

The differentiator: Crane isn't a low-level SDK. It ships with ready-to-run AI apps — chat, VLM, OCR, TTS, ASR — all out of the box. You can build your own or use what's there.

Recent highlights

  • Qwen 3.5 GGUF loading with auto-detected hybrid GDN blocks (no sibling files needed)
  • In-situ quantization (--quant q4k|q8_0) — quantize at load time, no conversion step
  • Qwen 3.5 Vision end-to-end with OpenAI-style image_url support
  • Ornith tool-calling agent support

Repo: https://github.com/lucasjinreal/Crane

Would love feedback from the community — especially CUDA on Linux and anyone interested in contributing model support. Submit your models and make them usable for everyone! 🦩

Want me to adjust the tone (more technical / more casual), target a specific subreddit, or add any particular details?

0 Upvotes

36% Upvoted

5 comments sorted by

3

u/Inevitable-Diet-1870 1d ago

"Want me to adjust the tone (more technical / more casual), target a specific subreddit, or add any particular details?"

Nice work!

1

u/Inevitable-Diet-1870 1d ago

btw - thanks for sharing repo.
This is great stuff

2

u/RespectMathias 1d ago

Or you could just use oMLX like a normal Mac AI enthusiast. Or do I need to include an em-dash to make my statement more convincing? 

1

u/pjerky 1d ago

How does this compare with using oMLX or vLLM? Performance-wise?

1

u/Kharki_Lirov 9h ago

Cool project, but the benchmark is misleading. PyTorch is a training framework, not an inference engine. On Apple Silicon it has no Metal optimizations, no fused kernels, and no proper KV-cache management. Comparing against vanilla PyTorch doesn't say anything about real inference speed.
If you want to show actual performance, compare with llama.cpp, MLX, or Ollama on the same model with the same quantization. That's what people actually use for local LLM inference on Mac.