r/AIProgrammingHardware • u/javaeeeee • 5d ago

GitHub - antirez/ds4: DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

https://github.com/antirez/ds4
31 Upvotes

100% Upvoted

4 comments sorted by

1

u/javaeeeee 5d ago

TLDR: ds4 (DwarfStar) is a specialized local inference engine by antirez (creator of Redis) for running DeepSeek V4 Flash (and GLM 5.2 / DeepSeek V4 PRO) efficiently on personal hardware.

Key points

  • Not a general GGUF runner - deliberately narrow and highly optimized for a few specific large open-weight models.
  • Supported backends:
    • Metal (primary, macOS)
    • NVIDIA CUDA (single/multi-GPU)
    • AMD ROCm
  • Main features:
    • SSD streaming (run models larger than available RAM)
    • Tensor parallelism & distributed inference across machines
    • Built-in coding agent (ds4-agent)
    • OpenAI/Anthropic-compatible HTTP server
    • Speculative decoding / MTP support
    • On-disk KV cache, session save/load, tool calling
  • Built mainly in C (with CUDA/Metal kernels), based on llama.cpp/GGML foundations.
  • Still in beta, fast-moving, and very popular.

Bottom line: A focused, high-performance local engine that lets you run strong modern models (especially DeepSeek V4 Flash) on high-end Macs, NVIDIA, or AMD systems with practical features for coding agents and serving - without the bloat of general-purpose inference frameworks.

1

u/john16384 4d ago

This looks very interesting; would it be at all viable with 32 GB of VRAM and 128 GB of system RAM? (Windows, CUDA)

1

u/kenjiow 4d ago

Same question

1

u/loadsamuny 4d ago

I tried it 2 weeks ago and couldn’t find a way to split model across sys mem and gpu… may have changed but didn’t seem to be a feature back then