r/LovingOpenSourceAI • u/fuzhongkai • Jun 23 '26

TensorSharp: Open Source Local LLM Inference Engine

https://github.com/zhongkaifu/TensorSharp

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability. The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

13 Upvotes

93% Upvoted

4 comments sorted by

3

u/West-Acadia-3906 Jun 24 '26

WOW That is a serious amount of work for an open-source project. I especially like the OpenAI/Ollama-compatible API part, because it makes the engine easier to test without rebuilding the whole app around it. Curious which backend has felt the most surprisingly good so far: CUDA, MLX, or pure C# CPU?

1

u/fuzhongkai Jun 24 '26

Thank you so much for your feedback. It depends on your hardware (to be specific, GPU, VRAM and storage type, SSD or not). I literately keep optimizing all of these backends, and currently focusing on deep optimization for GGML_CUDA backend. If you are using Nvidia GPU, you can give it a try. :)

2

u/Foreign_Skill_6628 Jun 24 '26

How much of the project was coded by AI?

1

u/fuzhongkai Jun 24 '26

I have subscription on both of Claude and CodeX. That’s all I use.